Emerging trends (smaller models, edge AI) in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Emerging trends (smaller models, edge AI)

Which metric matters for Emerging Trends (smaller models, edge AI) and WHY

For smaller models and edge AI, key metrics include model size, latency, and energy efficiency. Accuracy remains important but must be balanced with these constraints. We want models that are small and fast enough to run on devices like phones or sensors, while still making good predictions.

Confusion matrix example for edge AI classification

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 40 | False Negative (FN): 10 |
      | False Positive (FP): 5 | True Negative (TN): 45 |

      Total samples = 40 + 10 + 5 + 45 = 100

      Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
      Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
      Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85
    

This shows a balanced model that works well on-device with good precision and recall.

Precision vs Recall tradeoff in edge AI

Imagine a smart home camera detecting intruders. High precision means it rarely mistakes a family member for an intruder (few false alarms). High recall means it catches almost all real intruders (few misses). On edge devices, we must balance these because complex models that improve recall might be too slow or large.

Choosing the right tradeoff depends on what matters more: avoiding false alarms (precision) or catching every threat (recall).

Good vs Bad metric values for smaller models and edge AI

Good: Accuracy around 85%+, precision and recall balanced above 80%, model size under 10MB, latency under 100ms, and low power use.

Bad: Accuracy below 70%, very low recall (missing many cases), model size too large to run on device, or latency causing slow responses.

Common pitfalls in evaluating smaller models and edge AI

• Ignoring latency and size: A model with great accuracy but too big or slow is unusable on edge.
• Overfitting: Small models can overfit if not trained well, leading to poor real-world results.
• Data leakage: Using test data during training inflates accuracy falsely.
• Accuracy paradox: High accuracy on imbalanced data can be misleading if recall or precision is low.

Self-check question

Your edge AI model has 98% accuracy but only 12% recall on detecting faults. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most faults, which is critical to detect. High accuracy can be misleading if the data is imbalanced with many normal cases.

Key Result

For smaller models and edge AI, balance accuracy with model size, latency, and energy use to ensure practical, effective deployment.

Practice

1. What is a key benefit of smaller AI models in devices like smartphones?

easy

A. They need large servers to work

B. They require constant internet connection

C. They run faster and use less memory

D. They always produce more accurate results

Solution

1. Step 1: Understand smaller AI models

   Smaller AI models are designed to be lightweight, so they use less memory and compute power.

2. Step 2: Identify benefits for smartphones

   Because phones have limited memory and processing power, smaller models help them run AI tasks faster and more efficiently.

3. Final Answer:

   They run faster and use less memory -> Option C

4. Quick Check:

   Smaller models = faster, less memory [OK]

Hint: Smaller models save memory and speed up devices [OK]

Common Mistakes:

• Thinking smaller models need big servers
• Assuming smaller models require internet
• Believing smaller models always improve accuracy

2. Which code snippet correctly shows a simple way to run AI on an edge device?

easy

A. model = download_model('cloud_model')
result = model.predict_online(input_data)

B. model = load_model('big_model.h5')
result = model.train(input_data)

C. model = load_model('small_model.tflite')
result = model.upload(input_data)

D. model = load_model('small_model.tflite')
result = model.predict(input_data)

Solution

1. Step 1: Identify edge AI code

   Edge AI runs AI models locally, so loading a small model like 'small_model.tflite' fits this.

2. Step 2: Check correct method usage

   Using predict runs inference, which is typical for edge AI. Training or uploading is not common on edge devices.

3. Final Answer:

   model = load_model('small_model.tflite') result = model.predict(input_data) -> Option D

4. Quick Check:

   Load small model + predict locally = edge AI [OK]

Hint: Edge AI loads small models and predicts locally [OK]

Common Mistakes:

• Using training instead of prediction on edge
• Downloading models from cloud during inference
• Uploading data instead of predicting locally

3. Given this Python code simulating edge AI inference, what is the printed output?

class SmallModel:
    def predict(self, x):
        return x * 2

model = SmallModel()
result = model.predict(5)
print(result)

medium

A. 10

B. 5

C. 25

D. Error

Solution

1. Step 1: Understand the predict method

   The method multiplies input x by 2 and returns it.

2. Step 2: Calculate the output for input 5

   5 * 2 = 10, so result is 10.

3. Final Answer:

   10 -> Option A

4. Quick Check:

   5 times 2 equals 10 [OK]

Hint: Multiply input by 2 as per predict method [OK]

Common Mistakes:

• Confusing multiplication with addition
• Expecting input as output
• Assuming code causes error

4. This code tries to run AI on an edge device but has an error. What is the problem?

input_data = [1, 2, 3]
model = load_model('small_model.tflite')
result = model.train(input_data)
print(result)

medium

A. The model file name is incorrect

B. Edge devices usually do not train models, only predict

C. The print statement is missing parentheses

D. The input_data variable is not defined

Solution

1. Step 1: Understand edge AI capabilities

   Edge AI devices typically run inference (predict), not training, because training needs more resources.

2. Step 2: Identify incorrect method usage

   Calling train on the model is incorrect for edge AI; it should be predict.

3. Final Answer:

   Edge devices usually do not train models, only predict -> Option B

4. Quick Check:

   Edge AI = predict, not train [OK]

Hint: Edge AI predicts, does not train models [OK]

Common Mistakes:

• Thinking edge devices can train models
• Assuming file name causes error
• Ignoring method misuse

5. You want to build a voice assistant that works offline on a smartwatch. Which approach best fits this edge AI trend?

hard

A. Use a small AI model running locally on the watch

B. Use no AI and only pre-recorded responses

C. Stream audio to a server for processing

D. Use a large cloud AI model accessed via internet

Solution

1. Step 1: Understand offline edge AI needs

   Offline means no internet, so AI must run locally on the device.

2. Step 2: Choose model size for smartwatch

   Smartwatches have limited memory and power, so a small AI model is best to run locally.

3. Final Answer:

   Use a small AI model running locally on the watch -> Option A

4. Quick Check:

   Offline + smartwatch = small local model [OK]

Hint: Offline device needs small local AI model [OK]

Common Mistakes:

• Choosing cloud models needing internet
• Streaming audio defeats offline goal
• Ignoring device memory limits