Why architecture choices affect scalability in Prompt Engineering / GenAI - Why Metrics Matter

Metrics & Evaluation - Why architecture choices affect scalability

Which metric matters for this concept and WHY

When we talk about scalability in machine learning models, the key metrics to watch are throughput (how many predictions the model can make per second) and latency (how fast a single prediction is made). These metrics show if the model can handle more data or users without slowing down. Architecture choices affect these because some designs use more resources or take longer to compute, which limits how well the model scales.

Confusion matrix or equivalent visualization (ASCII)

Throughput (predictions/sec):
+-----------------+-----------------+
| Architecture A  |  1000 preds/sec |
| Architecture B  |   200 preds/sec |
+-----------------+-----------------+

Latency (ms per prediction):
+-----------------+-----------------+
| Architecture A  |       5 ms      |
| Architecture B  |      25 ms      |
+-----------------+-----------------+

This simple table shows how different architectures can handle different speeds and loads.

Precision vs Recall (or equivalent tradeoff) with concrete examples

In scalability, the tradeoff is often between model complexity and speed. A very complex model might be more accurate but slower, hurting throughput and latency. A simpler model runs faster but might lose some accuracy. For example, a deep neural network with many layers can catch subtle patterns but takes longer to run. A smaller model runs quickly but might miss details.

Choosing architecture means balancing these: do you want the model to be very accurate but slower, or fast but less detailed? This balance affects how well the system scales when many users or data points come in.

What "good" vs "bad" metric values look like for this use case

Good scalability metrics:

• High throughput (e.g., thousands of predictions per second)
• Low latency (e.g., under 10 milliseconds per prediction)
• Stable performance as load increases (no big slowdowns)

Bad scalability metrics:

• Low throughput (e.g., less than 100 predictions per second)
• High latency (e.g., over 100 milliseconds per prediction)
• Performance drops sharply when more data or users arrive

Good architecture choices help keep metrics in the good range.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

Common pitfalls when evaluating scalability include:

• Ignoring latency: A model might be accurate but too slow to use in real time.
• Overfitting to small data: Complex architectures might perform well on test data but fail to scale with more data.
• Resource bottlenecks: Not considering memory or CPU limits can cause crashes or slowdowns.
• Data leakage: If the model accidentally sees future data during training, it may seem fast and accurate but fail in real use.

Self-check question

Your model has 98% accuracy but takes 500 milliseconds per prediction and can only handle 50 predictions per second. Is it good for a real-time app with thousands of users? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the latency and throughput are too slow for real-time use with many users. The architecture needs to be changed to improve speed and scalability.

Key Result

Architecture choices impact throughput and latency, which are key to model scalability.

Practice

1. Why do architecture choices matter for the scalability of AI systems?

easy

A. Because they control the AI's ability to speak multiple languages

B. Because they decide the color scheme of the AI interface

C. Because they determine how well the system handles more data or users

D. Because they affect the AI's ability to connect to the internet

Solution

1. Step 1: Understand scalability in AI

   Scalability means how well an AI system can grow or handle more data and users without slowing down or failing.

2. Step 2: Link architecture to scalability

   The architecture defines the system's structure and resources, which directly affect its ability to scale efficiently.

3. Final Answer:

   Because they determine how well the system handles more data or users -> Option C

4. Quick Check:

   Architecture affects scalability = Because they determine how well the system handles more data or users [OK]

Hint: Think about growth and handling more users or data [OK]

Common Mistakes:

• Confusing UI design with architecture
• Thinking scalability is about language support
• Assuming internet connection affects scalability

2. Which of the following is the correct way to describe a model architecture that supports scalability?

easy

A. A model that uses fixed-size layers regardless of data size

B. A model that can adjust its layers or parameters based on data volume

C. A model that ignores data size and always uses the same resources

D. A model that only works on small datasets without changes

Solution

1. Step 1: Identify scalable architecture traits

   Scalable models can adjust resources like layers or parameters to handle more data efficiently.

2. Step 2: Compare options

   Only A model that can adjust its layers or parameters based on data volume describes a model that adapts to data volume, which supports scalability.

3. Final Answer:

   A model that can adjust its layers or parameters based on data volume -> Option B

4. Quick Check:

   Adaptive model = A model that can adjust its layers or parameters based on data volume [OK]

Hint: Look for adaptability to data size in the description [OK]

Common Mistakes:

• Choosing fixed-size models as scalable
• Ignoring the need to adjust resources
• Confusing scalability with model accuracy

3. Consider this Python code snippet for a simple AI model architecture choice:

class SimpleModel:
    def __init__(self, size):
        self.size = size
    def process(self, data):
        return [x * self.size for x in data]

model_small = SimpleModel(2)
model_large = SimpleModel(10)
data = [1, 2, 3]

output_small = model_small.process(data)
output_large = model_large.process(data)
print(output_small, output_large)

What will be the printed output?

medium

A. [2, 4, 6] [10, 20, 30]

B. [1, 2, 3] [1, 2, 3]

C. [2, 4, 6] [2, 4, 6]

D. Error due to missing method

Solution

1. Step 1: Understand the model's process method

   The process method multiplies each data element by the model's size attribute.

2. Step 2: Calculate outputs for both models

   For model_small (size=2), output is [1*2, 2*2, 3*2] = [2, 4, 6]. For model_large (size=10), output is [1*10, 2*10, 3*10] = [10, 20, 30].

3. Final Answer:

   [2, 4, 6] [10, 20, 30] -> Option A

4. Quick Check:

   Multiplying data by size = [2, 4, 6] [10, 20, 30] [OK]

Hint: Multiply each data item by model size [OK]

Common Mistakes:

• Confusing the size attribute with data values
• Assuming process method modifies data in place
• Expecting an error due to method misunderstanding

4. The following code tries to create a scalable AI model but has a bug:

class ScalableModel:
    def __init__(self, layers):
        self.layers = layers
    def forward(self, data):
        for i in range(self.layers):
            data = data + i
        return data

model = ScalableModel(3)
result = model.forward(5)
print(result)

What is the error and how to fix it?

medium

A. No error; output is 11

B. Error: Adding int to int is invalid; fix by converting i to string

C. Error: data should be a list for addition; fix by initializing data as list

D. Error: The loop should multiply data, not add

Solution

1. Step 1: Analyze the forward method

   The method adds i (0,1,2) to data (starting at 5) in each loop iteration.

2. Step 2: Calculate the final result

   5 + 0 = 5, then 5 + 1 = 6, then 6 + 2 = 8. So the final result is 8, not 11.

3. Step 3: Check for errors

   Adding integers is valid in Python, so no error occurs.

4. Final Answer:

   No error; output is 8 -> Option A

5. Quick Check:

   Integer addition valid, output 8 = No error; output is 11 [OK]

Hint: Add integers stepwise to find output [OK]

Common Mistakes:

• Expecting type error when adding ints
• Miscomputing the sum as 11 instead of 8
• Thinking data must be a list

5. You want to design an AI system that can handle a growing number of users without slowing down. Which architecture choice best supports this goal?

hard

A. Use a model that only works on a fixed dataset size

B. Use a small fixed-size model that never changes

C. Use a single large model that processes all data sequentially

D. Use a modular architecture that can add more processing units as needed

Solution

1. Step 1: Understand scalability for many users

   Handling more users means the system must grow resources or distribute work to avoid slowdowns.

2. Step 2: Evaluate architecture options

   A modular architecture allows adding processing units as demand grows, supporting scalability better than fixed or single large models.

3. Final Answer:

   Use a modular architecture that can add more processing units as needed -> Option D

4. Quick Check:

   Modular, expandable design = Use a modular architecture that can add more processing units as needed [OK]

Hint: Choose expandable, modular designs for growth [OK]

Common Mistakes:

• Picking fixed-size models thinking they are faster
• Choosing single large models that bottleneck
• Ignoring the need to add resources dynamically