Why architecture choices affect scalability in Prompt Engineering / GenAI - Why Metrics Matter

Metrics & Evaluation - Why architecture choices affect scalability

Which metric matters for this concept and WHY

When we talk about scalability in machine learning models, the key metrics to watch are throughput (how many predictions the model can make per second) and latency (how fast a single prediction is made). These metrics show if the model can handle more data or users without slowing down. Architecture choices affect these because some designs use more resources or take longer to compute, which limits how well the model scales.

Confusion matrix or equivalent visualization (ASCII)

Throughput (predictions/sec):
+-----------------+-----------------+
| Architecture A  |  1000 preds/sec |
| Architecture B  |   200 preds/sec |
+-----------------+-----------------+

Latency (ms per prediction):
+-----------------+-----------------+
| Architecture A  |       5 ms      |
| Architecture B  |      25 ms      |
+-----------------+-----------------+

This simple table shows how different architectures can handle different speeds and loads.

Precision vs Recall (or equivalent tradeoff) with concrete examples

In scalability, the tradeoff is often between model complexity and speed. A very complex model might be more accurate but slower, hurting throughput and latency. A simpler model runs faster but might lose some accuracy. For example, a deep neural network with many layers can catch subtle patterns but takes longer to run. A smaller model runs quickly but might miss details.

Choosing architecture means balancing these: do you want the model to be very accurate but slower, or fast but less detailed? This balance affects how well the system scales when many users or data points come in.

What "good" vs "bad" metric values look like for this use case

Good scalability metrics:

• High throughput (e.g., thousands of predictions per second)
• Low latency (e.g., under 10 milliseconds per prediction)
• Stable performance as load increases (no big slowdowns)

Bad scalability metrics:

• Low throughput (e.g., less than 100 predictions per second)
• High latency (e.g., over 100 milliseconds per prediction)
• Performance drops sharply when more data or users arrive

Good architecture choices help keep metrics in the good range.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

Common pitfalls when evaluating scalability include:

• Ignoring latency: A model might be accurate but too slow to use in real time.
• Overfitting to small data: Complex architectures might perform well on test data but fail to scale with more data.
• Resource bottlenecks: Not considering memory or CPU limits can cause crashes or slowdowns.
• Data leakage: If the model accidentally sees future data during training, it may seem fast and accurate but fail in real use.

Self-check question

Your model has 98% accuracy but takes 500 milliseconds per prediction and can only handle 50 predictions per second. Is it good for a real-time app with thousands of users? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the latency and throughput are too slow for real-time use with many users. The architecture needs to be changed to improve speed and scalability.

Key Result

Architecture choices impact throughput and latency, which are key to model scalability.