Why serving architecture affects latency and cost in MLOps - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

How does a centralized serving architecture impact latency?

Imagine a machine learning model served from a single central server to users worldwide. What is the main effect of this setup on latency?

A Latency increases for distant users due to longer network travel times.

B Latency decreases because the server is optimized for all users.

C Latency stays the same regardless of user location.

D Latency is eliminated by using a central server.

💡 Hint

Think about how distance affects travel time for data packets.

A centralized server means users far away have longer network paths, increasing latency.

🧠 Conceptual

intermediate

Why does distributed serving architecture reduce latency?

What is the main reason a distributed serving architecture can reduce latency for users?

A It uses more powerful servers in one location.

B It compresses data to speed up processing.

C It caches results on the client device.

D It places model servers closer to users, reducing network travel time.

💡 Hint

Consider how physical proximity affects data travel time.

Distributed serving places servers near users, so data travels shorter distances, lowering latency.

💻 Command Output

advanced

Cost impact of autoscaling in serving architecture

Given this autoscaling configuration snippet for a model serving deployment, what is the expected effect on cost when traffic spikes?

MLOps

autoscaling:
  min_replicas: 1
  max_replicas: 10
  target_cpu_utilization_percentage: 50

A Cost increases as replicas scale up to handle traffic spikes.

B Cost decreases because fewer replicas are used during spikes.

C Cost stays fixed regardless of traffic changes.

D Cost is eliminated by autoscaling.

💡 Hint

Autoscaling adds more replicas when CPU usage is high.

Autoscaling increases replicas during high traffic, raising cost due to more resources used.

❓ Troubleshoot

advanced

Troubleshooting high latency despite distributed serving

A distributed serving system still shows high latency for some users. Which issue is the most likely cause?

A Autoscaling is disabled.

B Too many replicas are running, causing overload.

C Network congestion between user and nearest server.

D Model size is too small to process requests quickly.

💡 Hint

Think about factors outside the server that affect latency.

Even with distributed servers, network congestion can cause delays for users.

✅ Best Practice

expert

Choosing serving architecture to balance cost and latency

Which serving architecture best balances low latency and controlled cost for a global user base with variable traffic?

A Use a single powerful central server running at full capacity all the time.

B Use distributed servers with autoscaling to add replicas only when needed.

C Use distributed servers without autoscaling, always running max replicas.

D Use client-side model inference to eliminate servers.

💡 Hint

Consider how autoscaling helps manage cost during low traffic.

Distributed servers reduce latency by proximity; autoscaling controls cost by adjusting replicas to traffic.