Why serving architecture affects latency and cost in MLOps - Why It Works

Introduction

Serving architecture is how your machine learning model is made available to users or applications. The way you set it up changes how fast responses come back (latency) and how much money you spend (cost).

When you want your app to respond quickly to user requests with predictions

When you need to handle many prediction requests at the same time without delays

When you want to save money by using resources efficiently during low traffic times

When you must balance between fast responses and keeping cloud costs low

When you plan to scale your model serving as your user base grows

Commands

This command deploys your model serving setup on Kubernetes. It starts the pods that will handle prediction requests.

Terminal

kubectl apply -f model-serving-deployment.yaml

Expected Output

deployment.apps/model-serving created

This command checks the status of the pods to make sure your model serving is running.

Terminal

kubectl get pods

Expected Output

NAME                                READY   STATUS    RESTARTS   AGE

model-serving-deployment-5d7f9c7d9f-abc12   1/1     Running   0          30s

This command sends a prediction request to your model server to test latency and response.

Terminal

curl -X POST http://localhost:8501/v1/models/my-model:predict -d '{"instances": [1.0, 2.0, 5.0]}'

Expected Output

{"predictions": [0.75]}

This command increases the number of pods serving your model to handle more requests and reduce latency.

Terminal

kubectl scale deployment model-serving-deployment --replicas=5

Expected Output

deployment.apps/model-serving-deployment scaled

→

--replicas - Sets the number of pod instances to run

This command shows the current number of pods and their status after scaling.

Terminal

kubectl get deployment model-serving-deployment

Expected Output

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE

model-serving-deployment   5/5     5            5           2m

Key Concept

If you remember nothing else from this pattern, remember: the way you set up your model serving affects how fast predictions come back and how much you pay for computing resources.

Common Mistakes

Running only one pod for model serving under heavy load

This causes slow responses because one pod cannot handle many requests at once, increasing latency.

Scale the deployment to multiple pods to share the load and reduce latency.

Keeping many pods running even when traffic is low

This wastes resources and increases cost because you pay for unused computing power.

Use autoscaling to adjust the number of pods based on traffic automatically.

Deploying the model on a large machine without considering request volume

You might pay more than needed if the traffic is low, or get slow responses if traffic is high and resources are insufficient.

Match your serving resources to your traffic patterns and scale as needed.

Summary

Deploy your model serving using Kubernetes to make it available for predictions.

Check pod status to ensure your model is running and ready to serve.

Send test prediction requests to measure latency and correctness.

Scale the number of pods to handle more requests and reduce latency.

Balance scaling to avoid unnecessary costs during low traffic.

Practice

1. Which serving architecture typically offers the lowest latency for model predictions?

easy

A. Offline serving

B. Batch serving

C. Edge serving

D. Cloud batch processing

Solution

1. Step 1: Understand latency in serving architectures

   Latency means the delay before a prediction is returned. Edge serving places the model close to the user, reducing delay.

2. Step 2: Compare architectures

   Batch serving processes data in groups and is slower. Edge serving is designed for fast responses near the user.

3. Final Answer:

   Edge serving -> Option C

4. Quick Check:

   Lowest latency = Edge serving [OK]

Hint: Edge serving is closest to users, so fastest response [OK]

Common Mistakes:

• Confusing batch serving as low latency
• Thinking cloud batch is fastest
• Ignoring edge location benefits

2. Which statement correctly describes batch serving in ML model deployment?

easy

A. Batch serving provides real-time predictions with high cost.

B. Batch serving processes data in groups and is usually cheaper but slower.

C. Batch serving always runs on edge devices.

D. Batch serving requires no compute resources.

Solution

1. Step 1: Define batch serving

   Batch serving processes multiple data points together, not one by one, which saves cost but adds delay.

2. Step 2: Evaluate options

   Batch serving processes data in groups and is usually cheaper but slower. correctly states batch serving is cheaper but slower. Other options are incorrect or unrealistic.

3. Final Answer:

   Batch serving processes data in groups and is usually cheaper but slower. -> Option B

4. Quick Check:

   Batch serving = cheaper, slower [OK]

Hint: Batch = groups, cheaper but slower [OK]

Common Mistakes:

• Thinking batch serving is real-time
• Assuming batch runs on edge devices
• Believing batch needs no compute

3. Given a model deployed with online serving and another with batch serving, which output best describes their latency and cost?

medium

A. Online serving: low latency, high cost; Batch serving: high latency, low cost

B. Online serving: high latency, low cost; Batch serving: low latency, high cost

C. Both have similar latency and cost

D. Online serving is always cheaper than batch serving

Solution

1. Step 1: Recall characteristics of online and batch serving

   Online serving provides predictions immediately (low latency) but requires more resources (high cost). Batch serving delays predictions but is cheaper.

2. Step 2: Match options to characteristics

   Online serving: low latency, high cost; Batch serving: high latency, low cost correctly matches low latency and high cost to online serving, and high latency and low cost to batch serving.

3. Final Answer:

   Online serving: low latency, high cost; Batch serving: high latency, low cost -> Option A

4. Quick Check:

   Online = fast & costly, Batch = slow & cheap [OK]

Hint: Online = fast+costly, Batch = slow+cheap [OK]

Common Mistakes:

• Swapping latency and cost roles
• Assuming both have same cost
• Thinking batch is faster

4. A team deployed a model using edge serving but notices high latency and cost. What is the most likely cause?

medium

A. Edge serving always causes high latency and cost

B. Batch processing was mistakenly used instead of edge serving

C. The model is deployed in a cloud data center far from users

D. The model is too large to run efficiently on edge devices

Solution

1. Step 1: Understand edge serving constraints

   Edge devices have limited resources. Large models can slow down processing and increase cost.

2. Step 2: Analyze options

   The model is too large to run efficiently on edge devices explains the likely cause. Batch processing was mistakenly used instead of edge serving is incorrect because batch serving is different. The model is deployed in a cloud data center far from users describes cloud serving, not edge. Edge serving always causes high latency and cost is false.

3. Final Answer:

   The model is too large to run efficiently on edge devices -> Option D

4. Quick Check:

   Large model on edge = high latency/cost [OK]

Hint: Large models slow edge devices, raising latency and cost [OK]

Common Mistakes:

• Confusing edge with cloud serving
• Assuming edge always has high latency
• Mixing batch and edge serving

5. A company wants to minimize prediction latency for users worldwide but has a limited budget. Which serving architecture balances latency and cost best?

hard

A. Combine edge serving for critical regions and batch serving elsewhere

B. Deploy models only in a central cloud data center

C. Use batch serving exclusively for all predictions

D. Deploy large models on every user device

Solution

1. Step 1: Analyze latency and cost trade-offs

   Central cloud has higher latency for distant users. Batch serving is cheap but slow. Edge serving is fast but costly.

2. Step 2: Evaluate hybrid approach

   Combining edge serving in key regions reduces latency where needed, while batch serving elsewhere controls costs.

3. Final Answer:

   Combine edge serving for critical regions and batch serving elsewhere -> Option A

4. Quick Check:

   Hybrid edge + batch balances latency and cost [OK]

Hint: Hybrid edge and batch serving balances speed and cost [OK]

Common Mistakes:

• Choosing only cloud causing high latency
• Using batch only causing slow responses
• Deploying large models on all devices is costly