Load balancing for AI services in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of load balancing in AI services?

Load balancing helps AI services handle many requests efficiently. What is its main goal?

A To reduce the accuracy of AI predictions to speed up processing

B To increase the size of the AI model automatically during high demand

C To evenly distribute incoming requests across multiple AI servers to prevent overload

D To store all AI data in a single server for faster access

💡 Hint

Think about how to avoid one server getting too busy while others are idle.

Load balancing spreads requests evenly to avoid any single server becoming a bottleneck, ensuring smooth and fast AI service.

❓ Predict Output

intermediate

What output does this load balancing code produce?

Consider this Python code simulating round-robin load balancing for AI requests:

servers = ['AI1', 'AI2', 'AI3']
requests = 5
assignments = []
for i in range(requests):
    server = servers[i % len(servers)]
    assignments.append(server)
print(assignments)

What is the printed output?

Prompt Engineering / GenAI

servers = ['AI1', 'AI2', 'AI3']
requests = 5
assignments = []
for i in range(requests):
    server = servers[i % len(servers)]
    assignments.append(server)
print(assignments)

A ['AI1', 'AI1', 'AI1', 'AI1', 'AI1']

B ['AI3', 'AI2', 'AI1', 'AI3', 'AI2']

C ['AI2', 'AI3', 'AI1', 'AI2', 'AI3']

D ['AI1', 'AI2', 'AI3', 'AI1', 'AI2']

💡 Hint

Look at how the modulo operator cycles through the server list.

The modulo (%) operator cycles through the servers list indices 0,1,2 repeatedly, assigning requests in order.

❓ Hyperparameter

advanced

Which hyperparameter affects load balancing efficiency in AI model serving?

When deploying AI models behind a load balancer, which hyperparameter most directly impacts how well the load is balanced?

A Batch size of requests processed by each AI server

B Learning rate of the AI model during training

C Number of layers in the AI model architecture

D Dropout rate used in the AI model

💡 Hint

Think about what controls how many requests a server handles at once.

Batch size controls how many requests a server processes together, affecting throughput and load distribution efficiency.

❓ Metrics

advanced

Which metric best indicates load balancing effectiveness in AI services?

You monitor AI servers behind a load balancer. Which metric best shows if load balancing is working well?

A Total number of AI model parameters

B Variance in CPU usage across all AI servers

C Training accuracy of the AI model

D Size of the AI model file on disk

💡 Hint

Good load balancing means servers work evenly. What shows uneven work?

Variance in CPU usage shows how evenly the load is spread. Low variance means balanced load.

🔧 Debug

expert

Why does this AI load balancer code cause uneven request distribution?

Review this Python code snippet for load balancing AI requests:

servers = ['AI1', 'AI2', 'AI3']
requests = 6
assignments = []
for i in range(requests):
    server = servers[(i // 2) % len(servers)]
    assignments.append(server)
print(assignments)

Why does this code cause uneven load distribution?

Prompt Engineering / GenAI

servers = ['AI1', 'AI2', 'AI3']
requests = 6
assignments = []
for i in range(requests):
    server = servers[(i // 2) % len(servers)]
    assignments.append(server)
print(assignments)

A Because each server gets two consecutive requests before switching, causing bursts

B Because the modulo operator is used incorrectly causing index errors

C Because the loop runs fewer times than the number of requests

D Because the servers list is empty, causing an error

💡 Hint

Look at how the index changes with integer division.

The integer division (i // 2) groups requests in pairs, so each server gets two requests in a row, causing uneven bursts.