Long polling and Server-Sent Events in HLD - Scalability & System Analysis

Scalability Analysis - Long polling and Server-Sent Events

Growth Table: Long Polling and Server-Sent Events

Users	Connections	Server Load	Network Usage	Latency
100 users	100 concurrent connections	Low CPU and memory	Low bandwidth	Low latency, near real-time
10,000 users	10,000 concurrent connections	High CPU and memory on few servers	Moderate bandwidth	Some delay due to server load
1,000,000 users	1,000,000 concurrent connections	Cannot handle on few servers; memory and CPU bottleneck	High bandwidth; network saturation risk	Increased latency; possible dropped connections
100,000,000 users	100,000,000 concurrent connections	Impossible on single data center; requires global distribution	Very high bandwidth; CDN and edge servers needed	Latency depends on geo-distribution; complex load balancing

First Bottleneck

The first bottleneck is the server's ability to maintain many concurrent open connections. Both long polling and Server-Sent Events (SSE) keep connections open, consuming memory and CPU resources. At around 10,000 to 50,000 concurrent connections per server, resource limits are reached. Network bandwidth also becomes a concern as each connection sends data periodically or stays open.

Scaling Solutions

• Horizontal scaling: Add more servers behind a load balancer to distribute connections.
• Connection multiplexing: Use protocols like HTTP/2 or WebSockets to reduce overhead per connection.
• Use of reverse proxies: Employ Nginx or Envoy to efficiently manage many open connections.
• Offload static content: Use CDNs to reduce server bandwidth for static assets.
• Sharding users: Distribute users across multiple servers or regions to reduce load per server.
• Switch to WebSockets: For very high scale, WebSockets can be more efficient than long polling or SSE.
• Optimize message frequency: Reduce how often servers send updates to reduce bandwidth and CPU.

Back-of-Envelope Cost Analysis

Assuming 10,000 users with SSE or long polling:

• Each server handles ~10,000 concurrent connections (high but possible with optimized servers).
• Each connection sends a small message every 5 seconds -> 0.2 messages per second per connection.
• Total messages per second = 10,000 users * 0.2 messages/sec = 2,000 messages/sec.
• Bandwidth per message ~1 KB -> 2,000 KB/s = ~2 MB/s bandwidth per server.
• Memory per connection ~2 KB -> 10,000 connections * 2 KB = ~20 MB RAM just for connections.
• CPU usage depends on message processing; expect moderate CPU load.

Scaling to 1 million users requires ~100 servers, 200,000 messages/sec, and ~200 MB/s bandwidth total.

Interview Tip

Start by explaining how long polling and SSE keep connections open and why that matters for scaling. Identify the server resource limits (memory, CPU, network). Discuss horizontal scaling and connection management techniques. Mention alternatives like WebSockets and CDNs. Always quantify with rough numbers to show understanding of scale.

Self Check

Your server handles 1,000 QPS with long polling connections. Traffic grows 10x to 10,000 QPS. What do you do first and why?

Key Result

Long polling and Server-Sent Events scale well up to tens of thousands of concurrent connections per server. Beyond that, server memory, CPU, and network bandwidth become bottlenecks. Horizontal scaling with load balancers and connection-efficient protocols is essential for millions of users.

Practice

1. What is the main difference between long polling and Server-Sent Events (SSE)?

easy

A. Long polling closes the connection after each response; SSE keeps the connection open.

B. Long polling uses WebSockets; SSE uses HTTP/2 only.

C. Long polling requires client to open multiple connections; SSE uses multiple connections.

D. Long polling pushes data continuously; SSE waits for client requests.

Solution

1. Step 1: Understand long polling behavior

   Long polling keeps the HTTP request open until the server has data, then closes it after sending the response.

2. Step 2: Understand SSE behavior

   SSE keeps a single HTTP connection open to continuously push updates without closing it.

3. Final Answer:

   Long polling closes the connection after each response; SSE keeps the connection open. -> Option A

4. Quick Check:

   Connection persistence = A [OK]

Hint: Remember: long polling reconnects, SSE stays connected [OK]

Common Mistakes:

• Confusing SSE with WebSockets
• Thinking SSE uses multiple connections
• Believing long polling pushes continuously

2. Which HTTP method is typically used by clients to initiate a Server-Sent Events connection?

easy

A. POST

B. GET

C. PUT

D. DELETE

Solution

1. Step 1: Identify HTTP method for SSE

   SSE connections are initiated by clients using the GET method to request a stream from the server.

2. Step 2: Confirm other methods are not used

   POST, PUT, DELETE are used for data modification, not for opening SSE streams.

3. Final Answer:

   GET -> Option B

4. Quick Check:

   SSE connection method = GET [OK]

Hint: SSE uses GET to open a streaming connection [OK]

Common Mistakes:

• Choosing POST because it sends data
• Confusing PUT or DELETE with SSE initiation
• Thinking SSE uses WebSocket handshake methods

3. Consider this simplified server behavior for long polling:

while True:
  wait_for_event()
  send_response_to_client()
  client_reconnects()

What is the main downside of this approach compared to SSE?

medium

A. It cannot send data to multiple clients simultaneously.

B. It keeps the connection open indefinitely, wasting server resources.

C. It uses UDP instead of TCP, causing unreliable delivery.

D. It requires the client to repeatedly reconnect, causing overhead.

Solution

1. Step 1: Analyze long polling loop

   The server waits for an event, sends a response, then the client reconnects to wait again.

2. Step 2: Identify overhead caused by reconnects

   Each reconnect requires new HTTP handshake, increasing latency and server load.

3. Final Answer:

   It requires the client to repeatedly reconnect, causing overhead. -> Option D

4. Quick Check:

   Repeated reconnects = overhead [OK]

Hint: Long polling reconnects cause overhead, SSE keeps one open [OK]

Common Mistakes:

• Thinking long polling keeps connection open indefinitely
• Confusing TCP/UDP protocols in HTTP context
• Assuming long polling can't handle multiple clients

4. A developer implements SSE but notices the client stops receiving updates after some time. What is a likely cause?

medium

A. The client is not reconnecting after each message.

B. The client is sending multiple GET requests simultaneously.

C. The server is closing the connection prematurely.

D. The server is using POST instead of GET for SSE.

Solution

1. Step 1: Understand SSE connection behavior

   SSE expects the server to keep the connection open to push updates continuously.

2. Step 2: Identify issue with connection closing

   If the server closes the connection early, the client stops receiving updates until it reconnects.

3. Final Answer:

   The server is closing the connection prematurely. -> Option C

4. Quick Check:

   Connection closed early = no updates [OK]

Hint: SSE needs server to keep connection open, not close early [OK]

Common Mistakes:

• Thinking client must reconnect after every message
• Confusing HTTP methods for SSE
• Assuming multiple GET requests improve SSE

5. You are designing a chat app that must support thousands of users receiving real-time messages. Which approach is best and why?
A) Use long polling for all clients.
B) Use Server-Sent Events for all clients.
C) Use WebSockets for all clients.
D) Use HTTP polling every 5 seconds.

hard

A. WebSockets provide full-duplex communication, best for chat scalability.

B. Long polling scales well and reduces server load compared to SSE.

C. SSE keeps one connection per client open, efficiently pushing updates.

D. HTTP polling every 5 seconds is the most real-time and efficient.

Solution

1. Step 1: Evaluate long polling and SSE for scale

   Long polling causes overhead reconnects; SSE keeps connections open but is unidirectional.

2. Step 2: Consider WebSockets for chat

   WebSockets allow two-way communication with a single persistent connection, ideal for chat apps.

3. Step 3: Assess HTTP polling

   Polling every 5 seconds causes high latency and unnecessary load.

4. Final Answer:

   WebSockets provide full-duplex communication, best for chat scalability. -> Option A

5. Quick Check:

   Best real-time chat = WebSockets [OK]

Hint: Chat needs two-way, low-latency: choose WebSockets [OK]

Common Mistakes:

• Choosing SSE for two-way chat communication
• Thinking polling is efficient for real-time
• Assuming long polling scales better than WebSockets