First design walkthrough (URL shortener) in HLD - Scalability & System Analysis

Scalability Analysis - First design walkthrough (URL shortener)

Growth Table: URL Shortener Scaling

Scale	Users	Requests per Second (QPS)	Data Stored	Key Changes
Small	100 users	~10 QPS	~10K URLs (~1MB)	Single server, single DB instance, simple hashing
Medium	10,000 users	~1,000 QPS	~1M URLs (~100MB)	DB read replicas, caching hot URLs, load balancer
Large	1,000,000 users	~100,000 QPS	~100M URLs (~10GB)	Sharded DB, distributed cache, CDN for redirects
Very Large	100,000,000 users	~10M QPS	~10B URLs (~1TB+)	Multi-region sharding, global CDN, asynchronous writes, tiered storage

First Bottleneck

At small to medium scale, the database is the first bottleneck. It handles all URL mappings and redirect lookups. As traffic grows, the DB CPU and I/O get overwhelmed by read and write queries.

At large scale, the application servers can become CPU-bound due to encoding/decoding URLs and handling many concurrent connections.

At very large scale, network bandwidth and data partitioning become critical challenges.

Scaling Solutions

• Database: Add read replicas to distribute read load. Use connection pooling. Implement sharding by URL hash ranges to split data across servers.
• Caching: Use in-memory caches (like Redis) for popular URLs to reduce DB hits.
• Application Servers: Horizontally scale by adding more servers behind a load balancer.
• CDN: Use a Content Delivery Network to cache redirect responses globally, reducing latency and load.
• Storage: Archive old or rarely used URLs to cheaper storage tiers.
• Asynchronous Processing: Use queues for write-heavy operations to smooth spikes.

Back-of-Envelope Cost Analysis

• At 1,000 QPS, DB needs to handle ~86 million requests per day.
• Each URL record ~100 bytes (short URL, original URL, metadata).
• 1 million URLs ~100 MB storage; 100 million URLs ~10 GB.
• Redirect responses are small (~100 bytes), so bandwidth at 100,000 QPS ~10 MB/s (800 Mbps).
• Network bandwidth and storage costs grow with scale; caching reduces DB and bandwidth load.

Interview Tip

Start by clarifying requirements: expected traffic, URL length, expiration, analytics.

Discuss data model and how to generate unique short URLs.

Explain bottlenecks at different scales and propose solutions step-by-step.

Use real numbers to justify design choices.

Show awareness of trade-offs like consistency vs availability.

Self Check Question

Your database handles 1,000 QPS. Traffic grows 10x to 10,000 QPS. What do you do first and why?

Answer: Add read replicas and implement caching to reduce DB load, because the DB is the first bottleneck at this scale.

Key Result

The database is the first bottleneck as traffic grows; scaling requires read replicas, caching, sharding, and horizontal scaling of app servers with CDN support for redirects.

Practice

1. What is the primary purpose of a URL shortener system?

easy

A. To generate random passwords for users

B. To increase the length of URLs for security

C. To store user passwords securely

D. To convert long URLs into short, easy-to-share links

Solution

1. Step 1: Understand the function of a URL shortener

   A URL shortener takes a long URL and creates a shorter alias that redirects to the original URL.

2. Step 2: Identify the correct purpose among options

   Only To convert long URLs into short, easy-to-share links describes this function correctly; others describe unrelated tasks.

3. Final Answer:

   To convert long URLs into short, easy-to-share links -> Option D

4. Quick Check:

   URL shortener purpose = convert long URLs to short [OK]

Hint: Focus on what URL shorteners do: shorten URLs [OK]

Common Mistakes:

• Confusing URL shortening with security features
• Thinking it stores passwords
• Assuming it generates random data unrelated to URLs

2. Which component is essential for generating unique short codes in a URL shortener?

easy

A. Cache server

B. Load balancer

C. Short code generator

D. User authentication module

Solution

1. Step 1: Identify the role of each component

   Short code generator creates unique short strings; load balancer distributes traffic; cache stores data temporarily; authentication manages users.

2. Step 2: Match component to short code creation

   Only the short code generator is responsible for creating unique short URLs.

3. Final Answer:

   Short code generator -> Option C

4. Quick Check:

   Unique code creation = short code generator [OK]

Hint: Short codes come from the short code generator [OK]

Common Mistakes:

• Confusing load balancer with code generation
• Thinking cache generates codes
• Assuming authentication creates short URLs

3. Given a URL shortener system, what happens when a user clicks a short URL http://short.ly/abc123?

medium

A. The system looks up abc123 in storage and redirects to the original long URL

B. The system generates a new short code for abc123

C. The system deletes the short URL from the database

D. The system asks the user to enter the long URL manually

Solution

1. Step 1: Understand the redirection process

   When a short URL is accessed, the system finds the original URL mapped to the short code and redirects the user.

2. Step 2: Evaluate each option

   The system looks up abc123 in storage and redirects to the original long URL correctly describes this lookup and redirect. Others describe incorrect or unrelated actions.

3. Final Answer:

   The system looks up abc123 in storage and redirects to the original long URL -> Option A

4. Quick Check:

   Click short URL = lookup and redirect [OK]

Hint: Clicking short URL triggers lookup and redirect [OK]

Common Mistakes:

• Thinking system creates new codes on click
• Assuming deletion happens on access
• Believing user must input original URL manually

4. In a URL shortener design, which issue arises if the short code generator does not check for duplicates?

medium

A. The system runs out of storage space quickly

B. Multiple long URLs map to the same short code causing incorrect redirects

C. Users cannot create new accounts

D. The system slows down due to too many cache hits

Solution

1. Step 1: Understand duplicate short code impact

   If duplicates exist, different long URLs share the same short code, causing wrong redirection.

2. Step 2: Analyze options for relevance

   Multiple long URLs map to the same short code causing incorrect redirects correctly identifies this problem. Others describe unrelated issues.

3. Final Answer:

   Multiple long URLs map to the same short code causing incorrect redirects -> Option B

4. Quick Check:

   Duplicate codes cause wrong redirects [OK]

Hint: Duplicates cause wrong URL redirects [OK]

Common Mistakes:

• Confusing storage issues with duplicate codes
• Thinking duplicates affect user accounts
• Assuming cache hits cause this problem

5. To scale a URL shortener for millions of users, which design choice improves performance and availability?

hard

A. Use distributed storage with caching and load balancing

B. Store all data on a single server to simplify access

C. Generate short codes sequentially without randomness

D. Disable caching to ensure fresh data every time

Solution

1. Step 1: Identify scalability needs

   Millions of users require fast access, high availability, and fault tolerance.

2. Step 2: Evaluate design choices

   Distributed storage spreads data load; caching speeds up lookups; load balancing distributes traffic. This combination supports scale.

3. Step 3: Reject poor choices

   Single server limits scale; sequential codes risk collisions; disabling cache slows response.

4. Final Answer:

   Use distributed storage with caching and load balancing -> Option A

5. Quick Check:

   Scaling needs distributed storage + cache + load balancer [OK]

Hint: Scale with distributed storage, caching, and load balancing [OK]

Common Mistakes:

• Choosing single server for large scale
• Ignoring caching benefits
• Using sequential codes risking collisions