Design a unique ID generator in HLD - System Design Guide

Problem Statement

When multiple systems or services generate IDs independently, collisions can occur, causing data corruption or overwrites. Without a reliable unique ID generator, distributed systems struggle to maintain data integrity and traceability across components.

Solution

A unique ID generator creates identifiers that are guaranteed to be distinct across time and space. It can use techniques like timestamp-based components, machine identifiers, and sequence numbers to ensure uniqueness even in distributed environments. This allows systems to generate IDs without coordination, avoiding bottlenecks and collisions.

Architecture

Client/Service

Instance 1

→Unique ID Gen

↓

Client/Service

Instance 2

→Unique ID Gen

This diagram shows multiple client or service instances requesting unique IDs from their local unique ID generator instances, which then store or use these IDs in a database. Each generator instance ensures IDs are unique without central coordination.

Trade-offs

✓ Pros

→

Enables distributed ID generation without a single point of failure or bottleneck.

→

Supports high throughput by allowing parallel ID generation across multiple nodes.

→

Ensures IDs are sortable by time if timestamp components are included.

✗ Cons

→

Requires careful design to avoid collisions, especially with clock skew or machine ID conflicts.

→

Complexity increases with the need to coordinate machine identifiers or handle clock rollback.

→

Debugging ID generation issues can be difficult in distributed environments.

Use when your system is distributed across multiple nodes or data centers and requires scalable, collision-free ID generation at high request rates (e.g., over thousands of IDs per second).

Avoid if your system is small-scale with a single database generating auto-increment IDs, or if strict sequential IDs are mandatory without gaps.

Real World Examples

Twitter

Twitter uses Snowflake, a distributed unique ID generator that encodes timestamp, machine ID, and sequence number to generate unique, sortable tweet IDs without coordination.

Instagram

Instagram uses a similar approach to generate unique IDs for posts and comments across distributed servers, ensuring no collisions and enabling efficient data sharding.

Amazon

Amazon uses unique ID generators to assign order IDs and transaction IDs across multiple services and regions, ensuring global uniqueness and traceability.

Alternatives

Centralized ID Generator

All ID requests go to a single central server that issues IDs sequentially.

Use when: Choose when system scale is low and simplicity is preferred over distributed scalability.

UUID (Universally Unique Identifier)

IDs are generated using random or pseudo-random numbers without coordination, relying on probability for uniqueness.

Use when: Choose when you need simple, decentralized ID generation without ordering guarantees.

Database Auto-Increment IDs

IDs are generated by the database as sequential numbers on insert.

Use when: Choose when a single database handles all writes and strict sequential IDs are required.

Summary

Unique ID generators prevent collisions in distributed systems by combining timestamps, machine IDs, and sequences.

They enable scalable, parallel ID creation without a central bottleneck or coordination.

Choosing the right ID generation pattern depends on system scale, ordering needs, and complexity tolerance.