Bird
Raised Fist0

Metrics Collection System Design - Push vs Pull Model Trade-offs

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Metrics Collection System Design - Push vs Pull Model Trade-offs
45 min · senior
senior

You are designing a metrics collection system for a startup with 50 million daily active users (DAU). The system must ingest, process, and store metrics from millions of client devices and backend services. The business requires encrypted data in transit and at rest, offline delivery support for intermittent connectivity, and a P99 ingestion latency under 200ms to enable near real-time monitoring and alerting.

Constraints
  • All metrics data must be encrypted end-to-end (TLS in transit, AES-256 at rest).
  • Support offline delivery: clients may batch and retry metrics when offline.
  • P99 ingestion latency must be under 200ms to support near real-time dashboards.
  • System must handle peak ingestion rates of at least 1 million metrics per second.
  • Data retention for raw metrics is 30 days, with aggregation stored for 1 year.
📋
Guided Steps
1/6
Step 1
Choosing Push vs Pull Ingestion Model
Should the system primarily use a push or pull model for metrics ingestion? What are the trade-offs at 50M DAU scale?
💡 Hint
Consider client connectivity patterns, offline support, and ingestion latency requirements.
Key Decision
Adopt a hybrid model where clients push metrics via HTTPS REST API (TLS encrypted) for low latency and offline buffering, while backend services can optionally pull aggregated metrics via gRPC for analytics.
Interviewer Expectation
Mid: Names HTTPS push for ingestion. Senior: Explains offline buffering benefits of push, pull's suitability for backend analytics, and how push reduces load spikes compared to pull at 50M DAU.
⚠️Common Wrong Answer
Relying solely on pull model - causes massive load spikes and high latency due to 50M clients polling frequently, collapsing system under load.
Step 2
Data Ingestion Pipeline and Messaging System
What messaging or streaming technology will you use to ingest and buffer metrics data before processing?
💡 Hint
Look for a system that supports high throughput, partitioning by client or metric type, and durable storage.
Key Decision
Use Apache Kafka with topic partitioning keyed by client_id to ensure ordered, scalable ingestion and replay capability.
Interviewer Expectation
Mid: Names Kafka for ingestion. Senior: Explains client_id partition key ensures ordering per client, avoids cross-partition coordination, and supports replay for offline delivery.
⚠️Common Wrong Answer
Using RabbitMQ or Redis Pub/Sub - these do not scale linearly to 1M msg/sec and lack durable replay needed for offline delivery.
Step 3
Storage Technology for Raw and Aggregated Metrics
Which storage system will you choose for raw metrics and aggregated data, and why?
💡 Hint
Consider write-heavy workloads, time-series data, and efficient range queries for aggregation.
Key Decision
Use Apache Cassandra for raw metrics with partition key (client_id, metric_name, date) for write scalability and range scans; use ClickHouse for aggregated metrics to support fast OLAP queries.
Interviewer Expectation
Mid: Names Cassandra for raw storage. Senior: Explains partition key design for even data distribution and efficient time-range queries, and why ClickHouse is chosen for analytical aggregation over Cassandra.
⚠️Common Wrong Answer
Using relational databases like MySQL for raw metrics - leads to write bottlenecks and poor horizontal scalability at 1M writes/sec.
Step 4
Ensuring Encryption and Security
How will you ensure encryption of metrics data in transit and at rest, and secure client authentication?
💡 Hint
Think about TLS, encryption keys management, and token-based authentication.
Key Decision
Enforce TLS 1.3 for all client-server communication; use AWS KMS-managed AES-256 encryption for data at rest in Cassandra and Kafka; authenticate clients with OAuth 2.0 bearer tokens.
Interviewer Expectation
Mid: Mentions TLS and encryption at rest. Senior: Details TLS 1.3 benefits, key rotation via KMS, and OAuth token scopes for fine-grained client permissions.
⚠️Common Wrong Answer
Ignoring encryption at rest or using self-signed TLS certificates - risks data breaches and non-compliance with security standards.
Step 5
Handling Offline Delivery and Retry Mechanisms
How will the system support clients that are intermittently offline and need to retry sending metrics?
💡 Hint
Consider client-side buffering, idempotency, and server-side deduplication.
Key Decision
Clients buffer metrics locally with unique IDs; server uses Kafka's offset and idempotent producer features to deduplicate; clients retry with exponential backoff to avoid overload.
Interviewer Expectation
Mid: Mentions client buffering and retries. Senior: Explains idempotent Kafka producers, offset tracking for exactly-once semantics, and backoff strategies to prevent thundering herd.
⚠️Common Wrong Answer
No deduplication or retry strategy - leads to duplicate metrics inflating data and system overload during reconnect storms.
Step 6
API Design for Metrics Collection
Design the 3 core API endpoints to support push ingestion, pull retrieval, and acknowledgments.
💡 Hint
Think about REST vs gRPC, payload formats, and authentication.
Key Decision
1) POST /metrics/push - HTTPS REST endpoint for clients to push encrypted metrics batches; 2) GET /metrics/pull - gRPC endpoint for backend services to pull aggregated metrics with filters; 3) POST /metrics/ack - REST endpoint for clients to acknowledge receipt of processing confirmation.
Interviewer Expectation
Mid: Names endpoints and HTTP methods. Senior: Explains REST for client compatibility, gRPC for efficient backend pulls, payload encryption, and idempotency tokens in acknowledgments.
⚠️Common Wrong Answer
Using only HTTP polling endpoints - causes excessive load and high latency at scale.
🔌
API Design Challenge
Sketch the 3 core API endpoints for the metrics collection system that support both push and pull ingestion models.
Hints
  • How does a client push metrics data securely and efficiently?
  • How can a client pull aggregated or raw metrics data for analysis?
  • How does the system acknowledge receipt and processing of metrics?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Correct choice of push vs pull ingestion model with scale reasoning
  • Use of Apache Kafka with appropriate partitioning for ingestion
  • Selection of Cassandra for raw metrics storage with partition key design
  • Encryption in transit and at rest with specific technologies
  • Support for offline delivery with client buffering and deduplication
Good to Have - 30%
  • Use of ClickHouse for analytical queries
  • OAuth 2.0 for client authentication
  • Exponential backoff retry strategies
  • Idempotent Kafka producers for exactly-once semantics
Bonus - 10%
  • Hybrid push-pull ingestion model
  • gRPC for backend pull APIs
  • Key rotation with AWS KMS
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The system uses a hybrid ingestion model combining HTTPS REST push from clients for low latency and offline support, and gRPC pull from backend services for analytics. Apache Kafka with client_id partitioning ensures ordered, scalable ingestion and replay. Raw metrics are stored in Apache Cassandra with a composite partition key (client_id, metric_name, date) for write scalability and efficient range queries. Aggregated metrics are stored in ClickHouse for fast OLAP queries. TLS 1.3 secures data in transit, and AWS KMS-managed AES-256 encryption secures data at rest. OAuth 2.0 bearer tokens authenticate clients. Clients buffer metrics locally with unique IDs and retry with exponential backoff; Kafka's idempotent producers and offset tracking ensure exactly-once ingestion. The API includes REST endpoints for push ingestion and acknowledgments, and a gRPC endpoint for pulling aggregated metrics.

Key Decisions
  • Hybrid push-pull ingestion model to balance latency and load
  • Kafka topic partitioned by client_id for ordered, scalable ingestion and replay
  • Cassandra with composite partition key (client_id, metric_name, date) for write scalability and range queries
  • ClickHouse chosen for fast OLAP queries on aggregated data
  • TLS 1.3 enforced for all client-server communication
  • AWS KMS-managed AES-256 encryption for data at rest in Kafka and Cassandra
  • OAuth 2.0 bearer tokens for secure client authentication
  • Client-side buffering with unique IDs and exponential backoff retry for offline support
  • Idempotent Kafka producers and offset tracking for exactly-once ingestion semantics
  • REST API for push ingestion and acknowledgments; gRPC API for backend pull retrieval
  • Apache Kafka with client_id partition key for ordered, scalable ingestion and replay support
  • Apache Cassandra with composite partition key (client_id, metric_name, date) for write scalability and efficient range queries
  • ClickHouse for fast OLAP queries on aggregated metrics data
  • TLS 1.3 enforced for all client-server communication to ensure encryption in transit
  • AWS KMS-managed AES-256 encryption for data at rest in Kafka and Cassandra
  • OAuth 2.0 bearer tokens for secure client authentication and authorization
  • Client-side buffering with unique IDs and exponential backoff retry to support offline delivery
  • Idempotent Kafka producers and offset tracking for exactly-once ingestion semantics
  • REST API for push ingestion and acknowledgments; gRPC API for backend pull retrieval

Practice

(1/5)
1. A global e-commerce platform requires zero downtime during data center failures and immediate failover without manual intervention. Which availability pattern best supports this requirement?
easy
A. Active-Active failover with synchronous replication
B. Single active node with periodic backups
C. Active-Passive failover with asynchronous replication
D. Active-Passive failover with manual DNS switch

Solution

  1. Step 1: Identify failover requirements

    Zero downtime and immediate failover require both sites to be active and data synchronized synchronously to avoid data loss.

  2. Step 2: Evaluate options

    Active-Active with synchronous replication ensures both data centers serve traffic simultaneously and failover is automatic without data loss.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Active-Active with sync replication -> zero downtime and instant failover [check]
Hint: Active-Active sync = zero downtime, instant failover [check]
Common Mistakes:
2. A deployment system must ensure zero downtime during releases by switching traffic instantly between two identical environments. Which deployment strategy best fits this requirement?
easy
A. Rolling deployment updating servers one by one
B. Canary deployment with gradual traffic shifting to new version
C. Feature flags toggling new features on the same environment
D. Blue-Green deployment with instant switch of load balancer target

Solution

  1. Step 1: Identify zero downtime requirement

    Blue-Green deployment uses two identical environments and switches traffic instantly, ensuring zero downtime.

  2. Step 2: Eliminate other options

    Canary is gradual, not instant; Feature flags don't switch entire environments; Rolling updates cause partial downtime.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Blue-Green switches environments instantly for zero downtime [check]
Hint: Instant switch between identical environments is Blue-Green [OK]
Common Mistakes:
3. A food delivery system must efficiently match nearby drivers to new orders in real-time, minimizing latency and ensuring fairness. Which component is best suited to handle this driver assignment under high load?
easy
A. A relational database with spatial indexing to query nearest drivers synchronously
B. A simple REST API that queries all drivers and filters by distance on the client side
C. A batch job that runs every 5 minutes to assign drivers to pending orders
D. A distributed in-memory geospatial index with pub/sub notifications for driver availability

Solution

  1. Step 1: Identify driver assignment requirements

    Real-time, low-latency matching requires fast spatial queries and event-driven updates.

  2. Step 2: Evaluate component suitability

    Relational DBs with spatial indexes (A) are too slow for high write/read concurrency; batch jobs (C) introduce unacceptable delay; client-side filtering (D) is inefficient and inconsistent.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Distributed in-memory geospatial index with pub/sub -> real-time, scalable driver assignment [check]
Hint: In-memory geospatial + pub/sub for real-time driver matching [check]
Common Mistakes:
4. If the quorum coordinator fails immediately after receiving write acknowledgments from a majority but before sending success to the client, what is the exact failure mode observed by the client?
medium
A. Client experiences a timeout and retries, risking duplicate writes
B. Client blocks indefinitely waiting for acknowledgment
C. Client receives duplicate success acknowledgments causing confusion
D. Client receives failure response despite successful write

Solution

  1. Step 1: Trace quorum coordinator failure after majority ack

    Coordinator crashes before responding, client waits but eventually times out.

  2. Step 2: Client retries write causing potential duplicates

    Client retries due to timeout; duplicates possible if write was committed.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Timeout triggers retry risking duplicate writes [check]
Hint: Coordinator crash before ack -> client timeout and retry risk [check]
Common Mistakes:
5. For global video delivery, which architecture better handles sudden traffic spikes in a specific region with minimal latency increase?
medium
A. Hierarchical CDN with regional edge caches and origin fallback
B. Centralized origin server cluster with global load balancer
C. Peer-to-peer streaming where clients share video chunks directly
D. Multiple independent origin servers in each region without caching

Solution

  1. Step 1: Evaluate architectures for traffic spike handling

    Centralized origin (Centralized origin server cluster with global load balancer) risks overload and latency spikes. Peer-to-peer (Peer-to-peer streaming where clients share video chunks directly) depends on client availability and is complex for global scale. Independent origins without caching (Multiple independent origin servers in each region without caching) lack scalability and increase origin load.

  2. Step 2: Assess hierarchical CDN benefits

    Regional edge caches (Hierarchical CDN with regional edge caches and origin fallback) absorb traffic spikes locally, reducing origin load and latency, providing scalable and resilient delivery.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Hierarchical CDN absorbs regional spikes with low latency [check]
Hint: Regional caches absorb spikes, origin fallback reduces load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard