Dashboards (Grafana) in Microservices - System Design Exercise

Design: Microservices Monitoring Dashboard with Grafana

In scope: Metrics collection, storage, visualization, alerting, and access control. Out of scope: Microservices implementation, detailed alert notification channels.

Functional Requirements

FR1: Display real-time metrics from multiple microservices

FR2: Support customizable dashboards for different teams

FR3: Visualize key performance indicators (KPIs) such as latency, error rates, and throughput

FR4: Allow alerting based on threshold breaches

FR5: Handle up to 100 microservices with 10,000 metrics per second

FR6: Provide historical data for at least 30 days

FR7: Secure access with role-based permissions

Non-Functional Requirements

NFR1: API response latency for dashboard queries should be under 500ms (p99)

NFR2: System availability must be 99.9% uptime

NFR3: Data retention for 30 days with efficient storage

NFR4: Support concurrent access by 500 users

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Metrics collection agents (e.g., Prometheus exporters)

Time-series database for storing metrics

Grafana for dashboard visualization

Authentication and authorization service

Alert manager for threshold-based alerts

Design Patterns

Pull vs push metrics collection

Caching for dashboard queries

Role-based access control (RBAC)

Data retention and downsampling

High availability and failover

Reference Architecture

                    +---------------------+
                    |  User Browsers       |
                    +----------+----------+
                               |
                               v
                    +----------+----------+
                    |      Grafana UI      |
                    +----------+----------+
                               |
                               v
                    +----------+----------+
                    | Authentication &    |
                    | Authorization (RBAC)|
                    +----------+----------+
                               |
                               v
                    +----------+----------+
                    |   Query Engine       |
                    +----------+----------+
                               |
                               v
          +--------------------+--------------------+
          |                                         |
+---------+---------+                     +---------+---------+
| Time-Series DB    |                     | Alert Manager     |
| (e.g., Prometheus |                     |                   |
| TSDB)             |                     +-------------------+
+---------+---------+
          |
          v
+---------+---------+
| Metrics Exporters  |
| (on microservices) |
+-------------------+

Components

Metrics Exporters

Prometheus Exporters

Collect metrics from each microservice and expose them for scraping

Time-Series Database

Prometheus TSDB or Cortex

Store collected metrics efficiently with time stamps

Grafana UI

Grafana

Visualize metrics in customizable dashboards

Authentication & Authorization

OAuth2 / LDAP / RBAC system

Secure dashboard access and enforce user permissions

Alert Manager

Prometheus Alertmanager

Send alerts when metrics cross defined thresholds

Query Engine

PromQL or equivalent

Process user queries to fetch metrics from TSDB

Request Flow

1. 1. Metrics exporters on each microservice collect and expose metrics endpoints.

2. 2. Prometheus server scrapes metrics from exporters at regular intervals (e.g., every 15 seconds).

3. 3. Scraped metrics are stored in the time-series database.

4. 4. Users access Grafana UI to view dashboards.

5. 5. Grafana authenticates users and checks permissions via the auth service.

6. 6. Grafana queries the time-series database using the query engine to fetch requested metrics.

7. 7. Metrics data is visualized on dashboards with graphs and charts.

8. 8. Alert manager monitors metrics and triggers alerts based on configured rules.

9. 9. Alerts are sent to users via configured notification channels (email, Slack, etc.).

Database Schema

Entities: - Metric: {metric_id, name, labels (key-value), timestamp, value} - Dashboard: {dashboard_id, name, owner_user_id, configuration_json} - User: {user_id, username, roles} - AlertRule: {alert_id, metric_name, threshold, duration, severity, notification_channels} Relationships: - User owns multiple Dashboards (1:N) - AlertRules linked to Metrics by metric_name - Roles define access permissions for Dashboards and Alerts

Scaling Discussion

Bottlenecks

High ingestion rate of metrics causing storage and processing overload

Slow query response times due to large data volume

Authentication service becoming a single point of failure

Alert manager overwhelmed by frequent alerts

Dashboard UI performance degradation with many concurrent users

Solutions

Use a horizontally scalable TSDB like Cortex or Thanos to distribute storage and ingestion load

Implement query caching and downsampling of older metrics to speed up queries

Deploy authentication service in a highly available cluster with load balancing

Rate-limit alerts and use deduplication in alert manager to reduce noise

Use Grafana’s built-in caching and optimize dashboard queries; scale Grafana instances behind a load balancer

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing and answering questions.

Explain the choice of Prometheus and Grafana as industry standards for metrics and dashboards

Discuss pull-based metrics collection for reliability and scalability

Highlight security with RBAC and authentication integration

Describe how alerting integrates with monitoring for proactive issue detection

Address scaling challenges with distributed TSDB and caching

Mention data retention and downsampling strategies for storage efficiency