How to Use Prometheus with Kubernetes for Monitoring

To use Prometheus with Kubernetes, deploy Prometheus as a set of pods in your cluster using the official Helm chart or manifests. Configure Prometheus to scrape metrics from Kubernetes components and your applications by setting up ServiceMonitors or PodMonitors with proper labels and annotations.

Syntax

Prometheus on Kubernetes is typically deployed using Helm or Kubernetes manifests. Key components include:

• Prometheus Server: Collects and stores metrics.
• ServiceMonitor: Custom resource to tell Prometheus which services to scrape.
• PodMonitor: Custom resource to scrape pods directly.
• ConfigMap: Holds Prometheus configuration.

Example Helm install command syntax:

helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

bash

helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

Example

This example shows how to deploy Prometheus using Helm and configure it to scrape metrics from Kubernetes nodes and pods.

bash and yaml

kubectl create namespace monitoring
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace

# Verify pods are running
kubectl get pods -n monitoring

# Example ServiceMonitor YAML to scrape a service
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-service-monitor
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: http-metrics
    interval: 30s

Output

NAME                                   READY   STATUS    RESTARTS   AGE
prometheus-server-0                    2/2     Running   0          1m
prometheus-alertmanager-0              2/2     Running   0          1m

# ServiceMonitor created successfully

Common Pitfalls

Common mistakes when using Prometheus with Kubernetes include:

• Not labeling services or pods correctly, so Prometheus cannot find targets.
• Forgetting to install the prometheus-operator which manages ServiceMonitor resources.
• Using incorrect scrape intervals or ports in ServiceMonitor definitions.
• Not exposing metrics endpoints properly in your applications.

yaml

### Wrong: Missing labels on service
apiVersion: v1
kind: Service
metadata:
  name: example-app
  namespace: default
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: example-app

### Right: Add labels for Prometheus to select
apiVersion: v1
kind: Service
metadata:
  name: example-app
  namespace: default
  labels:
    app: example-app
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: example-app

Quick Reference

Tips for using Prometheus with Kubernetes:

• Use the prometheus-community Helm chart for easy deployment.
• Label your services and pods to match ServiceMonitor selectors.
• Use ServiceMonitor or PodMonitor CRDs to configure scraping.
• Check Prometheus UI at http://prometheus-server.monitoring.svc.cluster.local:9090 inside the cluster.
• Use kubectl port-forward to access Prometheus UI locally.

Key Takeaways

Deploy Prometheus on Kubernetes using the official Helm chart for easy setup.

Use ServiceMonitor or PodMonitor resources to tell Prometheus what to scrape.

Label your Kubernetes services and pods properly for Prometheus discovery.

Verify Prometheus pods are running and targets are being scraped via the UI.

Avoid common mistakes like missing labels or incorrect scrape configurations.