Observability with service mesh in Kubernetes - Step-by-Step Execution

Process Flow - Observability with service mesh

Service Mesh Installed

↓

Sidecar Proxy Injected

↓

Traffic Routed Through Proxy

↓

Telemetry Data Collected

↓

Data Sent to Observability Tools

↓

Metrics, Logs, Traces Visualized

↓

Issues Detected and Debugged

This flow shows how a service mesh adds proxies to services, collects telemetry data, and sends it to tools for monitoring and debugging.

Execution Sample

Kubernetes

kubectl apply -f istio.yaml
kubectl get pods --namespace istio-system
kubectl label namespace default istio-injection=enabled
kubectl apply -f sample-app.yaml
kubectl get svc
kubectl logs -l app=sample-app -c istio-proxy

These commands install a service mesh, enable sidecar injection, deploy an app, and show proxy logs for observability.

Process Table

Step	Command	Action	Result/Output
1	kubectl apply -f istio.yaml	Install Istio service mesh components	istio-system namespace created, pods starting
2	kubectl get pods --namespace istio-system	Check Istio pods status	Pods: istiod, ingressgateway running
3	kubectl label namespace default istio-injection=enabled	Enable automatic sidecar injection	Namespace 'default' labeled
4	kubectl apply -f sample-app.yaml	Deploy sample app with sidecar proxies	Pods created with istio-proxy containers
5	kubectl get svc	List services including mesh-enabled ones	Services: sample-app listed
6	kubectl logs -l app=sample-app -c istio-proxy	View proxy logs for telemetry data	Logs show traffic routing and telemetry events
7	Exit	All components running and telemetry collected	Observability data ready for analysis

💡 All pods are running with sidecar proxies; telemetry data is collected and visible in logs.

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 4	Final
istio-system pods	0	Starting	Running	Running	Running
default namespace label	none	none	istio-injection=enabled	istio-injection=enabled	istio-injection=enabled
sample-app pods	0	0	0	Created with sidecar	Running with sidecar
telemetry logs	empty	empty	empty	populated	populated

Key Moments - 3 Insights

Why do we label the namespace with 'istio-injection=enabled'?

Labeling the namespace triggers automatic injection of sidecar proxies into pods created in that namespace, as shown in execution_table step 3 and 4.

What is the role of the 'istio-proxy' container in the sample app pods?

The 'istio-proxy' container acts as a sidecar proxy that intercepts traffic and collects telemetry data, visible in logs at step 6.

How do we know telemetry data is being collected?

Telemetry data appears in the logs of the istio-proxy container, as shown in execution_table step 6.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the sidecar proxy injected into the sample app pods?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Check the action and result columns for pod creation with sidecar proxies in step 4.

According to the variable tracker, what is the state of telemetry logs after step 4?

A populated

B empty

C starting

D unknown

💡 Hint

Look at the 'telemetry logs' row under 'After Step 4' column.

If the namespace label 'istio-injection=enabled' was missing, what would change in the execution table?

A Istio pods would not start in step 1

B Sidecar proxies would not be injected in step 4

C Sample app pods would fail to create in step 4

D Telemetry logs would be populated earlier

💡 Hint

Refer to step 3 and 4 where labeling triggers sidecar injection.

Concept Snapshot

Observability with Service Mesh:
- Install service mesh (e.g., Istio) in cluster
- Label namespaces to enable sidecar proxy injection
- Deploy apps; proxies collect telemetry
- Use logs, metrics, traces for monitoring
- Helps detect and debug service issues easily

Full Transcript

Observability with a service mesh means adding a layer that automatically inserts small proxy containers next to your app containers. These proxies watch all the network traffic going in and out. First, you install the service mesh components in your Kubernetes cluster. Then, you label the namespace where your apps run to enable automatic sidecar proxy injection. When you deploy your app, each pod gets a proxy container. This proxy collects telemetry data like logs, metrics, and traces. You can view this data by checking the proxy logs or using observability tools. This setup helps you see how your services communicate and quickly find problems.

Practice

1. What is the main purpose of using a service mesh for observability in Kubernetes?

easy

A. To replace Kubernetes networking completely

B. To deploy applications faster without monitoring

C. To automatically collect metrics, logs, and traces from microservices

D. To store application data persistently

Solution

1. Step 1: Understand service mesh role in observability

   A service mesh helps by automatically collecting data like metrics, logs, and traces from microservices without manual setup.

2. Step 2: Compare options with this role

   Only To automatically collect metrics, logs, and traces from microservices describes this automatic collection for observability. Other options describe unrelated tasks.

3. Final Answer:

   To automatically collect metrics, logs, and traces from microservices -> Option C

4. Quick Check:

   Service mesh observability = automatic data collection [OK]

Hint: Service mesh = automatic monitoring data collection [OK]

Common Mistakes:

• Thinking service mesh replaces Kubernetes networking
• Confusing observability with deployment speed
• Assuming service mesh stores application data

2. Which of the following is the correct command to install Istio's observability components using istioctl?

easy

A. istioctl install --set profile=demo

B. istioctl deploy --profile=observability

C. kubectl apply -f istio-observability.yaml

D. istioctl setup observability

Solution

1. Step 1: Recall Istio installation syntax

   The correct command to install Istio with observability features is 'istioctl install' with a profile like 'demo' that includes observability tools.

2. Step 2: Check options for correct syntax

   istioctl install --set profile=demo matches the correct syntax. Options A and B use invalid commands. kubectl apply -f istio-observability.yaml is generic and not specific to istioctl.

3. Final Answer:

   istioctl install --set profile=demo -> Option A

4. Quick Check:

   Istio install command = istioctl install --set profile=demo [OK]

Hint: Use 'istioctl install --set profile=demo' for observability [OK]

Common Mistakes:

• Using 'deploy' instead of 'install' with istioctl
• Trying kubectl apply without correct manifest
• Assuming 'setup observability' is a valid command

3. Given the following Istio configuration snippet for telemetry, what will be the effect?

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: example-telemetry
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
      prometheus:
        defaultHistogramBuckets: [0.1, 0.5, 1, 5]

medium

A. Prometheus will ignore histogram buckets and use defaults

B. Prometheus will collect metrics with custom histogram buckets 0.1, 0.5, 1, and 5

C. Telemetry resource will cause an error due to invalid syntax

D. Metrics will be sent to Jaeger instead of Prometheus

Solution

1. Step 1: Analyze the Telemetry resource configuration

   The snippet sets a Telemetry resource specifying Prometheus as the metrics provider and overrides histogram buckets to [0.1, 0.5, 1, 5].

2. Step 2: Understand the effect on Prometheus metrics

   This means Prometheus will collect metrics using these custom histogram buckets instead of defaults.

3. Final Answer:

   Prometheus will collect metrics with custom histogram buckets 0.1, 0.5, 1, and 5 -> Option B

4. Quick Check:

   Telemetry config with overrides = custom Prometheus buckets [OK]

Hint: Overrides in Telemetry change Prometheus buckets [OK]

Common Mistakes:

• Assuming default buckets remain unchanged
• Confusing metrics destination as Jaeger
• Thinking syntax is invalid without error

4. You deployed Istio with observability enabled but notice no traces appear in Jaeger UI. Which of the following is the most likely cause?

medium

A. The application logs are too verbose

B. Prometheus is not scraping metrics correctly

C. The Kubernetes cluster is out of storage

D. Istio sidecar proxy injection is missing on your application pods

Solution

1. Step 1: Identify cause of missing traces in Jaeger

   Jaeger receives traces from Istio sidecar proxies. If sidecars are missing, no traces are sent.

2. Step 2: Evaluate options for trace absence

   Istio sidecar proxy injection is missing on your application pods correctly identifies missing sidecar injection as the cause. Prometheus scraping affects metrics, not traces. Storage or log verbosity do not directly cause missing traces.

3. Final Answer:

   Istio sidecar proxy injection is missing on your application pods -> Option D

4. Quick Check:

   Missing sidecar = no traces in Jaeger [OK]

Hint: No Jaeger traces? Check sidecar injection on pods [OK]

Common Mistakes:

• Blaming Prometheus for trace issues
• Assuming storage issues cause missing traces
• Thinking log verbosity affects tracing

5. You want to monitor request latency across multiple microservices in your Kubernetes cluster using Istio and Prometheus. Which combination of configurations will best achieve this?

hard

A. Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization

B. Disable Istio sidecar injection and install Jaeger only

C. Use only Kubernetes native metrics without Istio or Prometheus

D. Configure Prometheus to scrape application logs directly

Solution

1. Step 1: Identify components needed for latency monitoring

   Istio sidecars collect telemetry data. Prometheus scrapes these metrics. Grafana visualizes latency metrics effectively.

2. Step 2: Evaluate options for best observability setup

   Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization combines sidecar injection, Prometheus scraping, and Grafana dashboards, which is the standard approach. Other options miss key components or use incorrect methods.

3. Final Answer:

   Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization -> Option A

4. Quick Check:

   Sidecar + Prometheus + Grafana = latency monitoring [OK]

Hint: Use sidecar, Prometheus, and Grafana for latency monitoring [OK]

Common Mistakes:

• Disabling sidecar injection breaks telemetry collection
• Relying only on Jaeger for latency metrics
• Scraping logs instead of metrics for latency