Observability with service mesh in Kubernetes - Commands & Configuration

Introduction

When you run many small services together, it can be hard to see how they talk and work. A service mesh helps by adding tools that watch and report on these connections automatically.

When you want to see detailed traffic flow between your microservices without changing their code

When you need to find slow or failing services quickly in a complex app

When you want automatic collection of logs, metrics, and traces from your services

When you want to control and secure service communication with visibility

When you want to add observability features without adding extra work to your developers

Config File - istio-demo.yaml

istio-demo.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: my-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-service
spec:
  hosts:
  - "my-service"
  gateways:
  - my-gateway
  http:
  - route:
    - destination:
        host: my-service
        port:
          number: 80

This file sets up an Istio Gateway to allow external traffic and a VirtualService to route that traffic to your service named my-service. The Gateway listens on port 80 for HTTP requests. The VirtualService tells Istio how to send requests to the service inside the mesh.

Commands

This command creates the Gateway and VirtualService in your Kubernetes cluster to enable traffic routing and observability features.

Terminal

kubectl apply -f istio-demo.yaml

Expected Output

gateway.networking.istio.io/my-gateway created

virtualservice.networking.istio.io/my-service created

Check that the Istio components, including the observability tools, are running properly in the istio-system namespace.

Terminal

kubectl get pods -n istio-system

Expected Output

NAME                                    READY   STATUS    RESTARTS   AGE

istio-ingressgateway-5d8f7f6f7b-abcde  1/1     Running   0          10m

istiod-7f9d8f7f7b-xyz12                 1/1     Running   0          10m

Find the external IP or port to access your services through the Istio ingress gateway.

Terminal

kubectl get svc istio-ingressgateway -n istio-system

Expected Output

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                      AGE

istio-ingressgateway   LoadBalancer   10.96.123.45    34.68.123.45    80:31380/TCP,443:31390/TCP                    10m

Open the Kiali dashboard, a visual tool that shows service mesh traffic and observability data in your browser.

Terminal

istioctl dashboard kiali

Expected Output

Starting Kiali dashboard proxy on http://localhost:20001

Press Ctrl+C to exit

Key Concept

If you remember nothing else from this pattern, remember: a service mesh adds automatic observability by managing and reporting on service communication without changing your app code.

Common Mistakes

Not installing the service mesh control plane before applying Gateway and VirtualService resources

The resources depend on the control plane to work; without it, they won't be recognized or function.

Install Istio or your chosen service mesh first using its official installation commands before applying configuration files.

Trying to access services without exposing the ingress gateway service externally

Without an external IP or port, you cannot reach your services from outside the cluster.

Ensure the ingress gateway service is of type LoadBalancer or NodePort and note its external IP or port.

Ignoring the namespace when checking pods or services

Istio components run in the istio-system namespace; commands without -n istio-system will not find them.

Always specify the correct namespace with -n istio-system when managing Istio resources.

Summary

Apply Gateway and VirtualService YAML files to configure traffic routing and observability.

Verify Istio system pods are running to ensure observability tools are active.

Check the ingress gateway service to find how to access your mesh-enabled services.

Use the Istio Kiali dashboard to visually monitor service communication and health.

Practice

1. What is the main purpose of using a service mesh for observability in Kubernetes?

easy

A. To replace Kubernetes networking completely

B. To deploy applications faster without monitoring

C. To automatically collect metrics, logs, and traces from microservices

D. To store application data persistently

Solution

1. Step 1: Understand service mesh role in observability

   A service mesh helps by automatically collecting data like metrics, logs, and traces from microservices without manual setup.

2. Step 2: Compare options with this role

   Only To automatically collect metrics, logs, and traces from microservices describes this automatic collection for observability. Other options describe unrelated tasks.

3. Final Answer:

   To automatically collect metrics, logs, and traces from microservices -> Option C

4. Quick Check:

   Service mesh observability = automatic data collection [OK]

Hint: Service mesh = automatic monitoring data collection [OK]

Common Mistakes:

• Thinking service mesh replaces Kubernetes networking
• Confusing observability with deployment speed
• Assuming service mesh stores application data

2. Which of the following is the correct command to install Istio's observability components using istioctl?

easy

A. istioctl install --set profile=demo

B. istioctl deploy --profile=observability

C. kubectl apply -f istio-observability.yaml

D. istioctl setup observability

Solution

1. Step 1: Recall Istio installation syntax

   The correct command to install Istio with observability features is 'istioctl install' with a profile like 'demo' that includes observability tools.

2. Step 2: Check options for correct syntax

   istioctl install --set profile=demo matches the correct syntax. Options A and B use invalid commands. kubectl apply -f istio-observability.yaml is generic and not specific to istioctl.

3. Final Answer:

   istioctl install --set profile=demo -> Option A

4. Quick Check:

   Istio install command = istioctl install --set profile=demo [OK]

Hint: Use 'istioctl install --set profile=demo' for observability [OK]

Common Mistakes:

• Using 'deploy' instead of 'install' with istioctl
• Trying kubectl apply without correct manifest
• Assuming 'setup observability' is a valid command

3. Given the following Istio configuration snippet for telemetry, what will be the effect?

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: example-telemetry
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
      prometheus:
        defaultHistogramBuckets: [0.1, 0.5, 1, 5]

medium

A. Prometheus will ignore histogram buckets and use defaults

B. Prometheus will collect metrics with custom histogram buckets 0.1, 0.5, 1, and 5

C. Telemetry resource will cause an error due to invalid syntax

D. Metrics will be sent to Jaeger instead of Prometheus

Solution

1. Step 1: Analyze the Telemetry resource configuration

   The snippet sets a Telemetry resource specifying Prometheus as the metrics provider and overrides histogram buckets to [0.1, 0.5, 1, 5].

2. Step 2: Understand the effect on Prometheus metrics

   This means Prometheus will collect metrics using these custom histogram buckets instead of defaults.

3. Final Answer:

   Prometheus will collect metrics with custom histogram buckets 0.1, 0.5, 1, and 5 -> Option B

4. Quick Check:

   Telemetry config with overrides = custom Prometheus buckets [OK]

Hint: Overrides in Telemetry change Prometheus buckets [OK]

Common Mistakes:

• Assuming default buckets remain unchanged
• Confusing metrics destination as Jaeger
• Thinking syntax is invalid without error

4. You deployed Istio with observability enabled but notice no traces appear in Jaeger UI. Which of the following is the most likely cause?

medium

A. The application logs are too verbose

B. Prometheus is not scraping metrics correctly

C. The Kubernetes cluster is out of storage

D. Istio sidecar proxy injection is missing on your application pods

Solution

1. Step 1: Identify cause of missing traces in Jaeger

   Jaeger receives traces from Istio sidecar proxies. If sidecars are missing, no traces are sent.

2. Step 2: Evaluate options for trace absence

   Istio sidecar proxy injection is missing on your application pods correctly identifies missing sidecar injection as the cause. Prometheus scraping affects metrics, not traces. Storage or log verbosity do not directly cause missing traces.

3. Final Answer:

   Istio sidecar proxy injection is missing on your application pods -> Option D

4. Quick Check:

   Missing sidecar = no traces in Jaeger [OK]

Hint: No Jaeger traces? Check sidecar injection on pods [OK]

Common Mistakes:

• Blaming Prometheus for trace issues
• Assuming storage issues cause missing traces
• Thinking log verbosity affects tracing

5. You want to monitor request latency across multiple microservices in your Kubernetes cluster using Istio and Prometheus. Which combination of configurations will best achieve this?

hard

A. Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization

B. Disable Istio sidecar injection and install Jaeger only

C. Use only Kubernetes native metrics without Istio or Prometheus

D. Configure Prometheus to scrape application logs directly

Solution

1. Step 1: Identify components needed for latency monitoring

   Istio sidecars collect telemetry data. Prometheus scrapes these metrics. Grafana visualizes latency metrics effectively.

2. Step 2: Evaluate options for best observability setup

   Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization combines sidecar injection, Prometheus scraping, and Grafana dashboards, which is the standard approach. Other options miss key components or use incorrect methods.

3. Final Answer:

   Enable Istio sidecar injection, configure Prometheus scrape for Istio metrics, and use Grafana dashboards for latency visualization -> Option A

4. Quick Check:

   Sidecar + Prometheus + Grafana = latency monitoring [OK]

Hint: Use sidecar, Prometheus, and Grafana for latency monitoring [OK]

Common Mistakes:

• Disabling sidecar injection breaks telemetry collection
• Relying only on Jaeger for latency metrics
• Scraping logs instead of metrics for latency