GKE monitoring and logging in GCP - Step-by-Step Execution

Process Flow - GKE monitoring and logging

Deploy GKE Cluster

↓

Enable Monitoring & Logging

↓

Collect Metrics & Logs

↓

Send Data to Cloud Operations

↓

View Dashboards & Alerts

↓

Analyze & Troubleshoot

This flow shows how a GKE cluster is set up with monitoring and logging enabled, data is collected and sent to Cloud Operations, and then viewed for analysis.

Execution Sample

GCP

gcloud container clusters create my-cluster \
  --zone us-central1-a \
  --enable-stackdriver-kubernetes

gcloud container clusters get-credentials my-cluster --zone us-central1-a

kubectl get pods --namespace=kube-system

Creates a GKE cluster with monitoring enabled, authenticates kubectl to the cluster, and checks system pods running for logging agents.

Process Table

Step	Command/Action	Result/Output	System State Change
1	gcloud container clusters create my-cluster --zone us-central1-a --enable-stackdriver-kubernetes	Cluster 'my-cluster' created with monitoring enabled	GKE cluster created with Stackdriver monitoring and logging agents installed
2	gcloud container clusters get-credentials my-cluster --zone us-central1-a kubectl get pods --namespace=kube-system	Credentials updated. Lists pods including 'stackdriver-agent' and 'fluentd-gcp-scaler'	kubectl authenticated; Monitoring and logging agents are running in the cluster
3	Generate workload (e.g., deploy app)	App pods start running and generate logs and metrics	Logs and metrics start flowing to Cloud Operations
4	Go to Google Cloud Console > Monitoring	View cluster metrics dashboards	Metrics visualized for CPU, memory, network, etc.
5	Go to Google Cloud Console > Logging	View logs from cluster pods and system components	Logs searchable and filterable for troubleshooting
6	Set alert policies in Monitoring	Alerts trigger on defined conditions	Proactive notifications for cluster health
7	Analyze dashboards and logs	Identify issues or performance bottlenecks	Improved cluster reliability and debugging
8	Exit	Monitoring and logging setup complete	System continuously collects and displays data

💡 Monitoring and logging agents are running; data flows to Cloud Operations for ongoing analysis

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
Cluster State	None	Created with monitoring enabled	Agents running	Workload generating data	Data flowing to Cloud Operations
Pods in kube-system	None	Created	stackdriver-agent and fluentd-gcp-scaler running	No change	No change
Metrics Data	None	None	None	Generated by workload	Available in Monitoring
Logs Data	None	None	None	Generated by workload and system	Available in Logging

Key Moments - 3 Insights

Why do we see 'stackdriver-agent' pods in the kube-system namespace?

These pods are the monitoring and logging agents installed automatically when creating the cluster with --enable-stackdriver-kubernetes, as shown in execution_table step 2.

How does enabling monitoring affect the cluster workload?

Enabling monitoring adds agents that collect metrics and logs without changing the workload itself, but workload pods generate data that these agents send to Cloud Operations (see steps 3 and 4).

What happens if we don't enable monitoring during cluster creation?

No monitoring or logging agents are installed automatically, so metrics and logs won't be collected or visible in Cloud Operations unless manually configured (not shown in the execution table).

Visual Quiz - 3 Questions

Look at the execution table at step 2, which pods indicate monitoring and logging are active?

A stackdriver-agent and fluentd-gcp-scaler

B kube-dns and coredns

C nginx-ingress-controller

D etcd and kube-apiserver

💡 Hint

Check the 'Result/Output' column at step 2 in the execution table.

At which step does the cluster start sending metrics and logs to Cloud Operations?

A Step 1

B Step 3

C Step 5

D Step 7

💡 Hint

Look for when workload pods generate data in the execution table.

If you remove the --enable-stackdriver-kubernetes-engine flag during cluster creation, what changes in the execution table?

A Step 4 dashboards will show more metrics

B Step 6 alerts will trigger more often

C Step 2 will not show stackdriver-agent pods running

D Step 3 workload pods will not start

💡 Hint

Refer to the presence of monitoring agents in step 2.

Concept Snapshot

GKE monitoring and logging setup:
- Create cluster with --enable-stackdriver-kubernetes
- Agents run in kube-system namespace
- Metrics and logs collected from workloads
- Data sent to Cloud Operations
- View dashboards and logs in Google Cloud Console
- Set alerts for proactive monitoring

Full Transcript

This visual execution shows how to enable monitoring and logging in a Google Kubernetes Engine cluster. First, you create the cluster with the flag --enable-stackdriver-kubernetes, which installs monitoring and logging agents automatically. Then, you verify these agents by listing pods in the kube-system namespace. When you deploy workloads, they generate metrics and logs that the agents collect and send to Google Cloud Operations. You can then view these metrics in Monitoring dashboards and logs in Logging. Alerts can be set up to notify you of issues. This setup helps you keep your cluster healthy and troubleshoot problems efficiently.

Practice

1. What is the main purpose of enabling monitoring and logging in a GKE cluster?

easy

A. To deploy new applications faster

B. To increase the cluster's CPU and memory automatically

C. To watch the health and performance of your cluster and applications

D. To backup your cluster data to Google Drive

Solution

1. Step 1: Understand monitoring and logging roles

   Monitoring and logging help track the health, performance, and errors in your cluster and apps.

2. Step 2: Identify the correct purpose

   Increasing resources, faster deployment, or backups are unrelated to monitoring and logging.

3. Final Answer:

   To watch the health and performance of your cluster and applications -> Option C

4. Quick Check:

   Monitoring and logging = watch cluster health [OK]

Hint: Monitoring = watch health; logging = record events [OK]

Common Mistakes:

• Confusing monitoring with resource scaling
• Thinking logging backs up data
• Assuming monitoring deploys apps

2. Which command correctly enables Cloud Logging and Monitoring on an existing GKE cluster named my-cluster in zone us-central1-a?

easy

A. gcloud container clusters create my-cluster --zone us-central1-a --enable-logging --enable-monitoring

B. gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM

C. kubectl enable logging my-cluster --zone us-central1-a

D. gcloud container clusters update my-cluster --zone us-central1-a --enable-logs --enable-metrics

Solution

1. Step 1: Identify the correct update command syntax

   The gcloud container clusters update command uses flags --logging and --monitoring with values like SYSTEM and WORKLOAD.

2. Step 2: Check each option

   gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM uses correct flags and values. gcloud container clusters create my-cluster --zone us-central1-a --enable-logging --enable-monitoring is for creating clusters, not updating. kubectl enable logging my-cluster --zone us-central1-a is invalid syntax. gcloud container clusters update my-cluster --zone us-central1-a --enable-logs --enable-metrics uses wrong flags.

3. Final Answer:

   gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM -> Option B

4. Quick Check:

   Update cluster with --logging and --monitoring flags [OK]

Hint: Use 'gcloud container clusters update' with --logging and --monitoring [OK]

Common Mistakes:

• Using 'create' instead of 'update' to enable logging
• Wrong flag names like --enable-logs
• Trying to use kubectl for cluster logging setup

3. You run kubectl logs -l app=webserver on your GKE cluster. What output should you expect?

medium

A. Logs from all pods labeled with app=webserver

B. A list of pods with label app=webserver

C. An error saying label selector is invalid

D. Metrics data about the webserver pods

Solution

1. Step 1: Understand kubectl logs -l usage

   The -l flag selects pods by label, and kubectl logs shows their logs.

2. Step 2: Analyze the command output

   The command outputs logs from all pods matching app=webserver. It does not list pods or show metrics.

3. Final Answer:

   Logs from all pods labeled with app=webserver -> Option A

4. Quick Check:

   kubectl logs -l = logs from matching pods [OK]

Hint: kubectl logs -l shows logs from pods matching label [OK]

Common Mistakes:

• Expecting pod list instead of logs
• Confusing logs with metrics
• Thinking label selector syntax is invalid

4. You enabled Cloud Monitoring on your GKE cluster, but metrics do not appear in the Cloud Console. What is a likely cause?

medium

A. The cluster is in a different Google Cloud project

B. kubectl logs command was not run

C. Cloud Logging is disabled, so metrics cannot show

D. The cluster was updated with incorrect monitoring flags

Solution

1. Step 1: Check monitoring setup flags

   If monitoring flags were set incorrectly during cluster update, metrics won't appear.

2. Step 2: Evaluate other options

   kubectl logs does not affect metrics. Cloud Logging can be separate from Monitoring. Different project would cause access issues but not necessarily missing metrics if viewed correctly.

3. Final Answer:

   The cluster was updated with incorrect monitoring flags -> Option D

4. Quick Check:

   Incorrect monitoring flags block metrics [OK]

Hint: Check monitoring flags in cluster update command [OK]

Common Mistakes:

• Assuming kubectl logs affects metrics
• Confusing logging with monitoring
• Ignoring project or permissions issues

5. You want to monitor a custom application metric in your GKE cluster and view it in Cloud Monitoring. Which steps should you take?

hard

A. Instrument your app to expose metrics, deploy Prometheus, and configure Cloud Monitoring to scrape them

B. Enable Cloud Logging only and check logs for metric data

C. Use kubectl logs to collect metrics from pods and upload manually

D. Create a new GKE cluster with monitoring enabled and redeploy your app

Solution

1. Step 1: Instrument the application

   Custom metrics require your app to expose them, usually via Prometheus format.

2. Step 2: Deploy Prometheus and configure scraping

   Prometheus collects metrics from your app. Cloud Monitoring can integrate with Prometheus to display metrics.

3. Step 3: Understand other options

   Cloud Logging alone does not handle custom metrics. kubectl logs is for logs, not metrics. Creating a new cluster is unnecessary.

4. Final Answer:

   Instrument your app to expose metrics, deploy Prometheus, and configure Cloud Monitoring to scrape them -> Option A

5. Quick Check:

   Custom metrics need app instrumentation + Prometheus + Cloud Monitoring [OK]

Hint: Use Prometheus with app instrumentation for custom metrics [OK]

Common Mistakes:

• Relying on logs for metrics
• Thinking kubectl logs collects metrics
• Creating new cluster unnecessarily