GKE monitoring and logging in GCP - Commands & Configuration

Introduction

When you run applications on Google Kubernetes Engine (GKE), you need to see how they perform and if any errors happen. GKE monitoring and logging help you watch your apps and find problems quickly by collecting data and showing it in easy dashboards.

When you want to check if your app on GKE is running smoothly without crashes.

When you need to find out why your app is slow or not responding.

When you want to keep track of resource use like CPU and memory for your containers.

When you want to collect logs from all your app parts in one place for easy searching.

When you want to get alerts if something goes wrong with your GKE cluster or apps.

Config File - gke-monitoring.yaml

gke-monitoring.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: kube-system
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      format json
    </source>
    <match kubernetes.**>
      @type stackdriver
      resource_type k8s_container
    </match>

This ConfigMap sets up Fluentd to collect logs from all containers in the GKE cluster. It tells Fluentd where to find the logs and how to send them to Google Cloud's logging service called Stackdriver (now Cloud Logging). This helps centralize logs for easy viewing and searching.

Commands

This command applies the logging configuration to your GKE cluster, setting up Fluentd to collect and send logs to Cloud Logging.

Terminal

kubectl apply -f gke-monitoring.yaml

Expected Output

configmap/fluentd-config created

This command checks if the Fluentd pods responsible for sending logs are running properly in the kube-system namespace.

Terminal

kubectl get pods -n kube-system -l k8s-app=fluentd-cloud-logging

Expected Output

NAME                                   READY   STATUS    RESTARTS   AGE

fluentd-cloud-logging-abc123           1/1     Running   0          2m

→

-n kube-system - Specifies the namespace where Fluentd runs

→

-l k8s-app=fluentd-cloud-logging - Filters pods by label to find Fluentd pods

This command shows which logging and monitoring services are enabled on your GKE cluster to confirm Cloud Logging and Cloud Monitoring are active.

Terminal

gcloud container clusters describe example-cluster --zone us-central1-a --format='value(loggingService,monitoringService)'

Expected Output

logging.googleapis.com/kubernetes

monitoring.googleapis.com/kubernetes

→

--zone us-central1-a - Specifies the cluster zone

→

--format='value(loggingService,monitoringService)' - Formats output to show only logging and monitoring services

This command shows the current CPU and memory usage of pods in the default namespace, helping you monitor resource use.

Terminal

kubectl top pods --namespace default

Expected Output

NAME                     CPU(cores)   MEMORY(bytes)

my-app-1234567890-abcde  50m          100Mi

→

--namespace default - Shows resource usage for pods in the default namespace

Key Concept

If you remember nothing else, remember: setting up Fluentd in GKE sends your app logs to Cloud Logging so you can easily watch and troubleshoot your apps.

Common Mistakes

Not applying the logging ConfigMap to the correct namespace.

Fluentd runs in the kube-system namespace, so applying config elsewhere means logs won't be collected.

Always apply logging configurations in the kube-system namespace where Fluentd runs.

Forgetting to check if Cloud Logging and Monitoring are enabled on the cluster.

Without these services enabled, logs and metrics won't be collected or visible in Google Cloud Console.

Use gcloud commands to verify loggingService and monitoringService are set to Kubernetes services.

Running kubectl top without metrics-server installed.

kubectl top needs metrics-server to provide resource usage data; without it, the command fails.

Ensure metrics-server is installed and running in your cluster before using kubectl top.

Summary

Apply a Fluentd ConfigMap to collect container logs and send them to Cloud Logging.

Check Fluentd pods are running in the kube-system namespace to confirm logging is active.

Verify Cloud Logging and Monitoring services are enabled on your GKE cluster.

Use kubectl top to monitor pod resource usage in real time.

Practice

1. What is the main purpose of enabling monitoring and logging in a GKE cluster?

easy

A. To deploy new applications faster

B. To increase the cluster's CPU and memory automatically

C. To watch the health and performance of your cluster and applications

D. To backup your cluster data to Google Drive

Solution

1. Step 1: Understand monitoring and logging roles

   Monitoring and logging help track the health, performance, and errors in your cluster and apps.

2. Step 2: Identify the correct purpose

   Increasing resources, faster deployment, or backups are unrelated to monitoring and logging.

3. Final Answer:

   To watch the health and performance of your cluster and applications -> Option C

4. Quick Check:

   Monitoring and logging = watch cluster health [OK]

Hint: Monitoring = watch health; logging = record events [OK]

Common Mistakes:

• Confusing monitoring with resource scaling
• Thinking logging backs up data
• Assuming monitoring deploys apps

2. Which command correctly enables Cloud Logging and Monitoring on an existing GKE cluster named my-cluster in zone us-central1-a?

easy

A. gcloud container clusters create my-cluster --zone us-central1-a --enable-logging --enable-monitoring

B. gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM

C. kubectl enable logging my-cluster --zone us-central1-a

D. gcloud container clusters update my-cluster --zone us-central1-a --enable-logs --enable-metrics

Solution

1. Step 1: Identify the correct update command syntax

   The gcloud container clusters update command uses flags --logging and --monitoring with values like SYSTEM and WORKLOAD.

2. Step 2: Check each option

   gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM uses correct flags and values. gcloud container clusters create my-cluster --zone us-central1-a --enable-logging --enable-monitoring is for creating clusters, not updating. kubectl enable logging my-cluster --zone us-central1-a is invalid syntax. gcloud container clusters update my-cluster --zone us-central1-a --enable-logs --enable-metrics uses wrong flags.

3. Final Answer:

   gcloud container clusters update my-cluster --zone us-central1-a --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM -> Option B

4. Quick Check:

   Update cluster with --logging and --monitoring flags [OK]

Hint: Use 'gcloud container clusters update' with --logging and --monitoring [OK]

Common Mistakes:

• Using 'create' instead of 'update' to enable logging
• Wrong flag names like --enable-logs
• Trying to use kubectl for cluster logging setup

3. You run kubectl logs -l app=webserver on your GKE cluster. What output should you expect?

medium

A. Logs from all pods labeled with app=webserver

B. A list of pods with label app=webserver

C. An error saying label selector is invalid

D. Metrics data about the webserver pods

Solution

1. Step 1: Understand kubectl logs -l usage

   The -l flag selects pods by label, and kubectl logs shows their logs.

2. Step 2: Analyze the command output

   The command outputs logs from all pods matching app=webserver. It does not list pods or show metrics.

3. Final Answer:

   Logs from all pods labeled with app=webserver -> Option A

4. Quick Check:

   kubectl logs -l = logs from matching pods [OK]

Hint: kubectl logs -l shows logs from pods matching label [OK]

Common Mistakes:

• Expecting pod list instead of logs
• Confusing logs with metrics
• Thinking label selector syntax is invalid

4. You enabled Cloud Monitoring on your GKE cluster, but metrics do not appear in the Cloud Console. What is a likely cause?

medium

A. The cluster is in a different Google Cloud project

B. kubectl logs command was not run

C. Cloud Logging is disabled, so metrics cannot show

D. The cluster was updated with incorrect monitoring flags

Solution

1. Step 1: Check monitoring setup flags

   If monitoring flags were set incorrectly during cluster update, metrics won't appear.

2. Step 2: Evaluate other options

   kubectl logs does not affect metrics. Cloud Logging can be separate from Monitoring. Different project would cause access issues but not necessarily missing metrics if viewed correctly.

3. Final Answer:

   The cluster was updated with incorrect monitoring flags -> Option D

4. Quick Check:

   Incorrect monitoring flags block metrics [OK]

Hint: Check monitoring flags in cluster update command [OK]

Common Mistakes:

• Assuming kubectl logs affects metrics
• Confusing logging with monitoring
• Ignoring project or permissions issues

5. You want to monitor a custom application metric in your GKE cluster and view it in Cloud Monitoring. Which steps should you take?

hard

A. Instrument your app to expose metrics, deploy Prometheus, and configure Cloud Monitoring to scrape them

B. Enable Cloud Logging only and check logs for metric data

C. Use kubectl logs to collect metrics from pods and upload manually

D. Create a new GKE cluster with monitoring enabled and redeploy your app

Solution

1. Step 1: Instrument the application

   Custom metrics require your app to expose them, usually via Prometheus format.

2. Step 2: Deploy Prometheus and configure scraping

   Prometheus collects metrics from your app. Cloud Monitoring can integrate with Prometheus to display metrics.

3. Step 3: Understand other options

   Cloud Logging alone does not handle custom metrics. kubectl logs is for logs, not metrics. Creating a new cluster is unnecessary.

4. Final Answer:

   Instrument your app to expose metrics, deploy Prometheus, and configure Cloud Monitoring to scrape them -> Option A

5. Quick Check:

   Custom metrics need app instrumentation + Prometheus + Cloud Monitoring [OK]

Hint: Use Prometheus with app instrumentation for custom metrics [OK]

Common Mistakes:

• Relying on logs for metrics
• Thinking kubectl logs collects metrics
• Creating new cluster unnecessarily