Debugging service connectivity in Kubernetes - Step-by-Step Execution

Process Flow - Debugging service connectivity

Start: Service unreachable

↓

Check Pod status

↓

Check Service endpoints

↓

Check Network Policies

↓

Check DNS resolution

↓

Check Logs and Events

↓

Fix issue or escalate

↓

End

This flow shows the step-by-step checks to find why a Kubernetes service is not reachable.

Execution Sample

Kubernetes

kubectl get pods
kubectl get svc my-service
kubectl describe svc my-service
kubectl get endpoints my-service
kubectl exec -it pod-name -- curl http://my-service
kubectl logs pod-name

Commands to check pod status, service details, endpoints, test connectivity, and view logs.

Process Table

Step	Command	Action	Result	Next Step
1	kubectl get pods	Check if pods are running	Pods are Running	Check service endpoints
2	kubectl get svc my-service	Check service existence and type	Service exists, ClusterIP assigned	Check endpoints
3	kubectl get endpoints my-service	Check if endpoints are ready	Endpoints list is empty	Check pod labels and selectors
4	kubectl describe svc my-service	Verify selector labels	Selector labels do not match pod labels	Fix pod labels or service selector
5	kubectl exec -it pod-name -- curl http://my-service	Test connectivity from pod	Connection refused	Check network policies
6	kubectl get networkpolicies	Check network policies	No restrictive policies found	Check DNS resolution
7	kubectl exec -it pod-name -- nslookup my-service	Check DNS resolution	DNS resolves to ClusterIP	Check pod logs
8	kubectl logs pod-name	Check pod logs for errors	No errors found	Fix labels and redeploy
9	Fix pod labels to match service selector	Update pod labels	Pods now match service selector	Recheck endpoints
10	kubectl get endpoints my-service	Verify endpoints after fix	Endpoints list shows pod IPs	Test connectivity again
11	kubectl exec -it pod-name -- curl http://my-service	Test connectivity from pod	Successful response	Issue resolved
12	-	-	-	End of debugging

💡 Connectivity issue resolved after fixing pod labels to match service selector, endpoints populated, and successful curl test.

Status Tracker

Variable	Start	After Step 3	After Step 9	Final
Pods status	Unknown	Running	Running	Running
Service selector labels	mismatch	mismatch	match	match
Endpoints list	Unknown	Empty	Populated	Populated
Connectivity test	Not tested	Failed	Failed	Successful

Key Moments - 3 Insights

Why do endpoints show empty even though pods are running?

Because the service selector labels do not match the pod labels, so Kubernetes cannot link pods to the service (see execution_table step 3 and 4).

Why does curl to the service fail even when pods are running?

Because no endpoints are associated with the service due to label mismatch, so traffic cannot reach pods (see execution_table step 5).

How do we confirm DNS is not the issue?

By running nslookup inside a pod and seeing the service resolves to the correct ClusterIP (see execution_table step 7).

Visual Quiz - 3 Questions

Look at the execution table, at which step do endpoints become populated?

A Step 3

B Step 10

C Step 9

D Step 7

💡 Hint

Check the 'Endpoints list' column in execution_table rows for when it changes from empty to populated.

According to variable_tracker, what is the state of 'Connectivity test' after step 5?

A Failed

B Successful

C Not tested

D Unknown

💡 Hint

Look at the 'Connectivity test' row and the value under 'After Step 3' and 'After Step 9' columns.

If pod labels were correct from the start, which step would be unnecessary?

A Step 4 - Describe service

B Step 6 - Check network policies

C Step 9 - Fix pod labels

D Step 7 - Check DNS resolution

💡 Hint

Refer to the step where pod labels are fixed and consider if that would be needed if labels matched initially.

Concept Snapshot

Debugging service connectivity in Kubernetes:
- Check pod status with 'kubectl get pods'
- Verify service and endpoints with 'kubectl get svc' and 'kubectl get endpoints'
- Ensure service selectors match pod labels
- Test connectivity inside pods using 'curl'
- Check network policies and DNS if needed
- Fix label mismatches to restore connectivity

Full Transcript

When a Kubernetes service is unreachable, start by checking if pods are running using 'kubectl get pods'. Then verify the service exists and has a ClusterIP with 'kubectl get svc'. Next, check if endpoints are populated with 'kubectl get endpoints'. If endpoints are empty, describe the service to check if its selector labels match pod labels. A mismatch means the service cannot route traffic to pods. Fix pod labels or service selectors accordingly. Test connectivity from a pod using curl to the service. If connection fails, check network policies and DNS resolution. Finally, check pod logs for errors. After fixing label mismatches, endpoints populate and connectivity succeeds, resolving the issue.

Practice

1. What is the primary command to check if a Kubernetes service has endpoints assigned?

easy

A. kubectl describe nodes

B. kubectl get pods

C. kubectl get endpoints

D. kubectl get configmaps

Solution

1. Step 1: Understand service connectivity basics

   Services route traffic to endpoints, so checking endpoints shows if pods are linked.

2. Step 2: Use the correct command to list endpoints

   kubectl get endpoints lists endpoints for services, showing if pods are ready.

3. Final Answer:

   kubectl get endpoints -> Option C

4. Quick Check:

   Check endpoints = kubectl get endpoints [OK]

Hint: Use 'kubectl get endpoints' to verify service pod connections [OK]

Common Mistakes:

• Using 'kubectl get pods' which shows pods but not service endpoints
• Checking nodes or configmaps which are unrelated to service endpoints
• Confusing 'kubectl describe svc' with listing endpoints

2. Which of the following commands correctly tests DNS resolution inside a Kubernetes pod named web-123?

easy

A. kubectl exec web-123 -- nslookup myservice

B. kubectl exec web-123 nslookup myservice

C. kubectl exec -it web-123 nslookup myservice

D. kubectl exec web-123 -- curl myservice

Solution

1. Step 1: Understand kubectl exec syntax

   The correct syntax to run a command inside a pod is kubectl exec [pod] -- [command].

2. Step 2: Identify the command to test DNS

   nslookup tests DNS resolution, so kubectl exec web-123 -- nslookup myservice is correct.

3. Final Answer:

   kubectl exec web-123 -- nslookup myservice -> Option A

4. Quick Check:

   Correct exec syntax + nslookup = kubectl exec web-123 -- nslookup myservice [OK]

Hint: Use '--' before command in kubectl exec to run inside pod [OK]

Common Mistakes:

• Omitting '--' which causes command to fail
• Using '-it' without need for interactive shell
• Using curl instead of nslookup for DNS test

3. You run kubectl describe svc myservice and see no endpoints listed. What will be the output of kubectl get endpoints myservice?

medium

A. Error from server (NotFound): endpoints "myservice" not found

B. NAME ENDPOINTS AGE
myservice 10.0.0.5:80,10.0.0.6:80 10m

C. NAME ENDPOINTS AGE
myservice 127.0.0.1:80 10m

D. NAME ENDPOINTS AGE
myservice <none> 10m

Solution

1. Step 1: Interpret service describe output

   No endpoints means no pods are linked to the service, so endpoints list is empty.

2. Step 2: Predict endpoints output

   kubectl get endpoints myservice will show the service name with <none> under ENDPOINTS.

3. Final Answer:

   NAME ENDPOINTS AGE myservice <none> 10m -> Option D

4. Quick Check:

   No endpoints = <none> shown [OK]

Hint: No endpoints in describe means endpoints show <none> [OK]

Common Mistakes:

• Assuming endpoints will list IPs even if none exist
• Expecting an error when endpoints exist but are empty
• Confusing endpoints with pod IPs

4. A pod cannot reach a service by its DNS name. You run kubectl exec pod1 -- nslookup myservice and get a timeout. What is the most likely cause?

medium

A. The pod is missing the DNS policy or DNS is misconfigured

B. The service has no endpoints, so DNS resolves but no response

C. The service selector labels do not match any pods

D. The pod is running in a different namespace without DNS search path

Solution

1. Step 1: Analyze DNS timeout symptom

   A DNS timeout means the pod cannot resolve the service name, indicating DNS issues.

2. Step 2: Identify DNS misconfiguration causes

   Missing DNS policy or broken DNS config in pod causes nslookup timeout, unlike no endpoints which still resolve DNS.

3. Final Answer:

   The pod is missing the DNS policy or DNS is misconfigured -> Option A

4. Quick Check:

   DNS timeout = DNS config issue [OK]

Hint: DNS timeout means DNS config or policy problem, not endpoints [OK]

Common Mistakes:

• Confusing DNS resolution failure with no endpoints
• Assuming label mismatch causes DNS timeout instead of no response
• Ignoring namespace DNS search path issues

5. You have a service myservice in namespace prod. A pod in namespace dev tries to connect using curl myservice but fails. Which is the best way to debug this connectivity issue?

hard

A. Run kubectl describe pod -n prod myservice to check pod details

B. Run kubectl exec -n dev pod -- curl myservice.prod.svc.cluster.local to test full DNS name

C. Run kubectl get svc -n dev myservice to check service in dev namespace

D. Run kubectl exec -n prod pod -- curl myservice to test from the service namespace

Solution

1. Step 1: Understand cross-namespace service access

   Pods in different namespaces must use the full DNS name including namespace to reach a service.

2. Step 2: Test connectivity using full DNS name from the pod in dev namespace

   Running kubectl exec -n dev pod -- curl myservice.prod.svc.cluster.local tests correct DNS and connectivity.

3. Final Answer:

   Run kubectl exec -n dev pod -- curl myservice.prod.svc.cluster.local to test full DNS name -> Option B

4. Quick Check:

   Cross-namespace access needs full DNS name [OK]

Hint: Use full DNS name with namespace for cross-namespace service access [OK]

Common Mistakes:

• Trying to curl service without namespace from another namespace
• Checking service in wrong namespace
• Describing pod instead of testing connectivity