Debugging service connectivity in Kubernetes - Step-by-Step Execution

Process Flow - Debugging service connectivity

Start: Service unreachable

↓

Check Pod status

↓

Check Service endpoints

↓

Check Network Policies

↓

Check DNS resolution

↓

Check Logs and Events

↓

Fix issue or escalate

↓

End

This flow shows the step-by-step checks to find why a Kubernetes service is not reachable.

Execution Sample

Kubernetes

kubectl get pods
kubectl get svc my-service
kubectl describe svc my-service
kubectl get endpoints my-service
kubectl exec -it pod-name -- curl http://my-service
kubectl logs pod-name

Commands to check pod status, service details, endpoints, test connectivity, and view logs.

Process Table

Step	Command	Action	Result	Next Step
1	kubectl get pods	Check if pods are running	Pods are Running	Check service endpoints
2	kubectl get svc my-service	Check service existence and type	Service exists, ClusterIP assigned	Check endpoints
3	kubectl get endpoints my-service	Check if endpoints are ready	Endpoints list is empty	Check pod labels and selectors
4	kubectl describe svc my-service	Verify selector labels	Selector labels do not match pod labels	Fix pod labels or service selector
5	kubectl exec -it pod-name -- curl http://my-service	Test connectivity from pod	Connection refused	Check network policies
6	kubectl get networkpolicies	Check network policies	No restrictive policies found	Check DNS resolution
7	kubectl exec -it pod-name -- nslookup my-service	Check DNS resolution	DNS resolves to ClusterIP	Check pod logs
8	kubectl logs pod-name	Check pod logs for errors	No errors found	Fix labels and redeploy
9	Fix pod labels to match service selector	Update pod labels	Pods now match service selector	Recheck endpoints
10	kubectl get endpoints my-service	Verify endpoints after fix	Endpoints list shows pod IPs	Test connectivity again
11	kubectl exec -it pod-name -- curl http://my-service	Test connectivity from pod	Successful response	Issue resolved
12	-	-	-	End of debugging

💡 Connectivity issue resolved after fixing pod labels to match service selector, endpoints populated, and successful curl test.

Status Tracker

Variable	Start	After Step 3	After Step 9	Final
Pods status	Unknown	Running	Running	Running
Service selector labels	mismatch	mismatch	match	match
Endpoints list	Unknown	Empty	Populated	Populated
Connectivity test	Not tested	Failed	Failed	Successful

Key Moments - 3 Insights

Why do endpoints show empty even though pods are running?

Because the service selector labels do not match the pod labels, so Kubernetes cannot link pods to the service (see execution_table step 3 and 4).

Why does curl to the service fail even when pods are running?

Because no endpoints are associated with the service due to label mismatch, so traffic cannot reach pods (see execution_table step 5).

How do we confirm DNS is not the issue?

By running nslookup inside a pod and seeing the service resolves to the correct ClusterIP (see execution_table step 7).

Visual Quiz - 3 Questions

Look at the execution table, at which step do endpoints become populated?

A Step 3

B Step 10

C Step 9

D Step 7

💡 Hint

Check the 'Endpoints list' column in execution_table rows for when it changes from empty to populated.

According to variable_tracker, what is the state of 'Connectivity test' after step 5?

A Failed

B Successful

C Not tested

D Unknown

💡 Hint

Look at the 'Connectivity test' row and the value under 'After Step 3' and 'After Step 9' columns.

If pod labels were correct from the start, which step would be unnecessary?

A Step 4 - Describe service

B Step 6 - Check network policies

C Step 9 - Fix pod labels

D Step 7 - Check DNS resolution

💡 Hint

Refer to the step where pod labels are fixed and consider if that would be needed if labels matched initially.

Concept Snapshot

Debugging service connectivity in Kubernetes:
- Check pod status with 'kubectl get pods'
- Verify service and endpoints with 'kubectl get svc' and 'kubectl get endpoints'
- Ensure service selectors match pod labels
- Test connectivity inside pods using 'curl'
- Check network policies and DNS if needed
- Fix label mismatches to restore connectivity

Full Transcript

When a Kubernetes service is unreachable, start by checking if pods are running using 'kubectl get pods'. Then verify the service exists and has a ClusterIP with 'kubectl get svc'. Next, check if endpoints are populated with 'kubectl get endpoints'. If endpoints are empty, describe the service to check if its selector labels match pod labels. A mismatch means the service cannot route traffic to pods. Fix pod labels or service selectors accordingly. Test connectivity from a pod using curl to the service. If connection fails, check network policies and DNS resolution. Finally, check pod logs for errors. After fixing label mismatches, endpoints populate and connectivity succeeds, resolving the issue.