Service discovery via DNS in Kubernetes - Deep Dive

Overview - Service discovery via DNS

What is it?

Service discovery via DNS is a way for applications to find and connect to other services using domain names instead of fixed IP addresses. In Kubernetes, services get a DNS name that other parts of the system can use to reach them easily. This means apps don't need to know where exactly a service runs, just its name. DNS handles the translation from the service name to the current IP address automatically.

Why it matters

Without DNS-based service discovery, applications would need to know exact IP addresses of services, which can change often in dynamic environments like Kubernetes. This would make apps fragile and hard to maintain. DNS service discovery solves this by providing a stable name that always points to the right service, enabling smooth communication and scaling in cloud-native systems.

Where it fits

Before learning this, you should understand basic Kubernetes concepts like pods, services, and networking. After this, you can explore advanced topics like service mesh, load balancing, and multi-cluster service discovery.

Mental Model

Core Idea

Service discovery via DNS lets applications find services by name, with DNS translating those names to current IP addresses automatically.

Think of it like...

It's like calling a friend by their name instead of their phone number; even if they change numbers, you just use their name and the phone system connects you correctly.

┌───────────────┐        ┌───────────────┐        ┌───────────────┐
│ Application A │───────▶│ DNS Server    │───────▶│ Service B IP  │
└───────────────┘        └───────────────┘        └───────────────┘
       │                        ▲                        │
       │                        │                        │
       └────────────────────────┴────────────────────────┘
                 Uses service name to get IP

Build-Up - 7 Steps

1

Foundation

What is Service Discovery

Concept: Introduce the basic idea of service discovery as a way for apps to find each other.

Service discovery means that when one app wants to talk to another, it needs to know where to find it. Instead of hardcoding IP addresses, service discovery lets apps use a name that always points to the right place, even if the IP changes.

Result

You understand why apps need a way to find each other dynamically.

Understanding that services move and change IPs in cloud environments explains why static addresses don't work well.

2

Foundation

Basics of DNS in Kubernetes

Concept: Explain how Kubernetes uses DNS to assign names to services.

Kubernetes automatically creates DNS entries for each service. These names follow a pattern like service-name.namespace.svc.cluster.local. When an app queries this name, DNS returns the current IP of the service's endpoints.

Result

You know that Kubernetes services have stable DNS names that apps can use.

Knowing that DNS names are stable while IPs can change helps you trust service names for communication.

3

Intermediate

DNS Query Flow in Kubernetes

🤔Before reading on: do you think DNS queries go directly to the service IP or through an intermediary? Commit to your answer.

Concept: Describe how DNS queries from pods are handled and resolved to service IPs.

When a pod queries a service DNS name, the query goes to the cluster DNS server (like CoreDNS). CoreDNS looks up the service name and returns the virtual IP (ClusterIP) of the service. The pod then sends traffic to this IP, which Kubernetes routes to the right pod endpoints.

Result

You see the full path from DNS query to service IP resolution and traffic routing.

Understanding the DNS query path clarifies how Kubernetes abstracts service locations from clients.

4

Intermediate

DNS Namespaces and Scoping

🤔Before reading on: do you think service DNS names are global or scoped by namespace? Commit to your answer.

Concept: Explain how DNS names include namespaces to avoid conflicts and scope resolution.

Service DNS names include the namespace, like myservice.mynamespace.svc.cluster.local. This means two services with the same name in different namespaces have different full DNS names. Pods can also use short names if they are in the same namespace.

Result

You understand how Kubernetes uses namespaces in DNS to organize services.

Knowing namespace scoping prevents confusion when services share names across namespaces.

5

Intermediate

Headless Services and DNS Records

Concept: Introduce headless services and how DNS returns multiple IPs for them.

A headless service has no ClusterIP. Instead, DNS returns the IPs of all pods backing the service. This lets clients do their own load balancing or discovery by getting all pod IPs directly.

Result

You learn how DNS can provide multiple IPs for a service, not just one.

Understanding headless services shows how DNS supports more flexible discovery patterns.

6

Advanced

DNS Caching and TTL Effects

🤔Before reading on: do you think DNS results in Kubernetes are cached forever or refreshed periodically? Commit to your answer.

Concept: Explain how DNS caching and TTL (time to live) affect service discovery freshness.

DNS responses have TTL values that tell clients how long to cache them. Kubernetes services usually have short TTLs to quickly reflect changes like pod restarts or scaling. However, caching can cause temporary stale results, so apps should handle retries or failures gracefully.

Result

You understand the tradeoff between DNS caching and discovery freshness.

Knowing TTL effects helps you design resilient apps that tolerate brief DNS inconsistencies.

7

Expert

DNS Service Discovery Limits and Workarounds

🤔Before reading on: do you think DNS alone can handle all service discovery needs in Kubernetes? Commit to your answer.

Concept: Discuss limitations of DNS-based discovery and how advanced tools complement it.

DNS service discovery works well for basic needs but has limits: it can't do health checks, weighted routing, or retries. Service meshes like Istio or Linkerd add these features by intercepting traffic and adding logic. Also, multi-cluster discovery often needs extra tools beyond DNS.

Result

You see why DNS is foundational but not always sufficient for complex service discovery.

Understanding DNS limits prepares you to adopt service meshes or other solutions when needed.

Under the Hood

Kubernetes runs a DNS server (usually CoreDNS) inside the cluster. When a pod makes a DNS query for a service name, the query is intercepted by CoreDNS. CoreDNS looks up the service in Kubernetes API, finds the ClusterIP or pod IPs (for headless services), and returns them as DNS answers. The pod then uses this IP to send traffic. Kubernetes networking routes the traffic to the correct pod endpoints transparently.

Why designed this way?

DNS was chosen because it is a well-known, scalable, and standardized system for name resolution. Using DNS leverages existing tools and protocols, avoiding the need to invent a new discovery mechanism. Kubernetes integrates DNS tightly with its API to keep service names and IPs in sync dynamically. Alternatives like static config or custom registries were less flexible and harder to maintain.

┌───────────────┐
│ Pod (Client)  │
└───────┬───────┘
        │ DNS query for service-name.namespace.svc.cluster.local
        ▼
┌───────────────┐
│ CoreDNS       │
│ (Cluster DNS) │
└───────┬───────┘
        │ Queries Kubernetes API for service info
        ▼
┌───────────────┐
│ Kubernetes API│
│ Server       │
└───────┬───────┘
        │ Returns ClusterIP or pod IPs
        ▼
┌───────────────┐
│ CoreDNS       │
│ Responds with │
│ IP addresses  │
└───────┬───────┘
        │ DNS response
        ▼
┌───────────────┐
│ Pod (Client)  │
│ Sends traffic │
│ to IP        │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Kubernetes DNS always return the IP of a single pod? Commit yes or no.

Common Belief: Kubernetes DNS returns the IP of one pod directly for a service name.

Reality: Kubernetes DNS returns the ClusterIP of the service, which is a virtual IP that load balances to pods. For headless services, it returns multiple pod IPs.

Why it matters: Assuming DNS returns a pod IP can cause confusion about load balancing and service stability.

Quick: Do you think DNS names in Kubernetes are global across all namespaces? Commit yes or no.

Common Belief: Service DNS names are global and unique across the whole cluster.

Reality: Service DNS names are scoped by namespace, so the same service name can exist in different namespaces with different full DNS names.

Why it matters: Ignoring namespace scoping can lead to wrong service connections and hard-to-debug errors.

Quick: Does DNS service discovery handle health checks and routing decisions? Commit yes or no.

Common Belief: DNS service discovery includes health checks and advanced routing like retries or weighted load balancing.

Reality: DNS only resolves names to IPs; it does not perform health checks or routing logic. These features require additional tools like service meshes.

Why it matters: Relying solely on DNS can cause traffic to unhealthy pods or miss advanced routing needs.

Quick: Is DNS caching in Kubernetes infinite? Commit yes or no.

Common Belief: DNS results are cached forever once resolved.

Reality: DNS responses have TTLs that limit caching duration, so clients refresh DNS records periodically.

Why it matters: Not understanding TTL can cause confusion about why service IP changes take time to propagate.

Expert Zone

1

CoreDNS plugins can customize DNS behavior, enabling features like stub domains or conditional forwarding.

2

Headless services combined with StatefulSets enable stable network IDs for pods, useful for databases or clustered apps.

3

DNS resolution in Kubernetes can be affected by network policies and pod DNS configuration, which can cause subtle connectivity issues.

When NOT to use

DNS-based service discovery is not suitable when you need fine-grained traffic control, health-aware routing, or multi-cluster service discovery. In those cases, use service meshes like Istio or Linkerd, or external service registries with API-based discovery.

Production Patterns

In production, teams use DNS for basic service discovery but layer service meshes on top for observability, retries, and security. Headless services are common for stateful workloads. DNS caching behavior is tuned to balance performance and freshness. Multi-cluster setups often combine DNS with external DNS or service mesh federation.

Connections

Load Balancing

DNS service discovery provides the IP that load balancers use to distribute traffic.

Understanding DNS resolution helps grasp how load balancing targets are found and updated dynamically.

Service Mesh

Service meshes build on DNS discovery by adding traffic control and health checks beyond DNS capabilities.

Knowing DNS limits clarifies why service meshes are needed for advanced service communication.

Telephone Directory Systems

Both DNS and telephone directories map human-friendly names to numeric addresses for routing calls or data.

Recognizing this shared pattern shows how naming systems simplify complex networks in different domains.

Common Pitfalls

#1 Hardcoding service IPs instead of using DNS names.

Wrong approach:

curl http://10.96.0.1:80/api

Correct approach:

curl http://myservice.mynamespace.svc.cluster.local/api

Root cause:

Not understanding that service IPs can change, making hardcoded IPs unreliable.

#2 Using short service names across namespaces without qualification.

Wrong approach:

curl http://myservice/api # from a pod in a different namespace

Correct approach:

curl http://myservice.mynamespace.svc.cluster.local/api

Root cause:

Ignoring namespace scoping in DNS names leads to failed service resolution.

#3 Expecting DNS to handle service health and routing logic.

Wrong approach:

Relying on DNS alone to avoid sending traffic to unhealthy pods.

Correct approach:

Use a service mesh or readiness probes combined with Kubernetes service endpoints.

Root cause:

Misunderstanding DNS's role as only a name resolver, not a traffic manager.

Key Takeaways

Service discovery via DNS lets applications find services by stable names instead of changing IPs.

Kubernetes automatically creates DNS names for services scoped by namespace, enabling easy and reliable communication.

DNS queries go through CoreDNS, which resolves service names to ClusterIPs or pod IPs for headless services.

DNS caching and TTL affect how quickly service changes propagate, so apps must handle transient inconsistencies.

DNS is foundational but limited; advanced service discovery needs like health checks and routing require service meshes.