On this page
DNS and service discovery
Using the Domain Name System to dynamically locate and route traffic to ephemeral services in distributed architectures.
What it is
Service discovery is the mechanism that allows network-connected applications to find the network locations (IP addresses and ports) of other services they need to communicate with. In modern, dynamic environments where infrastructure is ephemeral and containers constantly shift across physical hosts, hardcoding IP addresses is impossible.
The Domain Name System (DNS) is the foundational protocol used to resolve human-readable names to IP addresses. While DNS was originally designed for relatively static internet routing, it has been adapted to serve as the primary service discovery mechanism within internal clusters and microservice architectures.
Why it matters
Without dynamic service discovery, adding capacity to a web application would require manually updating configuration files on every dependent service and restarting them to recognize the new IP addresses. This makes auto-scaling and self-healing systems impossible.
DNS solves this by providing a highly available, decentralized directory. When an application needs to talk to the payment service, it simply queries payments.internal. The DNS server responds with the current, live IP addresses of the payment containers. If a container crashes and is replaced, the DNS server updates its records, and subsequent queries route to the new container automatically.
How to implement
A and AAAA Records
The most basic implementation uses A records (for IPv4) and AAAA records (for IPv6). When a service scales horizontally to three instances, the internal DNS server holds three A records for the service name. When a client queries the name, the DNS server returns all three IPs, often rotating their order (DNS Round Robin) to distribute the load.
SRV Records
Standard A records only return IP addresses. If services are running on dynamically assigned, random ports (common in containerized environments without dedicated IPs per container), A records are insufficient. SRV (Service) records solve this by returning both the IP address and the specific port number, along with priority and weight values for load balancing.
Internal DNS Servers
Cloud providers and orchestrators provide integrated DNS solutions. In Kubernetes, CoreDNS automatically watches the cluster API. When a new Pod starts, CoreDNS instantly creates a DNS record for it. Applications running inside the cluster do not need to know they are in Kubernetes; they simply make standard DNS queries.
// Client configuration
DATABASE_URL=postgres://user:[email protected]:5432/dbCommon mistakes
Ignoring DNS TTLs
DNS records have a Time To Live (TTL) value, which tells clients how long they can cache the response before querying the server again. If a service discovery DNS record has a TTL of 300 seconds (5 minutes), and a container crashes, clients will continue trying to send traffic to the dead container’s IP for up to 5 minutes. Service discovery requires very low TTLs (often 0 or 1 second) to react quickly to changes.
Application-Level Caching
Even if the DNS server uses a low TTL, many language runtimes and HTTP clients completely ignore it and cache DNS lookups indefinitely. Java’s JVM famously cached DNS lookups forever by default in older versions. If an application caches the IP, it will never discover new instances and will fail permanently if the original instance is replaced.
Overloading the DNS Server
If thousands of microservices constantly query the central DNS server every second because TTLs are set to zero, the DNS server will become a bottleneck and crash. This requires implementing local DNS caches (like NodeLocal DNSCache in Kubernetes) as daemonsets on every physical node to absorb the query volume.
Verification
To verify service discovery is functioning, use the dig or nslookup command-line tools from within the environment.
Query the service name and verify it returns multiple IP addresses if the service is scaled:
dig +short payments.internal.svc.cluster.localCheck the TTL of the records to ensure it is appropriately low for ephemeral infrastructure:
dig payments.internal.svc.cluster.localScale the target service down and immediately run the query again. The DNS response should update within seconds to exclude the terminated instance.