Keeping DNS sane on Ubuntu 22.04 servers
Ubuntu 22.04.5 ships with systemd-resolved running on 127.0.0.53
as a stub resolver. resolv.conf is a symlink to
/run/systemd/resolve/stub-resolv.conf. This is fine for most machines,
but it becomes a single point of failure for servers that depend on DNS for
boot-time services.
What systemd-resolved does
systemd-resolved reads the upstream DNS servers from
/etc/systemd/resolved.conf and caches results. It also manages
/etc/resolv.conf as a stub that forwards queries to systemd-resolved.
If the upstream DNS server is unreachable (e.g., a cloud provider's DNS is down),
systemd-resolved falls back to /etc/systemd/resolved.conf's
FallbackDNS setting, but that setting is often not set by default.
The failure mode
If the upstream DNS server is unreachable, systemd-resolved will
retry for 30 seconds, then fall back to the FallbackDNS if set. If
FallbackDNS is unset, the stub will keep retrying and eventually
timeout. This can cause services that depend on DNS (e.g., Nginx's resolver
directive) to fail after 30 seconds. The server may still be responsive, but any
service that needs to resolve a domain name will hang.
Fix: static resolv.conf
For servers where DNS is critical, replace the symlink with a static file:
rm -f /etc/resolv.conf
printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\noptions edns0 trust-ad\nsearch .\n' > /etc/resolv.confThis makes the server read DNS directly from the resolver without a stub.
systemd-resolved still runs, but it's only used for systemd's
own services. Most daemons (Nginx, Caddy, Python scripts, Go binaries) read
/etc/resolv.conf directly, so they get the static file.
Optional: upstream DNS
For most cloud providers, the default DNS servers are fine. For custom setups,
set FallbackDNS in /etc/systemd/resolved.conf to a
reliable upstream (e.g., 1.1.1.1 8.8.8.8). This is less common than
the static file approach but more "systemd-native".
Rule of thumb: for a server that boots with a static IP, a staticresolv.confis safer thansystemd-resolved.