How Slow DNS Puts Valkey Sentinel Into TILT
A team running BetterDB reported it: three Sentinels, hostname addressing, TLS on, logging +tilt and flagging +sdown on a primary that was demonstrably up. Raw IPs and plaintext made it stop, and nobody was happy with that as an answer. Here is the mechanism, the isolation matrix, and two things I got wrong on the first pass.
This is the reproduction, the isolation, and two findings I got wrong on the first pass. All numbers come from captured logs and prober CSVs. Versions tested: Valkey 8.1.8 and 9.0.6. The harness that produced them is open source: github.com/BetterDB-inc/valkey-sentinel-tilt-repro.
What TILT is
Sentinel is single threaded. Its work runs in a timer on the main event loop, under the same serverCron that everything else in the process uses. That timer is supposed to fire on a schedule. Sentinel checks the wall clock each time it runs. If the gap since the previous run is negative or larger than about two seconds, Sentinel assumes something is wrong with its own timing and enters TILT.
TILT is a self protection mode. Sentinel keeps running, but it stops trusting its own failure detection. Concretely, in sentinelHandleValkeyInstance it returns early while in TILT (8.1 sentinel.c:5157, 9.0 sentinel.c:5385), before the subjective down check that runs just after (8.1 sentinel.c:5163, 9.0 sentinel.c:5391). No sdown, no odown, no failover, until the loop has been stable again for a while.
So +tilt in the log means one thing: the main thread stopped for a couple of seconds. It does not mean the network is down. It does not mean the instance is down. It means Sentinel itself was not running.
The mechanism
Sentinel resolves hostnames synchronously, on the main loop, with a blocking getaddrinfo.
The chain in src/sentinel.c and src/anet.c is createSentinelAddr to anetResolve to getaddrinfo. On the 8.1 branch that is sentinel.c:563 into anet.c:348-366. On 9.0 it is sentinel.c:579 into anet.c:347-365. There is no async DNS, no resolver thread, and no thread pool anywhere in sentinel.c on either branch. I grepped.
If the resolver takes three seconds to answer, the main loop is frozen for three seconds. Actually longer. glibc's getaddrinfo issues both an A and an AAAA query, and if the resolver serializes them you block for roughly twice the per query latency. The harness measured 3004ms for a 1500ms setting, and the resolver log shows both queries taking the full delay:
query name=primary.sentrepro.test type=AAAA latency_ms=3000 answer=NODATA
query name=primary.sentrepro.test type=A latency_ms=3000 answer=10.83.7.10
A three second stall on a loop with a two second TILT threshold is a TILT every time.
There is one nuance that matters for reproduction. Resolution results are cached. Sentinel re-resolves a monitored instance in sentinelReconnectInstance, and only while that link is disconnected (8.1: 2338-2344, 9.0: 2357-2363). So a healthy loop with all links up does not resolve anything on every tick. The stall is driven by reconnect churn. A short down-after-milliseconds plus slow DNS bootstraps that churn into a feedback loop: a link drops, Sentinel resolves to reconnect, the resolve stalls the loop, other links time out, more resolves. That is why the harness sets down-after-milliseconds to 5000. Not to cheat, but to make the churn happen inside a three minute observation window.
There is a second resolution path that is not gated on reconnect at all. I did not find it until the second round. It gets its own section below.
How I reproduced it
Everything runs in Docker Compose, namespaced so it cannot touch anything else on the host. No host DNS, no host networking, no host resolv.conf involved. Topology: one primary, two replicas, three Sentinels, quorum 2, down-after-milliseconds 5000.
Three pieces make it work.
A controllable slow resolver. A small DNS server (built on dnslib) that answers for the test zone with a configurable per query delay. The compose dns: key points every container at it. Set the delay to 0 and it is a normal resolver. Set it to 3000 and every lookup takes three seconds. It also logs every query with its latency, which turns out to be the most useful evidence in the whole setup, because it tells you exactly which name was resolved and by whom.
A loop latency prober. A dependency free client that speaks raw RESP over a warm socket to sentinel1. It sends PING every 100ms and INFO every second and records the round trip of each. It is TLS aware. Crucially, it connects to the Sentinel by IP, so its own name resolution can never block and contaminate the measurement.
Why is PING RTT on a warm connection a genuine event loop stall signal? Because PING on Sentinel is answered on the same single thread that runs the Sentinel timer. There is no worker pool to hide behind. If the loop is blocked in getaddrinfo, the PING sits in the socket buffer until the loop comes back. A 20 second PING RTT is not network latency on a Docker bridge. It is 20 seconds of the main thread not running. The harness counts PINGs over 2000ms as "gaps", and a gap is by definition long enough to trip TILT.
Throwaway TLS. A local CA and per node certs with SANs covering both the FQDN and the static IP, so TLS can be turned on and off as an independent axis without cert errors muddying the result.
Each scenario runs for a 180 second observation window. A collector parses the Sentinel logs for +tilt, +sdown, +odown, computes RTT stats from the prober CSV, and joins them by timestamp so each Sentinel event has a nearest PING sample attached.
Before any break scenario, a baseline with fast DNS, hostnames, and TLS has to come back clean. It did: 0 tilt, 0 sdown, 1844 pings, RTT max 6.3ms, p99 0.6ms, zero gaps. The harness does not produce false positives on its own.
What actually breaks it
One variable at a time, on 8.1.8. "Slow" is 3000ms per query. RTT in ms.
| scenario | addressing | TLS | DNS | resolver | tilt | sdown | rtt max | rtt p99 | gaps >2s | pings |
|---|---|---|---|---|---|---|---|---|---|---|
| baseline | FQDN | on | fast | correct | 0 | 0 | 6.3 | 0.6 | 0 | 1844 |
| A | FQDN | on | slow | correct | 57 | 0 | 20019.3 | 19819.1 | 1 | 2 |
| B | FQDN | on | fast | correct | 0 | 0 | 3.9 | 0.6 | 0 | 1844 |
| C | raw IP | off | slow | correct | 0 | 0 | 1.0 | 0.4 | 0 | 1843 |
| D | raw IP | on | slow | correct | 0 | 0 | 2.2 | 0.6 | 0 | 1843 |
| E | FQDN | off | slow | correct | 20 | 2 | 20019.8 | 20019.8 | 8 | 9 |
Read it as pairs.
B versus A: same hostnames, same TLS. The only change is DNS latency. B is clean. A has 57 TILT events and the prober managed 2 PINGs in three minutes because the loop was so wedged it could not even complete a TLS handshake. That collapsed sample count is itself the signal.
A versus D: same slow DNS, same TLS. D addresses everything by raw IP. D is clean, RTT max 2.2ms. With no hostname there is nothing to resolve, so slow DNS never touches the loop.
D versus E is the direct TLS versus DNS test. E has slow DNS and no TLS: it breaks, 20 tilt. D has TLS and no resolution: it is clean. On this evidence, blocking DNS is both sufficient and necessary for the break, and TLS is neither.
C is the raw IP plus plaintext workaround from the setup I started with. It is clean for the same reason D is clean. Dropping TLS did nothing. Dropping hostname resolution did everything.
The prober trace from A shows what a wedged loop looks like from the outside:
15:29:48.155 ping ok 0.1ms
15:30:08.274 ping 20019.3ms TimeoutError <- loop wedged
15:29:45.147 1:X ... # +tilt #tilt mode entered (57 total across sentinels)
At 3000ms the stall is total, which is good for proving the break and bad for seeing the correlation. So I added two characterization runs at lower latency, labeled as such and not part of the original matrix:
| scenario | DNS latency | tilt | sdown | rtt max | rtt p99 | gaps >2s | pings |
|---|---|---|---|---|---|---|---|
| Amild | 1000ms | 175 | 0 | 10017.4 | 9707.3 | 31 | 32 |
| Amild2 | 500ms | 131 | 0 | 4017.0 | 4016.7 | 54 | 64 |
At 500ms the loop stalls intermittently instead of wedging, the prober gets a real time series, and every +tilt lands within tens of milliseconds of a multi second PING:
16:14:16.013 sentinel2 +tilt | nearest_ping rtt_ms=2509.6 dt_ms=14
16:14:19.010 sentinel3 +tilt | nearest_ping rtt_ms=4009.9 dt_ms=1027
# prober.csv around that window: ping 2007.7, 2509.4, 3508.8, 4013.5 ms
That is the mechanism, observed: DNS latency, loop RTT climbs past two seconds, TILT.
Two things that are easy to get wrong
TILT masks sdown
Look at the sdown column in every slow DNS run. A: 0. E: 2. Amild: 0. Amild2: 0. TILT ran into the dozens and hundreds, and sdown stayed near zero.
That is not noise. It is forced by the architecture. To mark an instance sdown, Sentinel needs to see it miss down-after-milliseconds worth of PINGs, 5000ms here. Any loop stall long enough to produce a 5 second gap is already far past the 2 second TILT trigger. TILT fires first, and TILT short circuits the sdown check. On 8.1 and 9, loop starvation from slow DNS essentially cannot produce sdown. It suppresses it.
So the original symptom, +sdown on instances that are actually up, is not explained by the stall. It is a different failure with a different signature. To show it, I ran a split horizon scenario: fast DNS, hostnames, TLS, but the resolver returns an unreachable address for the primary.
| scenario | DNS | resolver | tilt | sdown | rtt max | rtt p99 |
|---|---|---|---|---|---|---|
| split | fast | unreachable IP | 0 | 3 | 0.8 | 0.6 |
Three real sdown events, zero TILT, and a loop that never blinked:
15:49:16.194 sentinel3 +sdown master mymaster ... | nearest_ping rtt_ms=0.2 dt_ms=40
15:49:16.245 sentinel1 +sdown master mymaster ... | nearest_ping rtt_ms=0.2 dt_ms=-11
The two modes are distinguishable from the prober alone. Starvation: +tilt present, RTT in seconds, gaps over 2s. Misdirected or unreachable resolution: +sdown present, +tilt absent, RTT flat and sub millisecond. If you see both in the same log, you most likely have both problems at once: slow DNS stalling the loop, and some name resolving to something unreachable. They need separate fixes.
I also tested whether a broken cert SAN contributes. A CA signed cert with a deliberately wrong SAN stayed healthy with zero SSL errors, because these Sentinel links validate the chain to the CA but do not enforce hostname matching by default. Not a contributor here.
An FQDN in announce-ip bites the peers, not the announcer
The setup I started with had sentinel announce-ip set to an FQDN. My first pass through the source said this could not matter: announce-ip is stored literally and the announcer never resolves it. That reading was correct, and my conclusion from it was wrong.
The announcer does not resolve its own announce-ip. But it gossips it. Every Sentinel periodically publishes a hello message, and token 0 of that message is the announced address. Every other Sentinel receiving that hello runs sentinelProcessHelloMessage, which calls getSentinelValkeyInstanceByAddrAndRunID, which calls createSentinelAddr on the announced string (8.1: 2745 to 1467; 9.0: 2789 to 1486). That is the same createSentinelAddr that ends in a blocking getaddrinfo. And it runs inside the lookup, before the address change comparison, so it runs on every hello, not just when something changed. Whether it resolves at all is gated by the receiver's resolve-hostnames setting.
So with announce-hostnames yes on the announcer and resolve-hostnames yes on the receivers, one Sentinel announcing an FQDN puts a blocking lookup on every peer's main loop, on every hello period, with no reconnect churn required.
To isolate this from the monitored instance path, I addressed the primary and replicas by IP in all of these runs, so the announce path is the only thing that can resolve. Slow DNS at 3000ms, TLS on, resolve-hostnames yes.
| scenario | monitor targets | announce-hostnames | announce-ip | tilt | sdown | rtt max | gaps >2s | pings |
|---|---|---|---|---|---|---|---|---|
| R1 | IP | yes | FQDN | 50 | 0 | 20020.1 | 1 | 18 |
| R2 | IP | yes | IP | 0 | 0 | 0.6 | 0 | 1843 |
| R3 | IP | no | IP | 0 | 0 | 1.0 | 0 | 1843 |
| R4 | FQDN | yes | FQDN | 63 | 0 | 20019.4 | 1 | 136 |
R1 versus R2 is the decisive pair. Everything is identical and by IP except the announce-ip form. R1 breaks hard. R2 is clean, and the resolver logged zero slow queries in R2, because nothing in the system issued a DNS query at all. R4 shows the combined case, monitored FQDNs plus announced FQDN, is the worst of the set.
Then a surgical run to show the owner versus peer split directly. Only sentinel1 announces an FQDN. sentinel2 and sentinel3 announce IPs. The prober sits on sentinel2, a peer that has to resolve sentinel1's announced name.
| node | role | +tilt |
|---|---|---|
| sentinel1 | announces its own FQDN | 0 |
| sentinel2 | peer, resolves sentinel1's FQDN | 22 |
| sentinel3 | peer, resolves sentinel1's FQDN | 20 |
The resolver log for this run has 268 slow queries, and every single one is for sentinel1.sentrepro.test. No other name appears. The announcer, which never resolves its own address, records zero TILT. The two peers, which resolve it on every hello, stall and tilt.
This is why "we changed announce-ip to a literal IP and it stopped" is a real fix and not a coincidence. It removed a blocking lookup from every peer's loop. It just was not the lookup anyone expected.
Same on 8.1 and 9
I reran the matrix on Valkey 9.0.6 with the same windows and capture. The source is materially identical on every relevant point; the only diffs in these regions are library renames from redisAsync* to valkeyAsync*.
| scenario | axes | tilt | sdown | rtt max | gaps >2s | pings | 8.1.8 result |
|---|---|---|---|---|---|---|---|
| baseline_v9 | FQDN, TLS, fast DNS | 0 | 0 | 7.7 | 0 | 1844 | 0 / 0 |
| A_v9 | FQDN, TLS, slow | 59 | 0 | 20017.0 | 3 | 4 | 57 / 0 |
| B_v9 | FQDN, TLS, fast DNS | 0 | 0 | 3.8 | 0 | 1844 | 0 / 0 |
| C_v9 | raw IP, no TLS, slow | 0 | 0 | 1.7 | 0 | 1843 | 0 / 0 |
| D_v9 | raw IP, TLS, slow | 0 | 0 | 0.8 | 0 | 1843 | 0 / 0 |
| E_v9 | FQDN, no TLS, slow | 20 | 2 | 20020.1 | 8 | 251 | 20 / 2 |
| R1_v9 | monitor IP, announce FQDN, slow | 49 | 0 | 20018.2 | 1 | 207 | 50 / 0 |
| R2_v9 | monitor IP, announce IP, slow | 0 | 0 | 0.9 | 0 | 1843 | 0 / 0 |
Every cell matches its 8.1.8 counterpart within run to run noise. The per node pattern in R1_v9 is the same peer signature again: sentinel2 18, sentinel3 19, sentinel1 12. Upgrading to 9 does not change the behavior and does not need a different fix. Resolution is still synchronous on the main loop.
The fix
What worked in the reproduction is simple: get the blocking lookup off Sentinel's main loop.
Two things resolve hostnames. The monitored instance addresses, on reconnect. And the addresses peers announce, on every hello, when resolve-hostnames is on. Either address them by IP, or make sure those names answer instantly, from a local caching resolver or /etc/hosts entries on each Sentinel host, so getaddrinfo returns without waiting on the network. Scenarios C, D, and R2 are the IP form of that, all clean under 3000ms DNS.
Keep TLS. D and R2 both ran TLS on with slow DNS and stayed healthy. Dropping to plaintext gives up encryption for nothing. TLS may still add some handshake cost during reconnect churn, but that is a second order effect, not the trigger, and I did not measure it.
Tuning down-after-milliseconds or living with TILT is mitigation, not a fix. It moves the threshold and leaves the stall in place.
This is what the matrix isolated and what fixed the configuration I started with, on both versions. If your DNS latency is spiky rather than uniform, or you run more Sentinels each announcing an FQDN, the numbers shift but the mechanism does not.
Reproduce it yourself
The method is two pieces: a resolver you can make arbitrarily slow, and a client on a warm connection timing PINGs against a Sentinel addressed by IP. Put the resolver in front of a small containerized Sentinel topology, monitor the primary by hostname, and watch PING RTT against the +tilt lines. Then flip one axis at a time: DNS latency, hostname versus IP, TLS on or off, announce-ip form. The resolver's own query log tells you which names are being resolved, and by which node.
The whole harness is on GitHub: github.com/BetterDB-inc/valkey-sentinel-tilt-repro. The slow resolver, the prober, the throwaway TLS, and the scenario scripts are all there. bash scripts/validate.sh to check the setup, then bash scripts/run-scenario.sh baseline for the clean control, then bash scripts/run-matrix.sh A B C D E split san for the isolation matrix.
If you have hit Sentinel TILT in production and it was not this, I would like to hear what it was. And if you have seen the sdown-on-healthy-nodes version, check the loop RTT before you blame the stall.
Line references are against Valkey 8.1.8 and 9.0.6, pinned to commits 9b4ab3b and a100149 so the links stay valid as the source moves on.