Fix resolver list getting stuck after a partial bootstrap result

[emgeorrk]

The problem we ran into

We run https_dns_proxy on an OpenWrt 25.12.5 router (aarch64) with an IPv4-only uplink. The default route goes through a VPN tunnel. After a short tunnel flap, both instances started by the OpenWrt init script (Cloudflare and Google) stopped answering and stayed that way for 37 hours, until we killed them. The whole LAN had no DNS. Meanwhile the tunnel, the routes, plain UDP DNS to 1.1.1.1 and curl https://1.1.1.1/dns-query?... from the same router all worked. Both processes were alive and idle, had no upstream sockets open, their UDP listeners drained the queue with no drops, and nothing was logged at the default verbosity.

OpenWrt ships 0ba0525 as package 2026.05.06. Current master f8e13aa behaves the same way.

Root cause

Two things combine:

1. A partial poll result replaces the whole resolver list. c-ares has its query cache on by default since 1.31. While the network is down, ares_getaddrinfo() with AF_UNSPEC succeeds with whichever of the A/AAAA records is still cached, while the other query times out. doh_proxy_handle_resolver_update() installs that partial list.
2. The "unchanged" check only looks one way. addr_list_reduced(addr_list, old_addr_list) only checks that every old address is in the new list. When the network is back, the full list arrives, the shrunken old list is a subset of it, and the update is reported as unchanged and dropped. The list never grows back.

If the AAAA records are the ones that survive, an IPv4-only host is left with IPv6-only addresses. Every request then fails with CURLE_COULDNT_CONNECT before a connection is even attempted. No reply is sent, the reset timer is never armed, and nothing is logged at the default level.

Log from the router with the OpenWrt build, while all outbound traffic of the proxy was dropped with nftables for 6 minutes (times UTC+3):

17:59:27 dns_poller.c:96  Received 2 IPv4 and 2 IPv6 addresses
17:59:27 doh_proxy.c:135  Received new DNS server IP '104.16.248.249,104.16.249.249,2606:4700::6810:f9f9,2606:4700::6810:f8f9'
         (uplink cut at 17:59:30)
18:01:27 dns_poller.c:96  Received 0 IPv4 and 2 IPv6 addresses
18:01:27 doh_proxy.c:96   IP address missing: 104.16.248.249
18:01:27 doh_proxy.c:135  Received new DNS server IP '2606:4700::6810:f9f9,2606:4700::6810:f8f9'
18:01:30 https_client.c:364 34E2: curl request failed with 7: Error
         (from now on every request: "Failed to connect to cloudflare-dns.com:443 after 0 ms", CURLINFO_OS_ERRNO 101)
18:03:27 dns_poller.c:114 DNS lookup of 'cloudflare-dns.com' failed: Could not contact DNS servers
         (repeats every 5 s; uplink restored at 18:05:30)
18:05:32 dns_poller.c:96  Received 2 IPv4 and 2 IPv6 addresses
18:05:32 doh_proxy.c:128  DNS server IP address unchanged (104.16.248.249,104.16.249.249,2606:4700::6810:f8f9,2606:4700::6810:f9f9).
18:07:32 (the same again; the proxy never answers)

What this PR changes

• src/doh_proxy.c: compare the old and the new address list in both directions, and match addresses as whole list elements, so 192.0.2.1 no longer matches the start of 192.0.2.10. The prefix case only matters once the check is two-way: a reordered list would otherwise trigger a needless client reset.
• src/dns_poller.c: disable the c-ares query cache in the poller (ARES_OPT_QUERY_CACHE with qcache_max_ttl = 0, behind #ifdef for older c-ares). The poller re-resolves on its own schedule, so the cache only serves stale partial answers during an outage.
• Tests: tests/unit/test_resolver_update.c covers the update logic, and test_dns_poller.c checks that the query cache is off.

Reproduction

The bug is deterministic with a fake bootstrap DNS server that gives A records a 5 s TTL and AAAA records 3600 s, and then goes silent for a while. The poll at t=10 s is then answered from the c-ares cache with the AAAA records only. No real outage is needed, only an IPv4-only host: a Docker container on the default bridge network is one.

docker run --rm -it alpine:3.22 sh
apk add build-base cmake ninja git c-ares-dev libev-dev curl-dev python3 bind-tools
git clone https://github.com/aarond10/https_dns_proxy && cd https_dns_proxy
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCLANG_TIDY_EXE= && cmake --build build
# save the two scripts below as repro.sh and fake_bootstrap_dns.py, then:
VARIANTS="master|./build/https_dns_proxy|" \
BOOT="127.0.0.1:5300,127.0.0.2:5300,2606:4700:4700::1111,2606:4700:4700::1001" \
OUTAGE_END=64 sh repro.sh

BOOT mirrors the bootstrap list that OpenWrt passes: two reachable IPv4 servers and two IPv6 ones that fail at once. Output on master:

== before the outage
t=3s     master=OK
== bootstrap DNS silent from t=6s
== bootstrap DNS back at t=64s
== after the outage
t=80s    master=FAIL
== 30 s later
t=111s   master=FAIL
== master: resolver list updates (t relative to start)
t= 1.4s Received 2 IPv4 and 2 IPv6 addresses
t= 1.4s Received new DNS server IP '104.16.248.249,104.16.249.249,2606:4700::6810:f8f9,2606:4700::6810:f9f9'
t= 27.9s Received 0 IPv4 and 2 IPv6 addresses
t= 27.9s Received new DNS server IP '2606:4700::6810:f8f9,2606:4700::6810:f9f9'
t= 55.4s Received 0 IPv4 and 2 IPv6 addresses
t= 55.4s DNS server IP address unchanged (2606:4700::6810:f8f9,2606:4700::6810:f9f9).
t= 65.4s Received 2 IPv4 and 2 IPv6 addresses
t= 65.4s DNS server IP address unchanged (104.16.248.249,104.16.249.249,2606:4700::6810:f8f9,2606:4700::6810:f9f9).
t= 75.4s Received 2 IPv4 and 2 IPv6 addresses
t= 75.4s DNS server IP address unchanged (104.16.248.249,104.16.249.249,2606:4700::6810:f8f9,2606:4700::6810:f9f9).
   (the same every 10 s from here on; every request fails with "Failed to connect ... after 0 ms")

repro.sh

#!/bin/sh
# Deterministic reproduction of the https_dns_proxy resolver-list bug.
# Needs an IPv4-only host (a Docker container on the default bridge network is
# one), python3 and dig. Each line of VARIANTS is "name|binary|extra args".
# The fake bootstrap server gives A records a 5 s TTL and AAAA records 3600 s,
# then goes silent from t=4 s to t=OUTAGE_END. The poll at t=10 s is answered
# from the c-ares cache with the AAAA records only. With several bootstrap
# servers a lookup retries for longer, so OUTAGE_END must be larger (60+ s).
VARIANTS=${VARIANTS:-"master|./https_dns_proxy|"}
BOOT=${BOOT:-127.0.0.1:5300}
OUTAGE_END=${OUTAGE_END:-29}
HERE=$(cd "$(dirname "$0")" && pwd)
OUT=${OUT:-/tmp/hdp-repro}; rm -rf "$OUT"; mkdir -p "$OUT"; rm -f /tmp/outage
python3 "$HERE/fake_bootstrap_dns.py" --listen 0.0.0.0 --port 5300 > "$OUT/fake_dns.log" 2>&1 & DNSPID=$!
sleep 1
PIDS=""; PORTS=""; port=5400
echo "$VARIANTS" | while IFS='|' read -r name bin extra; do [ -n "$name" ] && echo "$name|$bin|$extra"; done > "$OUT/variants"
while IFS='|' read -r name bin extra; do
  port=$((port + 1))
  # shellcheck disable=SC2086
  "$bin" -r https://cloudflare-dns.com/dns-query -p $port -b "$BOOT" -i 10 -vvv $extra > "$OUT/$name.log" 2>&1 &
  PIDS="$PIDS $!"; PORTS="$PORTS $port:$name"
done < "$OUT/variants"
T0=$(date +%s)
at() { now=$(date +%s); [ $((T0 + $1)) -gt "$now" ] && sleep $((T0 + $1 - now)); }
check() {
  printf '%-8s' "t=$(( $(date +%s) - T0 ))s"
  for pn in $PORTS; do
    p=${pn%%:*}; n=${pn#*:}
    if dig +short +time=2 +tries=1 -p "$p" @127.0.0.1 example.com A | grep -q '^[0-9]'; then r=OK; else r=FAIL; fi
    printf ' %s=%s' "$n" "$r"
  done
  echo
}
at 3;  echo "== before the outage";  check
touch /tmp/outage; echo "== bootstrap DNS silent from t=$(( $(date +%s) - T0 ))s"
at "$OUTAGE_END"; rm -f /tmp/outage; echo "== bootstrap DNS back at t=$(( $(date +%s) - T0 ))s"
at $((OUTAGE_END + 16)); echo "== after the outage"; check
at $((OUTAGE_END + 46)); echo "== 30 s later";       check
kill $PIDS $DNSPID 2>/dev/null; wait 2>/dev/null
for pn in $PORTS; do n=${pn#*:}
  echo "== $n: resolver list updates (t relative to start)"
  grep -E "Received [0-9]+ IPv4|Received new DNS server IP|IP address unchanged" "$OUT/$n.log" |
    awk -v t0="$T0" '{ $2 = sprintf("t=%5.1fs", $2 - t0); $1 = ""; $3 = ""; print }' | sed 's/^ *//; s/  */ /g' | cut -c1-120
  echo "   instant connect failures: $(grep -c 'failed with 7' "$OUT/$n.log")"
done

fake_bootstrap_dns.py

#!/usr/bin/env python3
"""Tiny bootstrap DNS server for reproducing the https_dns_proxy resolver-list bug.

Answers A and AAAA queries for any name with fixed addresses and a separate TTL
per record type. While the file given by --outage-file exists, every query is
dropped, which looks like a network outage to the client.
"""
import argparse
import os
import socket
import struct
import time


def question_end(msg):
    off = 12
    while msg[off] != 0:
        if msg[off] & 0xC0:
            raise ValueError("compressed name in query")
        off += 1 + msg[off]
    return off + 1 + 4  # root label, QTYPE, QCLASS


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--listen", default="127.0.0.1")
    ap.add_argument("--port", type=int, default=5300)
    ap.add_argument("--a", default="104.16.248.249,104.16.249.249")
    ap.add_argument("--aaaa", default="2606:4700::6810:f8f9,2606:4700::6810:f9f9")
    ap.add_argument("--ttl-a", type=int, default=5)
    ap.add_argument("--ttl-aaaa", type=int, default=3600)
    ap.add_argument("--outage-file", default="/tmp/outage")
    args = ap.parse_args()

    records = {
        1: (args.ttl_a, [socket.inet_pton(socket.AF_INET, a) for a in args.a.split(",") if a]),
        28: (args.ttl_aaaa, [socket.inet_pton(socket.AF_INET6, a) for a in args.aaaa.split(",") if a]),
    }
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind((args.listen, args.port))
    print(f"listening on {args.listen}:{args.port}", flush=True)
    while True:
        msg, peer = sock.recvfrom(4096)
        try:
            end = question_end(msg)
            qid, flags = struct.unpack("!HH", msg[:4])
            qtype = struct.unpack("!H", msg[end - 4:end - 2])[0]
        except (IndexError, ValueError, struct.error):
            continue
        stamp = time.strftime("%H:%M:%S")
        if os.path.exists(args.outage_file):
            print(f"{stamp} drop  qtype={qtype}", flush=True)
            continue
        ttl, rdatas = records.get(qtype, (0, []))
        header = struct.pack("!HHHHHH", qid, 0x8080 | (flags & 0x0100), 1, len(rdatas), 0, 0)
        answers = b"".join(struct.pack("!HHHIH", 0xC00C, qtype, 1, ttl, len(r)) + r for r in rdatas)
        sock.sendto(header + msg[12:end] + answers, peer)
        print(f"{stamp} reply qtype={qtype} answers={len(rdatas)} ttl={ttl}", flush=True)


if __name__ == "__main__":
    main()

Five builds side by side in the same setup:

Build	Resolver list during the outage	After the outage (t=80 s and t=111 s)
0ba0525, the OpenWrt package	IPv6 only from t=28 s	stuck on IPv6 only, no answers
master f8e13aa	IPv6 only from t=28 s	stuck on IPv6 only, no answers
master with -4	unchanged	answers
#214 at c574e4b	IPv6 only from t=27 s	full list again at t=65 s, answers
this PR	unchanged	answers

With this PR the list was not touched during the outage. One lookup right after it returned only the A records, without any cache involved, and the full list was back 10 s later. That is why both parts are needed: disabling the cache removes stale answers during an outage, but a live lookup can still come back partial at its edge, and only the two-way comparison lets the list grow back.

Testing

• Unit tests on Alpine 3.22 (c-ares 1.34.8, libcurl 8.14.1). Without the source changes the new tests fail: 2 cases in resolver_update and the query-cache case in dns_poller_watchers. With them ctest passes, and valgrind reports no errors and no leaks.
• Built with gcc 14 and with clang 20 plus clang-tidy using the project's check list. No new warnings.
• The reproduction above, run in a fresh container exactly as written.

Relation to #214

#214 also makes the comparison two-way, with the same whole-element matching, but keeps the query cache. This PR is small and independent, so it can be merged on its own, and OpenWrt can pick it up without waiting for the larger refactor. If #214 lands first, I will rebase this down to the query-cache change and the tests.

Notes for users on affected versions

• On IPv4-only hosts, -4 avoids the problem: only A records are queried, so the list cannot shrink to the wrong family.
• On OpenWrt, /etc/init.d/https-dns-proxy restart does not help, because it is a reload and procd keeps an instance whose command line did not change. Killing the process does.
• With a single bootstrap server, 0ba0525 also exits with c-ares needed more IO event handler, than the number of provided nameservers: 1 when the bootstrap server stops answering. That is already fixed on master by 63c49b4, but OpenWrt still ships 0ba0525.

🤖 Generated with Claude Code

[baranyaib90]

this looks OK to me