Are you still getting these nxdomain errors?
If so, it might be interesting to use Wireshark to inspect the full request packets:
apt-get install --no-install-recommends tshark dnsutils
tshark -i eth0 -f "port 53"
Then, in a separate ssh session:
dig AAAA fly.io
You should see lines like the following:
59 498.862588529 fdaa:3:3169:a7b:21a1:56cf:c280:2 → fdaa::3 DNS 109 Standard query 0x3019 AAAA fly.io OPT
60 498.864110074 fdaa::3 → fdaa:3:3169:a7b:21a1:56cf:c280:2 DNS 125 Standard query response 0x3019 AAAA fly.io AAAA 2a09:8280:1::a:791 OPT
The first is the request, and the second the response—with addresses in <source> → <destination> format. [It always surprises me initially to see those two get swapped in the second line, but it does make sense; for responses, it’s the server—fdaa::3—that’s sending.]
The part at the end of the first line, after the AAAA, might be illuminating in the case of the nxdomaining query. Perhaps there are lingering glitches in how IPv6-related things are being parsed/reconstructed.
Another possibility is a bypass of the fdaa::3 resolver, generally similar to what was mentioned in the following:
“Some apps do their own DNS resolution and know nothing of the wireguard names”
In that case, it would be → 1.1.1.1 or the like in the request.
Anyway, hope this helps!