I’m trying to create a WireGuard tunnel to the 6PN private network, but DNS only works partially.
Also, fly wireguard status
outputs “Error: You hit a Fly API error with request ID: 01H1Y3BVHZF39MJ7V3VABEG4MB-ord” the first time, and later “Error: upstream service is unavailable”.
The following resolved: <app>.fly.dev, _apps.internal
The following didn’t produce any results: <app>.internal, _instances.internal, _peer.internal
I tried the same steps via https://fly.io/terminal and the results were a little different: <app>.internal and _instances.internal resolved (only after I set up WireGuard), and _peer.internal didn’t produce any results.
Finally, when running the same DNS queries from the demo app’s VM, everything resolved except _peer.internal.
(Look for the lines marked “PROBLEM” below.)
# OS and flyctl versions
$ lsb_release -d
Description: Debian GNU/Linux 11 (bullseye)
$ fly version
flyctl v0.1.24 linux/amd64 Commit: 86409d36fda482b419410a4af3113d12537a06d7 BuildDate: 2023-06-01T11:52:45Z
# install wireguard
$ sudo apt install wireguard openresolv
...
Unpacking openresolv (3.12.0-1) ...
Unpacking wireguard-tools (1.0.20210223-1) ...
Unpacking wireguard (1.0.20210223-1) ...
...
# demo app
$ mkdir app && cd app
$ cat > Dockerfile <<"EOF"
FROM debian:11
CMD sleep infinity
EOF
$ fly launch
...
? Choose a region for deployment: Chicago, Illinois (US) (ord)
...
Platform: machines
...
# disable auto-stop
$ sed -E -i '/^ auto_stop_machines = true$/ s/true/false/' fly.toml
# get the app name
$ app="$(sed -E -n 's/^app = "([^"]*)"$/\1/p' fly.toml)"
$ fly deploy --ha=false
...
$ fly ips list
VERSION IP TYPE REGION CREATED AT
v6 2a09:... public global ...s ago
v4 66.2... public (shared)
# try some DNS queries without wireguard
$ dig +short aaaa $app.fly.dev
2a09:...
$ dig +noall +answer _apps.internal txt # https://fly.io/docs/reference/private-networking/#testing-the-tunnel
# no output (as expected)
$ dig +short aaaa $app.internal
# no output (as expected)
# start wireguard
$ fly wireguard create
...
? Filename to store WireGuard configuration in, or 'stdout': basic.conf
$ sudo cp basic.conf /etc/wireguard/basic.conf
$ sudo wg-quick up basic
[#] ip link add basic type wireguard
[#] wg setconf basic /dev/fd/63
[#] ip -6 address add fdaa:...:202/120 dev basic
[#] ip link set mtu 1420 up dev basic
[#] resolvconf -a basic -m 0 -x
[#] ip -6 route add fdaa:...::/48 dev basic
# check the status of wireguard
$ sudo wg
interface: basic
public key: ...
private key: (hidden)
listening port: 58413
peer: ...
endpoint: 43.2...:51820
allowed ips: fdaa:...::/48
latest handshake: 2 seconds ago
transfer: 92 B received, 180 B sent
persistent keepalive: every 15 seconds
$ sudo systemctl status wg-quick@basic
● wg-quick@basic.service - WireGuard via wg-quick(8) for basic
Loaded: loaded (/lib/systemd/system/wg-quick@.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: ...
# not sure why it shows as "inactive"
$ sudo journalctl
...
... systemd-udevd[849]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
... kernel: wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
... kernel: wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
... systemd-udevd[849]: Using default interface naming scheme 'v247'.
...
$ cat /etc/resolv.conf
# Generated by resolvconf
nameserver fdaa:...::3
$ fly wireguard status
? Select peer: <select interactively>
Error: upstream service is unavailable
# PROBLEM (also get this for peer connections generated automatically by flyctl)
#
# Output with FLY_LOG=debug:
#
# DEBUG --> POST https://api.fly.io/graphql
#
# {
# "query": "query($slug: String!, $name: String!) { organization(slug: $slug) { wireGuardPeer(name: $name) { gatewayStatus } } }",
# "variables": {
# "name": "interactive-...",
# "slug": "personal"
# }
# }
#
# DEBUG {}
# DEBUG <-- 200 https://api.fly.io/graphql (311.56ms)
#
# {
# "data": {
# "organization": null
# },
# "errors": [
# {
# "message": "upstream service is unavailable",
# "locations": [
# {
# "line": 1,
# "column": 98
# }
# ],
# "path": [
# "organization",
# "wireGuardPeer",
# "gatewayStatus"
# ],
# "extensions": {
# "code": "SERVICE_UNAVAILABLE"
# }
# }
# ]
# }
# try some DNS queries with wireguard
$ dig +short aaaa $app.fly.dev
2a09:...
$ dig +noall +answer _apps.internal txt
_apps.internal. 5 IN TXT "fly-builder-...,<app>"
$ dig +noall +answer _instances.internal txt
# no output (PROBLEM)
$ dig +short txt _peer.internal
# no output (PROBLEM)
$ dig +short aaaa $app.internal
# no output (PROBLEM)
$ ping -4 $app.internal
ping: <app>.internal: No address associated with hostname
$ ping -6 $app.internal
ping: <app>.internal: No address associated with hostname
# try some DNS queries from the VM
$ fly ssh console
# apt update && apt install dnsutils
# dig +short aaaa $FLY_APP_NAME.fly.dev
2a09:...
# dig +noall +answer _apps.internal txt
_apps.internal. 5 IN TXT "fly-builder-...,<app>"
# dig +noall +answer _instances.internal txt
_instances.internal. 5 IN TXT "instance=...,app=<app>,ip=fdaa:...:2,region=ord"
# dig +short txt _peer.internal
# no output (PROBLEM?)
# dig +short aaaa $FLY_APP_NAME.internal
fdaa:...:2