I have successfully deployed an IKEv2 VPN server on Fly.io using strongSwan and dnsmasq. The VPN tunnel establishes and appears to be working (heavy ESP traffic, client gets an IP from the VPN pool). Although, my iOS device has no internet access after connecting. I strongly suspect the issue is related to DNS resolution, because an nslookup
from the device times out when connected also.
I’ve tried to find as much information from the forums and docs as I can that relates to UDP/TCP, VPN, DNS, but I’m at a loss on this.
What’s Working
- VPN Tunnel: Establishes and maintains connection reliably.
- Traffic Flow:
tcpdump
shows traffic flowing through the tunnel. - dnsmasq Responsiveness (Local):
dig @172.19.14.195 google.com
run from within the server shows dnsmasq responds perfectly to queries. - External DNS through Tunnel: If the client is configured to use external DNS (e.g., 8.8.8.8) through the VPN, it works (traffic is routed and responses received).
- NAT/Routing:
iptables -t nat -L -n -v
confirms MASQUERADE rule is active for10.8.0.0/16
outbound oneth0
.
What’s Broken
- No internet access: Despite VPN connection, the client device cannot browse the internet or resolve any domain names.
- DNS resolution via VPN server fails: When the VPN client is configured to use the VPN server’s DNS (172.19.14.195),
nslookup
on the client times out with “no servers could be reached”. - DNS queries fail with and without dnsmasq.
tcpdump Analysis (on VPN server)
When the client (10.8.0.1) attempts DNS resolution:
# VPN client queries to 172.19.14.195:53 ARRIVE at server:
10.8.0.1.52805 > 172.19.14.195.53: 26530+ A? google.com. (28)
# However, there are NO responses observed originating from 172.19.14.195:53 back to 10.8.0.1.
# Only DNS traffic to external servers (like 8.8.8.8) shows responses:
10.8.0.1.52647 > 8.8.8.8.53: 26530+ A? google.com. (28)
8.8.8.8.53 > 172.19.14.194.52647: 26530 1/0/0 google.com. A 142.250.189.14
Configuration Files
fly.toml
...
[experimental]
allowed_public_ports = [53]
[[services]]
protocol = 'udp'
internal_port = 500
processes = ['app']
[[services.ports]]
port = 500
[[services]]
protocol = 'udp'
internal_port = 4500
processes = ['app']
[[services.ports]]
port = 4500
[[services]]
protocol = 'udp'
internal_port = 53
processes = ['app']
[[services.ports]]
port = 53
[[services]]
protocol = 'tcp'
internal_port = 53
processes = ['app']
[[services.ports]]
port = 53
...
etc/ipsec.conf
config setup
conn %default
left=172.19.14.195
leftid=%any
ikelifetime=60m
keylife=20m
rekeymargin=3m
keyingtries=1
keyexchange=ikev2
authby=secret
# Modern encryption algorithms
ike=aes256-sha256-modp2048,aes128-sha256-modp2048,aes256-sha1-modp2048,aes128-sha1-modp2048!
esp=aes256-sha256,aes128-sha256,aes256-sha1,aes128-sha1!
conn rw
# http://wiki.loopop.net/doku.php?id=server:vpn:strongswanonopenvz
# https://wiki.strongswan.org/projects/strongswan/wiki/ForwardingAndSplitTunneling
leftsubnet=0.0.0.0/0,::/0
# end ref
leftfirewall=no
right=%any
rightid=%any
rightsourceip=10.8.0.0/16,fd6a:6ce3:c8d8:7caa::/64
auto=add
etc/strongswan.conf
charon {
load_modular = yes
install_routes = no
install_virtual_ip = no
syslog {
daemon {
}
auth {
default = 2
esp = 1
knl = 1
net = 1
enc = 1
}
}
plugins {
include strongswan.d/charon/*.conf
# https://wiki.strongswan.org/projects/strongswan/wiki/Attrplugin
# attr {
# dns = 8.8.8.8, 8.8.4.4
# }
# Kernel interface configuration for containers
kernel-netlink {
set_proto_port_transport_sa = yes
}
}
}
starter {
load_warning = no
}
include strongswan.d/*.conf
etc/ndppd.conf
proxy eth0 {
rule fd6a:6ce3:c8d8:7caa::/64 {
static
}
}
etc/dnsmasq.conf:
(Doesn’t matter what address is being listened on, localhost, 0.0.0.0, fly-global-services, or 172.19.14.195)
# dnsmasq configuration for VPN clients
listen-address=0.0.0.0
# Don't read /etc/hosts
no-hosts
# Don't read /etc/resolv.conf
no-resolv
# Use these upstream DNS servers
server=8.8.8.8
server=1.1.1.1
server=208.67.222.222
# Cache size
cache-size=1000
# Don't forward plain names (without a dot or domain part)
domain-needed
# Never forward addresses in the non-routed address spaces
bogus-priv
# Log queries for debugging
log-queries
# Don't daemonize
keep-in-foreground
.mobileconfig
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>PayloadDisplayName</key>
<string>{PAYLOAD_DISPLAY_NAME}</string>
<key>PayloadIdentifier</key>
<string>{PAYLOAD_IDENTIFIER}</string>
<key>PayloadUUID</key>
<string>{PAYLOAD_UUID}</string>
<key>PayloadType</key>
<string>Configuration</string>
<key>PayloadVersion</key>
<integer>1</integer>
<key>PayloadContent</key>
<array>
<dict>
<key>PayloadIdentifier</key>
<string>{PAYLOAD_IDENTIFIER}</string>
<key>PayloadUUID</key>
<string>{PAYLOAD_UUID}</string>
<key>PayloadType</key>
<string>com.apple.vpn.managed</string>
<key>PayloadVersion</key>
<integer>1</integer>
<key>UserDefinedName</key>
<string>My IKEv2 VPN</string>
<key>VPNType</key>
<string>IKEv2</string>
<key>IKEv2</key>
<dict>
<key>RemoteAddress</key>
<string>{PUBLIC_SERVER_IP}</string>
<key>RemoteIdentifier</key>
<string>{PUBLIC_SERVER_IP}</string>
<key>LocalIdentifier</key>
<string></string>
<key>OnDemandEnabled</key>
<integer>1</integer>
<key>OnDemandRules</key>
<array>
<dict>
<key>Action</key>
<string>Connect</string>
</dict>
</array>
<key>AuthenticationMethod</key>
<string>SharedSecret</string>
<key>SharedSecret</key>
<string>{SHARED_SECRET}</string>
<key>ExtendedAuthEnabled</key>
<integer>0</integer>
<key>AuthName</key>
<string></string>
<key>AuthPassword</key>
<string></string>
<key>IncludeAllNetworks</key>
<true />
<key>DisableIPv6</key>
<true />
<key>ChildSecurityAssociationParameters</key>
<dict>
<key>EncryptionAlgorithm</key>
<string>AES-256</string>
<key>IntegrityAlgorithm</key>
<string>SHA2-256</string>
<key>DiffieHellmanGroup</key>
<integer>14</integer>
<key>LifeTimeInMinutes</key>
<integer>1440</integer>
</dict>
<key>IKESecurityAssociationParameters</key>
<dict>
<key>EncryptionAlgorithm</key>
<string>AES-256</string>
<key>IntegrityAlgorithm</key>
<string>SHA2-256</string>
<key>DiffieHellmanGroup</key>
<integer>14</integer>
<key>LifeTimeInMinutes</key>
<integer>1440</integer>
</dict>
</dict>
</dict>
</array>
</dict>
</plist>
Questions
-
UDP Binding on Fly.io: Given the
tcpdump
output, should dnsmasq bind to the specificfly-global-services
IP (172.19.14.195
) or0.0.0.0
for VPN client traffic originating from the10.8.0.0/16
subnet? We initially tried172.19.14.195
and are about to test0.0.0.0
. -
VPN Traffic Routing/NAT: Are there any special considerations for UDP DNS traffic that originates from the VPN tunnel (e.g.,
10.8.0.x
to172.19.14.195:53
) on Fly.io’s network stack? Could Fly.io’s internal networking or firewall rules implicitly be dropping DNS responses to VPN clients, even ifiptables
shows no explicit blocks? -
Interface Binding & VPN: Does
bind-interfaces
+ specific IP (as originally configured in dnsmasq) behave differently for packets routed through the VPN tunnel compared to direct external traffic on Fly.io? -
Fly.io limitations: Are there any limitations that I should be aware of, or should be handling differently?
Any insights on UDP DNS handling for VPN clients on Fly.io, especially regarding internal routing and interface binding, would be greatly appreciated!