Running LiveKit (livekit-server + livekit-sip) on Fly: the `fly-global-services` UDP bind requirement, and a specific fix

[catchshubu]

If you’re trying to self-host LiveKit’s livekit-server and/or livekit-sip on Fly and finding that TCP/signaling works fine but every UDP path (TURN, the ICE mux, SIP signaling) is silently unreachable from the outside - this is (probably) not a Fly bug, and there’s a concrete fix at the LiveKit config level.

Root cause

Fly’s own docs ( Running apps on UDP and TCP · Fly Docs ) state that a UDP-listening app must bind the special fly-global-services address injected into /etc/hosts - not the wildcard (0.0.0.0/::) - for replies to go out with the correct source address. Binding wildcard lets you receive packets, but replies leave with the wrong source IP and never make it back to the client.

We confirmed this empirically with a minimal Python UDP echo server before touching LiveKit at all:

• bound to 0.0.0.0 → completely unreachable from a real external client
• bound explicitly to the resolved fly-global-services IP → worked immediately

LiveKit’s default config binds its UDP listeners to the wildcard, so it hits exactly this issue out of the box.

The fix (livekit-server)

LiveKit actually has two independent UDP bind paths, and both need the fix:

1. ICE mux (rtc.udp_port) - binds one socket per local interface IP unless restricted. Fix:

   rtc:
     ips:
       includes:
         - "<fly-global-services-ip>/32"
   

2. Built-in TURN server (turn.udp_port + the relay port range) - has a completely separate bind path (pkg/service/turn.go, plain net.ListenPacket) that rtc.ips does not cover. Fix:

   turn:
     bind_addresses:
       - "<fly-global-services-ip>"
   

fly-global-services resolves to a different IP per machine (it’s the “secondary” address in the app’s private /29), so we resolve it at boot from /etc/hosts via a small entrypoint wrapper and substitute it into the config before starting the real binary - it’s not something you can hardcode at build time.

Verified fixed via a raw STUN probe against the TURN port (real response) and a full lk room join --publish-demo from livekit-cli (real ICE connection + published media, where before we’d get could not connect after timeout).

livekit-sip — partial success, open question

For livekit-sip, the equivalent fix is listen_ip: <fly-global-services-ip> (governs the SIP signaling UDP/TCP bind in pkg/sip/server.go). We confirmed at the kernel level (/proc/net/udp) that this correctly binds the socket to the right address, with no leftover wildcard/primary-address socket.

However, unlike livekit-server, inbound UDP still isn’t reaching it - we polled /proc/net/snmp’s Udp: InDatagrams counter for several minutes and it stayed at 0 for our own probe traffic, even though background internet SIP-scanner noise (constant on port 5060) keeps showing up as unrelated NoPorts ICMP responses on the same machine. Same Fly org/region, same “single socket bound only to fly-global-services, no wildcard” shape that worked for livekit-server - just doesn’t work here.

Also worth noting for anyone using SIP trunking specifically: livekit-sip’s RTP media port listener (pkg/sip/media_port.go, NewMediaPortWith) hardcodes a bind to 0.0.0.0 with no config override in the current release - that path can’t be fixed from the yaml at all right now.

Question for anyone who’s gotten livekit-sip fully working on Fly: did you hit this, and if so what made the difference?