Hi,
If I deploy an app to multiple regions, is there an easy way to communicate between regions. Can I use the internal IP of eth0 to talk with each other?
You can indeed talk between regions.
You’ll use a special IP address, which is in /etc/hosts as fly-local-6pn — it’s an IPv6 address that starts with fdaa that can only be used to talk between apps you own, and nobody else.
That sounds complicated, so you should know we provide DNS for this, and that’s what most people use. As long as you’re using the default DNS server we set instances up with (fdaa::3), all instances of your app are addressable under $appname.internal. If you want to talk only to the instances in Japan, that’s nrt.$appname.internal.
If you want to know what regions are currently deployed, you can look that up in the DNS too: dig txt regions.$appname.internal.
More here: Incoming! 6PN Private Networks · Fly
2 Likes
Thanks for the solution, that’s really cool but does fly.io have any firewall between internal IPs? My application is trying to talk with each other on REG.$appname.internal. They are not using the public service port. ICMP works, TCP works, but it seems UDP packets are getting dropped?
Another question is it seems I cannot keep two console opening at the same time.
There’s no filter between fdaa addresses, but you won’t be able to talk to an fdaa address of another instance using anything but the fdaa address of your current system. We run DNS and UDP Serf over private networks here all the time, so UDP should work fine.
The two consoles thing is a known problem! ssh console does a ludicrous amount of stuff; it’s running a full TCP/IP and WireGuard stack inside flyctl itself. When you try to run the console twice, you’re essentially trying to bring up the same WireGuard connection in two different places, which of course doesn’t work.
We’re noodling about solutions for this. In the meantime, though, if you want to open multiple consoles, the thing to do is to use flyctl to generate an SSH certificate (or load one into your local SSH agent), and then use standard WireGuard to connect to your private network.
2 Likes