I’ve been discussing with the Yugabyte team how to deploy YugabyteDB to Fly. The big hurdle is that the DB masters need to have either a fixed IP or hostname that they can be referenced by, which persists across deploys.
Does Fly guarantee that the VM that claims a given storage volume will always have the same private IP?
The alternative we’ve settled on is either deploying one master per region so we can do iad.yb-masters.internal, or making each master a separate Fly app so we can do yb-master-1.internal.
Hi @spiffytech I’ll do a quick flyby, until someone with better chops comes along: would something like <region>.<appname>.internal work for you? Or one of the other possibilities at Fly internal addresses?
<region>.<appname>.internal sounds like it’ll work, but it’s not ideal. It forces a multi-region deployment (which forces cross-region latency inside the cluster), and treating per-region identity as a proxy for per-instance identity feels a bit like a kludge.
None of the other options in the internal networking doc sound like a reliable way to point to whatever instance is holding volume vol_abc. The per-VM addressing is the closest, but VM IDs change on every deploy.
Yugabyte masters expect to be configured with a list of all masters, like --master_address=host1,host2,host3. YB then expects to always connect with host 1 using the host1 address (or always using the same IP, if you configured it with IPs instead of hostnames). Meanwhile, host 1 has generated some state including a UUID, and YB assumes a permanent hostname<->UUID relationship like host1 <-> 1f5eafb2-b9e5-49d6-89f6-b72f6ddc4cf1.
So in Fly, that means I need a way to have a hostname or IP that never changes, which always resolves the an instance holding same storage volume (so the hostname <-> UUID relationship remains invariant).
The solution in that thread is interesting. So if I had a single Fly app, yb-masters, and I made each master listen on a different port, Fly would route yb-masters.internal:7001 to one instance and yb-masters.internal:7002 to a different instance, depending on which instance is actually listening?
Yugabyte does need me to specify the port for each master, but I’d have to ask them whether the port helps identify the master or only connect to it.
Is a given allocation ID permanently attached to a volume, or would it changed if I redeployed my app?
When you use volumes, you’ll get a fixed private IP. So you can grab the fdaa:: address when you boot with a volume and treat it as permanent.
One way to use this is to create the volumes you need, launch a stub app that does like sleep 36000, and then run fly ips private to see the IPs you were assigned.
@spiffytech I’ve tested this (fixed private IP after volume creation) and put an example with yugabyted: 🚀 YugabyteDB on fly.io - I - yugabyted - DEV Community @kurt if you can have a look and give me feedback is something looks wrong, I’ll appreciate. I’ll build a more production-like solution based on this.