I’ve been discussing with the Yugabyte team how to deploy YugabyteDB to Fly. The big hurdle is that the DB masters need to have either a fixed IP or hostname that they can be referenced by, which persists across deploys.
Does Fly guarantee that the VM that claims a given storage volume will always have the same private IP?
The alternative we’ve settled on is either deploying one master per region so we can do
iad.yb-masters.internal, or making each master a separate Fly app so we can do
Hi @spiffytech I’ll do a quick flyby, until someone with better chops comes along: would something like
<region>.<appname>.internal work for you? Or one of the other possibilities at Fly internal addresses?
<region>.<appname>.internal sounds like it’ll work, but it’s not ideal. It forces a multi-region deployment (which forces cross-region latency inside the cluster), and treating per-region identity as a proxy for per-instance identity feels a bit like a kludge.
None of the other options in the internal networking doc sound like a reliable way to point to whatever instance is holding volume
vol_abc. The per-VM addressing is the closest, but VM IDs change on every deploy.
Ah, I see, you want to distinguish the master from the replicas, sorry, coffee mumble mumble.
I have no idea how Yugabyte does leader election, but each instance knows its own allocation id thanks to an environment variable.
You may have noticed this possibly-interesting thread.
Ah! Sorry for my lack of clarity:
Yugabyte masters expect to be configured with a list of all masters, like
--master_address=host1,host2,host3. YB then expects to always connect with host 1 using the
host1 address (or always using the same IP, if you configured it with IPs instead of hostnames). Meanwhile, host 1 has generated some state including a UUID, and YB assumes a permanent hostname<->UUID relationship like
So in Fly, that means I need a way to have a hostname or IP that never changes, which always resolves the an instance holding same storage volume (so the hostname <-> UUID relationship remains invariant).
The solution in that thread is interesting. So if I had a single Fly app,
yb-masters, and I made each master listen on a different port, Fly would route
yb-masters.internal:7001 to one instance and
yb-masters.internal:7002 to a different instance, depending on which instance is actually listening?
Yugabyte does need me to specify the port for each master, but I’d have to ask them whether the port helps identify the master or only connect to it.
Is a given allocation ID permanently attached to a volume, or would it changed if I redeployed my app?
When you use volumes, you’ll get a fixed private IP. So you can grab the
fdaa:: address when you boot with a volume and treat it as permanent.
One way to use this is to create the volumes you need, launch a stub app that does like
sleep 36000, and then run
fly ips private to see the IPs you were assigned.
Oh, perfect! That makes easy. Thanks!
@spiffytech I’ve tested this (fixed private IP after volume creation) and put an example with yugabyted: 🚀 YugabyteDB on fly.io - I - yugabyted - DEV Community
@kurt if you can have a look and give me feedback is something looks wrong, I’ll appreciate. I’ll build a more production-like solution based on this.