Some questions on machine downtime and volume attachment

Questions / Help
,
1

@mayailurus says on the frequency downtime happens:

It’s pretty frequent. The Fly.io platform will migrate your Machine to a different underlying physical host, without you asking, which involves a shutdown for a while, and each deploy also stops the Machine for varying amounts of time.

Moreover, congestion can cause a Machine to not be able to start until an auto-migration can happen, and in the past some people have found their (single-Machine) app offline for multiple hours. Volumes can exacerbate this, although the details aren’t really documented. (Last I heard, volumes prevented auto-migration entirely, but I get the impression that’s been relaxed recently.)

I have some further questions:

  • Is there any data so I can know the actual frequency? Which timezone does downtime frequently happen?
  • Why does my machine need to be migrated to another physical host? Would that host be in the same region?
  • If the purpose of having a second machine is to avoid downtime, then it should also attach to the volume, right? So the fact that volumes are not shared between machines defeat that purpose?
  • What counts as rootfs? If I create a custom file or folder under root, is it a part of rootfs? If I set persist_rootfs as always or restart, then would that file or folder be kept, or just changes inside default Linux directories are kept? Would mounting the rootfs to the volume be a way to have it persistent while setting the config to never?
  • I have only one machine and want to start a new one, what do I need to make sure the data is shared? It seems to me that I need to set another machine dedicated for database. This is true if the backup machine is in the same region, and even true if I want it in another region. But then I have to worry that that database machine can have downtime of its own.
4

Just to emphasize, this is about the downtime of an individual Machine, part of the argument against single-Machine apps, :sweat_smile:…

(The platform specifically recommends against them, although maybe not loudly enough, given the weight of everyone’s assumptions in the opposite direction.)

[Also, this is the Machines platform, not the newer Sprites concept.]

The two Machines definitely don’t share the volume; i.e., the platform’s expectations themselves nearly lose their logical self-consistency (when you introduce volumes into the system). This is one reason why volumes are an advanced feature on the Fly.io platform (in my view), even if it doesn’t seem that hard at first glance…

The initial configuration is easy, but the broader design implications are not, in other words.

The app‡ itself, your own code or something you install alongside it within each Machine, would need to actively replicate between the two disks. There’s a sentence to that effect in the (long) list of volume caveats in the official docs. This is the only† way to satisfy the (wise) recommendation of redundant Machines while also having persistent local storage beyond temp files, things that you can re-download from S3 on reboot, etc. In general, this amounts to creating and sysadmining your own distributed database, which is really not simple. Most people should instead use Managed Postgres and/or Tigris, which handle all the multiple nodes and the synchronization among them.

†There can be niche exceptions, but only very experienced users should mount a volume on the Machines platform, in my opinion. (Apart from experiments, personal, single-user projects without any irreplaceable data stored, etc.)

‡“App” generally includes things like a separate multi-node MySQL cluster that you install and maintain yourself, too, which is slightly idiosyncratic terminology. I.e., it’s an app in the very expansive sense that the Machines platform defines. In those cases, you typically rely on the database’s own clustering mechanisms (which count as “app” code, from this perspective) to keep the various nodes up to date.

Exactly. You can look at the design of LiteFS to see the considerations that come into play there. It has at least two database Machines running at all times, and they’re constantly chatting with each other over the internal network, to make sure that both have record of all the latest modifications. Moreover, the freebie Consul cluster is the arbiter of who is the “primary” node (the one that can accept new writes), which is another critical gotcha in this setting.

The primary Machine will go down during deploys, auto-migrations, hardware failure, etc., and the other one needs to always be prepared to both notice that and then take over the main role itself.

(I don’t recommend using LiteFS, per se, at this time; it’s just a relatively small example system for reference.)

Aside: With LiteFS, the database Machines were actually the same as the web-app Machines, which was a nice cost-saving feature of its architecture. That doesn’t really affect the overall thrust here, though.

5

Beyond what @mayailurus has already answered above,

This can happen for a number of reasons. For example, if the original host is suffering a failure, or if we need to rebalance workload between hosts to avoid CPU steal and I/O issues. But,

Unless we specifically notified you about a region deprecation or similar changes, migrations are always local to regions.

Yes, if you don’t create a volume, then whatever you put in /, except in some Linux system mountpoints like /dev, /tmp, /proc, /sys, etc., are counted as rootfs. You can check what mountpoint a path is under by looking at mount output.

Yes but only if machines are restarted or updated, not migrated. These options are intended for cases where the rootfs is used as some kind of cache, in which case a volume is overkill (because cache doesn’t need real persistence), but default rootfs causes cache to be invalidated on every update. They are NOT intended for actual persistent data and there is no guarantee for long term persistency.

Not sure what this means, but you can definitely run a process from inside a volume and just have a barebones rootfs image that executes something in that volume.

The answer here depends on what type of data you have. If you need strongly consistent relational data, then you’d need a relational database, in which case you may consider Managed Postgres or another managed database solution, or you will have to administer your own. If the data you are storing is mostly media blobs, then an object store like AWS S3, Tigris, or any other S3-compatible service should suit your needs best.

Volumes are a low-level primitive that these services are built on top of. For example, our Managed Postgres service heavily depends on volumes. But because volumes are “just” ext4 filesystems, there is really no way to “share” them as-is without knowing your application logic. Managed Postgres replicates data using PostgreSQL’s replication protocol. If you run S3 on top of volumes, that S3 service needs to handle its own replication. An out-of-the-box distributed POSIX-compatible filesystem would be nice, except that it is almost certainly not what an application that actually requires POSIX-compatible semantics would expect, and your app probably would not function properly with “just” a shared filesystem, unless all you are storing is just some media blobs, in which case, as mentioned above, you should probably consider using an object storage service.

Granted I do think it would be fun to explore in the future if we can provide a somewhat volume-like experience but with reduced consistency guarantees to allow sharing. However I do not think that is likely what you actually need here, and I think there is a somewhat tiny subset of use cases that would actually benefit from this vs just using an object store + a managed database.

6

Multi-tenant 9P would be neat for this, :black_cat:.

(The kernel build used in Machines recently got support for 9P.)

It’s a simple enough protocol that a person can realistically implement their own server, with extra security (isolation) emphasis…