Some questions on machine downtime and volume attachment

Questions / Help
,
1

@mayailurus says on the frequency downtime happens:

It’s pretty frequent. The Fly.io platform will migrate your Machine to a different underlying physical host, without you asking, which involves a shutdown for a while, and each deploy also stops the Machine for varying amounts of time.

Moreover, congestion can cause a Machine to not be able to start until an auto-migration can happen, and in the past some people have found their (single-Machine) app offline for multiple hours. Volumes can exacerbate this, although the details aren’t really documented. (Last I heard, volumes prevented auto-migration entirely, but I get the impression that’s been relaxed recently.)

I have some further questions:

  • Is there any data so I can know the actual frequency? Which timezone does downtime frequently happen?
  • Why does my machine need to be migrated to another physical host? Would that host be in the same region?
  • If the purpose of having a second machine is to avoid downtime, then it should also attach to the volume, right? So the fact that volumes are not shared between machines defeat that purpose?
  • What counts as rootfs? If I create a custom file or folder under root, is it a part of rootfs? If I set persist_rootfs as always or restart, then would that file or folder be kept, or just changes inside default Linux directories are kept? Would mounting the rootfs to the volume be a way to have it persistent while setting the config to never?
  • I have only one machine and want to start a new one, what do I need to make sure the data is shared? It seems to me that I need to set another machine dedicated for database. This is true if the backup machine is in the same region, and even true if I want it in another region. But then I have to worry that that database machine can have downtime of its own.
4

Just to emphasize, this is about the downtime of an individual Machine, part of the argument against single-Machine apps, :sweat_smile:…

(The platform specifically recommends against them, although maybe not loudly enough, given the weight of everyone’s assumptions in the opposite direction.)

[Also, this is the Machines platform, not the newer Sprites concept.]

The two Machines definitely don’t share the volume; i.e., the platform’s expectations themselves nearly lose their logical self-consistency (when you introduce volumes into the system). This is one reason why volumes are an advanced feature on the Fly.io platform (in my view), even if it doesn’t seem that hard at first glance…

The initial configuration is easy, but the broader design implications are not, in other words.

The app‡ itself, your own code or something you install alongside it within each Machine, would need to actively replicate between the two disks. There’s a sentence to that effect in the (long) list of volume caveats in the official docs. This is the only† way to satisfy the (wise) recommendation of redundant Machines while also having persistent local storage beyond temp files, things that you can re-download from S3 on reboot, etc. In general, this amounts to creating and sysadmining your own distributed database, which is really not simple. Most people should instead use Managed Postgres and/or Tigris, which handle all the multiple nodes and the synchronization among them.

†There can be niche exceptions, but only very experienced users should mount a volume on the Machines platform, in my opinion. (Apart from experiments, personal, single-user projects without any irreplaceable data stored, etc.)

‡“App” generally includes things like a separate multi-node MySQL cluster that you install and maintain yourself, too, which is slightly idiosyncratic terminology. I.e., it’s an app in the very expansive sense that the Machines platform defines. In those cases, you typically rely on the database’s own clustering mechanisms (which count as “app” code, from this perspective) to keep the various nodes up to date.

Exactly. You can look at the design of LiteFS to see the considerations that come into play there. It has at least two database Machines running at all times, and they’re constantly chatting with each other over the internal network, to make sure that both have record of all the latest modifications. Moreover, the freebie Consul cluster is the arbiter of who is the “primary” node (the one that can accept new writes), which is another critical gotcha in this setting.

The primary Machine will go down during deploys, auto-migrations, hardware failure, etc., and the other one needs to always be prepared to both notice that and then take over the main role itself.

(I don’t recommend using LiteFS, per se, at this time; it’s just a relatively small example system for reference.)

Aside: With LiteFS, the database Machines were actually the same as the web-app Machines, which was a nice cost-saving feature of its architecture. That doesn’t really affect the overall thrust here, though.

5

Beyond what @mayailurus has already answered above,

This can happen for a number of reasons. For example, if the original host is suffering a failure, or if we need to rebalance workload between hosts to avoid CPU steal and I/O issues. But,

Unless we specifically notified you about a region deprecation or similar changes, migrations are always local to regions.

Yes, if you don’t create a volume, then whatever you put in /, except in some Linux system mountpoints like /dev, /tmp, /proc, /sys, etc., are counted as rootfs. You can check what mountpoint a path is under by looking at mount output.

Yes but only if machines are restarted or updated, not migrated. These options are intended for cases where the rootfs is used as some kind of cache, in which case a volume is overkill (because cache doesn’t need real persistence), but default rootfs causes cache to be invalidated on every update. They are NOT intended for actual persistent data and there is no guarantee for long term persistency.

Not sure what this means, but you can definitely run a process from inside a volume and just have a barebones rootfs image that executes something in that volume.

The answer here depends on what type of data you have. If you need strongly consistent relational data, then you’d need a relational database, in which case you may consider Managed Postgres or another managed database solution, or you will have to administer your own. If the data you are storing is mostly media blobs, then an object store like AWS S3, Tigris, or any other S3-compatible service should suit your needs best.

Volumes are a low-level primitive that these services are built on top of. For example, our Managed Postgres service heavily depends on volumes. But because volumes are “just” ext4 filesystems, there is really no way to “share” them as-is without knowing your application logic. Managed Postgres replicates data using PostgreSQL’s replication protocol. If you run S3 on top of volumes, that S3 service needs to handle its own replication. An out-of-the-box distributed POSIX-compatible filesystem would be nice, except that it is almost certainly not what an application that actually requires POSIX-compatible semantics would expect, and your app probably would not function properly with “just” a shared filesystem, unless all you are storing is just some media blobs, in which case, as mentioned above, you should probably consider using an object storage service.

Granted I do think it would be fun to explore in the future if we can provide a somewhat volume-like experience but with reduced consistency guarantees to allow sharing. However I do not think that is likely what you actually need here, and I think there is a somewhat tiny subset of use cases that would actually benefit from this vs just using an object store + a managed database.

6

Multi-tenant 9P would be neat for this, :black_cat:.

(The kernel build used in Machines recently got support for 9P.)

It’s a simple enough protocol that a person can realistically implement their own server, with extra security (isolation) emphasis…

7

It seems to me take that because a group of machines serving one same program share the same IP address regardless of their actual regions, Fly decided to name that group as “app”, and provide “volume” to attach to that app. On high-level, this preserves the mental model of how Docker works, and I like it. However, when you uncover the abstraction, then the analogy breaks down:

  • In Docker, the app runs inside a container, which runs inside a VM, which runs inside a physical machine. In this scenario, thinking that your volume is always there storing data for your program is reasonable, because it only runs in one physical machine.
  • But in Fly, what the app actually do is to control the VMs inside different physical machines, which will then control the containers, which will then run the program. In this scenario, you cannot think that your volume is always available to your program, because it runs on different physical machines.

I guess this is the difference in how we view what app is in the development phase and deployment phase.

I have some further questions:

  • Why can’t different VMs hosted under one physical machine attach to the same volume? In general, why doesn’t Fly auto sync the volumes in different machines of a same app? Is it because it’s not optimal in term of cost?
  • Can a machine (which is a VM) belong to different apps? Because if it’s only managed by one app (via fly.toml), then I don’t see the role of container in Fly. The role of machine in Fly seems to be the same with the role of container then, because all containers inside a machine should run the same image. We mostly discuss about machine and not container.

Also, can you share the downtime data? When does downtime frequently happen, especially in a particular region?

8

Fly Volumes are simple block devices mounted by the guest kernel, it’s not possible for two guest kernels to mount the same block device (at least read-write) at the same time. we don’t sync data between volumes because that’s out of scope for what the product is; for many use cases it wouldn’t make sense, and the complexity (especially between regions) would be huge. you can run your own code to sync data between volumes, or ideally use object storage or a database product instead.

a Fly Machine belongs to a single app. not all fly machines are managed via a fly.toml (indeed, I would say most aren’t) and you can definitely run different images both within the same app (by creating Machines via API and within the same machine using multi-container support. if you’re familiar with kubernetes, you could consider a Fly Machine to be akin to a Pod.

we don’t have downtime data available at the moment, but our reliability team is looking into compiling metrics to be publicly shared.

9

It’s because whether this auto-syncing works or not depends a lot on your app’s expectations. We can’t assume that a bog-standard userspace program works correctly when multi-homed and only filesystem data is synchronized (mind that this synchronization has its own consistency pitfalls; it probably can’t be immediately consistent). You wouldn’t be able to run any kind of database on this kind of volumes, for example, not even sqlite. So instead of “why volumes can’t be synchronized” we’d be receiving a lot of support requests about horribly broken apps when run on this kind of volumes instead.

This is not just a Fly Volumes limitation, most block storage products probably won’t do what you are describing here, and those that do probably do not function like a standard disk that most filesystems and apps expect of a block device. If your app requires less synchronization than a full block device + a POSIX filesystem on top, the product you’d be looking for is some kind of object storage and possibly a database to go along with it. If you still want something that looks like a local folder, you can use FUSE implementations to mount object storage at a local mountpoint, but then again, it will not look and behave like a block storage device.

There are definitely product discussions to be had about Fly Volumes, but so long as they stay a block storage product, honestly I do not see it offering any kind of auto-synchronization or multi-mount support.

This is mostly accurate, on the PaaS platform, a machine is indeed expected to function more like a Docker container in a multi-container setup. You can run further containers nested inside a machine, but those aren’t really visible to the Fly PaaS platform and we still treat your Machine as a single “container” of some sort.

10

So I take that Fly Machine is just VM. Is there any advantage of coining the word “Fly Machine” instead of just using VM? I’m not familiar with Kubernetes, but am familiar with Docker Compose. In my understanding, pod is just an advanced compose that provides more controls to you. So if machine is just like pod, then why not just call it pod at the first place?

11

It’s hard to choose names that can’t be improved in hindsight.

The ability to run multiple containers within a single Machine that @lillian mentioned is relatively new, and calling them “pods” before that would have increased confusion rather than reduced it.

(Even after Fly.io introduced a formal Kubernetes compatibility interface, there were still loud and repeated complaints that it wasn’t real Kubernetes, etc.)

“VM” is more debatable… They originally did say “micro-VM” a lot, to emphasize the fast booting. Later, at least a few people within Fly.io started to see it as a midpoint between a traditional VPS (on the one hand) and AWS Lambda (on the other):

I think of Machines as being like Lambda, but without the random restrictions (but also without the autoscaling features).

They can run apps, of course (24/7 or auto-stopping when idle), but they’re also great for shorter, bursty workloads.

So, there is arguably a benefit to having a third term, for that middle ground.

On top of that, an API/orchestration distinction was needed, during a certain (unclear to me) interval, between that way and the older (and subsequently shut down) Nomad system.

I don’t know the fully detailed chronology of this myself, but if you’re interested, I’d suggest reading through old blog posts and forum threads, which have a fair amount of what they were thinking back then. Here are a few starting points: