This is really exciting, but there are a few things they will certainly have to work through:
Services:
Kubernetes expects DNS records like {pod}.default.svc.cluster.local. In order to achieve this they will have to have some custom DNS records on the “pod” (fly machine) to resolve this with their metadata. Not impossible, but something that has to be take into account.
StatefulSets:
This has 2 major obstacles:
The first is dealing with disk. k8s expects that it can move disks to different logical pods when they lose them (e.g. mapping EBS to an EC2 node). The problem here is that fly has a fundamentally different model for storage. It means that it either has to decide not to schedule a pod because it can’t get the machine that the disk lives on, or not guarantee that the disk is the same. While this does exist as a setting currently, the former is a serious issue (yes this issue exists on EKS with local SSDs, but they are communicated as much more ephemeral).
The second major issue is again with DNS and services. StatefulSets have ordinal pod names (e.g. {ss-name}-{0…n}.default.sv.cluster.local). While this can be achieved with their machine metadata and custom DNS on the machine, it means that it either has to run a local DNS server to “translate” DNS records to the fly nomenclature, or have to constantly update local services on machines to tell them about new records. Both will incur some penalty.
I am also eager to see how load balancers are handled. I imagine it makes more sense to use Ingress manifests with custom labels for the fly-specific config.
I’m sure the fine folks at fly have this all in mind, but these are some things that I think are major obstacles that will have to be solved to make a good k8s (k3s?)