kubernetes on fly

You’re probably right with not exposing the kube api publicly. I was just thinking about trying kind on fly, but yeah in hindsight it maybe a little silly :slight_smile:

I have been meaning to try k8s on Fly, so I’m into silly ideas.

“Run your container clusters in other containers” sounds like a hilarious guide to me.

1 Like

With k3d even multi node clustering should be possible over the internal network I guess. I also think it can be configured to use postgres instead of etcd, so the managed postgres could also be used maybe

etcd is pretty easy to run on Fly! I know of at least one person who got it working.

If you’re interested in trying this stuff out and writing about it in a README on github, we’ll give you service credits.

Maybe I could start with dendrite (port 8443)?

This one I already had working (except for federation)

Docker compose would be helpful on this note. I know there are some build packs but it wasn’t clear how well they would work and some seem outdated.

Reason being is it would allow a simpler transition to fly at first, and also let you share the higher memory/cpu nodes a bit more efficiently.

On a side side note (maybe split into it’s own thread) could you post some plans in terms of future VMs? I’d be interested in high CPU optimized, and potentially in GPU.

1 Like

We’d like to support Docker Compose! docker-compose to “production” is a little tricky, but we built the PG apps specifically to make progress on that.

1 Like

Yes start with that! Port 8443 might work now. :slight_smile:

Has anybody had any success with running Kubernetes on Fly yet?

I’m trying to run a single node of k3s and not having much luck at the moment. I need to do a bit more investigation to figure out what is wrong, but I don’t think the API server ever actually starts.

While its probably a bit niche I’d love to see Fly do a managed Kubernetes offering. The Anycast IP’s and the private wireguard network (across clusters) really make this an interesting offering.

1 Like

I think I just managed to run k3s on fly with nats as the datastore. (note that it seems to require at least 1GB of ram).

❯ kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-b96499967-z4fw2                   1/1     Running   0          3h
kube-system   helm-install-traefik-5fbsx                0/1     Pending   0          3h
kube-system   helm-install-traefik-crd-bzq8f            0/1     Pending   0          3h
kube-system   local-path-provisioner-7b7dc8d6f5-98p54   1/1     Running   0          3h
kube-system   metrics-server-668d979685-d5qps           1/1     Running   0          3h

I think helm is Pending because I’m using --node-taint CriticalAddonsOnly=true:NoExecute.

3 Likes

Am also using kine :slight_smile:

What’s the Zement of ram per pod running ?

about 1G without any load (not in prod). just testing k3s+kine+nats (running with overmind).

I wonder if trying to make the docker image smaller could help, or maybe test without overmind.

yep exactly how i run it too.

overmind and kine - really easy

will check out the repo !!

1 Like

I got this working using k3d in a dind.

On fly or local ?

on fly

Are you able to share a GitHub link ?

Would like to try it out

This was part of a bigger integration project so I wasn’t able to put it on Github.

Use a dind container like docker:dind and install k3d, fuse-overlayfs. Ensure iptables-legacy is enabled (see below).
Once that is done, use k3d cluster create to bring up the cluster in the dind container.

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Hope this helps you.

1 Like

It’s does thanks.

I already use kine and nats , so should be able to work out the rest based on the extra info. Run k8 before.

K3 said is a single binary but using 1 gb RAM is shockingly large and that’s before anything is running on it.

I guess however it’s needed for those orgs that have projects already designed for k8 and wanting the features of fly like serverless , scale to zero, db replication etc .
Some of these will not work like scale to zero inside of K3s unless you also run Knative I expect.

So it would be I retesting to know your tabs on this , if only for others that go down this path . If you have time of course.

1 Like