You’re probably right with not exposing the kube api publicly. I was just thinking about trying kind on fly, but yeah in hindsight it maybe a little silly 
I have been meaning to try k8s on Fly, so I’m into silly ideas.
“Run your container clusters in other containers” sounds like a hilarious guide to me.
1 Like
With k3d even multi node clustering should be possible over the internal network I guess. I also think it can be configured to use postgres instead of etcd, so the managed postgres could also be used maybe
etcd is pretty easy to run on Fly! I know of at least one person who got it working.
If you’re interested in trying this stuff out and writing about it in a README on github, we’ll give you service credits.
Maybe I could start with dendrite (port 8443)?
This one I already had working (except for federation)
Docker compose would be helpful on this note. I know there are some build packs but it wasn’t clear how well they would work and some seem outdated.
Reason being is it would allow a simpler transition to fly at first, and also let you share the higher memory/cpu nodes a bit more efficiently.
On a side side note (maybe split into it’s own thread) could you post some plans in terms of future VMs? I’d be interested in high CPU optimized, and potentially in GPU.
1 Like
We’d like to support Docker Compose! docker-compose to “production” is a little tricky, but we built the PG apps specifically to make progress on that.
1 Like
Yes start with that! Port 8443 might work now. 
Has anybody had any success with running Kubernetes on Fly yet?
I’m trying to run a single node of k3s and not having much luck at the moment. I need to do a bit more investigation to figure out what is wrong, but I don’t think the API server ever actually starts.
While its probably a bit niche I’d love to see Fly do a managed Kubernetes offering. The Anycast IP’s and the private wireguard network (across clusters) really make this an interesting offering.
1 Like
I think I just managed to run k3s on fly with nats as the datastore. (note that it seems to require at least 1GB of ram).
❯ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-b96499967-z4fw2 1/1 Running 0 3h
kube-system helm-install-traefik-5fbsx 0/1 Pending 0 3h
kube-system helm-install-traefik-crd-bzq8f 0/1 Pending 0 3h
kube-system local-path-provisioner-7b7dc8d6f5-98p54 1/1 Running 0 3h
kube-system metrics-server-668d979685-d5qps 1/1 Running 0 3h
I think helm is Pending because I’m using --node-taint CriticalAddonsOnly=true:NoExecute.
4 Likes
Am also using kine
What’s the Zement of ram per pod running ?
about 1G without any load (not in prod). just testing k3s+kine+nats (running with overmind).
I wonder if trying to make the docker image smaller could help, or maybe test without overmind.
yep exactly how i run it too.
overmind and kine - really easy
will check out the repo !!
1 Like
I got this working using k3d in a dind.
On fly or local ?
on fly
Are you able to share a GitHub link ?
Would like to try it out
This was part of a bigger integration project so I wasn’t able to put it on Github.
Use a dind container like docker:dind and install k3d, fuse-overlayfs. Ensure iptables-legacy is enabled (see below).
Once that is done, use k3d cluster create to bring up the cluster in the dind container.
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
Hope this helps you.
1 Like
It’s does thanks.
I already use kine and nats , so should be able to work out the rest based on the extra info. Run k8 before.
K3 said is a single binary but using 1 gb RAM is shockingly large and that’s before anything is running on it.
I guess however it’s needed for those orgs that have projects already designed for k8 and wanting the features of fly like serverless , scale to zero, db replication etc .
Some of these will not work like scale to zero inside of K3s unless you also run Knative I expect.
So it would be I retesting to know your tabs on this , if only for others that go down this path . If you have time of course.
1 Like