Howdy; I’ve been testing the waters at Codeberg for git hosting, and wanted to set up their equivalent to Github Actions / CICD which require Forgejo Runners – I want to set up my runners on fly, because of course I should.
There’s examples of people doing this in docker compose using two images – Docker-In-Docker (dind) and the Runner image itself – Here’s an excellent writeup .
I’m trying to figure out the simplest way to map this to Fly – would it be two projects/apps, one for each Docker image, and then networking them to each other, or is there a reasonable path to doing it as a single app?
To remark/opine initially on the narrower part of the question about images… Most of the flyctl amenities assume a (mostly) single image shared by every Machine in an app, and some will even complain if they detect multiple, . That’s not a strict requirement, however, and the above docs cover almost all permutations: multiple images (long-term) in distinct Machines within a single app, multiple images within a single Machine, homogeneous images across an entire app (and hence multiple apps).
The first two have rough edges, though, so if you’re sitting on the fence, creating two separate apps would be the default choice…
And hopefully someone who knows more about Forgejo can comment about how that might affect the trade-offs, etc.
Thanks @mayailurus, those were helpful links – Fly now supports docker compose files for multi-container stuff – who knew?!? (New to me, maybe no one else =)
That’s a huge potential win for dev/prod parity. If I can simplify deployment to just a fly.toml and a docker-compose.yml I’ll be feeling pretty clever.
All that being said, I would trade the clever feelings for a working solution, so if anyone’s done the work or wants to take a crack at mapping this compose file to a running fly solution, I’d be super grateful. I’ve been trying, but I’m trying to get the forgejo runner to use a config.yml file that’s not sitting in the /data directory, which I think needs to be a fly persistent volume.
They also do some ownership changes on files that I’m not sure are required or just smart if you’re running it via compose rather than under a firecracker vm.
Out of curiosity, do CI workloads run on the Runner itself, or is that just concerned with starting them? If the latter, how does it normally create a new machine in its most preferred environments? I suppose in the Docker world it would just be the Docker API, but that won’t work in Fly - one would have to call one of the Fly APIs.
@halfer Based on the docker compose example, one of the containers is the runner, the other is an instance of docker-in-docker, which would then be able to spin up whatever it needed. At least that’s my understanding of the workload split.
I did a bit of research, and I think DinD isn’t going to cut it. My thought process is that proper usage of Docker here would spin up a new separate container i.e. the system scales horizontally. However DinD implies that you have a private Docker instance running in Fly, and containers are created inside the space that that single machine permits (sorta kinda vertical scaling, but rather inefficient).
However, happily, a robot kindly wrote a Fly connector for me. How correct it is, I’ve no idea, but I can paste it if you like (this one is in Python + FastAPI). Here is how it works: Forgejo can call web-hooks for events, supplying the event name in request header X-Forgejo-Event. A connector would listen for workflow_job events and then spawn a Fly machine, of the spec you would like, using the Fly REST API.
@halfer Very cool! That’s an interesting alternative approach, and may end up being the direction to go for fly or any service with a API for resource allocation.
For now I’m in the learning phase and need to build a baseline working version as close the documented solution as possible before trying to solve for scale or introducing new moving parts.
I’ve been watching this space for a while and there isn’t really a ready-to-use solution for this. the stock forgejo runner only supports scaling within the same Docker daemon as halfer mentioned; you can make a wrapper around it (this needs to poll for new jobs, not webhook receive) that spawns fly machines.
iirc there are some folks building a proper solution for this but I don’t recall where they’re at off the top of my head.
. That’s not a strict requirement, however, and the above docs cover almost all permutations: multiple images (long-term) in distinct Machines within a single app, multiple images within a single Machine, homogeneous images across an entire app (and hence multiple apps).