Multiple Processes App

,
4

tl;dr: you probably want to follow the link from rubys, above. perhaps start with a bash script that backgrounds things and then iterate

it’s not clear what the use-case is, but listing some tradeoffs might help you determine what makes sense (i spent a decent amount of time this past weekend on running multiple processes and settled on the first option, below, with hivemind–i’ll add some details for my setup which i don’t claim is optimal).

from my read on the lay of the land the options are:

  • use a process manager (bash/Procfile manager/init-like-supervisor)
    • this is what rubys links to, above
  • use machine processes
  • use apps v2 process groups in fly.toml
  • use different fly apps
    • this would be the traditional way to setup services that can be scaled independently

use a process manager (bash/Procfile manager/init-like-supervisor)

this is the approach i took because it gave me a http service, background worker, and cron scheduler which could all interact with the primary litefs db (and i didn’t have to figure out HALT locks, the litefs proxy, the fly replay header, or figure out a different option for the scheduler).

i ended up using litefs mount to mount the database, run any migrations, and then run hivemind. in turn, hivemind is responsible for running the http server, background worker, and supercronic which are all defined in my Procfile.

this was a simple-enough approach to getting all the processes to wait for the sqlite db to be ready before starting up. hivemind is pretty simple so i think it’s possible the cron and background worker could stop running and they wouldn’t get restarted (but it doesn’t pull in python or tmux so that seemed fine for now)

use machine processes

this works essentially the same way as above, but has the downside (or upside) where if any given process blows up the machine gets shutdown.

that said, this is already pretty cool. it feels more correct than stitching together a user-space process manager and the different processes can have different images (each http service could be running in a docker image with only the application code and then have a telemetry agent running in an isolated process on the same VM)

that said, i’m a big fan of being lazy and adding a curl command to a Dockerfile and using my Procfile seemed really, really easy.

use apps v2 process groups in fly.toml

this is a step along the way to separate apps. with process groups, one ends up with one Firecracker VM per process group replica. per the docs:

fly deploy creates at least one Machine for each process group, and destroys all the Machines that belong to any process group that isn’t defined, in your app’s fly.toml file.

so this would be nice in the case where one wants n http service replicas and for each region maybe a memcached process (though that could be done with a separate memcached fly.toml App, as well).

what follows, i believe to be the case but would love to be aggressively corrected if it’s incorrect:

  • each process group replica would need its own separate Volume, if a volume is necessary for the process
  • there’s no guarantee process group replicas are scheduled on the same underlying hardware as each other (i suspect some sleuthing with dig would be able to answer this).
  • because of the above, even if one figured out a way to use the volume mounted by another process group replica scheduled on the same underlying hardware

use different fly apps

this would be how one would stand up postgres or some other piece of shared something. as far as i can tell (i didn’t read things exhaustively, or do a bunch of sleuthing), process groups are some syntactic sugar around Apps for the extremely common use-case of having side-car like things users want to run logically associated with their application.

i could see a very common pattern being:

  • Team A wants something like memcached as part of their volcano eruption detection app “volcaNooo”
  • Team A deploys app + memcached as process groups
  • Team A gets volcaNooo deployed globally on tiny 128mb Machines and they have a memcached process running wherever volcaNooo is on bigger 512mb Machines
  • Team B is working on volcaNooo’s notifications, sees Team A’s success and Team B starts to go down the same path
  • someone does some math and realizes fewer, beefier memcached instances strategically geolocated will provide an acceptable UX and save dozens and dozens of dollars and keys can be namespaced where appropriate and shared where appropriate and everyone gets a better setup
  • Team B (hopefully, but it’s probably Team Z.a.1) then pulls the memcached process group into its own app and deploys it
  • Team B uses new memcached cluster app for volcaNooo’s notifications
  • Team A swaps their config to use new memcached cluster app
  • Team A removes old memcached process group
  • dozens of dollars are saved and everyone gets extra donuts
2 Likes