So here’s my update after following the advice in this thread. Might be useful as a reference for other people working to make a similar setup function.
It runs one process group, “app”, on one VM/Machine, which runs LiteFS, which in turn runs the overmind supervisor, which in turns runs the backend, worker and poller(poll whether to start or stop the worker process based on primary status) processes.
fly.toml
...
[mounts]
source = 'litefs'
destination = '/var/lib/litefs'
[vm]
cpu_kind = "shared"
cpus = 1
memory_mb = 512 # Shared by backend,worker,poller
processes = ["app"]
Dockerfile
# The process starts with litefs mount, which will afterwards boot overmind, which will boot individual processes
ENTRYPOINT litefs mount -config deployments/litefs.yml
litefs.yml
exec:
- cmd: "npm run db:migrate:deployed"
if-candidate: true
# Initiate the long running process supervisor "overmind" to start the application
# This path is relative to this file
- cmd: "./deployments/scripts/start.sh"
start.sh
#!/bin/sh
if [ "$PRIMARY_REGION" == "$FLY_REGION" ]; then
# In the primary region, we start both the backend, worker and poller processes
overmind start -f deployments/Procfile -l backend,worker,poller
else
# In the secondary region, we only start the backend and poller process
overmind start -f deployments/Procfile -l backend,poller
fi
Procfile
backend: npm run backend:start
worker: npm run worker:start
# poll LiteFS to start/stop the worker process based on primary status
poller: ./scripts/poll_litefs_events.sh
poll_litefs_events.sh
#!/bin/sh
# Poll the LiteFS event stream every second, and start/stop the worker process based on the primary status
while true; do
response=$(curl -s localhost:20202/events | jq -r '.type, .isPrimary')
# We're looking for a response such as https://fly.io/docs/litefs/events/#primarychange
# {
# "type": "primaryChange",
# "data": {
# "isPrimary": false,
# "hostname": "myPrimaryHost:20202"
# }
# }
# If we are not the primary anymore, stop the worker process
if [[ $response == "primaryChange false" ]]; then
echo "primaryChange is false, stopping worker"
overmind stop worker
# If we become the primary, start the worker process
elif [[ $response == "primaryChange true" ]]; then
echo "primaryChange is true, starting worker"
overmind start worker
fi
sleep 1
done
package.json
...
// Migrate commands
"db:migrate": "npx tsx src/db/migrate.ts",
"db:migrate:deployed": "SQLITE_PATH=/litefs/db npm run db:migrate"
// Boot commands
"backend:start": "NODE_ENV=production node dist/backend/index.js", // Listens on :8080
"worker:start": "NODE_ENV=production node dist/worker/index.js",
Endpoint handler code
handler.ts (GET /resources/:id)
// This code is reached if the data needs to be fetched externally and then written to the DB
// If we're running in deployed environments (dev, prod) we need to use the primary node
// Since the following logic relies on writing to the database
if (process.env.NODE_ENV === "production") {
const primaryNodeLocation = await getPrimaryNodeLocation();
console.log("Primary node location", primaryNodeLocation);
// This means we're not reaching this line of code on the primary node,
// which means we need to replay the request to the primary node
if (primaryNodeLocation) {
return res
.setHeader("Fly-Replay", `instance=${primaryNodeLocation}`)
.status(200)
.send();
}
}
litefs.ts
import fs from "fs/promises";
import { LITE_FS_PRIMARY_FILE_LOCATION } from "../config";
export async function getPrimaryNodeLocation() {
try {
const file = await fs.readFile(LITE_FS_PRIMARY_FILE_LOCATION, "utf8");
return file.trim();
} catch (err) {
console.error(
"An error occurred while reading LiteFS primary file location",
err
);
return null;
}
}
There’s nothing in the worker logic that has been adjusted, because I don’t think that should be needed.
I think that covers all of the important parts. I’ve been able to have this work in the primary region. The setup seems a bit finicky though.
Questions:
- Does anything stand out as problematic to you in this setup?
- When I run
fly m clone to replicate into a new region, it seems to not mount the volume that I already created, with the already written data. Is this a problem? Meaning every replica will have its own volume? It sounds like this will duplicate the data, making it costly in the long run.
- The poller runs every second, and it looks like it could potentially be picking up tasks from the queue (Redis in my case via node BullMQ) after it has lost its primary status. This will have the tasks fail. I guess this will be retried when the new worker starts up? Sounds like an edge case but still.
- I expected the DB to be connected through the
/var/lib/litefs path, but it seems to be litefs/db. Where is this decided?
Thanks again for all of the assistance.
…
, if the external source can be relied on to return the same thing.
.)