I’d like to configure fly so that one of my machines will auto-stop while the other will stay running. However, I need to specify which machine stays running! Can I not do this in my fly.toml?
For simplicity sake, say my machine ID’s are machineA and machineB. I want machineA to have auto-stop as off and machineB to have auto-stop as true.
A current work-around:
fly deploy with auto_stop_machines = "stop"
Turn off auto-stop on machineA with flyctl machines update machineA --autostop='off'
This is a bit annoying as now after every fly deploy I have to remember to turn off auto-stop for machineA after deploying.
Context: My app has a node package mailListener. We only want one of these to ever be listening, so my app implements this:
const machineName = os.hostname();
if (machineName === process.env.MAIL_LISTENER_MACHINE) {
logger.warn({ message: `Machine ${machineName} is listening for mail.` });
setupMailListener();
}
What is the repercussion of multiple machines listening to the mail? IMO, managing state of machines is a bad design - the machines should be agnostic. If you need to process mail once and only once, then using a queue and a distributed DB is the way to go.
If I have 2 machines both listening to mail, when incoming mail is received, we encounter a race condition where both machines are trying to process and move the same piece of mail
You wouldn’t want to shut down machineB, what if machineA dies? You want to run multiple machines for availability (unless the mailing has a log it can pick up where it left off). The quick and easy way is to write to a distributed DB and have a worker machine process it so there won’t be any duplicate work.
As it is now, it is working fine and as expected. Under light load, machineA runs the mailer and server requests. If traffic picks up and we need more power, machineB and others can spin up. The mailer is only ever run by machineA. If it dies, then we no longer are listening for incoming mail and we’d have to manually restart it which is acceptable since it is a rare edge case for it to be killed and then not restarted automatically.
I realize there are likely better, but much more complex, solutions out there such as using redis for locking or a filesystem lock. But this is a really simple solution for a small application. I was hoping I could configure fly.toml to specify policies for specific machines but it doesn’t seem like that is something that is supported.
To counter this, I’ve changed my deployment from running fly deploy to npm run deploy where I have the following script