The [deploy] section seems to be missing from the docs: App Configuration (fly.toml)
I found some examples using release_command so I’m trying it myself. Should I expect that my secrets will be available as environment variables to the release command? It doesn’t seem to work.
I’m attempting to use it as so:
Since I need to run this before I can launch my app, I’m not able to use fly ssh console -C in this case.
Secrets should be available but I can see the error in your release task logs. I’ll dig in and let you know what I find.
Okay they are missing here. I think this is another bug related to process groups. Going to see if there’s a quick fix.
Got it.
Process groups aren’t really documented either. Does each process get its own VM, or do they live in the VM together (ala Running Multiple Processes Inside A Fly.io App)
Each process gets it’s own VM. As you can see, we’re still working out the kinks between process groups and release tasks. We should have them added to docs pretty soon though.
I think we found the bug. New secrets aren’t making it to release tasks since release tasks are based on the previous config. We need to figure out how to handle it. In the mean time it’ll work if you deploy again.
I have a bit of a chicken-and-egg, because I need to run rails db:setup before I can successfully deploy the app. But I understand the behavior now, and was able to deploy a dummy image to get the environment setup first.
Now I’m running into an issue with process groups + mounts. Is there a way to enable mounts for just one process?
I added a sidekiq process in add sidekiq · tmm1/flyapp-mastodon@9a6e28a · GitHub and now deploys fail as so:
Error not enough volumes named mastodon_uploads (1) to run 2 processes
(This also seems to suggest you can have multiple volumes with the same name?)
Actually I wonder if that would even work, since it seems likely the sidekiq workers would also need to access data in that volume.
I may need to reconsider and use cloud storage instead of a fly volume.
Cloud storage is probably the way to go here (we need to bake in cloud storage). You can set mounts on one type of process though:
[mount]
processes = ["worker"]
source = "mastodon_uploads"
destination = "/opt/mastodon/public/system"
Awesome, thanks!
I’m still seeing issues with regions and my processes. Now that I added this sidekiq process, I see it coming up in Brazil for some reason instead of IAD like my volumes and rails process.
2021-10-28T00:16:14.749 app[024cab19] gru [info] Starting init (commit: 50ffe20)...
2021-10-28T00:16:14.806 app[024cab19] gru [info] Preparing to run: `bundle exec sidekiq` as mastodon
2021-10-28T00:16:14.850 app[024cab19] gru [info] 2021/10/28 00:16:14 listening on [fdaa:0:37a6:a7b:1f60:24c:ab19:2]:22 (DNS: [fdaa::3]:53)
It also fails to run there, because I guess my secrets aren’t in that regions vault? Or because its the first deployment of this process?
2021-10-28T00:16:21.354 app[024cab19] gru [info] 2021-10-28T00:16:21.354Z pid=510 tid=bua WARN: KeyError: key not found: "OTP_SECRET"
Oof this is all kinds of confusing UX. You have to set regions per process group:
fly regions set iad --group sidekiq
The secrets should have made it to the new group, I’ll double check that.
I ran this but it didn’t seem to help. It tried to come up in sjc and failed due to missing env secrets. Then it tried in mad and booted up there.
2021-10-28T00:55:18.070 app[aecbbf5a] mad [info] 2021-10-28T00:55:18.069Z pid=510 tid=2it2 WARN: Your Redis network connection is performing extremely poorly.
2021-10-28T00:55:18.070 app[aecbbf5a] mad [info] Last RTT readings were [98501, 98655, 98489, 98673, 98569], ideally these should be < 1000.
2021-10-28T00:55:18.070 app[aecbbf5a] mad [info] Ensure Redis is running in the same AZ or datacenter as Sidekiq.
I think that OTP_SECRET may not have been applied to your job at all, perhaps after a rollback or something. I think things getting rolled back are putting the thing in an inconsistent state. Give me a sec and I’ll see if I can clean it up.
I think your app is in a good spot not (and we have a list of fun new bugs to work through this week).
Will you have a look and see if it’s doing what you want?
Thanks it does look correct now:
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
dafd4583 rails 23 iad run running 2 total, 2 passing 0 17m55s ago
d2c59705 sidekiq 23 iad run running 0 17m55s ago
fly logs could use a way to filter by process group. Right now they all appear as “app”
Yeah that would be helpful. You can run fly logs -i d2c59705 for now to get specific process output.