This is great! I was recently thinking that it’s a missing piece on Fly and now it is not
TIL about [deploy] release_command
, I believe it’s missing from the docs?
This is great! I was recently thinking that it’s a missing piece on Fly and now it is not
TIL about [deploy] release_command
, I believe it’s missing from the docs?
For anyone wanting to scale an app created before this feature existed: The default process/group name is app
, so this should work:
fly scale count app=1 -a myapp
release_command
is missing from docs, yes, we have a big doc update we need to do.
Does [processes] syntax work for both docker built and buildpack apps?
Yes, this should work. If you have any problems, let us know.
Hello,
This is an awesome feature. I am currently using it to run a Django API, a Celery Worker, and a Celery Beat instance.
fly.toml for references
kill_signal = "SIGINT"
kill_timeout = 5
[processes]
web = "gunicorn appname.wsgi --log-level 'info'"
worker = "celery -A appname worker -l INFO"
beat = "celery -A appname beat -l INFO"
[build]
builtin = "python"
[deploy]
release_command = "sh release.sh"
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
processes = ["web"]
http_checks = []
internal_port = 8080
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 6
timeout = "2s"
Is there something special that I need to do to expose Fly Secrets to all of my processes? Does it differ in any way from just setting variables in [env]
?
I ask because:
When I specify a SECRET_KEY
variables locally on my computer in the env block and then deploy it all three instances get the variable and work fine
With that working, I moved on to trying to deploy via a Github Action, good docs around that too BTW, and moved all my sensitive values from the [env] block into the Fly Secrets.
However, this appears to not work, as my celery beat process cannot find the SECRET_KEY
variable now. but my main app instance can.
I sshed into my worker instance and my app instance and listed their environment variables and both had a SECRET_KEY
set. I would check the Beat instance in the same way but cannot SSH into as the deployment fails and gets rolled back, I guess I probably could disable health checks and then SSH in?
Thanks for your help!
Hey, great to see some Python love here! I’m going to steal your fly.toml
for our Python guide
Setting secrets should cut a new release. If that release failed for the beat instance, it may not get the secret assigned. With this config, your beat instance should not actually have any health checks. What does fly status
look like?
Thanks! Feel free to I just compiled stuff from the existing Flask guide and these forums haha.
Hmm so the secrets all already existed and are being used to deploy a worker and the API just fine. And I tested the API can send tasks to the worker and the worker executes them just fine
For some reason though that envar does not make it to the Beat process
Fly status shows that my worker and API are running fine because of a rollback
App
Name = little-frog-6396
Owner = personal
Version = 91
Status = running
Hostname = little-frog-6396.fly.dev
Deployment Status
ID = 0123d165-3eca-3eba-4afa-74dba5fcf96c
Version = v91
Status = successful
Description = Deployment completed successfully
Instances = 2 desired, 2 placed, 2 healthy, 0 unhealthy
Instances
ID TASK VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
529034bc web 91 lax run running 1 total, 1 passing 0 29m18s ago
5314c66d worker 91 lax run running 0 29m18s ago
My failure message when I try to deploy the beat scheduler is
The generic unhealthy allocations
***v90 failed - Failed due to unhealthy allocations - rolling back to job version 89 and deploying as v91
When I hop into the logs I see
2021-10-05T07:58:31.206074824Z app[dc0d5ad4] lax [info] SECRET_KEY = env('SECRET_KEY')
2021-10-05T07:58:31.206079407Z app[dc0d5ad4] lax [info] File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 186, in __call__
2021-10-05T07:58:31.206082605Z app[dc0d5ad4] lax [info] return self.get_value(
2021-10-05T07:58:31.206087637Z app[dc0d5ad4] lax [info] File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 367, in get_value
2021-10-05T07:58:31.206091098Z app[dc0d5ad4] lax [info] raise ImproperlyConfigured(error_msg) from exc
2021-10-05T07:58:31.206098486Z app[dc0d5ad4] lax [info] django.core.exceptions.ImproperlyConfigured: Set the SECRET_KEY environment variable
You could try setting the command to tail -f /dev/null
then logging in.
Hmm, Unsetting the SECRET_KEY secret and then resetting it seems to have fixed it, Maybe it was a weird ordering thing.
Nice trick with tail -f /dev/null
I will keep that in mind for future debugging.
Thanks!
Good catch there. A good way to avoid this in the future is to set the secrets before the first deployment. We’ll think about ways to improve this situation.
I have another couple of questions related to defining multiple processes in one fly.toml file
Thanks!
What are people using as a message broker for the worker? Is everyone just passing messages through the fly redis cache?
Most people are using small Redis instance or KeyDB the built in Redis cache is bad for messages, they will get lost! Maybe fine for development though.
Yeah the docs are pretty clear on not using the cache for message - how do you declare that redis insistence? An unconnected toml file with a redis docker image?
Yep! This project works to launch a Redis: GitHub - fly-apps/redis: Launch a Redis server on Fly
You can launch KeyDB through here, too: KeyDB - Launch on Fly.io · Fly
How does autoscaling work with multi process apps?
There seems to be some sort of bug with scaling, not sure if it only affects multi-process apps:
https://community.fly.io/t/an-unknown-error-occurred-during-deployment/2448/18
I am hoping this feature is the missing link to a use case. We rely on webhooks for some critical workflows and I am very interested in being able to exercise the system in development. This is hard because our providers require the receiving server to be on the public internet. We would like to tunnel from a running instance to our local machines.
I came to a project called FRP and think it makes our use-case possible. The secondary process would live as long as the primary process and receive traffic. That traffic would be received by development machines. Any thoughts on the feasibility of this model with the Fly infra would be great! Otherwise, I look forward to writing this up. Thanks for any help/guidance the community can share -
Hey @kurt, this is awesome! Exactly what I was looking for.
I know the documentation hasn’t been updated yet, but:
By way of an explanation, I am building a simple varnish docker container that caches the fly helloname express app. The toml config runs the node app on (8080:8080) and the varnish app (8080:80). It deploys just fine.
Unfortunately, the Varnish VCL is misconfiguration and unable to find the node app instance internally. Because these processes are separate instances with different ip addreses, the varnish VCL cannot use 127.0.0.1 for the backend host because the node app in on another instance.
Any ideas or suggestions?
Edit:
Holy smokes, Same app name / dns. Just different ports.
/ # curl app_name.internal:3000
<h1>Hello From Node on Fly!</h1>
That’s what I am talking about. Now to see if the VCL backend can use app_name.internal as a host.
Processes don’t work quite like we want with DNS. It’s part of why we haven’t “officially” launched this feature yet! Right now, all processes in the same app show on <app>.internal
. It is difficult to run different VMs that expose different services with this setup.
It’s usually better to run caches as separate apps in Fly. You can still use DNS between apps, but you can target things more specifically.
VCL can only use hostnames that return one single IP. I’m not sure how well that will work with Fly hosted apps, especially if you scale to two instances.