Preview: multi process apps (get your workers here!)

This is great! I was recently thinking that it’s a missing piece on Fly and now it is not :slight_smile:

TIL about [deploy] release_command, I believe it’s missing from the docs?

2 Likes

For anyone wanting to scale an app created before this feature existed: The default process/group name is app, so this should work:

fly scale count app=1 -a myapp
2 Likes

release_command is missing from docs, yes, we have a big doc update we need to do. :slight_smile:

Does [processes] syntax work for both docker built and buildpack apps?

1 Like

Yes, this should work. If you have any problems, let us know.

1 Like

Hello,

This is an awesome feature. I am currently using it to run a Django API, a Celery Worker, and a Celery Beat instance.

fly.toml for references

kill_signal = "SIGINT"
kill_timeout = 5

[processes]
web = "gunicorn appname.wsgi --log-level 'info'"
worker = "celery -A appname worker -l INFO"
beat = "celery -A appname beat -l INFO"

[build]
  builtin = "python"

[deploy]
  release_command = "sh release.sh"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  processes = ["web"]
  http_checks = []
  internal_port = 8080
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 6
    timeout = "2s"

Is there something special that I need to do to expose Fly Secrets to all of my processes? Does it differ in any way from just setting variables in [env]?

I ask because:

When I specify a SECRET_KEY variables locally on my computer in the env block and then deploy it all three instances get the variable and work fine

With that working, I moved on to trying to deploy via a Github Action, good docs around that too BTW, and moved all my sensitive values from the [env] block into the Fly Secrets.

However, this appears to not work, as my celery beat process cannot find the SECRET_KEY variable now. but my main app instance can.

I sshed into my worker instance and my app instance and listed their environment variables and both had a SECRET_KEY set. I would check the Beat instance in the same way but cannot SSH into as the deployment fails and gets rolled back, I guess I probably could disable health checks and then SSH in?

Thanks for your help!

2 Likes

Hey, great to see some Python love here! I’m going to steal your fly.toml for our Python guide :slight_smile:

Setting secrets should cut a new release. If that release failed for the beat instance, it may not get the secret assigned. With this config, your beat instance should not actually have any health checks. What does fly status look like?

1 Like

Thanks! Feel free to I just compiled stuff from the existing Flask guide and these forums haha.

Hmm so the secrets all already existed and are being used to deploy a worker and the API just fine. And I tested the API can send tasks to the worker and the worker executes them just fine

For some reason though that envar does not make it to the Beat process

Fly status shows that my worker and API are running fine because of a rollback

App
  Name     = little-frog-6396
  Owner    = personal
  Version  = 91
  Status   = running
  Hostname = little-frog-6396.fly.dev

Deployment Status
  ID          = 0123d165-3eca-3eba-4afa-74dba5fcf96c
  Version     = v91
  Status      = successful
  Description = Deployment completed successfully
  Instances   = 2 desired, 2 placed, 2 healthy, 0 unhealthy

Instances
ID       TASK   VERSION REGION DESIRED STATUS  HEALTH CHECKS      RESTARTS CREATED
529034bc web    91      lax    run     running 1 total, 1 passing 0        29m18s ago
5314c66d worker 91      lax    run     running                    0        29m18s ago

My failure message when I try to deploy the beat scheduler is
The generic unhealthy allocations

***v90 failed - Failed due to unhealthy allocations - rolling back to job version 89 and deploying as v91

When I hop into the logs I see

2021-10-05T07:58:31.206074824Z app[dc0d5ad4] lax [info]     SECRET_KEY = env('SECRET_KEY')
2021-10-05T07:58:31.206079407Z app[dc0d5ad4] lax [info]   File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 186, in __call__
2021-10-05T07:58:31.206082605Z app[dc0d5ad4] lax [info]     return self.get_value(
2021-10-05T07:58:31.206087637Z app[dc0d5ad4] lax [info]   File "/usr/local/lib/python3.8/site-packages/environ/environ.py", line 367, in get_value
2021-10-05T07:58:31.206091098Z app[dc0d5ad4] lax [info]     raise ImproperlyConfigured(error_msg) from exc
2021-10-05T07:58:31.206098486Z app[dc0d5ad4] lax [info] django.core.exceptions.ImproperlyConfigured: Set the SECRET_KEY environment variable

You could try setting the command to tail -f /dev/null then logging in.

Hmm, Unsetting the SECRET_KEY secret and then resetting it seems to have fixed it, Maybe it was a weird ordering thing.

Nice trick with tail -f /dev/null I will keep that in mind for future debugging.

Thanks!

Good catch there. A good way to avoid this in the future is to set the secrets before the first deployment. We’ll think about ways to improve this situation.

1 Like

I have another couple of questions related to defining multiple processes in one fly.toml file

  1. Is there a way to using the flyctl to easily view the logs only for a specific process
  2. Is there a way to view metrics for each process?
  3. When looking at the metrics page for my app with multiple processes am I only looking at the metrics for the first app I defined?

Thanks!

1 Like

What are people using as a message broker for the worker? Is everyone just passing messages through the fly redis cache?

Most people are using small Redis instance or KeyDB the built in Redis cache is bad for messages, they will get lost! Maybe fine for development though.

Yeah the docs are pretty clear on not using the cache for message - how do you declare that redis insistence? An unconnected toml file with a redis docker image?

Yep! This project works to launch a Redis: GitHub - fly-apps/redis: Launch a Redis server on Fly

You can launch KeyDB through here, too: KeyDB - Launch on Fly.io · Fly

2 Likes

How does autoscaling work with multi process apps?

There seems to be some sort of bug with scaling, not sure if it only affects multi-process apps:

https://community.fly.io/t/an-unknown-error-occurred-during-deployment/2448/18

1 Like

I am hoping this feature is the missing link to a use case. We rely on webhooks for some critical workflows and I am very interested in being able to exercise the system in development. This is hard because our providers require the receiving server to be on the public internet. We would like to tunnel from a running instance to our local machines.

I came to a project called FRP and think it makes our use-case possible. The secondary process would live as long as the primary process and receive traffic. That traffic would be received by development machines. Any thoughts on the feasibility of this model with the Fly infra would be great! Otherwise, I look forward to writing this up. Thanks for any help/guidance the community can share -

Hey @kurt, this is awesome! Exactly what I was looking for.

I know the documentation hasn’t been updated yet, but:

  • How do processes work with private networking? How does one find or address an app process / instance internally?
  • How does this work in a multi-region environment? Is it possible to address the “sibling” instances in the same region?

By way of an explanation, I am building a simple varnish docker container that caches the fly helloname express app. The toml config runs the node app on (8080:8080) and the varnish app (8080:80). It deploys just fine.

Unfortunately, the Varnish VCL is misconfiguration and unable to find the node app instance internally. Because these processes are separate instances with different ip addreses, the varnish VCL cannot use 127.0.0.1 for the backend host because the node app in on another instance.

Any ideas or suggestions?


Edit:
Holy smokes, Same app name / dns. Just different ports.

/ # curl app_name.internal:3000
<h1>Hello From Node on Fly!</h1>

That’s what I am talking about. Now to see if the VCL backend can use app_name.internal as a host.

2 Likes

Processes don’t work quite like we want with DNS. It’s part of why we haven’t “officially” launched this feature yet! Right now, all processes in the same app show on <app>.internal. It is difficult to run different VMs that expose different services with this setup.

It’s usually better to run caches as separate apps in Fly. You can still use DNS between apps, but you can target things more specifically.

VCL can only use hostnames that return one single IP. I’m not sure how well that will work with Fly hosted apps, especially if you scale to two instances.