Preview: multi process apps (get your workers here!)

Yeah the docs are pretty clear on not using the cache for message - how do you declare that redis insistence? An unconnected toml file with a redis docker image?

Yep! This project works to launch a Redis: GitHub - fly-apps/redis: Launch a Redis server on Fly

You can launch KeyDB through here, too: KeyDB - Launch on Fly.io · Fly

2 Likes

How does autoscaling work with multi process apps?

There seems to be some sort of bug with scaling, not sure if it only affects multi-process apps:

https://community.fly.io/t/an-unknown-error-occurred-during-deployment/2448/18

1 Like

I am hoping this feature is the missing link to a use case. We rely on webhooks for some critical workflows and I am very interested in being able to exercise the system in development. This is hard because our providers require the receiving server to be on the public internet. We would like to tunnel from a running instance to our local machines.

I came to a project called FRP and think it makes our use-case possible. The secondary process would live as long as the primary process and receive traffic. That traffic would be received by development machines. Any thoughts on the feasibility of this model with the Fly infra would be great! Otherwise, I look forward to writing this up. Thanks for any help/guidance the community can share -

Hey @kurt, this is awesome! Exactly what I was looking for.

I know the documentation hasn’t been updated yet, but:

  • How do processes work with private networking? How does one find or address an app process / instance internally?
  • How does this work in a multi-region environment? Is it possible to address the “sibling” instances in the same region?

By way of an explanation, I am building a simple varnish docker container that caches the fly helloname express app. The toml config runs the node app on (8080:8080) and the varnish app (8080:80). It deploys just fine.

Unfortunately, the Varnish VCL is misconfiguration and unable to find the node app instance internally. Because these processes are separate instances with different ip addreses, the varnish VCL cannot use 127.0.0.1 for the backend host because the node app in on another instance.

Any ideas or suggestions?


Edit:
Holy smokes, Same app name / dns. Just different ports.

/ # curl app_name.internal:3000
<h1>Hello From Node on Fly!</h1>

That’s what I am talking about. Now to see if the VCL backend can use app_name.internal as a host.

2 Likes

Processes don’t work quite like we want with DNS. It’s part of why we haven’t “officially” launched this feature yet! Right now, all processes in the same app show on <app>.internal. It is difficult to run different VMs that expose different services with this setup.

It’s usually better to run caches as separate apps in Fly. You can still use DNS between apps, but you can target things more specifically.

VCL can only use hostnames that return one single IP. I’m not sure how well that will work with Fly hosted apps, especially if you scale to two instances.

Hey thanks for the response @kurt. Yeah, I struggled to get the networking working between processes. Even though I could curl app_name.internal:PORT from the app, Varnish was not happy if I tried to use that as a hostname.

Instead, what worked, was the “gross” method of running multiple processes through an entrypoint.sh script.

It’s the same instance, so the varnish backend host can be set to 127.0.0.1, referring to it’s own app server, which means you can easily scale both horizontally, without having to worry about dns and networking. I even got varnishncsa logging Varnish requests to /dev/stdout (hits and misses) which prints to the fly prometheus logs.

Which is SUPER cool, but is that Kosher? Not sure we are supposed to be running requests logs to Prometheus?

Kudos to you and the team. I just discovered Fly.io a few weekends back, but it has been an absolute pleasure to work with. Thank you.

3 Likes

Yeah that’s how we want you to use logs! Prometheus is just metrics, logs go to a monster Elastic cluster that can handle quite a lot.

Now that @benbjohnson1 has some read replica bits live (https://litestream-read-replica-demo.fly.dev/), any rough sketches of those those sidecar-like-things? :heart_eyes:

We’ve started just putting tiny supervisors in our VMs that can run multiple processes. Here’s one for running nats-server with monitoring and a config updater running alongside it: nats-cluster/main.go at supercluster · fly-apps/nats-cluster · GitHub

If we ship sidecars as a feature, it will probably look something like this. Our init can potentially run multiple processes.

The big problem we have is that we’re not using Docker or another container tech that supports sidecars. Most of those rely on shared filesystems and Linux namespaces, we put everything in a much more isolated VM and can’t point two VMs at the same disk.

2 Likes

I see this is not a fly.toml feature but importing nats-cluster/pkg/supervisor at supercluster · fly-apps/nats-cluster · GitHub into a small go program that runs the desired processes.

1 Like

What would the fly.toml look like for a dockerized node.js express app + bull worker… for example. I’m having a hard time visualizing what the [processes] block would look like.

Something like:

[processes]
web = "npm start"
worker = "npm run bull" # or however you invoke the bull worker process

The [processes] block is a key / value group of process names + the command you use to run them.

1 Like

And would this command override the dockerfile CMD?

Yes! These run run in place of the defined CMD.

I was wondering how this works in the context of persistent volumes. I guess the volume does not get attached to both running vms right? If so, is there like a parent vm?

Hi @arttii, if you run a multi-process app, the processes run within a single VM. Does that answer your question? (This did not, in fact, answer the question.)

See Kurt’s answer below, which does.

1 Like

@catflydotio Ah oke I see. I somehow got the impression, that this was starting multiple vms. Like this, it makes sense. Thanks!

1 Like

Just to clarify, this method for running separate processes does start separate VMs. This feature is separate from the in-VM process managers we use elsewhere. It’s very confusing. :smiley:

To mount volumes, you’ll need to add a processes attribute to the mount definition. Something like:

[[mounts]]
processes = ["worker"]
source="myapp_data"
destination="/data"

This will attach the volumes only to worker VMs.

2 Likes

Hey @kurt thanks for the clarification! What’s the other one actually? I thought this was the only flyey method to do it.