exq Worker Not Starting Successfully...

,
1

Heyo. Trying to get exq going in Fly.io - split between to apps (the enqueuer, and the worker). Separating from a single repo/umbrella for reasons.

I’ve got the deploy pipelined up, but it doesn’t seem to start successfully:

2022-12-05T21:44:09Z runner[559dbcc0] iad [info]Pulling container image
2022-12-05T21:44:23Z runner[559dbcc0] iad [info]Pull failed, retrying (attempt #0)
2022-12-05T21:44:23Z runner[559dbcc0] iad [info]Unpacking image
2022-12-05T21:44:35Z runner[559dbcc0] iad [info]Preparing kernel init
2022-12-05T21:44:36Z app[559dbcc0] iad [info]Preparing to run: `launcher MIX_ENV=prod mix do app.start, exq.run` as heroku
2022-12-05T21:44:36Z app[559dbcc0] iad [info]2022/12/05 21:44:36 listening on [fdaa:0:33c6:a7b:92:559d:bcc0:2]:22 (DNS: [fdaa::3]:53)
2022-12-05T21:44:43Z runner[559dbcc0] iad [info]Starting instance
2022-12-05T21:44:48Z runner[559dbcc0] iad [info]Configuring virtual machine
2022-12-05T21:44:48Z runner[559dbcc0] iad [info]Pulling container image
2022-12-05T21:44:49Z runner[559dbcc0] iad [info]Starting virtual machine
2022-12-05T21:44:49Z app[559dbcc0] iad [info]Starting init (commit: f447594)...
2022-12-05T21:45:02Z runner[559dbcc0] iad [info]Unpacking image
2022-12-05T21:45:02Z runner[559dbcc0] iad [info]Preparing kernel init
2022-12-05T21:45:03Z runner[559dbcc0] iad [info]Starting virtual machine
2022-12-05T21:45:03Z app[559dbcc0] iad [info]Preparing to run: `launcher MIX_ENV=prod mix do app.start, exq.run` as heroku
2022-12-05T21:45:03Z app[559dbcc0] iad [info]2022/12/05 21:45:03 listening on [fdaa:0:33c6:a7b:92:559d:bcc0:2]:22 (DNS: [fdaa::3]:53)

My fly.toml is simple:

# fly.toml file generated for lei-worker on 2022-12-05T16:33:18-05:00

app = "lei-worker"
kill_signal = "SIGINT"
kill_timeout = 5

[build]
  builder = "heroku/buildpacks:20"
  buildpacks = ["https://cnb-shim.herokuapp.com/v1/hashnuke/elixir"]

[env]
  PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[processes]
  worker = "MIX_ENV=prod mix do app.start, exq.run"

[[services]]
  processes = ["worker"]

Am wondering if there’s a healthcheck that’s happening, or trying to but I’ve got nothing. This worker only ever talks to redis, so I could confirm the connection using redis-cli or something? Or does the worker need to respond to some sort of healthcheck to get declared OK to the VM?

TIA.
Kit

2

Hi @kitplummer,

I don’t have any experience deploying Elixir apps using the Heroku buildpacks. I use and recommend Dockerfiles and creating releases.

When you build a release, there is no “mix” available in the deployed application.

If you are deploying two separate applications, then they could be clustered together and you wouldn’t be using worker processes either.

I don’t know if separate workers get healthchecks. :thinking: This is just a very different way of deploying than what I’m familiar with. Perhaps others with more experience with this approach can weigh in?

3

I’d originally tried working with the container, but ran into a bunch of issues - so bailed. Can definitely figure that out, and work the the entrypoint to get it started. Just kinda figured the buildpack should work.

4

That all makes sense. Sorry I can’t be of more help there!

5

Well, for posterity - I was following this:

And in Googling I found this:

Which says this:

A: It is, with one caveat: your applications need to have a network service of some kind. Why is that? Well, when your app deploys, we run health checks to ensure you can connect to it. They are, by default, TCP connection checks but we do have an option for full HTTP checks. Anyway, unless there is a network service that can be connected to, the health checks will fail and Fly will kill off the app for being apparently faulty.

We do this ourselves for some of our internal services. When we run our Buildkite agents, we use the fact that they export Prometheus metrics on port 8080 and use that port for the health checks to look at.

And no, this isn't how things are going to stay. We have support for Worker processes on the way.

Would love to know if this is still the case. Thinking so, and am going to implement a simple TCP listener. Might seem like I’m working around containers - well, that’s because I am.