RabbitMQ on Fly not working?

Questions / Help
1

I’m trying to set up RabbitMQ on Fly and I thought it would be straightforward but I was wrong! I’ve had a look at this thread: RabbitMQ on Fly.io

I have the following fly.toml:

app = "rabbitmq-amplified"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
RABBITMQ_MNESIA_DIR = "/var/lib/rabbitmq/mnesia/data"

[experimental]
allowed_public_ports = []
auto_rollback = true

[build]
image = "rabbitmq:3-management"

[[services]]
http_checks = []
internal_port = 5672
processes = ["app"]
protocol = "tcp"
script_checks = []

[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"

# rabbitmq admin
[[services]]
http_checks = []
internal_port = 15672
protocol = "tcp"
script_checks = []

[[services.ports]]
handlers = ["http", "tls"]
port = "15672"

[[services.tcp_checks]]
grace_period = "1s"

But I can’t access anything. When I ssh in it seems all is functioning correctly and advertising on the right ports. Monitoring looks fine. But going to the app url:port gives me nothing. Maybe I’m missing something super obvious?

2

Use the hostname rabbitmq-amplified.internal to access the rabbitmq from other apps in your org. Let us know if you’re having issues with that.

In this part of the admin services config:

the handlers are processed in order. For a web app, terminate tls then handle http with:

handlers = ["tls", "http"]

You may not want that admin site to be on the public internet, which it will be with the current configuration. If not, something like tailscale or a vpn would enable remote access without exposing the admin page.

3

Thank you! Yeah… no luck. I tried accessing it via fly proxy as well and no dice. I did try flipping the handlers as suggested (good to know!) and still nothing.

4

Let me share my files, hope it helps you

Dockerfile

FROM rabbitmq:3.11-management
COPY prod.conf /etc/rabbitmq/rabbitmq.conf
RUN apt-get -o Acquire::Check-Date=false update && apt-get install -y curl
RUN curl -L https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases/download/3.11.1/rabbitmq_delayed_message_exchange-3.11.1.ez > $RABBITMQ_HOME/plugins/rabbitmq_delayed_message_exchange-3.11.1.ez
RUN chown rabbitmq:rabbitmq $RABBITMQ_HOME/plugins/rabbitmq_delayed_message_exchange-3.11.1.ez
RUN rabbitmq-plugins enable rabbitmq_delayed_message_exchange

prod.conf

# rabbitmq config file
listeners.tcp.default = 5672
default_user = user
default_pass = pass
log.file.level = warning
log.console.level = warning
vm_memory_high_watermark.relative = 0.9 # 90% of available RAM

fly.toml

app = "broker"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  dockerfile = "Dockerfile"

[env]
  PRIMARY_REGION = "iad"
  RABBITMQ_MNESIA_DIR = "/var/lib/rabbitmq/mnesia/data"

[[services]]
  http_checks = []
  internal_port = 5672
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

# rabbitmq admin
[[services]]
  http_checks = []
  internal_port = 15672
  protocol = "tcp"
  script_checks = []

  [[services.ports]]
    handlers = ["http", "tls"]
    port = "15672"

  [[services.tcp_checks]]
    grace_period = "1s"
5

Thanks Marco! That’s pretty close to identical to what I have and no luck.

6

This is my fly.toml

app = "some-app"
kill_signal = "SIGTERM"
kill_timeout = 300

[env]
  PRIMARY_REGION = "lax"
  RABBITMQ_MNESIA_DIR = "/var/lib/rabbitmq/mnesia/data"

[build]
    image = "rabbitmq:3-management"

[experimental]
  allowed_public_ports = []
  auto_rollback = false
  private_network = true

[[services]]
  http_checks = []
  internal_port = 5672
  protocol = "tcp"
  script_checks = []

  [[services.ports]]
    handlers = ["http"]
    port = "5672"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[services]]
  http_checks = []
  internal_port = 15672
  protocol = "tcp"
  script_checks = []

  [[services.ports]]
    handlers = ["http", "tls"]
    port = "15672"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[mounts]
  source="rabbitmq_data"
  destination="/var/lib/rabbitmq/mnesia/data"

The only difference is the mounts for the Mnesia data I think. This works and you can visit the management ui with some-app.fly.dev:15672.

What I’m having trouble with now is running rabbit in a cluster. I’m trying to use the new machines api + terrraform to manage this. Don’t know if anyone has been successful with running a cluster of rabbit nodes on fly.

7

Thanks! Tried copying that more or less exactly, including adding a volume. Still not working for me. I’m really not sure what’s going on. The container seems fine internally. I can’t access :15672 no matter what I seem to do.

8

FWIW we finally made this work - the issue is that the management plugin listens to IPv4 only by default, so was unreachable via IPv6. Adding management.tcp.ip = :: to the RabbitMQ configuration gets everything working as intended.

9

Nice. Not sure why mine works without that. I do have ipv6 mode enabled for rabbit and have fly wire guard on for my local setup. Btw are you folks running going to run a cluster?