Need help with deploying and connecting rabbit MQ to a deployed app

I have a node.js application I have deployed on fly.io that I am trying to connect to another deployed docker instance of rabbit MQ. However, I am having trouble connecting to the rabbit MQ app to the platform due to this error that pops up on my node.js application:

{
  errno: -4077,
  code: 'ECONNRESET',
  syscall: 'read'
}

I kind of have an idea of what this could be, but I’m not sure to be honest. Here is the docker file that I am trying to deploy:

FROM rabbitmq:3-management-alpine
COPY ./prod.conf /etc/rabbitmq/rabbitmq.conf
RUN rabbitmq-plugins enable rabbitmq_management

here is the fly TOML file:

# fly.toml app configuration file generated for rabbit-mq-fly-dep on 2024-09-25T14:16:38-04:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'rabbit-mq-fly-dep'
primary_region = 'atl'

[build]

[[services]]
  protocol = "tcp"
  internal_port = 5672

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

[[services]]
  protocol = "tcp"
  internal_port = 15672

  [[services.ports]]
    port = 15672
    handlers = ["tls", "http"]

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

here is the .cofig file for the rabbit mq deployment

listeners.tcp.default = 5672
default_user = [my_rabbit_mq_password]
default_pass = [may_rabbit_mq_password]
management.tcp.ip = ::

I should also say that I did use the correct format for the amqp URL to connect via tcp and tls to the deployed rabbit mq instance:

RABIT_MQ_URL=amqp://username:password@rabbit-mq-fly-dep.fly.dev:portnumber/vhost

(NOTE:I should also say that I tried amqps, but no dice, unfortunately.)

I tried executing the code again and monitor the fly logs, but they didn’t update. I don’t understand this, because it says that the application started listening on the expected port (5672) as you can see within the respective line in the logs I have here:

2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.111223+00:00 [info] <0.674.0> Ready to start client connection listeners
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.112324+00:00 [info] <0.832.0> started TCP listener on [::]:5672
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info] completed with 5 plugins.
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0> Server startup complete; 5 plugins started.
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0>  * rabbitmq_prometheus
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0>  * rabbitmq_federation
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0>  * rabbitmq_management
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0>  * rabbitmq_management_agent
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.173596+00:00 [info] <0.674.0>  * rabbitmq_web_dispatch
2024-10-01T16:59:09Z app[d8dd761ce00038] atl [info]2024-10-01 16:59:09.306912+00:00 [info] <0.9.0> Time to start RabbitMQ: 6214 ms
2024-10-01T16:59:18Z health[d8dd761ce00038] atl [info]Health check on port 5672 is now passing.
2024-10-01T16:59:19Z health[d8dd761ce00038] atl [info]Health check on port 15672 is now passing.
2024-10-01T17:00:03Z app[d8dd761ce00038] atl [info]2024-10-01 17:00:03.182211+00:00 [notice] <0.86.0>     alarm_handler: {set,{system_memory_high_watermark,[]}}

however, this resulted in the same error as above. I have been up and down the internet trying to solve this, but no luck. I would appreciate it if someone could give me a hand in this. LOL. Thanks!

Hi… Typically you use the 6PN address instead of the public one for internal services. This is both more secure and more convenient (simultaneously, :black_cat:). Try substituting rabbit-mq-fly-dep.internal for rabbit-mq-fly-dep.fly.dev in the URL—and then remove both [[services]] blocks from fly.toml (unless you do need external access for some other reason).

Hope this helps!

Added proxy

I removed the service blocks in the TOML to where it looks like this:

# fly.toml app configuration file generated for rabbit-mq-fly-dep on 2024-09-25T14:16:38-04:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'rabbit-mq-fly-dep'
primary_region = 'atl'

[build]

[[services]]
  protocol = "tcp"
  internal_port = 15672

  [[services.ports]]
    port = 15672
    handlers = ["tls", "http"]

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

I absolutely need access to my admin instance to add a new account, because I read somewhere that the default admin account is iffy if your trying to use it over a network and not with a local host. I also need it to create a virtual host.

I also updated my url:

RABIT_MQ_URL=amqp://username:password@rabbit-mq-fly-dep.internal:portnumber/vhost

However, the error remains the same

Wait, wait wait. Hold on xD I should say that I am trying to access the deployed rabbit mq instance over a LOCAL node.js api instance. I think this could be the problem, and as to why I’m getting kicked because of fly.io’s security policies? I looked over the docs for the 6PN address, and it said that fly.io can be picky about providing outside apps access. I think this could be it??? perhaps it will work after I deploy my API???

Right… Your local laptop/desktop doesn’t have access to the .internal network by default.

(fly proxy is the easiest way to change that temporarily.)

-_____- uuuuuugggggghhhhhh! Ok LOL let me try that.

I thought this was more straight forward like apparently other people who had slightly different problems with rabbitmq, but dang, I guess not! xD

So I would ESSENTIALLY use a proxied local host to serve this instance up locally AND THEN connect my local API instance to it locally. Then when I’m done, I terminate that proxy, reset my url BACK to the live one (or have another “live” deployment config with this url instead of the local host.) and then deploy it. Am I understanding this correctly?

Exactly… You could also use that for your admin operations. (That’s what I would do.)

Usually these are called something along the lines of “dev” and “prod” environments, but I don’t know what the specific Node idiom is.

(One of the local Node experts might chime in…)

Thanks! I will try this and get back to you.

To be honest, I know this makes sense, but its really weird xD

Sorry! :adhesive_bandage: I know I did promise “convenience” up there…

(For the things that are running within the Fly.io platform itself, it really is easier, though…)

Nah, its tech. Stuff is wierd some times LOL

1 Like

I will say right now I am trying to figure out how to get the fly.io cli onto my computer. (PC), this is apparently a bit of a challenge as well, but once I get this installed on my local machine, I will test it out, and let you know how it goes. I do have it on my macbook though, I just want to get it installed on my pc as well.

Crap, I forgot to mention, I also have this problem too:


WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.  
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:15672
Found these processes inside the machine with open listening sockets:
  PROCESS  

so I don’t think I can get access to the server through a proxy unfortunately :confused:

any help on how to fix this problem would be great as well.

PS. I would think you would need to use a service block like the following TOML lines as well if you want an outside application to access it as well such as a proxy server for local testing:

[[services]]
  protocol = "tcp"
  internal_port = 5672

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

because rabbit MQ listens by default on port 5672, of which, would need to be proxied in this case, (at least that is my understanding) however, I cannot get it to access this port either via fly proxy with my current configuration. If there is some other wizardry that I am missing to access this application via proxy, please let me know. (Also, let me know if I need to do something different if the whole thing with the 6PN addresses needs to be involved as well. Thanks.)

If it does help, here is the error with the aforementioned line added to the fly.toml file:

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.  

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.  
You can fix this by configuring your app to listen on the following addresses:
WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.  
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:5672
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:5672
  - 0.0.0.0:5672
  - 0.0.0.0:15672
  - 0.0.0.0:15672
Found these processes inside the machine with open listening sockets:

and here is the fly.toml file:

# fly.toml app configuration file generated for rabbit-mq-fly-dep on 2024-09-25T14:16:38-04:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'rabbit-mq-fly-dep'
primary_region = 'atl'

[build]

[[services]]
  protocol = "tcp"
  internal_port = 5672

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

[[services]]
  protocol = "tcp"
  internal_port = 15672

  [[services.ports]]
    port = 15672
    handlers = ["tls", "http"]

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "1m0s"
    grace_period = "1s"

I should also note that I tried to update my config file, but to no avail to try and accommodate this error:

here is my prod.config file for rabbit mq (let me know if I did this wrong):

listeners.tcp.default = 0.0.0.0:5672
default_user = admin
default_pass = admin
management.tcp.ip = ::

So basically, I have no idea how to configure these addresses.

I also got this error with a port that it was listening on for erlang (4369) as shown here with fly hallpass (whatever that is lol) after I deployed the latest prod.config file:

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:5672
  - 0.0.0.0:15672
Found these processes inside the machine with open listening sockets:
  PROCESS                                               | ADDRESSES
--------------------------------------------------------*---------------------------------------  
  /.fly/hallpass                                        | [fdaa:9:bf74:a7b:2d6:271b:4051:2]:22    
  /opt/erlang/lib/erlang/erts-14.2.5.3/bin/epmd -daemon | 0.0.0.0:4369, [::]:4369


-------
 ✔ Cleared lease for d8dd761ce00038
-------

update: I was able to get my deployed server to connect to a fly proxy on my local machine despite this error on the management port 15672, but if I try to use an account that I set up (that isn’t the admin account) I am unable to log into the management dashboard on the proxied instance. I also tried the admin account for the heck of it, but no dice. Next I tried to connect to the locally proxied instance on port 5672 with the following URL:

RABIT_MQ_URL=amqp://username:password@localhost:5672/vhost

bust STILL nothing, just the same ECONNRESET error as before

IDK I’m confused xD.

For my apps to communicate w/ my other apps internally, I have to bind my services on address [::]. I’m not sure if that’s relevant for your issue.

Oh! Looks like that worked! :slight_smile: Testing now…

Okay. Bit by bit this is starting to work! :slight_smile: The data that I added into the proxied instance, the new user, and the new vhost transferred over to the deployed instance on fly.io. However, I am still getting the ECONNRESET error.

Here is my config.prod files that I have tried:

attempt 1:

listeners.tcp.default = 5672
management.listener.port = 15672
management.listener.ip = ::
default_user = admin
default_pass = admin
deprecated_features.permit.management_metrics_collection = true
deprecated_features.permit.management_metrics_collection = false

attempt 2:

listeners.tcp.default = 0.0.0.0:5672
management.listener.port = 0.0.0.0:15672
management.listener.ip = ::
default_user = admin
default_pass = admin

and here is a list of values that I used for the rabbit mq instance that I have proxied on the localhost:

attempt 1:

RABIT_MQ_URL=amqp://username:password@localhost:5672/vhost

attempt 2:

RABIT_MQ_URL=amqp://localhost:5672
#I also tried adding the vhost name to the url as well it still didn't work
#I also tried the amqps:// protocal with these urls, and still nothing.

just so you know, I do have multiple instances of proxied ports running on separate terminals:

management port proxy:

PS [none_of_your_business_lol]\RabbitMQDep> fly proxy 15672:15672
Proxying local port 15672 to remote [rabbit-mq-fly-dep.internal]:15672

listening queue port proxy:

PS  [none_of_your_business_lol]\RabbitMQDep> fly proxy 5672:5672
Proxying local port 5672 to remote [rabbit-mq-fly-dep.internal]:5672

We’re getting closer though guys! :slight_smile: I just want to say how much I appreciate all of your help! You all have been wonderful!

@khuezy is right about IPv6, and it looks like you’ve maybe fixed that for the management port (15672) but not for the main port (5672).

A quick glance at the RabbitMQ networking documentation suggests the following instead:

listeners.tcp.default = :::5672

You can see what really is listening on your machine at present by combining SSH with ss. (Those are Debian instructions, but Alpine should have something similar.)


Aside: It’d be wise to remove all of the [[services]] blocks and all the public IP addresses, since those are exposing your message queue to the outside world unnecessarily. The warning that you quoted earlier about fly-proxy is a red herring; it’s referring to a different Fly Proxy (unfortunately). The one that you’re using (i.e., the flyctl subcommand) is oblivious to [[services]].