Machines we create at non standard ports with and without TLS are refusing connections over the app endpoint over http and https. I
Looks like machines are not responded to http and https request at non standard ports. HTTP and HTTPS both work at standard ports 80 and 443. IP address is created for the app prior to creating the machine.
Machines created listening on non standard ports are responding with
“Failed to connect to “appname”.fly.dev port 7048: Connection refused”
Example config below. Also tried without tls still no love.
Wow, you’re right (even if this example that connects to machines on port
4000 suggests otherwise). Non-HTTP ports (I tried
8055) don’t work for some test deploys (via
flyctl deploy --config ...) I did just a few moments ago. Only
Thanks for the confirmation. I thought is was us but the more I tested I realized there was an issue. We are using the machine API mostly creating multiple machines under an app at different ports. Was working great until last night. Maybe it’s a policy change, no more non standard public facing ports? I know that some edge function providers do not allow request to not standard ports but that’s not exactly the same thing.
@dfragnito I was looking at your config and I noticed that the mapping was from 8080 (your app) to $port (external world) so assuming you want to get
appname.fly.dev:7048 your app should listen to 8080 and $port should be 7048.
I’m assuming this app has public IPs too.
Does this help?
The internal port is 8080
that is what our app is listening on.
The external port is dynamical generated starts at 7000. Many machines are created for an apps each listening on different external ports.
Yes a public fly IP address is generated for the app. Each fly machine end point listens on the external port
appname.fly.dev:"externalport". The config is correct well at least it was correct and functioning until last night. It works if external port is 80 or 443 refuse all other ports. If the config is wrong I would think it woudl also fail for external ports 80 and 443.
“port”: ‘.$port.’, //external port. If this is 80 (without tls) or 443 (with tls) it works fails all other ports
@dfragnito unfortunately the way you are trying to do app multi-tenancy via external ports does not work. Our proxy doesn’t do a mapping of external port to a specific machine instance so it would in essence choose a random machine from the app to route traffic to in this case.
It appears the underlying service is being exposed via http/https and if so, you should consider using the fly-replay feature of our proxy. We have several customers using it now with success for the type of multi-tenancy you seem to be trying to achieve. The typical setup has a “router” machine where the proxy first routes all requests to and then the router responds with the
fly-replay header based on the request attributes to send the request to the correct machine(s).
Each Machine has a separate config with a different external port, different en var, and often different regions. Also non standard ports are not working when we have only 1 machine. The underlying service is listening on the internal port 8080.
The fly endpoints
appname.fly.dev:"port1" => machine 1 (config1 external port1)
appname.fly.dev:"port2" => machine 2 (config2 external port2)
appname.fly.dev:"port3" => machine 3 (config3 external port3)
terminating ssl and handing off the request over tcp(http) to the correct machine
to our internal app listening on port 8080 . This all worked up until last night. No need for a router. Its not really multi tenancy each machine are separate instances of our app each connecting in many case to different DBs outside of fly.
This all worked up until last night. Maybe something changed? Maybe this was a unattended use of machines and no longer allowed?
From Machines · Fly Docs
“For an application with multiple machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple machines.”
Each machine has a different config in our case we are not looking to have request distributed across machines although its an option we where considering. I have been testing this setup over the past week and it worked marvelously.
I think it was mostly luck that it worked so far.
I used the term “multi-tenancy” there to simply mean 1 app with different machines wherein each machine is for a specific purpose/use. Using fly-replay is our suggested approach for this type of setup.
I cannot get 1 machine to to listen on non standard port.
curl -X POST https://sfsql-demos.fly.dev:7057
One machine under this app
curl -X POST https://sfsql-demos.fly.dev:7057
To echo dfragnito, any traffic to non-HTTP port (
80) did not work a few min ago.
# port 4443 doesn't (internal_port is listening on 8055) ➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:4443/ -q instagram.com -v -i FATA could not do a GET request to 'https://udns.fly.dev:4443/', cause: Get "https://udns.fly.dev:4443/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF # port 8055 doesn't (this external port is same as internal_port) ➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:8055/ -q instagram.com -v -i FATA could not do a GET request to 'https://udns.fly.dev:10555/', cause: Get "https://udns.fly.dev:10555/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF # port 443 works (standard https, but internal_port is same as before, 8055) ➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:443/ -q instagram.com -v -i instagram.com. 2m1s A 188.8.131.52
It has miraculously started working as of few seconds ago!
Looking at the machine config for the
sfsql-demos app, I’m seeing port 7000 defined for external routing and 8080 for internal (granted port 7000 also appears to not work).
We’re looking at a few things now to see if it’s a more general issue with non-standard ports.
Hmm im still getting connection refused for non standard external, port tls (7059) 8080 internal
but all is good for standard secured port 443 8080 internal
yes it was port 7000 not 7057 sorry, but you now you see the issue with the non standard ports. Thanks You.
fly-replay header only works for
http, any timelines to support other protos (AWS GLA does with what they call custom routing, which is quite neat and enables applications where co-location of clients is paramount)?
I changed the machine app to never kill itself and deployed it… and now… after some while… it has automagically started responding to traffic sent to non-HTTP ports. Not sure, if not killing the machine and leaving it up for an hour or so, had this effect…
@dfragnito I saw you destroyed the machine for that app. Would you be able to deploy another one for the same app (same config) to
iad again and test?
curl -X POST https://sfsql-demos.fly.dev:7061/
Failed to connect to sfsql-demos.fly.dev port 7061: Connection refused
Failed at port 7061 Do I need to use the same port 7000?
No, port 7061 is what you have defined on the machine config so that’s good. My guess is you’re hitting an edge where we haven’t rolled out the fix yet. Will let you know when we’re done with the rollout.
ok great thanks
Ok, I was able to get a response so please try again when you can.
❯ curl -i https://sfsql-demos.fly.dev:7061 HTTP/2 404 date: Tue, 06 Sep 2022 20:48:40 GMT server: Fly/7140f6f3 (2022-09-06) via: 2 fly.io fly-request-id: 01GCA8ZWFVH5BPT5PMJVR3FG5N-iad