Machines we create at non standard ports with and without TLS are refusing connections over the app endpoint over http and https. I
More Info
Looks like machines are not responded to http and https request at non standard ports. HTTP and HTTPS both work at standard ports 80 and 443. IP address is created for the app prior to creating the machine.
Machines created listening on non standard ports are responding with
“Failed to connect to “appname”.fly.dev port 7048: Connection refused”
Example config below. Also tried without tls still no love.
http://“.$con->{‘FLY_API_HOSTNAME’}.”/v1/apps/“.$con->{‘app_name’}.”/machines";
$json=‘{
“name”: "’.$l[‘db_id’].‘",
“region”: "’.$region.‘",
“config”: {
“image”: "’.$con->{‘image’}.‘",
“env”: {
“SF_LICENSEKEY”: "’.$l[‘licensekey’].'"
},
“services”: [
{
“ports”: [
{
“port”: ‘.$port.’,
“handlers”: [
“tls”,
“http”
]
}
],
“protocol”: “tcp”,
“internal_port”: 8080
}
]
}
}
Wow, you’re right (even if this example that connects to machines on port 4000 suggests otherwise). Non-HTTP ports (I tried 853, 4443, 10555, 8055) don’t work for some test deploys (via flyctl deploy --config ...) I did just a few moments ago. Only 80 and 443 work.
cc: @lubien
2 Likes
Thanks for the confirmation. I thought is was us but the more I tested I realized there was an issue. We are using the machine API mostly creating multiple machines under an app at different ports. Was working great until last night. Maybe it’s a policy change, no more non standard public facing ports? I know that some edge function providers do not allow request to not standard ports but that’s not exactly the same thing.
@dfragnito I was looking at your config and I noticed that the mapping was from 8080 (your app) to $port (external world) so assuming you want to get appname.fly.dev:7048 your app should listen to 8080 and $port should be 7048.
I’m assuming this app has public IPs too.
Does this help?
The internal port is 8080
“protocol”: “tcp”,
“internal_port”: 8080
that is what our app is listening on.
The external port is dynamical generated starts at 7000. Many machines are created for an apps each listening on different external ports.
Yes a public fly IP address is generated for the app. Each fly machine end point listens on the external port appname.fly.dev:"externalport". The config is correct well at least it was correct and functioning until last night. It works if external port is 80 or 443 refuse all other ports. If the config is wrong I would think it woudl also fail for external ports 80 and 443.
external port
“port”: ‘.$port.’, //external port. If this is 80 (without tls) or 443 (with tls) it works fails all other ports
“handlers”: [
“tls”,
“http”
]
@dfragnito unfortunately the way you are trying to do app multi-tenancy via external ports does not work. Our proxy doesn’t do a mapping of external port to a specific machine instance so it would in essence choose a random machine from the app to route traffic to in this case.
It appears the underlying service is being exposed via http/https and if so, you should consider using the fly-replay feature of our proxy. We have several customers using it now with success for the type of multi-tenancy you seem to be trying to achieve. The typical setup has a “router” machine where the proxy first routes all requests to and then the router responds with the fly-replay header based on the request attributes to send the request to the correct machine(s).
1 Like
Each Machine has a separate config with a different external port, different en var, and often different regions. Also non standard ports are not working when we have only 1 machine. The underlying service is listening on the internal port 8080.
The fly endpoints
appname.fly.dev:"port1" => machine 1 (config1 external port1)
appname.fly.dev:"port2" => machine 2 (config2 external port2)
appname.fly.dev:"port3" => machine 3 (config3 external port3)
terminating ssl and handing off the request over tcp(http) to the correct machine
to our internal app listening on port 8080 . This all worked up until last night. No need for a router. Its not really multi tenancy each machine are separate instances of our app each connecting in many case to different DBs outside of fly.
This all worked up until last night. Maybe something changed? Maybe this was a unattended use of machines and no longer allowed?
From Machines · Fly Docs
“For an application with multiple machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple machines.”
Each machine has a different config in our case we are not looking to have request distributed across machines although its an option we where considering. I have been testing this setup over the past week and it worked marvelously.
1 Like
I think it was mostly luck that it worked so far.
I used the term “multi-tenancy” there to simply mean 1 app with different machines wherein each machine is for a specific purpose/use. Using fly-replay is our suggested approach for this type of setup.
1 Like
I cannot get 1 machine to to listen on non standard port.
curl -X POST https://sfsql-demos.fly.dev:7057
One machine under this app
curl -X POST https://sfsql-demos.fly.dev:7057
connection refused?
1 Like
To echo dfragnito, any traffic to non-HTTP port (443 / 80) did not work a few min ago.
DoH:
# port 4443 doesn't (internal_port is listening on 8055)
➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:4443/ -q instagram.com -v -i
FATA[0000] could not do a GET request to 'https://udns.fly.dev:4443/',
cause: Get "https://udns.fly.dev:4443/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF
# port 8055 doesn't (this external port is same as internal_port)
➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:8055/ -q instagram.com -v -i
FATA[0003] could not do a GET request to 'https://udns.fly.dev:10555/',
cause: Get "https://udns.fly.dev:10555/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF
# port 443 works (standard https, but internal_port is same as before, 8055)
➜ q git:(main) ✗ ./q POST -t A -s https://udns.fly.dev:443/ -q instagram.com -v -i
instagram.com. 2m1s A 157.240.16.174
But:
It has miraculously started working as of few seconds ago!
Looking at the machine config for the sfsql-demos app, I’m seeing port 7000 defined for external routing and 8080 for internal (granted port 7000 also appears to not work).
We’re looking at a few things now to see if it’s a more general issue with non-standard ports.
1 Like
Hmm im still getting connection refused for non standard external, port tls (7059) 8080 internal
but all is good for standard secured port 443 8080 internal
1 Like
yes it was port 7000 not 7057 sorry, but you now you see the issue with the non standard ports. Thanks You.
Since the fly-replay header only works for http, any timelines to support other protos (AWS GLA does with what they call custom routing, which is quite neat and enables applications where co-location of clients is paramount)?
I changed the machine app to never kill itself and deployed it… and now… after some while… it has automagically started responding to traffic sent to non-HTTP ports. Not sure, if not killing the machine and leaving it up for an hour or so, had this effect…
@dfragnito I saw you destroyed the machine for that app. Would you be able to deploy another one for the same app (same config) to iad again and test?
curl -X POST https://sfsql-demos.fly.dev:7061/
Failed to connect to sfsql-demos.fly.dev port 7061: Connection refused
Failed at port 7061 Do I need to use the same port 7000?
No, port 7061 is what you have defined on the machine config so that’s good. My guess is you’re hitting an edge where we haven’t rolled out the fix yet. Will let you know when we’re done with the rollout.
ok great thanks
Ok, I was able to get a response so please try again when you can.
❯ curl -i https://sfsql-demos.fly.dev:7061
HTTP/2 404
date: Tue, 06 Sep 2022 20:48:40 GMT
server: Fly/7140f6f3 (2022-09-06)
via: 2 fly.io
fly-request-id: 01GCA8ZWFVH5BPT5PMJVR3FG5N-iad