Machines not responding to Non Standard Ports

@dfragnito I was looking at your config and I noticed that the mapping was from 8080 (your app) to $port (external world) so assuming you want to get appname.fly.dev:7048 your app should listen to 8080 and $port should be 7048.

I’m assuming this app has public IPs too.

Does this help?

The internal port is 8080

“protocol”: “tcp”,
“internal_port”: 8080

that is what our app is listening on.

The external port is dynamical generated starts at 7000. Many machines are created for an apps each listening on different external ports.

Yes a public fly IP address is generated for the app. Each fly machine end point listens on the external port appname.fly.dev:"externalport". The config is correct well at least it was correct and functioning until last night. It works if external port is 80 or 443 refuse all other ports. If the config is wrong I would think it woudl also fail for external ports 80 and 443.

external port

“port”: ‘.$port.’, //external port. If this is 80 (without tls) or 443 (with tls) it works fails all other ports
“handlers”: [
“tls”,
“http”
]

@dfragnito unfortunately the way you are trying to do app multi-tenancy via external ports does not work. Our proxy doesn’t do a mapping of external port to a specific machine instance so it would in essence choose a random machine from the app to route traffic to in this case.

It appears the underlying service is being exposed via http/https and if so, you should consider using the fly-replay feature of our proxy. We have several customers using it now with success for the type of multi-tenancy you seem to be trying to achieve. The typical setup has a “router” machine where the proxy first routes all requests to and then the router responds with the fly-replay header based on the request attributes to send the request to the correct machine(s).

1 Like

Each Machine has a separate config with a different external port, different en var, and often different regions. Also non standard ports are not working when we have only 1 machine. The underlying service is listening on the internal port 8080.

The fly endpoints
appname.fly.dev:"port1" => machine 1 (config1 external port1)
appname.fly.dev:"port2" => machine 2 (config2 external port2)
appname.fly.dev:"port3" => machine 3 (config3 external port3)
terminating ssl and handing off the request over tcp(http) to the correct machine
to our internal app listening on port 8080 . This all worked up until last night. No need for a router. Its not really multi tenancy each machine are separate instances of our app each connecting in many case to different DBs outside of fly.

This all worked up until last night. Maybe something changed? Maybe this was a unattended use of machines and no longer allowed?

From Machines · Fly Docs
“For an application with multiple machines with the same configuration, requests will be distributed across them. Warm-start behavior in this situation is not well-defined now, so should not be relied upon for apps with multiple machines.”

Each machine has a different config in our case we are not looking to have request distributed across machines although its an option we where considering. I have been testing this setup over the past week and it worked marvelously.

1 Like

I think it was mostly luck that it worked so far.

I used the term “multi-tenancy” there to simply mean 1 app with different machines wherein each machine is for a specific purpose/use. Using fly-replay is our suggested approach for this type of setup.

1 Like

I cannot get 1 machine to to listen on non standard port.

curl -X POST https://sfsql-demos.fly.dev:7057

One machine under this app

curl -X POST https://sfsql-demos.fly.dev:7057

connection refused?

1 Like

To echo dfragnito, any traffic to non-HTTP port (443 / 80) did not work a few min ago.

DoH:

# port 4443 doesn't (internal_port is listening on 8055)
➜  q git:(main) ✗ ./q  POST -t A -s https://udns.fly.dev:4443/ -q instagram.com -v -i
FATA[0000] could not do a GET request to 'https://udns.fly.dev:4443/',
cause: Get "https://udns.fly.dev:4443/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF 

# port 8055 doesn't (this external port is same as internal_port)
➜  q git:(main) ✗ ./q  POST -t A -s https://udns.fly.dev:8055/ -q instagram.com -v -i
FATA[0003] could not do a GET request to 'https://udns.fly.dev:10555/',
cause: Get "https://udns.fly.dev:10555/?dns=AAABAAABAAAAAAAACWluc3RhZ3JhbQNjb20AAAEAAQ": EOF 

# port 443 works (standard https, but internal_port is same as before, 8055)
➜  q git:(main) ✗ ./q  POST -t A -s https://udns.fly.dev:443/ -q instagram.com -v -i
instagram.com. 2m1s A 157.240.16.174

But:

It has miraculously started working as of few seconds ago!

Looking at the machine config for the sfsql-demos app, I’m seeing port 7000 defined for external routing and 8080 for internal (granted port 7000 also appears to not work).

We’re looking at a few things now to see if it’s a more general issue with non-standard ports.

1 Like

Hmm im still getting connection refused for non standard external, port tls (7059) 8080 internal
but all is good for standard secured port 443 8080 internal

1 Like

yes it was port 7000 not 7057 sorry, but you now you see the issue with the non standard ports. Thanks You.

Since the fly-replay header only works for http, any timelines to support other protos (AWS GLA does with what they call custom routing, which is quite neat and enables applications where co-location of clients is paramount)?

I changed the machine app to never kill itself and deployed it… and now… after some while… it has automagically started responding to traffic sent to non-HTTP ports. Not sure, if not killing the machine and leaving it up for an hour or so, had this effect…

@dfragnito I saw you destroyed the machine for that app. Would you be able to deploy another one for the same app (same config) to iad again and test?

curl -X POST https://sfsql-demos.fly.dev:7061/

Failed to connect to sfsql-demos.fly.dev port 7061: Connection refused

Failed at port 7061 Do I need to use the same port 7000?

No, port 7061 is what you have defined on the machine config so that’s good. My guess is you’re hitting an edge where we haven’t rolled out the fix yet. Will let you know when we’re done with the rollout.

ok great thanks

Ok, I was able to get a response so please try again when you can.

❯ curl -i https://sfsql-demos.fly.dev:7061
HTTP/2 404
date: Tue, 06 Sep 2022 20:48:40 GMT
server: Fly/7140f6f3 (2022-09-06)
via: 2 fly.io
fly-request-id: 01GCA8ZWFVH5BPT5PMJVR3FG5N-iad

Looking good. I am going now test firing up multiple machines at different ports and store data in each and query each one independently and lets see if the correct results come back.

1 Like

FYI, this is something we’re probably going to restrict in the future. The proxy design we have in place is not good for running a whole bunch of isolated ports on a single IP. It’s sort of an oversight that it works at all. :slight_smile:

The “supported” path here is to run an app in front of machines to route traffic, either by proxying directly or using fly-replay.

1 Like

Ok I see. I just tested persisting data to each db and the correct data is being returned when hitting the machines at the different ports so you are correctly routing based on ports. or I am just very lucky today.

It was nice while it lasted. I think we will create a machine per app, clientID-App1, ClientID-App2 etc… machines all at standard https port (not feeling real confident with the non standard ports). The nice part about all customer machines under one app is that billing was easy since you break down our cost by app. From our end users perspective it makes no difference.

In case your wondering these demo machines are all connecting to a planetscale db.

Is there a billing API that return cost break down?

Following up on this. We are now creating multiple user apps representing a single “Object Store”. Each app has 1 machine reachable at the app endpoint at the standard port 443. Users can create and destroy apps, stop, start, destroy and recreate the machine. No more port mapping at a single IP. I guess that was a bit of a hack (our demos are still doing this)

We can now use secrets instead of env variables having multiple machines under the app made it difficult to use secrets, since machines inherit secrets from the app and in our case the values needed to be different for each machine.

I’m looking but cant find a “billing” API. We pass the fly fees over to our customers. Can you point me in the right direction. I see this Sign In · Fly but don’t see how to access this info from an API call. I hope it’s not a graphql query. I just find a rest API easier. I did check out Graphql but it did not seem to have a query we could use that would return our cost per app. Then again I’m not proficient in Graphql.

Thanks

1 Like