Cannot access app through fly.io subdomain

Questions / Help Build debugging
1

I deployed a remix application today on fly.io, and it’s deploying successfully. If I use the fly proxy 8080 command, I can access the app on localhost, as I should be able to. However, I cannot access the app through the subdomain: twelvepool.fly.dev.

here is my fly.toml:

# fly.toml file generated for twelvepool on 2022-03-24T14:41:03Z

app = "twelvepool"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
  PORT = "8080"
  PRIMARY_REGION = "ewr"

[deploy]
  release_command = "npx prisma migrate deploy"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80
    force_https = true

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

  [[services.http_checks]]
    interval = 10000
    grace_period = "5s"
    method = "get"
    path = "/healthcheck"
    protocol = "http"
    timeout = 2000
    tls_skip_verify = false
    [services.http_checks.headers]
2

I am looking at this! It looks like DNS for twelvepool.fly.dev is not resolving correctly, which is a problem on our end and not yours.

3

This should be fixed. Your app didn’t get IP addresses or a hostname setup when you deployed.

Did you deploy your app without a [[services]] block at first? I’m trying to figure out where this went wrong.

4

no, I used a normal config with services. the only thing it didnt have was http healthtests

5

Hi there. I’m having a similar issue; just deployed an App using pretty much the default generated configuration file. Deployment succeeded, App is healthy and I can access it using the fly proxy command, but using the subdomain results in a 404.

The fly.toml

# fly.toml file generated for summer-star-7550 on 2023-01-11T10:43:19Z

app = "summer-star-7550"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "requests"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
6

If you haven’t checked, then see if the app process is listening for plaintext http connections on 8080 (as defined with services.internal_port) and responding approp?

7

How do you mean? If I understood properly, app is the default process. That’s the service exposing the services.internal_port. If I can reach the service with the fly proxy command, doesn’t that mean it’s listening properly?
And apologies if I misunderstood your question.

8

Maybe your web server is misconfigured, specifically the virtual hosts? Requests via fly proxy will have the Host header set to localhost:8080, while request via the .fly.dev subdomain will have the Host header set to that domain (there will also be additional headers automatically added by Fly).

Another potential difference is IPv4/IPv6: fly proxy will probably use IPv6, while the subdomain might get accessed over IPv4.

The web server configuration would be helpful for debugging, ideally as a minimal reproducible example…

1 Like
9

Hi Tom, thanks for replying.

The server is binding the listener to http://*:8080 and http://*:9001 which should be ANY interface on ports 8080 and 9001. That should work for both IPv4/v6.

I’ve updated the config to bind to http://0.0.0.0:8080 and http://0.0.0.0:9001 to do only IPv4. Results are still the same, the service isn’t reachable. The internal health check still works.

Deploying this to a kubernetes cluster and exposing it as a service with a TLS domain works, so this feels like something to do with fly’s platform.

There isn’t any additional configuration for the web server really…

ENV PORT=8080
ENV ASPNETCORE_URLS="http://0.0.0.0:${PORT};http://0.0.0.0:9001"
10

Try deploying the same code to another app and see if it works? I recall a similar issue reported once before.

Btw, listening on all interfaces is the right thing to do (source of many a debug headaches if you ever find yourself accidentally using any of Fly’s IPv6-only features).

11

Try deploying the same code to another app and see if it works? I recall a similar issue reported once before.

Yeah, I’ve done this. I’ve even created a separate project with the bare minimum… basically just a server that can reply something static text content to :8080/ and :9001/healthz endpoints. I’ve published the code here. There really isn’t much to it.

I’ve then deployed a new app with this, but I’m still getting the same results, unfortunately.

Btw, listening on all interfaces is the right thing to do

Completely agree with that.

2 Likes
12

Ok… some updates… this has nothing to do with the platform, it’s just a code issue, so apologies to everyone involved.

I’ve removed the 9001 port from the equation, left only 8080. Deployed and success, got a 200 as expected.

I then removed the 0.0.0.0 from the binding and left * to listen to any interface, since I think that should be the correct setting. Deployed and success, still got a 200 as expected.

The issue here is when listening to two ports, even though that’s not exactly the issue; particularly the way the endpoints are being mapped. To sum up, here’s what I’m doing (not real code)

Map( "/" ).RequireHost( "*:8080" );
Map( "/healthz" ).RequireHost( "*:9001" );

The problem is that RequireHost that isn’t doing what I expected it to do. It works locally (I have no idea why it worked with Kubernetes) but it won’t work on fly.io (or [insert-competition-name]).

So… closing this issue, as it’s a code issue. Apologies and a thank you to all that replied.

1 Like
13

I am curious how the health checks were passing? The app wasn’t health checking 8080?

Is the competition any better at customer service? :wink:

Don’t worry about it… btw, time to mark your own reply as “solution” then… (:

14

This looks like the problem – the pattern will be matched against the Host header, not the local listening port; changing the pattern to "*:80" fixes the 404. See RequireHost and reverse proxy · Issue #42252 · dotnet/aspnetcore · GitHub; the later comments also describe how to match against the local port (it looks complicated).

Maybe Kubernetes is rewriting the Host header…

(Also worth mentioning the X-Forwarded-* headers, but I don’t think they are relevant here.)

The health check requests have the internal port in the Host header (unlike external requests).

I was about to post the same thing, but this is actually a hijacked thread :‍)

2 Likes
15

This looks like the problem – the pattern will be matched against the Host header, not the local listening port; changing the pattern to "*:80" fixes the 404. See RequireHost and reverse proxy · Issue #42252 · dotnet/aspnetcore · GitHub; the later comments also describe how to match against the local port (it looks complicated).

This is exactly it! It’s a complete mess, to be honest. But yes, changing to *:80 does work. Thank you for the link btw, it’s a great find!

later comments also describe how to match against the local port

Seems like this is the correct path, though it feels like this should be handled by ASPNET.

I’ve added another endpoint without the restrictions to dump the headers and this is what I get (removed some irrelevant values)

{
	"Host": ["fragrant-cherry-1703.fly.dev"],
	"Via": ["1.1 fly.io"],
	"x-request-start": ["t=1673534881646873"],
	"fly-forwarded-proto": ["https"],
	"fly-forwarded-ssl": ["on"],
	"x-forwarded-ssl": ["on"],
	"fly-forwarded-port": ["443"],
	"x-forwarded-port": ["443"],
	"X-Original-Proto": ["http"]
}

btw, time to mark your own reply as “solution” then… (:

As @tom93 mentioned it, it’s a hijacked thread, turns out it’s a different issue from the original comment.

Thanks again to all the comments and feedback.

1 Like
16

Ok, just sharing my success on this. Managed to get everything to work as expected.

Thanks to @tom93 and the links he provided, I was able to resolve the issue, which again, was a code issue, not a fly issue.

I’ve replaced the .RequireHost with a filter that verifies the local port instead. Not posting real code, since that’s out of scope, but this was what I had before

Map( "/" ).RequireHost( "*:8080" );
Map( "/healthz" ).RequireHost( "*:9001" );

Which didn’t work unless we replaced the first one with *:80. That would mean different code on different environments (and/or providers). So… instead, I’m doing this instead

Map( "/" ).RequireLocalPort( 8080 );
Map( "/healthz" ).RequireLocalPort( 9001 );

This will work on any environment.

Btw…

Is the competition any better at customer service?

If none of you work for fly, I believe you have your answer :smile: but then again, I’m not (yet) a paying customer. Just browsing :slight_smile:

Thank you all.

1 Like