[Outage?] "Connection reset by peer" when using non-default ports

1

I’m getting errors (connection reset by peer) on the Nomad platform when I try to use external ports other than 80/443. I’ve reproduced the error with the “hello world” app.

The error doesn’t happen on the Machines platform (see the very end of this message).

Steps (based on Hands-on with Fly.io · Fly Docs):

$ mkdir app && cd app

$ flyctl launch --image flyio/hellofly:latest
Creating app in <...>/app
Using image flyio/hellofly:latest
? Choose an app name (leave blank to generate one): *<blank>*
automatically selected personal organization: <redacted>
? Choose a region for deployment: *Chicago, Illinois (US) (ord)*
Created app dry-bush-2017 in organization personal
Admin URL: https://fly.io/apps/dry-bush-2017
Hostname: dry-bush-2017.fly.dev
Wrote config file fly.toml
? Would you like to set up a Postgresql database now? *No*
? Would you like to set up an Upstash Redis database now? *No*
? Would you like to deploy now? *No*
Your app is ready! Deploy with `flyctl deploy`

# get the app name
$ APP=$(sed -E -n 's/^app = "(.*)"$/\1/p' fly.toml)

# add external port 8080 (error also happens with e.g. port 5000)
$ cat >>fly.toml <<"EOF"
  [[services.ports]]
    handlers = ["http"]
    port = 8080
EOF

$ flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'flyio/hellofly:latest' remotely...
image found: img_z1nr0lpjz9v5q98w
==> Creating release
--> release v2 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment
Logs: https://fly.io/apps/dry-bush-2017/monitoring

 1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v0 deployed successfully

$ wget -O - http://$APP.fly.dev # succeeds (full output below)

$ wget -O - http://$APP.fly.dev:8080 # fails
--2022-12-21 13:05:56--  http://dry-bush-2017.fly.dev:8080/
Resolving dry-bush-2017.fly.dev (dry-bush-2017.fly.dev)... 66.241.124.43
Connecting to dry-bush-2017.fly.dev (dry-bush-2017.fly.dev)|66.241.124.43|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2022-12-21 13:05:58--  (try: 2)  http://dry-bush-2017.fly.dev:8080/
Connecting to dry-bush-2017.fly.dev (dry-bush-2017.fly.dev)|66.241.124.43|:8080... connected.
HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.
Retrying.

--2022-12-21 13:06:00--  (try: 3)  http://dry-bush-2017.fly.dev:8080/
Connecting to dry-bush-2017.fly.dev (dry-bush-2017.fly.dev)|66.241.124.43|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2022-12-21 13:06:03--  (try: 4)  http://dry-bush-2017.fly.dev:8080/
Connecting to dry-bush-2017.fly.dev (dry-bush-2017.fly.dev)|66.241.124.43|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

^C

Full output of wget -O - http://$APP.fly.dev (which succeeds):

--2022-12-21 12:54:04--  http://long-rain-6270.fly.dev/
Resolving long-rain-6270.fly.dev (long-rain-6270.fly.dev)... 66.241.125.103
Connecting to long-rain-6270.fly.dev (long-rain-6270.fly.dev)|66.241.125.103|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://long-rain-6270.fly.dev/ [following]
--2022-12-21 12:54:05--  https://long-rain-6270.fly.dev/
Connecting to long-rain-6270.fly.dev (long-rain-6270.fly.dev)|66.241.125.103|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 96 [text/html]
Saving to: 'STDOUT'
<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body>
<h1>Hello from Fly</h1>

</body>
</html>

2022-12-21 12:54:06 (55.4 MB/s) - written to stdout [96/96]

Diagnostic commands:

$ flyctl status
App
  Name     = dry-bush-2017          
  Owner    = personal               
  Version  = 0                      
  Status   = running                
  Hostname = dry-bush-2017.fly.dev  
  Platform = nomad                  

Deployment Status
  ID          = b87d9cf0-8b7d-a7b2-0e31-ed66bf407f60         
  Version     = v0                                           
  Status      = successful                                   
  Description = Deployment completed successfully            
  Instances   = 1 desired, 1 placed, 1 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS        CREATED   
55a84ec4        app     0       ord     run     running 1 total, 1 passing      0               3m30s ago

$ flyctl logs
2022-12-21T13:03:03Z runner[55a84ec4] ord [info]Starting instance
2022-12-21T13:03:03Z runner[55a84ec4] ord [info]Configuring virtual machine
2022-12-21T13:03:03Z runner[55a84ec4] ord [info]Pulling container image
2022-12-21T13:03:04Z runner[55a84ec4] ord [info]Unpacking image
2022-12-21T13:03:04Z runner[55a84ec4] ord [info]Preparing kernel init
2022-12-21T13:03:04Z runner[55a84ec4] ord [info]Configuring firecracker
2022-12-21T13:03:04Z runner[55a84ec4] ord [info]Starting virtual machine
2022-12-21T13:03:04Z app[55a84ec4] ord [info]Starting init (commit: f447594)...
2022-12-21T13:03:04Z app[55a84ec4] ord [info]Preparing to run: `/goapp/app` as root
2022-12-21T13:03:04Z app[55a84ec4] ord [info]2022/12/21 13:03:04 listening on [fdaa:0:36f5:a7b:9adb:55a8:4ec4:2]:22 (DNS: [fdaa::3]:53)
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
2022-12-21T13:03:04Z app[55a84ec4] ord [info] - using env:      export GIN_MODE=release
2022-12-21T13:03:04Z app[55a84ec4] ord [info] - using code:     gin.SetMode(gin.ReleaseMode)
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] Loaded HTML Templates (2):
2022-12-21T13:03:04Z app[55a84ec4] ord [info]   -
2022-12-21T13:03:04Z app[55a84ec4] ord [info]   - hellofly.tmpl
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] GET    /                         --> main.handleIndex (3 handlers)
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] GET    /:name                    --> main.handleIndex (3 handlers)
2022-12-21T13:03:04Z app[55a84ec4] ord [info][GIN-debug] Listening and serving HTTP on :8080
2022-12-21T13:03:38Z app[55a84ec4] ord [info][GIN] 2022/12/21 - 13:03:38 | 200 |    2.493988ms |   <redacted-ip> | GET      /
^C

$ cat fly.toml
# fly.toml file generated for dry-bush-2017 on 2022-12-21T12:56:57Z

app = "dry-bush-2017"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  image = "flyio/hellofly:latest"

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
  [[services.ports]]
    handlers = ["http"]
    port = 8080

Steps for creating an app using the Machines platform:

$ mkdir app && cd app
$ flyctl apps create --machines
$ APP=<name> # replace <name> with the app name
$ flyctl config save -a $APP
$ cat >>fly.toml <<"EOF"
  [[services.ports]]
    handlers = ["http"]
    port = 8080
EOF
$ flyctl ips allocate-v4
$ flyctl deploy --image flyio/hellofly:latest
? This feature is highly experimental and may produce unexpected results. Proceed? *Yes*
$ wget -O - http://$APP.fly.dev:8080 # succeeds
2

By default, we allocate a shared IP address on Nomad platform, which only supports HTTP/S on 80/443. Running flyctl ips allocate-v4, as you did with Machines, will set you up with a dedicated IP which you can use custom ports with. (Do note we will start charging $2/mo for dedicated IPs in January.)

1 Like
3

Oh wow, that’s sad news (though not entirely surprising). Thanks for explaining; I didn’t see the announcement.

I now see that the page Public Network Services · Fly Docs has already been updated, but there are several other pages that I think should mention this:

  • App Configuration (fly.toml) · Fly Docs says “port : An integer representing the external port to listen on (ports 1-65535).” I think the text below should mention that by default an app’s IPv4 address is shared, so it only supports ports 80 and 443 (running HTTP and TLS+HTTP respectively); other ports and protocols require a dedicated IPv4/IPv6 address.

  • I feel like Troubleshooting your Deployment · Fly Docs should also mention the deal with shared IPv4 addresses (and how to check whether the app’s IPv4 address is shared or dedicated).

  • Running Fly.io Apps On UDP and TCP · Fly Docs should mention that the demo requires a dedicated IPv4 address. Bonus typo: In the line echo hello world | nc udp-echo-sample.fly.dev 5000, nc should be changed to nc -u (otherwise it uses TCP).

I note that people who want to try Fly.io without spending money won’t be able to create UDP services, because (if I understand it correctly) UDP can’t work on a shared IPv4 address, and is not supported over IPv6, so it requires a (paid) dedicated IPv4 address.