Fly app with multiple external ports only working for port 443

Hi!

I’m currently trying to deploy an app that needs to expose multiple ports with the TLS handler. After some debugging I discovered that both of my exposed services are working, but only if they were exposed on port 443.

To be sure there wasn’t some something specific with my configuration I also tried deploying the fly-app-with-multiple-external-ports example application which has the same issue. It is currently deployed and running at https://multiport.fly.dev and it should also be available at https://multiport.fly.dev:9091 which doesn’t work for me:

$ curl https://multiport.fly.dev
<h1>Hello From the first app!</h1>
$ curl https://multiport.fly.dev:9091
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to multiport.fly.dev:9091

Is anyone able to confirm if this seems to be a problem with just me? Both my original app and the example app give the same error in flyctl doctor which could also potentially be related.

$ flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for multiport:
Checking that app has ip addresses allocated... PASSED
Ip address 66.241.124.52 has unexpected type 'shared_v4'. Please file a bug with this message at https://github.com/superfly/flyctl/issues/new?assignees=&labels=bug&template=flyctl-bug-report.md&title=Checking AAAA record for multiport.fly.dev... Nope
        These IPs are missing from the multiport.fly.dev. AAAA record: 2a09:8280:1::a:b471
        This likely means we had an operational issue when we tried to create the record.
        Post in https://community.fly.io/ or send us an email if you have a support plan, and we'll get this fixed


Build checks for multiport:
Checking docker context size (this may take little bit)... PASSED (30 kB)
Checking for .dockerignore... PASSED

This is probably due to our recent release of shared IPs. These only support tls + http over port 443. The API should error if you’re trying to deploy an app with additional services.

The quick fix is to run fly ips allocate-v4 to get a dedicated IP. Note that we’ll start billing for dedicated IPs in January ($2/mo each) so that may not be what you want.

1 Like

This is probably due to our recent release of shared IPs.

Awesome thanks! That would explain it. Looks like I just had some really unlucky timing running into this right in between the change being deployed and the docs being updated.

The readme in the example project has been updated in Update readme to allocate ipv4 · fly-apps/fly-app-with-multiple-internal-ports@89e1bef · GitHub

And the main reference docs have been updated in update services doc for shared ipv4 (#482) · superfly/docs@794579e · GitHub

The API should error if you’re trying to deploy an app with additional services.

I wasn’t getting an error before when I was deploying the example app. I just tested it again with a fresh destroy/deploy on the latest flyctl version but I’m still not getting any errors.

The quick fix is to run fly ips allocate-v4 to get a dedicated IP.

Is there a way to define this inside of fly.toml instead of needing to run the command manually after the first deploy? I think Allocate public IPv4/v6 to new apps if a service is added · Issue #1196 · superfly/flyctl · GitHub might be what I’m describing.

Hi @crazybus, sorry you had to find the updated docs yourself. I had a draft open here to make the note and forgot.

I didn’t even know about the multiple-ports demo app, so thanks for the report. :smile:

I ran into the same thing. I was in the process of replicating a set of apps from one organization to another. I also have my own TLS handler on a custom port that deals with self-signed certs, and it was working fine in my older organization. But when I deployed the same app to a new organization and attempted to make requests, it failed with errors like this:

curl -k -vvv https://abc:6081/

*   Trying a.b.c.d:6081...
* Connected to abc (a.b.c.d) port 6081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to abc:6081 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to abc:6081 

I also use two ports, but my setup is a little different: one port (6080) with an HTTP handler that I let Fly manage a certificate and terminate TLS on, and a second port (6081) with the in-app TLS handling and self-signed certificates.

Cheers