I can't establish an external connection to my brand new fly postgres app

What I’m trying to do

I’m trying to migrate my SaaS from Vercel to fly io. I want the whole infrastructure on flyio, starting from Postgres.

What’s working until now

  1. Created an account
  2. Installed the fly cli tool and logged in
  3. Successfully created an app (or a fly machine, idk)
  4. I chose the development instance from the list

The problem

I can’t stablish an external connection if my life depended on it.

When

13/08/2024

What I’ve tried

I followed the docs on how to stablish an external connection:

Most of the steps were not required as it was already configured from the beginning.

# fly config show --app diet-it
{
  "app": "diet-it",
  "primary_region": "gru",

  "env": {
    "FLY_SCALE_TO_ZERO": "1h",
    "PRIMARY_REGION": "gru"
  },

  "mounts": [
    {
      "source": "pg_data",
      "destination": "/data"
    }
  ],

  "services": [
    {
      "protocol": "tcp",
      "internal_port": 5432,
      "auto_start_machines": true,

      "ports": [
        {
          "port": 5432,

          "handlers": [
            "pg_tls"
          ]
        }
      ],

      "concurrency": {
        "type": "connections",
        "hard_limit": 1000,
        "soft_limit": 1000
      }
    },
    {
      "protocol": "tcp",
      "internal_port": 5433,
      "auto_start_machines": true,

      "ports": [
        {
          "port": 5433,

          "handlers": [
            "pg_tls"
          ]
        }
      ],

      "concurrency": {
        "type": "connections",
        "hard_limit": 1000,
        "soft_limit": 1000
      }
    }
  ],

  "checks": {
    "pg": {
      "port": 5500,
      "type": "http",
      "interval": "15s",
      "timeout": "10s",
      "path": "/flycheck/pg"
    },

    "role": {
      "port": 5500,
      "type": "http",
      "interval": "15s",
      "timeout": "10s",
      "path": "/flycheck/role"
    },

    "vm": {
      "port": 5500,
      "type": "http",
      "interval": "15s",
      "timeout": "10s",
      "path": "/flycheck/vm"
    }
  },

  "metrics": [
    {
      "port": 9187,
      "path": "/metrics"
    }
  ]
}

Here’s the output of some of the relevant commands:

# fly services list --app diet-it
  PROTOCOL        PORTS           HANDLERS        FORCE HTTPS     PROCESS GROUP   REGIONS MACHINES
  TCP             5432 => 5432    [PG_TLS]        False                           gru     1
  TCP             5433 => 5433    [PG_TLS]        False                           gru     1
# fly app list
  NAME    OWNER   STATUS          LATEST DEPLOY
  diet-it diet-it suspended

# fly machine restart d891974b429928
  Restarting machine d891974b429928
  Waiting for d891974b429928 to become healthy (started, 3/3)
  Machine d891974b429928 restarted successfully!

# fly app list
  NAME    OWNER   STATUS          LATEST DEPLOY
  diet-it diet-it deployed
# fly ip list --app diet-it
  VERSION IP                      TYPE    REGION  CREATED AT
  v6      fdaa:9:d488:0:1::3      private global  23h38m ago

My tests

I always restarted the machine and made sure they were running before testing them

# pg_isready --host=diet-it.fly.dev --username=postgres --port=5432
diet-it.fly.dev:5432 - no response

# pg_isready --host=diet-it.fly.dev --username=postgres --port=5433
diet-it.fly.dev:5433 - no response

# pg_isready --host=dietit.fly.dev --username=postgres --port=5432
diet-it.fly.dev:5432 - no response

# pg_isready --host=dietit.fly.dev --username=postgres --port=5433
diet-it.fly.dev:5433 - no response

# pg_isready --host=[fdaa:9:d488:a7b:136:3b13:34ec:2] --username=postgres --port=5432
[fdaa:9:d488:a7b:136:3b13:34ec:2]:5432 - no response

# pg_isready --host=fdaa:9:d488:0:1::3 --username=postgres --port=5432
fdaa:9:d488:0:1::3:5432 - no response

Question

What am I don’t wrong and how can I fix it? I need to be able to connect to this database with other fly instances, but also with external connections not whithing the fly io infrastructure

This output from fly ip list shows you didn’t add a public IP address as indicated in the external access document. You only have one address and it’s private. Your output should look something like this:

$ fly ips list
VERSION	IP                  	TYPE              	REGION	CREATED AT        
v6     	2ax9:8280:1::5b:46d5	public (dedicated)	global	Jul 24 2023 23:28	
v6     	fdxa:2:7d1e:0:1::68 	private           	global	Feb 7 2024 20:54 

try fly ips allocate-v6 --app diet-it if an IPv6 address is sufficient for your needs, or fly ips allocate-v4 --app diet-it if you need an IPv4 for some reason. Note that IPv4 dedicated addresses have an extra cost while IPv6 ones do not :slight_smile:

  • Daniel

Thank you for your answer roadmr!
I did not had a public ipv6. But now I do and I’m still unable to connect to it.
Do you think my server is configured correctly now and this is an error on my connection test process?

# fly status --app diet-it
Updates available:
Machine "d891974b429928" flyio/postgres-flex:16.3 (v0.0.58) -> flyio/postgres-flex:16.3 (v0.0.59)
Run `flyctl image update` to migrate to the latest image version.

ID              STATE   ROLE    REGION  CHECKS                  IMAGE                                   CREATED                 UPDATED
d891974b429928  started primary gru     3 total, 3 passing      flyio/postgres-flex:16.3 (v0.0.58)      2024-08-14T13:48:35Z    2024-08-15T14:53:00Z


# fly ip list --app diet-it
VERSION IP                      TYPE                    REGION  CREATED AT
v6      2a09:8280:1::40:ef21:0  public (dedicated)      global  14m24s ago
v6      fdaa:9:d488:0:1::3      private                 global  Aug 14 2024 13:48


# pg_isready --host=2a09:8280:1::40:ef21:0 --username=postgres --port=5432
2a09:8280:1::40:ef21:0:5432 - no response

Sorry for the noob questions, I’m really bad at deploying stuff :sweat_smile:

HI, the app is now gone so I wasn’t able to check it :slight_smile:
Let me know if you recreate it and I can have another look.

Everything you have looks correct, at this point the only thing to check is whether you do have ipv6 connectivity - if not, you’ll probably need an IPv4 address instead.

  • Daniel

Sorry, I had just decided to delete it and restart from scratch.
I really did not manage to make it work. Now I chose to never scale it to zero.

# fly machine status --app master-db
? Select a machine: 48ed461c771948 weathered-fire-9258 (started, region gru, role 'primary')
Machine ID: 48ed461c771948
Instance ID: 01J5C4XTFFJQ9T9SEDHDKRRD08
State: started
HostStatus: ok

VM
  ID            = 48ed461c771948
  Instance ID   = 01J5C4XTFFJQ9T9SEDHDKRRD08
  State         = started
  Image         = flyio/postgres-flex:16.3 (v0.0.60)
  Name          = weathered-fire-9258
  Private IP    = fdaa:9:d488:a7b:145:658e:bd23:2
  Region        = gru
  Process Group =
  CPU Kind      = shared
  vCPUs         = 1
  Memory        = 256
  Created       = 2024-08-15T23:10:52Z
  Updated       = 2024-08-15T23:11:04Z
  Entrypoint    =
  Command       =
  Volume        = vol_40k082zy3omn0k2v

PG
  Role = primary

Checks [3/3]
NAME    STATUS  LAST UPDATED    OUTPUT
pg      passing 1h49m ago       [✓] connections: 7 used, 3 reserved, 300 max (4.12ms)
                                [✓] cluster-locks: No active locks detected (25.53µs)
                                [✓] disk-capacity: 11.0% - readonly mode will be enabled at 90.0% (9.35µs)
vm      passing 1h49m ago       [✓] checkDisk: 877.51 MB (89.0%) free space on /data/ (104.45µs)
                                [✓] checkLoad: load averages: 0.00 0.00 0.00 (85.41µs)
                                [✓] memory: system spent 0s of the last 60s waiting on memory (37.01µs)
                                [✓] cpu: system spent 630ms of the last 60s waiting on cpu (29.67µs)
                                [✓] io: system spent 306ms of the last 60s waiting on io (19.18µs)
role    passing 1h49m ago       primary

Event Logs
STATE   EVENT   SOURCE  TIMESTAMP                       INFO
started start   flyd    2024-08-15T20:11:04.085-03:00
created launch  user    2024-08-15T20:10:52.468-03:00
# fly ips list --app master-db
VERSION IP                      TYPE                    REGION  CREATED AT
v6      2a09:8280:1::41:2e87:0  public (dedicated)      global  1h47m ago
v6      fdaa:9:d488:0:1::3      private                 global  1h50m ago

What’s the best way to test the db using an external connection string?
All psql, telnet and dbeaver (which uses a JDBC connection string) says its unreacheable

# In example
# telnet 2a09:8280:1::41:2e87:0 5432
Trying 2a09:8280:1::41:2e87:0...
telnet: Unable to connect to remote host: Network is unreachable

Other things I’ve tried

Enabling ipv6

As seen in this topic

I checked if I had ipv6 disabled. It was enabled on wsl2 but not on windows, I enabled it on windows but it’s still not connecting

Scaling up the machine

I though the machine might be starving from resources, I made 1024 instead of 256

Testing postgres

When I connect using the fly cli proxy tool, I can successfully connect to postgres:

#fly proxy 5432:5432 -a master-db
Proxying local port 5432 to remote [master-db.internal]:5432

# telnet 127.0.0.1 5432
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

Postgres is running correctly

Just checking, did you also follow our process for connecting to Postgres externally?

Specifically, it requires pulling down the fly.toml for your Postgres app, modifying the service to be reachable externally, and redeploying. If you haven’t done that, your service would be reachable via proxy but not externally via IP, which is consistent with the behavior you’re seeing here.

If you have done that, could you please post the modified fly.toml you redeployed? Thanks!

I did not modify my fly.toml file, but the default fly.toml file for fly postgres already includes the required properties mentioned on the documentation guide:

# fly.toml app configuration file generated for master-db on 2024-08-16T10:29:37-03:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'master-db'
primary_region = 'gru'

[env]
  PRIMARY_REGION = 'gru'

[[mounts]]
  source = 'pg_data'
  destination = '/data'

[[services]]
  protocol = 'tcp'
  internal_port = 5432
  auto_start_machines = false

  [[services.ports]]
    port = 5432
    handlers = ['pg_tls']

  [services.concurrency]
    type = 'connections'
    hard_limit = 1000
    soft_limit = 1000

[[services]]
  protocol = 'tcp'
  internal_port = 5433
  auto_start_machines = false

  [[services.ports]]
    port = 5433
    handlers = ['pg_tls']

  [services.concurrency]
    type = 'connections'
    hard_limit = 1000
    soft_limit = 1000

[checks]
  [checks.pg]
    port = 5500
    type = 'http'
    interval = '15s'
    timeout = '10s'
    path = '/flycheck/pg'

  [checks.role]
    port = 5500
    type = 'http'
    interval = '15s'
    timeout = '10s'
    path = '/flycheck/role'

  [checks.vm]
    port = 5500
    type = 'http'
    interval = '15s'
    timeout = '10s'
    path = '/flycheck/vm'

[[metrics]]
  port = 9187
  path = '/metrics'

# Properties required by the documentation
[[services]]
  internal_port = 5432 # Postgres instance
  protocol = "tcp"

[[services.ports]]
  handlers = ["pg_tls"]
  port = 5432

# Section fo the default fly.toml file equivalente to the required properties
[[services]]
  protocol = 'tcp'
  internal_port = 5433
  auto_start_machines = false

  [[services.ports]]
    port = 5433
    handlers = ['pg_tls']

By the docs, I don’t think auto_start_machines or the order of the properties matter, but please correct me if I’m wrong.

Hmm, OK, next thought: does it work over an IPV4 address? I know you’re working with and enabled IPV6, I’m just trying to rule out other possibilities.

Also, what do you see when running grep -i listen /data/postgresql/*.conf from an SSH session in the postgres app (I.e. running fly ssh console -a <postgres-app>)? It’s a bit of a long shot, but maybe your configs aren’t listening on any remote interfaces.

These are the results of those commands:

# fly ssh console -a master-db
Connecting to fdaa:9:d488:a7b:145:658e:bd23:2... complete

# grep -i listen /data/postgresql/*.conf
/data/postgresql/postgresql.conf:#listen_addresses = 'localhost'                # what IP address(es) to listen on;
/data/postgresql/postgresql.internal.conf:listen_addresses = '*'

I don’t know grep so I don’t know what it means

Actually, if you want to help me run a quick experiment:

I think /data/postgresql/postgresql.internal.conf sets listen_addresses = '*' but that’s for the internal postgres listening on 5433. You might try setting that in /data/postgresql/postgresql.conf which I think is for the main instance–looks like it defaults to ‘localhost’ which would be why you can’t connect externally. Note that this will make your Postgres publically reachable, which I think is what you want but be sure you’re OK with that before restarting the app. If you can use .internal hostnames or a Wireguard endpoint that’d be better.

I’m new to running Postgres in any setup more complex than single-instance, so I don’t know what the best and most secure path forward here is, but I’m sure it doesn’t involve docs that don’t seem to work. If this fixes it, I’ll figure out something so it doesn’t happen again–either tweaking the image or updating the docs. Apologies for the frustration!

I’m not sure if I was meant to to it directly on the system or using a configuration file. So this is how I did it:

# vim /data/postgresql/postgresql.conf
############# Changed and saved from
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

#listen_addresses = 'localhost'         # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
############## to
#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '*'                     # what IP address(es) to listen on;
                                           # comma-separated list of addresses;

Now, I get the following when I run grep:

# grep -i listen /data/postgresql/*.conf
/data/postgresql/postgresql.conf:listen_addresses = '*' # what IP address(es) to listen on;
/data/postgresql/postgresql.internal.conf:listen_addresses = '*'

I tried to connect using telnet and it did not work.

# telnet 2a09:8280:1::41:2e87:0 5432
Trying 2a09:8280:1::41:2e87:0...
telnet: Unable to connect to remote host: Network is unreachable

Then I restarted the machine and tried again

# fly machine restart 48ed461c771948
Restarting machine 48ed461c771948
Waiting for 48ed461c771948 to become healthy (started, 3/3)
Machine 48ed461c771948 restarted successfully!

# telnet 2a09:8280:1::41:2e87:0 5432
Trying 2a09:8280:1::41:2e87:0...
telnet: Unable to connect to remote host: Network is unreachable

# The ip is correct
# fly ips list --app master-db
VERSION IP                      TYPE                    REGION  CREATED AT
v6      2a09:8280:1::41:2e87:0  public (dedicated)      global  16h18m ago
v6      fdaa:9:d488:0:1::3      private                 global  16h22m ago

It’s still unreachable. Let me know if I did anything wrong.

Ok, no problems. If this works it will be worth the amount of effort. We want to migrate one of our companies infrastructure to fly.io in order to be compliant with local internet laws.

“network is unreachable” suggests your host/network doesn’t have native IPv6 connectivity. You can try for example pinging Yahoo’s ipv6 address (yeah, I know, Yahoo, yuck, but I’ve found they consistently respond to pings and are good for IP diagnostics if nothing else :rofl: ). You’ll get this if your host/network doesn’t speak IPv6:

$ ping6 2001:4998:24:120d::1:1
ping6: connect: Network is unreachable

On a host with native IPv6 support and a compliant network:

$ ping6 2001:4998:24:120d::1:1
PING 2001:4998:24:120d::1:1(2001:4998:24:120d::1:1) 56 data bytes
64 bytes from 2001:4998:24:120d::1:1: icmp_seq=1 ttl=53 time=40.4 ms
64 bytes from 2001:4998:24:120d::1:1: icmp_seq=2 ttl=53 time=40.4 ms

If IPv6 doesn’t work then you’ll definitely need an IPv4 address (fly ips allocate-v4), though note that as I mentioned, these cost $2/month since they are quite scarce.

  • Daniel

You are correct, @roadmr , it was not connectiong due to ipv6 not working (even tho I had enabled the ipv6 config on windows).
I just created a dedicated ipv4 and now I have a different error. That’s some progress. I will try to figure out the rest.
Thank you very much!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.