How to setup and use PGBouncer with Fly Postgres

Will check, this is in ord? Could you share the app ID? I can try a trace.

Yes, ORD - better-cart-pg-bouncer

Found the error, thanks, will fix it and get back to you in a bit.

Gotcha, just curious, what is the issue?

Also, a warning I also spotted in bouncer logs:

WARNING C-0x7f60b075c330: (nodb)/(nouser)@[fdaa:0:190:a7b:bea:0:a:202]:64664 unsupported startup parameter: options=-c extra_float_digits=3

Mean anything to you?

Also, no errors or warning in postgres logs.

I am also seeing this warning, but not sure if this is from a broken deployment:

WARNING DNS lookup failed: better-cart-postgres.internal: result=0

@sudhir.j - this def seems like a ipv6 internal dns issue.

The error is bug in the code that executes the deployment, we found it on our error tracker / monitoring tool.

WARNING DNS lookup failed: better-cart-postgres.internal: result=0 looks very relevant, will see if there’s anything in the image that prevents IPv6 from working.

1 Like

Based on a quick look at the error, it might be happening when doing a deploy where nothing changed. Just to unblock us, could you try adding a dummy environment variable and trying again?

Oh I should have mentioned, I am now able to deploy, and have done so a few times since getting the error.

I tried:

LISTEN_ADDR = "*"

No luck

The LISTEN_ADDR needs to be ::, that’s correct. This is more likely that the pgBouncer image’s DNS resolver can’t understand IPv6 addresses, trying to double check that. Think the configs are all correct on both apps (as of now).

Okay, the latest deployment is using “::”

I’m gonna try to either port the Dockerfile to something other than alpine or find a different pgBouncer container.

In the meanwhile, you could run fly ips private -a better-cart-postgres to get the actual IP address of the database, and put that in as DB_HOST on the pgBouncer app. That should bypass this problem, at least for now.

It seems using IPv6 for internal DNS has been causing non stop issues within the fly ecosystem, I must ask, if every app has a unique IPv4 address that is reserved and static, why is the internal dns pointing to the IPv6 address if this is pretty incompatible at the moment?

Just a random thought, maybe create multiple internal dns records, one for IPv6 and one for IPv4?

Just took a look and there is no IPv4 addresses assigned to the fly postgres app. I only see 2 IPv6 addresses.

IPv6 is definitely causing problems, but for better or worse the way forward is IPv6. My personal opinion is that if all software on the internet needs to be dragged kicking and screaming into the future, so be it :stuck_out_tongue: How to connect to a 3rd party image that listens for IPv4 addresses using Fly's private networking? - #16 by uncvrd

And yeah, setting the IPv6 address to the host should work. The problem is not with using IPv6 itself, it’s the resolution of these addresses from a abc.internal domain name.

2 Likes

Each VM gets a unique IPv6 address. The only IPv4 addresses we have are public, load balanced IPs that work through our load balancers. Postgres is only available on the internal private network, thus IPv6 only.

It’s unfortunate that Alpine has these DNS resolution bugs (assuming that’s what this is), but we can’t fix that. :confused:

1 Like

Ah, that makes sense!

I have switched to use the internal IPv6 address and we are now able to connect :tada:

1 Like

It blows my mind that Alpine has not fixed their images to support IPv6 DNS resolutions :exploding_head:

I think they have? Not sure. But just shifting the package to slim or something ought to handle it. I don’t really want to have Fly versions of every app on Github.

The direct IP setting should hold up reasonably well. The IP is tied to the volume, not the instance itself, so it’ll survive a restart, but not necessarily a change of the primary DB. So until we figure this alpine thing out you might have to switch it to the other address in case it becomes the leader.

This is something I was worried about and why I figured it was just a quick fix for now.

I would really not feel comfortable deploying this at the moment, it basically kills the idea of HA :confused:

Sort of. Since you’re connecting to port 5432 the HA setup that Fly uses will always send the connection over to the primary even if the IP is pointing at the replica, so you’re 99% ok and not too far from where we started.

But I can see how this would be worrying, so it’ll make sense to run the connections straight to Postgres for now until the bouncer is cleanly deployed. Either way you’re still in the single digit application connection limits, so I’m not sure the bouncer is an immediate necessity. I usually don’t add the bouncer until the first major connection pool related outage :stuck_out_tongue:

1 Like