Flycast for postgres

Hey folks! We have some new fresh produce for you full of some of our favorite things: flycast and postgres!

Tldr

You can switch from DNS based resolution for your postgres instances to flycast based connections by running flyctl pg add_flycast against your machines based postgres cluster. Future flyctl pg attach invocations will use the flycast address instead of the .internal domain. You can manually find your new flycast ip by running flyctl ips list as well.

What is flycast again?

Good question! A flycast ip is a private ipv6 that goes through the proxy instead of directly to instances in your app. Meaning you get all the load balancing, rate-limiting, and other sweet proxy features you know and love.

Why?

  1. It removes DNS from the critical path of PG connections
  2. We get get the safety/service level checks/smart routing from the proxy for free
  3. It opens the gateway for things like scale to zero postgres (flycast will already start up stopped postgres machines)

Going forward

This will become the default for new postgres clusters sometime in the future

3 Likes

Hi there,

Is there any chance that this release/deploy could be causing issues for existing apps using flycast?

We have a service that was just deployed ~15 mins ago that uses flycast for ingress/egress, and currently can’t connect to other apps on the network (hangs, then connection reset)

Rolling back to the previous version didn’t fix it, which is what’s leading me to believe it’s related to the infra.

We’re able to work around it by using the .internal DNS namespace, but this is causing degradation in other ways (occasionally trying to connect to processes that aren’t listening for http connections)

Could someone please look into this? Or direct me to a better forum to report?

1 Like

Seems to be an unrelated network problem that we are aware of. We’re looking into it.

1 Like

This should now be fixed. Can you confirm?

Only a few hosts (new) were affected: Fly.io Status - Flycast connectivity broken from certain (new) hosts

Confirmed that it’s fixed for our app, thanks!

Are post-mortems conducted/publicized for these incidents? This caused ~30 mins of downtime during key hours for us - not good. We were lucky that we eventually found a workaround with the .internal address, but until then we were totally hosed.

We don’t have a great system for public-postmortems yet but we frequently share details on the forum and emails, or when people ask!

We are in the process of adding lots of new regions/workers to keep up with demand, which is a good problem to have! In the process of provisioning a new bunch of hosts today some new deploys were ending up relying on those hosts before they were ready. Specifically not all our networking provisioning had happened yet, hence the broken flycast. We manually deployed the proxy on those hosts which fixed flycast routing.

@DAlperin What is possible today for scale to zero postgres?

You’d need to install your own Postgres version that knows how to exit when it’s idle. Our proxy will happily start Postgres up when a new connection comes in, but it won’t every shut down after that.

Thanks @kurt I’ll experiment with possible approaches.