Early look: PostgreSQL on Fly. We want your opinions.

1

We’re launching PostgreSQL on Fly soon. We’d like some feedback, and volunteers to test it out. Here’s what we’re planning:

Launch high availability PostgreSQL clusters

These will be two instance PostgreSQL clusters (using stolon, if you’re a DB nerd). The smallest option will be a cluster on 2x shared-cpu-1x VMs with 256MB of RAM and 10GB of disk space. Total cost, about $7.50 per month. You can scale these like any Fly app, all the way up to 8 CPUs, 64GB RAM, 500GB disks.

You will have admin rights to these clusters, so you can add as many PostgreSQL DBs and users as you want, share them between apps, etc.

Notably, we will not have free databases because multitenant, free PostgresSQL clusters are an atrocity.

We also will not offer single instance PostgreSQL databases. This will save both us and you headaches.

DB management

You’ll notice that prices are only the cost of the underlying VMs. This is on purpose, so you can buy cheap DB clusters and use them for side projects, development, etc. This is not a managed postgres service.

It’s best to think of it as an automated postgres cluster with built in health checks, metrics and alerts. You can wire up PagerDuty and get paged when something does go awry with the cluster (like you fill up the disk). And you can hook up Grafana to see how it’s doing. You can even fork and modify the Postgres app we install for you.

Later on, you’ll be able to pay $$ to have us (or community members) respond to alerts and provide other management services. For now, automated postgres is enough for many kinds of apps.

Comments?

What we’re doing is a little different than most self service PostgreSQL products. Does this sound useful to you all? And do any of you want to try it out in the next few weeks?

18 Likes
2

:heart_eyes: how exciting! Looking forward to giving this a shot at Draftbit :smile:

4 Likes
3

Fantastic timing - I’m working on migrating the open source Core distribution of Prefect (the latest hotness in dataflow automation platforms) to run on Fly and integrate along with some other goodies. Swapping the Hasura postgres node that is backing the workflow scheduler with something that can really scale is a game-changer for enabling colocation of data ingestion and processing on the edge.

IMO this coupled with the templated Minio, NATS, and the very new KeyDB launch templates and I’m seeing a very compelling story for a zero-ops turnkey Unified@Edge Data Infrastructure.

3 Likes
4

People are going to think we bribed you to say all that because it’s 100% on the nose. :smiley:

1 Like
5

Now this is cool!

What we’re doing is a little different than most self service PostgreSQL products.

How is this different from something like Google Cloud SQL? Here you own it of course, but it seems like they’re both Postgres with management made easier.

Secondly, why not use a distributed SQL database like Cockroach? It seems like it would fit really well with the competitive advantages of fly.

1 Like
6

Great timing. I resorted to deploying an app on Heroku yesterday because I didn’t want to manage Postgres myself on Fly.

Where do we sign up? :grinning_face_with_smiling_eyes:

1 Like
7

This feels like maybe the last shoe to drop. I’m convinced that a lot of people who are getting sucked into serverless basically just want something less black-box-y than Heroku, cheaper, and easier to push out to the edge. For me at least, I want to be able to put up a simple Rails/Redis/PostgreSQL stack and know that if traffic starts shooting up I’m not suddenly caught flat-footed. Up until now, Heroku was the obvious choice (and maybe Render lately as a #2 choice). But honestly, I’d much rather run containers on the edge and just spin up more containers if/when load demands it. I’m super excited about this and can’t wait to try it out!

6 Likes
8

I second this question. I saw in your HN thread you mentioned Cockroach a few times. Is the familiarity for users at play in choosing a db? (Though of course, having Postgres at first doesn’t mean Cockroach will never come)

9

What we’re doing is similar to Google Cloud SQL (and AWS RDS) because you’re getting Postgres, but delivered differently. On Fly, postgres will just be another app. It’ll be the first kind of app to use new health check + alert features, and it’ll be something you launch from our configuration (or fork and change if you want). flyctl will work like you’re used to. In fact, you’ll be able to add geo replicas with flyctl just like you add regions for your other apps.

Which brings me to Cockroach. Cockroach is great, but it’s not quite as drop-in of a Postgres alternative as we wanted. Our goal is to get a DB option that people can use for their normal full stack frameworks, figure out how to do it across regions (hello geo replicas), and then see what’s next.

Also, Timescale was a big factor. You’ll be able to run Timescale in your Fly Postgres. Having time series in a ready to go DB is pretty swanky.

7 Likes
10

How do you all want to be notified of alerts? We’re thinking:

  • PagerDuty
  • Slack
  • Webhook

That seems like 90% of what people will need, and then webhooks to make anything else possible with a little work.

11

As long as there’s a webhook, it’ll be fine :slight_smile:

12

If you’re still looking for beta testers, I’m up for it!

13

Any thoughts on how upgrades are going to work? Since users could still manage the instances themselves are they expected to self-serve, or will there likely be hand-holding here as well?

14

Minor release upgrades will just be a quick flyctl deploy -i <new-version> away, and will be graceful. We haven’t decided how to do major upgrades yet, but the plan is to automate that as well. Major releases will probably require a preboot option for deploys apps can run some logic for an upgrade process.

1 Like
15

Hi,

An example of HA using Hasura + PostreSQL would be really great!

Instant graphql API with a great Database.

:slight_smile:

1 Like
16

Another +1 for me if you’re still looking for volunteers to try this out. Would love to get my hands on it!

17

Yes! Give us a day or two and we’ll post some instructions. :slight_smile: We have a show stopper bug we’re fixing before we get anyone else using it.

1 Like
18

Howdy folks. After you update flyctl to the latest version (0.0.166+) you’ll find some handy new commands to launch and manage a 2 node HA postgres cluster.

This is a preview, you probably don’t want to use this in production yet. :wink:

Launch

$ flyctl pg create

You’ll be prompted for name, organization, region, vm size, and volume size. After a few moments you’ll have a shiney new pg cluster and be presented with some help text like this:

Creating postgres cluster pg-demo in organization personal
Postgres cluster pg-demo created
  Username:    postgres
  Password:    f02313012f9ab47297651bd12b294623b780ee
  Hostname:    pg-demo.internal
  Proxy Port:  5432
  Leader Port: 5433
Save your credentials in a secure place, you won't be able to see them again!

Connect to postgres
Any app within the personal organization can connect to postgres using the above credentials and the hostname "pg-demo.internal."
For example: postgres://postgres:f02313012f9ab47297651bd12b294623b780ee@pg-demo.internal:5432

See the postgres docs for more information on next steps, managing postgres, connecting from outside fly:  https://fly.io/docs/reference/postgres/

It’ll take about 1-2 minutes for the cluster to be ready for connections.

Connecting

Once the cluster is up, you can connect to it however you want. Fly apps within the same organization or a wireguard peer from outside fly can use the app-name.internal hostname to find an instance over the private network.

For example, from my mac with wireguard, I can use psql to connect as a superuser:

psql postgres://postgres:f02313012f9ab47297651bd12b294623b780ee@pg-demo.internal:5432

Using the superuser credentials, you can create databases, users, and whatever else you need to configure for your apps.

We also have a shortcut for attaching postgres to another fly app:

flyctl postgres attach -a another-app --postgres-app pg-demo

This will create a user for the app, optionally create a database, grant permissions, and set a DATABASE_URL secret.

Here’s a few other helpful commands:

# list users
flyctl postgres users list pg-demo

# list databases
flyctl postgres databases list pg-demo

# detach an app
flyctl postgres detach -a another-app --postgres-app pg-demo

Monitoring

You can keep an eye on your postgres app just like any other fly app since these are normal fly apps, just with extra functionality sprinked on top…

flyctl status shows which VMs are replicas and leader.

$ flyctl status -a pg-demo
Instances
ID       VERSION REGION DESIRED STATUS            HEALTH CHECKS      RESTARTS CREATED
c66364b4 206     ord    run     running (replica) 3 total, 3 passing 0        7m31s ago
dba8a5cf 206     ord    run     running (leader)  3 total, 3 passing 0        8m50s ago

Other commands like flyctl status instance <vm>, flyctl checks list, flyctl logs etc all work as you’d expect.

Source code for the postgres fly app is available here: GitHub - fly-apps/postgres-ha: Postgres + Stolon for HA clusters.. We’ll share more technical bits on how it works soon along with documentation soon.

If you run into any issues or have feedback, let us know with a reply here. Or if it’s working flawlessly, you can let us know that too :smiley:

4 Likes
19

Spun up a test Postgres and it works great!

Any ideas what the backup story will look like once this is out of preview (or even still in preview)?
My ideal would be the ability to backup to an S3 compatible storage on a schedule that is customisable, but has good defaults. For a basic backup, saving to a fly volume would be okay too.

Does stolon support anything by default that would make your lives easier?

1 Like
20

Backup off-site would be nice. Once a day full backup + streaming updates to allow PITR. Would be nice if you can “clone” a new postgres cluster from a backup timestamp. Or, for staging, fork an existing database by using the last full backup and then streaming the last sub one day.

That said, would be nice if you can support Redis clusters too.

Everything you do with flyctl works as well as through the GraphQL API I guess?

How many IOPS will postgres get? Are there ways to e.g choose SSD as storage?

2 Likes