Why is cross-region routing of database queries slower than http?

Questions / Help
,
#1

This is more of an engineering and architecture question than a specific technical inquiry. From the Fly.io article (Globally Distributed Postgres · Fly),

It is much, much faster to ship the whole HTTP request where it needs to be than it is move the database away from an app instance and route database queries directly.

My question is about why routing database queries is slower than http.

A bit of context, I am trying to deploy a CloudFlare worker-backed serverless SPA (think Sveltekit, Remix etc., the app logic is tightly coupled with the SSR layer and not as a separate backend API microservice). The database layer would either be Supabase/Nhost or a CloudFlare VPN tunnel into a Fly.io postgres instance. My question is, would it be faster to just skip the whole serverless-backend architecture entirely and just put everything on a single Fly.io node with CloudFlare acting as a dumb CDN? I would prefer to go with serverless because of the lower DevOps overhead but Fly.io’s blog made me question the wisdom of the distributed serverless + centralised database architecture. The app is a CRUD app, balanced between read/write heaviness.

I know there are serverless-specific databases like Cockroachdb/Spanner/PlanetScale but they are out of my budget for my use cases.

Most single region database solutions like Supabase expose their database connection string over https/TLS, how much latency would this add in practice for a serverless app? Is a CloudFlare VPN tunnel into Postgres a better solution? A CloudFlare VPN + postgres would require two containers: the postgres DB + another container for the postgres pool and the VPN. There will be three network jumps, user <-> CloudFlare <-> pool+VPN container <-> database container.

#2

@kurt

#3

It’s not slower per-query. Most applications do a bundle of queries, though. A typical POST request might have a sequence like read-read-write-read-write-read. It’s quicker to handle that whole HTTP request close to the database.

If you run that sequence far away from the database, you might get round trips like read (1ms) - read (1ms) - write (60ms) - read (60ms) - write(60ms) - read(60ms). This is because you need to read from the primary after the first write.

Most single region setups like Supabase should provide geo read replicas. :slight_smile: We’re getting there! I would not use a TCP database protocol over long distances. If your database is stuck in one region, you should probably just run your app in one region.

2 Likes
#4

Is a CloudFlare VPN tunnel into Postgres a better solution?

Because you’re on Workers, is using Durable Objects an option (if you’re data isn’t in need of Relational or Postgres-specific semantics)?

In our case, for where Durable Objects isn’t an option, we are looking to experiment with litestream.io read-replicas on fly.io and/or multi-region redis cache on fly.io where approp.

Like you, I prefer zero-devops, too… and litestream.io’s read-replica has me all hyped up. PlanetScale is enticing nevertheless with 100bn (!) row reads in their free tier.

Edit: Btw, apart from using cloudflared + postgREST, Cloudflare (like Deno Deploy) has announced private beta support for outgoing TCP connections to databases like Postgres, so you could use cloudflared in the interim, safe in the knowledge that a better solution is right around the corner.

1 Like