Hasura and Multi-Region Postgres

I am evaluating fly.io and before I dig in wanted to make sure there aren’t any major issues I should be aware of when using fly.io with Hasura and multi-region Postgres. So to my questions:

  1. When using Hasura hosted on fly.io and the multi-region Postgres, are there any issues with the replay logic? I am not sure how it impacts this.
  2. In this configuration, would there be a Hasura instance in each region we decide to use?
  3. Are there issues with the Hasura Event triggers when running in multi-regions?

I hope that makes sense. I like what I am seeing with fly.io but I also love Hasura so trying to figure out if I can maximize the usage of both.

In principle there shouldn’t be a problem. But when I looked at Hasura, it didn’t seem to make it easy to add middleware like other frameworks. Here’s an Apollo Server experiment that catches write errors and replays requests. You could also, in theory, try to detect mutations and replay those. If you want to dig into this with Hasura, we’d love to know how it turns out!

There would have to be a Hasura instance in each region along with a Postgres replica.

Event triggers, in theory, should work fine since you’d only see events from writes, which only happen in your primary region. My guess is that you’d need to run the Hasura console in the primary region for anything to work as expected.

I think my biggest concern is the Event triggers. How Hasura actually works is it uses actual database triggers to put the event trigger information into a table, then it will trigger. So in theory if each replica is firing on those triggers, events could potentially be duplicated. Or is the replication of Postgres happening another way in fly.io?

Do you know how it’s actually getting notified on event triggers? If they’re using NOTIFY, that only fires from the primary Postgres instance.

It seems like Hasura Cloud can handle read replicas, so their notification system will likely work: Read replicas | Hasura GraphQL Docs

I am not sure with that. Also the Hasura Cloud is their hosted solution and they don’t put that feature in their docker version. I was thinking about using Hasura on fly.io directly to get it as close to the data. Which may mean I just use Hasura Cloud and put the Postgres on fly.io.

It looks like there’s some read replica support available in open source too: Read-Replicas-on-Hasura-OSS (2020-12-21)

I don’t think Hasura Cloud runs in multiple regions, so you wouldn’t get much use out of our multi region Postgres with it. We should see if they’ll work with us on this. Multi region Hasura would work amazingly well.

Yeah I think it would be awesome to combine multi-region Postgres with Hasura to provide a lot of speed and redundancy.

Their structure makes sense:

image

With the read-write instance being in the primary region. With local and subscriptions be in the local. I assume the events would be triggered in the primary only.

It would require a little work on the query side based on their caveats section. This doesn’t seem impossible and could provide a really nice solution and better than their Hasura Cloud

1 Like

Or maybe need a trick set env for region on fly.io =]]

@jmarbutt I have a solution for this.
in Hasura:

  • PG_DATABASE_URL: postgres://<user>:<password>@my-db.internal:5432/db?sslmode=disable
  • HASURA_GRAPHQL_READ_REPLICA_URLS: postgres://<user>:<password>@top1.nearest.of.my-db.internal:5433/db?sslmode=disable

See: High Availability & Global Replication · Fly Docs

top<number>.nearest.of.<appname>.internal : top number closest app instances