Early look at Elixir packages for globally distributed deployments with Postgres DBs

Here is an early release of two libraries to help globally distributed Phoenix applications using Postgres DB with read-replicas.

  • fly_rpc - Library for making RPC calls to nodes in other Fly.io regions.
  • fly_postgres - Library for working with local read-replica postgres databases and performing writes through RPC calls to other nodes in the primary Fly.io region.

The short notes:

  • Prereq: Your Phoenix app nodes are clustered together
  • Prereq: You have primary/replica Postgres DBs setup
  • Use fly_postgres in your app (Follow the README installation steps)
  • DB read operations like MyApp.Repo.all(User) are executed locally on the replica
  • DB write operations like MyApp.insert(changeset) are proxied to one of your apps running in the primary region. The call to insert will block until it completes and the database replication that includes your change is received on the calling node.

This means you can write “normal” code that performs writes on the primary, then blocks and waits for the async replication to complete before continuing. Your app doesn’t have to be completely re-designed or reworked for dealing with primary and replica databases. :smile:

More complex operations that do many inserts or updates will become slow while it pauses to wait for each insert or update to be replicated locally. For these operations, you can do a simple refactor of your code to make an explicit RPC call to the primary. It essentially says, “run this complex operation on the primary, close to the database”.

As an example, this version of a LiveView event is totally valid when you are close to the database. It iterates through a loop and performs many inserts.

  def handle_event("do_hard_work", %{"list_of_stuff" => stuff_ids} = _params, socket) do
    # create list of things from a set of IDs. May be a very large list and many inserts
    Enum.each(stuff_ids, fn item_id ->
      {:ok, _created} = ImportantStuff.create_stuff_from_item(item_id)
    end)

    {:noreply, assign(socket, :stuff_list, ImportantStuff.all()}
  end

While it could be completely re-written to be more efficient for batch operations, etc. We can make a small change to let it work well in our system.

  def handle_event("do_hard_work", %{"list_of_stuff" => stuff_ids} = _params, socket) do
    Fly.Postgres.rpc_and_wait(__MODULE__, :create_list_of_stuff, [stuff_ids])

    {:noreply, assign(socket, :stuff_list, ImportantStuff.all()}
  end

  def create_list_of_stuff(stuff_ids) do
    # create list of things from a set of IDs. May be a very large list and many inserts
    Enum.each(stuff_ids, fn item_id ->
      {:ok, _created} = ImportantStuff.create_stuff_from_item(item_id)
    end)
  end

This refactors the work part out into a separate public function. We can then RPC to the primary and execute the function there! We still need to pass the arguments. The RPC waits for the execution to finish and the data replication to sync. Finally, we continue on as normal.

Using this approach, your Phoenix LiveView can be physically close to your users giving really snappy and responsive behavior. Then “writes” are a little slower as they are performed in the primary region. This works well for read-heavy applications, which most web applications tend to be.

This is still early but seems to be working well for me so far! I’d love to get your feedback.

12 Likes

Tagging @aswinmohanme @nathanwillson @joshua as you might be interested in this.

2 Likes

Hey @brainlid, I did visit the fly_rpc project a few days ago before you extracted PG functionalities to it’s own app. Great work!

I like that your approach differs from the Ecto primary and replica guide and makes it much easier to integrate with a dynamic cluster. MyApp.Repo working as a proxy and delegating to the primary for write operations is brillant.

Also I think it should be more lightweight, by not having to keep a bunch of PG connections open to the primary from all running instances.

Now I’m not really familiar with PG and replicas so I have a few questions:

  1. What kind of latency should I except when waiting for the replica to sync?
  2. Is waiting for the replica to synchronise writes something I need all the time? I’d imagine that getting the result from the primary would suffice in a lot of cases.
  3. Would it be possible to bind this feature to the :read_after_write field option? Or only activate it when necessary?

Fly.Postgres.rpc_and_wait/3 seems a nice addition for running more complex queries involving reads and writes. Basically a great use case for Ecto.Multi. But for some cases it might make things more difficult to resonate about. For example using Ecto.Multi.run/3 might be executed on a different node which is not what you want most of the time.

I’ve never used it but dynamic repositories could be handy for similar tasks. Basically dynamically starting a single connection on the primary, run a few queries and close the connection.

2 Likes

Thanks! I’m glad you like it! All the cool ideas came from @chrismccord. :smile: But I had a blast implementing things.

Replication is async. The latency depends on a number of factors. Like how far away is the primary from the replica? I’ve been doing my experiments going from lax (Los Angeles) to syd (Sydney Australia) where syd is the primary region. The lag I see varies but it’s typically less than 200ms. That lag includes built-in lag in the library because it’s currently polling the DB every 100ms to see where the replication log is.

When you start playing with it, I think you’ll be pleasantly surprised with the feeling of it.

So this is the first pass at solving this problem. You are right that sometimes you don’t need to wait for replication and can just take the result from the RPC call. That could be an option that we pass in to disable it. I think having it enabled by default makes the most sense for people because it “just works”.

I like the :read_after_write idea! The repo operations we’re doing are just wrapping the whole repo. It’s not doing anything at the schema level. I’m not sure what would happen there. That’s interesting!

The explicit calls for rpc_and_wait/3 are intended for making numerous queries/changes, explicit transactions and the use of Mutli. Because Elixir is functional, it’s easier to pass that work off to another node. Just thinking back over my usages of Ecto.Multi.run/3, I can’t think of any typical scenario where running on a different node would be a problem. Most of the time people aren’t doing things with state in GenServers or local ETS tables during a Multi. It’s possible, but I don’t believe it’s common.

I looked into dynamic repositories as well. I think that’s a better solution for separate databases per customer. You’d still have to wait for replication, but yes, it helps keep down the number of constant open connections to the primary.

3 Likes

This looks awesome, I’ll definitely try this out! Do both libraries work correctly in local dev with a single node, or do you need to do a sort of am-I-in-fly check when using them? Also both their Github links from the hex pages (https://github.com/superfly/fly_rpc_elixir/fly_rpc and https://github.com/superfly/fly_rpc_elixir/fly_postgres) are 404s for me.

Great. Will give it a try soon :+1:

I think Ecto default behaviour with writes aka. Ecto.Repo.insert/2, Ecto.Repo.update/2, etc. is fire-and-forget. The :returning option is set to false so the client does not except the database to change anything on it own (except with :read_after_write or :returning enabled).

{:ok, post} = MyRepo.insert(%Post{title: "this is unique"})

Here post is not read from the DB. Instead it state is deduced from the given changeset or struct.

The example given in Ecto.Multi.run/3 is writting a file to the filesystem. I personally have quite a few cases where I use run/3 in order to do similar things. Of course it can be refactored in order to keep Ecto.Multi transactions pure (without side-effects) and apply side-effects later on.

Thanks! I’ll fix the links.

Yes, this works with a single node and for local development. With local development, you just need to add some ENV variables like this:

export PRIMARY_REGION=xyz
export FLY_REGION=xyz

EDIT: I fixed the project links. They share the same github repo and you can find it here:

1 Like

Good point on the Ecto.Multi.run/3 file system example. Yes, things like that would best be refactored or at least the developer should be aware that it might need to happen on the primary.

this looks amazing, I’ve tried to take it for a spin this evening, but unable to get out of the door, as I heavily use Ash for my datalayer, Ash expects the Repo to implement many things, i.e: in_transaction? etc etc, it seems as though Fly.Postgres is just stubbing out a few core methods, would it be possible to have Fly.Postgres extend the entire repo instead of only stubbing out a few function calls?

Yes, it needs to support more. :slight_smile: Interested to hear how it interacts with the Ash framework too.

The fly_postgres library wraps all of the Repo module’s function calls. I think macro’s bring in other functions like query/4. What are some functions that the Ash framework needs here?

one that I can remember without installing it again is in_transaction?. I’ve got a deadline to meet this week, but will try and get a more comprehensive list soon :slight_smile:

1 Like

Just letting people know that fly_postgres v0.1.11 updated the polling logic to be smarter so it only polls for updates when a process requests to be notified.

Let me know how it works for you!

2 Likes