I wanted to get an idea of what typical latency can be when running writes to the main leader postgres database in let’s say ORD when the app server is running in lets say SYD.
I would only assume 200-300ms increase, but we are seeing speeds hitting 15-20 seconds, compared to when run in ORD they are 150ms.
Is it possible by having more app servers around the world and only the one read/write database in a single region worse than keeping the app servers close by and the database’s region?
We are not quite ready to start testing the read replicas within our app, so for the time being, we are looking for the best configuration possible with our postgres only running in a single region but receiving traffic from all over the world.
Okay, so I thought I would start sharing some of my findings and they do tend to make sense.
As I started to think about it more, it probably makes sense to have app servers running close to the postgres app and pushing the latency to the first request from the user, rather than potentially thousands of requests with latency to the databse.
So far it seems to make more sense having less app servers around the world until we have the read replica’s implemented.
We will still need somewhat of a better solution even for writes, as we were also seeing pretty large delays when performing writes as well when app servers were deployed in SYD and postgres in ORD.
Yes, the latency numbers are per-query, and most applications run many queries in sequence. There’s a Fly-Replay header that tells the Fly proxy to send a request somewhere else, but even this is useful mostly when there’s a read replica in SYD that can handle reads locally, and it can replay all writes on ORD.