litestream automatic failover strategies

1

Hi there. First, thank you to Ben for putting together litestream.

Let’s say you are serving a webapp: a golang server single binary that contains a SPA (frontend) and a backend that persists data to SQLite protected with litestream. The simplest solution is to spawn a virtual machine (maybe via fly.io) and run your golang tls server. Then you add a dns entry for your custom domain and you are in business. Great.

Now it could happen that your machine goes down. At that point you can manually spawn a new instance, replicate your db and restart the server. That may be ok in some scenarios but you may need to bring your app back online asap.

My question is, are you using litestream and have an automatic failover setup in place? If so, do you mind sharing your approach? Are there any resources out there about this topic you can share?

Thank you!
-drd

1 Like
2

Hi @drio. Litestream is purely for disaster recovery so it doesn’t have the ability to fail over. You can try looking at another project we are working on called LiteFS that will replicate databases across a cluster. If you use the Consul leasing with it then you can automatically failover when the primary node goes down.

The LiteFS Example Application docs page has an example of getting it up and running with Consul. Alternatively, if you’re just looking for live read replication and single, static primary, you can use static leasing and there’s an example of that in the Getting Started with LiteFS guide.

3 Likes
3

Hey Ben, fantastic work. I’m watching the development of LiteFS, and I’m excited to try it out. I created a small project to test it on Fly, but it looks like $FLY_CONSUL_URL is not set on machines (enable_consul is ignored?). Does it only work on apps?

1 Like
4

Fly Machines Workaround

For future readers: I was able to manually obtain a Consul URL by using the GraphQL API (see enablePostgresConsul, it returns a consulUrl).

2 Likes
5

That’s currently the best way to do it. We’re considering adding a CLI command for it as well in the future.

6

experimental.enable_consul ignored for Machines · Issue #1593 · superfly/flyctl · GitHub hopefully u/tvdfly gets to it sooner (:

1 Like