New to Fly.io tbh… and I’ve been learning/playing around with CockroachDB. I’ve been testing it on GKE for the time being, but was really curious if it would be possible to run on Fly!
I know that Fly offers persistant disk space… right? Are these disks fast enough to run CockroachDB? I know GKE has different types of disks you can choose from…
Has anyone experimented with CockroachDB on Fly yet, and if so what was your experience?
Is there any way we could get an Official single node and HA example on Fly? I think a lot of people would be interested in deploying CockroachDB on Fly as well!
Turns out, it’s not all that hard to get going! Here’s an example multi region cockroachdb setup on Fly. It doesn’t use haproxy or anything, your app will need to connect directly to the nodes for it to work right.
You will need today’s release of flyctl for these instructions to work.
CockroachDB requires more than 50% of the nodes in the cluster to be alive for the cluster to operate. With the current example there is no way to ensure more than 50% of the nodes are kept alive during a deploy / upgrade.
In the example no health checks are setup to ensure that unhealthy nodes are replaced as a node can become unhealthy without the CockroachDB process exiting.
I think this was a quick first pass a how it’s possible to deploy CockroachDB on Fly.
The service is not exposed through the public internet, only through our private encrypted network.
However, the caveats in that list aren’t too good:
Any user, even root , can log in without providing a password.
Yikes.
Our deployment process uses a rolling strategy whenever volumes are attached. Meaning we’ll upgrade 1-by-1 each node, waiting for them to come back up as healthy.
That’s a good point, we can add a script check for cockroach’s unhealthiness. A check failure means a restart.
It’s true! This is just an example (although PRs are acceptable).
CockroachDB is working on making --insecureless of a bazooka. mTLS is dumb and not necessary on Fly. Once they get that cleaned up we should be able to add auth and keep the example simple.
I would disagree that mTLS is dumb, it’s a straightforward way to verify that two nodes are indeed part of the same cluster and have the correct authority. Its especially useful when you expose the database to the Internet so that third-party services / vendors can connect with the database (CockroachDB shares the SQL and Node comms on the same port).
The referenced issue wouldn’t necessarily remove mTLS, simply make it easier to work with, more transparent and less cumbersome.
I might get some time over the next week where I can upgrade the example to a full secure cluster in a way that’s similar to our deployment on AWS Fargate.
One issue / limitation that I’ve encountered with Fargate (which as far as I know fly.io has too) is the ability to add secret files to the image. In the case of CockroachDB, we base64 the certificates and keys before storing them into the secrets, then the image entry script will decode them and store them on disk and pass those paths to the CockroachDB process.
Ok you’re right. mTLS is generally dumb but in Cockroach land it’s useful.
--sql-addr and --accept-sql-without-tls will be helpful at least. Fingers crossed the remove the mtls requirement between cockroach nodes soon and just let you do a pre shared RPC key.
