Sneak peak: global lock service

1

As part of our Postgres rollout, we needed global locking for leader election.

We launched a shared Consul cluster to make this work, and are using the ACL system to provide app level access to it. To enable it, add this to your fly.toml:

[experimental]
enable_consul=true

This will inject a FLY_CONSUL_URL environment variable in your app, like this:

https://:<token>@consul-na.fly-shared.net/app-name-23sxdf

There’s no username on that URL (note the leading :), but the token is in the password field, and the path section (/app-name-23sxdf) is the prefix that token has write access to.

The ACL policies for apps look like this:

key_prefix "app-name-23sxdf/" {
  policy = "write"
}

node_prefix "app-name-23sxdf/" {
  policy = "write"
}

session_prefix "app-name-23sxdf/" {
  policy = "write"
}

This lets you register with the catalog, write to the KV store, and create lock sessions for your registered nodes. You can’t run a consul agent with it, but you can make consul API calls.

Here’s how you’d register an agent:

curl -H "X-Consul-Token: <token>" \
    "https://consul-na.fly-shared.net/v1/catalog/register" \
    -XPOST \
    -d '{"Name":"app-name-23sxdf/kurt","Address":"fdaa:0:1:a7b:ab6:0:a:2"}'

We’re using it with Stolon. We forked docker/libkv to add support for Consul auth + ACLs, then had to fork Stolon to use the new libkv. Fun times.

If you are trying to launch clusters that need elections on Fly, this might be a useful service for you. Feel free to poke at it and see how it goes.

6 Likes
2

Will all edge locations be part of the same datacentre as defined by consul? Curious to know how the KV store will work; e.g. single global store, regional stores, replicated stores, etc.

3

Right now we just run this as one datacenter. We’ll probably run one per continent, possibly one per region in Singapore/Sydney/Tokyo/HKG. I don’t think federated sessions ever shipped in Consul.