An Oban Odyssey

Fresh Produce
1

Recently, I was trying to use our internal Oban dashboard, and getting frustrated by various performance issues. Looking into how to address these led me to the bottom of a rabbit hole of Oban improvements, and I thought y’all might find some of it interesting!

The Original Sin

We’d originally setup the Oban Web dashboard to use longpoll as the transport.

Why we were using longpoll

Honestly, I’m a bit hazy on the details here, but my rough sense is as follows:

  • We have a single Elixir app that serves the Fly.io dashboard, and all of our Oban code (job definitions, etc.) lives in the repo for this app.
  • We have process_group=app Machines for serving the dashboard, and process_group=oban_worker Machines for processing job queue work.
  • The app Machines don’t run Oban, because we don’t want dashboard-serving Machines also trying to do job queue work on the side, making our public dashboard slow. The oban_worker Machines weren’t publicly accessible, because there was no need for them to be - they’re dedicated to processing job queue work.
  • So: oban_worker Machines are the only ones running Oban, but they’re inaccessible from the internet. To get our Oban dashboard working, we needed some way for requests to be made to an app Machine and then end up on an oban_worker Machine.
  • We opted to use ReverseProxyPlug with .internal DNS to reverse proxy Oban dash requests to oban_worker.process.OUR_APP_NAME.internal:4000
  • If I had to guess, this was probably just much simpler at the time than implementing a websocket relay, but this forced us to use longpoll transport.

Some of us like to have lots of Oban dash browser tabs open at the same time. The Oban dash makes one request per second (by default) in order to update the UI in real time. If you keep lots of tabs open, that can add up to a fair number of requests per second, each of which results in a database query.

At some point, someone had noticed that this had the potential to accidentally cause a self-DoS, and they had chosen to implement a rate-limit as a coarse backstop. Fair enough - as a quick fix, it works. longpoll means every query the page makes is an HTTP request, so it’s a reasonably effective way to rate-limit database queries.

The Inciting Incident

Unfortunately, in the wrong circumstances, this can make the dashboard pretty impossible to use. The rate-limit was implemented as local-only, so you have to exceed the limit on a specific Machine in order to actually be rate-limited.

Fine, on our production app, where there are lots of Machines. That makes it hard to ever exceed the rate limit on any one Machine. Not so fine on our staging apps, where one Machine is the entire app.

This was a recipe for frustration if you were trying to test something Oban-related in a staging environment.

Agents make it really easy to look into this stuff, so I fired one up in a Sprite and set it about looking into why our Oban dash was so slow.

The first thing it suggested was switching the transport to websocket, which, yeah, why not? It won’t fix any of the core issues (in fact, it subverts the rate limit). But once they are fixed, it’ll make the Oban dash smoother.

The First Facepalm

Then it noticed that we were running PromEx.Plugins.Oban. And I mean we were running it. On every Machine in our primary region (which is where our database is).

This plugin ran the following query every 5 seconds on every primary-region node:

SELECT queue, state, count(id) FROM oban_jobs GROUP BY queue, state

It amounted to about 2.8 full scans per second of a ~5.4M-row table, in our primary database. The Oban connection pools in our app accounted for about one third of our database’s load. :grimacing:

We changed the period to be 60 seconds, and configured the plugin to run only on Oban worker Machines.

The Second, Also Pretty Big, Facepalm

After deploying these changes, I still found that I would get timeouts when trying to view a specific job’s page in the dash. This was particularly baffling because I would’ve assumed query the data for a single job would be fairly snappy.

The reason for this turned out to be due to the query that Oban uses to load a worker’s history, so that it can show the “Recent Executions” graph. The fragment of interest is:

WHERE worker = $1
  AND id <= $2
  AND state IN ('executing','completed','cancelled','discarded')
ORDER BY id DESC LIMIT 60

pg_indexes on the database showed nothing on oban_jobs that starts with worker. So Postgres has two options when performing the query above:

  • walk the primary key backwards from the job’s id, checking each row’s worker until it finds 60 matches
  • scan the table.

Both of these can end up being glacially slow. Particularly for a worker that executes infrequently, because of that LIMIT clause.

The solution here is simple: add a (worker, id) index to the table. I’m actually not sure why this isn’t something that’s built into Oban.

Discovering An Iceberg

The other huge finding in this second round of investigation was that we had a 460 GB (!) index on a ~31 GB table. What was going on here?

The index was created as:

CREATE INDEX oban_jobs_args_index ON public.oban_jobs USING gin (args);

Oban Web only ever searches args using the @> Postgres operator, so it’s possible to use the index below instead:

CREATE INDEX oban_jobs_args_index ON public.oban_jobs USING gin (args jsonb_path_ops);

This index requires far fewer bytes, but it will only support @> queries; it won’t work for key-existence queries (? operator) or searching for empty structures, e.g. a column with contents {"a": {}} would not appear in the index. Luckily, neither of these restrictions matters for the queries that Oban Web makes.

This change to the index reduces the overall size, but it still doesn’t explain why the existing index got so ridiculously big. It turns out that, even though we’d included the Reindexer plugin in our configuration, there were two issues:

  • This index takes a while to build, because of the cardinality of our job queue. “A while” as in ~20 minutes, which greatly exceeds the 15 second default.
  • Oban was using database credentials that did not own/could not assume a role that owned the oban_jobs table. Reindexing requires being the owner of the table to which the index applies, so the reindexing had never even been attempted.

The index had been valid the whole time, but much more bloated than necessary, which had made it very slow. Better than a full scan, but still basically unusable.

We made the Oban user the owner of oban_jobs. We increased the timeout configured for the Reindexer plugin. We migrated the index to use jsonb_path_ops.

All told, the index we have now is ~2 GB, meaning we cut ~458 GB of bloat. And the Oban dash is snappier than it has ever been!

edit: Oh, one thing I forgot to mention is that the Reindexer can be interrupted by deploys. I have made a note to look into wrapping that plugin, so that it runs the reindexing in a Machine that won’t be touched by deploys.

1 Like