It looks like we may have an old instance that was never shut down, which is still processing jobs from our job queue.
We have a Phoenix app that uses the Oban job queue. We recently noticed that, in the oban_jobs table in Postgres, the “attempted_at” timestamp is sometimes almost 2 minutes earlier than the “scheduled_at” timestamp, which indicated clock drift. We logged into the 2 instances currently in our cluster + the database server and the system times appear to be in sync. Then we checked the “attempted_by” column and found that the the rows with “attempted_by” timestamps in the past always have the same instance ID: production@fdaa:0:309a:[…]:55b5:2. These started on Oct. 28 and the drift has kept increasing.
So, is it possible to check to see if there’s some instance out there that wasn’t shut down properly?