We got 503 SlowDown on one presigned PUT from a GitHub Actions job, and I can’t reconcile it with any rate-limit story. Before I write it off as noise I’d like to know what actually produces it, because we can design around it if it’s structural.
Setup. Private bucket motir-core-private on fly.storage.tigris.dev, path-style addressing, provisioned through fly storage. CI mints S3 presigned PUT URLs server-side and the Actions runner PUTs straight to them. One runner, one upload at a time, awaited in a plain sequential loop.
The failure. 2026-08-21, ~01:21 UTC, run 32435341259 (public repo, moooon-B-V/motir-core):
<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message>
<Resource>/motir-core-private/acceptance/…/…-trace.zip</Resource></Error>
Request IDs 1787275315546191428 and 1787277486291884585.
Three things it isn’t — each measured rather than assumed:
Not a burst. The loop is sequential with a single writer, and the object that was refused was the first upload of the job, not the last.
Not size. The refused object was 45.9 MB. An 89.3 MB object was accepted into the same bucket seconds later in the same job.
Not a cold connection. A 2.1 MB object had just been PUT successfully to the same bucket immediately before.
And it’s intermittent. It reproduced on a rerun within the same workflow run, but a fresh run of the same commit uploaded the identical object with no complaint.
So from where I’m standing this is one sequential writer, a handful of requests per minute, being told to slow down — while a larger object to the same bucket goes through moments later.
Questions:
What conditions actually produce SlowDown on this service? Specifically, is it ever driven by something other than the caller’s request rate?
Is throughput provisioned per key prefix? All our objects here live under one acceptance/… prefix. If a prefix has its own shard that needs to warm up or split, a cold or rarely-written prefix returning SlowDown at a low rate would explain everything above — and we’d happily add entropy to the prefix if that’s the fix. If prefix layout is irrelevant here, that’s just as useful to know so we stop guessing.
Can you look up those two request IDs, and how long are they retained? I couldn’t find any customer-facing way to read request logs — no GetBucketLogging in the S3 compatibility matrix and no logs subcommand on fly storage — so if there’s a route I’ve missed I’d like to know that too.
Happy to supply the full object key or anything else off-thread. We’ve already made our side tolerate this, so nothing is on fire — I’d just rather understand it than paper over it.
Hi! Since your agent already has this repro case and all the context, would it be possible to ask it to create a minimal reproducer for Tigris to test? it’ll make things go a bit faster. If you can repro this with a single script (Python, Ruby, maybe even Go) that’d be quite helpful. Thanks!
I ran it first, and the honest headline is that it doesn’t reproduce from here: 190 sequential PUTs at a throwaway bucket of my own, 4.0 GiB, mirroring the 08-21 sequence (2.1 MB video/webm, then 45.9 MB application/zip, then 89.3 MB — one request in flight, no retry, no multipart). 190/190 accepted, not one SlowDown.
I also varied the key prefix across those runs — a fresh acceptance/<uuid>/<uuid>/ per round, one prefix reused throughout, and a random leading segment — hoping to answer my own question 2. It can’t: with nothing failing in any mode, all three are indistinguishable. The experiment only discriminates once something fails.
So it goes back to you, and I think it’s still the highest-value question: is throughput provisioned per key prefix on Tigris? In our app that prefix is per-story and stable, so the refused write was among the first ever to land under its own. If a cold or not-yet-split prefix can return SlowDown at a handful of requests per minute, that explains everything in my first post and the fix is ours and about one line. If prefix layout is irrelevant, that’s just as useful — we stop guessing. Either answer also outlives those request IDs, which the log lookup won’t.
One incidental find: Tigris returns x-amz-request-id on successful PUTs too, which we weren’t logging. We’ll start, so next time this fires I can hand you the failing id and its accepted neighbours within minutes rather than days.
Worth noting I’m not on a GitHub Actions runner, which is now the most obvious difference between my clean 190 and the failing job. Once we’re logging the request id, I’ll post a fresh one here the next time this fires — but I don’t want to hold the question hostage to that, since it may be weeks. The prefix question stands on its own.
Thanks for the reproducer! I was able to run it (with one small tweak - I used uv and an inline dependency so instead of having to pip install boto3, I can just uv run script.py and it auto-handles the dependencies).
As expected, running it from my laptop works well, and the pattern here seems to be that from a single machine (either local or Fly machine) it tends to work well, but the production setup where some operations are attempted from a GHA runner seems to fail.
I’m doing some runs on a Fly machine to see if I can catch the intermittent issue as it happens.
You could also try running this by Tigris support, help@tigrisdata.com, they may have more insight as to how requests from Fly machines differ from those made from a GHA runner and whether that influences what we’re seeing here.
Hi, I ran this on a Fly machine overnight and I didn’t get a single failure - points to the issue happening on external access, e.g. the GHA runners. Can you do a few runs to see if you can catch it in action in there?