Tigris lookups for files that don't exist fail about 60% of the time when sent from inside Fly

Tigris lookups for files that don’t exist fail about 60% of the time when sent from inside Fly

  1. Plain curl reproduces it on the Fly machine, so it isn’t our server code. I sent 10 signed HEAD requests for random keys: 4 returned 404 (the normal “not found”) and 6 returned 408.
  2. The error body names it. It’s Tigris’s RequestCanceled: Request is canceled, with a Tigris request ID.
  3. Files that do exist come back fine (200). Only “not found” lookups break.
  4. It doesn’t happen from my machine. The same requests return a clean 404 every time from my machine. So it looks specific to the Fly region my app runs in (iad).

Tigris bucket dry-bird-5370, region iad, and some request IDs: 1792524228072895086 and 1792524582750057914 (from my app), 1792524670444110257 and 1792524683584487806 (my curl tests).

Any idea what is going on or how to fix it?

Hi, I tried a simple repro (write some objects, then try to read them), I got a 404 every time from Fly machines in iad and yyz.

I think it’d be reasonable to report this to Tigris instead, and if you’re able to provide a minimal reproducer (if you do, try posting it here and I can run it too - assume the AWS_ACCESS_KEY_ID AWS_ENDPOINT_URL_S3 AWS_REGION AWS_SECRET_ACCESS_KEY BUCKET_NAME variables are set) it would be quite helpful.