I’ve attempted a few more things to resolve the issue:
I’ve tried changing region from fra to ams which didn’t yield any different results.
Default laravel application:
To make sure there was nothing in my codebase that was causing issues so I created a new Laravel application, using composer create-project laravel/laravel example-app. Once installed I proceeded to run flyctl launch and deployed right after.
This also caused the same error, with serversideup/php:7.4-fpm-nginx instead of 8.1.
Remote builder [REDACTED] ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 0.0s (0/1)
[+] Building 1.5s (6/6) FINISHED
=> [internal] load remote build context 0.0s
=> copy /context / 0.1s
=> resolve image config for docker.io/docker/dockerfile:experimental 0.8s
=> CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5 0.0s
=> [internal] load metadata for docker.io/library/node:14 0.3s
=> ERROR [internal] load metadata for docker.io/serversideup/php:7.4-fpm-nginx 0.4s
> [internal] load metadata for docker.io/serversideup/php:7.4-fpm-nginx:
Error error building: failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = unexpected status code [manifests 7.4-fpm-nginx]: 500 Internal Server Error
… and it worked. Deployed (in my case to lhr), loaded, all good. Strange!
May be worth fly destroy name-of-your-remote-builder to force a new one to be created on the next deploy, just in case that old one had any network issues or was out of space? Not sure what else would differ. I’m using the latest Fly CLI (fly version update).
As mentioned in the original post I’ve already attempted destroying the builder. And when I attempted a different region I made sure to also destroy the builder such that the new builder would be in the new region properly. Sadly, no dice.
I want to make it clear it wasn’t docker hub this time! Our registry mirror had a configuration issue that prevented disk cache from getting pruned correctly, and a few regions filled up and began erroring. Why docker bombs completely on a 500 from a known mirror is beyond me, but we fixed the issue and haven’t seen any issues for about 90 minutes now.