Was stuck at waiting for depot builder when running on SEA last week, but let me check if any region still has this issue.
1 Like
Encountered an odd error, is it because I tried to deploy multiple apps that shared cache at once?
1 Like
yep, seeing a cache copy error persistently across multiple app deploys Merge branch 'manawiki:main' into main ¡ manasites/zzz@b693cf6 ¡ GitHub
The error is reproduceable by rerunning the job, and there doesnât seem to be any way to nuke the cache in userland, so weâll have to revert using Depot for now. Hope this gets fixed soon!
hey! it looks like your builder volume got full. Normally, Depot will clear old layers in the build cache as it gets more full, but running the three deploys at once seems to have seems to have filled the cache completely.
Iâm planning on adding a command to clear the build cache in Depot, but in the mean time, you can set --depot-scope=app to use a separate build cache for each app of yours, provided youâre using the v0.2.118 of flyctl or higher.
Also, I believe that we cleared that specific volume, so future deploys should work as is.
On an unrelated note - what app are you using here to monitor network traffic? (It looks nice!)
Thanks @billy, will give it a whirl tomorrow!
The way our wiki apps are built, theyâre all forks off the same trunk with the same package.json and dockerfile in every app, so the org-shared cache layer is why weâre so excited about adopting Depot.
Would you recommend staggering deploys on Depot then, since it sounds like this issue could crop up again.
Surge http://nssurge.com/
But I donât think itâs worth to purchase it outside China (since the most value of it is to bypass the GFW in China)
If you like the interface, I bet you may want to see Proxyman https://proxyman.io/
1 Like
For now, yea staggering deploys would probably make the most sense. We should hopefully have options to increase build cache size in the future, but I donât wanna make promises now.
Oh and thatâs a really cool use case!
Still encountering build errors on Depot:
If you have the time to, reach out to me directly on github or Discord @xhomu and maybe we can narrow down its causes?
1 Like
hey! iâll send you a DM on Discord to try to figure it out 
Thanks for the help, billy! I think the ENOSPC: no space left on device, write bugâs fixed now.
Everything looks green, I will move all our deploys to Depot today, and report back if anything else crops up!
1 Like
Ops, jobs begin to fail again this morning: add meta canonicalURL ¡ manawiki/mana@31d23df ¡ GitHub
Error: failed to fetch an image or build from source: error building: failed to solve: failed to prepare sha256:2a776155e9afff7cacd607d8c3df199185f2279006273fb939d5514ea8f6639a as k0wlybhzu99ljfj6ckgx4c7iy: failed to create prepare snapshot dir: mkdir /var/lib/buildkit/runc-stargz/snapshots/snapshotter/snapshots/new-2305413024/fs: no space left on device
FYI After consulting Depot devs, the out-of-space bugs have largely disappeared after we updated dockerfile COPY --link to COPY.
This looks like a bug in buildkit as it is not always accounting for the size of --link layers correctly. Sometimes buildkit thinks those layers are zero bytes in size despite being multiple gigabytes. When GC runs, it thinks the disk has far more space than it really does!
1 Like