Cannot deploy application

[rlivingston]

Hello,

I currently cannot deploy to a production application. I am seeing the below issue when pushing a docker image. I have not seen this issue before.

==> Pushing image to fly
The push refers to repository [registry.fly.io/REDACTED]
5b9e7d1828b7: Pushing [==================================================>]  54.71MB/54.71MB
e25e4334ac3d: Layer already exists
ee4ab924d0ec: Layer already exists
7dfab0bb3e34: Layer already exists
8d7be0d3cbdd: Layer already exists
cc2447e1835a: Layer already exists
Error: failed to fetch an image or build from source: error rendering push status stream: blob upload unknown

[kaz]

Sorry for the inconvenience. We are actively working on the issue.

Fly.io registry errors

Please follow Error: failed to fetch an image or build from source: error rendering push status stream: blob upload unknown for updates.

[yawnxyz]

I see the status has been updated to “fixed” but I’m still getting problems building Docker images:

Remote builder fly-builder-purple-wave-5076 ready
Error: failed to fetch an image or build from source: error fetching docker server info: Get "http://[fdaa:0:a5af:a7b:1b1:6d76:1f48:2]:2375/v1.41/info": context deadline exceeded
error Command failed with exit code 126.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[khuezy]

Have you tried:
FLY_REGISTRY_HOST=registry-iad.fly.io fly deploy

[yawnxyz]

@khuezy

FLY_REGISTRY_HOST=registry-iad.fly.io fly deploy

whoa, that worked like magic. It’s much much faster, and the deployment actually worked…!

[yawnxyz]

ok, that registry is now also not working…

[kaz]

That is odd. Did you still have the error you’ve posted above?

Error: failed to fetch an image or build from source: error fetching docker server info: Get “http://[fdaa:0:a5af:a7b:1b1:6d76:1f48:2]:2375/v1.41/info”: context deadline exceeded

[yawnxyz]

Hi Kaz!

It only happens occasionally (sometimes 20-30 minutes it just can’t seem to pull from the registry), which makes it hard to diagnose. I’m also running into occasional build issues for some code which builds fine on adapter-node locally (on Sveltekit) but just gets a Command failed with signal "SIGKILL" error on your Docker. I’m building on local but unfortunately it seems to hang on Fetching packages (not sure why it takes so long) so… still figuring it out!

[yawnxyz]

Alright I’ve tracked it down to a file that imports a 2.5MB JSON object w/ a long array (of names of bacteria!). If I shrink the array down, it builds properly. Leaving the entire JSON file up hangs the build process and eventually #0 127.1 error Command failed with signal "SIGKILL". — but without any error messages. I’ve tried to up the size ENV NODE_OPTIONS="--max-old-space-size=32768" to no avail haha. But a 2.5MB JSON file shouldn’t cause the headaches for the build process right??

Edit: here’s the file. Uncomment the large object and it breaks…!

/*

It’s probably a Docker memory error. The package mammoth (2mb; mammoth - npm) is also triggering a timeout / sigkill

[kaz]

Yeah. Just loading a 2.5MB JSON file shouldn’t kill a Docker daemon. I also installed mammoth in my sample Node app, but it worked fine.

Can you check your dashboard? There should be an app with “Free builder” label and you can check its logs from https://fly.io/apps/[name of this app]/monitoring.

There would be a lot of logs like below. They are not important, but others log messages may have some information to troubleshoot.

2023-10-27T23:09:51.677 app[9080e1e9c12208] sea [info] time="2023-10-27T23:09:51.677722482Z" level=debug msg="checking docker activity"
2023-10-27T23:09:51.677 app[9080e1e9c12208] sea [info] time="2023-10-27T23:09:51.677981325Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

[yawnxyz]

Hmmm yep definitely “Out of memory” — I’m stuffing a lot of various node modules in there, which is probably causing this problem. But building Sveltekit locally, the build folder is only 8mb. And because it’s the Docker builder running out of memory, provisioning more memory using scale won’t really do anything right?

Edit: just kidding, the build folder is kind of big (57.3MB) lol… I should probably fix that. Curious though, for us who are really into big projects, is this usually bad practice? Should we be breaking up these monoliths? What’s usually the upper bound of a build folder size?

2023-10-28T13:25:04.585 app[6e82de00b03698] syd [info] [ 646.082064] Out of memory: Killed process 1254 (node) total-vm:35319584kB, anon-rss:3615348kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:103252kB oom_score_adj:0

Umpteenth edit: setting this in Docker makes it deploy! Any other number makes it not deploy… ENV NODE_OPTIONS="--max-old-space-size=4096"

I don’t really know what’s going on, but if it works, it works…

Edit: It kind of works, about 3 of 4 tries end up with ✖ Machine 9185e50a413483 [app] update failed: timed out waiting for machine to reach started state: failed to wait for VM 9185e50a413483 in started sta

Edit #??: The Fly.io site is now giving me error 500’s… maybe that’s a sign I should go to bed. Quite often (80% of the time) when trying to deploy, even to a new project, I get Machine e2865944b70518 [app] update failed: failed to update VM e2865944b70518: request returned non-2xx status, 504

About half of my deployments also get: PCI: Fatal: No config space access function found and the Sveltekit Node app fails to launch

Final Edit: Finally got everything to deploy! It was a combination of user error (didn’t set .env properly for Sveltekit; need to set ORIGIN), incorrect configs (needed ENV NODE_OPTIONS="--max-old-space-size=4096" and FROM node:18.15.0-alpine (not latest) in Dockerfile, and removing this mysterious line I don’t know where I got from: BODY_SIZE_LIMIT=20000000), and deploying with a different registry (FLY_REGISTRY_HOST=registry-iad.fly.io fly deploy bc of some registry issues at Fly), and a few other things that might be Fly-side I never figured out (Dockerfile running out of memory sometimes; Docker images failing to fetch, the Fly machine not starting properly). Honestly I’m not really sure why some of it didn’t work for so long, and why it later worked, especially the Docker deployments hanging, or Sveltekit reporting “not found” or responding “invalid request body” or thinking my deployment size to be way too big, but actually kind of small in comparison to “real” monolithic apps.

There was a lot of trial and error, hammering your build machine for the last week (sorry!) and finally got it working — but what makes me kind of uneasy is that I don’t think I solved all the problems of why things never worked but I did figure out how to get a new project / deployment working, with the Docker and fly.toml and .env configurations. Here’s to hoping it doesn’t break again when Node 20 rolls around…! Thanks for everything, Cheers!