Out of Memory on Deploy

I’m trying to deploy a node gql app with a Dockerfile from a somewhat large monorepo. I keep running into an error ( Error: command finished with error: exit status 137), which typically means the vm ran out of memory trying to build the image. Is there a way to increase the memory for building? I’ve already increased the memory for my app, but I’m still running into the 137 issue. Any help would be appreciated, as we’re trying to determine if we’re going to move all of our apps and apis to fly.

Hi @Charlie_Blackstock

Are you using the remote builder or local?

If you’re building during CI, you might benefit from building locally on the CI server instead of using remote builder. You can use the --local-only flag to ensure it doesn’t use the remote builder.

Did you figure this out? Are you on an M1 Mac by chance? The QEMU virtualization it uses do do Docker builds has a memory limit. I would try --remote-only in addition to what @charsleysa said. I would be surprised if you’re hitting memory limits on remote builders, but we definitely need to know if local or remote are what’s breaking. :slight_smile:

1 Like

Hey kurt! No I still haven’t firgured this out. I am on an new M1Max. I can build the docker file just fine locally, and we have it getting built remotely in CI. So i’m just running flyctl --app app deploy --config ./apps/graphql/fly.toml --dockerfile ./apps/graphql/Dockerfile, which I’d assume is a remote context.

That would mean you’re building on your CI server. You’d need to add --remote-only for it to build on the Fly remote builders, which have 2GB of memory. Which CI service are you using?

1 Like

Gotcha I added --remote-only add that got me deploy building! So it looks like it builds and starts correctly now, but if failing the health checks.

It’s hard to see why it’d be failing just from the health check logs. I can see the server starts up properly and logs the port. I’ve made sure to add the port as an env to the fly.toml and I’ve exposed the port in my dockerfile, but I’m just getting failed health checks with this as the only error, “Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1”

Thank you for all the help so far, I really appreciate it!

1 Like

For what it’s worth, building for amd64 on M1 macs is different than just a normal Docker build. So you were probably hitting memory limits related to QEMU or whatever it’s using for the underlying virtualization when building locally.