# ERROR after fly deploy

**URL:** <https://community.fly.io/t/error-after-fly-deploy/19313>\
**Category:** General\
**Tags:** login-only\
**Created:** [April 16, 2024, 8:31am UTC](https://community.fly.io/t/error-after-fly-deploy/19313 "2024-04-16T08:31:41Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![mikemoore13](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mikemoore13/32/16080_2.png) [@mikemoore13](https://community.fly.io/u/mikemoore13)\
**Post date:** [April 16, 2024, 8:31am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/1 "2024-04-16T08:31:41Z")

</div>

impossible to restart the machine after fly deploy  
2024-04-16T08:27:16.452 proxy[91857507cd6618] cdg [error] machines API returned an error: “process not found”

2024-04-16T08:27:16.519 runner[91857507cd6618] cdg [info] machine exited with exit code 0, not restarting

2024-04-16T08:27:18.404 proxy[91857507cd6618] cdg [info] Starting machine

2024-04-16T08:27:18.482 app[91857507cd6618] cdg [error] 2024-04-16T08:27:18.482615475 [01HVJZWT7NQTYY44A8MKHQYD7G:main:ERROR:src/firecracker/src/main.rs:321] Configuration for VMM from one single json failed: InvalidJson

2024-04-16T08:27:18.485 app[91857507cd6618] cdg [warn] Virtual machine exited abruptly

2024-04-16T08:27:20.708 runner[91857507cd6618] cdg [info] machine exited with exit code 0, not restarting

2024-04-16T08:27:23.405 proxy[91857507cd6618] cdg [error] could not wake up machine due to a timeout requesting from the machines API

2024-04-16T08:27:25.413 proxy[91857507cd6618] cdg [info] Starting machine

2024-04-16T08:27:25.480 app[91857507cd6618] cdg [error] 2024-04-16T08:27:25.480677941 [01HVJZWT7NQTYY44A8MKHQYD7G:main:ERROR:src/firecracker/src/main.rs:321] Configuration for VMM from one single json failed: InvalidJson

2024-04-16T08:27:25.482 app[91857507cd6618] cdg [warn] Virtual machine exited abruptly

---

<div class="post-metadata">

**Author:** ![somt0xhi](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/somt0xhi/32/42794_2.png) [@somt0xhi](https://community.fly.io/u/somt0xhi)\
**Post date:** [April 16, 2024, 9:17am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/2 "2024-04-16T09:17:22Z")

</div>

Hello @mikemoore13 ,

Thanks for reporting this. We are currently working to resolve this issue.  
Can you try redeploying your app?

---

<div class="post-metadata">

**Author:** ![mikemoore13](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mikemoore13/32/16080_2.png) [@mikemoore13](https://community.fly.io/u/mikemoore13)\
**Post date:** [April 16, 2024, 9:21am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/3 "2024-04-16T09:21:51Z")

</div>

seems solved after a new fly deploy

---

<div class="post-metadata">

**Author:** ![al0xd](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/al0xd/32/14383_2.png) [@al0xd](https://community.fly.io/u/al0xd)\
**Post date:** [April 16, 2024, 9:32am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/4 "2024-04-16T09:32:57Z")

</div>

I’ve encountered a similar issue as well. I’ve tried redeploying several times, but it still stops at the image building stage

```auto
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-morning-moon-1135`
WARN Failed to start remote builder heartbeat: remote builder app unavailable
Waiting for remote builder fly-builder-morning-moon-1135...

```

---

<div class="post-metadata">

**Author:** ![rugwiro](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/rugwiro/32/49239_2.png) [@rugwiro](https://community.fly.io/u/rugwiro)\
**Post date:** [April 16, 2024, 10:25am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/5 "2024-04-16T10:25:33Z")

</div>

Hey @al0xd, the fastest way to return to deploying is to delete the fly-builder app `fly-builder-morning-moon-1135`. A new one should be created on your next deployment.

---

<div class="post-metadata">

**Author:** ![al0xd](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/al0xd/32/14383_2.png) [@al0xd](https://community.fly.io/u/al0xd)\
**Post date:** [April 16, 2024, 11:12am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/6 "2024-04-16T11:12:56Z")

</div>

Hi @rugwiro Thank you, after deleting, I was able to deploy successfully. Although the specific reason is unclear, it worked.

---

<div class="post-metadata">

**Author:** ![mathiasn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mathiasn/32/35151_2.png) [@mathiasn](https://community.fly.io/u/mathiasn)\
**Post date:** [April 16, 2024, 11:25am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/7 "2024-04-16T11:25:08Z")

</div>

Even after the deletion, the deployment fails. I’ve already sent an email to your support as this becomes a major problem now.

---

<div class="post-metadata">

**Author:** ![rugwiro](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/rugwiro/32/49239_2.png) [@rugwiro](https://community.fly.io/u/rugwiro)\
**Post date:** [April 16, 2024, 11:59am UTC](https://community.fly.io/t/error-after-fly-deploy/19313/8 "2024-04-16T11:59:53Z")

</div>

Your machine was stuck in a weird in-between state, even though the incident had been resolved. I tried to manually unclog in but it seems to have fallen to a new bug we are still investigating. Destroying the builder was the fastest way to get you unblocked. Glad it worked out.

---

<div class="post-metadata">

**Author:** ![rugwiro](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/rugwiro/32/49239_2.png) [@rugwiro](https://community.fly.io/u/rugwiro)\
**Post date:** [April 16, 2024, 12:06pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/9 "2024-04-16T12:06:48Z")

</div>

Hey @mathiasn, have received a response over there yet?

EDIT: I just responded to the email.

---

<div class="post-metadata">

**Author:** ![mathiasn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mathiasn/32/35151_2.png) [@mathiasn](https://community.fly.io/u/mathiasn)\
**Post date:** [April 16, 2024, 1:28pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/10 "2024-04-16T13:28:02Z")

</div>

Unfortunately the issue still persists…

```auto
> Created release_command machine 683324ef7de028
> Waiting for 683324ef7de028 to have state: started
> Machine 683324ef7de028 has state: started
> Waiting for 683324ef7de028 to have state: destroyed
> Machine 683324ef7de028 has state: destroyed
> Waiting for 683324ef7de028 to get exit event
✔ release_command 683324ef7de028 completed successfully
Updating existing machines in 'my-app' with rolling strategy
> [1/2] Updating 1781975b277458 [app]
> [1/2] Updating 1781975b277458 [app]
> [1/2] Waiting for 1781975b277458 [app] to have state: started
> [1/2] Machine 1781975b277458 [app] has state: started
> [1/2] Checking that 1781975b277458 [app] is up and running
Smoke checks for 1781975b277458 failed: the app appears to be crashing
Check its logs: here's the last lines below, or run 'fly logs -i 1781975b277458':
  Successfully prepared image registry.fly.io/my-app:deployment-01HVKGV0EAQ5399ZDRSZS7N05A (5.063147614s)
✖ [1/2] Machine 1781975b277458 [app] update failed: smoke checks for 1781975b277458 failed: the app appears to be crashing
  Configuring firecracker
  [0.032764] PCI: Fatal: No config space access function found
   INFO Starting init (commit: 5b8fb02)...
   INFO Preparing to run: `/app/docker-entrypoint bin/my-app start` as root
   INFO [fly api proxy] listening at /.fly/api
  2024/04/16 13:21:29 listening on [fdaa:0:4682:a7b:b9b8:5234:8318:2]:22 (DNS: [fdaa::3]:53)
  setting swap...
  Setting up swapspace version 1, size = 5528579685 bytes
  UUID=a7edc778-6178-4250-b0db-ecb7213fd384
  swapon: /swapfile: Invalid argument
  Machine created and started in 5.961s
   INFO Main child exited normally with code: 1
   INFO Starting clean up.
   WARN could not unmount /rootfs: EINVAL: Invalid argument
  [1.255180] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
  [0.032905] PCI: Fatal: No config space access function found
   INFO Starting init (commit: 5b8fb02)...
   INFO Preparing to run: `/app/docker-entrypoint bin/my-app start` as root
   INFO [fly api proxy] listening at /.fly/api
  Machine started in 354ms
  2024/04/16 13:21:30 listening on [fdaa:0:4682:a7b:b9b8:5234:8318:2]:22 (DNS: [fdaa::3]:53)
  setting swap...
  Setting up swapspace version 1, size = 5528579685 bytes
  UUID=0aa2a45c-ccad-4e43-acfe-abbf66a5a9e6
  swapon: /swapfile: Invalid argument
   INFO Main child exited normally with code: 1
   INFO Starting clean up.
   WARN could not unmount /rootfs: EINVAL: Invalid argument
  [1.255370] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
  Machine started in 379ms
Checking DNS configuration for my-app.fly.dev
Error: smoke checks for 1781975b277458 failed: the app appears to be crashin

```

We have this `docker-entrypoint` defined:

```auto
#!/bin/sh -e

if [$(id -u) -eq 0] && [! -z "$SWAP"] && [-z "$RELEASE_COMMAND"]; then
  # release_command: We are NOT in a temporary VM, run as normal...
  echo "setting swap..."
  # allocate swap space
  fallocate -l $(($(stat -f -c "(%a*%s/10)*7" .))) /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
  free -m
  exec su appuser $0 $@
else
  exec "${@}"
fi

```

That wasn’t an issue so far. Interestingly the deployment on our Fly staging environment was successfully with the same code!! Just Production does not deploy… We deleted fly app builder already couple of times

---

<div class="post-metadata">

**Author:** ![ben-io](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/ben-io/32/35648_2.png) [@ben-io](https://community.fly.io/u/ben-io)\
**Post date:** [April 16, 2024, 3:44pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/11 "2024-04-16T15:44:52Z")

</div>

Hey @mathiasn. Some recent [changes we made](https://community.fly.io/t/a-new-way-to-rootfs/19196) have affected how swap is activated. To configure swap space, simply set [`swap_size_mb`](https://fly.io/docs/reference/configuration/#swap_size_mb-option) in your fly.toml.

---

<div class="post-metadata">

**Author:** ![mathiasn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mathiasn/32/35151_2.png) [@mathiasn](https://community.fly.io/u/mathiasn)\
**Post date:** [April 16, 2024, 3:46pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/12 "2024-04-16T15:46:41Z")

</div>

Yeah got that already. Can confirm that this works.

For us the issue was that our staging environment continued working with old config, only production didn’t like it. Swap and RAM size was the same. Not clear yet why it behaved differently then.

---

<div class="post-metadata">

**Author:** ![JP\_Phillips](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jp_phillips/32/18337_2.png) [@JP\_Phillips](https://community.fly.io/u/JP_Phillips)\
**Post date:** [April 16, 2024, 3:51pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/13 "2024-04-16T15:51:44Z")

</div>

> [@mathiasn](#):
>
> For us the issue was that our staging environment continued working with old config, only production didn’t like it

Do you use a different organization to separate your staging and production environments? Our rollout of the new rootfs relies on the organization identifier to determine which orgs will use the new rootfs config.

---

<div class="post-metadata">

**Author:** ![mathiasn](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mathiasn/32/35151_2.png) [@mathiasn](https://community.fly.io/u/mathiasn)\
**Post date:** [April 16, 2024, 3:52pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/14 "2024-04-16T15:52:25Z")

</div>

Yes, it’s a different org.

---

<div class="post-metadata">

**Author:** ![cloudmall](https://avatars.discourse-cdn.com/v4/letter/c/87869e/32.png) [@cloudmall](https://community.fly.io/u/cloudmall)\
**Post date:** [April 19, 2024, 1:12pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/15 "2024-04-19T13:12:02Z")

</div>

I’m facing this same issue, tried destroying the machines and redeploying repeatedly but still the same restart loop.

Starting machine

2024-04-19T12:56:18.797 app[4d89d965b74638] sjc [info] [0.037462] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!

2024-04-19T12:56:18.823 app[4d89d965b74638] sjc [info] [0.040924] PCI: Fatal: No config space access function found

2024-04-19T12:56:19.047 app[4d89d965b74638] sjc [info] INFO Starting init (commit: 65db7f7)…

2024-04-19T12:56:19.065 app[4d89d965b74638] sjc [info] INFO Preparing to run: `docker-entrypoint.sh node` as root

2024-04-19T12:56:19.073 app[4d89d965b74638] sjc [info] INFO [fly api proxy] listening at /.fly/api

2024-04-19T12:56:19.080 app[4d89d965b74638] sjc [info] 2024/04/19 12:56:19 listening on [fdaa:0:ce68:a7b:247:28ce:25f1:2]:22 (DNS: [fdaa::3]:53)

2024-04-19T12:56:19.084 runner[4d89d965b74638] sjc [info] Machine started in 445ms

2024-04-19T12:56:20.074 app[4d89d965b74638] sjc [info] INFO Main child exited normally with code: 0

2024-04-19T12:56:20.087 app[4d89d965b74638] sjc [info] INFO Starting clean up.

2024-04-19T12:56:20.089 proxy[4d89d965b74638] sjc [info] machine started in 1.459134603s

2024-04-19T12:56:20.093 app[4d89d965b74638] sjc [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2024-04-19T12:56:20.093 app[4d89d965b74638] sjc [info] [1.307619] reboot: Restarting system

2024-04-19T12:56:20.995 runner[4d89d965b74638] sjc [info] machine exited with exit code 0, not restarting

2024-04-19T12:56:23.525 proxy[4d89d965b74638] sjc [info] Starting machine

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/fly1/original/3X/6/e/6e8955d00cb218e186ebbe3a6ab06fefdce15290.svg) [@system](https://community.fly.io/u/system)\
**Post date:** [April 26, 2024, 1:12pm UTC](https://community.fly.io/t/error-after-fly-deploy/19313/16 "2024-04-26T13:12:09Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.