impossible to restart the machine after fly deploy
2024-04-16T08:27:16.452 proxy[91857507cd6618] cdg [error] machines API returned an error: “process not found”
2024-04-16T08:27:16.519 runner[91857507cd6618] cdg [info] machine exited with exit code 0, not restarting
2024-04-16T08:27:18.482 app[91857507cd6618] cdg [error] 2024-04-16T08:27:18.482615475 [01HVJZWT7NQTYY44A8MKHQYD7G:main:ERROR:src/firecracker/src/main.rs:321] Configuration for VMM from one single json failed: InvalidJson
2024-04-16T08:27:25.480 app[91857507cd6618] cdg [error] 2024-04-16T08:27:25.480677941 [01HVJZWT7NQTYY44A8MKHQYD7G:main:ERROR:src/firecracker/src/main.rs:321] Configuration for VMM from one single json failed: InvalidJson
I’ve encountered a similar issue as well. I’ve tried redeploying several times, but it still stops at the image building stage
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-morning-moon-1135`
WARN Failed to start remote builder heartbeat: remote builder app unavailable
Waiting for remote builder fly-builder-morning-moon-1135...
Hey @al0xd, the fastest way to return to deploying is to delete the fly-builder app fly-builder-morning-moon-1135. A new one should be created on your next deployment.
Your machine was stuck in a weird in-between state, even though the incident had been resolved. I tried to manually unclog in but it seems to have fallen to a new bug we are still investigating. Destroying the builder was the fastest way to get you unblocked. Glad it worked out.
> Created release_command machine 683324ef7de028
> Waiting for 683324ef7de028 to have state: started
> Machine 683324ef7de028 has state: started
> Waiting for 683324ef7de028 to have state: destroyed
> Machine 683324ef7de028 has state: destroyed
> Waiting for 683324ef7de028 to get exit event
✔ release_command 683324ef7de028 completed successfully
Updating existing machines in 'my-app' with rolling strategy
> [1/2] Updating 1781975b277458 [app]
> [1/2] Updating 1781975b277458 [app]
> [1/2] Waiting for 1781975b277458 [app] to have state: started
> [1/2] Machine 1781975b277458 [app] has state: started
> [1/2] Checking that 1781975b277458 [app] is up and running
Smoke checks for 1781975b277458 failed: the app appears to be crashing
Check its logs: here's the last lines below, or run 'fly logs -i 1781975b277458':
Successfully prepared image registry.fly.io/my-app:deployment-01HVKGV0EAQ5399ZDRSZS7N05A (5.063147614s)
✖ [1/2] Machine 1781975b277458 [app] update failed: smoke checks for 1781975b277458 failed: the app appears to be crashing
Configuring firecracker
[ 0.032764] PCI: Fatal: No config space access function found
INFO Starting init (commit: 5b8fb02)...
INFO Preparing to run: `/app/docker-entrypoint bin/my-app start` as root
INFO [fly api proxy] listening at /.fly/api
2024/04/16 13:21:29 listening on [fdaa:0:4682:a7b:b9b8:5234:8318:2]:22 (DNS: [fdaa::3]:53)
setting swap...
Setting up swapspace version 1, size = 5528579685 bytes
UUID=a7edc778-6178-4250-b0db-ecb7213fd384
swapon: /swapfile: Invalid argument
Machine created and started in 5.961s
INFO Main child exited normally with code: 1
INFO Starting clean up.
WARN could not unmount /rootfs: EINVAL: Invalid argument
[ 1.255180] reboot: Restarting system
machine did not have a restart policy, defaulting to restart
[ 0.032905] PCI: Fatal: No config space access function found
INFO Starting init (commit: 5b8fb02)...
INFO Preparing to run: `/app/docker-entrypoint bin/my-app start` as root
INFO [fly api proxy] listening at /.fly/api
Machine started in 354ms
2024/04/16 13:21:30 listening on [fdaa:0:4682:a7b:b9b8:5234:8318:2]:22 (DNS: [fdaa::3]:53)
setting swap...
Setting up swapspace version 1, size = 5528579685 bytes
UUID=0aa2a45c-ccad-4e43-acfe-abbf66a5a9e6
swapon: /swapfile: Invalid argument
INFO Main child exited normally with code: 1
INFO Starting clean up.
WARN could not unmount /rootfs: EINVAL: Invalid argument
[ 1.255370] reboot: Restarting system
machine did not have a restart policy, defaulting to restart
Machine started in 379ms
Checking DNS configuration for my-app.fly.dev
Error: smoke checks for 1781975b277458 failed: the app appears to be crashin
We have this docker-entrypoint defined:
#!/bin/sh -e
if [ $(id -u) -eq 0 ] && [ ! -z "$SWAP" ] && [ -z "$RELEASE_COMMAND" ]; then
# release_command: We are NOT in a temporary VM, run as normal...
echo "setting swap..."
# allocate swap space
fallocate -l $(($(stat -f -c "(%a*%s/10)*7" .))) /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo 10 > /proc/sys/vm/swappiness
swapon /swapfile
echo 1 > /proc/sys/vm/overcommit_memory
free -m
exec su appuser $0 $@
else
exec "${@}"
fi
That wasn’t an issue so far. Interestingly the deployment on our Fly staging environment was successfully with the same code!! Just Production does not deploy… We deleted fly app builder already couple of times
Yeah got that already. Can confirm that this works.
For us the issue was that our staging environment continued working with old config, only production didn’t like it. Swap and RAM size was the same. Not clear yet why it behaved differently then.
Do you use a different organization to separate your staging and production environments? Our rollout of the new rootfs relies on the organization identifier to determine which orgs will use the new rootfs config.
I’m facing this same issue, tried destroying the machines and redeploying repeatedly but still the same restart loop.
Starting machine
2024-04-19T12:56:18.797 app[4d89d965b74638] sjc [info] [ 0.037462] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-04-19T12:56:18.823 app[4d89d965b74638] sjc [info] [ 0.040924] PCI: Fatal: No config space access function found
2024-04-19T12:56:19.047 app[4d89d965b74638] sjc [info] INFO Starting init (commit: 65db7f7)…
2024-04-19T12:56:19.065 app[4d89d965b74638] sjc [info] INFO Preparing to run: docker-entrypoint.sh node as root
2024-04-19T12:56:19.073 app[4d89d965b74638] sjc [info] INFO [fly api proxy] listening at /.fly/api