app down, log shows "failed to compare firecracker", "no such file or directory"

Questions / Help
1

Recently my app was returning 502. When I went to see the logs to try to understand why the app was not discoverable by the reverse proxy… I found this in the log.

machines API returned an error: "failed to compare \"/opt/firecracker-v0.24.x/bin/firecracker\" and \"/opt/flyd/firefly/01HTZHVVD3AF4HZS62J4B357Q3/root/firecracker\": stat /opt/firecracker-v0.24.x/bin/firecracker: no such file or directory"

See bellow more context around this line.

2024-04-12 13:43:39.000	Starting machine
2024-04-12 13:43:10.000	machine exited with exit code 0, not restarting
2024-04-12 13:43:09.000	machine exited with exit code 0, not restarting
2024-04-12 13:40:51.000	could not find a good candidate within 90 attempts at load balancing
2024-04-12 13:40:51.000	timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
...... 3 MINUTES SAME SOL ....
2024-04-12 06:47:50.000	timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)
2024-04-12 06:47:48.000	Starting machine
2024-04-12 06:47:48.000	machines API returned an error: "failed to compare \"/opt/firecracker-v0.24.x/bin/firecracker\" and \"/opt/flyd/firefly/01HTZHVVD3AF4HZS62J4B357Q3/root/firecracker\": stat /opt/firecracker-v0.24.x/bin/firecracker: no such file or directory"
2024-04-12 06:47:48.000	Starting machine
2024-04-12 06:47:48.000	machines API returned an error: "failed to compare \"/opt/firecracker-v0.24.x/bin/firecracker\" and \"/opt/flyd/firefly/01HTZHVNV4XFS2PQVP6GYS52YT/root/firecracker\": stat /opt/firecracker-v0.24.x/bin/firecracker: no such file or directory"
2024-04-12 06:47:48.000	Starting machine
2024-04-12 06:44:55.000	[  426.855180] reboot: Restarting system
2024-04-12 06:44:55.000	 INFO Starting clean up.
2024-04-12 06:44:55.000	 INFO Main child exited normally with code: 0
2024-04-12 06:44:54.000	[2024-04-12 05:44:54 +0000] [314] [INFO] Shutting down: Master
2024-04-12 06:44:54.000	[2024-04-12 00:44:54 +0000] [323] [INFO] Worker exiting (pid: 323)
2024-04-12 06:44:54.000	[2024-04-12 00:44:54 +0000] [324] [INFO] Worker exiting (pid: 324)
2024-04-12 06:44:54.000	[2024-04-12 05:44:54 +0000] [314] [INFO] Handling signal: int
2024-04-12 06:44:54.000	 INFO Sending signal SIGINT to main child process w/ PID 314
2024-04-12 06:44:54.000	Downscaling app webapp from 1 machines to 0 machines, stopping machine ******* (region=***, process group=***)

Does any one know what happened with this machine at this time?

Thank you in advance.

2

Hi @gtv—sorry that you ran into this! A bad update was pushed to our servers yesterday morning, causing issues with Machine creations and starts. After becoming aware of the problem, we quickly implemented a fix, and the incident was marked resolved at 07:47 UTC yesterday. Machine creations and starts should be back to normal since then. If you’re still experiencing trouble, though, please let us know.

3

Hi @MatthewIngwersen ,
Thank you very much for your quick reply and explanation.
I haven’t noticed that happening with my app anymore. I’ll let you know in case the same issue takes place again.
Cheers,

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.