Similar to this similar issue fly-app-servers-down/19229/1
The steps I took.
- Fly launch (OK
- Flay deploy (Not OK, app crash at startup, restarts until it gives up)
At that point the deploy command hangs. I have to kill the cli
- Fly deploy (Not OK, now always hangs after building the image just fine).
At that point all I can do is destroy the app. the log never show it start, even unsuccessfully.
The other thread says to destroy the machine, not sure it’s the same issue.
the app shows no more activity in the live logs, ending with the 10 count reached.
2024-06-27T06:43:19Z app[3d8dd764b2e5e8] waw [info] INFO Main child exited normally with code: 1
2024-06-27T06:43:19Z app[3d8dd764b2e5e8] waw [info] INFO Starting clean up.
2024-06-27T06:43:19Z app[3d8dd764b2e5e8] waw [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-06-27T06:43:19Z app[3d8dd764b2e5e8] waw [info][ 1.567748] reboot: Restarting system
2024-06-27T06:43:19Z runner[3d8dd764b2e5e8] waw [info]machine has reached its max restart count (10)
Destroying the app and launching it again. solves the issue
Hi @Hirakosan !
Checking logs for machine 3d8dd764b2e5e8 I can see the following error right before the machine dying:
Jun 27, 2024 @ 06:43:19.674116457 [32m INFOe[0m Main child exited normally with code: 1
Jun 27, 2024 @ 06:43:18.790230457 {"level":"fatal","error":"can't setup store: could not migrate datastore: unable to open database file: no such file or directory","time":"2024-06-27T06:43:18Z","message":"error running server"}
Jun 27, 2024 @ 06:43:18.788775271 {"level":"warn","time":"2024-06-27T06:43:18Z","message":"no sqlite3 file found, will create one at '/var/lib/woodpecker/woodpecker.sqlite'"}
Jun 27, 2024 @ 06:43:18.788635905 {"level":"info","time":"2024-06-27T06:43:18Z","message":"log level: info"}
Jun 27, 2024 @ 06:43:18.694326000 machine started in 669.190377ms
I’d say the app is dying because it’s not able to “migrate datastore”. Not sure how recreating the app can solve the issue though, would be possible on first run the migrate sequence is not executed?
The app dying is not the issue though.
Yes it is dying. Nothing to do with fly that it fails and dies.
What is maybe a problem is that the fly cli when hangs and would not redeploy it (with a new image that fixes that migration issue which is an app specific error due to some files not present on the fs, again nothing to do with fly of course).
When things hang like that, I have to resort to:
kill -3 the cli process (it would not respond to a normal SIGINT to stop it gracefullyfly destroy that app- (re)
fly launch that app
And hope I troubleshoot any runtime issue before it jam again. I also fear i’m shamelessly leaving some builder machine ghosting behind, doing I don’t know what but hanging.