Deployment hangs at "reboot: Restarting System"

Hey there!

I’m trying to deploy a Phoenix LiveView app but the deployment hangs. I’ve launched the app with fly launch (this worked flawlessly) and now I’m trying to deploy some changes with fly deploy. The client CLI output stops at

Running my-app release_command: /app/bin/migrate
  release_command 683d92dc11dd48 completed successfully
Updating existing machines in 'my-app' with rolling strategy
  [1/2] Updating e28671ef729038 [app]

and if I look at fly logs the last message is
app[080e475b621558] lhr [info][ 380.737148] reboot: Restarting system

It’s been stuck like this for hours.
Thanks for any help!

Hey there,

That is definitely a bit odd, especially because from the cli ouput it seems to show that everything is running smoothly. Until you looked deeper into the logs. With it being stuck I wonder if a simple restart would do the trick? Maybe try taking a look at these docs and see if they end up helping at all?

  1. Restart an App or a Machine · Fly Docs.
  2. fly machine restart · Fly Docs

hi @timkonieczny

Just popping in here to add that the log:

app[080e475b621558] lhr [info][ 380.737148] reboot: Restarting system

is a totally normal log to see. It’s the last log when a Machine is stopped by the auto start and stop feature set in your fly.toml file. Logs resume when the Machine is started again (by the same feature, when there’s traffic to your app).

Let us know if the app restart helps with the stuck deploy!

I seem to have the same problem. I do not have the auto start and stop feature enabled though. I did not use the fly launch command but created the app manually with fly apps create and assigned a shared ipv4 address. Afterwards, I created and destroyed a few machines to try things out. Now with a fly.toml and all machines destroyed, I am ready for an initial fly deploy, but the command hangs:

==> Verifying app config
Validating ******/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'nginx' remotely...
image found: img_0lq747eo3e8p6x35

Watch your deployment at https://fly.io/apps/paperless-home/monitoring

skipping provisioning

Ctrl+C does not help stopping the hang, I have to kill the terminal application.

EDIT: The monitoring shows nothing at all after doing fly deploy.

For me, fly apps restart myapp does nothing and produces no log. I guess that is what I expect since there are no machines.

After releasing the shared ipv4 and doing fly deploy again, I finally get some insight in what is going wrong:
Error: input:4: createRelease We need your payment information to continue! Add a credit card or buy credit: https://fly.io/dashboard/paperless-home/billing
I noticed something odd earlier on already: I was not able to create a second machine because apparently I have not provided payment information. But I definitely did provide my credit card. Not I just noticed that somehow I got bounced back from “good standing” to “trial”. After entering my credit card details again, the deployment worked fine.

Thanks for the suggestions.

I’ve tried deploying again after fly apps restart but it’s still not working.
My Account Status is Good Standing, so that’s not the issue either.

EDIT
I did a bit of trial and error, but still no luck. Here’s what I’ve tried:

fly apps restart
fly deploy → Unsuccessful, same problem
deleted all apps
fly launch → Successful deployment
made some changes in the code
fly deploy → Unsuccessful with error:

Updating existing machines in 'my-app' with rolling strategy
  [1/2] Updating d89169da440438 [app]
Error: failed to update VM d89169da440438: You have reached the maximum number of machines for this app.

To unlock more features please add credit to your org (using an admin account) from your billing page:

 https://fly.io/dashboard/name/billing

Or contact us at billing@fly.io and we'll take a closer look at your account.

deleted all unused / destroyed apps
fly launch → Unsuccessful because db wasn’t ready in time for some reason
deleted everything again
fly launch → Unsuccessful. Problem: Cannot deploy because of insufficient memory - #48 by andie
deleted everything again
fly launch with other region → Successful
made some changes in the code
fly deploy → Unsuccessful

So in the end the same problem remains.

@timkonieczny

Suggest contacting billing@fly.io. Maybe there’s a problem with your card.

2 Likes

Thanks @andie. There really seemed to have been a problem with billing. I’ve just deployed successfully.

3 Likes

I had the same issue and it was indeed a billing issue, thanks.
Flyio should just throw an error in this case but instead I have been debugging for the whole day.

2 Likes

hi @Terry

I understand how frustrating that can be. Did you get a different error from the one above that suggests contacting billing? Or was the deploy just hanging on the first Machine update?

Was just hanging and no mention to billing

Thanks for the feedback. We’ll look into the error (or non-error)

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.