Our web service is down When will it be back to normal?..

[image] scottyeung: CPU load maxed out once I redeployed again I sort of get this as well, causing downtime for 15 minutes and then it comes back to “sort-of” normal. [CleanShot 2024-11-26 at 13.24.27@2x]

Yeah, I’m still having issues. Error: release command failed - aborting deployment. error running release_command machine: error updating release_command machine: failed to update VM 3d8d503b97e568: invalid_argument: unable to update machine configured for auto destroy (Request ID: 01JDM8DT8X7SRAW…

For anyone with something that looks similar, I found a machine in the dashboard that was not in a started state and I destroyed it and my deploy went through.

[image] licarth-switchgrid: I sort of get this as well, causing downtime for 15 minutes and then it comes back to “sort-of” normal. We’ve just seen our two instances recover after about 1 hour and 15 minutes. The two other instances I’d tried creating (1 in an existing region, 1 in a new one)…

Confirmed that this worked for me! Thanks.

I’m now thinking that the problems we’ve experienced aren’t related the downtime, but that CPU quotas have been turned on: [image] Predictable Processor Performance Fresh Produce Continuing with the rollout schedule, we’re moving to 100% enforcement of quotas today. I’m s…

My app’s now back online. Haven’t done anything myself so it seems like either they’ve done something in the past few hours or whatever they did during the night took a while to fully take effect.

I think it is a mixture of both. I do see the quota was reset back to 75% earlier today or at least that is what the graphs show so maybe there is something to it there.

Looks like you’re right, the baseline jumped back up from ~6% to ~30% during one of my restarts.

Yes, it has been confirmed. [image] Predictable Processor Performance Fresh Produce We did. Sorry for not posting an update here. A number of apps are having issues today and we’re trying to isolate the cause.

fly.io site is currently inaccessible...

rodolfo November 26, 2024, 4:07pm 74

Yes, it has been confirmed.