Is SJC Down?

I noticed that my app isnt using SJC and is using a backup server. Something wrong with SJC?

Instances going to backup regions usually means we’re full in the non-backup region. It could also mean it’s down or “marked as down” as we troubleshoot problems.

In this case, SJC is full :slight_smile:! We’ll need to provision more servers in that region.

@nahtnam actually, I misspoke. There is still capacity there.

You can remove backup regions if you precisely want you app to only run in SJC, but it means that in the future if we don’t have capacity, you might not be able to deploy. You can do that with flyctl regions backup sjc and that will set your backup regions the same as your regions. If you have more than 1 region and don’t want backups for them, you’ll have to input the whole list of regions at once with flyctl regions backup.

Which app is this? There should still be 50+ available CPUs in that region and more than enough memory.

I wanted to keep the backup regions for this purpose but since it didn’t deploy to sjc then I assumed that it was down or full. The app is caddie-api, I haven’t changed anything yet if you want to take a look. Since SJC is not full, how can I move the server back to SJC while retaining my backups?

I just took a look.

Since you have 3 regions specified, you should definitely turn off backup regions. You can do this by setting flyctl regions backup sjc, though it’s a bit of a hack.

This will work fine, if SJC fails it will reschedule the SJC VM to one of your other regions.

When a VM lands in a backup region, the only way to move it (currently) is to trigger a deploy. You can either run flyctl deploy to do this, or change the region config. We’ll have a command to reschedule an instance someday!

Makes sense. I decided to set the backup region to lax just to see what would happen and it deployed to LAX instead of SJC so there might be something not allowing SJC deployments to go through? Might be worth investigation. Settings SJC as the backup ended up pushing it to SJC so that fixed my issue.

Thanks!

This is a problem with the way we do balancing right now. SJC is busier than LAX, and the scoring system we use (via Nomad) will weight that a lot higher than it should. We need to rebuild all of that, though, it’s woefully unpredictable sometimes.

1 Like