Issues with machines restart policy

1

A couple of days ago, I migrated a project from v1 to v2.

It’s a service with no HTTP so the [[services]] block in the fly.toml is empty.

Today a user mentioned something wasn’t working right and turns out this machine was stopped.

I’m not using the machines API and I certainly didn’t stop it manually.

In the dashboard I can see this:

exit	2023-04-01T14:12:23.496	flyd

image
image2794×918 133 KB

So it was stopped by flyd a couple of days after the machine 91857293c74328 was created and started. But why?

I can’t see any errors in the logs. This project has been running for months as a v1 app, I doubt there’s anything wrong with the code that would have caused this.

Maybe it’s because it has no services?

1 Like
2

The exit event indicates the VM shutdown on its own with exit code 0. It’s possible this machine was on a host that rebooted and we currently can’t easily detect if a machine stopped due to the host shutdown signal or some other reason. If the machine is meant to always be running, it’s a good practice to set a restart policy of “always” so flyd will restart the machine regardless of exit code.

1 Like
3

Thanks JP.

So how/where do I set this?

I can’t seem to find any reference in the docs or the new migration guide :thinking:

https://fly.io/docs/apps/migrate-to-v2/

4

So I did fly machines update 91857293c74328 to see if there were any options to set the restart policy and it automatically showed me these changes to the config:


  	... // 10 identical lines
  			"fly_release_version": "1"
  		},
- 		"restart": {},
+ 		"restart": {
+ 			"policy": "always"
+ 		},
  		"services": [
  			{}
  	... // 3 identical lines
  			"cpus": 1,
  			"memory_mb": 256
- 		}
+ 		},
+ 		"dns": {}
  	}

So I accepted.

It’s a bit weird this isn’t the default though?

Now I have to do this for like 10 more machines…

5

Yea, machines created before we made it the default had a different restart policy.

6

So is there an easy way to see the config of a machine with the fly CLI that doesn’t involve using the API?

Edit:

Also would be great if you could update all the machines in an app at once :grimacing:

1 Like
7

When doing fly machines update to the PG machines, nothing showed up.

1 Like
8

How do I check my machines configs for restart policy? fly machine update or fly machine status don’t show anything interesting

1 Like
9

This one I can answer!

$ fly m status <machine-id> -d will give you the whole Machine config as JSON.

You’ll see the "restart": entry like in @pier’s post.

4 Likes
10

@catflydotio @JP_Phillips

Can someone please update Fly Launch configuration (fly.toml) · Fly Docs ? I ran into this…and I’m sure also many others.

11

How do I set this in the fly.toml?

12

I just created a app and it defaulted to empty.

13

Did you create a new v2 app or just a machine?

14

A new app, sorry.

15

The Machine restart policy isn’t yet a part of app-wide configuration. It will get there.

In the meantime, you don’t have to go all the way to the API: flyctl can do it with fly machine update. Unfortunately, yes, once you’ve scaled up, that currently means you have to run it on each Machine separately.

1 Like
16

What command do I need to run exactly?

17 
fly machines update <machine-id>
18

Yes I know that, but to change the restart policy specifically, when I run this it tells me there is no change to the config.

19

If you’re certain your machine doesn’t have that policy then no idea. That’s what worked for me.

To check the JSON config you can do:

fly m status <machine-id> -d
20

With this my restart section is this:

 "restart": {},