# Instance or service not restarted when I expected it to **URL:** \ **Category:** Questions / Help\ **Created:** [July 25, 2022, 8:01pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155 "2022-07-25T20:01:55Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![davidhodge](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/davidhodge/32/9908_2.png) [@davidhodge](https://community.fly.io/u/davidhodge)\ **Post date:** [July 25, 2022, 8:01pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/1 "2022-07-25T20:01:55Z")
Hey all, Starting Saturday evening, one of my Fly services running a single instance on a shared-cpu-1x with 512MB RAM was killed due to OOM. I was surprised by what happened next and would love some guidance on how to think about how Fly operates here and on what to do in the future. In case it’s relevant, this is essentially a worker where we only want one of them running at a time. 1. I expected [Fly.io](http://Fly.io) to try to boot a new instance or restart the existing instance, but it did not. 2. If it was not able restart the old instance or start a new one, I’d have expected to be notified in some manner. 3. Further, the status of the service, according to the fly dashboard seemed to either show “pending” and/or I think showed something in green, neither of which make sense for a service that wasn’t even running. Mitigations so far: 1. I’ve upped the instance size to be dedicated and have 4GB of RAM. 2. We’ve put in better alerting on our existing systems. Thoughts / ideas? Thanks! ![Unknown-5](https://global.discourse-cdn.com/fly1/original/3X/b/8/b8c451949a5bc891ae1633eff991a74fdb9dd701.jpeg) ---
**Author:** ![greg](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/greg/32/2119_2.png) [@greg](https://community.fly.io/u/greg)\ **Post date:** [July 25, 2022, 8:17pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/2 "2022-07-25T20:17:45Z")
This _may_ not be the case here (as it shows as pending) however I found that when my app randomly failed (in my case a random spike but would be the same for memory) they did _not_ get restarted. I too was expecting they would have. > [@Any thoughts on why my app randomly died?](https://community.fly.io/t/any-thoughts-on-why-my-app-randomly-died/4427): > > Hello, I happened to notice one of my dev web apps (nodejs) stopped responding earlier today. I haven’t touched it for days. Last deploy a week ago, ish. It is behind Cloudflare’s orange-cloud proxy and so all requests to it from its domain currently return a 525. But as I’ve found in the past, that status code is misleading. It’s not an “SSL” error. That is just what Cloudflare returns if it can’t get a response back from the app. Sure enough: Instances ID PROCESS VERSION REGION DESIRE… The fix was simple: add a healthcheck to your `fly.toml` (tcp and/or http, depending on your app) _and_ make sure that has a `restart_limit` set. Since the default is 0. Which disables restarts: > - `restart_limit`: The number of consecutive TCP check failures to allow before attempting to restart the VM. The default is `0`, which disables restarts based on failed TCP health checks. See: > **[Fly Launch configuration (fly.toml)](https://fly.io/docs/reference/configuration/#services-tcp_checks)** > > Documentation and guides from the team at Fly.io. Upping the RAM to 4GB will also fix it. Just a more expensive way 🙂 ---
**Author:** ![davidhodge](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/davidhodge/32/9908_2.png) [@davidhodge](https://community.fly.io/u/davidhodge)\ **Post date:** [July 25, 2022, 9:25pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/3 "2022-07-25T21:25:53Z")
Really helpful, thanks Greg. I tried to investigate by triggering an OOM myself, and what I found was that the instance was restarted. And I hadn’t yet made a change to restart\_limit! So I’m still a bit puzzled. Perhaps they’ll restart failed instances if they fail health checks within a certain amount of time of boot up? Additionally, I’m having a bit of trouble making sense of how health checks work in the context where we have 1 instance vs how I would expect them to act when we have many instances as a part of an autoscaled service. With a multi-instance service, if there is an instance that is no longer responding, I’d think they’d stop sending traffic to it and replace it with a new instance. So why isn’t that automatic when there’s just one service. Surely I’m missing something, but this is my mental model at least. ---
**Author:** ![tj1](https://avatars.discourse-cdn.com/v4/letter/t/22d042/32.png) [@tj1](https://community.fly.io/u/tj1)\ **Post date:** [July 25, 2022, 9:33pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/4 "2022-07-25T21:33:39Z")
Yah, this is a bit confusing. The restart\_limit hint is good to know though. ---
**Author:** ![greg](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/greg/32/2119_2.png) [@greg](https://community.fly.io/u/greg)\ **Post date:** [July 25, 2022, 9:46pm UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/5 "2022-07-25T21:46:05Z")
Hmm … Well that’s surprising. If you haven’t specified a `restart_limit` _my_ understanding was that vms would not be restarted. It’s _possible_ that has been changed and the docs not updated to reflect it, or maybe there are circumstances where the stars align and it does happen regardless. Like you say, perhaps there is a time/age component to it. Given your vm showed as _pending_ it may point again to that actually _is_ happening, or at least tried to. Not sure. As regards the load balancer model, I guess again it comes back to what “should” happen and whether people’s expectations may differ. _Personally_ I would say the default should be to replace an instance that is not responding to a healthcheck. And yes, replace it. As an app with one instance that is now not available to handle requests … well, that’s not much use. That would be different to auto-scaling where the instance would be added or removed based on a load metric, rather than based on health. I was trying to think how AWS did it back when I had one instance behind an ELB … I vaguely recall when that instance failed (and hence failed the LB check on it), it too did not automatically get replaced/restarted by AWS. It needed an additional healthcheck to _tell_ it to. Which would explain Fly’s approach. ---
**Author:** ![wjordan](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/wjordan/32/1280_2.png) [@wjordan](https://community.fly.io/u/wjordan)\ **Post date:** [July 26, 2022, 5:19am UTC](https://community.fly.io/t/instance-or-service-not-restarted-when-i-expected-it-to/6155/6 "2022-07-26T05:19:41Z")
Hi @davidhodge, One of our hosts in the `sjc` region triggered a bug in Nomad (our VM-orchestration service) which was preventing some instances from transitioning correctly, which may have been why your instance was stuck in ‘pending’. We restarted the service on the affected host which unblocked the stuck instances. If this was the issue, things should be cleared up by now and let us know if you continue to see any unexpected behavior. As for the restart policy, note that the `restart_limit` setting only configures restarts triggered by _health-check failures_. Application-process crashes (including OOM-triggered exits) are triggered by a separate internal (not configurable) restart policy. The current policy \ will restart any exited processes up to 2 times within 5 minutes, then re-deploys the instance on another host if the process exits again. If the new deploy continues to fail, the instance will continue to get re-deployed indefinitely, with an exponential delay between 15 seconds - 15 minutes, capped at 15 restarts every 2 hours. This is all very tied to Nomad’s [built-in restart behavior](https://learn.hashicorp.com/tutorials/nomad/failures-restart), so the exact restart-policy details may change with Machine-based apps.