# No Keeper Available, runaway WAL **URL:** \ **Category:** Questions / Help\ **Tags:** postgres\ **Created:** [September 14, 2022, 9:13am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153 "2022-09-14T09:13:07Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![LeoAdamek](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/leoadamek/32/39302_2.png) [@LeoAdamek](https://community.fly.io/u/LeoAdamek)\ **Post date:** [September 14, 2022, 9:13am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/1 "2022-09-14T09:13:07Z")
I have a postgres cluster which I created, scaled up to a couple of instances and then scaled back down to a single vm. However the vm is now reporting in the logs: > cmd/sentinel.go:276 no keeper info available This message appears every few seconds, there are also two replications slots which appear to be stale, causing a runway WAL which eventually led to a full disk. How do I recover this cluster so that there are no stale replication slots and the WAL is properly cleared? ---
**Author:** ![LeoAdamek](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/leoadamek/32/39302_2.png) [@LeoAdamek](https://community.fly.io/u/LeoAdamek)\ **Post date:** [September 14, 2022, 10:11am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/2 "2022-09-14T10:11:08Z")
I had to expand the disk to recover availability, so now my disk is also oversized and needs reducing. The WAL is growing rapidly and this isn’t sustainable. This seems to come up quite often on the forums and no one has provided an actual solution beyond “we fixed it now” Which is no good. ---
**Author:** ![LeoAdamek](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/leoadamek/32/39302_2.png) [@LeoAdamek](https://community.fly.io/u/LeoAdamek)\ **Post date:** [September 14, 2022, 10:49am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/3 "2022-09-14T10:49:33Z")
I figured out how to reduce the disk size: - Create a new volume of the desired size. - Scale up the DB cluster - Wait for it to become stable. - Stop the VMs on the old disk size and scale down the cluster. - Delete the old (oversized) volumes. ---
**Author:** ![LeoAdamek](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/leoadamek/32/39302_2.png) [@LeoAdamek](https://community.fly.io/u/LeoAdamek)\ **Post date:** [September 14, 2022, 10:50am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/4 "2022-09-14T10:50:53Z")
However there are still stale replica slots and dropping them just causes them to be re-created so this is very much time sensitive as the WAL will fill the disk again. ---
**Author:** ![LeoAdamek](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/leoadamek/32/39302_2.png) [@LeoAdamek](https://community.fly.io/u/LeoAdamek)\ **Post date:** [September 14, 2022, 1:03pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/5 "2022-09-14T13:03:08Z")
So after some faffing about I think I finally managed to resovle it. I used `fly ssh console` to access the db instance. I then spent a while figuring out what params I needed to pass into `stolonctl` to get the status: ``` stolonctl status --cluster-name $FLY_APP --store-backend consul --store-url $FLY_CONSUL_URL ``` Then I could identify and remove the unhealthy (Removed) keepers and use `stolonctl removekeeper` This appears to have resolved the issue but took an unnecessarily long time to figure out. ---
**Author:** ![iangcarroll](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/iangcarroll/32/15462_2.png) [@iangcarroll](https://community.fly.io/u/iangcarroll)\ **Post date:** [September 29, 2022, 9:30pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/6 "2022-09-29T21:30:58Z")
> [@LeoAdamek](#): > > Then I could identify and remove the unhealthy (Removed) keepers and use `stolonctl removekeeper` Thanks, I have been running into this too and this saved my cluster (only after scaling the volumes all the way to 100GB each). Hope Fly will fix this natively as this is a pretty sharp edge case. ---
**Author:** ![shaun](https://avatars.discourse-cdn.com/v4/letter/s/f07891/32.png) [@shaun](https://community.fly.io/u/shaun)\ **Post date:** [September 29, 2022, 9:54pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/7 "2022-09-29T21:54:21Z")
Here’s some additional information about this issue: [Reconfigure Stolon's "deadKeeperRemovalInterval · Issue #34 · fly-apps/postgres-ha · GitHub](https://github.com/fly-apps/postgres-ha/issues/34) ---
**Author:** ![iangcarroll](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/iangcarroll/32/15462_2.png) [@iangcarroll](https://community.fly.io/u/iangcarroll)\ **Post date:** [September 29, 2022, 10:34pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/8 "2022-09-29T22:34:38Z")
Thanks! I see 48 hours is mentioned in this issue but just want to note this impacted my app for weeks and did not go away on its own. It might be worth a doc update to the HA Postgres section, to note that scaling has this edge case right now. ---
**Author:** ![shaun](https://avatars.discourse-cdn.com/v4/letter/s/f07891/32.png) [@shaun](https://community.fly.io/u/shaun)\ **Post date:** [September 30, 2022, 1:07am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/9 "2022-09-30T01:07:28Z")
@iangcarroll Would you be able to let us know what image version you’re running? ```auto fly image show --app ``` ---
**Author:** ![iangcarroll](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/iangcarroll/32/15462_2.png) [@iangcarroll](https://community.fly.io/u/iangcarroll)\ **Post date:** [September 30, 2022, 1:25am UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/10 "2022-09-30T01:25:11Z")
@shaun ```auto Image Details Registry = registry-1.docker.io Repository = flyio/postgres Tag = 14.4 Version = v0.0.25 Digest = sha256:e60ad3f6f1e33c07d3b8a353df0243de5a27d6541961464023943bcae2eb080a ``` ---
**Author:** ![shaun](https://avatars.discourse-cdn.com/v4/letter/s/f07891/32.png) [@shaun](https://community.fly.io/u/shaun)\ **Post date:** [October 3, 2022, 8:12pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/11 "2022-10-03T20:12:27Z")
We discovered an issue with Stolon that was preventing failed keepers from getting cleaned up as expected. This issue has been patched and addressed with release `v0.0.29`. You can upgrade to the latest release via: `fly image update --app ` Also, as a side note: With this upgrade, you should also no longer need to export any additional environment variables in order to leverage `stolonctl` commands. If you have any questions on this, just let us know! cc:// @iangcarroll@LeoAdamek ---
**Author:** ![Berndinox](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/berndinox/32/8243_2.png) [@Berndinox](https://community.fly.io/u/Berndinox)\ **Post date:** [January 1, 2023, 5:39pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/12 "2023-01-01T17:39:35Z")
v0.0.33 - problem still exists… ```auto stolonctl status --cluster-name $FLY_APP --store-backend consul --store-url $FLY_CONSUL_URL nil cluster data: ``` The ENV Vars are set. ---
**Author:** ![armiiller](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/armiiller/32/18456_2.png) [@armiiller](https://community.fly.io/u/armiiller)\ **Post date:** [July 31, 2023, 6:20pm UTC](https://community.fly.io/t/no-keeper-available-runaway-wal/7153/13 "2023-07-31T18:20:44Z")
I can also confirm that on 0.0.41 this issue still exists. I have a [write up related to an incident](https://pagertree.com/blog/fly.io-migrate-to-v2-postgres-stuck-in-read-only-mode#runway-wal-log) about it. When you have the runaway log I found these commands helpful to kill the orphaned replica. ```auto # connect to postgres machine fly ssh console -a # set the environment variables export $(cat /data/.env | xargs) # check the stolon status stolonctl status # kill the runway stolon. stolonctl removekeeper ```