Deleted inode reference in multiple databases

While deploying this morning we noticed two of our databases were dead (one since the weekend). A system reboot revealed the following errors in redis:

2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=537, just started
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # Configuration loaded
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.456 * monotonic clock: POSIX clock_gettime
2022-10-31T16:10:06.458 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.457 * Running mode=standalone, port=6379.
2022-10-31T16:10:06.458 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.458 # Server initialized
2022-10-31T16:10:06.459 app[9372dd40] ewr [info] [ 0.134625] EXT4-fs error (device vdc): ext4_lookup:1709: inode #2: comm redis-server: deleted inode referenced: 12
2022-10-31T16:10:06.459 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.459 # Fatal error loading the DB: No error information. Exiting.

and the following in psql:

2022-10-31T16:28:44.925 app[36bf5ac7] ewr [info] Starting init (commit: 249766e)...
2022-10-31T16:28:44.945 app[36bf5ac7] ewr [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-10-31T16:28:44.948 app[36bf5ac7] ewr [info] could not resize filesystem to fill device size: Operation not permitted (os error 1)
2022-10-31T16:28:44.957 app[36bf5ac7] ewr [info] Preparing to run: `docker-entrypoint.sh start` as root
2022-10-31T16:28:44.972 app[36bf5ac7] ewr [info] 2022/10/31 16:28:44 listening on [fdaa:0:6ee5:a7b:ab2:1:1fd3:2]:22 (DNS: [fdaa::3]:53)
2022-10-31T16:28:45.025 app[36bf5ac7] ewr [info] [ 0.185383] EXT4-fs error (device vdc): ext4_lookup:1709: inode #8193: comm chown: deleted inode referenced: 1537
2022-10-31T16:28:45.031 app[36bf5ac7] ewr [info] panic: exit status 1
2022-10-31T16:28:45.031 app[36bf5ac7] ewr [info] goroutine 1 [running]:
2022-10-31T16:28:45.965 app[36bf5ac7] ewr [info] Starting clean up.
2022-10-31T16:28:45.979 app[36bf5ac7] ewr [info] Umounting /dev/vdc from /data

Are there any steps we could follow to troubleshoot this, or is it related to a possible outage?

1 Like