Redis deploy failing

Questions / Help Build debugging
1

We had a running redis instance that appears to have crashed and can’t be restarted. It was deployed using

# fly.toml file generated for app-redis on 2022-06-14T19:19:34-04:00

app = "app-redis-dev"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  image = "flyio/redis:6.2.6"

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  [[mounts]]
    destination = "/data"
    source = "app_redis_dev"

About five months ago and had no issues. In the logs all we’re getting is:

2022-10-31T16:10:05.110 runner[9372dd40] ewr [info] Starting instance
2022-10-31T16:10:05.600 runner[9372dd40] ewr [info] Configuring virtual machine
2022-10-31T16:10:05.605 runner[9372dd40] ewr [info] Pulling container image
2022-10-31T16:10:05.734 runner[9372dd40] ewr [info] Unpacking image
2022-10-31T16:10:05.739 runner[9372dd40] ewr [info] Preparing kernel init
2022-10-31T16:10:05.998 runner[9372dd40] ewr [info] Setting up volume 'app_redis_dev'
2022-10-31T16:10:06.003 runner[9372dd40] ewr [info] Opening encrypted volume
2022-10-31T16:10:06.143 runner[9372dd40] ewr [info] Configuring firecracker
2022-10-31T16:10:06.159 runner[9372dd40] ewr [info] Starting virtual machine
2022-10-31T16:10:06.396 app[9372dd40] ewr [info] Starting init (commit: 249766e)...
2022-10-31T16:10:06.415 app[9372dd40] ewr [info] Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2022-10-31T16:10:06.417 app[9372dd40] ewr [info] could not resize filesystem to fill device size: Operation not permitted (os error 1)
2022-10-31T16:10:06.423 app[9372dd40] ewr [info] Preparing to run: `docker-entrypoint.sh /usr/bin/start-redis-server.sh` as root
2022-10-31T16:10:06.434 app[9372dd40] ewr [info] 2022/10/31 16:10:06 listening on [fdaa:0:6fc0:a7b:ab2:1:2413:2]:22 (DNS: [fdaa::3]:53)
2022-10-31T16:10:06.436 app[9372dd40] ewr [info] vm.overcommit_memory = 1
2022-10-31T16:10:06.436 app[9372dd40] ewr [info] net.core.somaxconn = 1024
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=537, just started
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:C 31 Oct 2022 16:10:06.456 # Configuration loaded
2022-10-31T16:10:06.456 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.456 * monotonic clock: POSIX clock_gettime
2022-10-31T16:10:06.458 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.457 * Running mode=standalone, port=6379.
2022-10-31T16:10:06.458 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.458 # Server initialized
2022-10-31T16:10:06.459 app[9372dd40] ewr [info] [ 0.134625] EXT4-fs error (device vdc): ext4_lookup:1709: inode #2: comm redis-server: deleted inode referenced: 12
2022-10-31T16:10:06.459 app[9372dd40] ewr [info] 537:M 31 Oct 2022 16:10:06.459 # Fatal error loading the DB: No error information. Exiting.
2022-10-31T16:10:07.429 app[9372dd40] ewr [info] Starting clean up.
2022-10-31T16:10:07.444 app[9372dd40] ewr [info] Umounting /dev/vdc from /data
1 Like
2

What you get when you call:

flyctl status -a app-redis-dev

3

do you create [mounts] in sub services ?

i think you need setup fly.toml like this

[[services]]
...
[mounts]
    destination = "/data"
    source = "app_redis_dev"
1 Like
4

@byku @bogordesain this ended up being related to the outage. we had to create a new redis volume from a snapshot of our pre-existing one before the outage this previous post helped