Fly Postgres create with FORK option stopped working

,
1

Hello everyone,

I am not able to fork the postgres DB cluster we are using for our production environment.
We have been using the fork feature mainly in one of our Github Action workflows ( for provisioning staging environments on demand)
However, one day (friday) it suddenly stopped working. Here is the error message:


2024-04-08T06:38:10.333 runner[e82d620a7154e8] fra [info] Pulling container image docker-hub-mirror.fly.io/flyio/postgres-flex:15.6

2024-04-08T06:38:11.079 runner[e82d620a7154e8] fra [info] Successfully prepared image docker-hub-mirror.fly.io/flyio/postgres-flex:15.6 (745.590562ms)

2024-04-08T06:38:22.689 runner[e82d620a7154e8] fra [info] Configuring firecracker

2024-04-08T06:38:23.416 app[e82d620a7154e8] fra [info] [ 0.171197] PCI: Fatal: No config space access function found

2024-04-08T06:38:23.635 app[e82d620a7154e8] fra [info] INFO Starting init (commit: 5b8fb02)...

2024-04-08T06:38:23.709 app[e82d620a7154e8] fra [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755

2024-04-08T06:38:23.712 app[e82d620a7154e8] fra [info] [ 0.463873] EXT4-fs error (device vdb): ext4_get_journal_inode:5146: inode #8: comm init: iget: bad extra_isize 48447 (inode size 256)

2024-04-08T06:38:23.712 app[e82d620a7154e8] fra [info] [ 0.465092] EXT4-fs (vdb): no journal found

2024-04-08T06:38:23.713 app[e82d620a7154e8] fra [info] ERROR Error: couldn't mount /dev/vdb onto /data, because: EINVAL: Invalid argument

2024-04-08T06:38:23.714 app[e82d620a7154e8] fra [info] [ 0.466675] reboot: Restarting system

2024-04-08T06:38:23.824 app[e82d620a7154e8] fra [warn] Virtual machine exited abruptly

I also want to mention that the fly postgres create --fork-from <pg-fly-app> still runs with postgres fly application, but not anymore for our production.

anyone has a clue what this might be about and how to fix it?
Thanks.

2

I have managed to create a new postgres cluster from the same database using fly postgres create --snapshot-id <snapshot_id> and passing the latest snapshot of that volume.

But when I use fly postgres create --fork-from <postgres-db-app-name>:<vol_id> it fails with the error above. ( same with fly postgres create --fork-from <postgres-db-app-name> without the volume ID.

anyone has an idea what this might be about?

3

Got the same error since last Friday, April 5th. This is impacting our CI.

4

Like you did, I created a new postgres app from a previous snapshot I’m certain worked, and also got the same checksum errors:

2024-04-08T17:54:43Z runner[080e1e9c653d28] sea [info]Machine started in 1.304s
2024-04-08T17:54:43Z app[080e1e9c653d28] sea [info] INFO Main child exited normally with code: 2
2024-04-08T17:54:43Z app[080e1e9c653d28] sea [info] INFO Starting clean up.
2024-04-08T17:54:43Z app[080e1e9c653d28] sea [info] INFO Umounting /dev/vdb from /data
2024-04-08T17:54:43Z app[080e1e9c653d28] sea [info][    1.471278] reboot: Restarting system
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.147069] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.171018] PCI: Fatal: No config space access function found
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info] INFO Starting init (commit: 5b8fb02)...
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info] WARN could not resize filesystem to fill device size: Operation not permitted (os error 1)
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info] INFO [fly api proxy] listening at /.fly/api
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]2024/04/08 17:54:49 listening on [fdaa:4:e9ed:a7b:123:9b1a:2955:2]:22 (DNS: [fdaa::3]:53)
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.483007] EXT4-fs error (device vdb): htree_dirblock_to_tree:1072: inode #393217: comm chown: Directory block failed checksum
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.490383] EXT4-fs error (device vdb): htree_dirblock_to_tree:1072: inode #528385: comm chown: Directory block failed checksum
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.511244] EXT4-fs error (device vdb): ext4_dx_csum_verify:499: inode #528421: comm chown: dir seems corrupt?  Run e2fsck -D.
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info][    0.512523] EXT4-fs error (device vdb): dx_probe:818: inode #528421: comm chown: Directory index failed checksum
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]panic: failed to set directory ownership: exit status 1
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]goroutine 1 [running]:
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]main.panicHandler({0x9a7460?, 0xc000023790})
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]	/go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:188 +0x55
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]main.main()
2024-04-08T17:54:49Z app[080e1e9c653d28] sea [info]	/go/src/github.com/fly-apps/fly-postgres/cmd/start/main.go:65 +0xe5e
2024-04-08T17:54:50Z runner[080e1e9c653d28] sea [info]Machine started in 1.279s

I emailed Fly.io’s support and they said they’re aware and are working on it.

5

@eventinc-gmbh FYI this was fixed today

6

We are also tryng to fork a volume and redeploy on that, but we also got a similar error.
The context is that a server in AMS got deprecated and the volume could not be used anymore.
So we were suggested to fork and deploy on the forked volume, unfortunatly this looks impossible.
We are in contact with support.

 2024-04-10T13:20:11.248 app[2875371c029598] fra [info] INFO Starting init (commit: 5b8fb02)...
2024-04-10T13:20:11.317 app[2875371c029598] fra [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2024-04-10T13:20:11.317 app[2875371c029598] fra [info] [ 0.301671] EXT4-fs (vdb): VFS: Can't find ext4 filesystem
2024-04-10T13:20:11.318 app[2875371c029598] fra [info] ERROR Error: couldn't mount /dev/vdb onto /data, because: EINVAL: Invalid argument
2024-04-10T13:20:11.319 app[2875371c029598] fra [info] [ 0.303337] reboot: Restarting system
2024-04-10T13:20:11.385 app[2875371c029598] fra [warn] Virtual machine exited abruptly
2024-04-10T13:20:11.424 runner[2875371c029598] fra [info] machine exited with exit code 0, not restarting
2024-04-10T13:20:11.758 proxy[2875371c029598] fra [error] machines API returned an error: "machine exited abruptly"
2024-04-10T13:20:15.563 proxy[2875371c029598] fra [info] Starting machine
2024-04-10T13:20:15.754 app[2875371c029598] fra [info] [ 0.039401] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
2024-04-10T13:20:15.843 app[2875371c029598] fra [info] [ 0.055277] PCI: Fatal: No config space access function found
2024-04-10T13:20:16.025 app[2875371c029598] fra [info] INFO Starting init (commit: 5b8fb02)...
2024-04-10T13:20:16.097 app[2875371c029598] fra [info] INFO Mounting /dev/vdb at /data w/ uid: 0, gid: 0 and chmod 0755
2024-04-10T13:20:16.098 app[2875371c029598] fra [info] [ 0.309358] EXT4-fs (vdb): VFS: Can't find ext4 filesystem
2024-04-10T13:20:16.099 app[2875371c029598] fra [info] ERROR Error: couldn't mount /dev/vdb onto /data, because: EINVAL: Invalid argument
2024-04-10T13:20:16.100 app[2875371c029598] fra [info] [ 0.311025] reboot: Restarting system
2024-04-10T13:20:16.160 app[2875371c029598] fra [warn] Virtual machine exited abruptly
2024-04-10T13:20:16.199 runner[2875371c029598] fra [info] machine exited with exit code 0, not restarting
2024-04-10T13:20:16.564 proxy[2875371c029598] fra [error] machines API returned an error: "machine exited abruptly"
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.