Volume Pending to destroy

Mine07 · September 26, 2024, 6:09pm

Hi, today after the outage on some servers (QRO, MX) I checked the volumes on my db app and noticed that one, the one I think has all the info stored (vol id vol_q4q800w3xnx6o984) is marked as pending_destroy, does that mean that the volume is indeed being destroyed?

I tried to clone/fork/restore snapshot into another volume but all them created empty (I guess??)

Does that mean that once (if it does) the volumes ends up destroyed im losing all my data?
Tried to download it but since the machine wont start I cant access it trough sftp.

Is any way I can retrieve it? (tried to connect with my sql but well its down lol)

ben-ang · September 26, 2024, 7:07pm

Hi @Mine07 , I’m looking into this now and will update here with further information.

In the meanwhile, am I understanding correctly that cloning/forking from the volume is failing, and also restoring from snapshots? Could you try the steps in Manage volume snapshots · Fly Docs if you haven’t already, or tell me what output you’re seeing if you have?

Mine07 · September 26, 2024, 7:20pm

Hi, thx Ben, yea already tried that but all new volumes end up being created with a size of 0, also tried to create a new machine with the volume attached but It says the volume is not detected, also if I run fly volume listed the volume is not there, i guess because the status, but if I run fly volume <volume_id> the vol is shown there.

ben-ang · September 26, 2024, 7:41pm

I’m still investigating why snapshot restores don’t seem to be working as expected here, but I’ve recovered your volume from pending_destroy so your data is not in danger.

ben-ang · September 26, 2024, 7:48pm

@Mine07 from the backend, it looks like the volumes that your machines are currently attached to do have data, but that this is somehow not being propagated to the dashboard UI. Would you be able to try starting those machines and tell me what happens?

Mine07 · September 26, 2024, 7:53pm

Looks the maching is trying to start up but failing to do so:

2024-09-26T19:51:03.372 app[080e395c100d48] qro [info] [ 0.305016] PCI: Fatal: No config space access function found

2024-09-26T19:51:03.728 app[080e395c100d48] qro [info] INFO Starting init (commit: 8cdad48f)...

2024-09-26T19:51:03.826 app[080e395c100d48] qro [info] INFO Checking filesystem on /data

2024-09-26T19:51:03.829 app[080e395c100d48] qro [info] /dev/vdc: clean, 193/655360 files, 113280/2617344 blocks

2024-09-26T19:51:03.830 app[080e395c100d48] qro [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755

2024-09-26T19:51:03.831 app[080e395c100d48] qro [info] INFO Resized /data to 10720641024 bytes

2024-09-26T19:51:03.834 app[080e395c100d48] qro [info] INFO Preparing to run: `docker-entrypoint.sh --datadir /data/mysql --default-authentication-plugin mysql_native_password` as root

2024-09-26T19:51:03.840 app[080e395c100d48] qro [info] INFO [fly api proxy] listening at /.fly/api

2024-09-26T19:51:03.843 app[080e395c100d48] qro [info] 2024/09/26 19:51:03 INFO SSH listening listen_address=[fdaa:7:a954:a7b:1ab:9a58:be4a:2]:22 dns_server=[fdaa::3]:53

2024-09-26T19:51:03.868 app[080e395c100d48] qro [info] 2024-09-26 19:51:03+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:03.893 runner[080e395c100d48] qro [info] Machine started in 953ms

2024-09-26T19:51:04.218 app[080e395c100d48] qro [info] 2024-09-26 19:51:04+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2024-09-26T19:51:04.236 app[080e395c100d48] qro [info] 2024-09-26 19:51:04+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:04.439 app[080e395c100d48] qro [info] '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'

2024-09-26T19:51:04.795 app[080e395c100d48] qro [info] 2024-09-26T19:51:04.451095Z 0 [System] [MY-015015] [Server] MySQL Server - start.

2024-09-26T19:51:04.795 app[080e395c100d48] qro [info] 2024-09-26T19:51:04.791506Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.4.2) starting as process 331

2024-09-26T19:51:04.802 app[080e395c100d48] qro [info] 2024-09-26T19:51:04.802536Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2024-09-26T19:51:05.168 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.167204Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2024-09-26T19:51:05.301 app[080e395c100d48] qro [info] mysqld: Table 'mysql.plugin' doesn't exist

2024-09-26T19:51:05.301 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.301677Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.

2024-09-26T19:51:05.302 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.302599Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.302 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.302807Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.303 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.302996Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.303 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.303667Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.303 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.303836Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.304 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.304001Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.304 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.304204Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.353 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.352913Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:05.364 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.364405Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:05.371 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.371017Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2024-09-26T19:51:05.371 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.371117Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.

2024-09-26T19:51:05.374 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.374430Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.

2024-09-26T19:51:05.374 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.374791Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:05.375 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.375004Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist

2024-09-26T19:51:05.375 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.375069Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.

2024-09-26T19:51:05.375 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.375205Z 0 [ERROR] [MY-000068] [Server] unknown option '--default-authentication-plugin'.

2024-09-26T19:51:05.376 app[080e395c100d48] qro [info] 2024-09-26T19:51:05.376211Z 0 [ERROR] [MY-010119] [Server] Aborting

2024-09-26T19:51:06.897 app[080e395c100d48] qro [info] 2024-09-26T19:51:06.897014Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.4.2) MySQL Community Server - GPL.

2024-09-26T19:51:06.897 app[080e395c100d48] qro [info] 2024-09-26T19:51:06.897098Z 0 [System] [MY-015016] [Server] MySQL Server - end.

2024-09-26T19:51:07.843 app[080e395c100d48] qro [info] INFO Main child exited normally with code: 1

2024-09-26T19:51:07.857 app[080e395c100d48] qro [info] INFO Starting clean up.

2024-09-26T19:51:07.858 app[080e395c100d48] qro [info] INFO Umounting /dev/vdc from /data

2024-09-26T19:51:07.860 app[080e395c100d48] qro [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2024-09-26T19:51:07.861 app[080e395c100d48] qro [info] [ 4.792994] reboot: Restarting system

2024-09-26T19:51:08.460 app[080e395c100d48] qro [info] 2024-09-26T19:51:08.460640002 [01J8QNC1003M8YJQCNSARPDR37:main] Running Firecracker v1.7.0

2024-09-26T19:51:08.822 app[080e395c100d48] qro [info] [ 0.303432] PCI: Fatal: No config space access function found

2024-09-26T19:51:09.174 app[080e395c100d48] qro [info] INFO Starting init (commit: 8cdad48f)...

2024-09-26T19:51:09.275 app[080e395c100d48] qro [info] INFO Checking filesystem on /data

2024-09-26T19:51:09.279 app[080e395c100d48] qro [info] /dev/vdc: clean, 194/655360 files, 113281/2617344 blocks

2024-09-26T19:51:09.280 app[080e395c100d48] qro [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755

2024-09-26T19:51:09.281 app[080e395c100d48] qro [info] INFO Resized /data to 10720641024 bytes

2024-09-26T19:51:09.283 app[080e395c100d48] qro [info] INFO Preparing to run: `docker-entrypoint.sh --datadir /data/mysql --default-authentication-plugin mysql_native_password` as root

2024-09-26T19:51:09.289 app[080e395c100d48] qro [info] INFO [fly api proxy] listening at /.fly/api

2024-09-26T19:51:09.293 app[080e395c100d48] qro [info] 2024/09/26 19:51:09 INFO SSH listening listen_address=[fdaa:7:a954:a7b:1ab:9a58:be4a:2]:22 dns_server=[fdaa::3]:53

2024-09-26T19:51:09.315 app[080e395c100d48] qro [info] 2024-09-26 19:51:09+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:09.327 runner[080e395c100d48] qro [info] Machine started in 936ms

2024-09-26T19:51:09.652 app[080e395c100d48] qro [info] 2024-09-26 19:51:09+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2024-09-26T19:51:09.670 app[080e395c100d48] qro [info] 2024-09-26 19:51:09+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:09.885 app[080e395c100d48] qro [info] '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'

2024-09-26T19:51:10.242 app[080e395c100d48] qro [info] 2024-09-26T19:51:09.897849Z 0 [System] [MY-015015] [Server] MySQL Server - start.

2024-09-26T19:51:10.242 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.237855Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.4.2) starting as process 331

2024-09-26T19:51:10.250 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.249921Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2024-09-26T19:51:10.625 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.624206Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2024-09-26T19:51:10.755 app[080e395c100d48] qro [info] mysqld: Table 'mysql.plugin' doesn't exist

2024-09-26T19:51:10.755 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.755621Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.

2024-09-26T19:51:10.756 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.756631Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.756 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.756839Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.757 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.757025Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.757 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.757673Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.757 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.757840Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.758 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.758006Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.758 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.758173Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.806 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.806135Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:10.816 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.816428Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:10.823 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.822987Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2024-09-26T19:51:10.823 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.823063Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.

2024-09-26T19:51:10.826 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.826339Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.

2024-09-26T19:51:10.826 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.826671Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:10.826 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.826876Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist

2024-09-26T19:51:10.827 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.826941Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.

2024-09-26T19:51:10.827 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.827063Z 0 [ERROR] [MY-000068] [Server] unknown option '--default-authentication-plugin'.

2024-09-26T19:51:10.827 app[080e395c100d48] qro [info] 2024-09-26T19:51:10.827663Z 0 [ERROR] [MY-010119] [Server] Aborting

2024-09-26T19:51:12.350 app[080e395c100d48] qro [info] 2024-09-26T19:51:12.350175Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.4.2) MySQL Community Server - GPL.

2024-09-26T19:51:12.350 app[080e395c100d48] qro [info] 2024-09-26T19:51:12.350253Z 0 [System] [MY-015016] [Server] MySQL Server - end.

2024-09-26T19:51:13.294 app[080e395c100d48] qro [info] INFO Main child exited normally with code: 1

2024-09-26T19:51:13.306 app[080e395c100d48] qro [info] INFO Starting clean up.

2024-09-26T19:51:13.307 app[080e395c100d48] qro [info] INFO Umounting /dev/vdc from /data

2024-09-26T19:51:13.308 app[080e395c100d48] qro [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2024-09-26T19:51:13.309 app[080e395c100d48] qro [info] [ 4.789532] reboot: Restarting system

2024-09-26T19:51:17.192 app[080e395c100d48] qro [info] 2024-09-26T19:51:17.192581698 [01J8QNC1003M8YJQCNSARPDR37:main] Running Firecracker v1.7.0

2024-09-26T19:51:17.555 app[080e395c100d48] qro [info] [ 0.302349] PCI: Fatal: No config space access function found

2024-09-26T19:51:17.900 app[080e395c100d48] qro [info] INFO Starting init (commit: 8cdad48f)...

2024-09-26T19:51:18.003 app[080e395c100d48] qro [info] INFO Checking filesystem on /data

2024-09-26T19:51:18.006 app[080e395c100d48] qro [info] /dev/vdc: clean, 195/655360 files, 113282/2617344 blocks

2024-09-26T19:51:18.007 app[080e395c100d48] qro [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755

2024-09-26T19:51:18.008 app[080e395c100d48] qro [info] INFO Resized /data to 10720641024 bytes

2024-09-26T19:51:18.011 app[080e395c100d48] qro [info] INFO Preparing to run: `docker-entrypoint.sh --datadir /data/mysql --default-authentication-plugin mysql_native_password` as root

2024-09-26T19:51:18.016 app[080e395c100d48] qro [info] INFO [fly api proxy] listening at /.fly/api

2024-09-26T19:51:18.021 app[080e395c100d48] qro [info] 2024/09/26 19:51:18 INFO SSH listening listen_address=[fdaa:7:a954:a7b:1ab:9a58:be4a:2]:22 dns_server=[fdaa::3]:53

2024-09-26T19:51:18.044 runner[080e395c100d48] qro [info] Machine started in 927ms

2024-09-26T19:51:18.045 app[080e395c100d48] qro [info] 2024-09-26 19:51:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:18.384 app[080e395c100d48] qro [info] 2024-09-26 19:51:18+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2024-09-26T19:51:18.400 app[080e395c100d48] qro [info] 2024-09-26 19:51:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:18.609 app[080e395c100d48] qro [info] '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'

2024-09-26T19:51:18.977 app[080e395c100d48] qro [info] 2024-09-26T19:51:18.620986Z 0 [System] [MY-015015] [Server] MySQL Server - start.

2024-09-26T19:51:18.977 app[080e395c100d48] qro [info] 2024-09-26T19:51:18.973209Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.4.2) starting as process 331

2024-09-26T19:51:18.984 app[080e395c100d48] qro [info] 2024-09-26T19:51:18.984362Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2024-09-26T19:51:19.388 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.386872Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2024-09-26T19:51:19.529 app[080e395c100d48] qro [info] mysqld: Table 'mysql.plugin' doesn't exist

2024-09-26T19:51:19.529 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.529116Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.

2024-09-26T19:51:19.530 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.530085Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.530 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.530314Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.530 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.530513Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.531 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.531168Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.531 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.531336Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.531 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.531502Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.531 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.531686Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.583 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.582902Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:19.645 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.645321Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:19.652 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.651978Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2024-09-26T19:51:19.652 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.652039Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.

2024-09-26T19:51:19.655 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.655405Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.

2024-09-26T19:51:19.655 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.655728Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:19.656 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.655952Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist

2024-09-26T19:51:19.656 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.656029Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.

2024-09-26T19:51:19.656 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.656155Z 0 [ERROR] [MY-000068] [Server] unknown option '--default-authentication-plugin'.

2024-09-26T19:51:19.656 app[080e395c100d48] qro [info] 2024-09-26T19:51:19.656826Z 0 [ERROR] [MY-010119] [Server] Aborting

2024-09-26T19:51:21.184 app[080e395c100d48] qro [info] 2024-09-26T19:51:21.184072Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.4.2) MySQL Community Server - GPL.

2024-09-26T19:51:21.184 app[080e395c100d48] qro [info] 2024-09-26T19:51:21.184157Z 0 [System] [MY-015016] [Server] MySQL Server - end.

2024-09-26T19:51:22.021 app[080e395c100d48] qro [info] INFO Main child exited normally with code: 1

2024-09-26T19:51:22.035 app[080e395c100d48] qro [info] INFO Starting clean up.

2024-09-26T19:51:22.037 app[080e395c100d48] qro [info] INFO Umounting /dev/vdc from /data

2024-09-26T19:51:22.038 app[080e395c100d48] qro [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2024-09-26T19:51:22.039 app[080e395c100d48] qro [info] [ 4.786673] reboot: Restarting system

2024-09-26T19:51:27.726 app[080e395c100d48] qro [info] 2024-09-26T19:51:27.726916193 [01J8QNC1003M8YJQCNSARPDR37:main] Running Firecracker v1.7.0

2024-09-26T19:51:28.087 app[080e395c100d48] qro [info] [ 0.302802] PCI: Fatal: No config space access function found

2024-09-26T19:51:28.435 app[080e395c100d48] qro [info] INFO Starting init (commit: 8cdad48f)...

2024-09-26T19:51:28.535 app[080e395c100d48] qro [info] INFO Checking filesystem on /data

2024-09-26T19:51:28.538 app[080e395c100d48] qro [info] /dev/vdc: clean, 196/655360 files, 113283/2617344 blocks

2024-09-26T19:51:28.538 app[080e395c100d48] qro [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755

2024-09-26T19:51:28.540 app[080e395c100d48] qro [info] INFO Resized /data to 10720641024 bytes

2024-09-26T19:51:28.542 app[080e395c100d48] qro [info] INFO Preparing to run: `docker-entrypoint.sh --datadir /data/mysql --default-authentication-plugin mysql_native_password` as root

2024-09-26T19:51:28.547 app[080e395c100d48] qro [info] INFO [fly api proxy] listening at /.fly/api

2024-09-26T19:51:28.551 app[080e395c100d48] qro [info] 2024/09/26 19:51:28 INFO SSH listening listen_address=[fdaa:7:a954:a7b:1ab:9a58:be4a:2]:22 dns_server=[fdaa::3]:53

2024-09-26T19:51:28.572 app[080e395c100d48] qro [info] 2024-09-26 19:51:28+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:28.581 runner[080e395c100d48] qro [info] Machine started in 921ms

2024-09-26T19:51:28.905 app[080e395c100d48] qro [info] 2024-09-26 19:51:28+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2024-09-26T19:51:28.923 app[080e395c100d48] qro [info] 2024-09-26 19:51:28+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.4.2-1.el9 started.

2024-09-26T19:51:29.131 app[080e395c100d48] qro [info] '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'

2024-09-26T19:51:29.495 app[080e395c100d48] qro [info] 2024-09-26T19:51:29.142745Z 0 [System] [MY-015015] [Server] MySQL Server - start.

2024-09-26T19:51:29.495 app[080e395c100d48] qro [info] 2024-09-26T19:51:29.491017Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.4.2) starting as process 332

2024-09-26T19:51:29.502 app[080e395c100d48] qro [info] 2024-09-26T19:51:29.502429Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

2024-09-26T19:51:29.913 app[080e395c100d48] qro [info] 2024-09-26T19:51:29.911954Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.

2024-09-26T19:51:30.049 app[080e395c100d48] qro [info] mysqld: Table 'mysql.plugin' doesn't exist

2024-09-26T19:51:30.050 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.049934Z 0 [ERROR] [MY-010735] [Server] Could not open the mysql.plugin table. Please perform the MySQL upgrade procedure.

2024-09-26T19:51:30.051 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.051259Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.051 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.051482Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.051 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.051693Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.052 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.052416Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.052 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.052592Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.052 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.052771Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.053 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.052986Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.105 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.105010Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:30.115 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.115465Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.

2024-09-26T19:51:30.122 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.122204Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.

2024-09-26T19:51:30.122 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.122294Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.

2024-09-26T19:51:30.126 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.126063Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.

2024-09-26T19:51:30.126 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.126474Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables

2024-09-26T19:51:30.126 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.126695Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist

2024-09-26T19:51:30.126 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.126742Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.

2024-09-26T19:51:30.126 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.126888Z 0 [ERROR] [MY-000068] [Server] unknown option '--default-authentication-plugin'.

2024-09-26T19:51:30.127 app[080e395c100d48] qro [info] 2024-09-26T19:51:30.127559Z 0 [ERROR] [MY-010119] [Server] Aborting

2024-09-26T19:51:31.648 app[080e395c100d48] qro [info] 2024-09-26T19:51:31.648024Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.4.2) MySQL Community Server - GPL.

2024-09-26T19:51:31.648 app[080e395c100d48] qro [info] 2024-09-26T19:51:31.648105Z 0 [System] [MY-015016] [Server] MySQL Server - end.

2024-09-26T19:51:32.551 app[080e395c100d48] qro [info] INFO Main child exited normally with code: 1

2024-09-26T19:51:32.565 app[080e395c100d48] qro [info] INFO Starting clean up.

2024-09-26T19:51:32.566 app[080e395c100d48] qro [info] INFO Umounting /dev/vdc from /data

2024-09-26T19:51:32.568 app[080e395c100d48] qro [info] WARN could not unmount /rootfs: EINVAL: Invalid argument

2024-09-26T19:51:32.568 app[080e395c100d48] qro [info] [ 4.783670] reboot: Restarting system

ben-ang · September 26, 2024, 9:11pm

I’m going to have to step away from this for now unfortunately; I haven’t been able to determine if this is a platform issue or something to do with your app configuration, but as far as we can tell, it is not a volumes issue. The original volume will be left intact until we can sort this out.

Mine07 · September 26, 2024, 9:15pm

Okk thx, in this case, should I try to create a new machine, install mysql and try to setup a backup?

ben-ang · September 26, 2024, 9:34pm

Yes, that might be worth a try

Mine07 · September 26, 2024, 10:35pm

Ok I managed to take the machines up and running, changed the proccess on fly.toml to this:

# Uncomment and use the following for 8.4:
# [processes]
#  app = """--datadir /data/mysql \
#    --mysql-native-password=ON"""

I think at some point the machines updated mysql or smth like that? anyway its running now BUT I cannot attach the original volume to an existing machine, I cant find documentation about that plus I when I try to create a new machine with that volume attached to It I get the next error:

insufficient resources to create new machine with existing volume 'vol_q4q800w3xnx6o984' (Request ID: 01J8R7A0RAPN08ACNNXAT8HRX2-qro)

Apparently the region is full now? Any suggestion what can I do to get that specific volume up and running? The one machine I have running with an attached volume has an empty database.

Mine07 · September 27, 2024, 3:39am

Hi @ben-ang, I have been trying all day to restore the volume in a new machine without succes, I mean, I can create a new machine and mount the a fork of the original volume on it and run mysql, but the database is empty.

I can see the original database in data/mysql using ssh, but apparently mysql is being installed on var/lib/mysql but on my original volume its in data/mysql, so I guess thats why I cant access my data on a new machine.

Im using the following fly.toml file

[build]
  image = 'mysql:8.0.3'

[[vm]]
  cpu_kind = 'shared'
  cpus = 1
  memory_mb = 2048

# The [processes] section is different for 8+, 8.4, and 5.7. Use the one that matches your version.
# Use the following for versions 8 to 8.3:
[processes]
  app = """--datadir /data/mysql \
    --default-authentication-plugin mysql_native_password"""

[mounts]
  source = "mysqldata"
  destination = "/data"

[env]
  MYSQL_DATABASE = "@db@"
  MYSQL_USER = "@user@"

Wich apparently should point mysql to --datadir /data/mysql but Its not working, any idea how can I restore the volume or retrieve the original data?

Mine07 · September 27, 2024, 2:25pm

Ok so I could fix this, in case someone is in the same predicament, what I think happened is that at some point the machine upgraded from mysql 8.3 to 8.4 since my .toml file asks for a mysql:8 img and it kinda broke the up to date volume,
what I had to do is to take a 1 day old snapshot of my volume, and mount it in a machine specifying for mysql 8.3 and it worked without issues.

mayailurus · October 1, 2024, 12:05pm

Added mysql, volumes

Topic		Replies	Views
App stuck in Pending, Volume "Migrated", maybe for a week already	1	214	November 4, 2022
Volume stuck in pending_destroy for 11 hours	4	633	August 24, 2023
Volume Restores are now available!	13	1822	January 30, 2023
Volume stuck in "migrated" status, cannot delete.	5	317	April 18, 2022
volumes disappeared after hardware failure volumes	9	289	May 21, 2024

Volume Pending to destroy

Related Topics