My region shouldn’t be affected, but somehow it’s impacting my app — my services are currently down because of it. How can I avoid this kind of issue in the future?
I’m running MySQL 5.6, and my apps can’t afford to be down for an hour.
Yikes, again… This looks like a problem with a single physical host machine, rather than the region as a whole.
(This is bad luck after the spate of I/O errors a couple weeks ago.)
What do you see in the output of fly m list -a i9-database (or the current name of the MySQL app)?
The could not umount /rootfs always happens, unfortunately. I.e., it’s not really an error.
The actual cause should be further up…
There is the log for error
[ERROR] InnoDB: Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint 8084523422754 and the end 8084525126144.
2025-05-16T17:55:25.663 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:25.662705Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163645Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163685Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163693Z 0 [ERROR] Failed to initialize builtin plugins.
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163696Z 0 [ERROR] Aborting
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163715Z 0 [Note] Binlog end
2025-05-16T17:55:26.163 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.163800Z 0 [Note] Shutting down plugin 'CSV'
2025-05-16T17:55:26.164 app[e286501dbd70d8] gru [info] 2025-05-16T17:55:26.164207Z 0 [Note] mysqld: Shutdown complete
This looks like disk corruption again, unfortunately…
And I’m not enough of a MySQL expert to say whether it could be nudged into recovering on its own at this point; it may be necessary to restore from a disk snapshot (+ rsync backup, if you have one).
This would mean some data loss, of course, but that’s unfortunately hard to avoid with single-Machine databases on the Fly.io platform, in general.
(Overall, they recommend that you always have ≥2 database Machines, in a cluster—although there are halfway points with streaming backups that some people find (mostly) tolerable.)
I want to restore data from the snapshot vs_nq7L5xQw82wHpQYQya94 to my new volume vol_vj5o8dy8k55xlnz4, it is possible? My app name is i9-mysql-prod
You can’t do that directly, but you can create a completely new volume from the snapshot. I think the easiest command would be the combined fly m clone --from-snapshot vs_nq7<long-string> e286501dbd70d8 -a i9-mysql-prod, which would give you a new volume attached to a new MySQL Machine. (I haven’t tried this exact variant myself, though.)
You can destroy the old, stopped Machine once you’re satisfied that everything is fully working again.
(If you’re using the .internal addresses, then the old one should stay asleep.)
1 Like
Thats solve my problem, thanks, I will work on automated backups. Thanks!
I’m pleased you’ve been able to restore your db; just adding a few thoughts.
If you’re really relying on a single host then hopefully it’s now obvious that situation needs repair. I’d probably add that if you aren’t a DBA, becoming competent enough to run your own cluster is a distraction from your core project, and it’d be better to switch to a managed database.
Also, it’s worth moving away from 5.6; that’s now ancient. Can you bump up to 8.4?
1 Like
It’s a high cost for my client, as we have hundreds of customers still using this version locally. We’ll need to upgrade sooner or later, but now isn’t the right time.
We have an ERP system, and I’ve already migrated parts of it—like the BI dashboard, mobile app, and new APIs—to the cloud. This setup has been running for about a year.
We’re planning to migrate to MySQL 8 in Q3, but we’re a small team—just me and my business partner (who now mostly handles legacy systems).