I am annoyed I lost everything in my redis database when y’all automatically updated my app to v2.
Instead of shutting off my instance and losing my data, it would have been great if you properly migrated my database (via replication or even snapshots the RAM and disk).
I know the v2 was announced a few months ago, but I didn’t realize you would force the migration (and lose my data) if I didn’t migrate it myself.
Hi Kevin,
Can you give us more details as to what was lost? Both your original and cloned volume are still there - as long as your data was on those volumes you should be good to go!
If you’re seeing something different, let us know and we can probably figure out the issue.
I used redis for a sidekiq queue, following the documentation here:
GitHub - fly-apps/redis: Launch a Redis server on Fly almost 1 year ago.
On July 24th (2 days ago), I received an email saying FlyIO would:
Starting next week, we need to migrate some of your apps from our deprecated v1 platform to v2. We’ll be migrating the following apps:
It is July 26 (2 days after the “next week”) warning. I received an email this morning telling me the migration was complete and I verified that all data was lost from my instance.
I thought I had at least 5 days to verify my volumes were configured properly or manually migrate the instance myself.
I setup redis up over almost a year ago. At the time, I was trying to be low budget and was just using redis for short-lived tasks.
If the service rebooted, while not a great user experience, you get what you pay for.
It would have been nice if the email said, “Hey, we are about to reboot/restart your instance, resulting in full data loss because there are no volumes attached.”
While I admit, I should have configured the service for durability, I could have been warned when flyio was intentionally going to delete my data.
Got it, I can absolutely see how how frustrating it is to not get a warning, even if the service wasn’t configured with durability in mind as you said.
Most of Fly’s apps (like 95%) are now on machines, and we’re not planning on doing any more migrating any time soon, so this shouldn’t happen again.
What can we do to make this better for you? We can’t get the queue data back, but at least we can give you credits and/or extra support - I’m going to add some to your account right now.
I certainly would appreciate credit. I also don’t understand why the email told me I had 6+ days, when I actually only had less than 2 days?
If you don’t want to migrate your apps yourself, they will be migrated to v2 automatically over the next couple weeks. We’ll keep you updated on our progress so you know what apps are up for migration next.
This announcement didn’t provide clear deadlines when the migration would happen. The only warning I had for exactly when the change would go through was that email and the email lied about when it would happen.
How can you expect companies to trust flyio for their hosting if they give inaccurate and unclear warnings about migrations with “weeks” instead of “months” time horizons?
Problem 1: Email lied about when the upgrade would happen
Yea this is ridiculous. I got the email on the 25th and migrations started on the 26th.
Well you are both in luck! I’m the person who ran the jobs to send all the heads up emails and most of the migrations, so you can direct as many creative insults in my direction as you’d like 
In all seriousness, we’re a small team and we’re doing the best we can to move with forward without breaking anyone. I appreciate your patience.
I appreciate that you are a small team but this is not cool
What does the team size have to do with not honoring your commitments? I understand that “things change” but what changed so drastically that you didn’t even send out a warning email about the change before it happened?
Hi Kevin, I heard you loud and clear, we’ve long since fixed the emails, and I applied a $500 credit to your account last week as an apology. If there’s anything else you would like us to do to make this right, please let me know and I’ll try to make it happen.
Regardless of the migrations, you already know that Fly VMs are rebuilt on every restart. Any data stored in the root fs will be destroyed at reboot. I understand this was triggered by a migration, but there’s many others reasons a VM would restart that don’t come with a warning. We strongly suggest you (and anyone else building on Fly) use a volume going forward.
I totally agree that I messed up by not attaching volumes, which led to the unexpected instance restart. Definitely a mistake I wont make again.
Thanks a bunch for the credit! I accidentally ended up with two flyio accounts – one authenticated with GitHub (u:KevinColemanInc), which is used for hosting my company’s machines (where the impact occurred) and one with email/password. Both accounts should have the same email address.
Is there any chance you could move the credit to the github account? So sorry for the inconvenience. Thanks again.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.