Is there an email list or something where changes like this get announced on? I haven’t had time to keep up and just got a bunch of emails saying things had migrated.
(They all migrated successfully - it was just an unexpected “hey something changed”).
Right now it’s just the forum. For what it’s worth, most product updates are announced in the “Fresh Produce” category of the forum, so if you subscribe to email notifications for that category it might be helpful (albeit a little noisy, we post a lot!)
If you want to do that, you can
Click this link to go view the “Fresh Produce” category on this forum
Click the subscription bell in the top right corner, and select “Watching First Post”.
Few of our apps have now been successfully converted to v2, looks good!
Are we the only ones having some problems with logging? For few apps, it seems app is doing allright, but on dashboard and/or flyctl logs there are only old v1 logs available.
@allison - it appears I have apps that have been automatically migrated to V2 overnight. However, I suspect one of them has failed the migration and is now in some sort of strange state (re: Fly’s back-end).
As far as I can tell (is this a pre-req to/part of the migration process?) the scale count has been reduced to 1, it was previously 2.
I noticed it was reported as suspended, so I restarted it. No dice - still suspended, although it appears it is actually up and working.
I’m not sure if suspended is a valid state for Fly-Nomad app status (is the Nomad equivalent dead)? The platform for the app is still reported as being nomad.
Logs for the app appear to have stopped since the (I assume) failed automated migration.
Though I appreciate the business and reputational imperative in force-migrating to people to Apps V2, in the interests of transparency I think it would be useful if Fly could create a new post documenting known feature shortfalls between the two (example from this thread: “bluegreen strategy, with health checks”).
Another example (TBC): with Nomad I believe it was possible, albeit subject to Fly-Nomad deficiencies (potential 15 minute delay/etc), to have a single always(read:mostly)-available instance. In the event of hardware failure Nomad would automatically move the VM to another host(?). I understand that machine instances are tied to hardware - I’m not sure if Apps V2 supports this use case? It would need two machines - but with Fly’s in-house orchestration ensuring only one is running at any point in time, and without reliance on external connections to trigger the proxy to bring up the 2nd/backup machine.
I’ll refrain from speculating too much, but it does sound like we tried to auto migrate your app and it failed.
For a little context, “suspended” is real old terminology from when you could suspend/resume nomad apps, and when machines was first built the “suspended” flag was overloaded to mean a machines app with no machines. During migration, at some point the app had no machines, so that flag got set. It just never got unset when things failed and it tried to restore your app to the previous state. You should be able to fly resume <appname> to get it back in working order.
We’ve seen a few people mention logs being strange after migration attempts. We’re looking into it.
Bluegreen is sadly not supported right now. It’s being looked into, but honestly, most of the people working on the apps platform are working on making sure this migration goes well right now. In the meantime, we do support canary deployments, which are pretty close!
As for having a single instance of an app that is relatively resilient, you might be looking for standby machines? Essentially, these are machines that are pointed at another machine, and turn on when their target machine is unreachable. Two caveats here, though: I don’t know what happens when/if the original machine comes back up, and they only get added for processes that do not expose a service.
And how do I scale down to 0 and back to 1 machine?
I simply run flyctl scale count 0 and then flyctl scale count 1, which immediately fails. With this message:
Error: there are no active machines for this app. Run fly deploy to create one and rerun this command
I tried runnin flyctl deploy this a couple of times, but every time I lose my machine settings, e.g. I have a machine with 512MB of RAM, scale down to 0, deploy, and it’s restarted on a machine with 256MB of RAM, which is not enough for startup and gets stuck until I scale the RAM back up manually.
Hmmm, I kinda feel like I’ve lost a bit though . I’ve been automatically upgraded, and now my deploys don’t work and it has taken my site down.
I’m getting a timeout Error for health checks. I’m running a Rails 7 app.
Just looking through the logs and this seems to be an issue with ActiveSupport.
Does anyone have any suggestions on how I could fix this?
=> Booting Puma
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Rails 7.0.4 application starting in production
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Run `bin/rails server --help` for more startup options
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] Exiting
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:209:in `rescue in _decrypt': ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:186:in `_decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:160:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/messages/rotator.rb:22:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:104:in `decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:66:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:21:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:33:in `config'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:48:in `options'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/core_ext/module/delegation.rb:303:in `method_missing'
@judel You’re probably deploying with a v1 fly.toml. If that’s the problem, you can run fly config save to pull in a current, v2-compatible fly.toml and then try deploying again.
There’s a known bug where an older copy of your RAILS_MASTER_KEY may have been set during migration. Can you try setting it again like: fly secrets set RAILS_MASTER_KEY=$(cat config/master.key)? This will ensure the value is the correct one.
Resetting the Rails MASTERKEY has worked Thank you! I always struggle with this type of stuff (devops/infrastructure). Where could I have found out about this known bug, or any others?
I also stopped receiving logs on my v2 app. But I think this is a broader issue. My colleague ran into an issue where logs stopped coming in on a v1 app.
Hopefully you can keep the documentation up to date. It still says that V2 apps do not support the canary strategy in several parts of the documentation. Will it be possible to make that the default deployment strategy? Just like they were in the v1 apps