Hmmm, I kinda feel like I’ve lost a bit though . I’ve been automatically upgraded, and now my deploys don’t work and it has taken my site down.
I’m getting a timeout Error for health checks. I’m running a Rails 7 app.
Just looking through the logs and this seems to be an issue with ActiveSupport.
Does anyone have any suggestions on how I could fix this?
=> Booting Puma
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Rails 7.0.4 application starting in production
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Run `bin/rails server --help` for more startup options
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] Exiting
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:209:in `rescue in _decrypt': ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:186:in `_decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:160:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/messages/rotator.rb:22:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:104:in `decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:66:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:21:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:33:in `config'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:48:in `options'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/core_ext/module/delegation.rb:303:in `method_missing'
@judel You’re probably deploying with a v1 fly.toml. If that’s the problem, you can run fly config save to pull in a current, v2-compatible fly.toml and then try deploying again.
There’s a known bug where an older copy of your RAILS_MASTER_KEY may have been set during migration. Can you try setting it again like: fly secrets set RAILS_MASTER_KEY=$(cat config/master.key)? This will ensure the value is the correct one.
Resetting the Rails MASTERKEY has worked Thank you! I always struggle with this type of stuff (devops/infrastructure). Where could I have found out about this known bug, or any others?
I also stopped receiving logs on my v2 app. But I think this is a broader issue. My colleague ran into an issue where logs stopped coming in on a v1 app.
Hopefully you can keep the documentation up to date. It still says that V2 apps do not support the canary strategy in several parts of the documentation. Will it be possible to make that the default deployment strategy? Just like they were in the v1 apps
So, as somebody not reading all forum posts I was a little bit surprised by this migration. Apparently, 15-20 of our apps have been migrated over night. Since I did not hear any complaints, yet, I guess it worked fine.
So I went ahead to update our .toml files by running fly config save and I wanted to ask, if it is a known issue that [build.args] are not included in the saved file?
It’s been known that [build] is not stored remotely, but that behavior is particularly bad when paired with automatic migrations.
The latest release of flyctl offers to merge in the [build] section from an existing config when running config save, so that this doesn’t break things for anyone else. Thanks for bringing this to our attention!
If you have experimental / enable_consul = true on apps v1 fly.toml, migration goes well but you have to run flyctl consul attach to maintain similar environment
On automatic migrations one thing which is probably just our own problem, but we have wrote few comments here and there and config save does not have those comments anymore. Just needs little manual work to merge those.
Also, if previously fly.toml had empty [services], it seems that migrate-to-v2 puts some default 8080 service there.
Apart from weird things happening with secrets, we are happy with v1->v2 migration for now.
Yep - we’re going to send some emails today to give folks more warning. We’re migrating in batches to make sure we can fix issues as we go and minimize impact
Thanks. What prevents the default from being canary, as it was before? Downtimes are higher now with the default value, people expect to have 0 downtime.
I tried fly config save and now I get this upon deploy:
Error error loading appv2 config: failed loading appv2 config from fly.toml: json: cannot unmarshal array into Go struct field Config.mounts of type appv2.Volume
I found the problem: the fly version.
Not as easy to fix as ti sounds. First of all, I never saw anywhere we were supposed to upgrade it.
Second, brew refuses to install it on my older MacOS, so I had to use the alternate method with curl.