Get In Losers*, We're Getting Off Nomad

You can kill one of the machines with fly m destroy and the next fly deploy will respect the machine count.

If you need to scale down to 0 and then scale up, run fly deploy --ha=false , that will launch only one machine if there aren’t any.

At last, we recommend to run 2 machines but enable autostart/autostop to reduce costs while keeping availability.

See Automatically starting/stopping Apps v2 instances and Setting a minimum number of instances to keep running when using auto start/stop

Hmmm, I kinda feel like I’ve lost a bit though :sweat_smile:. I’ve been automatically upgraded, and now my deploys don’t work and it has taken my site down.

I’m getting a timeout Error for health checks. I’m running a Rails 7 app.

Just looking through the logs and this seems to be an issue with ActiveSupport.

Does anyone have any suggestions on how I could fix this?

=> Booting Puma
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Rails 7.0.4 application starting in production
2023-05-25T02:58:55.100 app[9080526b60ee87] sin [info] => Run `bin/rails server --help` for more startup options
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] Exiting
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:209:in `rescue in _decrypt': ActiveSupport::MessageEncryptor::InvalidMessage (ActiveSupport::MessageEncryptor::InvalidMessage)
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:186:in `_decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:160:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/messages/rotator.rb:22:in `decrypt_and_verify'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:104:in `decrypt'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:66:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:21:in `read'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:33:in `config'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:48:in `options'
2023-05-25T02:58:55.135 app[9080526b60ee87] sin [info] from /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/core_ext/module/delegation.rb:303:in `method_missing'

@judel You’re probably deploying with a v1 fly.toml. If that’s the problem, you can run fly config save to pull in a current, v2-compatible fly.toml and then try deploying again.

Ran fly config save, and tried to deploy again, and I’m getting a similar error, but it’s now related to sidekiq.

This is my .toml file.

# fly.toml app configuration file generated for trade on 2023-05-25T08:19:37+04:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "trade"
kill_signal = "SIGINT"
kill_timeout = "5s"

[experimental]
  auto_rollback = true

[build]
  [build.args]
    BUILD_COMMAND = "bin/rails fly:build"
    SERVER_COMMAND = "bin/rails fly:server"

[processes]
  web = "bin/rails server"
  worker = "bundle exec sidekiq"

[[services]]
  protocol = "tcp"
  internal_port = 8080
  processes = ["web"]

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]
  [services.concurrency]
    type = "connections"
    hard_limit = 25
    soft_limit = 20

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "2s"
    grace_period = "1s"
    restart_limit = 0

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/"

It may be that your RAILS_MASTER_KEY secret is not set. If you don’t see it in fly secrets list, can you try setting it again?

Both of the secrets I set before are still there.
RAILS_MASTER_KEY
REDIS_URL

This is what I’m getting in the logs at the moment.

Updating existing machines in 'trade' with rolling strategy
  Machine 3287dd0a642585 [worker] has state: started
  [1/2] Checking that 3287dd0a642585 [worker] is up and running
Smoke checks for 3287dd0a642585 failed: the app appears to be crashing
Check its logs: here's the last lines below, or run 'fly logs -i 3287dd0a642585':
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:302:in `require'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:302:in `boot_application'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:42:in `run'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/bin/sidekiq:31:in `<top (required)>'
  /app/vendor/bundle/ruby/3.1.0/bin/sidekiq:25:in `load'
  /app/vendor/bundle/ruby/3.1.0/bin/sidekiq:25:in `<top (required)>'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `load'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:23:in `run'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:484:in `exec'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:31:in `dispatch'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:25:in `start'
  /usr/local/bundle/gems/bundler-2.3.7/exe/bundle:48:in `block in <top (required)>'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
  /usr/local/bundle/gems/bundler-2.3.7/exe/bundle:36:in `<top (required)>'
  /usr/local/bundle/bin/bundle:25:in `load'
  /usr/local/bundle/bin/bundle:25:in `<main>'
  Starting clean up.
  hallpass exited, pid: 514, status: signal: 15
  2023/05/25 04:23:19 listening on [fdaa:1:e40:a7b:81:6ead:f927:2]:22 (DNS: [fdaa::3]:53)
  [    4.159064] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
  Starting init (commit: 9bb7ee8)...
  Preparing to run: `bundle exec sidekiq` as root
  2023/05/25 04:23:21 listening on [fdaa:1:e40:a7b:81:6ead:f927:2]:22 (DNS: [fdaa::3]:53)
  ActiveSupport::MessageEncryptor::InvalidMessage
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:209:in `rescue in _decrypt'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:186:in `_decrypt'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/message_encryptor.rb:160:in `decrypt_and_verify'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/messages/rotator.rb:22:in `decrypt_and_verify'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:104:in `decrypt'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_file.rb:66:in `read'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:21:in `read'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:33:in `config'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/encrypted_configuration.rb:48:in `options'
  /app/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.4/lib/active_support/core_ext/module/delegation.rb:303:in `method_missing'
  /app/config/environments/production.rb:74:in `block in <main>'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/railtie.rb:257:in `instance_eval'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/railtie.rb:257:in `configure'
  /app/config/environments/production.rb:3:in `<main>'
  /app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
  /app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
  /app/vendor/bundle/ruby/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in `require'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/engine.rb:562:in `block (2 levels) in <class:Engine>'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/engine.rb:561:in `each'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/engine.rb:561:in `block in <class:Engine>'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:32:in `instance_exec'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:32:in `run'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:61:in `block in run_initializers'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:228:in `block in tsort_each'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:431:in `each_strongly_connected_component_from'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:50:in `each'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:50:in `tsort_each_child'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:415:in `call'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:415:in `each_strongly_connected_component_from'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:349:in `block in each_strongly_connected_component'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:347:in `each'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:347:in `call'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:347:in `each_strongly_connected_component'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:226:in `tsort_each'
  /usr/lib/fullstaq-ruby/versions/3.1.2-jemalloc/lib/ruby/3.1.0/tsort.rb:205:in `tsort_each'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/initializable.rb:60:in `run_initializers'
  /app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/application.rb:372:in `initialize!'
  /app/config/environment.rb:5:in `<top (required)>'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:302:in `require'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:302:in `boot_application'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/lib/sidekiq/cli.rb:42:in `run'
  /app/vendor/bundle/ruby/3.1.0/gems/sidekiq-7.0.3/bin/sidekiq:31:in `<top (required)>'
  /app/vendor/bundle/ruby/3.1.0/bin/sidekiq:25:in `load'
  /app/vendor/bundle/ruby/3.1.0/bin/sidekiq:25:in `<top (required)>'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `load'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli/exec.rb:23:in `run'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:484:in `exec'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:31:in `dispatch'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/cli.rb:25:in `start'
  /usr/local/bundle/gems/bundler-2.3.7/exe/bundle:48:in `block in <top (required)>'
  /usr/local/bundle/gems/bundler-2.3.7/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
  /usr/local/bundle/gems/bundler-2.3.7/exe/bundle:36:in `<top (required)>'
  /usr/local/bundle/bin/bundle:25:in `load'
  /usr/local/bundle/bin/bundle:25:in `<main>'
  Starting clean up.
  hallpass exited, pid: 514, status: signal: 15
  2023/05/25 04:23:24 listening on [fdaa:1:e40:a7b:81:6ead:f927:2]:22 (DNS: [fdaa::3]:53)
  [    4.139955] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
  Starting init (commit: 9bb7ee8)...
  Preparing to run: `bundle exec sidekiq` as root
  2023/05/25 04:23:25 listening on [fdaa:1:e40:a7b:81:6ead:f927:2]:22 (DNS: [fdaa::3]:53)
Error: smoke checks for 3287dd0a642585 failed: the app appears to be crashing

There’s a known bug where an older copy of your RAILS_MASTER_KEY may have been set during migration. Can you try setting it again like: fly secrets set RAILS_MASTER_KEY=$(cat config/master.key)? This will ensure the value is the correct one.

2 Likes

Not to be funny, but perhaps a smaller number of apps have been included in the migration.

Also, I would have appreciated more emails warning of this, as if there was one I missed it.

1 Like

Resetting the Rails MASTERKEY has worked Thank you! I always struggle with this type of stuff (devops/infrastructure). Where could I have found out about this known bug, or any others?

The only way we noticed that some env secrets were not correct was we had offsite backups and compared them.

1 Like

I also stopped receiving logs on my v2 app. But I think this is a broader issue. My colleague ran into an issue where logs stopped coming in on a v1 app.

1 Like

Hopefully you can keep the documentation up to date. It still says that V2 apps do not support the canary strategy in several parts of the documentation. Will it be possible to make that the default deployment strategy? Just like they were in the v1 apps

So, as somebody not reading all forum posts I was a little bit surprised by this migration. Apparently, 15-20 of our apps have been migrated over night. Since I did not hear any complaints, yet, I guess it worked fine.

So I went ahead to update our .toml files by running fly config save and I wanted to ask, if it is a known issue that [build.args] are not included in the saved file?

3 Likes

It’s been known that [build] is not stored remotely, but that behavior is particularly bad when paired with automatic migrations.
The latest release of flyctl offers to merge in the [build] section from an existing config when running config save, so that this doesn’t break things for anyone else. Thanks for bringing this to our attention!

Also wanted to share here few notes:

  • If you have experimental / enable_consul = true on apps v1 fly.toml, migration goes well but you have to run flyctl consul attach to maintain similar environment
  • On automatic migrations one thing which is probably just our own problem, but we have wrote few comments here and there and config save does not have those comments anymore. Just needs little manual work to merge those.
  • Also, if previously fly.toml had empty [services], it seems that migrate-to-v2 puts some default 8080 service there.

Apart from weird things happening with secrets, we are happy with v1->v2 migration for now.

2 Likes

Yep - we’re going to send some emails today to give folks more warning. We’re migrating in batches to make sure we can fix issues as we go and minimize impact :slight_smile:

rolling is the default for v2, but canary is now supported. we’ll make sure to update the docs.

1 Like

Thanks. What prevents the default from being canary, as it was before? Downtimes are higher now with the default value, people expect to have 0 downtime.

I tried fly config save and now I get this upon deploy:
Error error loading appv2 config: failed loading appv2 config from fly.toml: json: cannot unmarshal array into Go struct field Config.mounts of type appv2.Volume

What’s the solution for this?

I found the problem: the fly version.
Not as easy to fix as ti sounds. First of all, I never saw anywhere we were supposed to upgrade it.
Second, brew refuses to install it on my older MacOS, so I had to use the alternate method with curl.