Unable to deploy on v2 - Timeout reached waiting for healthchecks

Questions / Help
1

Hi,

I’m unable to deploy my Rails app due to the following error:
Error: timeout reached waiting for healthchecks to pass for machine 3d8d314fe23418 failed to get VM 3d8d314fe23418: Get "https://api.machines.dev/v1/apps/bishbashbooked/machines/3d8d314fe23418": net/http: request canceled note: you can change this timeout with the --wait-timeout flag

I found this in the logs, but don’t really know what it means or if it’s relevant to this problem:
ERROR libinit::api_proxy: [fly api proxy] failed to start listener: Address in use (os error 98)

This is my fly.toml:

app = "bishbashbooked"
primary_region = "lhr"
kill_signal = "SIGINT"
kill_timeout = "5s"

[experimental]
  auto_rollback = true

[build]
  [build.args]
    BUILD_COMMAND = "bin/rails fly:build"
    SERVER_COMMAND = "bin/rails fly:server"

[deploy]
  release_command = "bin/rails fly:release"

[env]
  PORT = "8080"
  SERVER_COMMAND = "bin/rails fly:server"

[[services]]
  protocol = "tcp"
  internal_port = 8080
  processes = ["app"]

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]
  [services.concurrency]
    type = "connections"
    hard_limit = 25
    soft_limit = 20

  [[services.tcp_checks]]
    interval = "15s"
    timeout = "2s"
    grace_period = "1s"
    restart_limit = 0

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/"

Please let me know if you need any more info. Any help would be appreciated, thanks in advance.

2

I have tried deleting the builder and re-deploying but no luck. Also changed my fly.toml slightly:

[[services.tcp_checks]]
    interval = 10000
    timeout = 2000
    grace_period = "1m"

I am now also getting this error:

Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

Any ideas?

3

Hi,

The first place to check would be the logs. Does flyctl logs reveal any issues? You can often see the reason why the healthcheck has failed in there (undefined variable, could not connect to database … etc). The healthcheck should run every 10 seconds and so you shouldn’t have to wait long to see what code/message appears in that tail.

4

Thanks for the reply. It seems to be unable to start the rails server:


2023-06-13T20:30:57.920 app[5683dd6f157178] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.14.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

2023-06-13T20:30:57.920 app[5683dd6f157178] lhr [info] from bin/rails:5:in `<main>'

2023-06-13T20:30:57.935 app[5683dd6f157178] lhr [info] rails aborted!

2023-06-13T20:30:57.935 app[5683dd6f157178] lhr [info] Command failed with status (1): [bin/rails server...]

2023-06-13T20:30:57.935 app[5683dd6f157178] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/file_utils.rb:67:in `block in create_shell_runner'

I’m not sure why though because the app works locally, and the fly.toml/dockerfile has not been changed since the last successful deploy.

5

Strange :thinking:

Not sure, I’m afraid. There are lots of Rails people in this community so hopefully (armed with those log lines) someone will be able to assist.

6

I have no idea what’s going on either. If I delete the app and re-launch it, do you know if I can I link the database to the new app?

7

I would need to see more log lines that that. Try the Monitoring tab in the dashboard.

8

@harold You can but before doing anything like that, see what @rubys has to say. You’ve found the ideal person to assist.

9

@rubys I’ve tried to paste in as much as I can here:

2023-06-13T21:01:07.824 app[3287910a0126e8] lhr [info] bin/rails server

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: previous definition of ProtocRetryError was here

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: previous definition of BUFSIZE was here

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket

2023-06-13T21:01:08.474 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: previous definition of Socket was here

2023-06-13T21:01:08.897 app[3287910a0126e8] lhr [info] => Booting Puma

2023-06-13T21:01:08.897 app[3287910a0126e8] lhr [info] => Rails 7.0.4 application starting in production

2023-06-13T21:01:08.897 app[3287910a0126e8] lhr [info] => Run `bin/rails server --help` for more startup options

2023-06-13T21:01:09.037 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/will_paginate-3.1.7/lib/will_paginate/deprecation.rb:35: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead

2023-06-13T21:01:09.037 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/will_paginate-3.1.7/lib/will_paginate/deprecation.rb:35: warning: Capturing the given block using Proc.new is deprecated; use `&block` instead

2023-06-13T21:01:09.152 app[3287910a0126e8] lhr [info] [Warning] After upgrading RailsAdmin to 3.x you haven't set asset_source yet, using :sprockets as the default.

2023-06-13T21:01:09.152 app[3287910a0126e8] lhr [info] To suppress this message, run 'rails rails_admin:install' to setup the asset delivery method suitable to you.

2023-06-13T21:01:09.622 app[3287910a0126e8] lhr [info] Exiting

2023-06-13T21:01:09.622 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/activestorage-7.0.4/lib/active_storage/service/registry.rb:18:in `block in fetch': Missing configuration for the digitalocean Active Storage service. Configurations available for the test, local, and cloudinary services. (KeyError)

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activestorage-7.0.4/lib/active_storage/service/registry.rb:11:in `fetch'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activestorage-7.0.4/lib/active_storage/service/registry.rb:11:in `fetch'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activestorage-7.0.4/lib/active_storage/engine.rb:155:in `block (2 levels) in <class:Engine>'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:95:in `class_eval'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:95:in `block in execute_hook'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:85:in `with_execution_control'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:90:in `execute_hook'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:76:in `block in run_load_hooks'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:75:in `each'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activesupport-7.0.4/lib/active_support/lazy_load_hooks.rb:75:in `run_load_hooks'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/activestorage-7.0.4/app/models/active_storage/blob.rb:410:in `<main>'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:37:in `require'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:37:in `require'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.14.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/kernel.rb:30:in `require'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/helpers.rb:135:in `const_get'

2023-06-13T21:01:09.623 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/helpers.rb:135:in `cget'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:176:in `block in actual_eager_load_dir'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/helpers.rb:40:in `block in ls'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/helpers.rb:25:in `each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/helpers.rb:25:in `ls'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:164:in `actual_eager_load_dir'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:17:in `block (2 levels) in eager_load'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:16:in `each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:16:in `block in eager_load'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:10:in `synchronize'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader/eager_load.rb:10:in `eager_load'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader.rb:296:in `block in eager_load_all'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader.rb:294:in `each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.6.6/lib/zeitwerk/loader.rb:294:in `eager_load_all'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/application/finisher.rb:74:in `block in <module:Finisher>'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/initializable.rb:32:in `instance_exec'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/initializable.rb:32:in `run'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/initializable.rb:61:in `block in run_initializers'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:347:in `each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:347:in `call'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/initializable.rb:60:in `run_initializers'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/application.rb:372:in `initialize!'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/config/environment.rb:5:in `<main>'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from config.ru:3:in `require_relative'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from config.ru:3:in `block in <main>'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:116:in `eval'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:116:in `new_from_string'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:105:in `load_file'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/builder.rb:66:in `parse_file'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:349:in `build_app_and_options_from_config'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:249:in `app'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:422:in `wrapped_app'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:312:in `block in start'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:379:in `handle_profiling'

2023-06-13T21:01:09.624 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.4/lib/rack/server.rb:311:in `start'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:38:in `start'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:143:in `block in perform'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:134:in `tap'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:134:in `perform'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/command/base.rb:87:in `perform'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/command.rb:48:in `invoke'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:37:in `require'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:37:in `require'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.14.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

2023-06-13T21:01:09.625 app[3287910a0126e8] lhr [info] from bin/rails:5:in `<main>'

2023-06-13T21:01:09.642 app[3287910a0126e8] lhr [info] rails aborted!

2023-06-13T21:01:09.642 app[3287910a0126e8] lhr [info] Command failed with status (1): [bin/rails server...]

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/file_utils.rb:67:in `block in create_shell_runner'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/file_utils.rb:57:in `sh'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/lib/tasks/fly.rake:21:in `block (2 levels) in <main>'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:18:in `perform'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/command.rb:51:in `invoke'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.14.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] Tasks: TOP => fly:server

2023-06-13T21:01:09.643 app[3287910a0126e8] lhr [info] (See full trace by running task with --trace)

2023-06-13T21:01:09.770 app[3287910a0126e8] lhr [info] INFO Main child exited normally with code: 1

2023-06-13T21:01:09.770 app[3287910a0126e8] lhr [info] INFO Starting clean up.

2023-06-13T21:01:09.771 app[3287910a0126e8] lhr [info] WARN hallpass exited, pid: 515, status: signal: 15 (SIGTERM)

2023-06-13T21:01:09.771 app[3287910a0126e8] lhr [info] 2023/06/13 21:01:09 listening on [fdaa:0:9579:a7b:8e:b2c8:ceb8:2]:22 (DNS: [fdaa::3]:53)

2023-06-13T21:01:10.767 app[3287910a0126e8] lhr [info] [ 5.500617] reboot: Restarting system

2023-06-13T21:01:10.951 runner[3287910a0126e8] lhr [info] machine did not have a restart policy, defaulting to restart

2023-06-13T21:01:11.457 app[3287910a0126e8] lhr [info] INFO Starting init (commit: 0b28cec)...

2023-06-13T21:01:11.478 app[3287910a0126e8] lhr [info] INFO starting statics vsock server

2023-06-13T21:01:11.479 app[3287910a0126e8] lhr [info] INFO Preparing to run: `/bin/bash -c ${SERVER_COMMAND}` as root

2023-06-13T21:01:11.487 app[3287910a0126e8] lhr [info] ERROR [fly api proxy] failed to start listener: Address in use (os error 98)

2023-06-13T21:01:11.501 app[3287910a0126e8] lhr [info] 2023/06/13 21:01:11 listening on [fdaa:0:9579:a7b:8e:b2c8:ceb8:2]:22 (DNS: [fdaa::3]:53)

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:68: warning: previous definition of ProtocRetryError was here

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:208: warning: previous definition of BUFSIZE was here

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /usr/lib/fullstaq-ruby/versions/2.7.0-jemalloc/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket

2023-06-13T21:01:13.037 app[3287910a0126e8] lhr [info] /app/vendor/bundle/ruby/2.7.0/gems/net-protocol-0.1.3/lib/net/protocol.rb:504: warning: previous definition of Socket was here

2023-06-13T21:01:13.705 app[3287910a0126e8] lhr [info] bin/rails server
10

Is it possible that not all of your secrets migrated?

Check either using fly secrets list or using the dashboard.

11

I think it is a secrets issue, but I can’t set the correct secrets because the health checks don’t pass. Is there any way of setting secrets and skipping the health checks?

12

Try:

flyctl secrets set --stage NAME=VALUE
1 Like
13

@rubys I’ve got a few more issues now. If I deleted the app and launched it again, how could I attach the exisiting database to the new app?

Would this command work?

flyctl postgres attach --app <app-name> <postgres-app-name>
14

Fly postgres attach will create a new database and user which is fine… if that’s what you want. But I suspect that you want the current database and user.

Before deleting your app, try running:

fly ssh console -C printenv DATABASE_URL

Save that secret (and don’t share it) as it has your user and database info. For good measure, run the same command without DATABASE_URL and save the result someplace safe just in case you need another secret later.

If you now delete and re-create your app you can set this secret and you will be up and running. You won’t be able to run detach later, but that’s fine.

15

I think I’ve f-ed up here, I had a database url saved for my app on my computer from when I originally created the app. I then deleted the app and saved this into my app secrets as you suggested, but get this error when deploying:

Error release_command failed running on machine 148e121c7706d8 with exit code 1.
Check its logs: here's the last 100 lines below, or run 'fly logs -i 148e121c7706d8':
  - Has the database name changed? Check your database.yml config has the correct database name.
  To create your database, run:
          bin/rails db:create
  /app/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:81:in `rescue in new_client'
  /app/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'
  /app/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
  /app/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `public_send'
  /app/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:656:in `new_connection'
1.14.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
  Caused by:
  PG::ConnectionBad: FATAL:  password authentication failed for user "bishbashbooked"

This is saying I have the wrong password for the database. Is there a way of using pg_dump to backup the data and then restore it into a new db? Or another way to attach the old db to the new app? Sorry for giving you all this hassle!

Edit (my database.yml file):

production:
  url: <%= ENV['DATABASE_URL'] %>
16

I’m pretty sure that’s not the problem, see here: Configuring Rails Applications — Ruby on Rails Guides

Hang on while I try to find somebody who knows more about postgres to answer this question.

17

@rubys thanks for all the help, I have managed to backup my old database and upload in into a new one for the newly created app. For anyone that is as stupid as I am, these are the steps I followed:

  • Set up Wireguard on your local machine so that you can access your Fly database. Click this link for the tutorial.
  • Backup the database from Fly using the old database credentials and the following command:
pg_dump -h HOST -d $DATABASE -U $USER -p $PORT -W -F t > latest.dump
  • Type the password when prompted.
  • Find the database connection URL for your new Fly PG database, using the following commands:
fly ssh console
echo $DATABASE_URL
  • Restore the database using the following command:
pg_restore -v -d [FLY_POSTGRES_URL] < latest.dump
18

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.