After moving organizations, the 'metrics' page seems broken

qqwy · November 8, 2021, 9:47am

I’ve moved an app (consisting of an application node and and a postgres database node) over from the ‘Personal’ organization to a ‘production’ one which my coworkers can get access to.

(To be precise: I first suspended them, then moved them, and then resumed them, to make sure the app would not complain for a long time about the database not being available).

However, now the metrics-page within Fly seems to no longer show any data.
What is going on here? Is there one more step I need to do, or is this a bug within Fly?

sudhir.j · November 8, 2021, 10:18am

@qqwy I don’t think you should have to do anything else for this, we’ll check what’s going on internally.

qqwy · November 8, 2021, 12:52pm

Thanks for the quick response!

Extra info:

The metrics pages of both the main app and the ‘database app’ are affected.
I had set up a Grafana dashboard before, and this is still receiving the data correctly (of both Fly’s main metrics as well as extra Prometheus/PromEx metrics).

sudhir.j · November 8, 2021, 1:06pm

Thanks for the extra info, we’re looking into it.

In the meanwhile can I suggest a re-deploy of the apps to try to unblock your dashboard? The configs are likely still pointing at whichever org they were last deployed in, would be good to see if that fixes it.

qqwy · November 8, 2021, 1:09pm

OK, I’ll try that.

How do I redeploy the postgres database ‘app’? In the documentation I was able to find options to suspend/resume and restart, but no clear way to ‘redeploy’.

sudhir.j · November 8, 2021, 1:11pm

Yeah, that’s more complex, can I suggest just the application servers first? Let make sure that handles the problem, don’t want to touch the database unless we know it’ll fix things.

qqwy · November 8, 2021, 1:15pm

The problem is not resolved. Rather, an error is raised during a redeploy, which seems to indicate that there are more problems:

Command: /app/bin/oberx eval Oberx.Release.migrate
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: 7943db6)...
         Preparing to run: `/app/bin/oberx eval Oberx.Release.migrate` as nobody
         2021/11/08 13:13:19 listening on [fdaa:0:38ed:a7b:23c2:4cd:d8be:2]:22 (DNS: [fdaa::3]:53)
         Reaped child process with pid: 561 and signal: SIGUSR1, core dumped? false
         13:13:21.135 [error] Postgrex.Protocol (#PID<0.137.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
         13:13:21.135 [error] Postgrex.Protocol (#PID<0.136.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
         13:13:23.076 [error] Could not create schema migrations table. This error usually happens due to the following:
           * The database does not exist
             migrations, was defined by another library
           * There is a deadlock while migrating (such as using concurrent
         To fix the first issue, run "mix ecto.create".
         To address the second, you can run "mix ecto.drop" followed by
         "mix ecto.create". Alternatively you may configure Ecto to use
         another table and/or repository for managing migrations:
               migration_repo: AnotherRepoForSchemaMigrations
         ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2978ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
           2. Tracking down slow queries and making sure they are running fast enough
           3. Increasing the pool_size (although this increases resource consumption)
           4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
             (ecto_sql 3.5.0) lib/ecto/adapters/sql.ex:770: Ecto.Adapters.SQL.execute_ddl/4
             (ecto_sql 3.5.0) lib/ecto/migrator.ex:495: Ecto.Migrator.lock_for_migrations/4
             (ecto_sql 3.5.0) lib/ecto/migrator.ex:413: Ecto.Migrator.run/4
         Reaped child process with pid: 563 and signal: SIGUSR1, core dumped? false
         Starting clean up.

Error Release command failed, deployment aborted

sudhir.j · November 8, 2021, 1:26pm

Thanks for that log, we’ll debug this a bit internally and get back to you.

Just want to confirm that the app itself is working correctly with no issues after the migration? I can see how the metrics and release task having trouble with the new org, I just want to confirm that there’s no issues with the running of the application itself.

qqwy · November 8, 2021, 1:49pm

The app is currently running correctly.

However, not being able to deploy a new release seems rather scary.

If you want, I can try turning off the release task for now and see how well a deploy without doing any release tasks will go?

Luckily this currently is only a ‘staging’ instance of our application (more by chance than by planning; originally I had planned to migrate our production application over to Fly earlier today, but the problem with the metrics made me investigate before doing so).

sudhir.j · November 8, 2021, 2:01pm

We can try turning off the release task, yes. That should complete successfully and push up a stable config under the new org. Should be easier to re-enable it once we have that.

qqwy · November 8, 2021, 2:10pm

I have done a new release (v24) with the release task turned off.

Unfortunately this does not resolve the issue. Instead, flyctl logs shows, from the time the new release starts:

2021-11-08T14:05:23.502 app[22da29e2] ams [info] Preparing to run: `bin/oberx start` as nobody
2021-11-08T14:05:23.519 app[22da29e2] ams [info] 2021/11/08 14:05:23 listening on [fdaa:0:38ed:a7b:aa0:22da:29e2:2]:22 (DNS: [fdaa::3]:53)
2021-11-08T14:05:24.513 app[22da29e2] ams [info] Reaped child process with pid: 545, exit code: 0
2021-11-08T14:05:28.043 app[22da29e2] ams [info] 14:05:28.042 [warn] GIT_SHA environment variable has not been defined
2021-11-08T14:05:28.045 app[22da29e2] ams [info] 14:05:28.042 [warn] GIT_AUTHOR environment variable has not been defined
2021-11-08T14:05:28.045 app[22da29e2] ams [info] 14:05:28.043 [warn] GIT_SHA environment variable has not been defined
2021-11-08T14:05:28.045 app[22da29e2] ams [info] 14:05:28.043 [warn] GIT_AUTHOR environment variable has not been defined
2021-11-08T14:05:28.051 app[22da29e2] ams [info] 14:05:28.049 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.049 [error] Postgrex.Protocol (#PID<0.2522.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.049 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.049 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.050 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.050 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.058 app[22da29e2] ams [info] 14:05:28.050 [error] Postgrex.Protocol (#PID<0.2532.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.060 app[22da29e2] ams [info] 14:05:28.051 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.060 app[22da29e2] ams [info] 14:05:28.053 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.060 app[22da29e2] ams [info] 14:05:28.054 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:28.094 app[22da29e2] ams [info] 14:05:28.093 [info] Running OberxWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
2021-11-08T14:05:28.119 app[22da29e2] ams [info] 14:05:28.116 [info] Access OberxWeb.Endpoint at http://oberx.fly.dev
2021-11-08T14:05:28.294 app[22da29e2] ams [info] 14:05:28.293 [info] PromEx.LifecycleAnnotator successfully created start annotation in Grafana.
2021-11-08T14:05:28.344 app[22da29e2] ams [info] 14:05:28.342 [info] PromEx.DashboardUploader successfully uploaded /app/lib/prom_ex-1.4.1/priv/application.json.eex to Grafana.
2021-11-08T14:05:28.485 app[22da29e2] ams [info] 14:05:28.484 [info] PromEx.DashboardUploader successfully uploaded /app/lib/prom_ex-1.4.1/priv/beam.json.eex to Grafana.
2021-11-08T14:05:28.519 app[22da29e2] ams [info] Reaped child process with pid: 566 and signal: SIGUSR1, core dumped? false
2021-11-08T14:05:28.640 app[22da29e2] ams [info] 14:05:28.639 [info] PromEx.DashboardUploader successfully uploaded /app/lib/prom_ex-1.4.1/priv/phoenix.json.eex to Grafana.
2021-11-08T14:05:28.792 app[22da29e2] ams [info] 14:05:28.791 [info] PromEx.DashboardUploader successfully uploaded /app/lib/prom_ex-1.4.1/priv/phoenix_live_view.json.eex to Grafana.
2021-11-08T14:05:29.088 app[22da29e2] ams [info] 14:05:29.086 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:29.190 app[22da29e2] ams [info] 14:05:29.188 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:29.262 app[22da29e2] ams [info] 14:05:29.261 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:29.871 app[22da29e2] ams [info] 14:05:29.870 [error] Postgrex.Protocol (#PID<0.2532.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:29.876 app[22da29e2] ams [info] 14:05:29.875 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.174 app[22da29e2] ams [info] 14:05:30.173 [error] Postgrex.Protocol (#PID<0.2522.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.263 app[22da29e2] ams [info] 14:05:30.262 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.264 app[22da29e2] ams [info] 14:05:30.262 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.528 app[22da29e2] ams [info] 14:05:30.527 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.732 app[22da29e2] ams [info] 14:05:30.731 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.861 app[22da29e2] ams [info] 14:05:30.860 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:30.889 app[22da29e2] ams [info] 14:05:30.888 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:31.876 app[22da29e2] ams [info] 14:05:31.874 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:32.451 app[22da29e2] ams [info] 14:05:32.450 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:32.702 app[22da29e2] ams [info] 14:05:32.701 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:32.888 app[22da29e2] ams [info] 14:05:32.887 [info] Elixir.LogflareLogger.HttpBackend v0.11.0 started.
2021-11-08T14:05:34.320 app[22da29e2] ams [info] 14:05:34.318 [error] Postgrex.Protocol (#PID<0.2532.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:34.505 app[22da29e2] ams [info] 14:05:34.504 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:34.873 app[22da29e2] ams [info] 14:05:34.872 [error] Postgrex.Protocol (#PID<0.2522.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:35.062 app[22da29e2] ams [info] 14:05:35.060 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:35.105 app[22da29e2] ams [info] 14:05:35.104 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:36.095 app[22da29e2] ams [info] 14:05:36.093 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:36.186 app[22da29e2] ams [info] 14:05:36.185 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:36.527 app[22da29e2] ams [info] 14:05:36.525 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:37.336 app[22da29e2] ams [info] 14:05:37.335 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:37.432 app[22da29e2] ams [info] 14:05:37.431 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:40.091 app[22da29e2] ams [info] 14:05:40.090 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:41.785 app[22da29e2] ams [info] 14:05:41.784 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:41.903 app[22da29e2] ams [info] 14:05:41.901 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:42.939 proxy[22da29e2] ams [warn] Health check status changed 'passing' => 'warning'
2021-11-08T14:05:43.492 app[22da29e2] ams [info] 14:05:43.491 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:43.703 app[22da29e2] ams [info] 14:05:43.702 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:45.280 app[22da29e2] ams [info] 14:05:45.279 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:46.259 app[22da29e2] ams [info] 14:05:46.257 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:46.673 app[22da29e2] ams [info] 14:05:46.672 [error] Postgrex.Protocol (#PID<0.2532.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:47.137 app[22da29e2] ams [info] 14:05:47.136 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:48.536 app[22da29e2] ams [info] 14:05:48.534 [error] Postgrex.Protocol (#PID<0.2522.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:55.348 app[22da29e2] ams [info] 14:05:55.346 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:56.205 app[22da29e2] ams [info] 14:05:56.203 request_id=FrWXQ5jpPpg0StgAABaR [info] HEAD /
2021-11-08T14:05:56.222 app[22da29e2] ams [info] 14:05:56.221 request_id=FrWXQ5jpPpg0StgAABaR [info] Sent 200 in 17ms
2021-11-08T14:05:56.361 app[22da29e2] ams [info] 14:05:56.360 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:57.803 app[22da29e2] ams [info] 14:05:57.801 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:59.031 app[22da29e2] ams [info] 14:05:59.029 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:05:59.110 app[22da29e2] ams [info] 14:05:59.109 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:06.902 app[22da29e2] ams [info] 14:06:06.900 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:08.045 app[22da29e2] ams [info] 14:06:08.044 [error] Postgrex.Protocol (#PID<0.2533.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:09.552 app[22da29e2] ams [info] 14:06:09.550 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:09.559 app[22da29e2] ams [info] 14:06:09.558 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:10.499 app[22da29e2] ams [info] 14:06:10.497 [error] Postgrex.Protocol (#PID<0.2532.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:13.427 app[a2e2af9d] ams [info] 14:06:13.427 request_id=FrWXR5uEHp0KZ6gAAGmB [info] GET /
2021-11-08T14:06:13.429 app[a2e2af9d] ams [info] 14:06:13.429 request_id=FrWXR5uEHp0KZ6gAAGmB [info] Sent 302 in 2ms
2021-11-08T14:06:13.459 app[a2e2af9d] ams [info] 14:06:13.458 request_id=FrWXR51np6DxH7QAAGmx [info] GET /restaurants
2021-11-08T14:06:13.462 app[a2e2af9d] ams [info] 14:06:13.461 request_id=FrWXR51np6DxH7QAAGmx [info] Sent 200 in 3ms
2021-11-08T14:06:15.382 app[22da29e2] ams [info] 14:06:15.381 [error] Postgrex.Protocol (#PID<0.2530.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:15.655 app[a2e2af9d] ams [info] 14:06:15.654 request_id=FrWXSCBJDiJdG08AAGnh [info] GET /restaurants
2021-11-08T14:06:15.657 app[a2e2af9d] ams [info] 14:06:15.657 request_id=FrWXSCBJDiJdG08AAGnh [info] Sent 200 in 2ms
2021-11-08T14:06:17.482 app[a2e2af9d] ams [info] 14:06:17.481 request_id=FrWXSI0syR696hYAAGoR [info] GET /restaurants/ad113006-3609-482c-b860-47b1ffa35b55
2021-11-08T14:06:17.492 app[a2e2af9d] ams [info] 14:06:17.491 request_id=FrWXSI0syR696hYAAGoR [info] Sent 200 in 10ms
2021-11-08T14:06:17.529 app[22da29e2] ams [info] 14:06:17.527 [error] Postgrex.Protocol (#PID<0.2522.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:18.120 app[22da29e2] ams [info] 14:06:18.118 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:18.481 app[a2e2af9d] ams [info] 14:06:18.480 request_id=FrWXSMi1e9hMFssAAGpB [info] GET /restaurants/ad113006-3609-482c-b860-47b1ffa35b55/detail
2021-11-08T14:06:18.492 app[a2e2af9d] ams [info] 14:06:18.491 request_id=FrWXSMi1e9hMFssAAGpB [info] Sent 200 in 10ms
2021-11-08T14:06:18.899 app[a2e2af9d] ams [info] 14:06:18.899 [info] CONNECTED TO Phoenix.LiveView.Socket in 33µs
2021-11-08T14:06:18.899 app[a2e2af9d] ams [info]   Transport: :websocket
2021-11-08T14:06:18.899 app[a2e2af9d] ams [info]   Serializer: Phoenix.Socket.V2.JSONSerializer
2021-11-08T14:06:18.899 app[a2e2af9d] ams [info]   Parameters: %{"_csrf_token" => "KhYOBCJQLWEwBRpTQXM1JGYReTQYEFgmGeAJe1TRzK_elKdRPr8CiJ1h", "_mounts" => "0", "_track_static" => %{"0" => "https://staging.oberx.nl/semantic-ui-css/semantic.min-979091fc4831c0cdf5dcecb2746c982f.css?vsn=d", "1" => "https://staging.oberx.nl/css/app-dd3f79ebb83bb1bd5306d1cf33f281d2.css?vsn=d", "2" => "https://staging.oberx.nl/js/app-6500af22d8f21de5e267b66a624221e4.js?vsn=d"}, "vsn" => "2.0.0"}
2021-11-08T14:06:18.937 proxy[22da29e2] ams [info] Health check status changed 'warning' => 'passing'
2021-11-08T14:06:19.920 app[22da29e2] ams [info] 14:06:19.919 [error] Postgrex.Protocol (#PID<0.2524.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:21.324 app[22da29e2] ams [info] 14:06:21.323 [error] Postgrex.Protocol (#PID<0.2534.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:22.492 runner[a2e2af9d] ams [info] Shutting down virtual machine
2021-11-08T14:06:22.639 app[a2e2af9d] ams [info] Sending signal SIGTERM to main child process w/ PID 510
2021-11-08T14:06:22.640 app[a2e2af9d] ams [info] 14:06:22.639 [info] SIGTERM received - shutting down
2021-11-08T14:06:25.471 app[22da29e2] ams [info] 14:06:25.469 [error] Postgrex.Protocol (#PID<0.2527.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:26.642 app[22da29e2] ams [info] 14:06:26.641 [error] Postgrex.Protocol (#PID<0.2536.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:27.249 app[22da29e2] ams [info] 14:06:27.248 [error] Postgrex.Protocol (#PID<0.2528.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
2021-11-08T14:06:29.844 app[22da29e2] ams [info] 14:06:29.842 [error] Postgrex.Protocol (#PID<0.2535.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (oberx-staging-db.internal:5432): non-existing domain - :nxdomain
nats: Permissions Violation for Subscription to logs.oberx.*.* on connection [28]

Two things seem odd:

The health checks pass even though the app cannot connect to the database, and the broken release ends up deployed.
- This includes a ‘HTTP’ health check which I would expect to fail, as visiting the page manually in my browser shows a ‘500 internal server error’. EDIT: This is actually not that strange, as the HTTP health check is visiting the home page which does not involve any database queries.
Not only can the connection to the database not be established, also we see nats: Permissions Violation for Subscription to logs.oberx.*.* on connection [28].

sudhir.j · November 8, 2021, 2:18pm

@qqwy this has re-deployed your app server in the new org — and the Postgres app wasn’t deployed in the new org’s network, which we’ve rectified by manually re-deploying your Postgres in the new org.

If you try a restart / re-deploy again you should see everything running fine now. And we can then re-enable the release task as well.

sudhir.j · November 8, 2021, 2:21pm

The move command should have done all this, but we think suspending the app actually interfered with the normal moving operation, which is rather ironic

qqwy · November 8, 2021, 2:28pm

Thank you! I have done so (including re-enabling the release task). The application indeed is chugging along nicely again.

Heh, yes, this is rather ironic ! It’s the big red warning messages that were shown when I tried moving which made me suspend it in the first place.

Thank you very much for your quick response and your diligent resolution of this problem!

sudhir.j · November 8, 2021, 2:49pm

we have a fix going out to make sure apps moved when suspended wake up into the correct org network as well, so you shouldn’t have this problem again.

qqwy · November 9, 2021, 12:13pm

The app and database are running properly again, and there are metrics for the main app.

However, I just noticed that the metrics page of the database-app is still empty:

sudhir.j · November 9, 2021, 1:23pm

Will take a look at that and let you know.

sudhir.j · November 9, 2021, 3:13pm

@qqwy we’ve manually reconfigured the app (oberx-staging-db, right?) with the correct metrics ID for the new org — you should start seeing metrics soon.

qqwy · November 9, 2021, 3:14pm

Yes, oberx-staging-db is correct.

fedeotaran · November 9, 2021, 6:52pm

Hi @sudhir.j! I have the same issue for the application: koderitdb-dev.

Topic		Replies	Views
No postgres metrics after moving organizations Questions / Help	10	466	June 14, 2022
Dashboard metrics are not available	12	759	December 4, 2023
Lack of Prometheus metrics, sometimes, after deploy (host specific?)	20	2246	June 22, 2022
Postgres metrics on Fly dashboard have "no data available" Questions / Help postgres	6	613	May 4, 2022
Prometheus not collecting or exposing metrics? Questions / Help grafana	6	2597	September 26, 2022

After moving organizations, the 'metrics' page seems broken

Related topics