fra [warn] Could not proxy HTTP request, app went into infinite redeploy loop after db upgrade

Questions / Help
1

I have just upgraded postgres-flex image, i can connect to the db, but my app cannot, went into infinite redeploy loop

Crash dump is being written to: erl_crash.dump..."connection not available and request was dropped from queue after 2984ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:

  1. Ensuring your database is available and that you can connect to it
  2. Tracking down slow queries and making sure they are running fast enoug
  3. Increasing the pool_size (although this increases resource consumption)
  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2

2023-03-02T19:59:04Z app[83a6ca78] ams [info]done
2023-03-02T19:59:05Z app[83a6ca78] ams [info]Starting clean up.
2023-03-02T19:59:08Z proxy[83a6ca78] ams [error]timed out while connecting to instance
2023-03-02T19:59:09Z proxy[83a6ca78] ams [error]timed out while connecting to instance
2023-03-02T19:59:09Z proxy[83a6ca78] fra [warn]Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-03-02T19:59:10Z proxy[83a6ca78] fra [warn]Could not proxy HTTP request. Retrying in 1000 ms (attempt 60)
2023-03-02T19:59:10Z runner[83a6ca78] ams [info]Starting instance
2023-03-02T19:59:10Z runner[83a6ca78] ams [info]Configuring virtual machine
2023-03-02T19:59:10Z runner[83a6ca78] ams [info]Pulling container image
2023-03-02T19:59:10Z proxy[83a6ca78] ams [error]timed out while connecting to instance
2023-03-02T19:59:11Z runner[83a6ca78] ams [info]Unpacking image
2023-03-02T19:59:11Z runner[83a6ca78] ams [info]Preparing kernel init
2023-03-02T19:59:11Z runner[83a6ca78] ams [info]Configuring firecracker
2023-03-02T19:59:12Z runner[83a6ca78] ams [info]Starting virtual machine
2023-03-02T19:59:12Z app[83a6ca78] ams [info]Starting init (commit: 08b4c2b)...
2023-03-02T19:59:12Z app[83a6ca78] ams [info]Preparing to run: `/app/bin/server` as nobody
2023-03-02T19:59:12Z proxy[83a6ca78] ams [error]instance refused connection
2023-03-02T19:59:12Z proxy[83a6ca78] ams [error]timed out while connecting to instance

2023-03-02T20:06:03Z app[b9e42ad0] ams [info]Crash dump is being written to: erl_crash.dump...done
2023-03-02T20:06:04Z app[b9e42ad0] ams [info]Starting clean up.
2023-03-02T20:06:08Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:08Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:12Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:12Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:16Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:16Z proxy[b9e42ad0] ams [error]timed out while connecting to instance
2023-03-02T20:06:19Z proxy[b9e42ad0] fra [warn]Could not proxy HTTP request. Retrying in 1000 ms (attempt 50)
2

Hey Elder,

Which database is this?

4

@shaun I’ve changed app’s DATABASE_URL to use app-db.internal instead of ip6, it helped, app got connected to the db.

App lost connection to the db after the db image upgrade, crashed on start and went into infinite redeploy loop.

That was some stressful 30 minutes of downtime, and another error monitoring spike protection kicked in.

5

We tracked this down to a networking issue that was recently deployed. We are working on a fix.

1 Like
6
1 Like
7

Thanks for the quick response and for the updates

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.