Site went down: "Code: 0 - Reason: Timeout / Connection Refused"w

Questions / Help
1

Hi everyone!

Our website went down at 2:33 PM with this error :

Code: 0 - Reason: Timeout / Connection Refused

Since then, it looks like it’s trying to restart again and again, but it always falls back on :

2022-12-15T13:46:08.591 app[6e754474] cdg [info] 13:46:08.591 [debug] Monitored server Postgrex terminate due to: :shutdown
2022-12-15T13:46:08.595 app[6e754474] cdg [info] 13:46:08.594 [notice] Application encheres_immo exited: shutdown
2022-12-15T13:46:08.604 app[6e754474] cdg [info] [os_mon] memory supervisor port (memsup): Erlang has closed
2022-12-15T13:46:08.605 app[6e754474] cdg [info] [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
2022-12-15T13:46:10.124 app[6e754474] cdg [info] {"Kernel pid terminated",application_controller,"{application_terminated,encheres_immo,shutdown}"}
2022-12-15T13:46:10.126 app[6e754474] cdg [info] Kernel pid terminated (application_controller) ({application_terminated,encheres_immo,shutdown})
2022-12-15T13:46:10.298 app[6e754474] cdg [info] Crash dump is being written to: erl_crash.dump...done
2022-12-15T13:46:10.664 app[6e754474] cdg [info] Starting clean up.
Error: could not find an instance to route to
Error: could not find an instance to route to
Error: could not find an instance to route to
Error: could not find an instance to route to

Our last release was two days ago and didn’t change anything major. https://status.flyio.net/ says “All Systems Operational”.

Here are the metrics :

image
image1892×2731 347 KB

Is this a temporary outage, or is there a real problem? Any idea of the reasons for this outage?

Thanks for your help!

2

Are there any other logs prior to the application shutdown? This looks like an issue with your specific release either failing to start or consistently exiting before health checks.

3

Thanks @chrismccord for taking the time to answer me,

The current release dates from two days ago. And has worked flawlessly ever since.

I am currently investigating an error that occurred one hour before the crash. But I find it hard to see how that could have caused the application to crash, and especially why it now prevents it from relaunching.

Fly.io monitoring logs 
$ fly logs -a encheres-immo-production

Waiting for logs...

2022-12-15T15:04:12.946 app[f7494cf6] cdg [info] data: %EncheresImmo.Auctions.Events.AuctionOutbid{

2022-12-15T15:04:12.946 app[f7494cf6] cdg [info] amount: 1300000,

2022-12-15T15:04:12.946 app[f7494cf6] cdg [info] auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2",

2022-12-15T15:04:12.946 app[f7494cf6] cdg [info] created_at: "2022-12-15T13:30:08.299622Z",

2022-12-15T15:04:12.948 app[f7494cf6] cdg [info] 15:04:12.947 [error] GenServer {EncheresImmo.Auctions.EventStore.EventStore.Subscriptions.Registry, {"$all", "Auctions.Projectors.Auction"}} terminating

2022-12-15T15:04:12.971 app[f7494cf6] cdg [info] 15:04:12.970 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:12.971 app[f7494cf6] cdg [info] 15:04:12.970 [debug] EncheresImmo.Auctions.Projectors.Auction has successfully subscribed to event store

2022-12-15T15:04:12.994 app[f7494cf6] cdg [info] 15:04:12.992 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:12.994 app[f7494cf6] cdg [info] 15:04:12.993 [debug] EncheresImmo.Auctions.Projectors.Auction received events: [%Commanded.EventStore.RecordedEvent{causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d", correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7", created_at: ~U[2022-12-15 13:30:08.310397Z], data: %EncheresImmo.Auctions.Events.AuctionOutbid{amount: 1300000, auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", created_at: "2022-12-15T13:30:08.299622Z", new_end_date: nil, postponed_version: nil, registration_id: "2d802a85-2e97-4694-ba7c-ce3cc91cf3df"}, event_id: "3795c180-d0cb-4efb-8cd5-b5bf088a5574", event_number: 6628, event_type: "Elixir.EncheresImmo.Auctions.Events.AuctionOutbid", metadata: %{}, stream_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", stream_version: 4}]

2022-12-15T15:04:13.000 app[f7494cf6] cdg [info] 15:04:12.999 [debug] QUERY OK db=5.3ms queue=0.5ms idle=893.5ms

2022-12-15T15:04:13.000 app[f7494cf6] cdg [info] begin []

2022-12-15T15:04:13.005 app[f7494cf6] cdg [info] 15:04:13.004 [debug] QUERY OK source="projection_versions" db=5.2ms

2022-12-15T15:04:13.005 app[f7494cf6] cdg [info] SELECT p0."projection_name", p0."last_seen_event_number", p0."inserted_at", p0."updated_at" FROM "projection_versions" AS p0 WHERE (p0."projection_name" = $1) ["Auctions.Projectors.Auction"]

2022-12-15T15:04:13.010 app[f7494cf6] cdg [info] 15:04:13.010 [debug] QUERY OK db=5.2ms

2022-12-15T15:04:13.010 app[f7494cf6] cdg [info] UPDATE "projection_versions" SET "last_seen_event_number" = $1, "updated_at" = $2 WHERE "projection_name" = $3 [6628, ~N[2022-12-15 15:04:13.004851], "Auctions.Projectors.Auction"]

2022-12-15T15:04:13.018 app[f7494cf6] cdg [info] 15:04:13.017 [debug] QUERY ERROR db=7.1ms

2022-12-15T15:04:13.018 app[f7494cf6] cdg [info] INSERT INTO "bids" ("amount","created_at","is_retained","registration_id","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) [1300000, ~U[2022-12-15 13:30:08.299622Z], false, <<45, 128, 42, 133, 46, 151, 70, 148, 186, 124, 206, 60, 201, 28, 243, 223>>, ~U[2022-12-15 15:04:13.010332Z], ~U[2022-12-15 15:04:13.010332Z], <<180, 23, 143, 145, 35, 183, 64, 93, 159, 50, 185, 28, 88, 15, 32, 69>>]

2022-12-15T15:04:13.024 app[f7494cf6] cdg [info] 15:04:13.023 [debug] QUERY OK db=5.6ms

2022-12-15T15:04:13.024 app[f7494cf6] cdg [info] rollback []

2022-12-15T15:04:13.025 app[f7494cf6] cdg [info] 15:04:13.024 [error] EncheresImmo.Auctions.Projectors.Auction failed to handle event %Commanded.EventStore.RecordedEvent{

2022-12-15T15:04:13.028 app[f7494cf6] cdg [info] 15:04:13.027 [debug] Subscription "Auctions.Projectors.Auction"@"$all" has no subscribers, shutting down

2022-12-15T15:04:13.050 app[f7494cf6] cdg [info] 15:04:13.049 [debug] EncheresImmo.Auctions.Projectors.Auction has successfully sub

2022-12-15T15:04:13.073 app[f7494cf6] cdg [info] 15:04:13.072 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.073 app[f7494cf6] cdg [info] 15:04:13.072 [debug] EncheresImmo.Auctions.Projectors.Auction received events: [%Commanded.EventStore.RecordedEvent{causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d", correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7", created_at: ~U[2022-12-15 13:30:08.310397Z], data: %EncheresImmo.Auctions.Events.AuctionOutbid{amount: 1300000, auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", created_at: "2022-12-15T13:30:08.299622Z", new_end_date: nil, postponed_version: nil, registration_id: "2d802a85-2e97-4694-ba7c-ce3cc91cf3df"}, event_id: "3795c180-d0cb-4efb-8cd5-b5bf088a5574", event_number: 6628, event_type: "Elixir.EncheresImmo.Auctions.Events.AuctionOutbid", metadata: %{}, stream_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", stream_version: 4}]

2022-12-15T15:04:13.079 app[f7494cf6] cdg [info] 15:04:13.078 [debug] QUERY OK db=5.0ms queue=0.4ms idle=319.2ms

2022-12-15T15:04:13.079 app[f7494cf6] cdg [info] begin []

2022-12-15T15:04:13.085 app[f7494cf6] cdg [info] 15:04:13.084 [debug] QUERY OK source="projection_versions" db=5.3ms

2022-12-15T15:04:13.085 app[f7494cf6] cdg [info] SELECT p0."projection_name", p0."last_seen_event_number", p0."inserted_at", p0."updated_at" FROM "projection_versions" AS p0 WHERE (p0."projection_name" = $1) ["Auctions.Projectors.Auction"]

2022-12-15T15:04:13.090 app[f7494cf6] cdg [info] 15:04:13.089 [debug] QUERY OK db=5.3ms

2022-12-15T15:04:13.090 app[f7494cf6] cdg [info] UPDATE "projection_versions" SET "last_seen_event_number" = $1, "updated_at" = $2 WHERE "projection_name" = $3 [6628, ~N[2022-12-15 15:04:13.084180], "Auctions.Projectors.Auction"]

2022-12-15T15:04:13.096 app[f7494cf6] cdg [info] 15:04:13.095 [debug] QUERY ERROR db=5.3ms

2022-12-15T15:04:13.096 app[f7494cf6] cdg [info] INSERT INTO "bids" ("amount","created_at","is_retained","registration_id","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) [1300000, ~U[2022-12-15 13:30:08.299622Z], false, <<45, 128, 42, 133, 46, 151, 70, 148, 186, 124, 206, 60, 201, 28, 243, 223>>, ~U[2022-12-15 15:04:13.090053Z], ~U[2022-12-15 15:04:13.090053Z], <<109, 93, 209, 125, 72, 2, 65, 75, 179, 98, 254, 63, 118, 118, 48, 239>>]

2022-12-15T15:04:13.102 app[f7494cf6] cdg [info] 15:04:13.101 [debug] QUERY OK db=5.1ms

2022-12-15T15:04:13.102 app[f7494cf6] cdg [info] rollback []

2022-12-15T15:04:13.102 app[f7494cf6] cdg [info] 15:04:13.102 [error] EncheresImmo.Auctions.Projectors.Auction failed to handle event %Commanded.EventStore.RecordedEvent{

2022-12-15T15:04:13.103 app[f7494cf6] cdg [info] 15:04:13.102 [debug] EncheresImmo.Auctions.Projectors.Auction is shutting down due to %Ecto.ConstraintError{constraint: "bids_registration_id_fkey", message: "constraint error when attempting to insert struct:\n\n * bids_registration_id_fkey (foreign_key_constraint)\n\nIf you would like to stop this constraint violation from raising an\nexception and instead add it as an error to your changeset, please\ncall `foreign_key_constraint/3` on your changeset with the constraint\n`:name` as an option.\n\nThe changeset has not defined any constraint.\n", type: :foreign_key}

2022-12-15T15:04:13.108 app[f7494cf6] cdg [info] 15:04:13.107 [debug] Subscription "Auctions.Projectors.Auction"@"$all" attempting to connect subscriber #PID<0.3885.0>

2022-12-15T15:04:13.129 app[f7494cf6] cdg [info] 15:04:13.128 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.129 app[f7494cf6] cdg [info] 15:04:13.128 [debug] EncheresImmo.Auctions.Projectors.Auction has successfully subscribed to event store

2022-12-15T15:04:13.152 app[f7494cf6] cdg [info] 15:04:13.151 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.152 app[f7494cf6] cdg [info] 15:04:13.151 [debug] EncheresImmo.Auctions.Projectors.Auction received events: [%Commanded.EventStore.RecordedEvent{causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d", correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7", created_at: ~U[2022-12-15 13:30:08.310397Z], data: %EncheresImmo.Auctions.Events.AuctionOutbid{amount: 1300000, auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", created_at: "2022-12-15T13:30:08.299622Z", new_end_date: nil, postponed_version: nil, registration_id: "2d802a85-2e97-4694-ba7c-ce3cc91cf3df"}, event_id: "3795c180-d0cb-4efb-8cd5-b5bf088a5574", event_number: 6628, event_type: "Elixir.EncheresImmo.Auctions.Events.AuctionOutbid", metadata: %{}, stream_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", stream_version: 4}]

2022-12-15T15:04:13.157 app[f7494cf6] cdg [info] 15:04:13.156 [debug] QUERY OK db=4.9ms queue=0.2ms idle=301.0ms

2022-12-15T15:04:13.157 app[f7494cf6] cdg [info] begin []

2022-12-15T15:04:13.164 app[f7494cf6] cdg [info] 15:04:13.164 [debug] QUERY OK source="projection_versions" db=6.8ms

2022-12-15T15:04:13.164 app[f7494cf6] cdg [info] SELECT p0."projection_name", p0."last_seen_event_number", p0."inserted_at", p0."updated_at" FROM "projection_versions" AS p0 WHERE (p0."projection_name" = $1) ["Auctions.Projectors.Auction"]

2022-12-15T15:04:13.170 app[f7494cf6] cdg [info] 15:04:13.170 [debug] QUERY OK db=5.7ms

2022-12-15T15:04:13.170 app[f7494cf6] cdg [info] UPDATE "projection_versions" SET "last_seen_event_number" = $1, "updated_at" = $2 WHERE "projection_name" = $3 [6628, ~N[2022-12-15 15:04:13.164089], "Auctions.Projectors.Auction"]

2022-12-15T15:04:13.177 app[f7494cf6] cdg [info] 15:04:13.176 [debug] QUERY ERROR db=5.8ms

2022-12-15T15:04:13.177 app[f7494cf6] cdg [info] INSERT INTO "bids" ("amount","created_at","is_retained","registration_id","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) [1300000, ~U[2022-12-15 13:30:08.299622Z], false, <<45, 128, 42, 133, 46, 151, 70, 148, 186, 124, 206, 60, 201, 28, 243, 223>>, ~U[2022-12-15 15:04:13.170134Z], ~U[2022-12-15 15:04:13.170134Z], <<216, 232, 158, 19, 189, 176, 77, 117, 161, 92, 101, 81, 139, 88, 235, 104>>]

2022-12-15T15:04:13.182 app[f7494cf6] cdg [info] 15:04:13.181 [debug] QUERY OK db=5.1ms

2022-12-15T15:04:13.182 app[f7494cf6] cdg [info] rollback []

2022-12-15T15:04:13.183 app[f7494cf6] cdg [info] 15:04:13.182 [error] EncheresImmo.Auctions.Projectors.Auction failed to handle event %Commanded.EventStore.RecordedEvent{

2022-12-15T15:04:13.183 app[f7494cf6] cdg [info] causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d",

2022-12-15T15:04:13.184 app[f7494cf6] cdg [info] 15:04:13.183 [error] GenServer {EncheresImmo.Auctions.Application, {EncheresImmo.Auctions.Application, Commanded.Event.Handler, "Auctions.Projectors.Auction"}} terminating

2022-12-15T15:04:13.207 app[f7494cf6] cdg [info] 15:04:13.206 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.207 app[f7494cf6] cdg [info] 15:04:13.206 [debug] EncheresImmo.Auctions.Projectors.Auction has successfully subscribed to event store

2022-12-15T15:04:13.230 app[f7494cf6] cdg [info] 15:04:13.229 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.230 app[f7494cf6] cdg [info] 15:04:13.229 [debug] EncheresImmo.Auctions.Projectors.Auction received events: [%Commanded.EventStore.RecordedEvent{causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d", correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7", created_at: ~U[2022-12-15 13:30:08.310397Z], data: %EncheresImmo.Auctions.Events.AuctionOutbid{amount: 1300000, auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", created_at: "2022-12-15T13:30:08.299622Z", new_end_date: nil, postponed_version: nil, registration_id: "2d802a85-2e97-4694-ba7c-ce3cc91cf3df"}, event_id: "3795c180-d0cb-4efb-8cd5-b5bf088a5574", event_number: 6628, event_type: "Elixir.EncheresImmo.Auctions.Events.AuctionOutbid", metadata: %{}, stream_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", stream_version: 4}]

2022-12-15T15:04:13.237 app[f7494cf6] cdg [info] 15:04:13.236 [debug] QUERY OK db=6.7ms queue=0.3ms idle=284.9ms

2022-12-15T15:04:13.237 app[f7494cf6] cdg [info] begin []

2022-12-15T15:04:13.243 app[f7494cf6] cdg [info] 15:04:13.242 [debug] QUERY OK source="projection_versions" db=5.6ms

2022-12-15T15:04:13.243 app[f7494cf6] cdg [info] SELECT p0."projection_name", p0."last_seen_event_number", p0."inserted_at", p0."updated_at" FROM "projection_versions" AS p0 WHERE (p0."projection_name" = $1) ["Auctions.Projectors.Auction"]

2022-12-15T15:04:13.249 app[f7494cf6] cdg [info] 15:04:13.248 [debug] QUERY OK db=5.8ms

2022-12-15T15:04:13.249 app[f7494cf6] cdg [info] UPDATE "projection_versions" SET "last_seen_event_number" = $1, "updated_at" = $2 WHERE "projection_name" = $3 [6628, ~N[2022-12-15 15:04:13.242877], "Auctions.Projectors.Auction"]

2022-12-15T15:04:13.257 app[f7494cf6] cdg [info] 15:04:13.257 [debug] QUERY ERROR db=7.5ms

2022-12-15T15:04:13.257 app[f7494cf6] cdg [info] INSERT INTO "bids" ("amount","created_at","is_retained","registration_id","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) [1300000, ~U[2022-12-15 13:30:08.299622Z], false, <<45, 128, 42, 133, 46, 151, 70, 148, 186, 124, 206, 60, 201, 28, 243, 223>>, ~U[2022-12-15 15:04:13.249042Z], ~U[2022-12-15 15:04:13.249042Z], <<86, 245, 63, 242, 145, 235, 72, 131, 157, 72, 73, 35, 82, 185, 121, 237>>]

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] 15:04:13.262 [debug] QUERY OK db=5.1ms

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] rollback []

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] 15:04:13.262 [error] EncheresImmo.Auctions.Projectors.Auction failed to handle event %Commanded.EventStore.RecordedEvent{

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d",

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7",

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] created_at: ~U[2022-12-15 13:30:08.310397Z],

2022-12-15T15:04:13.263 app[f7494cf6] cdg [info] amount: 1300000,

2022-12-15T15:04:13.287 app[f7494cf6] cdg [info] 15:04:13.286 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.287 app[f7494cf6] cdg [info] 15:04:13.287 [debug] EncheresImmo.Auctions.Projectors.Auction has successfully subscribed to event store

2022-12-15T15:04:13.312 app[f7494cf6] cdg [info] 15:04:13.311 [debug] Subscription "Auctions.Projectors.Auction"@"$all" catching-up

2022-12-15T15:04:13.312 app[f7494cf6] cdg [info] 15:04:13.311 [debug] EncheresImmo.Auctions.Projectors.Auction received events: [%Commanded.EventStore.RecordedEvent{causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d", correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7", created_at: ~U[2022-12-15 13:30:08.310397Z], data: %EncheresImmo.Auctions.Events.AuctionOutbid{amount: 1300000, auction_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", created_at: "2022-12-15T13:30:08.299622Z", new_end_date: nil, postponed_version: nil, registration_id: "2d802a85-2e97-4694-ba7c-ce3cc91cf3df"}, event_id: "3795c180-d0cb-4efb-8cd5-b5bf088a5574", event_number: 6628, event_type: "Elixir.EncheresImmo.Auctions.Events.AuctionOutbid", metadata: %{}, stream_id: "02d5c58a-ceae-445e-be90-f6e27b1939d2", stream_version: 4}]

2022-12-15T15:04:13.319 app[f7494cf6] cdg [info] 15:04:13.317 [debug] QUERY OK db=5.5ms queue=0.4ms idle=288.3ms

2022-12-15T15:04:13.319 app[f7494cf6] cdg [info] begin []

2022-12-15T15:04:13.324 app[f7494cf6] cdg [info] 15:04:13.323 [debug] QUERY OK source="projection_versions" db=5.5ms

2022-12-15T15:04:13.324 app[f7494cf6] cdg [info] SELECT p0."projection_name", p0."last_seen_event_number", p0."inserted_at", p0."updated_at" FROM "projection_versions" AS p0 WHERE (p0."projection_name" = $1) ["Auctions.Projectors.Auction"]

2022-12-15T15:04:13.330 app[f7494cf6] cdg [info] 15:04:13.329 [debug] QUERY OK db=5.3ms

2022-12-15T15:04:13.330 app[f7494cf6] cdg [info] UPDATE "projection_versions" SET "last_seen_event_number" = $1, "updated_at" = $2 WHERE "projection_name" = $3 [6628, ~N[2022-12-15 15:04:13.323829], "Auctions.Projectors.Auction"]

2022-12-15T15:04:13.337 app[f7494cf6] cdg [info] 15:04:13.337 [debug] QUERY ERROR db=6.8ms

2022-12-15T15:04:13.337 app[f7494cf6] cdg [info] INSERT INTO "bids" ("amount","created_at","is_retained","registration_id","inserted_at","updated_at","id") VALUES ($1,$2,$3,$4,$5,$6,$7) [1300000, ~U[2022-12-15 13:30:08.299622Z], false, <<45, 128, 42, 133, 46, 151, 70, 148, 186, 124, 206, 60, 201, 28, 243, 223>>, ~U[2022-12-15 15:04:13.329633Z], ~U[2022-12-15 15:04:13.329633Z], <<38, 217, 111, 245, 140, 53, 72, 156, 135, 195, 105, 21, 167, 64, 0, 80>>]

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] 15:04:13.342 [debug] QUERY OK db=5.1ms

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] rollback []

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] 15:04:13.343 [error] EncheresImmo.Auctions.Projectors.Auction failed to handle event %Commanded.EventStore.RecordedEvent{

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] causation_id: "8427b34e-fc38-4534-9ea5-4db0321ecf6d",

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] correlation_id: "cc551332-d60d-4d4b-800b-9c3bc0bd1bf7",

2022-12-15T15:04:13.343 app[f7494cf6] cdg [info] created_at: ~U[2022-12-15 13:30:08.310397Z],

2022-12-15T15:04:13.348 app[f7494cf6] cdg [info] 15:04:13.348 [debug] Monitored server Postgrex terminate due to: :shutdown

2022-12-15T15:04:13.354 app[f7494cf6] cdg [info] 15:04:13.353 [notice] Application encheres_immo exited: shutdown

2022-12-15T15:04:13.365 app[f7494cf6] cdg [info] [os_mon] memory supervisor port (memsup): Erlang has closed

2022-12-15T15:04:13.379 app[f7494cf6] cdg [info] [os_mon] cpu supervisor port (cpu_sup): Erlang has closed

2022-12-15T15:04:14.894 app[f7494cf6] cdg [info] Kernel pid terminated (applilication_controller,"{application_terminated,encheres_immo,shutdown}"}

2022-12-15T15:04:14.895 app[f7494cf6] cdg [info] Kernel pid terminated (application_controller) ({application_terminated,encheres_immo,shutdown})

2022-12-15T15:04:15.066 app[f7494cf6] cdg [info] Crash dump is being written to: erl_crash.dump...done

2022-12-15T15:04:15.366 app[f7494cf6] cdg [info] Starting clean up.

We have had hundreds of sales organized on our site by more than 200 real estate agencies, without ever having encountered this problem. And the latest releases haven’t touched that part of the application.

Nevertheless, I tried forcing a new release just now to see if that would change anything, and I was faced with the “Failed due to unhealthy allocations” error.

Fly.io deploy logs 
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-little-thunder-4065 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 11.1s (0/1)                                                                                                                                                         
[+] Building 182.2s (30/30) FINISHED                                                                                                                                             
 => [internal] load remote build context                                                                                                                                    0.0s
 => copy /context /                                                                                                                                                         1.9s
 => [internal] load metadata for docker.io/library/alpine:3.15.3                                                                                                            0.6s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.4-erlang-24.3.4-alpine-3.15.3                                                                                  0.6s
 => [build  1/17] FROM docker.io/hexpm/elixir:1.13.4-erlang-24.3.4-alpine-3.15.3@sha256:54f467278615de7dd821aeecbee425537508f19241950ee121e7a4b6c7b11991                    0.0s
 => [app 1/8] FROM docker.io/library/alpine:3.15.3@sha256:f22945d45ee2eb4dd463ed5a431d9f04fcd80ca768bb1acf898d91ce51f7bf04                                                  0.0s
 => CACHED [build  2/17] RUN apk add --no-cache build-base npm git                                                                                                          0.0s
 => CACHED [build  3/17] WORKDIR /app                                                                                                                                       0.0s
 => CACHED [build  4/17] RUN mix local.hex --force &&     mix local.rebar --force                                                                                           0.0s
 => [build  5/17] COPY mix.exs mix.lock ./                                                                                                                                  0.2s 
 => [build  6/17] COPY config/config.exs config/prod.exs config/                                                                                                            0.0s
 => [build  7/17] RUN mix deps.get --only prod &&     mix deps.compile                                                                                                    131.2s
 => [build  8/17] COPY assets/package.json assets/package-lock.json ./assets/                                                                                               0.0s 
 => [build  9/17] RUN npm --prefix ./assets ci --progress=false --no-audit --loglevel=error                                                                                 2.1s
 => [build 10/17] COPY priv priv                                                                                                                                            0.1s
 => [build 11/17] COPY lib lib                                                                                                                                              0.0s
 => [build 12/17] COPY assets assets                                                                                                                                        0.0s
 => [build 13/17] RUN mix assets.deploy                                                                                                                                    31.2s
 => [build 14/17] RUN mix compile                                                                                                                                           8.0s
 => [build 15/17] COPY config/runtime.exs config/                                                                                                                           0.0s
 => [build 16/17] COPY rel rel                                                                                                                                              0.0s
 => [build 17/17] RUN mix release                                                                                                                                           4.4s
 => CACHED [app 2/8] RUN apk add --no-cache libstdc++ openssl ncurses-libs imagemagick                                                                                      0.0s
 => CACHED [app 3/8] RUN apk --no-cache add msttcorefonts-installer fontconfig &&     update-ms-fonts &&     fc-cache -f                                                    0.0s
 => CACHED [app 4/8] WORKDIR /app                                                                                                                                           0.0s
 => CACHED [app 5/8] RUN chown nobody:nobody /app                                                                                                                           0.0s
 => CACHED [app 6/8] RUN mkdir /tmp/ftp_transfers                                                                                                                           0.0s
 => CACHED [app 7/8] RUN chown nobody:nobody /tmp/ftp_transfers                                                                                                             0.0s
 => [app 8/8] COPY --from=build --chown=nobody:nobody /app/_build/prod/rel/encheres_immo ./                                                                                 0.5s
 => exporting to image                                                                                                                                                      0.8s
 => => exporting layers                                                                                                                                                     0.8s
 => => writing image sha256:6b008aaa870984c17c5873e988979369b33254e3e443d865c44093adc95213d2                                                                                0.0s
 => => naming to registry.fly.io/encheres-immo-production:deployment-01GMB700AD20NHX2KFE54CJEG9                                                                             0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/encheres-immo-production]
240df040e822: Pushed 
3d9e0e66db4e: Mounted from encheres-immo-staging 
98fb6a6e1fe4: Mounted from encheres-immo-staging 
ba7d6d7afbc6: Mounted from encheres-immo-staging 
6165c009d884: Mounted from encheres-immo-staging 
7ab40bdb0830: Mounted from encheres-immo-staging 
27b6892302df: Mounted from encheres-immo-staging 
a1c01e366b99: Layer already exists 
deployment-01GMB700AD20NHX2KFE54CJEG9: digest: sha256:e9a7aee4d1c498f7566cd6846653e73828f4cfaf7c28f076ac2240220dfb9fe3 size: 1991
--> Pushing image done
image: registry.fly.io/encheres-immo-production:deployment-01GMB700AD20NHX2KFE54CJEG9
image size: 235 MB
==> Creating release
--> release v200 created

--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: /app/bin/encheres_immo eval EncheresImmo.Release.migrate

--> This release will not be available until the release command succeeds.
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: f447594)...
         Setting up swapspace version 1, size = 536866816 bytes
         UUID=c411d3a2-a47d-45a0-bc53-500e05dc495c
         Preparing to run: `/app/bin/encheres_immo eval EncheresImmo.Release.migrate` as nobody
         2022/12/15 15:38:18 listening on [fdaa:0:33f5:a7b:aec1:1206:14d9:2]:22 (DNS: [fdaa::3]:53)
         15:38:20.320 [warning] Description: 'Authenticity is not established by certificate path validation'
              Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'
         15:38:20.320 [warning] Description: 'Authenticity is not established by certificate path validation'
              Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'
         15:38:21.005 [info] Migrations already up
         Starting clean up.
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: f447594)...
         Setting up swapspace version 1, size = 536866816 bytes
         UUID=c411d3a2-a47d-45a0-bc53-500e05dc495c
         Preparing to run: `/app/bin/encheres_immo eval EncheresImmo.Release.migrate` as nobody
         2022/12/15 15:38:18 listening on [fdaa:0:33f5:a7b:aec1:1206:14d9:2]:22 (DNS: [fdaa::3]:53)
         15:38:20.320 [warning] Description: 'Authenticity is not established by certificate path validation'
              Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'
         15:38:20.320 [warning] Description: 'Authenticity is not established by certificate path validation'
              Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'
         15:38:21.005 [info] Migrations already up
         Starting clean up.
==> Monitoring deployment

 1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v200 failed - Failed due to unhealthy allocations - rolling back to job version 199 and deploying as v201 

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort
4

Good news @chrismccord : I managed to resolve the issue. I still can’t make new deployments because “Failed due to unhealthy allocations” but I’ll make a new topic + an email to support about it tomorrow.