"Error 1: Undocumented" after deploy & missing logs

I’m facing a strange issue with my LiveView app: I just did a fresh staging deploy and immediately after got the “Error 1 Undocumented” issue. I’ve seen this refer to hitting a concurrency limit (e.g. in proxy error.message=["Undocumented"]) but the metrics on the fly dashboard say I have 1 connection to staging.

Here’s my staging logs

2021-11-23T16:47:29.645 proxy[ab37eb8b] chi [error] Error 1: Undocumented
2021-11-23T16:47:30.837 app[72d6c75a] ord [info] 16:47:30.836 request_id=Fro6yVG39f9AWpsAAANh [info] GET /campaigns
2021-11-23T16:47:30.910 app[72d6c75a] ord [info] 16:47:30.909 request_id=Fro6yVG39f9AWpsAAANh [info] Sent 200 in 73ms
2021-11-23T16:47:38.655 app[72d6c75a] ord [info] 16:47:38.655 [info] CONNECTED TO Phoenix.LiveView.Socket in 36µs
2021-11-23T16:47:38.655 app[72d6c75a] ord [info]   Transport: :websocket
2021-11-23T16:47:38.655 app[72d6c75a] ord [info]   Serializer: Phoenix.Socket.V2.JSONSerializer
2021-11-23T16:47:38.655 app[72d6c75a] ord [info]   Parameters: %{"_csrf_token" => "DwkwHy4_KhpDOxUGFDEeDxYOaFtVUzZPzKzxatRMtHFPPVFYzyZ02jix", "_mounts" => "0", "vsn" => "2.0.0"}

I think this is related to the issue I had in production this morning where after that error my app was down

2021-11-23T14:27:24.637 proxy[2daf3929] chi [error] error.code=1 error.message="Undocumented" request.method="GET" request.url="/live/websocket?_csrf_token=AUVZZzk0NQIkWzY8LytTLxcmOTkXc3Mtxr53ZgZjMjDSfMaxzwjRmG8N&_mounts=6&vsn=2.0.0" request.id="01FN6K2GN5H3NY674P1BAXJCFE" response.status=502
2021-11-23T14:32:45.052 proxy chi [error] error.code=1002 error.message="No suitable (healthy) instance found to handle request" request.method="GET" request.url="/live/websocket?_csrf_token=ZwE4CwFhExsnYUdsUmpVZUAABi5YQTBG6lyCJ8PHASr4f9x39FdDk3Ev&_mounts=40&vsn=2.0.0" request.id="01FN6KBFS9GB4EGKHK9H2JV5CS" response.status=502
2021-11-23T14:32:45.095 proxy chi [error] error.code=1002 error.message="No suitable (healthy) instance found to handle request" request.method="GET" request.url="/live/websocket?_csrf_token=P3UdGAobIj0PYEEcewoEYyo8GS5XXy8GFBqLiHMUfQ3s2l64GmJE-kde&_mounts=7&vsn=2.0.0" request.id="01FN6KBFTDTJ1TE0ZFPJD0CFEX" response.status=502

Production is back up and working fine, there hasn’t been another deploy or any restarts, but I no longer get any logs for the instance (2daf3929) even as the app is responding to traffic.

This all happened after what I thought was a trivial change/deploy last night. Is it possible that there was some downtime/incident on fly’s end or is there a bug with my code or config I’m hitting?

For reference:

There’s a little more information here Router errors 502 of POST requests "Undocumented"

What’s the fly status on the app? It’s useful to know how many instances are running and where they are.

In this particular case, it seems possible that the connection was cut ( the umbrella / catch all error.code=1) as the app went down / was terminated for whatever reason (fly checks list should tell you which health checks are failing). There seem to have been no good instances available on the next couple of requests (error.code=1002).

fly status says

Instances
ID       PROCESS VERSION REGION DESIRED STATUS  HEALTH CHECKS      RESTARTS CREATED
2daf3929 app     122     yyz    run     running 1 total, 1 passing 0        15h15m ago

The uptime corresponds with the deploy last night and there are zero restarts. The app is up (e.g. when I open it in a browser), but I’m no longer getting any recent logs in flyctl logs, and my test with deploying to staging indicates that I should be getting normal logs.