Seeing requests to the Fly proxy in the logs


Over the months I’ve had some issues with requests (seemingly) failing at the Fly proxy. So the app never gets them and so they don’t appear in the app logs.

Like today I got a report of a random 502 error. Nothing in the logs to show that error code. All 200s. So I assume the request failed at the Fly proxy, and so never made it to an app instance. As then it would appear in the app logs.

Is there a way to see requests from the outside world?

I get it’s a bit more complicated than say, an AWS load balancer, which is owned by each separate user, not shared between all users. However you proxy does know which app a request is destined for and so shouldn’t those requests appear in the app logs?

I not, maybe a:
flyctl --app [name] --proxy logs
… type command to see logs at the proxy level?

Errors caused by our proxy do show up in logs. They look like:

2021-06-30T00:22:34.487955794Z proxy lhr [error] error.code=1002 error.message="No suitable (healthy) instance found to handle request" request.method="HEAD" request.url="/admin/signin""01F9D4NTQM2PJX6XF2SW08A786" response.status=502

(this was taken from one of your other posts: 502 errors from app ... possibly related to 525? :slight_smile:)

If there are none present, then our proxy has not produced it. Or the log got lost somewhere in transit (but I haven’t heard of such reports).

We don’t log all HTTP requests because this would mean and extra hundreds of millions of log lines per day.

If you don’t mind sending me your app name, I can take a closer look.

Ah ha … good point.

So it seems a proxy 525 does not get logged (which I guess makes sense, as the handshake fails). A proxy 502 does get logged if the cause was instance not found. As that logged line proves that. But I don’t see that same line today. So I guess the question then is whether a 502 can be caused by something else.

I’ll send a message with the details in. No hurry. Was just curious what caused it.

We can actually generate a 502 for a bunch of reasons, but they’ll all get logged like that. We don’t, however, log 502s that pass through from the app itself. I can’t remember what kind of app you’re running but it could be the source of 502s.

That makes sense. I agree that if the app makes the error, it should be logging it. I mean it shouldn’t be error-ing here given the healthcheck that returned that 502 does nothing but return a little JSON back (no database lookup, no logic, or anything that could have broken). And the app didn’t log an error here. So this particular request was seemingly lost somewhere before it got to the app.

In the subsequent messages with Jerome it appears to just be some random packet loss. Pesky internet.