Hey folks.
We are sometimes seeing httpx.RemoteProtocolError: Server disconnected without sending a response. when trying when trying to POST to our fly app, but the logs for the app don’t show any sort of errors related to that (for context/clarity: we run a fastapi app on fly). Looked around and didn’t see anything similar in the forum here, but maybe this is some kind of known issue or you might have hints? Thanks!
Apart from your app logs, there should be logs from the Fly proxy in your log stream. Do you see anything from the proxy when these errors happen?
Hm, in the normal log stream? Like in grafana and live logs on dashboard? I don’t think I’ve ever seen any logs that are not from my app there. Googled around, there seem to be some traces of fly logs --proxy , but I don’t see it in fly logs · Fly Docs and my local installation also clearly doesn’t know about it. How do I view my proxy logs?
Proxy logs come right along with your app logs, in Grafana and also in fly logs output.
Hm, not seeing any messages other than my fastapi ones. Should I constantly see proxy logs?
You should only see proxy logs if there’s something unusual going on.
Where are you posting from? Another Fly app, another cloud? And in which region?
I’m POSTing from another app (non fly) on google cloud (in some US region, can clarify further if needed), but I also saw this when POSTing dozens requests rapidly from EMEA before. Same issue though, haven’t seen any errors in my fastapi app or any other errors in the logs. Some context: this app is a DIY cdn with only two endpoints one POST to initiate the upload (quick, just returns you a new pre-signed tigris url to PUT to) and GET to get your image. It seems like sometimes when there is a spike in number of images we need to upload (e.g. someone needs to upload a few dozen and calls upload/initiate a dozen times quickly) we get this error but we also saw some connect timeouts before too in such cases from time to time. This makes me believe that something is going on on the proxy side, but I can’t really seem to find any proxy logs to read.
One more bit of context: we have 4 instances with 2 in iad and one of those being the primary node (the only one allowed to write to litefs). In our google cloud rest service we specify the preferred region to iad to avoid too many redirects (non-primary instances are made to use fly-replay if they get an upload request), but don’t specify the machine. So proxy probably has to do some work there. But before we had 2 iad instances we also had the same problem, so I don’t think adding a second one made it any worse.
How often do you see the client error?
Your app probably does have proxy logs and you’re just not seeing them ‘live’. Inside your dash you can visit “Log Search” and enter this search term:
event.provider:"proxy"
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.