At first I thought this might be because I wasn’t returning the Content-Length header in the response, but that doesn’t seem to be it. I’m not really sure what the proxy is expecting. Would someone be able to help me diagnose?
I understand this isn’t very exciting, but I’d be super grateful if someone from Fly could look up these error IDs and give me anymore information. Not understanding how to work with the proxy at times like this is the only thing stopping me from going all-in on Fly.
Sorry for the delay, we’ve had unrelated issues Yesterday. We usually respond pretty fast.
“App connection idle” means your app’s connection was idle (no bytes sent or received) for 60s and we closed it.
“App connection problem” can mean a few things, usually it means the connection (between your app and us) was broken, timed out connecting or closed at an unexpected time.
In this case, I see it timed out connecting. This would mean your app was unable to accept a connection within a few seconds. Connecting to an instance from the local host (which is what we do) should only ever take a few milliseconds. The timeout is set at 2 seconds if I recall correctly. This should be ample time, if your app cannot accept the connection within that time, something is very wrong.
@jerome I continue to be baffled by this. I think the proxy may be changing POST requests in some subtle way. The reason my app is returning ‘app connection problem’ is because it is choking when I call request.json() or request.text() inside my request handlers. I’ve tried putting them inside try/catch blocks, but they simply time out.
The same handlers work as expected when run locally. As soon as they’re up on Fly, things stop working. Does the proxy modify requests in any particular way? The bodies of my post requests are large, but not exceptionally so.
The body should be passed as-is. Can you provide a curl request that reproduces the issue? So I can test this myself.
If this is sensitive information, please send via DM.
I looked at your current apps and couldn’t find much. There were connection timeouts attempting to reach your app. Looks like the event loop is blocked on your node app?
There’s a tight timeout for connection to an app locally, mostly because it should take ~1-5ms to connect (the timeout is ~2 seconds iirc, roughly 400-2000x what it should normally take an app to accept a connection).
What kind of app is this? I assume node.js because of request.text() and request.json(). Are you manually accepting connections in a loop and something is holding it up from accepting more connections concurrently?
I just tried the curl you send me via DM again and it worked this time.
We did make some changes to how much data we’ll buffer. However, I’m pretty sure the initial limit of 1MiB wasn’t being hit by that request you provided. Our changes may have fixed it as a side effect
@gwil are you still seeing issues? It could be specific regions.
@Jerome out of curiosity, though, what’s the buffer on the fly-replay feature? I don’t think the replay header is sent back to the browser, so how big a request can Fly hold and replay?
@gwil we just figured out that if an app is writing too much to stdout, it can stall VMs. In our case, Rails was logging all the JSON body we were passing it and everything was locked up trying to write to stdout.
Is it possible your app is logging a lot to stdout / stderr when you get these large bodies?
We’ve bumped it to 10MiB. It may or may not stay that way
