Error logs saying "Internal problem" result in 502s

1

My health checks are all passing, however I am getting this in my flyctl logs as of 10-15 minutes ago:

error.code=2 error.message="Internal problem" request.method="GET" request.url="[redacted]" response.status=502

Looks like the Fly HTTP connection handler is down? Can anyone else confirm?

2

Which app is this? That error isn’t from an outage, but could be caused from some weird state when your app VMs changed / deployed.

3

Hi Kurt,

Thanks for the reply! App info sent to you as a PM. I did do a deployment 30min ago, so perhaps that may of done it…

4

What region is that error message coming from?

5

Hi Kurt,

Another PM sent to you with the full log excerpt. Looks like all the gateways…

6

Ok we’re digging, it’s unclear if this is a problem with the proxy or a response that we can’t handle from the app. I’ll let you know what we uncover.

7

Hi kurt,

Thanks!

More info: Little changed between my previous deploy and the current deploy – both were pushed via the fly builder. I did not change my package.json lock (it’s a Node app) at all in this deploy either. Therefore, it’ll be very very odd and unusual that this deployment is returning responses that were different than the previous build.

Also, for what it’s worth, I also deleted another app today … exact same app name as the current one, but with a -2 concatenated at the end (e.g. if the current apps name is fly-app, the one I deleted is named fly-app-2). Perhaps the proxy is getting confused with the other app that’s now destroyed?

8

This is a problem on our end, the fix is going out now. The errors are caused by stale data in our proxy state. They’re triggered on some deploys, which is why they’re intermittent like that.

9

Hi Kurt,

Wonderful to hear! So this whole issue was triggered because of my recent deployment, correct?

10

Yes. More specifically, removing instances caused it. These requests were getting routed to instances that no longer existed.

11

Okay thanks! I see our app is working again. I also did another deployment just now, and the app is still working. So this issue has been resolved – thanks!!

1 Like