All the sudden 502 on external access after deploying my app

Questions / Help
1

Hello, I just deployed a new version but I get all a sudden a 502 from the fly proxy server. The internal health checks (both TCP as well as HTTP) are all clear and based on the (granted limited) logs the app seems to have started as expected.

My frontend app that uses the internal route to the backend service can also connect to the service as expected.

Also just for testing purposes tried to restart my app but that didn’t work either. Still seeing the same behavior.

2

Oh and in case it helps this are example response headers for one of this requests:

date: Mon, 28 Nov 2022 17:25:19 GMT
fly-request-id: 01GJZM8HRZ8RG3P2B9YRYPZJYC-sea
server: Fly/b76dc087 (2022-11-18)
via: 2 fly.io
3

This is now fixed. Looking into what happened. It appears the particular host you were on had some issues. Can’t tell quite what for sure yet.

4

Thank you so much. Yes it is now working. It would be great to get some more details when you had a chance. Also in case, this happens again is there a better/faster way to get it resolved? We are planning to release our prod application early December and there an outage like this would be a lot worst for us then our test system.

5

A little while ago we turned the switch on on our new state replication systems. It’s vastly different than what we had before and a lot more efficient.

That said, it’s essentially a completely different thing. Even if it’s better, it breaks in different, unforeseen ways :). Such is the life when you do rewrites. In this case we didn’t have much choice because of how much was changing: everything.

As far as I can tell, for many hours this specific node was not participating in state replication.

I can’t yet pinpoint what happened because most everything else was working just fine.

Now, I’m going to add alerts for this so we don’t miss it next time.

All I can give you here, without lying or overpromising is: it gets better every day. I’m keeping a close eye on it and shipping improvements almost every day.

We have plans that include priority support. We can’t guarantee instant responses, but we do go through them faster.

2 Likes