Hello, I just deployed a new version but I get all a sudden a 502 from the fly proxy server. The internal health checks (both TCP as well as HTTP) are all clear and based on the (granted limited) logs the app seems to have started as expected.
My frontend app that uses the internal route to the backend service can also connect to the service as expected.
Also just for testing purposes tried to restart my app but that didn’t work either. Still seeing the same behavior.
Thank you so much. Yes it is now working. It would be great to get some more details when you had a chance. Also in case, this happens again is there a better/faster way to get it resolved? We are planning to release our prod application early December and there an outage like this would be a lot worst for us then our test system.
A little while ago we turned the switch on on our new state replication systems. It’s vastly different than what we had before and a lot more efficient.
That said, it’s essentially a completely different thing. Even if it’s better, it breaks in different, unforeseen ways :). Such is the life when you do rewrites. In this case we didn’t have much choice because of how much was changing: everything.
As far as I can tell, for many hours this specific node was not participating in state replication.
I can’t yet pinpoint what happened because most everything else was working just fine.
Now, I’m going to add alerts for this so we don’t miss it next time.
All I can give you here, without lying or overpromising is: it gets better every day. I’m keeping a close eye on it and shipping improvements almost every day.