Redis time out issue (CDG)

Hi, I’m writing it here because for some reason I have no response via email for 5 days despite I’m on Scale Plan (which promises quick email support) :confused:

One year ago I had an issue with Redis connection timeout and it was fixed. More than one week ago it started reproducing again.

The issue with comments here: Redis started throwing Read timed out

Please help to resolve this.

1 Like

Hello @eyouu

Can you provide us some recent timestamps? So we can specifically check that range.

Best

Thanks for the response. I will let you know as soon as it reproducing again because I switched from cdglhr region and at this point of time it works fine (typically it starts in the evening)

If you want some exact timestamp before then for example: Aug 7, 4:21 PM (Kyiv timezone) it was reproducing with a frequency of approximately once every 5 minutes. 30 seconds I only receive connection timeout then 5 minutes it works fine and this was the way how it worked for a few days.

@bi1 if timestamps in the past is not enough for you for analysis then I can try to switch back to cdg region and I think it will be reproducing in the evening again. Please let me know

That would be very nice, I informed internally the SRE team, we will keep observing

@bi1 I just switched back to cdg and it reproduces again

2024-08-13T12:24:39.123 app[178151d7be6e78] cdg [info] Caused by: java.net.SocketTimeoutException: Read timed out

Please let me know if you are on it right now. Because I don’t want to be in this region for a long time because it inconveniences the users but if you are analyzing it now I will stay for now

@bi1 FYI: I switched back to lhr, errors were in place during whole hour when I stayed at cdg. Please let me know if you need me to switch to cdg again.

I’m not sure if it can be related, but I’m facing high HTTP response time in CDG since few days. This topic is related Really slow responses from Remix JS App on Fly.io (CDG) - #9 by alyx1 and a friend also have the same issue.

1 Like

Sorry @eyouu I don’t want to steal your topic, but the time it started to happen on my web app is matching your timing too (as you said ~a week ago)

From the HTTP app:

Fly edge:

I wonder: are you hitting the Redis instance from another fly.io machine in CDG, or externally?

1 Like

@alyx1 No problem, feel free to post comments here. I think your topic is related because in addition to my Redis problem, I also noticed high HTTP response time in CDG.

Regarding your question: I’m hitting Redis instance from fly machine in CDG.

3 Likes

Well I guess we will have to move out of CDG :frowning: It’s been happening for so many days now, no replies from support here or on Mastodon. I don’t get why not more people are impacted, or reporting the issue. Nothing was changed on our applications, the traffic is the same as before, and we all have the same issue during the same time range so…

Same. I have no response for a week already despite being on the most expensive plan :slight_smile:
Moved to LHR already but I’m afraid the issue can spread to other regions sooner or later.

1 Like

Bumping because the issue is still ongoing and the team still hasn’t addressed the issue.

Hey everyone - we managed to fix this issue. Thanks again for bringing it to our attention.

3 Likes

Hi Joshua, thank you for looking into it. The ongoing fix is working well so far, thank you so much :bowing_woman:

@eyouu Did you ever get a response in support?

@joshua-fly recently - no (my few messages were ignored). Half a year ago - yes :slight_smile:

Thank you!
I’m switching back to CDG to check if everything is ok and will give you a feedback soon.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.