Redis started throwing Read timed out

1

Hi, my app that is using Redis just randomly started throwing Read timed out for no reason
Are there any issues with redis at this moment?

2

already automatically resolved :slight_smile:

1 Like
3

and once again, in the evening I have this problem…
any ideas what could be the root cause?

4

Hello @eyouu
Do you have any specific timestamp or time range for the timeouts?

So we can focus that time range.

Thanks

5

Hi @bi1
It was around [22:30 - 02:30]
One day before it started near [01:00 - 02:00]
All times in Europe/Kiev timezone.

I’m using 90$ plan and my app has high cache usage (yesterday it was around 84GB Bandwidt) and I started thinking maybe it’s because of this load but before I was using even more than 84GB and everything was ok.

6

Also, I had this high usage during the whole month but only past 2 days in a row I get this Read TIme Out error and only in the evening.

7

Once again at 22:00 (Kiev) is started throwing errors…
Before 22:00 it worked fine

8

Sorry for the trouble. Are you still seeing these issues?

9

@jsierles thanks for the response

Luckily no. After I moved my app instances from Amsterdam to France (it immediately helped), this error disappeared.

10

Now I have this issue in France. I tried to do the same workaround and moved to Frankfurt and now this issue acts this way:

10:53-10:54 → it doesn’t work
10:54-10:55 → it works
10:55-10:56 → it doesn’t work
10:56-10:57 → it works

So 1 minute I received timeouts and 1 minute it works fine. I don’t know what to do anymore :frowning_face:

Pls help @jsierles @bi1

11

image
image1972Ă—478 135 KB

On the graph you can see how every 1-2 minutes commands per sec = 0 because of that error

12

hi @eyouu those graphs might be misleading.

We are looking internally now.

I will inform you soon.

13

Thanks, but it’s not only about graphs. I also check it in my app and I can confirm that at that down points, my app doesn’t work because Upstash Redis doesn’t response.

14

@bi1 @jsierles Additional info to help with analysis:

I just switched all my instances to [Frankfurt, Germany] region and I just stopped receiving timeouts. So, sometimes switching to other region helps.
As a guess, maybe there is some limits on Upstash side that blocks requests or something like this…

Please let me know if there is any investigation outcomes since it’s very crucial for me to get it fixed.

15

This is how cache latency changed after switching to Frankfurt

image
image1930Ă—498 58.3 KB

16

Hey @eyouu
We have internally worked with fly team. The issue has been identified underlying network infrastructure. The fly team applied the fix.

You can move back to CDG region.

Best

1 Like
17

Here’s the incident report: Fly.io Status - Disk issue on application hosts

1 Like
18

@jsierles @bi1
I switched back to CDG and now I don’t see the errors, big thanks for the high quality service :heart:

2 Likes
19

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.