upstash redis timeouts

Questions / Help
61

I’m having some similar issues. On my Rail 7 app with Upstash redis, watching the /cable route - the app tries to subscribe to a channel. Initially after a deploy, it succeeds. But, after an hour or two, the frontend is unable to receive a confirmation of subscription to /cable - even though the server is sending ping messages through.

I tried adding action-cable-redis-backport but it doesn’t seem to succeed yet. I just tried upgrading to the paid Upstash, even though I don’t seem to be triggering any limits - so I’ll report back on that.

I’m not seeing any errors reported, which is suspicious. I suspect that means it’s just timing out.

I’ve been debugging this for too much time, so I may go the route of self-hosting Redis.

62

I’m seeing Redis connection errors and timeouts in LHR today in a few orgs and apps to upstash Redis.

Using fly redis connect throws an I/O error
Error: Server closed the connection

63

Did some research around sidekiq and redis issues on GitHub and here’s what I found –

Default timeout is 1 second. Diagnostic tool is redis-cli --latency-history:

redis-cli -u (REDACTED / $REDIS_URL) --latency-history
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
min: 1, max: 17, avg: 1.60 (1270 samples) – 15.00 seconds range
min: 1, max: 20, avg: 1.58 (1275 samples) – 15.01 seconds range
min: 1, max: 21, avg: 1.57 (1275 samples) – 15.01 seconds range
min: 1, max: 18, avg: 1.53 (1278 samples) – 15.01 seconds range
min: 1, max: 22, avg: 1.62 (1269 samples) – 15.01 seconds range
min: 1, max: 25, avg: 2.28 (92 samples)^C

I did this by spinning up a fly ssh console and running the command above to my REDIS_URL. So with this in mind, it looks like the latency is quite high to my upstash.

Docs say to increase the timeout, but it may need some digging with the upstash and fly teams:

64

I followed the steps above with a redislabs instance, and I see better latency, but I still occasionally see timeouts:

root@0806603b1463e8:/app# redis-cli -u $REDIS_URL --latency-history
Warning: Using a password with ‘-a’ or ‘-u’ option on the command line interface may not be safe.
min: 0, max: 3, avg: 0.83 (1365 samples) – 15.01 seconds range
min: 0, max: 1, avg: 0.78 (1233 samples)