upstash redis timeouts

Hello! Thanks for you help!

I just deployed a new Rails application and I started to see the same exact error. Found your post and installed the gem, but feels like didn’t help.

Should I create a complete new instance with a dedicated Redis installation or there is other workaround I can look?

I also gave up and deployed keydb to fly as a separate app instead of using upstash. Haven’t seen the issue since.

Hey @tello , when did you experience this problem? Can you share your redis endpoint (without credentials) so we can check the health of the your DB in Upstash side?

@bi1 I would say since yesterday

Private URL = redis://default:TOKEN@fly-lifi-redis.upstash.io

I will say that I’m not seeing timeouts with a simple application. I’d be curious if you could reproduce my results and if so, identify what might be different about your application.

mkdir demo
cd demo
curl https://fly.io/docs/rails/cookbooks/databases/Dockerfile1 -o Dockerfile
fly launch

Accept all of the defaults (you don’t need a postgres database, and you don’t need an additional redis instance).

Now run

fly secrets set REDIS_URL=redis://default:TOKEN@fly-lifi-redis.upstash.io`

(with your TOKEN of course)

Finally:

fly deploy
fly open
fly logs

If you open a second browser window you can see updates being pushed in realtime using web sockets.

I haven’t let it go idle for a full hour, but if there is no activity for an hour the socket will be closed and that will take an unpatched Rails down.

Not sure if this helps, but this timeout thing improved in the past days. Still this is an issue every now and then.

Not sure if I can help in any way to completely dismiss this or is something I need to live with at least for some time. Any advise is appreciated. @bi1

hey @tello
Do you see any pattern for the connection issues?
Like is it happening randomly or with same interval like every hour etc?

Thanks for your answer @bi1. Seems like it happens randomly. Not sure if this helps in some way, but most of the errors come from background jobs with Sidekiq. I’m executing around 10K jobs daily.

Can you share with me latest timestamp that happened so we can specifically check for that date?

Last error seen at: Mar 3, 2023 10:51:19 AM CST
First error logged at: Feb 13, 2023 5:14:09 AM CST

@bi1 sorry it took me long to reply. For some reason I did receive the notification email.