Intermittent Connection Errors / Timeouts connecting Django to Redis

Django
1

Hi there,

I have a Django app running on Fly. I am using upstash redis as a cache and message broker for a celery worker process.

The problem is, no matter what I try I seem to get seemingly random connection errors - usually a handful every hour. The error message I get is: ConnectionError: Connection closed by server.. Retrying always seems to work. This happens randomly on different operations that involve redis. Here is a graph of the errors in the last day:

image

This ranges from 0 to 5 per hour.

I’ve read lots of threads on this topic on the forum but haven’t found a solution. Things I have tried:

  1. Upgrading my app to v2.
  2. Upgrading to a paid redis instance.
  3. Adding a ping/heartbeat function to my app to hit redis every 5 minutes.
  4. Appending /0 to my connection string.

Any other suggestions on how I can resolve this? Is Django + upstash just inherently not reliable on fly?

thanks,
Cory

1 Like
2

Just to update on this, I’ve now gone the route of bypassing upstash entirely and setting up my own Redis instance. Since cutting over have seen no new errors:

image

Documenting the steps I took in case anyone else finds this:

  1. Cloned and deployed this repo, by following these steps
  2. Figured out the internal DNS from this guide, which was <region>.<app_name>.internal
  3. Set my connection string in my Django app to redis://:<password>@region.appname.internal:6379/0

Too bad upstash is so unreliable. It would have saved me a lot of time. But at least the problem is solved now.

Cory

3 Likes
3

Just wanted to say thanks for posting this. Was running into the same issue and never would have found this solution without your post!

1 Like
4

I’ve also encountered this issue the last few weeks. I’m using Redis-py + FastAPI with Upstash, so its definitely not just a Django issue. Most likely the Fly proxy or the Upstash instance has some sort of disconnect rule after a certain amount of time. It would be good if someone from the Fly/Upstash team could take a look at why its happening, and/or suggest ways of handling these exceptions, as it seems intermittent. Sometimes my cache pings get through, and then sometimes they result in:

raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
1 Like
5

I’ve also been experiencing connection errors with Sidekiq. They started occurring in July. Before that I hadn’t experienced any issues. I would expect more stability from a paid offering.

Are there any plans to address this?

6

Thanks for this. Facing exact same issue, curious why no one else is complaining since its pretty glaring and bad for any app!

All documented here: upstash redis timeouts - #53 by chachra

Might just go with self redis (or keydb or firefly?). Thanks!

2 Likes
7

Yes, I find it very strange that people haven’t complained more about this. Self-hosting redis has been very reliable for my needs!

2 Likes