Hello! Thanks for you help!
I just deployed a new Rails application and I started to see the same exact error. Found your post and installed the gem, but feels like didn’t help.
Should I create a complete new instance with a dedicated Redis installation or there is other workaround I can look?
I also gave up and deployed keydb to fly as a separate app instead of using upstash. Haven’t seen the issue since.
Hey @tello , when did you experience this problem? Can you share your redis endpoint (without credentials) so we can check the health of the your DB in Upstash side?
@bi1 I would say since yesterday
Private URL = redis://default:TOKEN@fly-lifi-redis.upstash.io
I will say that I’m not seeing timeouts with a simple application. I’d be curious if you could reproduce my results and if so, identify what might be different about your application.
mkdir demo
cd demo
curl https://fly.io/docs/rails/cookbooks/databases/Dockerfile1 -o Dockerfile
fly launch
Accept all of the defaults (you don’t need a postgres database, and you don’t need an additional redis instance).
Now run
fly secrets set REDIS_URL=redis://default:TOKEN@fly-lifi-redis.upstash.io`
(with your TOKEN of course)
Finally:
fly deploy
fly open
fly logs
If you open a second browser window you can see updates being pushed in realtime using web sockets.
I haven’t let it go idle for a full hour, but if there is no activity for an hour the socket will be closed and that will take an unpatched Rails down.
Not sure if this helps, but this timeout thing improved in the past days. Still this is an issue every now and then.
Not sure if I can help in any way to completely dismiss this or is something I need to live with at least for some time. Any advise is appreciated. @bi1
hey @tello
Do you see any pattern for the connection issues?
Like is it happening randomly or with same interval like every hour etc?
Thanks for your answer @bi1. Seems like it happens randomly. Not sure if this helps in some way, but most of the errors come from background jobs with Sidekiq. I’m executing around 10K jobs daily.
Can you share with me latest timestamp that happened so we can specifically check for that date?
Last error seen at: Mar 3, 2023 10:51:19 AM CST
First error logged at: Feb 13, 2023 5:14:09 AM CST
@bi1 sorry it took me long to reply. For some reason I did receive the notification email.
Feels like I am facing random upstash issues and timeouts as well. Connecting seems fine from console or from my local terminal 
@chachra Can you provide more details about your issues? For example: exception logs and the Fly.io region you’re using.
EWR region, Redis paid $10/month plan (200mb I think), remove stale data is on.
exception ConnectionError: Connection closed by server.
Error 18:10:38
redis
GET ':1:ddddd_6_1692161079.566111_44_1691726290.182672'
{db.operation: GET, redis.command: GET, redis.is_cluster: False, redis.key: :1:ddddd_6_1692161079.566111_44_1691726290.182672}
If this helps, on the application logs in fly UI:
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] [2023-08-16 09:05:30 +0000] [267] [ERROR] Socket error processing request.
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] Traceback (most recent call last):
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py”, line 271, in handle
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] keepalive = self.handle_request(req, conn)
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py”, line 343, in handle_request
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] util.reraise(*sys.exc_info())
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/util.py”, line 626, in reraise
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] raise value
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py”, line 329, in handle_request
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] resp.write(item)
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/http/wsgi.py”, line 346, in write
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] util.write(self.sock, arg, self.chunked)
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] File “/usr/local/lib/python3.11/site-packages/gunicorn/util.py”, line 287, in write
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] sock.sendall(data)
2023-08-16T09:05:30.199 app[e784eee6f396e8] ewr [info] TimeoutError: [Errno 110] Connection timed out
2023-08-16T13:54:14.326 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T15:08:37.308 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T15:09:37.850 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T15:09:52.681 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T15:20:11.009 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T16:29:17.944 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T16:29:44.070 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:01:08.277 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:01:52.144 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:02:05.945 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:45:09.672 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:50:10.795 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T17:55:58.964 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T20:58:05.455 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T21:01:46.925 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
2023-08-16T21:04:24.242 proxy[e784eee6f396e8] ewr [error] could not make HTTP request to instance: connection error: timed out
sometimes it works as well. I’ve got everything down to one machine per app. so one redis (no replication region), one postgres, one (performance) webapp container etc.
After failing a bunch of times it starts working again… till it doesn’t 
My issue seems related to or similar to this? Upstash Redis and "could not send HTTP request to instance: connection error: timed out" logs
The logs posted don’t mention Redis - is the previous error from an exception reporting tool? I also don’t see that the timestamps in the logs match with the Redis error timestamp.
hmm ok. Someone reported the same: Upstash Redis - Connection lost (ECONNRESET) (Redis::ConnectionError - #25 by cluzstudio
Also this thread: Intermittent Connection Errors / Timeouts connecting Django to Redis
All my timeouts are not recorded. This came from fly logs… sometimes it leads to an exception which Sentry is catching. Once it starts working it works fine! Seems to be related to a connection / proxy / warm up issue to me.
