Upstash Redis and "could not send HTTP request to instance: connection error: timed out" logs

thewilkybarkid · January 18, 2023, 2:05pm

I’ve made a few changes to improve the end-user experience (well, make it less bad). So the app now serves error pages if Redis commands take longer than 2 seconds, rather than hanging and eventually timing out (prereview.org/index.ts at 49914fe6994bb279116d761603344a16d763bcd9 · PREreview/prereview.org · GitHub).

I’m also now logging the ioredis events, so I can see it connecting/disconnecting/reconnecting etc (prereview.org/index.ts at 49914fe6994bb279116d761603344a16d763bcd9 · PREreview/prereview.org · GitHub).

Hopefully, this should be enough to determine if the issue is the Redis connection.

thewilkybarkid · January 18, 2023, 3:57pm

I believe it is a connection/latency problem. I’ve just had Redis timeouts (set to 2s) when connecting through the replica in fra. A deployment happened, and the fresh connection worked fine.

Not long beforehand, the other app instance in iad successfully restarted its connection according to my app’s logs.

ebenoist · January 18, 2023, 5:15pm

I’m having the same issue. I can’t deploy my applications because of timeouts to redis.

matthewford · January 18, 2023, 6:33pm

We also see the same issue with Redis using rails/sidekiq

thewilkybarkid · January 18, 2023, 7:15pm

Seems to be happening again (‘prereview’ app, connecting to the instance in fra using the Redis replica in fra).

Stephen · January 18, 2023, 9:24pm

After thinking I’d solved my similar problem in the past I’m suddenly getting it again now as well.

I’m seeing lots of Redis connection timeout errors and my app isn’t successfully getting anything out of Redis. I haven’t changed anything relevant for quite a while so don’t believe I’ve broken it.

kurt · January 18, 2023, 9:30pm

Just to clarify, you’re seeing timeouts when you try to connect to Redis? Or you’re seeing timeouts when a connection has been alive for a while.

It sounds like the former, which shouldn’t be happening. The latter is somewhat normal.

Stephen · January 18, 2023, 10:13pm

@kurt I can’t speak for the others but for me I think it is the latter (i.e. timeouts after connection alive for a while). If this latest instance of it is same as before for me then it works for a while after a deploy (like it does for @thewilkybarkid) and then starts failing after some period (not sure how long, but hours not minutes). It had been fine for me after I tweaked my Redis connection settings about 2 weeks ago, but today I’m getting problems again which I assumed might be the same as the others in this thread.

I understand it might have to reconnect if a connection has been open too long, but it seems like that re-connection might be failing. I’ve just bumped up connect and command timeouts a little and redeployed. It’s working again now so I’ll wait and see what happens.

thewilkybarkid · January 18, 2023, 10:36pm

It’s the latter for me. With half a day’s worth of logs (low traffic though), I can see the connection is closed every so often and ioredis reconnects. Other times, however, the connection remains open but is unusable (i.e. commands time out).

I might add a ping to my app’s health check to see if that helps. (It’ll be called far more often than should be needed, but it would keep it active.)

kurt · January 18, 2023, 10:38pm

ioredis might have a ping function. I think the hangs are concerning, but I’m guessing those happen because ioredis hasn’t detected the closed connection. If you can get a command to traverse the connection every ten seconds or so, I think it’ll be fine.

Can you share your ioredis config?

joncallahan · January 19, 2023, 12:00am

I am also seeing this issue and using ioredis with fly / upstash redis. My ioredis config is dead simple const redis = new Redis(process.env.REDIS_URL). I did add ?family=6 to the end of the connection string printed to my terminal after I created the redis instance in order for the initial connection to work, but now after after a stretch of no traffic I cannot connect anymore.

thewilkybarkid · January 19, 2023, 7:35am

We have the same as @joncallahan, except for setting commandTimeout (and enableAutoPipelining, which had no effect on this).

github.com

PREreview/prereview.org/blob/092a4be4ec226bc06f98e559271ad078bd69068e/src/index.ts#L23


      
          
          
const env = decodeEnv(process)()
          
          
const loggerEnv: L.LoggerEnv = {
            clock: SystemClock,
            logger: pipe(C.log, L.withShow(L.getColoredShow(L.ShowLogEntry))),
          }
          
          
const redis =
            env.REDIS_URI instanceof URL
              ? new Redis(env.REDIS_URI.href, { commandTimeout: 2 * 1000, enableAutoPipelining: true })
              : undefined
          const keyvStore = redis ? new KeyvRedis(redis) : undefined
          
          
redis?.on('connect', () => L.debug('Redis connected')(loggerEnv)())
          redis?.on('close', () => L.debug('Redis connection closed')(loggerEnv)())
          redis?.on('reconnecting', () => L.info('Redis reconnecting')(loggerEnv)())
          redis?.removeAllListeners('error')
          redis?.on('error', (error: Error) => L.errorP('Redis connection error')({ error: error.message })(loggerEnv)())
          
          
const server = app({

Stephen · January 19, 2023, 10:00am

Regarding pinging Redis… see Upstash - Redis not reachable sometimes - #8 by pontus

Looks like it might have worked for them (albeit with node-redis). Hopefully ioredis has something similar. If so, I’ll try it and see.

thewilkybarkid · January 19, 2023, 10:04am

It doesn’t seem to, but I’ve added a ping as part of our health check.

github.com/luin/ioredis

ioredis connection timeout

opened 01:21PM - 17 Nov 22 UTC

ShravaniSree

I am creating an ioredis client connection as shown below: ![image](https://use…r-images.githubusercontent.com/68904744/202457109-87776a98-6c5c-46c7-87f9-4154da7a0060.png) I am using this client in a webapp, I want to re-use the same connection forever. I have read in microsoft documentation that by default there is an idle timeout of 10 minutes and some client libraries send a ping to keep the connection alive. I wanted to understand if this is the case with ioredis. If not and a connection is lost, what is the error code?

github.com/PREreview/prereview.org

Ensure Redis responds when checking if the instance is healthy

committed 09:29AM - 19 Jan 23 UTC

thewilkybarkid

+6 -4

We're seeing cases where ioredis thinks the connection is still active, but ther…e's no response. Keeping it active might help as a workaround: the connection shouldn't then be closed due to inactivity. This change also means that the load balancer will pick up that the app isn't working correctly. Refs #589, https://community.fly.io/t/upstash-redis-and-could-not-send-http-request-to-instance-connection-error-timed-out-logs/10104/16

jsierles · January 19, 2023, 12:16pm

We’ve bumped the Redis idle timeout to a day (instead of one hour) which should help with debugging. If you’re seeing timeouts throughout a single day after a deployment, do report back here.

jsierles · January 19, 2023, 12:18pm

Sidekiq should reconnect to Redis automatically when disconnects happen. Are you seeing different behavior?

matthewford · January 19, 2023, 12:49pm

Will need to check; perhaps the error is on the rails side when sending a job. In either case moved to self hosting as it was causing too much of an issue.

thewilkybarkid · January 20, 2023, 9:33am

After 24 hours of the ping being in our health check, we had no reconnections or problems. (The idle timeout looks to have been changed a few hours after, it was fine before then though.)

I’ve opened an ioredis issue since there are a few things there:

github.com/luin/ioredis

Not always detecting closed connections on Fly.io

opened 09:31AM - 20 Jan 23 UTC

thewilkybarkid

This follows on from: https://community.fly.io/t/upstash-redis-and-could-not-sen…d-http-request-to-instance-connection-error-timed-out-logs/10104 The Redis service on Fly.io (provided by Upstash) closed idled connections after an hour, but ioredis didn't always recognise it. (They [increased it to a day](https://community.fly.io/t/upstash-redis-and-could-not-send-http-request-to-instance-connection-error-timed-out-logs/10104/22?u=thewilkybarkid) to reduce the number of errors. I've added a ping to our app's health check, which keeps the connection alive.) I added logging for the various events ioredis emits and could see that sometimes it would recognise the connection had closed and reconnect, but sometimes it didn't. When the connection was then used, any command would just time out. There are a few things here: - ioredis doesn't always recognise closed connections. - A 'command timeout event would be useful (for logging). - A regular ping setting would be useful to keep connections alive.

I don’t know if there are any details that others could provide; please do if you can!

Stephen · January 22, 2023, 7:51pm

I tried a variety of things with my ioredis settings but continued to have re-connection problems. This seems to be a recurring theme in ioredis GitHub issues and various other discussions online. Interestingly I also discovered that BullMQ (I’m using older Bull) explicitly checks if you are using Upstash and throws an error immediately if so as it doesn’t support them. In short… it’s all a bit of a confusing muddle.

So… I have put a ping to my Redis connection into a healthcheck call like thewikybarkid did. Since doing that it’s been OK, but it’s only been 24hrs so fingers crossed still.

jsierles · January 23, 2023, 8:25am

BullMQ uses Redis Streams which isn’t supported yet by Upstash. They’re working on it!