Redis: I/O error

1

My Redis instance is not accessible:

~/dev/project ❯ fly redis connect
? Select a database to connect to <name> (ams) <project>
Proxying local port 16379 to remote [<ipv6>]:6379
I/O error
127.0.0.1:16379>

Not from flyctl and my application is also unable to connect (from ams, den and scl) causing constant restarts until it gives up causing downtime unfortunately.

~/dev/project ❯ fly redis status <name>
Redis
  ID             = 7zo2y0Pz5oVQ7u9K2k6
  Name           = <name>
  Plan           = Free
  Primary Region = ams
  Read Regions   =
  Private URL    = redis://default:<password>@<name>.upstash.io

Both Fly (https://status.flyio.net/) and Upstash (https://status.upstash.com/) indicate no issues but unless I am doing something wrong it is broken.

What do I do to fix this?

2

Thanks - we’ve identified this as a more general issue and are looking into it.

3

Can you give your Redis a try again?

4

I had this issue a minute ago as well. Redis was unavailable to connect from the outside. Seems back now.

Is there monitoring available for Fly managed redis? Fly-metrics does not show the app.

5

We’re working on adding Redis to Fly Metrics. Normally we would have caught this, but some internal monitoring was disabled for a pending upgrade.

1 Like
6

That is unfortunate, but it sounds like this will be caught quicker next time?

It seems to be all working again :ok_hand:

7

Yep, we’re still working through some rough edges. Thanks for your patience!

1 Like
8

@jsierles I scaled my app down to 0 processes when this ocurred (personal app, no big deal) and have checked the status page several times since then to see when the issue was resolved and I could scale back up. After seeing this thread, I scaled the app back up and everything seems to be working normally. However, https://status.flyio.net/ still says the issue is identified but unresolved? Just wanted to point that out in case it has been fully resolved.

9

@z1lk We were monitoring to make sure things continued, and left the incident in place just in case. Now it’s been marked as resolved. Thanks!

1 Like