Multi-region deployment of redis and having trouble

kentcdodds · July 15, 2021, 8:39pm

I just had a great live stream with @kurt to get my app, database, and redis cache deployed to multiple regions. After having deployed, I’m getting this error:

2021-07-15T20:29:13.941018529Z app[19947c62] dfw [info] REDIS REPLICA (dfw) ERROR: Error: getaddrinfo ENOTFOUND dfw.kcd-redis.fly.dev
2021-07-15T20:29:13.943340513Z app[19947c62] dfw [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-15T20:29:13.943949686Z app[19947c62] dfw [info]   errno: -3008,
2021-07-15T20:29:13.944733607Z app[19947c62] dfw [info]   code: 'ENOTFOUND',
2021-07-15T20:29:13.945287398Z app[19947c62] dfw [info]   syscall: 'getaddrinfo',
2021-07-15T20:29:13.946055643Z app[19947c62] dfw [info]   hostname: 'dfw.kcd-redis.fly.dev'
2021-07-15T20:29:13.946209687Z app[19947c62] dfw [info] }

I’m not sure why this isn’t working, because I think it should be. My repo is here: GitHub - kentcdodds/redis-geo-cache: A global Redis cache

Also, output for fly volumes list:

ID                   NAME         SIZE REGION ATTACHED VM CREATED AT  
vol_okgj545p51vy2wzp redis_server 10GB dfw    cf9e4f3e    2 weeks ago

I think the problem is that trying to connect with the region subdomain is not working. If I remove the region subdomain then it connects fine. Am I misunderstanding how this is supposed to work?

Another interesting fact is that there’s a key for fly_region that was added to my redis db and the value is dfw which makes sense I suppose. But I would still expect to be able to connect to it via the region subdomain right?

kurt · July 15, 2021, 8:46pm

Ah! Try dfw.kcd-redis.internal. The regional subdomains only work over internal addresses. You’ll also need to connect to port 6379.

kentcdodds · July 15, 2021, 8:51pm

Hmmm… That’s not working. I tried dfw.kcd-redis.internal as well as dfw.kcd-redis.fly.internal and dfw.kcd-redis.fly.dev.internal. All with port 6379. I’m connected to wireguard as well and I’ve verified that my wireguard is working by connecting to my production postgres db and that works fine. So I think I’m just getting the redis URL wrong somehow.

kurt · July 15, 2021, 9:12pm

I had the wrong port in the Redis readme until someone fixed it for me a few hours ago.

It should be something like: redis://x:password@kcd-redis.internal:6379.

This same node app can resolve the postgres address fine, right?

kentcdodds · July 15, 2021, 9:16pm

Two things confuse me about this…

x:password@, unless I’m mistaken, there’s no username for redis, so that should be simply password@ right?
kcd-redis.internal, I thought I’m supposed to have the region subdomain. So shouldn’t that be dfw.kcd-redis.internal?

I’m using AnotherRedisDesktopManager to test my connection to this.

kurt · July 15, 2021, 9:22pm

Oh yes, kcd-redis.internal returns all the Redis instance. dfw.kcd-redis.internal will only return IPs in Dallas. It’s useful to try the top level private name for debugging, but if one works they both should.

Are you connecting from your local machine over wireguard? DNS over wireguard is really brittle, try this from your local machine when you’re connected:

ping6 dfw.kcd-redis.internal
If that doesn’t work, open a new terminal and try again.
If that doesn’t work, try ping6 fdaa:0:23df:a7b:7f:0:281c:2

AnotherRedisDesktopManager may not be using the proper Wireguard DNS for lookups, but redis-cli will work if ping does.

The password needs a : in front of it in the URL. redis://:<password> or redis://x:<password> are both fine.

kentcdodds · July 15, 2021, 9:25pm

PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:7f:0:281c:2
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=0 hlim=62 time=72.432 ms
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=1 hlim=62 time=71.015 ms
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=2 hlim=62 time=72.978 ms
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=3 hlim=62 time=71.461 ms
^C
--- dfw.kcd-redis.internal ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 71.015/71.972/72.978/0.775 ms

I’ll trust that means that it’s actually working and try to deploy with that set up.

Would be a real bummer if I can’t use AnotherRedisDesktopManager Interestingly, I can still connect to my redis db via kcd-redis.fly.dev:10000.

kentcdodds · July 15, 2021, 9:32pm

Unfortunately, updating my redis URL doesn’t work:

fly secrets set REDIS_URL="redis://x:my-password@kcd-redis.internal:6379"

After it’s deployed, when I hit the app I get these logs:

2021-07-15T21:30:49.570077747Z runner[c9bc91d9] dfw [info] Shutting down virtual machine
2021-07-15T21:30:49.617766815Z app[c9bc91d9] dfw [info] Sending signal SIGINT to main child process w/ PID 507
2021-07-15T21:31:09.493072783Z app[0c5c5e17] dfw [info] REDIS REPLICA (dfw) ERROR: Error: getaddrinfo EAI_AGAIN dfw.kcd-redis.internal
2021-07-15T21:31:09.495600135Z app[0c5c5e17] dfw [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-15T21:31:09.496317655Z app[0c5c5e17] dfw [info]   errno: -3001,
2021-07-15T21:31:09.497136421Z app[0c5c5e17] dfw [info]   code: 'EAI_AGAIN',
2021-07-15T21:31:09.497720371Z app[0c5c5e17] dfw [info]   syscall: 'getaddrinfo',
2021-07-15T21:31:09.498240765Z app[0c5c5e17] dfw [info]   hostname: 'dfw.kcd-redis.internal'
2021-07-15T21:31:09.498358468Z app[0c5c5e17] dfw [info] }

thomas · July 15, 2021, 9:45pm

Hey, Kent! Can you give me the name of the app you’re deploying there? (A DM is fine). Also: is there a way you can deploy it so that it’ll hang and stay up after that lookup failure?

kentcdodds · July 15, 2021, 9:54pm

Thanks! Here’s the app: remix-kentcdodds/fly.toml at 9ab90898318dae85d33ff7e14900f5b8f6e203a5 · kentcdodds/remix-kentcdodds · GitHub

I could probably eventually get it working without connecting to redis if you need that.

thomas · July 15, 2021, 9:59pm

Weird. I see it running, and I can see that it can resolve kcd-redis.internal from the container context. Lemme look at the code, back in a bit.

thomas · July 15, 2021, 10:21pm

My suspicion right now — I’m still working to clarify this — is that you need to pass the address family through to the Redis library, because Node (or the Redis library) defaults to IPv4-only.

thomas · July 15, 2021, 10:56pm

Can you try something like:

redis.createClient({url: primary.toString(), family: "IPv6"})

kentcdodds · July 15, 2021, 11:01pm

Sure, deploying that now.

kentcdodds · July 15, 2021, 11:16pm

Sorry it’s taking so long… Got :face_palm: · kentcdodds/remix-kentcdodds@c4bae12 · GitHub so I did: --remote-only still doesn't work · kentcdodds/remix-kentcdodds@a703f5c · GitHub

Hopefully this one deploys

kentcdodds · July 15, 2021, 11:27pm

Hey! It works! https://kent.dev

kentcdodds · July 16, 2021, 4:19am

Ok, so the primary region works fine, but the other regions aren’t working. I’ve got this filling up my logs:



2021-07-16T04:18:31.200958488Z app[63c472b1] hkg [info] REDIS REPLICA (hkg) ERROR: Error: getaddrinfo ENOTFOUND hkg.kcd-redis.internal
2021-07-16T04:18:31.202644305Z app[63c472b1] hkg [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:31.202935990Z app[63c472b1] hkg [info]   errno: -3008,
2021-07-16T04:18:31.203240890Z app[63c472b1] hkg [info]   code: 'ENOTFOUND',
2021-07-16T04:18:31.203625487Z app[63c472b1] hkg [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:31.204191128Z app[63c472b1] hkg [info]   hostname: 'hkg.kcd-redis.internal'
2021-07-16T04:18:31.204289620Z app[63c472b1] hkg [info] }
2021-07-16T04:18:36.037865063Z app[63c472b1] hkg [info] REDIS REPLICA (hkg) ERROR: Error: getaddrinfo ENOTFOUND hkg.kcd-redis.internal
2021-07-16T04:18:36.039962627Z app[63c472b1] hkg [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:36.040282615Z app[63c472b1] hkg [info]   errno: -3008,
2021-07-16T04:18:36.040633152Z app[63c472b1] hkg [info]   code: 'ENOTFOUND',
2021-07-16T04:18:36.041004660Z app[63c472b1] hkg [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:36.041521031Z app[63c472b1] hkg [info]   hostname: 'hkg.kcd-redis.internal'
2021-07-16T04:18:36.041619490Z app[63c472b1] hkg [info] }
2021-07-16T04:18:44.286905631Z app[63c472b1] hkg [info] REDIS REPLICA (hkg) ERROR: Error: getaddrinfo ENOTFOUND hkg.kcd-redis.internal
2021-07-16T04:18:44.289875390Z app[63c472b1] hkg [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:44.290700962Z app[63c472b1] hkg [info]   errno: -3008,
2021-07-16T04:18:44.291773251Z app[63c472b1] hkg [info]   code: 'ENOTFOUND',
2021-07-16T04:18:44.293035517Z app[63c472b1] hkg [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:44.294159338Z app[63c472b1] hkg [info]   hostname: 'hkg.kcd-redis.internal'
2021-07-16T04:18:44.294261261Z app[63c472b1] hkg [info] }
2021-07-16T04:18:45.278506358Z proxy[63c472b1] hkg [info] Health check status changed 'critical' => 'passing'
2021-07-16T04:18:45.609588814Z app[8e3d8178] hkg [info] REDIS REPLICA (hkg) ERROR: Error: getaddrinfo ENOTFOUND hkg.kcd-redis.internal
2021-07-16T04:18:45.612738100Z app[8e3d8178] hkg [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:45.613033333Z app[8e3d8178] hkg [info]   errno: -3008,
2021-07-16T04:18:45.613395241Z app[8e3d8178] hkg [info]   code: 'ENOTFOUND',
2021-07-16T04:18:45.613829746Z app[8e3d8178] hkg [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:45.614504223Z app[8e3d8178] hkg [info]   hostname: 'hkg.kcd-redis.internal'
2021-07-16T04:18:45.614633464Z app[8e3d8178] hkg [info] }
2021-07-16T04:18:46.131659290Z app[902c25cb] scl [info] REDIS REPLICA (scl) ERROR: Error: getaddrinfo ENOTFOUND scl.kcd-redis.internal
2021-07-16T04:18:46.134104957Z app[902c25cb] scl [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:46.134750162Z app[902c25cb] scl [info]   errno: -3008,
2021-07-16T04:18:46.135803839Z app[902c25cb] scl [info]   code: 'ENOTFOUND',
2021-07-16T04:18:46.136661846Z app[902c25cb] scl [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:46.137660356Z app[902c25cb] scl [info]   hostname: 'scl.kcd-redis.internal'
2021-07-16T04:18:46.137774781Z app[902c25cb] scl [info] }
2021-07-16T04:18:50.567226593Z app[4b76b228] syd [info] REDIS REPLICA (syd) ERROR: Error: getaddrinfo ENOTFOUND syd.kcd-redis.internal
2021-07-16T04:18:50.569416097Z app[4b76b228] syd [info]     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-07-16T04:18:50.570019420Z app[4b76b228] syd [info]   errno: -3008,
2021-07-16T04:18:50.570833261Z app[4b76b228] syd [info]   code: 'ENOTFOUND',
2021-07-16T04:18:50.571774418Z app[4b76b228] syd [info]   syscall: 'getaddrinfo',
2021-07-16T04:18:50.573021217Z app[4b76b228] syd [info]   hostname: 'syd.kcd-redis.internal'
2021-07-16T04:18:50.573252672Z app[4b76b228] syd [info] }

I’m not sure what to try here…

kentcdodds · July 16, 2021, 4:23am

I was able to successfully ping each of those redis dbs:

~/Desktop/redis-geo-cache (main) 🦍
$ ping6 dfw.kcd-redis.internal
PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:7f:0:281c:2
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=0 hlim=62 time=101.535 ms
16 bytes from fdaa:0:23df:a7b:7f:0:281c:2, icmp_seq=1 hlim=62 time=73.152 ms
^C
--- dfw.kcd-redis.internal ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 73.152/87.344/101.535/14.191 ms
~/Desktop/redis-geo-cache (main) 🦍
$ ping6 hkg.kcd-redis.internal
PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:acc:0:2b55:2
16 bytes from fdaa:0:23df:a7b:acc:0:2b55:2, icmp_seq=0 hlim=62 time=394.897 ms
16 bytes from fdaa:0:23df:a7b:acc:0:2b55:2, icmp_seq=1 hlim=62 time=225.799 ms
^C
--- hkg.kcd-redis.internal ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 225.799/310.348/394.897/84.549 ms
~/Desktop/redis-geo-cache (main) 🦍
$ ping6 scl.kcd-redis.internal
PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:6b:0:2b56:2
16 bytes from fdaa:0:23df:a7b:6b:0:2b56:2, icmp_seq=0 hlim=62 time=194.249 ms
16 bytes from fdaa:0:23df:a7b:6b:0:2b56:2, icmp_seq=1 hlim=62 time=190.601 ms
^C
--- scl.kcd-redis.internal ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 190.601/192.425/194.249/1.824 ms
~/Desktop/redis-geo-cache (main) 🦍
$ ping6 syd.kcd-redis.internal
PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:b92:0:2b57:2
16 bytes from fdaa:0:23df:a7b:b92:0:2b57:2, icmp_seq=0 hlim=62 time=535.388 ms
16 bytes from fdaa:0:23df:a7b:b92:0:2b57:2, icmp_seq=1 hlim=62 time=356.874 ms
^C
--- syd.kcd-redis.internal ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 356.874/446.131/535.388/89.257 ms
~/Desktop/redis-geo-cache (main) 🦍
$ ping6 ams.kcd-redis.internal
PING6(56=40+8+8 bytes) fdaa:0:23df:a7b:bea:0:a:2 --> fdaa:0:23df:a7b:aa3:0:2b54:2
16 bytes from fdaa:0:23df:a7b:aa3:0:2b54:2, icmp_seq=0 hlim=62 time=146.757 ms
16 bytes from fdaa:0:23df:a7b:aa3:0:2b54:2, icmp_seq=1 hlim=62 time=157.533 ms
^C
--- ams.kcd-redis.internal ping6 statistics ---
3 packets transmitted, 2 packets received, 33.3% packet loss
round-trip min/avg/max/std-dev = 146.757/152.145/157.533/5.388 ms

So I know they’re up. I’m just having a hard time connecting I guess… Ideas are welcome.

kentcdodds · July 16, 2021, 4:50am

Actually it seems to be working now. Maybe it just took a second for the connection to be happy? Not sure. But I did a WebPageTest from Sydney and it worked! WebPageTest Test Result - Sydney : kent.dev/blog - 07/16/21 04:49:35

Topic		Replies	Views
Deploying redis image to multiple regions Questions / Help	4	741	August 10, 2022
The multi-region application does not work in one region. Questions / Help	8	529	October 19, 2021
Host redis on fly (or some other distributed caching mechanism?) Build debugging	11	809	June 1, 2024
do apps deploy to multiple regions automatically? Questions / Help distributed	2	666	March 1, 2022
DNS error using Redis subscribe over Fly	4	389	June 1, 2021

Multi-region deployment of redis and having trouble

Related topics