Connect to Redis app via internal network failing

1

After following this guide: Redis - standalone Redis Server I have a redis deployment successfully running, but can not get my app server to connect to it.

Using redisio (nodejs) to connect and getting the following error:

Error: getaddrinfo ENOTFOUND app_server_name.internal

Is there something I am missing? I am a bit confused by the ports from the guide above 6379 vs 10000.

Here is my connection URL:

REDIS_URL=“redis://:REDIS_PASSWORD@redis_app_name.internal:10000”

Thanks!

app = "redis-app-name"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[[mounts]]
source      = "redis_server"
destination = "/data"

[[services]]
  http_checks = []
  internal_port = 6379
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [[services.ports]]
    handlers = []
    port = "10000"
2

Internally you need to connect to port 6379. Port 10000 is exposed to the internet, so you may want to remove the whole [[services]] block if you want to restrict access!

3

Ah that makes more sense, I thought that could be the case, let me give that a go and report back!

@jsierles - you are always a HUGE help and SUPER responsive, we really appreciate it!

4

Hey @jsierles gave this a try and connection refused, I believe the only time I saw it working when guessing the REDIS_URL was when i used the IPv4 address, still can’t seem to get the .internal address working :frowning:

REDIS_URL=“redis://:REDIS_PASSWORD@redis-server-app-name.internal:6379”

app = "redis-server-app-name"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[[mounts]]
source      = "redis_server"
destination = "/data"
5

Also you’ll need to remove --bind 0.0.0.0 to get IPv6 working. We need to update that guide as it’s very outdated! This repo has a good config: GitHub - fly-apps/redis: Launch a Redis server on Fly

6

Hey Josh, thanks for the quick reply on the weekend!

I am not sure I follow the last message, I believe I basically have the exact same config as the repo you mentioned above, could you explain further?

Just confirmed, using ipv4 vs app-name.internal is working, but seems I should be able to use the internal dns based on docs.

Also, side question, why can we simply not just use the image=alpine/redis image over dockerfile and the shell script?

Being that this guide is outdated, is there possibly a simpler setup for the redis deploy as a whole?

Thanks again,

Dan

7

If your startup script has the argument --bind 0.0.0.0, Redis will not correctly bind to the Ipv6 address which is what internal DNS will resolve to. The linked repo fixed that, but the guide still includes the bind argument.

You can use the image, but the script offers few improvements you couldn’t get otherwise, such as setting sysctl vm.overcommit_memory=1. That’s important for low memory situations to not cause data loss. Also, it allows you to enable password auth, or make any other config change.

We will soon have a better answer for setting up Redis - among other things - with good defaults. Stay tuned!

8

Gotcha, so at the moment this argument doesn’t exist anywhere in the code base, assuming this will need to be confused via ioredis?

Side note, any interest internally that would present a service / app that is similar to fly-Postgres but for redis to simplify redis deployments?

Thanks,

9

Yes we’ll definitely have something like fly postgres for Redis.

Which argument are you referring to with ioredis?

1 Like
10

Awesome to hear!

I completely missed the --bind argument in the shell script. I was thinking this would be a argument in the ioredis client.

Here is my latest script that is deployed without a password or bind 0.0.0.0

#!/bin/sh
sysctl vm.overcommit_memory=1
sysctl net.core.somaxconn=1024
redis-server --dir /data/ --appendonly yes

After doing so, the app-name.internal is still not working.

11

So in the mist of trying a million different configurations time after time, I am not 100% sure what works and does not work anymore, but it does seem there is something wrong with the configuration that allows private network access.

I believe its now working with the [[services]] block added back in with the exposed port=“10000” and using ipv4 address, the internal address won’t work no matter what I try.

REDIS_URL = redis://:PASSWORD@IPv4_ADDRESS:10000

Can’t seem to get connected from another app in the same org without the [[services]] block. Any ideas?

12

@danwetherald have you managed to keep Redis running? If so, could you share your config?

I’ve spent the last hour tweaking a new redis config and simply cannot connect to the running redis server successfully. Every time I connect to the server it crashes. I’ve rolled-back to using keydb for now, but I’m having to manually restart such instances a couple of times a day because it also crashes without any error output.

13

Our Redis examples and docs are all over the place, but this Redis app should work as is over the private network on Fly: GitHub - fly-apps/redis-geo-cache: A global Redis cache

You must set a redis password to connect from other VMs (or externally). If you don’t have a redis password set as a secret, it won’t allow connections from remote IPs.

You probably should not use [[services]] with Redis on Fly. Redis really shouldn’t be exposed to the public internet, and that’s what services do.

Debugging this can be painful, though. If you have VMs crashing for no reason, try running:

fly status --all
fly vm status <failing-vm-id>

The event log near the top will tell you why it exited. If you have services defined, it might exit because health checks are failing. It might also be exiting due to OOMs or the process exiting for some other reason.

14

Ah, this makes sense, I was trying with and without a password in the hopes to simplify the setup and get a successful connection within the private network, as I 100% agree with you, redis should never need to be available publically.

@kurt - When ever I removed the [[services]] I was not able to connect at all, any ideas? The only way I have been able to get this working at the moment is with [[services]] configured to open up on port 10000 and using public IPv4 address.

I am going to duplicate the repo above config and give it a shot, are you using the app-name.internal:6379 from your app servers to connect to this redis instance?

Thanks,

Dan

15

Yes, <name>.internal:6379 is the right address.

You might have to tweak your Node driver config to use that, though, I seem to remember Node drivers not looking for IPv6 addresses by default. You might need to pass this through to your connection code:

'family' : '6'
16

Ah you are probably right! - GitHub - redis/ioredis: 🚀 A robust, performance-focused, and full-featured Redis client for Node.js. but it looks like its 6 not IPv6 for anyone else landing here for a solution.

'family' : 6

Let me give this a try, this is a much more robust setup with multiple region support, etc.

Will report back soon,

Thanks

1 Like
17

Hey @kurt - just wanted to share an update, after replicating the configuration from the repo mentioned above with an updated strategy for deploying redis not only in a single region but also in multiple clustering regions, we have lift off :rocket:

I believe the “main” issue we were having was probably related to the IPv6 configuration in our redis node client, but also not always requiring the password to allow the app to accept other internal apps within the fly private netowrk.

Thanks again!

18

Hit this too, so for anyone who comes across this in the future, yes, take care to set the family attribute, like so:

... new Redis({
    port: 6379,
    family: 6,
    host: 'cool-fly-redis-app.internal',
    password: '123',
});
1 Like
19

What is one supposed to do if they dont have access to this Redis config to allow for IPv6? Is there any other workaround?