Redis reference not found by app

Questions / Help
1

Hello!

I’ve got a workflow now, where I am deploying via GH Actions :champagne: and I am seeing an error preventing deployment. I use the internal Redis URL found at FLY_REDIS_CACHE_URL and on every deployment, I see the following error and failure for my ‘web’ app:

2021/09/10 02:22:48 dial tcp: lookup redis on [fdaa::3]:53: no such host

Any idea what I should do to move past this roadblock? Is the flyctl tool used in the action the same as the version found on Hombrew? I am currently running:

flyctl v0.0.238 darwin/amd64 Commit: 4f402ad BuildDate: 2021-09-09T16:00:46Z

Thanks for the :eyes:

2

I created a new service and can confirm I see the redis logs. However the no such host error still occurs.

3

It looks like it’s trying to look up the hostname redis, which wouldn’t work. If you’re trying to point to a Redis in the same org, you’ll need to use <appname>.internal.

4

Hey Kurt! As always thanks for being available for CFT work. I already have the app doing exactly as you said. I am now concerned that my secrets from another environment are being read in this environment causing the issue. The redis reference is from the development docker network DNS entry docker-compose created. Let’s assume I put it there on the remote and want it off!

So I tried that - unsetting all secrets. I called flyctl secrets unset k=v ....

The response was:

Error No change detected to secrets

I also tried to check that I could ping Redis (I set up my own) and did not get what I expected.

echo "PING" | nc <app-name>.internal 6379 did not return any response.

I’d like to reset my secrets for the ‘web’ application and then redeploy them. What are your thoughts?

The biggest area of concern is the stickiness of the secrets.

Possibly related:

5

Can you try flyctl secrets unset k (without the =v)? I believe this tried unseting a secret named k=v.

6

I will do this now. Thanks @jerome

7

Alright I made the change I am getting the same state from the link in the ‘Possibly related’ comment. Trying to set the secrets now results in ‘no change detected to secrets’

8

Digging more into communicating with redis.

Using netcat I expected to get a response but I do not (assume my tunnel works). Here are the results:

❯ echo "PING" | nc <app-name>.internal 6379

❯ echo "PING" | nc 0.0.0.0 6379
+PONG

Note the lack of PONG response in the first. The second invocation is my workstation docker-compse setup.

This is strange because my connection appears to be accepted but no response. Is that a reasonable assessment?

Are these different representations of the same problem?

Even after updating the secrets it still shows the following:

2021-09-10T17:21:46.309510640Z app[6d997346] dfw [info] 2021/09/10 17:21:46 dial tcp: lookup redis on [fdaa::3]:53: no such host

Thanks team!

9

You probably need nc -6 to use IPv6.

This is a wild guess, but is it possible your Dockerfile or Docker compose are setting an env variable that overrides the secret / env variable set through Fly?

10

Same result (no response):

echo "PING" | nc -6 sendcoffee-redis.internal 6379

is it possible your Dockerfile or Docker compose are setting an env variable that overrides the secret / env variable set through Fly

The only way I could determine this (I believe) is by retrieving the secrets current running in the VM. My secrets workflow is done entirely from localhost. I decrypt secrets run something like this:

cat .env | tr '\n' ' ' | xargs flyctl secrets set --app <appname>

Wait for the process to finish. The GH Action step I take it literally from the Readme

Run superfly/flyctl-actions@1.1
  with:
    args: deploy . --app <appname> --dockerfile ./Dockerfile --remote-only --build-arg buildNumber=1222137295 --build-arg gitRevision=<rev>
  env:
    FLY_API_TOKEN: ***
    GOPRIVATE: github.com/...
    GOROOT: /opt/hostedtoolcache/go/1.17.0/x64
    GOMODCACHE: /home/runner/go/pkg/mod
11

Does ping -6 sendcoffee-redis.internal work?

You can see what secrets/env variables are set in your VM by connecting over ssh, then running env:

> fly ssh console
env

It is suspicious that it’s trying to resolve the name that docker compose uses for your local redis service. It seems unlikely that you ever set a secret like FLY_REDIS_URL=redis://redis, but that redis name it’s trying to resolve is coming from somewhere.

12

Does ping -6 sendcoffee-redis.internal work?

Perfectly.

Since I’ve moved to using my own internal redis I use this:

REDIS_URL=redis://sendcoffee-redis.internal:6379

The env command has hung for the last 60 secs. I will retry now. I suspect the fact that it is not running is the cause.

13

Woohoo - better news. It still fails, but it does appear to no longer reference redis

2021-09-10T18:42:59Z [info] 2021/09/10 18:42:59 dial tcp [fdaa:0:2071:a7b:12de:0:3b5b:2]:6379: connect: connection refused

This to me implies something in the app may not be functioning. But perhaps the lack of a pong response gives me pause.

14

Here is what I have learned so far.

Since redis is running I assumed it was a network issue.

I have added these lines, but the API VM still cannot connect:

redis-cli -h ipv4 => network timeout 
redis-cli -u redis://<redissvcname>.internal  => Could not connect to Redis at <redissvcname>.internal:6379: Connection refused

[[services]]
  internal_port = 6379
  protocol      = "tcp"

  [[services.tcp_checks]]
    interval = 10000
    timeout  = 200

I then tried to the diagnostics:

ping6

ping6 <appname>.internal
PING6(56=40+8+8 bytes) fdaa:0:2071:a7b:bea:0:a:2 --> fdaa:0:2071:a7b:12de:0:3b5b:2
16 bytes from fdaa:0:2071:a7b:12de:0:3b5b:2, icmp_seq=0 hlim=62 time=87.726 ms
16 bytes from fdaa:0:2071:a7b:12de:0:3b5b:2, icmp_seq=1 hlim=62 time=69.468 ms
16 bytes from fdaa:0:2071:a7b:12de:0:3b5b:2, icmp_seq=2 hlim=62 time=67.744 ms

dig

$ dig -t TXT <appname>.internal => no response
$ dig -t TXT <appname>.internal
;; Warning, extra type option

And then I re-ran the commands I normally run on Github Actions on my workstation.

2021-09-10T21:32:51Z [info] Preparing kernel init
2021-09-10T21:32:51Z [info] Configuring firecracker
2021-09-10T21:32:51Z [info] Starting virtual machine
2021-09-10T21:32:51Z [info] Starting init (commit: 721b5c7)...
2021-09-10T21:32:51Z [info] Running: `/usr/local/<appname>/app` as nonroot
2021-09-10T21:32:51Z [info] 2021/09/10 21:32:51 listening on [fdaa:0:2071:a7b:2203:10c4:3071:2]:22 (DNS: [fdaa::3]:53)
2021-09-10T21:32:53Z [info] 2021/09/10 21:32:53 dial tcp [fdaa:0:2071:a7b:12de:0:3b5b:2]:6379: connect: connection refused
2021-09-10T21:32:54Z [info] Main child exited normally with code: 1
2021-09-10T21:32:54Z [info] Starting clean up.
2021-09-10T21:32:55Z [info] Health check status changed 'warning' => 'passing'
***vXXX failed - Failed due to unhealthy allocations - rolling back to job version XXX and deploying as vXXX

Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
15

You’ll need dig aaaa <appname>.internal to get addresses.

What Redis did you deploy? It seems like maybe it’s not listening on IPv6, but is on IPv4?

You may need to try passing --bind :: through on the redis start command.

16 
redis-server --bind 0.0.0.0 --dir /data/ --appendonly yes

This is what I have now.

17

0.0.0.0 will only bind to ipv4. You will need to use :: to also bind to ipv6

18

I have updated my configuration and deployed. I will try the redis CLI with the ipv6 address.