Stale DNS entries again

Questions / Help
1

We’ve encountered stale DNS records again on our app and deploying new versions seems to be not working
Here are some of the latest logs:

2021-08-14T13:19:07.680324565Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.6683795,"logger":"http.log.error.log0","msg":"dial tcp: lookup payments.internal on [fdaa::3]:53: no such host","request":{"remote_addr":"185.40.232.122:40406","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Content-Length":["146"],"Fly-Forwarded-Ssl":["on"],"Fly-Dispatch-Start":["t=1628947147604283;instance=d6b74b0f"],"Accept":["*/*"],"X-Request-Start":["t=1628947147604219"],"Content-Type":["application/json"],"Fly-Forwarded-Proto":["https"],"X-Forwarded-Proto":["https"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"Accept-Encoding":["gzip, deflate"],"Fly-Client-Ip":["84.244.69.73"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Fly-Request-Id":["01FD2D1TTMRCE61V8B082Q31XE"],"Via":["1.1 fly.io"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"]}},"duration":0.053964321,"status":502,"err_id":"7e2kmgrtd","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}
2021-08-14T13:19:07.693711039Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.668471,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"185.40.232.122:40406","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Content-Type":["application/json"],"Fly-Forwarded-Proto":["https"],"X-Forwarded-Proto":["https"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"Accept-Encoding":["gzip, deflate"],"X-Request-Start":["t=1628947147604219"],"Fly-Client-Ip":["84.244.69.73"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Via":["1.1 fly.io"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"],"Fly-Request-Id":["01FD2D1TTMRCE61V8B082Q31XE"],"Fly-Forwarded-Ssl":["on"],"Fly-Dispatch-Start":["t=1628947147604283;instance=d6b74b0f"],"Accept":["*/*"],"Content-Length":["146"]}},"common_log":"185.40.232.122 - - [14/Aug/2021:13:19:07 +0000] \"POST /api/auth/agents/authorize HTTP/1.1\" 502 0","duration":0.053964321,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
2021-08-14T13:19:07.706992846Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.668471,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"185.40.232.122:40406","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Fly-Client-Ip":["84.244.69.73"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"],"Fly-Request-Id":["01FD2D1TTMRCE61V8B082Q31XE"],"Via":["1.1 fly.io"],"Fly-Dispatch-Start":["t=1628947147604283;instance=d6b74b0f"],"Accept":["*/*"],"Content-Length":["146"],"Fly-Forwarded-Ssl":["on"],"Fly-Forwarded-Proto":["https"],"X-Forwarded-Proto":["https"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"Accept-Encoding":["gzip, deflate"],"X-Request-Start":["t=1628947147604219"],"Content-Type":["application/json"]}},"common_log":"185.40.232.122 - - [14/Aug/2021:13:19:07 +0000] \"POST /api/auth/agents/authorize HTTP/1.1\" 502 0","duration":0.053964321,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
2021-08-14T13:19:07.801205478Z app[d6b74b0f] lhr [info] {"level":"debug","ts":1628947147.7860308,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"payments.internal:8080","request":{"remote_addr":"185.40.232.122:40432","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Content-Length":["146"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"],"Via":["1.1 fly.io"],"Accept-Encoding":["gzip, deflate"],"Fly-Client-Ip":["84.244.69.73"],"Fly-Dispatch-Start":["t=1628947147738915;instance=d6b74b0f"],"Accept":["*/*"],"Content-Type":["application/json"],"X-Forwarded-Proto":["https"],"Fly-Request-Id":["01FD2D1TYTQ6SGR3DEVMW4DNV5"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3, 185.40.232.122"],"Fly-Forwarded-Proto":["https"],"Fly-Forwarded-Ssl":["on"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"X-Request-Start":["t=1628947147738810"]}},"duration":0.038522101,"error":"dial tcp: lookup payments.internal on [fdaa::3]:53: no such host"}
2021-08-14T13:19:07.812699517Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.7861497,"logger":"http.log.error.log0","msg":"dial tcp: lookup payments.internal on [fdaa::3]:53: no such host","request":{"remote_addr":"185.40.232.122:40432","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Accept-Encoding":["gzip, deflate"],"Content-Length":["146"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"],"Via":["1.1 fly.io"],"Accept":["*/*"],"Content-Type":["application/json"],"Fly-Client-Ip":["84.244.69.73"],"Fly-Dispatch-Start":["t=1628947147738915;instance=d6b74b0f"],"X-Forwarded-Proto":["https"],"Fly-Request-Id":["01FD2D1TYTQ6SGR3DEVMW4DNV5"],"User-Agent":["python-requests/2.25.1"],"X-Request-Start":["t=1628947147738810"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Fly-Forwarded-Proto":["https"],"Fly-Forwarded-Ssl":["on"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"]}},"duration":0.038726511,"status":502,"err_id":"nt86y43cp","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}
2021-08-14T13:19:07.823517418Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.786191,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"185.40.232.122:40432","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Accept":["*/*"],"Content-Type":["application/json"],"Fly-Client-Ip":["84.244.69.73"],"Fly-Dispatch-Start":["t=1628947147738915;instance=d6b74b0f"],"X-Forwarded-Proto":["https"],"Fly-Request-Id":["01FD2D1TYTQ6SGR3DEVMW4DNV5"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"X-Request-Start":["t=1628947147738810"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Fly-Forwarded-Proto":["https"],"Fly-Forwarded-Ssl":["on"],"Via":["1.1 fly.io"],"Accept-Encoding":["gzip, deflate"],"Content-Length":["146"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"]}},"common_log":"185.40.232.122 - - [14/Aug/2021:13:19:07 +0000] \"POST /api/auth/agents/authorize HTTP/1.1\" 502 0","duration":0.038726511,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
2021-08-14T13:19:07.834156373Z app[d6b74b0f] lhr [info] {"level":"error","ts":1628947147.786191,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"185.40.232.122:40432","proto":"HTTP/1.1","method":"POST","host":"payments.paypack.rw","uri":"/api/auth/agents/authorize","headers":{"Fly-Client-Ip":["84.244.69.73"],"Fly-Dispatch-Start":["t=1628947147738915;instance=d6b74b0f"],"Accept":["*/*"],"Content-Type":["application/json"],"X-Forwarded-Proto":["https"],"Fly-Request-Id":["01FD2D1TYTQ6SGR3DEVMW4DNV5"],"X-Forwarded-For":["84.244.69.73, 213.188.209.3"],"Fly-Forwarded-Proto":["https"],"Fly-Forwarded-Ssl":["on"],"X-Forwarded-Ssl":["on"],"Fly-Region":["fra"],"User-Agent":["python-requests/2.25.1"],"X-Request-Start":["t=1628947147738810"],"Content-Length":["146"],"Fly-Forwarded-Port":["443"],"X-Forwarded-Port":["443"],"Via":["1.1 fly.io"],"Accept-Encoding":["gzip, deflate"]}},"common_log":"185.40.232.122 - - [14/Aug/2021:13:19:07 +0000] \"POST /api/auth/agents/authorize HTTP/1.1\" 502 0","duration":0.038726511,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
2

I don’t see any stale entries for that app, but I ran some cleanup scripts anyway. Did that help?

3

It looks like the payments app is crashed, so there are no instances in DNS.

If you run fly status --all -a payments you’ll see a bunch of failed VMs. You can run fly vm status <id> on any of those and see what caused them to fail, they seem to be failing health checks and stopping.

4

Does doing a restart solve the problem

5

You’ll likely need to deploy to get it started again.

6

after deploy these are the logs:

Instance
  ID            = add00dd6
  Version       = 190
  Region        = lhr
  Desired       = stop
  Status        = failed
  Health Checks = 1 total, 1 critical
  Restarts      = 2
  Created       = 15m57s ago

Recent Events
TIMESTAMP            TYPE             MESSAGE
2021-08-14T20:32:15Z Received         Task received by client
2021-08-14T20:32:15Z Task Setup       Building Task Directory
2021-08-14T20:32:35Z Started          Task started by client
2021-08-14T20:33:28Z Restart Signaled healthcheck: check "c2768b7fa5857809b03634a3a5ad0428" unhealthy
2021-08-14T20:33:29Z Terminated       Exit Code: 130
2021-08-14T20:33:29Z Restarting       Task restarting in 1.095819437s
2021-08-14T20:33:31Z Started          Task started by client
2021-08-14T20:34:24Z Restart Signaled healthcheck: check "c2768b7fa5857809b03634a3a5ad0428" unhealthy
2021-08-14T20:34:25Z Terminated       Exit Code: 130
2021-08-14T20:34:25Z Restarting       Task restarting in 1.038820151s
2021-08-14T20:34:28Z Started          Task started by client
2021-08-14T20:35:20Z Restart Signaled healthcheck: check "c2768b7fa5857809b03634a3a5ad0428" unhealthy
2021-08-14T20:35:22Z Terminated       Exit Code: 130
2021-08-14T20:35:22Z Not Restarting   Exceeded allowed attempts 2 in interval 5m0s and mode is "fail"
2021-08-14T20:35:22Z Alloc Unhealthy  Unhealthy because of failed task

Checks
ID                               SERVICE  STATE    OUTPUT

c2768b7fa5857809b03634a3a5ad0428 tcp-8080 critical Get "http://172.19.1.98:8080/healthz": dial tcp 172.19.1.98:8080: connect: connection refused

Recent Logs
7

@kurt I destroyed the app and started from scratch and now when i try to attach/connect it to the database it fails with the following error:

LOG_LEVEL=debug flyctl pg attach --app  payments  --postgres-app database --database-name payments
Attaching...⣾ DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: AttachPostgresClusterInput!) { attachPostgresCluster(input: $input) { app { name } postgresClusterApp { name } environmentVariableName connectionString environmentVariableName } }","variables":{"input":{"appId":"payments","postgresClusterAppId":"database","databaseName":"payments"}}}
}
Attaching...⣻ DEBUG <-- 500 https://api.fly.io/graphql (1.89s) {"errors":[{"message":"An unknown error occured.","extensions":{"code":"SERVER_ERROR"}}],"data":{}}
Error An unknown error occured.
8

This is a known issue when you attach an app and a DB with that app name already exists. Do you still have the DB connection string from before you deleted the app? You need to set that as the DATABASE_URL to reuse the existing database.