.internal DNS returns large number of invalid IP addresses

Questions / Help
#1

fly dig is returning 65 IP addresses for one of our apps which only has one instance (and has only ever had 9 instances in total). Most of these IP addresses do not appear to work, and other Fly services cannot talk to this service as a result.

I have tried restarting and re-deploying the app, but neither fixed the problem. Every other app I have checked on Fly seems fine.

$ fly status
App
  Name     = codeday-labs-gql          
  Owner    = codeday                   
  Version  = 9                         
  Status   = running                   
  Hostname = codeday-labs-gql.fly.dev  

Deployment Status
  ID          = f93ceb2d-c9b1-5ea1-9147-976101fd24da         
  Version     = v9                                           
  Status      = successful                                   
  Description = Deployment completed successfully            
  Instances   = 1 desired, 1 placed, 1 healthy, 0 unhealthy  

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS	RESTARTS	CREATED   
2b6af384	app    	9      	lax(B)	run    	running	             	0       	3m13s ago

fly dig codeday-labs-gql.internal -a codeday-gql
;; opcode: QUERY, status: NOERROR, id: 12671
;; flags: qr rd; QUERY: 1, ANSWER: 65, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;codeday-labs-gql.internal.	IN	 AAAA

;; ANSWER SECTION:
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:fcd3:b003:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:74c4:6969:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:7191:a733:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:86ad:aa70:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:64fa:4645:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:68e1:b83d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:2f1a:8960:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:5aed:177a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:e4b3:dfe:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:15cc:fe26:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:1fec:bcb:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:d494:fbb1:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:e431:1e96:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:15f8:8e2:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:6473:eb20:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:78bd:b793:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:d240:1db:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:c799:526e:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:a132:6e29:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:8d96:f71a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:7007:6e01:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:9447:29e7:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:6943:94a8:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:4c0:4362:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:14e2:2244:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:531:c744:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d0ef:f510:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:489b:360:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:637d:29d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e392:b38b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d0f9:570b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e573:55bd:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:a69e:2b0d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:4c48:6fd5:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e933:e658:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:eff:e61b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:c7da:e035:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:a86d:7b82:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:8adb:b8b2:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:863c:bf17:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:50a1:cda7:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d337:dc84:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:54c2:e32e:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:b142:2d37:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:a52c:4a6a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:68cf:dada:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:8c69:bbee:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:5c70:fe6c:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:7de2:b971:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:21be:1404:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:5763:2867:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:b9e9:8e72:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e769:c802:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:f022:f5bf:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:c2f5:1232:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:803e:bac8:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:f42e:dd50:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:3daa:b495:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:b13e:649c:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:6f53:a8e0:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:4516:62da:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:8f4:1254:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:b907:6cac:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:58d1:204a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:85:2b6a:f384:2

Other DNS (e.g. top5.nearest.of) also return invalid IPs.

#2

That’s super weird! I bounced the DNS on the worker you’re deployed on, which looks like it’s cleared that up, but I’m investigating now to see what’s happening there.

1 Like
#3

@thomas bug is back again, and once again it is returning exactly 65 entries:

fly dig -a codeday-labs-gql codeday-labs-gql.internal                                                                                                                                                                                ✘ 1
;; opcode: QUERY, status: NOERROR, id: 29899
;; flags: qr rd; QUERY: 1, ANSWER: 65, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;codeday-labs-gql.internal.	IN	 AAAA

;; ANSWER SECTION:
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:fcd3:b003:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:74c4:6969:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:7191:a733:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:86ad:aa70:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:64fa:4645:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:68e1:b83d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:2f1a:8960:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:5aed:177a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:e4b3:dfe:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:15cc:fe26:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:1fec:bcb:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:d494:fbb1:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:e431:1e96:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:15f8:8e2:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:6473:eb20:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:78bd:b793:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:d240:1db:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:c799:526e:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:a132:6e29:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:8d96:f71a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:7007:6e01:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:9447:29e7:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:6943:94a8:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:4c0:4362:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:14e2:2244:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:531:c744:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d0ef:f510:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:489b:360:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:637d:29d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e392:b38b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d0f9:570b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e573:55bd:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:a69e:2b0d:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:4c48:6fd5:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e933:e658:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:eff:e61b:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:c7da:e035:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:a86d:7b82:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:8adb:b8b2:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:863c:bf17:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:50a1:cda7:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:d337:dc84:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:54c2:e32e:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:b142:2d37:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:a52c:4a6a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:68cf:dada:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:8c69:bbee:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:5c70:fe6c:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:7de2:b971:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:21be:1404:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:5763:2867:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:b9e9:8e72:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:e769:c802:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:f022:f5bf:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:c2f5:1232:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:803e:bac8:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:f42e:dd50:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:3daa:b495:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2d30:b13e:649c:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2c60:6f53:a8e0:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:4516:62da:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:8f4:1254:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:b907:6cac:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:2dbb:58d1:204a:2
codeday-labs-gql.internal.	5	IN	AAAA	fdaa:0:39fa:a7b:85:2b6a:f384:2
#4

Folks, this is still broken for us and you’ve already said this is an internal problem for you. Can someone please take a look at restarting our DNS worker?

#5

Kicked it, it looks OK now. We’re investigating whether this is something distinctive about the way your app is running.

#6

Thanks; is there a way to force the app to associate to a new DNS worker without destroying and re-creating the app from scratch?

I’d be surprised if this is anything special about the app – the container is pretty stock Alpine and the TS code is based on the same template of most of the rest of the apps we’re using on Fly.

#7

Hey! I think we’ve tracked this down (thanks!). We’re getting phantom updates from worker servers we decommissioned. Kurt says we’re going to give up building platforms and do to-do lists instead.

I’m writing some code real quick to scrub these entries off all the servers in our fleet.

1 Like
#8

Glad to hear it, (well other than the to-do lists). thank you!