Unable to reach recently deployed apps at .fly.dev address (502s)

Trying to reach my apps at their .fly.dev addresses, but they just load for a really long time then I get a 502 http code. Seems like it’s not on my end, but I can’t be sure.

I’m seeing the start-logs in the apps, but I’m not seeing any logs which indicate that the apps are ever reached by http requests.

updates:
I’m not sure how much information I should give, but the certificates are still valid and the apps were working a couple days ago. Looking at the network info in dev tools, there’s no response for minutes.

Response Headers in dev tools after request fails:

date: Wed, 12 Oct 2022 19:30:42 GMT
fly-request-id: 01GF6TRC4CE232HCRMAY1FNGJ4-chi
server: Fly/54d1d920f (2022-09-30)
via: 2 fly.io

Tried pinging the .fly.dev addresses, and they’re reachable. Tried curling them as well, but they just hang for a long time.

Barring any Fly-related proxy issues, 502 means the app (upstream) is non-responsive. And so, check if your app process is healthy, up, and running, if you haven’t already?

fly status --all -a <app-name> shows app status (incl health checks, if defined).

➜ fly status --all -a app      
App
  Name     = app          
  Owner    = me           
  Version  = 2               
  Status   = running           
  Hostname = app.fly.dev  
  Platform = nomad             

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS      	RESTARTS	CREATED              
deadbeef	app    	434 ⇡  	aws   	run    	running	2 total, 2 passing 	0       	2022-09-28T21:38:31Z
beefdead	app    	425    	aws   	evict  	failed 	2 total, 2 critical	0       	2022-08-18T13:25:20Z	

…then, one may check for individual VM statuses, like so:

➜  fly vm status deadbeef -a app      
Instance
  ID            = deadbeef              
  Process       = app                   
  Version       = 2                   
  Region        = aws                   
  Desired       = run                   
  Status        = running               
  Health Checks = 2 total, 2 passing    
  Restarts      = 0                     
  Created       = 2020-08-18T01:08:31Z  

Events
TIMESTAMP           	TYPE      	MESSAGE                 
2020-08-18T01:08:11Z	Received  	Task received by client	
2020-08-18T01:08:41Z	Task Setup	Building Task Directory	
2020-08-18T01:09:01Z	Started   	Task started by client 	

Checks
ID                      SERVICE  	STATE  	     OUTPUT                                 
<some-id-0>	tcp-880 	passing	    TCP connect  172.x.y.z:880: Success 	
<some-id-1>	tcp-100	    passing	    TCP connect  172.a.b.c:100: Success	

fly ssh console -s -a <app-name> to gain access to the VM and verify if netstat -alpn or top show up expected app processes listening on expected port numbers.

1 Like

@ignoramous, thank you! I’ve checked app and vm statuses, and they both look fine (very similar to the ones you posted). But, when I attempt to ssh console in using fly ssh console -s -a <app-name>, I get:
Error look up <app-name>: failed fetching instances for "<app-name>": can't find deployed regions for <app-name>, I’m looking into why this is now.

Also here are edited statuses where is appify.

flyctl status -all -a appify:

App
  Name     = appify          
  Owner    = personal             
  Version  = 9                    
  Status   = running              
  Hostname = appify.fly.dev  
  Platform = nomad                

Deployment Status
  ID          = b89eb79d-b8a6-aa7c-3108-a4b991800c28         
  Version     = v9                                           
  Status      = successful                                   
  Description = Deployment completed successfully            
  Instances   = 1 desired, 1 placed, 1 healthy, 0 unhealthy  

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS 	HEALTH CHECKS     	RESTARTS	CREATED  
35e31070	app    	9      	ord   	run    	running	1 total, 1 passing	0       	1h5m ago	

fly vm status 35e31070 -a appify:

Instance
  ID            = 35e31070            
  Process       = app                 
  Version       = 9                   
  Region        = ord                 
  Desired       = run                 
  Status        = running             
  Health Checks = 1 total, 1 passing  
  Restarts      = 0                   
  Created       = 1h7m ago            

Events
TIMESTAMP           	TYPE      	MESSAGE                 
2022-10-12T18:42:45Z	Received  	Task received by client	
2022-10-12T18:42:45Z	Task Setup	Building Task Directory	
2022-10-12T18:42:59Z	Started   	Task started by client 	

Checks
ID                              	SERVICE 	STATE  	OUTPUT                                 
3df2415693844068640885b45074b954	tcp-8080	passing	TCP connect 172.19.0.122:8080: Success	

Recent Logs
1 Like

Was able to console in using the info you posted here: fly ssh console fails with `Error host unavailable: host was not found in DNS` - #3 by ldrndll.

I’m seeing the process when running top, but I can see startup logs in the fly.io web UI, just not logs indicating any requests make it to the app (this wasn’t clear in my original post, I’ve since updated).

netstat -alpn shows:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 :::8080                 :::*                    LISTEN      580/node
tcp        0      0 fdaa:0:ad6c:a7b:e770:35e3:1070:2:22 :::*                    LISTEN      522/hallpass
tcp        0     36 fdaa:0:ad6c:a7b:e770:35e3:1070:2:22 fdaa:0:ad6c:a7b:9016:0:a:0:30147 ESTABLISHED 522/hallpass
tcp        0      0 fdaa:0:ad6c:a7b:e770:35e3:1070:2:56628 fdaa:0:ad6c:a7b:9ad9:2:23e4:2:5432 ESTABLISHED 580/node
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  3      [ ]         STREAM     CONNECTED       4395 1/init              
unix  3      [ ]         STREAM     CONNECTED       4394 1/init 

My understanding of these logs is pretty shaky, but it looks like it’s showing what is expected, the app is listening on 8080.

1 Like

Ugh… Just went ahead and tried a redeploy and it’s working now. Didn’t change anything, and startup logs seem identical to the previous deployment.

1 Like

Interesting. Seems this isn’t a one-off problem. I just deployed a bugfix and the app became unreachable again. So, this time I redeployed straight away, and now it’s working again.

1 Like

This could be an issue with our state being inconsistent and getting jammed. We’re looking into this and making improvements.

2 Likes

@jerome, wonderful to hear!

Also, since this is my first post, I want to mention that fly.io has been my favorite platform to use for side-project deployments. I’ve also used aws, gcp, azure, heroku, digital ocean, and maybe some others that I’m forgetting. Digital ocean is a close second, but fly.io seems to be much faster and more flexible.

It’s really been a joy to use.

3 Likes