3 apps deployed fine, 4th app won't deploy

1

I am migrating some apps over from Heroku, and in the process I’m also moving them from buildpacks to Dockerfiles (as I’ve found that buildpacks on Fly are very slow compared to Heroku, and it seems like Docker is the more “fly native” method anyway).

To replicate Heroku pipelines, I’m creating two Fly apps for each Heroku app:

  • one for “staging”
  • one for “production”

The same Dockerfile and fly.toml are shared by both the staging & production apps.
The app value is omitted from the fly.toml file, and passed manually to the fly deploy command, e.g.

  • fly deploy -a my-app-staging
  • fly deploy -a my-app-production

The only differences between staging and production are:

  1. the app name
  2. the DATABASE_URL secret (staging and prod each have their own Postgres db, still hosted on Heroku for now).

So far I have successfully deployed to Fly:

  1. a Ruby/Sinatra app to both “staging” and “production”
  2. a Rails app to “staging”

The problem I’m having is deploying that same Rails app to “production”.
To reiterate: this is the exact same image that deployed successfully to staging (the build logs even show each layer as “Mounted from {name of staging app}”).

When I deploy, the only output shown in the “Monitor” page of the dashboard is:

$ fly logs -a lootprod

Waiting for logs...

Eventually the deployment fails with

--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

Running fly status --all returns

❯ fly status --all -a lootprod    
App
  Name     = lootprod          
  Owner    = personal          
  Version  = 4                 
  Status   = dead              
  Hostname = lootprod.fly.dev  
  Platform = nomad             

Deployment Status
  ID          = 09cc6ddb-6d60-11ef-1f63-19f2bb259235                                           
  Version     = v4                                                                             
  Status      = failed                                                                         
  Description = Failed due to unhealthy allocations - no stable job version to auto revert to  
  Instances   = 1 desired, 1 placed, 0 healthy, 1 unhealthy                                    

Instances
ID      	PROCESS	VERSION	REGION	DESIRED	STATUS	HEALTH CHECKS	RESTARTS	CREATED              
965fd436	app    	4 ⇡    	syd   	run    	failed	             	0       	6m42s ago           	
21bb4079	app    	3      	syd   	stop   	failed	             	0       	49m38s ago          	
11db4822	app    	1      	syd   	stop   	failed	             	0       	56m20s ago          	
a669b7e9	app    	0      	syd   	run    	failed	             	0       	2022-09-20T23:16:24Z

Running fly checks list -a lootprod returns

❯ fly checks list -a lootprod 
Health Checks for lootprod
  NAME | STATUS | ALLOCATION | REGION | TYPE | LAST UPDATED | OUTPUT  
-------*--------*------------*--------*------*--------------*---------

Running fly logs -i 965fd436 -a lootprod to view the logs of one of the failed VMs returns nothing (and I hit CTRL-C to exit)

❯ fly logs -i 965fd436 -a lootprod  
^C

I can’t use fly ssh console to debug, as there’s no VM to connect to.

I have triple checked the DATABASE_URL secret is correctly configured for the production app I’m trying to deploy.

Not sure if this is relevant, but I’ll mention it now just in case:
As per above, my “personal” org currently has three apps successfully deployed (2 x “staging” and 1 x “production”).
The app I am having trouble with will be my fourth, and I’m aware that this will take me out of the free tier; however I do have a payment method set up and my account status is in “good standing”.
Could it be something to do with billing?

I’m running out of ideas on how to troubleshoot this deployment. Can anyone think of what else could be preventing a second deployment of the same app, under a different name?

2

Throwing out ideas which are likely not anything to do with it.

  1. Are you connecting any volumes to the application? If they are setup in staging, but not prod, that may be the issue.
  2. You are using the personal org, so why not try connecting the staging db to lootprod and see what happens?
  3. Do you have to run any migrations first? I see there are no checks, so it shouldn’t be killed by healthchecks?
1 Like
3

Great suggestions, thanks.

  1. No, no volumes are used
  2. That’s a good idea (probably should have thought of that myself). Will try and report back.
  3. No migrations (well, normally there would be, but I’ve temporarily removed the release_command in case that was the problem. There are no pending schema changes anyway, so migrations would have been a no-op for this release)

Appreciate the ideas.

4

You were spot on, @tj1.

I set DATABASE_URL for the production app to match the staging app, and it deployed successfully.

So clearly there’s a problem connecting to the production DATABASE_URL…and I just need to figure out why (I’m 100% certain that the value was correct, as it was copied directly from Heroku).

Anyway, that’s my problem to figure out. You’ve been a great help, thanks.

2 Likes