Deployment stuck at Running release task (pending)

Questions / Help Build debugging
1

Hi!

My deployment is stuck at Running release task (pending).
It’s not just a CLI monitoring issue, as the release does not show up under the Activity tab in my app. The release tasks runs the usual Elixir migrations similar to your guide and has worked before. DATABASE_URL is set correctly through fly pg attach.

Can anyone please check really quick?

Cheers

2

Hi @christoph! Could you post the logs that you see during the deploy? It helps to see the details and figure things out quickly.

3 
❯ flyctl deploy
Deploying stitchkeeper-web
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.6 linux x86_64

--- cut docker build log ---

Successfully built ab0c075d8c88
Successfully tagged registry.fly.io/stitchkeeper-web:deployment-1635857014
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/stitchkeeper-web]
fe0a57fbbeef: Pushed 
4a2712f291d3: Layer already exists 
a90d11189292: Layer already exists 
c7b7cb3e3b42: Layer already exists 
e2eb06d8af82: Layer already exists 
deployment-1635857014: digest: sha256:396c673f841918254ebf8df1e7001bfee6a62717dcdf8c06544cdf3ecdfae51f size: 1363
--> Pushing image done
Image: registry.fly.io/stitchkeeper-web:deployment-1635857014
Image size: 26 MB
==> Creating release
Release v4 created
Release command detected: this new release will not be available until the command succeeds.

You can detach the terminal anytime without stopping the deployment
==> Release command
Command: /app/bin/stitchkeeper eval Stitchkeeper.Release.migrate
Running release task (pending)

Then it stops, doesn’t do anything for the past hour or so.
Sign In · Fly does not show the v4 release mentioned by the CLI.

4

@christoph Thanks! That narrows it down a bit. Can you try with the --verbose flag? That should tell us the exact problem. Actually, don’t bother, sorry. Don’t think it’ll add anything useful. Will check and get back to you.

5

Tried it anyways for good measure, and the output is exactly the same. Thanks!

6

@christoph trying to see if this is a CLI problem or a release task problem, could you run flyctl status to see what the current state of the VMs are? It’s possible the deploy went through fine but the CLI isn’t updating.

7 
❯ flyctl status
App
  Name     = stitchkeeper-web          
  Owner    = stitchkeeper              
  Version  = 3                         
  Status   = dead                      
  Hostname = stitchkeeper-web.fly.dev  

Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED

I guess the problem is as follows:

  • The release tasks needs DATABASE_URL to be set
  • Initial fly launch fails because of it not being set
  • fly pg attach needs launch to be run before (is this true? I assumed it is)
❯ flyctl secrets list
NAME                   DIGEST                           DATE      
DATABASE_URL           --- removed --- 1h42m ago

Attach seems to have worked fine.
Now that the release task has its necessary environment, it seems to fail because the initial launch has put it in an unhealthy state.

I can completely recreate and document all of my steps for you, if that helps.

8

You’re probably right about the DATABASE_URL not being picked up yet. We’ll debug this internally as well, but can you try setting a few test secrets and then a deploy again? Just fly secrets set K1=V1 and K2=V2 a couple of times and then run again?