Any idea why my deployment fails?

victorbjorklund · April 9, 2022, 7:30am

I have a standard phoenix app (no other packages than default) using the fly launch command that I’m trying to deploy but deploys keep failing on fly (for different reasons but all seems to fail with the db). I tried since yesterday (I took a break because thought perhaps temp probs with fly.io).

This is latest error message:

3f1f62edf862: Pushed 
41a3e795badb: Pushed 
7279d2fe4228: Pushed 
ba5a5fe43301: Pushed 
deployment-1649488866: digest: sha256:b8e5eecddd9758f0840482b3ae0b89bcfbd0ae0c9bcbac9b2c80fb738e66c8c6 size: 1576
--> Pushing image done
Image: registry.fly.io/superchat:deployment-1649488866
Image size: 118 MB
==> Creating release
Release v2 created
Release command detected: this new release will not be available until the command succeeds.

You can detach the terminal anytime without stopping the deployment
==> Release command
Command: /app/bin/migrate
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting init (commit: 6f9865f)...
         Preparing to run: `/app/bin/migrate` as nobody
         2022/04/09 07:23:16 listening on [fdaa:0:59ad:a7b:23c3:3162:7ff:2]:22 (DNS: [fdaa::3]:53)
         07:23:19.206 [info] == Running 20220407173957 LiveviewChat.Repo.Migrations.CreateUsersAuthTables.change/0 forward
         07:23:19.213 [info] execute "CREATE EXTENSION IF NOT EXISTS citext"
         ** (Postgrex.Error) ERROR 42501 (insufficient_privilege) permission denied to create extension "citext"
             (ecto_sql 3.7.2) lib/ecto/adapters/sql.ex:852: Ecto.Adapters.SQL.execute_ddl/4
             (ecto_sql 3.7.2) lib/ecto/migration/runner.ex:343: Ecto.Migration.Runner.log_and_execute_ddl/3
             (elixir 1.13.3) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
         Starting clean up.
Error Release command failed, deployment aborted

Not sure why I wouldnt have premissions to add citext. I deployed several phoenix apps before and not had this issue.

zblanco · April 9, 2022, 11:51pm

Running into the same issue - logging in directly to the database and giving the default postgres user permissions via ALTER USER user WITH SUPERUSER; as well as adding citext manually via CREATE EXTENSION IF NOT EXISTS citext; also results in the same error upon deploying the app.

The DB logs themselves are cycling through:

 2022-04-09T23:50:08.146 app[c0e11638] lax [info] keeper   | 2022-04-09T23:50:08.145Z INFO cmd/keeper.go:1504 our db requested role is master

2022-04-09T23:50:08.147 app[c0e11638] lax [info] keeper   | 2022-04-09T23:50:08.146Z INFO cmd/keeper.go:1542 already master

2022-04-09T23:50:08.164 app[c0e11638] lax [info] keeper   | 2022-04-09T23:50:08.164Z INFO cmd/keeper.go:1675 postgres parameters not changed

2022-04-09T23:50:08.165 app[c0e11638] lax [info] keeper   | 2022-04-09T23:50:08.164Z INFO cmd/keeper.go:1702 postgres hba entries not changed

But that might be some kind of intended polling behavior?

zblanco · April 10, 2022, 12:05am

I got it working with what I tried above by swapping to the other db within postgres and adding the extension manually.

flyctl postgres connect -a my_app-db

\c my_app
ALTER USER postgres WITH SUPERUSER;
CREATE EXTENSION IF NOT EXISTS citext;

Doesn’t seem like this should be necessary however - it worked yesterday for me with similarly configured phoenix app on a previous flyctl CLI version.

victorbjorklund · April 10, 2022, 12:44pm

Yea, I guess it might some bug with latest cli because worked for me on multiple porjects until latest cli. Anyone knows if there is a way to “downgrade” the fly cli to a prev version?

hunter · February 16, 2023, 11:04pm

I’m getting the same error now and this thread appears to be the only match. I was able to manually add the extension.

edeleon · February 18, 2023, 5:53am

I am also seeing this error.

ethanrjones97 · February 19, 2023, 1:52am

I am seeing this issue as well.
I’m trying to make a db migration on a new hosted postgres db with Pleroma:

bash-5.1$ /opt/pleroma/bin/pleroma_ctl migrate

01:46:04.890 [info] == Running 20170522160642 Pleroma.Repo.Migrations.CaseInsensivtivity.up/0 forward

01:46:04.895 [info] execute "create extension if not exists citext"
** (Postgrex.Error) ERROR 42501 (insufficient_privilege) permission denied to create extension "citext"

    hint: Must have CREATE privilege on current database to create this extension.
    (ecto_sql 3.9.0) lib/ecto/adapters/sql.ex:905: Ecto.Adapters.SQL.raise_sql_call_error/1
    (elixir 1.11.4) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
    (ecto_sql 3.9.0) lib/ecto/adapters/sql.ex:997: Ecto.Adapters.SQL.execute_ddl/4
    (ecto_sql 3.9.0) lib/ecto/migration/runner.ex:326: Ecto.Migration.Runner.log_and_execute_ddl/3
    (ecto_sql 3.9.0) lib/ecto/migration/runner.ex:117: anonymous fn/6 in Ecto.Migration.Runner.flush/0
    (elixir 1.11.4) lib/enum.ex:2193: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto_sql 3.9.0) lib/ecto/migration/runner.ex:116: Ecto.Migration.Runner.flush/0
    (ecto_sql 3.9.0) lib/ecto/migration/runner.ex:289: Ecto.Migration.Runner.perform_operation/3
bash-5.1$

Manually adding to the db does not fix the issue for me but that may be an Elixir / Pleroma bug.

mplatts · February 19, 2023, 6:10am

Also having this:

  21:31:11.829 [info] == Running 20211119215456 PetalPro.Repo.Migrations.CreateUsersAuthTables.change/0 forward
     21:31:11.838 [info] execute "CREATE EXTENSION IF NOT EXISTS citext"
     ** (Postgrex.Error) ERROR 42501 (insufficient_privilege) permission denied to create extension "citext"

ppiechota · February 20, 2023, 6:08pm

I’m having the same issue.

ethanrjones97 · February 21, 2023, 1:35am

I have a theory it’s related to the application reading the DATABASE_URL= environment variable which is seemingly incorrect. Mine was setting it as a different user than postgres

Running env will help determining that.

ethanrjones97 · February 21, 2023, 2:18am

I did redeploy the db before I discovered this but it does look like DATABASE_URL is incorrectly set by fly (it uses a user based on the db sever’s name).

Setting the proper DATABASE_URL via fly secrets appears to be a workaround.

siros123 · February 24, 2023, 4:13am

I am having this same problem when deploying a fresh Phoenix 1.7.0-rc.3 app with phx.gen.auth .

Starting from a clean “fly launch”, everything works until in deployment it gets to the database and then it blocks on

==> Release command detected: /app/bin/migrate

--> This release will not be available until the release command succeeds.

     Starting instance
     Configuring virtual machine
     Pulling container image
     Unpacking image
     Preparing kernel init
     Configuring firecracker
     Starting virtual machine
     Preparing to run: `/app/bin/migrate` as nobody
     2023/02/24 04:04:27 listening on [fdaa:0:5127:a7b:a1:3f3a:6c6e:2]:22 (DNS: [fdaa::3]:53)
     04:04:29.732 [info] execute "CREATE EXTENSION IF NOT EXISTS citext"
     ** (Postgrex.Error) ERROR 42501 (insufficient_privilege) permission denied to create extension "citext"
         hint: Must have CREATE privilege on current database to create this extension.
         (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:913: Ecto.Adapters.SQL.raise_sql_call_error/1
         (elixir 1.14.1) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
         (ecto_sql 3.9.2) lib/ecto/migration/runner.ex:326: Ecto.Migration.Runner.log_and_execute_ddl/3
         (ecto_sql 3.9.2) lib/ecto/migration/runner.ex:117: anonymous fn/6 in Ecto.Migration.Runner.flush/0
         (elixir 1.14.1) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
         (ecto_sql 3.9.2) lib/ecto/migration/runner.ex:116: Ecto.Migration.Runner.flush/0
         nofile:1: (file)
     Starting clean up.
Error release command failed, deployment aborted

I tried to install citext manually as per @zblanco but that didn’t seem to do anything, and it made no difference on a subsequent deploy.

Removing the “CREATE EXTENSION IF NOT EXISTS citext” command from the migration file does allow migration to continue but - unurprisingly - it generates a new error right afterwards:

==> Release command detected: /app/bin/migrate

--> This release will not be available until the release command succeeds.
     Starting instance
     Configuring virtual machine
     Pulling container image
     Unpacking image
     Preparing kernel init
     Configuring firecracker
     Starting virtual machine
     2023/02/24 04:03:11 listening on [fdaa:0:5127:a7b:a1:804f:683f:2]:22 (DNS: [fdaa::3]:53)
         (ecto_sql 3.9.2) lib/ecto/adapters/sql.ex:1005: Ecto.Adapters.SQL.execute_ddl/4
         (ecto_sql 3.9.2) lib/ecto/migration/runner.ex:117: anonymous fn/6 in Ecto.Migration.Runner.flush/0
         (elixir 1.14.1) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
         (ecto_sql 3.9.2) lib/ecto/migration/runner.ex:116: Ecto.Migration.Runner.flush/0
         nofile:1: (file)
     Starting clean up.
Error release command failed, deployment aborted

I am blocked on this now, does anyone have a workaround?

ethanrjones97 · February 24, 2023, 4:41am

Were you able to investigate the environment variable DATABASE_URL? I found it was incorrectly setting the user to something other than “postgres”.

cliffsky · February 25, 2023, 2:28pm

I’m having the exact same problem since today on a new app I’m trying to deploy.
Setting the DATABASE_URL didn’t work and recreating a new db and attaching it manually through flyctl postgres attach doesn’t work either.

Error message in the terminal:

ERROR 42501 (insufficient_privilege) permission denied to create extension "citext"
             hint: Must have CREATE privilege on current database to create this extension.

I’m using flyctl v0.0.469

srbaker · February 25, 2023, 3:23pm

Exactly the same issue. Changing DATABASE_URL didn’t help.

I absolutely love the service here when it works. But fly is either super ridiculously unstable, or I just keep catching bad weather days because I haven’t been able to get a successful app deployment out of it in about three months.

shaun · February 25, 2023, 3:34pm

It looks like you’ll need to grant your attached user SUPERUSER privileges to make that work. Extensions typically can’t be installed with normal privileges.

To grant superuser privileges:

Connect to your postgres app with fly pg connect --pg-app-name
ALTER user <target-user> with SUPERUSER;
Confirm the change with \du

Let me know if that does the trick for you.

vivek-guerdon · February 25, 2023, 3:50pm

I am having trouble deploying my app and my app server is becoming dead after every restart, apart from it while detaching and creating a new DB I cannot see the maa region in the option anymore. My app server is already in the maa region. I am not sure what is going on in the system but seems highly unreliable as of now.

shaun · February 25, 2023, 4:10pm

I just cut a new release of flyctl v0.0.470, which will address this issue. It should be available shortly.

srbaker · February 25, 2023, 5:10pm

Thank you!

I am waiting patiently to deploy this app.

-Steven

cliffsky · February 25, 2023, 5:13pm

This worked for me, thanks @shaun!

Will the new release of flyctl v0.0.470 fix and apply this automatically from now on, or are we expected to run the command ourselves on first deploy?

Topic		Replies	Views
Testing Postgres create locally first? Phoenix elixir , postgres	4	526	July 9, 2022
Error Release command failed, deployment aborted Questions / Help elixir	18	4509	October 26, 2021
Phoenix app - Unable to deploy due to app/bin/migrate error and dbconnection failed with non existing domain Phoenix elixir , postgres	2	1010	August 26, 2022
fly deploy returns error for new app Build debugging elixir , postgres	7	599	July 20, 2023
Cannot deploy due to Postgres db being unreachable Build debugging elixir , postgres	4	702	May 24, 2023

Any idea why my deployment fails?

Related topics