Django migrations are not running

Django
1

Hello, I’ve got a small django app in a fly instance. Recently, I added some migrations and wanted to run them. When I tried connecting through ssh after I entered the migrations command the command is stuck. Specifically I get the following output:

/app# python manage.py migrate
Operations to perform:
  Apply all migrations: account, admin, auth, cal, contenttypes, core, foods, reversion, sessions, sites, socialaccount, users
Running migrations:
  Applying cal.0005_dietday_dietdayfood...

The run migrations command is stuck there, nothing happening no matter how long I leave it running. Taking a look at the logs, the only thing I see is output similar to this, every 15 seconds:

2022-02-27T10:05:49Z app[4407a7ef] lhr [info][2022-02-27 12:05:49 +0200] [542] [DEBUG] Closing connection.

I also tried with a

[deploy]
  release_command = "python /app/manage.py migrate"

cmd in my fly.toml but got the same behavior.

Any idea what’s going on? How can I run my migrations?

TIA

2

At the end of the deploy, it should tell you it’s running the release command. Do you see this in the deploy logs?
The one thing I noted was that you executed python manage.py migrate in /app, maybe the release command is being executed at /?

3

No, no… Tt said that it started executing the migrations (i.e the command started fine) but got stuck in the same point.

4

This is most likely because the release command is hanging. We don’t handle that state well, though.

Can you try removing the release command, deploying, then SSH into your app and run it manually to see what happens?

5

I tried it without the release command and the result is exactly the same:

# python manage.py migrate
Operations to perform:
  Apply all migrations: account, admin, auth, cal, contenttypes, core, foods, reversion, sessions, sites, socialaccount, users
Running migrations:
  Applying cal.0005_dietday_dietdayfood...

stucks there for ever.

6

It seems like python manage.py migrate isn’t able to connect to your database.
When you attach a pg instance, fly.io places a DATABASE_URL onto your instance.

Is manage.py using DATABASE_URL to connect to the instance?

7

Hello @Alex_Piechowski thanks for replying but that’s not the problem. From the same shell where the migrations are not run I can do a python manage.py shell and query the database without problems.

8

Ooh, what if it’s just a really long migration and it’s not actually hanging, but doing work?
What are the contents of the migration file that it is hanging on?

Could you try running the migration, then also querying pg_stat_activity to see if thee’s a hung DDL operation? :

select * from pg_stat_activity;
9

Hello @Alex_Piechowski no it ain’t a long migration (the tables are very small). You are right, when I try the migration I get a hung DDL Operation:

datid            | 16387
datname          | xxx
pid              | 23781
leader_pid       |
usesysid         | 16386
usename          | xxx
application_name |
client_addr      | xxx
client_hostname  |
client_port      | 50608
backend_start    | 2022-03-01 16:19:57.604851+00
xact_start       | 2022-03-01 16:19:57.722498+00
query_start      | 2022-03-01 16:19:57.731432+00
state_change     | 2022-03-01 16:19:57.731433+00
wait_event_type  | Lock
wait_event       | transactionid
state            | active
backend_xid      | 728
backend_xmin     | 714
query            | CREATE TABLE "cal_dietday" ("id" bigserial NOT NULL PRIMARY KEY, "created_on" timestamp with time zone NOT NULL, "modified_on" timestamp with time zone NOT NULL, "name" varchar(128) NOT NULL, "created_by_id" bigint NOT NULL, "modified_by_id" bigint NOT NULL)
backend_type     | client backend

this never stops.

10

What happens when. You execute that directly in SQL console?

11

@Alex_Piechowski exactly the same! The DDL query hangs forever until the database stops it. I even tried suspending the app and connecting to the database to execute the query. Still the same behavior !

Is there a way to restart the database ?

12

https://fly.io/docs/flyctl/restart/

1 Like
13

@Alex_Piechowski it worked immediately after I restarted the db :frowning:

Sorry for taking so much of your time. Thank you for all the help.

Kind regards,
Serafeim

1 Like