Several months ago, I migrated my Django application with PG hosted on Heroku to Fly.io. Everything was working great.
Today, I made very minor code changes to my Django application and attempted to deploy it to fly.io.
$ flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-summer-smoke-8026 ready
==> Building image with Buildpacks
--> docker host: 20.10.12 linux x86_64
full: Pulling from paketobuildpacks/builder
Digest: sha256:5cc6dddc3b1cd5c50465e3386cf63cf6bec209d60d5761bd470d1a9d9005fe1b
Status: Image is up to date for paketobuildpacks/builder:full
full-cnb: Pulling from paketobuildpacks/run
Digest: sha256:155f74770739865d08f3f7ae52c745cf3ca774f8b065bc5592492ad6ce6e5d85
Status: Image is up to date for paketobuildpacks/run:full-cnb
latest: Pulling from paketo-buildpacks/python
Digest: sha256:ec31e670be640d75a982469503a7ef9b09cf3f9109f96ebd2daef64fb678d74e
Status: Image is up to date for gcr.io/paketo-buildpacks/python:latest
===> DETECTING
6 of 9 buildpacks participating
paketo-buildpacks/ca-certificates 3.4.0
paketo-buildpacks/cpython 1.7.2
paketo-buildpacks/pip 0.16.1
paketo-buildpacks/pip-install 0.5.7
paketo-buildpacks/python-start 0.14.0
paketo-buildpacks/procfile 5.4.0
===> ANALYZING
Restoring metadata for "paketo-buildpacks/ca-certificates:helper" from app image
Restoring metadata for "paketo-buildpacks/cpython:cpython" from app image
Restoring metadata for "paketo-buildpacks/pip:pip" from cache
Restoring metadata for "paketo-buildpacks/pip-install:packages" from app image
Restoring metadata for "paketo-buildpacks/pip-install:cache" from cache
===> RESTORING
Restoring data for "paketo-buildpacks/cpython:cpython" from cache
Restoring data for "paketo-buildpacks/pip:pip" from cache
Restoring data for "paketo-buildpacks/pip-install:cache" from cache
Restoring data for "paketo-buildpacks/pip-install:packages" from cache
===> BUILDING
Paketo Buildpack for CA Certificates 3.4.0
https://github.com/paketo-buildpacks/ca-certificates
Launch Helper: Reusing cached layer
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Paketo Buildpack for CPython 1.7.2
Resolving CPython version
Candidate version sources (in priority order):
-> ""
<unknown> -> ""
Selected CPython version (using ): 3.10.8
Reusing cached layer /layers/paketo-buildpacks_cpython/cpython
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Paketo Buildpack for Pip 0.16.1
Resolving Pip version
Candidate version sources (in priority order):
<unknown> -> ""
Selected Pip version (using <unknown>): 22.3.1
Reusing cached layer /layers/paketo-buildpacks_pip/pip
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Warning: BOM table is deprecated in this buildpack api version, though it remains supported for backwards compatibility. Buildpack authors should write BOM information to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>.
Paketo Buildpack for Pip Install 0.5.7
Executing build process
Running 'pip install --requirement requirements.txt --exists-action=w --cache-dir=/layers/paketo-buildpacks_pip-install/cache --compile --user --disable-pip-version-check'
Completed in 2.381s
Generating SBOM for /layers/paketo-buildpacks_pip-install/packages
Completed in 464ms
Configuring build environment
PYTHONPATH -> "/layers/paketo-buildpacks_pip-install/packages/lib/python3.10/site-packages:$PYTHONPATH"
Configuring launch environment
PYTHONPATH -> "/layers/paketo-buildpacks_pip-install/packages/lib/python3.10/site-packages:$PYTHONPATH"
Paketo Buildpack for Python Start 0.14.0
Assigning launch processes:
web (default): python
Paketo Buildpack for Procfile 5.4.0
https://github.com/paketo-buildpacks/procfile
Process types:
release: python manage.py migrate
web: gunicorn O.wsgi --log-file -
===> EXPORTING
Reusing layer 'paketo-buildpacks/ca-certificates:helper'
Reusing layer 'paketo-buildpacks/cpython:cpython'
Reusing layer 'paketo-buildpacks/pip-install:packages'
Reusing 1/1 app layer(s)
Reusing layer 'launcher'
Reusing layer 'config'
Reusing layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving registry.fly.io/X:cache...
*** Images (062c80cc12e1):
registry.fly.io/X:cache
registry.fly.io/X:deployment-01GJH9RT03JHR8PQ2KDSSE7FQ9
Reusing cache layer 'paketo-buildpacks/cpython:cpython'
Reusing cache layer 'paketo-buildpacks/pip:pip'
Reusing cache layer 'paketo-buildpacks/pip-install:cache'
Reusing cache layer 'paketo-buildpacks/pip-install:packages'
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/X]
13657e53d91b: Layer already exists
6fac1dfdb433: Layer already exists
858993968edd: Layer already exists
2c5ca480df98: Layer already exists
a115e3160295: Layer already exists
d6293bae7bfb: Layer already exists
c23057e83cef: Layer already exists
5f0cddb10d7f: Layer already exists
c3cd2fc2ba6c: Layer already exists
b8cc49313e8d: Layer already exists
03faab0841db: Layer already exists
249e27abf619: Layer already exists
055c77a47129: Layer already exists
69f57fbceb1b: Layer already exists
deployment-01GJH9RT03JHR8PQ2KDSSE7FQ9: digest: sha256:9259ff954bd3c7a365e1a1542744fcfb6d2d806f91339db18f7c7302e39c62f3 size: 3250
--> Pushing image done
image: registry.fly.io/X:deployment-01GJH9RT03JHR8PQ2KDSSE7FQ9
image size: 965 MB
==> Creating release
--> release v9 created
--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: python manage.py migrate
--> This release will not be available until the release command succeeds.
Starting instance
Configuring virtual machine
Pulling container image
Unpacking image
Preparing kernel init
Starting init (commit: 81d5330)...
2022/11/23 03:52:39 listening on [fdaa:0:88d8:a7b:e824:678:f60f:2]:22 (DNS: [fdaa::3]:53)
Starting instance
Configuring virtual machine
Pulling container image
Unpacking image
Preparing kernel init
Configuring firecracker
Starting virtual machine
Starting init (commit: 81d5330)...
Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
no label, UUID=0a050430-3769-485d-986b-12c6e955a9d3
Preparing to run: `launcher python manage.py migrate` as 1000
2022/11/23 03:52:39 listening on [fdaa:0:88d8:a7b:e824:678:f60f:2]:22 (DNS: [fdaa::3]:53)
repertoire_browser.Composer.attributes: (fields.W340) null has no effect on ManyToManyField.
System check identified some issues:
WARNINGS:
repertoire_browser.Composer.attributes: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, repertoire_browser, sessions
Running migrations:
No migrations to apply.
Your models in app(s): 'repertoire_browser' have changes that are not yet reflected in a migration, and so won't be applied.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, repertoire_browser, sessions
Running migrations:
No migrations to apply.
Your models in app(s): 'repertoire_browser' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Starting clean up.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Starting clean up.
Starting instance
Configuring virtual machine
Pulling container image
Unpacking image
Preparing kernel init
Configuring firecracker
Starting virtual machine
Starting init (commit: 81d5330)...
Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
no label, UUID=0a050430-3769-485d-986b-12c6e955a9d3
Preparing to run: `launcher python manage.py migrate` as 1000
2022/11/23 03:52:39 listening on [fdaa:0:88d8:a7b:e824:678:f60f:2]:22 (DNS: [fdaa::3]:53)
System check identified some issues:
WARNINGS:
repertoire_browser.Composer.attributes: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
Apply all migrations: admin, auth, contenttypes, repertoire_browser, sessions
Running migrations:
No migrations to apply.
Your models in app(s): 'repertoire_browser' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
Starting clean up.
==> Monitoring deployment
1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical]
Failed Instances
Failure #1
Instance
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
a25c2ab5 app 9 ord run running 1 total, 1 critical 0 4m50s ago
Recent Events
TIMESTAMP TYPE MESSAGE
2022-11-23T03:53:01Z Received Task received by client
2022-11-23T03:53:01Z Task Setup Building Task Directory
2022-11-23T03:53:10Z Started Task started by client
2022-11-23T03:53:08Z [info]Configuring virtual machine
2022-11-23T03:53:08Z [info]Pulling container image
2022-11-23T03:53:09Z [info]Unpacking image
2022-11-23T03:53:09Z [info]Preparing kernel init
2022-11-23T03:53:09Z [info]Configuring firecracker
2022-11-23T03:53:09Z [info]Starting virtual machine
2022-11-23T03:53:10Z [info]Starting init (commit: 81d5330)...
2022-11-23T03:53:10Z [info]Preparing to run: `launcher gunicorn O.wsgi --log-file -` as 1000
2022-11-23T03:53:10Z [info]2022/11/23 03:53:10 listening on [fdaa:0:88d8:a7b:e824:a25c:2ab5:2]:22 (DNS: [fdaa::3]:53)
2022-11-23T03:53:12Z [info][2022-11-23 03:53:12 +0000] [520] [INFO] Starting gunicorn 20.1.0
2022-11-23T03:53:12Z [info][2022-11-23 03:53:12 +0000] [520] [INFO] Listening at: http://127.0.0.1:8000 (520)
2022-11-23T03:53:12Z [info][2022-11-23 03:53:12 +0000] [520] [INFO] Using worker: sync
2022-11-23T03:53:12Z [info][2022-11-23 03:53:12 +0000] [542] [INFO] Booting worker with pid: 542
--> v9 failed - Failed due to unhealthy allocations - not rolling back to stable job version 9 as current job has same specification and deploying as v10
--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort
Following the Troubleshooting Guide, I ran the following commands:
$ flyctl status --all
App
Name = X
Owner = O
Version = 9
Status = running
Hostname = X.fly.dev
Platform = nomad
Deployment Status
ID = 562bd905-70d3-f9df-2bc0-aad1927ad574
Version = v9
Status = failed
Description = Failed due to unhealthy allocations - not rolling back to stable job version 9 as current job has same specification
Instances = 1 desired, 1 placed, 0 healthy, 1 unhealthy
Instances
ID PROCESS VERSION REGION DESIRED STATUS HEALTH CHECKS RESTARTS CREATED
a25c2ab5 app 9 ⇡ ord stop complete 1 total, 1 critical 0 10m23s ago
ff8466db app 8 ord run running 1 total, 1 passing 0 2022-08-26T06:51:04Z
adab043f app 7 ord stop complete 1 total, 1 critical 0 19m9s ago
$ flyctl vm status a25c2ab5
Instance
ID = a25c2ab5
Process = app
Version = 9
Region = ord
Desired = stop
Status = complete
Health Checks = 1 total, 1 critical
Restarts = 0
Created = 10m56s ago
Events
TIMESTAMP TYPE MESSAGE
2022-11-23T03:53:01Z Received Task received by client
2022-11-23T03:53:01Z Task Setup Building Task Directory
2022-11-23T03:53:10Z Started Task started by client
2022-11-23T03:58:01Z Alloc Unhealthy Task not running for min_healthy_time of 10s by deadline
2022-11-23T03:58:01Z Killing Sent interrupt. Waiting 5s before f*emphasized text*orce killing
2022-11-23T03:58:18Z Terminated Exit Code: 0
2022-11-23T03:58:18Z Killed Task successfully killed
Checks
ID SERVICE STATE OUTPUT
3df2415693844068640885b45074b954 tcp-8080 critical dial tcp 172.19.39.138:8080: connect: connection refused
Recent Logs
X = application name
O = owner name
I am trying to understand what is going wrong as I am very new to fly.io. There were no material changes to the Django application.
I would appreciate any pointers or suggestions as to what might be happening here.
Specifically, I want to know what 3df2415693844068640885b45074b954 tcp-8080 critical dial tcp 172.19.39.138:8080: connect: connection refused is doing. Why is it connecting to 8080 when WSGI is serving my application on 8000? Is that a red herring?
Thank you!