Deployment Debugging

I’m running into this error: “Failed due to unhealthy allocations”

It seems that it keeps terminating with exit code 0.


FROM  python:3.8.13-bullseye



COPY Pipfile /

COPY Pipfile.lock /

RUN pip install pipenv

RUN pipenv install --deploy --ignore-pipfile

COPY . ./



    build: ./        
      - "8000:8000"
      - /db.sqlite3
    command: bash -c "pipenv run python runserver"

    build: ./frontend
      - "3000:3000"
      - ./frontend:/frontend


# fly.toml file generated for chefs-notebook on 2022-12-29T13:55:20-05:00

app = "chefs-notebook"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []


  allowed_public_ports = []
  auto_rollback = true

  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
    hard_limit = 25
    soft_limit = 20
    type = "connections"

    force_https = true
    handlers = ["http"]
    port = 80

    handlers = ["tls", "http"]
    port = 443

    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Looks like your dockerfile is exposing 8000 but you have fly set to proxy to port 8080. I’m not sure if EXPOSE matters but what ports are you opening inside the docker container ?

I’m still pretty new to Docker, so as far as I know the only port I’m opening is 8000.

I tried changing the fly.toml internal port to 8000.

Here’s the status output for the VM that’s failing:

  ID            = b5423f2b             
  Process       = app                  
  Version       = 8                    
  Region        = iad                  
  Desired       = stop                 
  Status        = complete             
  Health Checks = 1 total, 1 critical  
  Restarts      = 0                    
  Created       = 12m18s ago           

TIMESTAMP           	TYPE           	MESSAGE                                                  
2023-01-05T22:25:06Z	Received       	Task received by client                                 	
2023-01-05T22:25:06Z	Task Setup     	Building Task Directory                                 	
2023-01-05T22:27:02Z	Started        	Task started by client                                  	
2023-01-05T22:30:06Z	Alloc Unhealthy	Task not running for min_healthy_time of 10s by deadline	
2023-01-05T22:30:06Z	Killing        	Sent interrupt. Waiting 5s before force killing         	
2023-01-05T22:30:28Z	Terminated     	Exit Code: 0                                            	
2023-01-05T22:30:28Z	Killed         	Task successfully killed                                	
2023-01-05T22:30:28Z	Killing        	Sent interrupt. Waiting 5s before force killing         	

ID                              	SERVICE 	STATE   	OUTPUT                                                 
8bff2a62d889b306be259e512d22e6a4	tcp-8000	critical	dial tcp connect: connection refused	

And here is the output that fly deploy gives me:

2023-01-05T22:09:04Z runner[92e89f5d] iad [info]Starting instance
2023-01-05T22:10:10Z runner[92e89f5d] iad [info]Configuring virtual machine
2023-01-05T22:10:17Z runner[92e89f5d] iad [info]Unpacking image
2023-01-05T22:12:32Z runner[92e89f5d] iad [info]Preparing kernel init
2023-01-05T22:12:33Z runner[92e89f5d] iad [info]Configuring firecracker
2023-01-05T22:12:33Z runner[92e89f5d] iad [info]Starting virtual machine
2023-01-05T22:12:33Z app[92e89f5d] iad [info]Starting init (commit: f447594)...
2023-01-05T22:12:34Z app[92e89f5d] iad [info]Preparing to run: `/bin/sh -c gunicorn conf.wsgi --bind --log-file - --log-level debug --timeout 100` as root
2023-01-05T22:12:34Z app[92e89f5d] iad [info]2023/01/05 22:12:34 listening on [fdaa:1:14ca:a7b:93:92e8:9f5d:2]:22 (DNS: [fdaa::3]:53)
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [525] [DEBUG] Current configuration:
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  config: ./
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  wsgi_app: None
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  bind: ['']
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  backlog: 2048
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  workers: 1
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  worker_class: sync
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  threads: 1
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  worker_connections: 1000
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  max_requests: 0
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  max_requests_jitter: 0
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  timeout: 100
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  graceful_timeout: 30
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  keepalive: 2
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  limit_request_line: 4094
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  limit_request_fields: 100
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  limit_request_field_size: 8190
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  reload: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  reload_engine: auto
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  reload_extra_files: []
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  spew: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  check_config: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info]  print_config: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [527] [INFO] Booting worker with pid: 527
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [525] [DEBUG] 1 workers
2023-01-05T22:25:06Z runner[b5423f2b] iad [info]Starting instance
2023-01-05T22:25:13Z runner[b5423f2b] iad [info]Configuring virtual machine
2023-01-05T22:25:13Z runner[b5423f2b] iad [info]Pulling container image
2023-01-05T22:25:19Z runner[b5423f2b] iad [info]Unpacking image
2023-01-05T22:27:01Z runner[b5423f2b] iad [info]Configuring firecracker
2023-01-05T22:27:01Z runner[b5423f2b] iad [info]Starting virtual machine
2023-01-05T22:27:02Z app[b5423f2b] iad [info]Starting init (commit: f447594)...
2023-01-05T22:27:02Z app[b5423f2b] iad [info]Preparing to run: `/bin/sh -c gunicorn conf.wsgi --bind --log-file - --log-level debug --timeout 100` as root
2023-01-05T22:27:02Z app[b5423f2b] iad [info]2023/01/05 22:27:02 listening on [fdaa:1:14ca:a7b:93:b542:3f2b:2]:22 (DNS: [fdaa::3]:53)
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [525] [DEBUG] Current configuration:
2023-01-05T22:27:04Z app[b5423f2b] iad [info]  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [527] [INFO] Booting worker with pid: 527
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [525] [DEBUG] 1 workers
2023-01-05T22:30:22Z runner[b5423f2b] iad [info]Shutting down virtual machine
2023-01-05T22:30:22Z app[b5423f2b] iad [info]Sending signal SIGINT to main child process w/ PID 520

I think I had the same problem too. You need to bind gunicorn to ipv6 addresses because fly internally doesn’t use ipv4 (as I understand it). So for gunicorn that looks like gunicorn --bind [::]8000 .... I do think your fly.toml internal_port should also be port 8000 or whatever you are exposing via your Dockerfile/app.

edited to specify that internal_port setting should match Dockerfile/app

@ramonacodes As I understand it things should work like this: proxy (80 or 443 in services.ports, 80 redirects to 443) – ipv6 → gunicorn on 8000 proxy (80 or 443 in services.ports, 80 redirects to 443) – ipv6 → gunicorn on 8000

Ah! Okay, I was wondering what the square brackets and colons were about.

So I changed the internal port in fly.toml to 8000 and changed the gunicorn binding to the ipv6 address [::]8000. And I was able to get a successful deployment that way.

Thank you for your help!