I’m running into this error: “Failed due to unhealthy allocations”
It seems that it keeps terminating with exit code 0.
Dockerfile
FROM python:3.8.13-bullseye
ENV PYTHONUNBUFFERED=1
WORKDIR /
COPY Pipfile /
COPY Pipfile.lock /
RUN pip install pipenv
RUN pipenv install --deploy --ignore-pipfile
COPY . ./
EXPOSE 8000
Docker-compose.yaml:
services:
api:
build: ./
ports:
- "8000:8000"
volumes:
- /db.sqlite3
command: bash -c "pipenv run python manage.py runserver"
web:
build: ./frontend
ports:
- "3000:3000"
volumes:
- ./frontend:/frontend
fly.toml
# fly.toml file generated for chefs-notebook on 2022-12-29T13:55:20-05:00
app = "chefs-notebook"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
allowed_public_ports = []
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
Looks like your dockerfile is exposing 8000 but you have fly set to proxy to port 8080. I’m not sure if EXPOSE matters but what ports are you opening inside the docker container ?
I’m still pretty new to Docker, so as far as I know the only port I’m opening is 8000.
I tried changing the fly.toml internal port to 8000.
Here’s the status output for the VM that’s failing:
Instance
ID = b5423f2b
Process = app
Version = 8
Region = iad
Desired = stop
Status = complete
Health Checks = 1 total, 1 critical
Restarts = 0
Created = 12m18s ago
Events
TIMESTAMP TYPE MESSAGE
2023-01-05T22:25:06Z Received Task received by client
2023-01-05T22:25:06Z Task Setup Building Task Directory
2023-01-05T22:27:02Z Started Task started by client
2023-01-05T22:30:06Z Alloc Unhealthy Task not running for min_healthy_time of 10s by deadline
2023-01-05T22:30:06Z Killing Sent interrupt. Waiting 5s before force killing
2023-01-05T22:30:28Z Terminated Exit Code: 0
2023-01-05T22:30:28Z Killed Task successfully killed
2023-01-05T22:30:28Z Killing Sent interrupt. Waiting 5s before force killing
Checks
ID SERVICE STATE OUTPUT
8bff2a62d889b306be259e512d22e6a4 tcp-8000 critical dial tcp 172.19.72.2:8000: connect: connection refused
And here is the output that fly deploy gives me:
2023-01-05T22:09:04Z runner[92e89f5d] iad [info]Starting instance
2023-01-05T22:10:10Z runner[92e89f5d] iad [info]Configuring virtual machine
2023-01-05T22:10:17Z runner[92e89f5d] iad [info]Unpacking image
2023-01-05T22:12:32Z runner[92e89f5d] iad [info]Preparing kernel init
2023-01-05T22:12:33Z runner[92e89f5d] iad [info]Configuring firecracker
2023-01-05T22:12:33Z runner[92e89f5d] iad [info]Starting virtual machine
2023-01-05T22:12:33Z app[92e89f5d] iad [info]Starting init (commit: f447594)...
2023-01-05T22:12:34Z app[92e89f5d] iad [info]Preparing to run: `/bin/sh -c gunicorn conf.wsgi --bind 127.0.0.1:8080 --log-file - --log-level debug --timeout 100` as root
2023-01-05T22:12:34Z app[92e89f5d] iad [info]2023/01/05 22:12:34 listening on [fdaa:1:14ca:a7b:93:92e8:9f5d:2]:22 (DNS: [fdaa::3]:53)
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [525] [DEBUG] Current configuration:
2023-01-05T22:12:35Z app[92e89f5d] iad [info] config: ./gunicorn.conf.py
2023-01-05T22:12:35Z app[92e89f5d] iad [info] wsgi_app: None
2023-01-05T22:12:35Z app[92e89f5d] iad [info] bind: ['127.0.0.1:8080']
2023-01-05T22:12:35Z app[92e89f5d] iad [info] backlog: 2048
2023-01-05T22:12:35Z app[92e89f5d] iad [info] workers: 1
2023-01-05T22:12:35Z app[92e89f5d] iad [info] worker_class: sync
2023-01-05T22:12:35Z app[92e89f5d] iad [info] threads: 1
2023-01-05T22:12:35Z app[92e89f5d] iad [info] worker_connections: 1000
2023-01-05T22:12:35Z app[92e89f5d] iad [info] max_requests: 0
2023-01-05T22:12:35Z app[92e89f5d] iad [info] max_requests_jitter: 0
2023-01-05T22:12:35Z app[92e89f5d] iad [info] timeout: 100
2023-01-05T22:12:35Z app[92e89f5d] iad [info] graceful_timeout: 30
2023-01-05T22:12:35Z app[92e89f5d] iad [info] keepalive: 2
2023-01-05T22:12:35Z app[92e89f5d] iad [info] limit_request_line: 4094
2023-01-05T22:12:35Z app[92e89f5d] iad [info] limit_request_fields: 100
2023-01-05T22:12:35Z app[92e89f5d] iad [info] limit_request_field_size: 8190
2023-01-05T22:12:35Z app[92e89f5d] iad [info] reload: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info] reload_engine: auto
2023-01-05T22:12:35Z app[92e89f5d] iad [info] reload_extra_files: []
2023-01-05T22:12:35Z app[92e89f5d] iad [info] spew: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info] check_config: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info] print_config: False
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [527] [INFO] Booting worker with pid: 527
2023-01-05T22:12:35Z app[92e89f5d] iad [info][2023-01-05 22:12:35 +0000] [525] [DEBUG] 1 workers
2023-01-05T22:25:06Z runner[b5423f2b] iad [info]Starting instance
2023-01-05T22:25:13Z runner[b5423f2b] iad [info]Configuring virtual machine
2023-01-05T22:25:13Z runner[b5423f2b] iad [info]Pulling container image
2023-01-05T22:25:19Z runner[b5423f2b] iad [info]Unpacking image
2023-01-05T22:27:01Z runner[b5423f2b] iad [info]Configuring firecracker
2023-01-05T22:27:01Z runner[b5423f2b] iad [info]Starting virtual machine
2023-01-05T22:27:02Z app[b5423f2b] iad [info]Starting init (commit: f447594)...
2023-01-05T22:27:02Z app[b5423f2b] iad [info]Preparing to run: `/bin/sh -c gunicorn conf.wsgi --bind 127.0.0.1:8000 --log-file - --log-level debug --timeout 100` as root
2023-01-05T22:27:02Z app[b5423f2b] iad [info]2023/01/05 22:27:02 listening on [fdaa:1:14ca:a7b:93:b542:3f2b:2]:22 (DNS: [fdaa::3]:53)
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [525] [DEBUG] Current configuration:
2023-01-05T22:27:04Z app[b5423f2b] iad [info] access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [527] [INFO] Booting worker with pid: 527
2023-01-05T22:27:04Z app[b5423f2b] iad [info][2023-01-05 22:27:04 +0000] [525] [DEBUG] 1 workers
2023-01-05T22:30:22Z runner[b5423f2b] iad [info]Shutting down virtual machine
2023-01-05T22:30:22Z app[b5423f2b] iad [info]Sending signal SIGINT to main child process w/ PID 520
I think I had the same problem too. You need to bind gunicorn to ipv6 addresses because fly internally doesn’t use ipv4 (as I understand it). So for gunicorn that looks like gunicorn --bind [::]8000 .... I do think your fly.toml internal_port should also be port 8000 or whatever you are exposing via your Dockerfile/app.
edited to specify that internal_port setting should match Dockerfile/app
@ramonacodes As I understand it things should work like this:
– ipv4 → fly.io proxy (80 or 443 in services.ports, 80 redirects to 443) – ipv6 → gunicorn on 8000
– ipv6 → fly.io proxy (80 or 443 in services.ports, 80 redirects to 443) – ipv6 → gunicorn on 8000
Ah! Okay, I was wondering what the square brackets and colons were about.
So I changed the internal port in fly.toml to 8000 and changed the gunicorn binding to the ipv6 address [::]8000. And I was able to get a successful deployment that way.
Thank you for your help!