Flask/Prometheus private networking

Questions / Help
1

I’m trying to set up a metrics stack with flask, prometheus, and grafana. I want the apps to communicate over the private network and am using the .internal DNS. When running the docker containers locally, everything works fine. When deployed to fly however, Grafana can connect to Prometheus as a data source, but Prometheus can’t connect to my flask app.

I get the following error:

Error scraping target: Get "http://flask-app.internal:5000/metrics": dial tcp [ip]:5000: connect: connection refused

The flask app is definitely running, with [INFO] Listening at: http://0.0.0.0:5000 (323) in the logs, and if I add an http_service to the fly.toml I can curl the /metrics endpoint successfully.

I’ve read through the docs on private networking but haven’t been able to figure out what’s wrong and I’m not sure what I’m missing here. Could use a point in the right direction for fixing this!

Flask App Dockerfile:

FROM python:3.11.2

COPY src /app/src
COPY requirements.txt /app/requirements.txt

RUN pip3 install -r /app/requirements.txt

WORKDIR /app/src

EXPOSE 5000

ENV PROMETHEUS_MULTIPROC_DIR /tmp
ENV prometheus_multiproc_dir /tmp

# expose env variables
RUN eval $(printenv | sed -n "s/^\([^=]\+\)=\(.*\)$/export \1=\2/p" | sed 's/"/\\\"/g' | sed '/=/s//="/' | sed 's/$/"/' >> /etc/profile)

ENTRYPOINT ["gunicorn", "-c", "config.py", "-w", "5", "--bind=0.0.0.0:5000", "--timeout", "600", "web.app:create_app()"]

Flask App fly.toml:

app = 'flask-app'
primary_region = 'lhr'

[build]

[http_service]
  internal_port = 5000
  force_https = true
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

Prometheus Dockerfile:

FROM prom/prometheus:latest
COPY prometheus_config.yaml /etc/prometheus/prometheus.yaml
EXPOSE 9090
ENTRYPOINT [ "/bin/prometheus" ]
CMD ["--config.file=/etc/prometheus/prometheus.yaml"]

Prometheus Config:

global:
  scrape_interval: 30s
scrape_configs:
  - job_name: prometheus
    static_configs:
    - targets:
      - 'prometheus.internal:9090'
  - job_name: 'webapp'
    static_configs:
    - targets:
      - 'flask-app.internal:5000'

Prometheus fly.toml:

app = 'prometheus'
primary_region = 'lhr'

[build]

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1
2

Hi!

.internal is ipv6-ony, so you need to get your gunicorn bound to :: and not to 0.0.0.0.

This should do it in your Dockerfile:

ENTRYPOINT ["gunicorn", "-c", "config.py", "-w", "5", "--bind=[::]:5000", "--timeout", "600", "web.app:create_app()"]

3

Ah of course! That worked, thanks.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.