Hi!
First of all, I’m a Fly.io noobie, so please forgive me for any mistakes in this post.
A few weeks ago we’ve received an email that the hardware we hosted our apps on received irreparable damage and thus the machines on this host would be marked as destroyed. We are running a Strapi app together with a postgres app. The email specified that only the Strapi app machines would be destroyed. However, the postgres app hasn’t been able to restart since.
As a result, we can’t get the Strapi app running as it can’t connect to the postgres app due to it being unable to start.
Some stuff we’ve tried so far:
- restart the machines
- redeploy the postgres app
- scale the machine’s ram and memory
- attach a consul cluster
- revert to older images from before the hardware damage
- get a backup from the postgres database (not possible because the machine needs to be online for this)
But so far none of this had any effect. We see that the health checks do not pass, but we’ve not found out why yet. See output of different commands below
$ fly version
fly v0.2.120 darwin/arm64 Commit: f0232893ca84c56f874780d3bbf427a654c563d3 BuildDate: 2024-08-22T04:42:51Z
$ fly machine list -a ***-db
***-db
ID NAME STATE CHECKS REGION ROLE IMAGE IP ADDRESS VOLUME CREATED LAST UPDATED PROCESS GROUP SIZE
d891d64c696268 withered-sun-197 started 1/3 ams 500 Internal Server Error flyio/postgres:14.6 (v0.0.41) fdaa:0:4a11:a7b:3a:c4b5:6b6d:2 vol_vpp9xnyyx2p273wv 2024-05-16T15:49:11Z 2024-08-23T13:33:58Z app shared-cpu-2x:512MB
failed to connect to local node: context deadline exceeded
d8d9731ae6e0d8 white-snow-6687 stopped 0/3 ams the machine hasn't started flyio/postgres: (v0.0.41) fdaa:0:4a11:a7b:10e:6b40:e4b0:2 vol_d42w0ky78l9g2634 2023-09-25T09:07:51Z 2024-08-23T13:34:19Z app shared-cpu-4x:8192MB
$ fly checks list -a ***-db
Health Checks for ***-db
NAME | STATUS | MACHINE | LAST UPDATED | OUTPUT
-------*---------*----------------*--------------*-----------------------------
pg | warning | d891d64c696268 | 1m30s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
role | warning | d891d64c696268 | 1m30s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
vm | warning | d891d64c696268 | 1m30s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
pg | warning | d8d9731ae6e0d8 | 7m7s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
role | warning | d8d9731ae6e0d8 | 7m7s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
vm | warning | d8d9731ae6e0d8 | 7m7s ago | the machine hasn't started
-------*---------*----------------*--------------*-----------------------------
$ fly status -a ***-db
Error: unable to fetch latest image details for flyio/postgres:: "Unable to resolve latest upgradeable tag for flyio/postgres:."
fly.toml
# fly.toml app configuration file generated for ***-db on 2023-09-25T11:11:20+02:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "***-db"
primary_region = "ams"
kill_signal = "SIGTERM"
kill_timeout = "5m0s"
[experimental]
enable_consul = true
[env]
PRIMARY_REGION = "ams"
[[mounts]]
source = "pg_data_machines"
destination = "/data"
[checks]
[checks.pg]
port = 5500
type = "http"
interval = "15s"
timeout = "10s"
grace_period = "30s"
method = "get"
path = "/flycheck/pg"
protocol = "http"
tls_skip_verify = false
[checks.role]
port = 5500
type = "http"
interval = "15s"
timeout = "10s"
grace_period = "30s"
method = "get"
path = "/flycheck/role"
protocol = "http"
tls_skip_verify = false
[checks.vm]
port = 5500
type = "http"
interval = "1m0s"
timeout = "10s"
grace_period = "1s"
method = "get"
path = "/flycheck/vm"
protocol = "http"
tls_skip_verify = false
[[metrics]]
port = 9187
path = "/metrics"
Dockerfile
# Use the official lightweight Node.js 12 image.
# https://hub.docker.com/_/node
FROM node:16-slim
# Create and change to the app directory.
WORKDIR /usr/src/app
# Copy application dependency manifests to the container image.
# A wildcard is used to ensure both package.json AND package-lock.json are copied.
# Copying this separately prevents re-running npm install on every code change.
COPY package*.json ./
# Install production dependencies.
RUN npm install --only=production
# Copy local code to the container image.
COPY favicon.png .
COPY config/ config/
COPY public/robots.txt public/
COPY src/ src/
COPY service-account-prod.json .
ENV NODE_ENV=production
RUN npm run build
# Run the web service on container startup.
CMD [ "npm", "start" ]
Logs:
2024-08-23T15:07:48Z app[d891d64c696268] ams [info]2024-08-23T15:07:48.094837396 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:07:48Z app[d891d64c696268] ams [info][ 0.276329] PCI: Fatal: No config space access function found
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:07:48Z app[d891d64c696268] ams [info]2024/08/23 15:07:48 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:07:48Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:07:48Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:07:48Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:07:48Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:07:49Z runner[d891d64c696268] ams [info]Machine started in 1.325s
2024-08-23T15:07:49Z health[d891d64c696268] ams [warn]Health check for your postgres database is warning. Your database might be malfunctioning.
2024-08-23T15:07:49Z health[d891d64c696268] ams [warn]Health check for your postgres vm is warning. Your instance might be hitting resource limits.
2024-08-23T15:07:49Z health[d891d64c696268] ams [warn]Health check for your postgres role is warning. Your cluster's membership might be affected.
2024-08-23T15:07:49Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:07:49Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:07:49Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:07:49Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:07:49Z app[d891d64c696268] ams [info][ 1.671535] reboot: Restarting system
2024-08-23T15:07:50Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:07:50Z app[d891d64c696268] ams [info]2024-08-23T15:07:50.687724716 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:07:51Z app[d891d64c696268] ams [info][ 0.276466] PCI: Fatal: No config space access function found
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:07:51Z app[d891d64c696268] ams [info]2024/08/23 15:07:51 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:07:51Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:07:51Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:07:51Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:07:51Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:07:51Z runner[d891d64c696268] ams [info]Machine started in 1.188s
2024-08-23T15:07:52Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:07:52Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:07:52Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:07:52Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:07:52Z app[d891d64c696268] ams [info][ 1.668530] reboot: Restarting system
2024-08-23T15:07:52Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:07:53Z app[d891d64c696268] ams [info]2024-08-23T15:07:53.660336120 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:07:54Z app[d891d64c696268] ams [info][ 0.279145] PCI: Fatal: No config space access function found
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:07:54Z app[d891d64c696268] ams [info]2024/08/23 15:07:54 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:07:54Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:07:54Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:07:54Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:07:54Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:07:54Z runner[d891d64c696268] ams [info]Machine started in 1.474s
2024-08-23T15:07:55Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:07:55Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:07:55Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:07:55Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:07:55Z app[d891d64c696268] ams [info][ 1.637977] reboot: Restarting system
2024-08-23T15:07:55Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:07:56Z app[d891d64c696268] ams [info]2024-08-23T15:07:56.166369636 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:07:56Z app[d891d64c696268] ams [info][ 0.280851] PCI: Fatal: No config space access function found
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:07:56Z app[d891d64c696268] ams [info]2024/08/23 15:07:56 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:07:56Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:07:56Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:07:56Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:07:56Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:07:57Z runner[d891d64c696268] ams [info]Machine started in 1.158s
2024-08-23T15:07:57Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:07:57Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:07:57Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:07:57Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:07:57Z app[d891d64c696268] ams [info][ 1.656202] reboot: Restarting system
2024-08-23T15:07:58Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:07:58Z health[d891d64c696268] ams [error]Health check for your postgres role has failed. Your cluster's membership is inconsistent.
2024-08-23T15:07:59Z app[d891d64c696268] ams [info]2024-08-23T15:07:59.099145915 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:07:59Z app[d891d64c696268] ams [info][ 0.274267] PCI: Fatal: No config space access function found
2024-08-23T15:07:59Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:07:59Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:07:59Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:07:59Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:07:59Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:07:59Z app[d891d64c696268] ams [info]2024/08/23 15:07:59 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:00Z runner[d891d64c696268] ams [info]Machine started in 1.163s
2024-08-23T15:08:00Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:00Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:00Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:00Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:00Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:00Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:00Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:00Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:00Z app[d891d64c696268] ams [info][ 1.713729] reboot: Restarting system
2024-08-23T15:08:01Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:02Z app[d891d64c696268] ams [info]2024-08-23T15:08:02.982416913 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:08:03Z app[d891d64c696268] ams [info][ 0.276590] PCI: Fatal: No config space access function found
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:08:03Z app[d891d64c696268] ams [info]2024/08/23 15:08:03 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:03Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:03Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:03Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:03Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:03Z runner[d891d64c696268] ams [info]Machine started in 1.445s
2024-08-23T15:08:04Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:04Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:04Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:04Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:04Z app[d891d64c696268] ams [info][ 1.641842] reboot: Restarting system
2024-08-23T15:08:05Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:08Z app[d891d64c696268] ams [info]2024-08-23T15:08:08.601078430 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:08:08Z app[d891d64c696268] ams [info][ 0.275701] PCI: Fatal: No config space access function found
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:08:09Z app[d891d64c696268] ams [info]2024/08/23 15:08:09 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:09Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:09Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:09Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:09Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:09Z runner[d891d64c696268] ams [info]Machine started in 1.084s
2024-08-23T15:08:10Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:10Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:10Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:10Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:10Z app[d891d64c696268] ams [info][ 1.661025] reboot: Restarting system
2024-08-23T15:08:10Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:16Z app[d891d64c696268] ams [info]2024-08-23T15:08:16.241579146 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:08:16Z app[d891d64c696268] ams [info][ 0.277482] PCI: Fatal: No config space access function found
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:08:16Z app[d891d64c696268] ams [info]2024/08/23 15:08:16 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:16Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:16Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:16Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:16Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:17Z runner[d891d64c696268] ams [info]Machine started in 1.174s
2024-08-23T15:08:17Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:17Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:17Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:17Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:17Z app[d891d64c696268] ams [info][ 1.633490] reboot: Restarting system
2024-08-23T15:08:18Z health[d891d64c696268] ams [error]Health check for your postgres database has failed. Your database is malfunctioning.
2024-08-23T15:08:18Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:20Z app[d891d64c696268] ams [info]2024-08-23T15:08:20.455795686 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:08:20Z app[d891d64c696268] ams [info][ 0.277244] PCI: Fatal: No config space access function found
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:08:21Z app[d891d64c696268] ams [info]2024/08/23 15:08:21 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:21Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:21Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:21Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:21Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:21Z runner[d891d64c696268] ams [info]Machine started in 1.162s
2024-08-23T15:08:22Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:22Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:22Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:22Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:22Z app[d891d64c696268] ams [info][ 1.637764] reboot: Restarting system
2024-08-23T15:08:22Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:33Z app[d891d64c696268] ams [info]2024-08-23T15:08:33.412011261 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:08:33Z app[d891d64c696268] ams [info][ 0.285692] PCI: Fatal: No config space access function found
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:08:34Z app[d891d64c696268] ams [info]2024/08/23 15:08:34 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:08:34Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:08:34Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:08:34Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:08:34Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:08:34Z runner[d891d64c696268] ams [info]Machine started in 1.264s
2024-08-23T15:08:35Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:08:35Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:08:35Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:08:35Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:08:35Z app[d891d64c696268] ams [info][ 1.707565] reboot: Restarting system
2024-08-23T15:08:35Z runner[d891d64c696268] ams [info]machine did not have a restart policy, defaulting to restart
2024-08-23T15:08:41Z health[d891d64c696268] ams [error]Health check for your postgres vm has failed. Your instance has hit resource limits. Upgrading your instance / volume size or reducing your usage might help.
2024-08-23T15:09:04Z app[d891d64c696268] ams [info]2024-08-23T15:09:04.420348467 [01J5ZVVYFBHB19XP1JHD62XRRJ:main] Running Firecracker v1.7.0
2024-08-23T15:09:04Z app[d891d64c696268] ams [info][ 0.269182] PCI: Fatal: No config space access function found
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] INFO Starting init (commit: 20f21dc5f)...
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] INFO Resized /data to 10737418240 bytes
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] INFO Preparing to run: `docker-entrypoint.sh start` as root
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] INFO [fly api proxy] listening at /.fly/api
2024-08-23T15:09:05Z app[d891d64c696268] ams [info]2024/08/23 15:09:05 INFO SSH listening listen_address=[fdaa:0:4a11:a7b:3a:c4b5:6b6d:2]:22 dns_server=[fdaa::3]:53
2024-08-23T15:09:05Z app[d891d64c696268] ams [info]panic: FLY_CONSUL_URL or CONSUL_URL are required
2024-08-23T15:09:05Z app[d891d64c696268] ams [info]goroutine 1 [running]:
2024-08-23T15:09:05Z app[d891d64c696268] ams [info]main.main()
2024-08-23T15:09:05Z app[d891d64c696268] ams [info] /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:34 +0x1c13
2024-08-23T15:09:05Z runner[d891d64c696268] ams [info]Machine started in 1.202s
2024-08-23T15:09:06Z app[d891d64c696268] ams [info] INFO Main child exited normally with code: 2
2024-08-23T15:09:06Z app[d891d64c696268] ams [info] INFO Starting clean up.
2024-08-23T15:09:06Z app[d891d64c696268] ams [info] INFO Umounting /dev/vdc from /data
2024-08-23T15:09:06Z app[d891d64c696268] ams [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2024-08-23T15:09:06Z app[d891d64c696268] ams [info][ 1.657376] reboot: Restarting system
2024-08-23T15:09:06Z runner[d891d64c696268] ams [info]machine has reached its max restart count (10)
What can we still do to fix this?