Fly app servers down!

smarties · April 12, 2024, 6:14am

All our phoenix apps are suspended. start 43 minutes ago. the db servers are still up.
We tried restarting:

$ fly apps restart lyn-pay
Restarting machine 18579eda435778
Error: could not stop machine 18579eda435778: failed to restart VM 18579eda435778: internal: internal server error (Request ID: 01HV8EKW05MPDG72JJBT8P9EW2-fra)

We also tried redeploying:

$ fly deploy -c prod.fly.toml
==> Verifying app config
Validating prod.fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-falling-thunder-5451`

WARN Failed to start remote builder heartbeat: remote builder app unavailable

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-falling-thunder-5451`

Error: failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

czue · April 12, 2024, 6:22am

Seeing something similar on my machines. App server went down and I can’t restart them. Here’s the message I’m getting on fly apps restart.

Error: could not stop machine e784927a630738: failed to restart VM e784927a630738: unknown: failed to compare "/opt/firecracker-v0.24.x/bin/firecracker" and "/opt/flyd/firefly/01HV3EC3HVV5J2GRBMVE2TPSRH/root/firecracker": stat /opt/firecracker-v0.24.x/bin/firecracker: no such file or directory (Request ID: 01HV8F8PFQHPAZWRJEBXA3JGPA-jnb)

AkshitGarg · April 12, 2024, 6:24am

There is an ongoing incident. A fix has been deployed. Can you please try again and let us know?

smarties · April 12, 2024, 6:34am

Still same error here are the logs for deployment:

$ flyctl logs -a fly-builder-falling-thunder-5451
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.970325086Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.970347617Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.970392568Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971596020Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971626290Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971649651Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971667672Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971684812Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971705243Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971722323Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971740453Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.971759684Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972215556Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972243007Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972262327Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972277558Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972297828Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972321029Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972350230Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972370390Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972402121Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972420001Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972437302Z" level=info msg="skipping tracing processor initialization (no tracing plugin)" error="no OpenTelemetry endpoint: skip plugin"
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.972977586Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.973045788Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.973115420Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
2024-04-11T10:24:25Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:25.973156011Z" level=info msg="containerd successfully booted in 0.034920s"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.015778209Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.033005485Z" level=info msg="Loading containers: start."
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.101324353Z" level=error msg="Could not add route to IPv6 network 2001:db8:1::1/64 via device docker0: network is down"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.461856114Z" level=error msg="Could not add route to IPv6 network 2001:db8:1::1/64 via device docker0: network is down"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.515644687Z" level=info msg="Loading containers: done."
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.549388670Z" level=warning msg="WARNING: API is accessible on http://0.0.0.0:2375 without encryption.\n         Access to the remote API is equivalent to root access on the host. Refer\n         to the 'Docker daemon attack surface' section in the documentation for\n         more information: https://docs.docker.com/go/attack-surface/"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.549433031Z" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2 version=24.0.7
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.549744770Z" level=info msg="metrics API listening on [::]:9323"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.550024267Z" level=info msg="Daemon has completed initialization"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.603679487Z" level=info msg="API listen on /var/run/docker.sock"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.603754769Z" level=info msg="API listen on [::]:2375"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.604929440Z" level=info msg="disk space used: 3.11%"
2024-04-11T10:24:26Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:26.605397592Z" level=info msg="Listening on :8080"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.123974460Z" level=debug msg="--> POST https://api.fly.io/graphql\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.124008371Z" level=debug msg="{\"query\":\"query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug } postgresAppRole: role { name } imageDetails { repository version } } }\",\"variables\":{\"appName\":\"yumly-dev\"}}\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.124029661Z" level=debug msg="{}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.381348791Z" level=debug msg="<-- 200 https://api.fly.io/graphql (257.35ms)\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.381429333Z" level=debug msg="{\"data\":{\"appcompact\":{\"id\":\"yumly-dev\",\"name\":\"yumly-dev\",\"hostname\":\"yumly-dev.fly.dev\",\"deployed\":true,\"status\":\"deployed\",\"appUrl\":\"https://2a09:8280:1::31:2a6a:0\",\"platformVersion\":\"machines\",\"organization\":{\"id\":\"P7lZB0nw2ylg8smzmMLA9eVLAQuXGz\",\"slug\":\"personal\"},\"postgresAppRole\":null,\"imageDetails\":{\"repository\":\"unknown\",\"version\":\"unknown\"}}}}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.383141238Z" level=debug msg="--> POST https://api.fly.io/graphql\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.383163689Z" level=debug msg="{\"query\":\"query ($appName: String!) { appcompact:app(name: $appName) { id name hostname deployed status appUrl platformVersion organization { id slug } postgresAppRole: role { name } imageDetails { repository version } } }\",\"variables\":{\"appName\":\"fly-builder-falling-thunder-5451\"}}\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.383173279Z" level=debug msg="{}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.522258670Z" level=debug msg="<-- 200 https://api.fly.io/graphql (139.1ms)\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.522334452Z" level=debug msg="{\"data\":{\"appcompact\":{\"id\":\"fly-builder-falling-thunder-5451\",\"name\":\"fly-builder-falling-thunder-5451\",\"hostname\":\"fly-builder-falling-thunder-5451.fly.dev\",\"deployed\":true,\"status\":\"suspended\",\"appUrl\":null,\"platformVersion\":\"machines\",\"organization\":{\"id\":\"P7lZB0nw2ylg8smzmMLA9eVLAQuXGz\",\"slug\":\"personal\"},\"postgresAppRole\":{\"name\":\"remote-docker-builder\"},\"imageDetails\":{\"repository\":\"unknown\",\"version\":\"unknown\"}}}}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.522457175Z" level=debug msg="--> POST https://api.fly.io/graphql\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.522467995Z" level=debug msg="{\"query\":\"query($slug: String!) { organization(slug: $slug) { id internalNumericId slug name type } }\",\"variables\":{\"slug\":\"personal\"}}\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.522475355Z" level=debug msg="{}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.924750671Z" level=debug msg="<-- 200 https://api.fly.io/graphql (402.27ms)\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.924811643Z" level=debug msg="{\"data\":{\"organization\":{\"id\":\"P7lZB0nw2ylg8smzmMLA9eVLAQuXGz\",\"internalNumericId\":\"60236\",\"slug\":\"personal\",\"name\":\"Simon El Nahas\",\"type\":\"PERSONAL\"}}}"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.924918736Z" level=debug msg="--> POST https://api.fly.io/graphql\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.924928486Z" level=debug msg="{\"query\":\"query($slug: String!) { organization(slug: $slug) { id internalNumericId slug name type } }\",\"variables\":{\"slug\":\"personal\"}}\n"
2024-04-11T10:24:28Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:28.924936116Z" level=debug msg="{}"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.170344691Z" level=debug msg="<-- 200 https://api.fly.io/graphql (245.4ms)\n"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.170405052Z" level=debug msg="{\"data\":{\"organization\":{\"id\":\"P7lZB0nw2ylg8smzmMLA9eVLAQuXGz\",\"internalNumericId\":\"60236\",\"slug\":\"personal\",\"name\":\"Simon El Nahas\",\"type\":\"PERSONAL\"}}}"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.170458704Z" level=debug msg="authorized from api"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.170468864Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.31"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.170506265Z" level=info msg="fdaa:0:8231:a7b:163a:0:a:e00 - - [11/Apr/2024:10:24:28 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.211278584Z" level=debug msg="authorized from cache"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.211324775Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.31"
2024-04-11T10:24:29Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:29.211368106Z" level=info msg="fdaa:0:8231:a7b:163a:0:a:e00 - - [11/Apr/2024:10:24:29 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-11T10:24:30Z app[32871727cd1385] ams [info] INFO Sending signal SIGUSR1 to main child process w/ PID 343
2024-04-11T10:24:35Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:35.875556626Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-11T10:24:35Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:35.875847833Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-11T10:24:43Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:43.573022348Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-11T10:24:43Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:43.573463800Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-11T10:24:43Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:43.673696702Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-11T10:24:43Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:43.673955169Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-11T10:24:44Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:44.514641897Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-11T10:24:44Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:44.514961065Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-11T10:24:44Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:44.613804071Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-11T10:24:44Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:44.614074898Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-11T10:24:46Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:46.249039805Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-11T10:24:46Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:46.249505428Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-11T10:24:46Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:46.888982961Z" level=warning msg="no trace recorder found, skipping"
2024-04-11T10:24:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:59.262421740Z" level=debug msg="authorized from cache"
2024-04-11T10:24:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:59.262454621Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.31"
2024-04-11T10:24:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:24:59.262488222Z" level=info msg="fdaa:0:8231:a7b:163a:0:a:e00 - - [11/Apr/2024:10:24:59 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236284443Z" level=info msg="deadline reached, no active builds, shutting down"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236394296Z" level=info msg="init shutdown"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236436707Z" level=info msg="shutting down proxy"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236553770Z" level=info msg="shutting down proxy"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236585911Z" level=info msg="disk space used: 3.37%"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.236712114Z" level=info msg="Processing signal 'interrupt'"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.237580307Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.237801963Z" level=info msg="Daemon shutdown complete"
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.237860594Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
2024-04-11T10:34:59Z app[32871727cd1385] ams [info]time="2024-04-11T10:34:59.237881415Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
2024-04-11T10:35:00Z app[32871727cd1385] ams [info]time="2024-04-11T10:35:00.241422352Z" level=info msg="dockerd has exited"
2024-04-11T10:35:00Z app[32871727cd1385] ams [info]time="2024-04-11T10:35:00.241463063Z" level=info msg="shutdown complete"
2024-04-11T10:35:00Z app[32871727cd1385] ams [info] INFO Main child exited normally with code: 0
2024-04-11T10:35:00Z app[32871727cd1385] ams [info] INFO Starting clean up.
2024-04-11T10:35:00Z app[32871727cd1385] ams [info] INFO Umounting /dev/vdb from /data
2024-04-11T10:35:00Z app[32871727cd1385] ams [info][  636.242968] reboot: Restarting system
2024-04-11T10:35:00Z runner[32871727cd1385] ams [info]machine exited with exit code 0, not restarting

AkshitGarg · April 12, 2024, 6:36am

Hey! Can you try starting the machine again and let us know if it works. This was an issue from our side.

smarties · April 12, 2024, 6:36am

$ fly apps restart lyn-pay
Restarting machine 18579eda435778
Error: could not stop machine 18579eda435778: failed to restart VM 18579eda435778: internal: internal server error (Request ID: 01HV8G88MNQ8C7TT5EZ48HZR60-fra)

Ziwen · April 12, 2024, 6:36am

Same for me. The status page showed “A fix has been implemented…” but didn’t seem to be working.

2024-04-12T06:36:14.412 proxy[918571d0a924d8] hkg [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)

2024-04-12T06:36:18.476 proxy[918571d0a924d8] hkg [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)

2024-04-12T06:36:22.517 proxy[918571d0a924d8] hkg [error] timed out while connecting to your instance. this indicates a problem with your app (hint: look at your logs and metrics)

smarties · April 12, 2024, 6:37am

$ fly apps restart lyn-pay --force-stop
Restarting machine 18579eda435778
Error: could not stop machine 18579eda435778: failed to restart VM 18579eda435778: internal: internal server error (Request ID: 01HV8GCW1PM6R4KJEVYEG0ENJ5-fra)

xwiles · April 12, 2024, 6:41am

Also facing an issue. My app went down and i am unable to get a machine to start. Seeing this in the logs:

2024-04-12T06:39:06Z runner[080e091f650218] sin [info]Configuring firecracker
2024-04-12T06:39:07Z app[080e091f650218] sin [info][    0.070891] PCI: Fatal: No config space access function found
2024-04-12T06:39:07Z app[080e091f650218] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:39:07Z app[080e091f650218] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:39:07Z app[080e091f650218] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:39:07Z app[080e091f650218] sin [info]2024/04/12 06:39:07 listening on [fdaa:0:4d47:a7b:185:eaf5:5fd:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:39:07Z runner[080e091f650218] sin [info]Machine created and started in 4.47s
2024-04-12T06:39:12Z app[080e091f650218] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:eaf5:5fd:2
2024-04-12T06:39:12Z app[080e091f650218] sin [info][    5.543704] reboot: Restarting system
2024-04-12T06:39:13Z runner[080e091f650218] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T06:39:13Z app[080e091f650218] sin [info][    0.078333] PCI: Fatal: No config space access function found
2024-04-12T06:39:14Z app[080e091f650218] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:39:14Z app[080e091f650218] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:39:14Z app[080e091f650218] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:39:14Z runner[080e091f650218] sin [info]Machine started in 777ms
2024-04-12T06:39:18Z app[080e091f650218] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:eaf5:5fd:2
2024-04-12T06:39:19Z app[080e091f650218] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:39:19Z app[080e091f650218] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:39:19Z app[080e091f650218] sin [info] INFO Starting clean up.
2024-04-12T06:39:19Z app[080e091f650218] sin [info][    5.591877] reboot: Restarting system
2024-04-12T06:39:19Z runner[080e091f650218] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T06:39:19Z app[080e091f650218] sin [info][    0.068293] PCI: Fatal: No config space access function found
2024-04-12T06:39:20Z app[080e091f650218] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:39:20Z app[080e091f650218] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:39:20Z app[080e091f650218] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:39:20Z app[080e091f650218] sin [info]2024/04/12 06:39:20 listening on [fdaa:0:4d47:a7b:185:eaf5:5fd:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:39:20Z runner[080e091f650218] sin [info]Machine started in 722ms
2024-04-12T06:39:24Z app[080e091f650218] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:eaf5:5fd:2
2024-04-12T06:39:25Z app[080e091f650218] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:39:25Z app[080e091f650218] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:39:25Z app[080e091f650218] sin [info] INFO Starting clean up.

I have tried restarting existing machines and creating new machines. Have been trying repeatedly since the status page said the fix has been deployed.

AkshitGarg · April 12, 2024, 6:44am

Hey @smarties and @Ziwen. Looks like your machines got stuck in an invalid state. Can you please try force destroying them and re-creating them if possible?

xwiles · April 12, 2024, 6:47am

My app suddenly went down with this incident and i cannot get a machine to successfully start again.

Machine stuck in a stopped state

fly machines list
1 machines have been retrieved from app reggu-prod.
View them in the UI here (https://fly.io/apps/reggu-prod/machines/)

reggu-prod
ID            	NAME            	STATE  	REGION	IMAGE      	IP ADDRESS                     	VOLUME	CREATED             	LAST UPDATED        	APP PLATFORM	PROCESS GROUP	SIZE
683d610a716418	cold-violet-4297	stopped	sin   	reggu-prod:	fdaa:0:4d47:a7b:185:c50b:a577:2	      	2024-04-12T06:43:08Z	2024-04-12T06:44:16Z	v2          	app          	shared-cpu-1x:1024MB

I force destroy it

fly machines destroy 683d610a716418 --force
machine 683d610a716418 was found and is currently in stopped state, attempting to destroy...
683d610a716418 has been destroyed

Create a new machine

fly scale count 1 --region sin
App 'reggu-prod' is going to be scaled according to this plan:
  +1 machines for group 'app' on region 'sin' of size 'shared-cpu-1x'
? Scale app reggu-prod? Yes
Executing scale plan
  Created 48ed192a71ded8 group:app region:sin size:shared-cpu-1x

Machine fails to start repeatedly.

2024-04-12T06:45:01Z runner[48ed192a71ded8] sin [info]Successfully prepared image registry.fly.io/reggu-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3 (2.970973845s)
2024-04-12T06:45:02Z runner[48ed192a71ded8] sin [info]Configuring firecracker
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info][    0.077384] PCI: Fatal: No config space access function found
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info]2024/04/12 06:45:03 listening on [fdaa:0:4d47:a7b:185:c1c6:c461:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:45:03Z runner[48ed192a71ded8] sin [info]Machine created and started in 4.627s
2024-04-12T06:45:07Z app[48ed192a71ded8] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:c1c6:c461:2
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Starting clean up.
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info][    5.532757] reboot: Restarting system
2024-04-12T06:45:08Z runner[48ed192a71ded8] sin [info]machine did not have a restart policy, defaulting to restart

Nothing changed on my end

czue · April 12, 2024, 6:50am

Also seeing this erorr. What are the steps for force destroying and re-creating?

xwiles · April 12, 2024, 6:50am

My app went down with this incident too and i cannot get machines up again.

I created a machine and it is stuck in a stopped state

fly machines list
1 machines have been retrieved from app reggu-prod.
View them in the UI here (https://fly.io/apps/reggu-prod/machines/)

reggu-prod
ID            	NAME            	STATE  	REGION	IMAGE      	IP ADDRESS                     	VOLUME	CREATED             	LAST UPDATED        	APP PLATFORM	PROCESS GROUP	SIZE
683d610a716418	cold-violet-4297	stopped	sin   	reggu-prod:	fdaa:0:4d47:a7b:185:c50b:a577:2	      	2024-04-12T06:43:08Z	2024-04-12T06:44:16Z	v2          	app          	shared-cpu-1x:1024MB

So I destroy the machine

fly machines destroy 683d610a716418 --force
machine 683d610a716418 was found and is currently in stopped state, attempting to destroy...
683d610a716418 has been destroyed

Create a new machine

fly scale count 1 --region sin
App 'reggu-prod' is going to be scaled according to this plan:
  +1 machines for group 'app' on region 'sin' of size 'shared-cpu-1x'
? Scale app reggu-prod? Yes
Executing scale plan
  Created 48ed192a71ded8 group:app region:sin size:shared-cpu-1x

Then the machine fails to start repeatedly.

2024-04-12T06:45:01Z runner[48ed192a71ded8] sin [info]Successfully prepared image registry.fly.io/reggu-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3 (2.970973845s)
2024-04-12T06:45:02Z runner[48ed192a71ded8] sin [info]Configuring firecracker
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info][    0.077384] PCI: Fatal: No config space access function found
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info]2024/04/12 06:45:03 listening on [fdaa:0:4d47:a7b:185:c1c6:c461:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:45:03Z runner[48ed192a71ded8] sin [info]Machine created and started in 4.627s
2024-04-12T06:45:07Z app[48ed192a71ded8] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:c1c6:c461:2
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Starting clean up.
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info][    5.532757] reboot: Restarting system
2024-04-12T06:45:08Z runner[48ed192a71ded8] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info][    0.066917] PCI: Fatal: No config space access function found
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:09Z runner[48ed192a71ded8] sin [info]Machine started in 702ms

Nothing has changed on my end

AkshitGarg · April 12, 2024, 6:58am

flyd m destroy --force <your machine id> and then starting a machine on your app as usual. The issue was caused due to a bad configuration being deployed on some workers. We rolled back the configuration change, but some machines got stuck in an invalid state. We are sorry for the inconvenience caused.

xwiles · April 12, 2024, 7:01am

My app went down with this incident too and i cannot get machines up again.

I created a machine and it is stuck in a stopped state

fly machines list
1 machines have been retrieved from app reggu-prod.
View them in the UI here (https://fly.io/apps/reggu-prod/machines/)

reggu-prod
ID            	NAME            	STATE  	REGION	IMAGE      	IP ADDRESS                     	VOLUME	CREATED             	LAST UPDATED        	APP PLATFORM	PROCESS GROUP	SIZE
683d610a716418	cold-violet-4297	stopped	sin   	reggu-prod:	fdaa:0:4d47:a7b:185:c50b:a577:2	      	2024-04-12T06:43:08Z	2024-04-12T06:44:16Z	v2          	app          	shared-cpu-1x:1024MB

So I destroy the machine

fly machines destroy 683d610a716418 --force
machine 683d610a716418 was found and is currently in stopped state, attempting to destroy...
683d610a716418 has been destroyed

Create a new machine

fly scale count 1 --region sin
App 'reggu-prod' is going to be scaled according to this plan:
  +1 machines for group 'app' on region 'sin' of size 'shared-cpu-1x'
? Scale app reggu-prod? Yes
Executing scale plan
  Created 48ed192a71ded8 group:app region:sin size:shared-cpu-1x

Then the machine fails to start repeatedly.

2024-04-12T06:45:01Z runner[48ed192a71ded8] sin [info]Successfully prepared image registry.fly.io/reggu-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3 (2.970973845s)
2024-04-12T06:45:02Z runner[48ed192a71ded8] sin [info]Configuring firecracker
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info][    0.077384] PCI: Fatal: No config space access function found
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info]2024/04/12 06:45:03 listening on [fdaa:0:4d47:a7b:185:c1c6:c461:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:45:03Z runner[48ed192a71ded8] sin [info]Machine created and started in 4.627s
2024-04-12T06:45:07Z app[48ed192a71ded8] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:c1c6:c461:2
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Starting clean up.
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info][    5.532757] reboot: Restarting system
2024-04-12T06:45:08Z runner[48ed192a71ded8] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info][    0.066917] PCI: Fatal: No config space access function found
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:09Z runner[48ed192a71ded8] sin [info]Machine started in 702ms

Nothing has changed on my end

smarties · April 12, 2024, 7:03am

$ fly m destroy --force 784e935f435378
machine 784e935f435378 was found and is currently in starting state, attempting to destroy...
784e935f435378 has been destroyed

But how do we start them again?

restart command still doesn’t work:

$ fly apps restart lyn-pay
Restarting machine 18579eda435778
Error: could not stop machine 18579eda435778: failed to restart VM 18579eda435778: failed_precondition: unable to restart machine, not currently started or stopped (Request ID: 01HV8HTP2Y11T8R5HGFHBRHX83-fra)

xwiles · April 12, 2024, 7:05am

I am still having an issue getting machines to start and my app is fully down. I have tried force deleting machines and starting new ones. None actually start and I am seeing this restart loop in the logs:

2024-04-12T07:02:10Z runner[185716eb427568] sin [info]Machine started in 770ms
2024-04-12T07:02:10Z app[185716eb427568] sin [info]2024/04/12 07:02:10 listening on [fdaa:0:4d47:a7b:185:6d4d:9726:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T07:02:14Z app[185716eb427568] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:6d4d:9726:2
2024-04-12T07:02:15Z app[185716eb427568] sin [info] INFO Main child exited normally with code: 1
2024-04-12T07:02:15Z app[185716eb427568] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T07:02:15Z app[185716eb427568] sin [info] INFO Starting clean up.
2024-04-12T07:02:15Z app[185716eb427568] sin [info][    5.587404] reboot: Restarting system

xwiles · April 12, 2024, 7:09am

My app went down with this incident too and i cannot get machines up again.

I created a machine and it is stuck in a stopped state

fly machines list
1 machines have been retrieved from app reggu-prod.
View them in the UI here (https://fly.io/apps/reggu-prod/machines/)

reggu-prod
ID            	NAME            	STATE  	REGION	IMAGE      	IP ADDRESS                     	VOLUME	CREATED             	LAST UPDATED        	APP PLATFORM	PROCESS GROUP	SIZE
683d610a716418	cold-violet-4297	stopped	sin   	reggu-prod:	fdaa:0:4d47:a7b:185:c50b:a577:2	      	2024-04-12T06:43:08Z	2024-04-12T06:44:16Z	v2          	app          	shared-cpu-1x:1024MB

So I destroy the machine

fly machines destroy 683d610a716418 --force
machine 683d610a716418 was found and is currently in stopped state, attempting to destroy...
683d610a716418 has been destroyed

Create a new machine

fly scale count 1 --region sin
App 'reggu-prod' is going to be scaled according to this plan:
  +1 machines for group 'app' on region 'sin' of size 'shared-cpu-1x'
? Scale app reggu-prod? Yes
Executing scale plan
  Created 48ed192a71ded8 group:app region:sin size:shared-cpu-1x

Then the machine fails to start repeatedly.

2024-04-12T06:45:01Z runner[48ed192a71ded8] sin [info]Successfully prepared image registry.fly.io/reggu-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3 (2.970973845s)
2024-04-12T06:45:02Z runner[48ed192a71ded8] sin [info]Configuring firecracker
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info][    0.077384] PCI: Fatal: No config space access function found
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:03Z app[48ed192a71ded8] sin [info]2024/04/12 06:45:03 listening on [fdaa:0:4d47:a7b:185:c1c6:c461:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T06:45:03Z runner[48ed192a71ded8] sin [info]Machine created and started in 4.627s
2024-04-12T06:45:07Z app[48ed192a71ded8] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:c1c6:c461:2
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Main child exited normally with code: 1
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info] INFO Starting clean up.
2024-04-12T06:45:08Z app[48ed192a71ded8] sin [info][    5.532757] reboot: Restarting system
2024-04-12T06:45:08Z runner[48ed192a71ded8] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info][    0.066917] PCI: Fatal: No config space access function found
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T06:45:09Z app[48ed192a71ded8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T06:45:09Z runner[48ed192a71ded8] sin [info]Machine started in 702ms

Nothing has changed on my end

AkshitGarg · April 12, 2024, 7:11am

@smarties 18579eda435778 was the machine that was in the bad state. You can destroy that one and bring up more machines first in your preferred region using fly scale -r <region> count <number of machines you want>.

@xwiles looks like your machine 48ed192a71ded8 was destroyed. Can you try to scale up again, and let me know if it solves the issue.

xwiles · April 12, 2024, 7:13am

I have repeated the process described and I am still having the issue.

The new machine id is 78113d5f9420d8 and I am still seeing a restart loop (logs below). I will keep this machine around

2024-04-12T07:11:15Z runner[78113d5f9420d8] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T07:11:16Z app[78113d5f9420d8] sin [info][    0.066774] PCI: Fatal: No config space access function found
2024-04-12T07:11:16Z app[78113d5f9420d8] sin [info] INFO Starting init (commit: 5b8fb02)...
2024-04-12T07:11:16Z app[78113d5f9420d8] sin [info] INFO Preparing to run: `/app/bin/server` as nobody
2024-04-12T07:11:16Z app[78113d5f9420d8] sin [info] INFO [fly api proxy] listening at /.fly/api
2024-04-12T07:11:16Z app[78113d5f9420d8] sin [info]2024/04/12 07:11:16 listening on [fdaa:0:4d47:a7b:185:6577:c25b:2]:22 (DNS: [fdaa::3]:53)
2024-04-12T07:11:16Z runner[78113d5f9420d8] sin [info]Machine started in 749ms
2024-04-12T07:11:21Z app[78113d5f9420d8] sin [info]Protocol 'inet6_tcp': invalid node name: reggu-prod-prod@sha256:3e1793d3272cef58b31c512fd85176f33e2f4610bb7176373525235a0ab586e3@fdaa:0:4d47:a7b:185:6577:c25b:2
2024-04-12T07:11:21Z app[78113d5f9420d8] sin [info] INFO Main child exited normally with code: 1
2024-04-12T07:11:21Z app[78113d5f9420d8] sin [info] WARN Reaped child process with pid: 363 and signal: SIGUSR1, core dumped? false
2024-04-12T07:11:21Z app[78113d5f9420d8] sin [info] INFO Starting clean up.
2024-04-12T07:11:21Z app[78113d5f9420d8] sin [info][    5.515875] reboot: Restarting system
2024-04-12T07:11:22Z runner[78113d5f9420d8] sin [info]machine did not have a restart policy, defaulting to restart
2024-04-12T07:11:22Z app[78113d5f9420d8] sin [info][    0.073521] PCI: Fatal: No config space access function found

Topic		Replies	Views
Application VMs down without any change, can't deploy Phoenix	16	1305	October 3, 2022
Trying to deploy a Phoenix app, nothing works, close to giving up	6	729	December 11, 2021
fly deploy passes - but no activity on Endpoint Phoenix	12	407	October 3, 2022
Sudden networking issue from app to db	5	441	May 7, 2023
500: Server Error - App Activity (/apps/app-name/activity) Questions / Help	9	704	December 23, 2021

Fly app servers down!

Related topics