Failed due to unhealthy allocations in syd

Questions / Help Build debugging
21

Bit of a hobby project and very little traffic so low numbers :slight_smile:

Uptime robot says time out. Here’s the trace route:

Tracing route to 109.105.216.45
hop no  -  node ip - ms
1 → 216.245.214.73 (1 ms)
2 → 63.143.63.145 (1 ms)
3 → 208.115.252.21 (1 ms)
4 → 172.22.1.17 (2 ms)
5 → 184.105.11.129 (39 ms)
6 → 184.105.81.209 (32 ms)
7 → 109.105.216.45 (15001 ms) Request timed out
8 → 109.105.216.45 (15000 ms) Request timed out
9 → 109.105.216.45 (15002 ms) Request timed out

I didn’t think to try the .fly.dev address! Maybe I’ll make another uptime robot check for that one as well.

22

There was a brief bit of temporary unavailability on a single syd host after we restarted an internal system service. This update should be seamless, and we’re actively investigating why it wasn’t so these kinds of blips won’t happen again. Sorry for the inconvenience!

23

It’s probably worth noting that I’m now mainly seeing these failures when deploying via GH Actions. Deployment from local seems to be working more reliably now, but still fails sometimes.

Here’s the fly logs while the GH Action is running:

2022-05-13T00:22:04Z runner[868efff6] syd [info]Starting instance
2022-05-13T00:22:04Z runner[868efff6] syd [info]Configuring virtual machine
2022-05-13T00:22:04Z runner[868efff6] syd [info]Pulling container image
2022-05-13T00:22:12Z runner[868efff6] syd [info]Unpacking image
2022-05-13T00:22:14Z runner[868efff6] syd [info]Preparing kernel init
2022-05-13T00:22:15Z runner[868efff6] syd [info]Configuring firecracker
2022-05-13T00:22:15Z runner[868efff6] syd [info]Starting virtual machine
2022-05-13T00:22:15Z app[868efff6] syd [info]Starting init (commit: 252b7bd)...
2022-05-13T00:22:15Z app[868efff6] syd [info]Preparing to run: `/app/bin/vex_liveview_prototype eval VexLiveviewPrototype.Release.migrate` as nobody
2022-05-13T00:22:15Z app[868efff6] syd [info]2022/05/13 00:22:15 listening on [fdaa:0:59b1:a7b:9c3e:868e:fff6:2]:22 (DNS: [fdaa::3]:53)
2022-05-13T00:22:18Z app[868efff6] syd [info]00:22:18.744 [info] Migrations already up
2022-05-13T00:22:19Z app[868efff6] syd [info]Main child exited normally with code: 0
2022-05-13T00:22:19Z app[868efff6] syd [info]Reaped child process with pid: 569 and signal: SIGUSR1, core dumped? false
2022-05-13T00:22:19Z app[868efff6] syd [info]Starting clean up.
2022-05-13T00:22:29Z runner[dee7fb29] syd [info]Starting instance
2022-05-13T00:22:30Z runner[dee7fb29] syd [info]Configuring virtual machine
2022-05-13T00:22:30Z runner[dee7fb29] syd [info]Pulling container image
2022-05-13T00:22:31Z runner[dee7fb29] syd [info]Unpacking image
2022-05-13T00:22:31Z runner[dee7fb29] syd [info]Preparing kernel init
2022-05-13T00:22:32Z runner[dee7fb29] syd [info]Configuring firecracker
2022-05-13T00:22:33Z runner[dee7fb29] syd [info]Starting virtual machine
2022-05-13T00:22:33Z app[dee7fb29] syd [info]Starting init (commit: 252b7bd)...
2022-05-13T00:22:33Z app[dee7fb29] syd [info]Preparing to run: `/app/bin/server` as nobody
2022-05-13T00:22:33Z app[dee7fb29] syd [info]2022/05/13 00:22:33 listening on [fdaa:0:59b1:a7b:9c3e:dee7:fb29:2]:22 (DNS: [fdaa::3]:53)
2022-05-13T00:22:34Z app[dee7fb29] syd [info]Reaped child process with pid: 555, exit code: 0
2022-05-13T00:22:37Z app[dee7fb29] syd [info]00:22:37.253 [info] Running VexLiveviewPrototypeWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
2022-05-13T00:22:37Z app[dee7fb29] syd [info]00:22:37.256 [info] Access VexLiveviewPrototypeWeb.Endpoint at http://vex.fly.dev
2022-05-13T00:22:37Z app[dee7fb29] syd [info]Reaped child process with pid: 576 and signal: SIGUSR1, core dumped? false
2022-05-13T00:22:41Z app[dee7fb29] syd [info]00:22:41.297 [info] tzdata release in place is from a file last modified Fri, 22 Oct 2021 02:20:47 GMT. Release file on server was last modified Wed, 16 Mar 2022 13:36:02 GMT.
2022-05-13T00:22:43Z app[dee7fb29] syd [info]00:22:43.211 [info] Tzdata has updated the release from 2021e to 2022a

#####################    App just sits here for several minutes

2022-05-13T00:27:44Z runner[dee7fb29] syd [info]Shutting down virtual machine
2022-05-13T00:27:44Z app[dee7fb29] syd [info]00:27:44.260 [notice] SIGTERM received - shutting down
2022-05-13T00:27:44Z app[dee7fb29] syd [info]Sending signal SIGTERM to main child process w/ PID 516
2022-05-13T00:27:45Z app[dee7fb29] syd [info]Reaped child process with pid: 579 and signal: SIGUSR1, core dumped? false
2022-05-13T00:27:46Z app[dee7fb29] syd [info]Main child exited normally with code: 0
2022-05-13T00:27:46Z app[dee7fb29] syd [info]Reaped child process with pid: 560, exit code: 0
2022-05-13T00:27:46Z app[dee7fb29] syd [info]Starting clean up.
24

Anything else we can try here @jsierles ? Our deploys are working fine from local, but every time we try to deploy via a GH Action it fails and ends up rolling back or down. Do you an example of a working GH action to deploy a Phoenix app to fly?

Thanks!

25

Hi @martin1

Have you tried clearing your local environment and going through the GH Action steps locally? There might be something in your local environment thats missing from the GH Action environment.

26

Hi @charsleysa,

Looks like there was an issue trying to get the Git revision as an env var and removing that has fixed the issue. Thanks for the tip. Now to figure out how to do that with breaking the build…