Networking Problem in Release Command

Questions / Help Build debugging
,
1

Hi there,
since months we have on and off issues with deploying our app inside Fly. Sometimes it bugs for hours where it cannot connect from the release command to our database.

Our machines all connect to our Tailscale network to get access. In the past it worked perfectly, but then it started getting more more apparent. Today the issue persists since hours. But the thing is - all running machines work ok. There is no problem with their connection to the database server. I can even scale more machines.

I added a pg_isready to our entrypoint now and it seems like the release command firecracker cannot get a connection at all (container is waiting for a connection since half an hour now).

I’d have loved to analyze this issue further now, but I was surprised to see that I cannot ssh into a release instance? I really don’t know what else to debug now. The machine itself is pingable inside my tailscale network.

I hope somebody can help me here… :confused:

P.S: If I clone the release machine, it has the exact same problem

2

Hm… This part should be possible, at least, via the non-obvious --address flag. (And you can check the RELEASE_COMMAND environment variable once you’re in there, to verify.)

$ fly m list  # note IP ADDRESS
$ fly ssh console --address fdaa::<hex-digits>
# printenv RELEASE_COMMAND
1

Hope this helps a little!

Aside: There was a cluster of reports of the egress-ip feature failing recently. I wonder if you guys are all in the same region, :thought_balloon:…

4

Thank you very much for telling me about the address option! Though I literally found it myself 2 minutes before you posted it :smiley: Analyzing it now.

1 Like
5

It really has been a race condition between starting Tailnet and doing the database connect! Thanks again for the tip with the address. Tbh it’s a bummer that there has been work put into hiding the release containers from the machines response in the CLI. This is crucial in debugging deployments! I hope this decision is being revisited :slightly_smiling_face: