fly deploy to false "remote builder app unavailable"

Questions / Help Build debugging
1

Hello Fly.io Support Team,

During the deployment process, I noticed the following warnings:

Firstly, I want to highlight that my application is hosted in the Tokyo region. However, while running the fly deploy command, I noticed that the logs indicate the builder is located in Hong Kong (hkg). This discrepancy between the application’s intended region and the builder’s location raises some questions and concerns for me.

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-wispy-voice-7350`

WARN Failed to start remote builder heartbeat: remote builder app unavailable

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-wispy-voice-7350`

Here are a few examples from the logs:

[Log entries]

2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.050449533Z" level=debug msg="authorized from api"
2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.050461053Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.33"
2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.050521903Z" level=info msg="fdaa:6:7e8b:a7b:8f98:0:a:700 - - [16/Apr/2024:05:28:18 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.143373598Z" level=debug msg="authorized from cache"
2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.143404448Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.33"
2024-04-16T05:28:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:20.143439838Z" level=info msg="fdaa:6:7e8b:a7b:8f98:0:a:700 - - [16/Apr/2024:05:28:20 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-16T05:28:21Z app[9080e331a9d598] hkg [info] INFO Sending signal SIGUSR1 to main child process w/ PID 342
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.258543392Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.258883562Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.725577564Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.725972834Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.827387458Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-16T05:28:48Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:48.827829288Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-16T05:28:49Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:49.115237821Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-16T05:28:49Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:49.115634481Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-16T05:28:50Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:50.283660071Z" level=debug msg="authorized from cache"
2024-04-16T05:28:50Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:50.283723711Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.33"
2024-04-16T05:28:50Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:28:50.283790401Z" level=info msg="fdaa:6:7e8b:a7b:8f98:0:a:700 - - [16/Apr/2024:05:28:50 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-16T05:29:05Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:05.419078830Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:05Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:05.419445890Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:20.238413852Z" level=debug msg="authorized from cache"
2024-04-16T05:29:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:20.238449332Z" level=info msg="extendDeadline called with user agent: flyctl/0.2.33"
2024-04-16T05:29:20Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:20.238489732Z" level=info msg="fdaa:6:7e8b:a7b:8f98:0:a:700 - - [16/Apr/2024:05:29:20 +0000] \"GET /flyio/v1/extendDeadline HTTP/1.1\" 202 0"
2024-04-16T05:29:23Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:23.284932801Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:23Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:23.285206461Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:23Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:23.409658503Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-16T05:29:23Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:23.410193213Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.507567007Z" level=warning msg="Failed to delete conntrack state for 10.100.0.3: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.507997327Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:3: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.604447312Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.604776292Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.824109569Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:25Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:25.824704959Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:28Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:28.547173366Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:28Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:28.547508016Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:29Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:29.188092318Z" level=warning msg="Failed to delete conntrack state for 10.100.0.2: invalid argument"
2024-04-16T05:29:29Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:29.188493538Z" level=warning msg="Failed to delete conntrack state for 2001:db8:1::242:a64:2: invalid argument"
2024-04-16T05:29:31Z app[9080e331a9d598] hkg [info]time="2024-04-16T05:29:31.112483733Z" level=warning msg="no trace recorder found, skipping"

I would greatly appreciate your guidance on how to address these warnings and ensure a smooth deployment process. Specifically:

  1. Could you provide more information on the recommended approach for handling sensitive environment variables such as AWS_SECRET_ACCESS_KEY, DB_PASSWORD, and REDIS_PASSWORD? Should I set them as secrets instead of including them in the [env] section of the configuration file?
  2. Regarding the remote builder warnings, I followed the suggestion to check the remote builder logs using flyctl logs -a fly-builder-wispy-voice-7350, but I am unsure how to interpret the logs or resolve the issue. Could you provide any insights or steps I should take to troubleshoot the remote builder not starting in time?

Thank you for your time and assistance. I look forward to your guidance on these matters to ensure a secure and successful deployment of my application on the Fly.io platform.

2

Hi,

Regarding

Firstly, I want to highlight that my application is hosted in the Tokyo region. However, while running the fly deploy command, I noticed that the logs indicate the builder is located in Hong Kong (hkg)

… that is expected. If using a remote builder, that will generally run close to you rather than close to the chosen region being deployed to:

You would normally want that. If it’s an issue you could either:

a) build locally and avoid using it at all (by doing fly deploy --local-only). ~Assuming you have Docker running locally
b) try and move the remote builder somewhere else, by moving its volume (which in turn pins the app). I vaguely recall people mentioning dong that … but I don’t think that is officially supported so I wouldn’t

As for the

Could you provide more information on the recommended approach for handling sensitive environment variables

… yep, any secrets should not be in fly.toml. Things like passwords and API keys should instead be stored encrypted using the command: fly secrets set NAME=VALUE.

For more, see:

That will still make them available to your app as environment variables, but will store them securely and avoid revealing them in plain text.

The logs of the builder would be more useful to Fly’s people for debugging. Like I say, a workaround is to avoid using it at all and build locally, at least for now. Docker is free to download and install for personal/small companies if you don’t already have it.