Node app stuck with STATUS PENDING

Questions / Help Build debugging
1

I tried deploying a Node app but I receive the following error message:

Deploying delicate-field-6353
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Waiting for remote builder fly-builder-cool-morning-5938... 
WARN Remote builder did not start on time. Check remote builder logs with `flyctl logs -a fly-builder-cool-morning-5938`
Error error connecting to docker: remote builder app unavailable

I’m not sure why the app is stuck or how to get the deploy to go through.

Output of flyctl status:

App
  Name     = delicate-field-6353          
  Owner    = personal                     
  Version  = 0                            
  Status   = pending                      
  Hostname = delicate-field-6353.fly.dev

The logs command hangs because the app hasn’t started, so there isn’t any useful output.

But here is the latest logs output for the builder:

2021-06-01T22:51:38.098591384Z app[7ca81435] dfw [info] time="2021-06-01T22:51:38.090481222Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2021-06-01T22:51:39.098593301Z app[7ca81435] dfw [info] time="2021-06-01T22:51:39.094915897Z" level=debug msg="checking docker activity"
2021-06-01T22:51:39.103549060Z app[7ca81435] dfw [info] time="2021-06-01T22:51:39.096617412Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2021-06-01T22:51:40.104407577Z app[7ca81435] dfw [info] time="2021-06-01T22:51:40.100338383Z" level=debug msg="checking docker activity"
2021-06-01T22:51:40.109747919Z app[7ca81435] dfw [info] time="2021-06-01T22:51:40.100833073Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

Here is the autogenerated fly.toml file if that’s useful:

# fly.toml file generated for delicate-field-6353 on 2021-06-01T16:27:36-06:00

app = "delicate-field-6353"

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  builtin = "node"

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 6
    timeout = "2s"
2

I can’t tell at quick glance why it couldn’t connect. Could you try deploying again to see if it works this time?

3

@michael I tried another deploy and received the same error.

4

What version of flyctl are you using?

5

v0.0.220

Output of flyctl version:

flyctl v0.0.220 linux/amd64 Commit: 2f06aed BuildDate: 2021-05-14T17:07:51Z
6

@tory are you able to resolve fly-builder-cool-morning-5938.internal over wireguard? Do you have a firewall running?

7

@michael I don’t have a firewall running, and I don’t have wireguard set up yet.

I’m evaluating Fly for a project and this is my first deployment. I’m running through the getting started guide for node and wireguard wasn’t mentioned. Do you need to have an active wireguard connection to run deployments?

8

Ooof I’m sorry for a bad first experience. You don’t need wireguard to build and deploy. flyctl opens a tunnel to the builder as needed, but something isn’t working here. I can’t reproduce the issue, but I’m going to remove your builder so you get a new one on the next try. Maybe that’ll get you unblocked.

9

No worries! I appreciate all the help!

10

@michael I tried another deploy with a new builder and got the same error. Are you seeing delicate-field-6353 stuck in PENDING on your side?

11

Yeah, delicate-field-6353 is still pending because it hasn’t been deployed yet. The image that gets deployed is built on a remote docker daemon (unless you have docker running locally) but flyctl can’t connect to it for whatever reason.

You can try using a wireguard tunnel to connect to your org’s private network, then run dig _apps.internal to see if DNS is an issue.

Is there anything out of the ordinary with the network you’re connecting from? Usually when this happens it’s either a local firewall, a corporate network, or another VPN that’s blocking connections.

12

I’m just working from my home network in Idaho. It’s wide open (no firewall, no VPN, etc…) And I have no problem communicating with the builder, but it seems like the builder can’t communicate with the app?

I’ll try wireguard and see if that get’s me unblocked.

13

What do you mean you can communicate with the builder? Is there anything more to the flyctl logs than you shared in your first post?

14

No, nothing more. The logs I shared in the first post were from the builder. So assume that since I can access the logs, I’m able to communicate with the builder. But maybe that assumption is incorrect?

15

Ah okay. The logs are coming through our API rather than directly from the VM. Since flyctl printed WARN Remote builder did not start on time. and the builder’s logs show it’s healthy, I think the issue is your machine can’t ping the builder within 5 minutes, so it’s giving up.

16

Could you try deploying with debug logs, like LOG_LEVEL=debug flyctl deploy and DM the flyctl output to me? I’m hoping there’s some more info being printed there.

17

@tory if you’re up for installing Docker locally, it would also help debug. You can run this to force it to use local docker:

fly deploy --local-only

And then force remote builds:

fly deploy --remote-only

Our bet is local builds will work, and remote builds aren’t due to some issue connecting to the wireguard network over UDP.

18

@tory since you can ping your builder with an active wireguard tunnel, the issue might be with our DNS that we’ll investigate.

We just added a way to unblock you by using the host wireguard tunnel for builds. Upgrade flyctl to 0.0.222 and run like this:

FLY_REMOTE_BUILDER_HOST_WG=1 flyctl deploy

Let me know how that goes.

19

@michael I’m sorry for all the trouble, and appreciate all the help :pray:

I upgraded and ran that command, but unfortunately it still isn’t working:

➜  torybriggsdev-remix-prisma-fly flyctl version
flyctl v0.0.222 linux/amd64 Commit: bc585e6 BuildDate: 2021-06-03T17:25:08Z
➜  torybriggsdev-remix-prisma-fly FLY_REMOTE_BUILDER_HOST_WG=1 flyctl deploy --build-arg REMIX_TOKEN=${REMIX_TOKEN} 
Deploying delicate-field-6353
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080

WARN Remote builder did not start on time. Check remote builder logs with `flyctl logs -a fly-builder-restless-thunder-2331`
Error error connecting to docker: remote builder app unavailable

I may explore the local build option with Docker. Docker just wasn’t something we wanted to use as part of the build for this app.

20

It’s working!

Turns out WSL 1 for Windows has firewall restrictions. I’ve never run into an issue like this, so I had no idea. Upgrading to WSL 2 solved the issue.

Sorry for taking up so much time for this, but hopefully it can help anyone else who may be running WSL 1.

1 Like