Can't complete flyctl deploy process - context deadline exceeded

Questions / Help
1

I seem to have broken my ability to deploy a new app. Whenever I tried, it goes from
Waiting for remote builder fly-builder-summer-dawn-8254… connecting ⣟
to
Error error connecting to docker: unable to connect WireGuard tunnel: context deadline exceeded

When I check the status it says pending for the app and when I list apps, pending for the fly-builder as well. I destroyed my existing fly-builder and tried again, but it hasn’t made a difference. I thought something in my environment was causing a timeout, but I just tried to deploy the hellostatic-builtin example and that failed in the same way.

I was able to deploy an app last week! Please let me know what I need to tear down in order to get things moving and shaking again.

Thank you,
-James

2

What version of flyctl are you running?

3

flyctl v0.0.240 linux/amd64 Commit: 0ed4d26 BuildDate: 2021-09-15T16:52:14Z

It prompted me to upgrade this morning.

4

I wonder if your WireGuard peer got into a bad state. Let’s try removing them and see if it helps.

  1. run flyctl wg list <org> to see a list of your peers
  2. run flyctl wg remove <org> <name> on the ones with a prefix of “interactive-”.

Once that’s done, try deploying again.

5

Removed the WireGuard peer, tried to deploy again, still getting the context deadline exceeded.

I destroyed the fly-builder instance, just in case. The new one also seems to stay in Pending and I’m not able to get any logs from it, which is the behavior I was seeing from the old one.

6

@CallJCW can you post or DM me your builder app name?

7

Updated to 0.0.241, still getting this problem. I tried flyctl 0.0.238, which is the last time it worked, but the waiting for remote builder still timed out. I’m trying to nail down if it is a problem in my environment and if so, where.

I’m going to try another machine from within my network and see if that makes a difference.

Thank you,
-James

8

I’m currently seeing this error from CircleCI on about 30% of my deploys, but it’s inconsistent. If I retry the build, it works but then another deploy will fail a few minutes later.

I’m running the latest version of flyctl:

flyctl v0.0.256 linux/amd64 Commit: f39ea62 BuildDate: 2021-11-21T19:59:38Z

The failures look like this:

#!/bin/bash -eo pipefail
"${HOME}/.fly/bin/flyctl" deploy --access-token "${FLY_ACCESS_TOKEN}"

WARN no config file found at /home/circleci/.fly/config.yml
Deploying whatgotdone
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Waiting for remote builder fly-builder-spring-voice-5703... connecting ⣻ Error error connecting to docker: unable to connect WireGuard tunnel: context deadline exceeded



Exited with code exit status 126

My fly.toml is here: whatgotdone/fly.toml at 73c148aab96f65f29462473253b23fd156fd5c5a · mtlynch/whatgotdone · GitHub

1 Like
9

I just hit this again last night. It’s no longer blocking 30% of my deploys, but it’s probably still affecting 10% of them, which is fairly high.

Is the workaround to just switch to local builds?

10

Can you open ~/.fly/config.yml and remove everything under wire_guard_state: and try again? I think your wg peer is in a bad state and this will wipe it and get you a new one on the next attempt.

11

It shouldn’t be anything with persistent state because I’m deploying from a fresh Docker container in CI every time:

12

I can confirm the same happening on my Github Actions. It’s not a blocker, but makes me want to double-check all deploys.

Here is my action.yaml

name: Deploy master branch to indiepaper on fly
on:
  push:
    branches:
      - master
env:
  FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
jobs:
  deploy:
    name: Deploy Production App
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: superfly/flyctl-actions@1.1
        with:
          args: "deploy -a indiepaper-production --remote-only"
13

If you’re using Github Actions (or most CI, really), you’re much better off building with local Docker. CI doesn’t seem to love our wireguard setups and we haven’t figure out why yet.

You can see a great example of how to build with Docker in Github Actions, courtesy of @kentcdodds: kentcdodds.com/deployment.yml at main · kentcdodds/kentcdodds.com · GitHub

This is nice because it uses Github’s caching, too, it should perform much better than remote only for most apps.

2 Likes
14

I’m having this problem as well atm:

❯ fly version
fly v0.0.276 darwin/amd64 Commit: 0e35938 BuildDate: 2022-01-01T17:54:59Z

I have no CI or WireGuard setup

15

Same for me. Deleted everything and run the process from scratch.

Once I got

--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  108.9kB
[+] Building 597.1s (0/1)                                                                                                                                      
 => [internal] load remote build context                                                                                                                597.1s
Error error building: unexpected EOF

and the other time

--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  108.9kB
[+] Building 598.1s (0/1)                                                                                                                                      
 => [internal] load remote build context                                                                                                                598.1s
Error error building: error during connect: Post "http://[***]:2375/v1.41/build?buildargs=null&buildid=***&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&target=&ulimits=null&version=2": EOF
16

FYI, this is happening for me also as of today. To confirm, I am deploying using a Github action. However I’ve been doing this with all my private and commercial apps, and it has been working fine. I’ll take a look at building the container on GH’s side, but I’m wondering whether there’s a way to increase the timeout?

17

@OldhamMade If by this you are only getting a timeout (rather than e.g “error during connect” or “EOF” or one of the other issues/errors mention in this thread) the good news is thatone may be fixed as soon as tomorrow, based on this post. It’s because of the time taken for the peer.

Until then the solution I found was to persist the config.yml file between deploys (you may see the fly deploy command complain about it missing if not). As with CI, generally that file is not persisted, resulting in a big delay. Not sure how you do that with Github Actions specifically though.

18

I’m running into the same issue as well. Using the latest flytctl, I can’t run deploy --remote-only.

it tries to connect to the builder, but fails. I’m on a non-VPNed MacOS Silicon, using a FIOS wifi access point.

I tried running the same command on a brand new linux VM inside MacOS, and it also did not work.

I tried both of the above switching to a T-Mobile hotspot, and still no luck.

I tried running the same command on a GCP instance, and it worked just fine.

Happy to try all sorts of commands to debug this, but it’s really frustrating. It just gets stuck on:

Waiting for remote builder fly-builder-proud-sound-5393… connecting ⢿

Then eventually times out.

19

Note: I can run flyctl --app fly-builder-proud-sound-5393 logs just fine, but I imagine that doesn’t invoke wireguard.

20

Looks like the timeout issues have resolved themselves today. :+1:t2: