I cannot deploy

Hi, we’ve been unable to deploy for a few days. Generally we build via our CI but deployments kept failing with:

Error failed to fetch an image or build from source: error rendering push status stream: name unknown: app repository not found

I tried again by running fly deploy manually and got up to…

 => [build  6/17] RUN mix deps.get --only prod &&   mix deps.compile     519.3s
 => => # remote: Total 249 (delta 55), reused 66 (delta 26), pack-reused 124   
 => => # * Getting phoenix_live_view (https://github.com/phoenixframework/phoen
 => => # ix_live_view.git)                                                     
 => => # remote: Enumerating objects: 17332, done.                             
 => => # remote: Counting objects:  61% (2475/4056)      

…but it got stuck here for about 10 minutes before I killed it.

Next I tried destroying our builder and ran manually again and this happened:

[...]
==> Verifying app config
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-withered-fog-8661... connecting ⣷ Error failed to fetch an image or build from source: error connecting to docker: unable to connect WireGuard tunnel: context deadline exceeded

Any idea what’s happening? We haven’t changed anything on our end. Our last successful deployment was last Friday (Jan. 14).

Not sure about the first half of your post, that error, but regarding the second issue, I’ve diagnosed with @kurt : it seems that time-out is caused by a lack of a config.yml file.

If you run a deploy locally, that is present from any prior successful builds. You should see it in your home folder, in a .fly folder. Its got a load of wireguard stuff in.

But if you run a deploy from CI, that config.yml does not persist from prior builds and so it is very slow and can time out before all the details are gathered (as i understand it).

See:

(so don’t destroy the builder. I was doing that and found it helped but I now think that was just a coincidence that it appeared to be helping, and actually was unconnected to the underlying config yml issue)

Thanks, unfortunately after adding --local-only to the fly deploy command in my CI config I’m still getting the first error…

==> Pushing image to fly
The push refers to repository [registry.fly.io/production]
c15ee4d375c1: Preparing
8dd379b875dc: Preparing
bb84c9222e03: Preparing
9a18678f8624: Preparing
d761cd9de8e5: Preparing
72e830a4dff5: Preparing
72e830a4dff5: Waiting
Error failed to fetch an image or build from source: error rendering push status stream: name unknown: app repository not found

I have tried your suggestion and still can’t deploy.

Error

Waiting for remote builder fly-builder-morning-forest-110
1... connecting ⣾ Error error connecting to docker: 
unable to connect WireGuard tunnel: context deadline exceeded

Is deployment working ?

Deployments are working for me.

I’m not sure why @enaia is getting that other error as I’ve not come across that issue, with repository not found. Think that is one for Fly.

But as regards the:

... unable to connect WireGuard tunnel: context deadline exceeded

error, which I used to get a lot and tried e.g deleting the builder and other random stuff, I’ve solved it by making sure there is a config.yml file present. If there isn’t, I get that error. As it waits several minutes and gives up. If there is, it proceeds immediately to deploy.

So look in the output of the deploy command, a bit further up. Do you see a line about a missing config.yml? It should show where it is expecting it to be found.

It seems like that config.yml is made on a successful deploy, so if you don’t have one, it may be you need to deploy until you get one, and then keep it. That is made much harder with CI builds as generally the server is made fresh each time, losing prior files, and hence this happens way more times. I persisted the config.yml to solve that:

… the other option is not using a remote builder at all, bypassing the issue. That requires you to have docker installed. If you do, great, change --remote-only to --local-only, and that will use your local docker, and so again, avoid the connection to the wireguard tunnel etc.

It seems like the networking for the remote builder isn’t working all that well for you. You will have better success if you install and run Docker locally, then deploy with fly deploy --remote-only.

I don’t know where you’re connecting from, but remote builders work best with reliable and fast internet. They’re kind of fragile when internet is unstable. So if you’re a long distance away from one of our gateways, you’re going to end up with a worse experience.

@enaia this error is probably caused by an auth error.

Try running fly agent stop, and then fly auth login. Then see if that helps!

2 Likes

Hmm this is a deploy using your GitHub action and an API token.

It looks like you’re trying to push to the wrong app repository: registry.fly.io/production

Are you running flyctl deploy or building+pushing separately?

Hmm odd - we haven’t changed our CI config at all, and this worked until last Friday. This is what we’re doing. We added the --local-only after the deploy started failing, based on an earlier suggestion in this thread.

  deploy:
    runs-on: ubuntu-latest
    needs: [int_test, build]
    if: github.ref == 'refs/heads/main'
    env:
      FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          ref: main # Check out main instead of the latest commit
          fetch-depth: 0 # Checkout the whole branch

      - uses: superfly/flyctl-actions@1.1
        with:
          args: "deploy --local-only -c fly.production.toml"

I just realized my mistake, I found your app so that isn’t the problem. We’ll keep looking and see what might be happening.

Have you revoked a personal access token recently? Can you try creating a new one and updating the secret in GitHub?

OK just verifying that you were referring to the FLY_API_TOKEN, not a github personal access token? Anyway, I hadn’t revoked either but I just generated a new fly token via flyctl auth token and updated the FLY_API_TOKEN secret, and deploys are working again.

I also have an app that is recently no longer able to deploy, and failing when building image with buildpacks:

Error failed to fetch an image or build from source: failed to fetch builder image 'index.docker.io/heroku/buildpacks:20': Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp 34.200.175.181:443: i/o timeout

I saw someone mention using remote builders requires a good connection, and I’m on a wired connection in the Netherlands.

This particular issue appears to be a transient network problem. There was a few issues this morning (eastern time).

Can you try again?

This would cause a different kind of issue.

I’ve tried again, still doesn’t work.

Error failed to fetch an image or build from source: failed to fetch builder image 'index.docker.io/heroku/buildpacks:20': Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Maybe it’s less transient. That error is saying our remote builder can’t pull from Docker Hub.

You can try removing the fly-builder-* app from the logs. That’ll let it create a new one on a new IP address, which might help.

I wish I had a better answer for this, but until we figure out why Docker Hub is failing from some VMs and regions, we can’t fix it quick enough for you.

You can also workaround this problem by building the docker image locally. fly deploy --local-only is worth a try.

Thanks, that did it. There have been a couple times I’ve had issues where the solution has been to delete the builder, and it’s not always clear that that’s what’s needed.

It has deployed after trying this one