Fly API error....

Hi,

Not sure why that would be, but maybe there is a random fault with that particular builder?

You could try deleting it. Run fly destroy fly-builder-dark-silence-168.

Then when you next run fly deploy, Fly will see you have no remote builder. And make you another one. Which may solve the issue. Of course if that new remote builder also fails, well there is something else going on! But that’s worth a try first.

I’m having the same issue.

I have destroy and recreate builder app but no luck, the builder stuck at pulling the image:

2023-01-28T20:31:28.924 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:30.079 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:32.641 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:34.907 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:36.911 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:37.927 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:39.407 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:41.482 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:43.979 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:45.980 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:47.702 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:50.028 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:53.005 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:55.094 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:56.634 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:31:59.043 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:00.830 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:02.849 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:05.393 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:06.551 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:09.393 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:11.807 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:13.421 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:14.544 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:16.193 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:17.224 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:20.168 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:21.384 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:23.178 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:25.922 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:28.650 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:30.309 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:32.829 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:35.587 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:36.969 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:39.433 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:41.259 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:44.016 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:46.853 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:48.316 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:50.172 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:53.034 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:54.284 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:55.758 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:56.986 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:32:59.049 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:33:01.814 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:33:03.099 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:33:05.074 runner[1781121f546789] sin [info] Pulling container image

2023-01-28T20:33:06.852 runner[1781121f546789] sin [info] Pulling container image

Hi Greg,
I have tried your suggestions but I find myself with the same errors…

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GQXN1GHMD43W08QQAJFR847P-maa
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-summer-violet-1286`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable


Also I’m having my builder logs which indicate its stuck on pulling the container image.same as @pyk

My app used to work fine during launch/deployment in the past week,but these started showing up since yesterday…

Er … :thinking:

I’m afraid I don’t know what to suggest. My hack/fix is to delete the remote builder and hope its replacement works … but you’ve tried that and it hasn’t helped here. Hmm. I don’t know Fly’s innards. If the issue only started yesterday I wonder if there is some temporary issue with Fly in that region? Hopefully someone will be able to look up that API request ID to debug.

I have successfully deploy my app using fly deploy --local-only, this will use local docker to build the image instead of remote builder. If you want to try it, make sure you have docker running in your machine.

Ah, that’s a good idea to bypass the remote builder step entirely @pyk .

Perhaps there is an issue with sin. Only other people have mentioned issues with that region too e.g

Hey thanks for the suggestion!
But the issue has been resolved and I’m able to deploy using the fly CLI…
Will surely try your solution next time if issues with remote-builder persist!

Hello again,it seems sin region is down again!
This saved me today!
(Used the --local-only flag)

Thanks @greg. I had the same issue in AMS today. I destroyed the builder with fly destroy BUILDER_APP_NAME and ran fly deploy --remote-only and that worked for me :+1:

1 Like

Having issues with fly machines… I’ve tried @PeterUllrich’s solution. Doesn’t seem to work for machines…

  "errors": [
    {
      "message": "You hit a Fly API error with request ID: 01GRDXG0NEXE08M8XC0G3ZCZV7-ams",
      "extensions": {
        "code": "SERVER_ERROR",
        "fly_request_id": "01GRDXG0NEXE08M8XC0G3ZCZV7-ams"
      }
    }
  ],
  "data": {}
}
DEBUG result image:<nil> error:error connecting to docker: You hit a Fly API error with request ID: 01GRDXG0NEXE08M8XC0G3ZCZV7-ams
DEBUG --> POST https://api.fly.io/graphql```

```Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for <app>:
Checking that app has ip addresses allocated... PASSED
Checking A record for <app>... PASSED
Oops, something went wrong! Could you try that again?```

Deleting the builder machine does not help:

$ /root/.fly/bin/flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GRDYGMW93AXWZ81HDNBDA0B2-iad
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GRDYGQSSXJS0772T6CCDZR4P-iad

To be honest this is unacceptable. Fly is doing around for a few years already, but stability issues like this still popping up out of nowhere. I’ve used to recommend fly in my community but I can not anymore. While I’m capable of figuring out and troubleshooting issues like this, a lot of folks isn’t.

I know that in a large scale those errors are rare, but I’ve never had issues with Heroku, for example. Issues like this make experience very bad and made me think to move out.

Random builder failures, capacity issues in regions, random DNS failures, things like these… I know you guys doing a very good job on developing your own software to orchestrate all the things, and I really appreciate your work, but when I need to deploy my stuff and something is not working makes me super disappointed.

1 Like

Could you try @pyk 's solution?(Docker must be installed & configured for this to work)

pyk

I have successfully deploy my app using fly deploy --local-only, this will use local docker to build the image instead of remote builder. If you want to try it, make sure you have docker running in your machine.

1 Like

so after destroying the builder 100 times, now am able to update my machines…

i tried pushing manually to fly registry and update the image through cli, but it’s a pain in the ass since am on osx and fly won’t accept arm64 builds. had to set up docker buildx.

anyway, the whole thing was a waste of time. this is just disappointing even though it worked after a few hours.

Could you try @pyk 's solution?(Docker must be installed & configured for this to work)

Yeah, it didn’t worked at the time, but now it is; thank you. But remote build & deploy still didn’t, and frankly, it’s unacceptable.

I have the same error in a GitHub Actions deploy. You can see the full run here:

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GREZ8KPGWD4KYFE9DBB5KJA3-iad

Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GREZ8MY6R7Y3QME9KYAPT6NH-iad

Looks like sin region is down today too!

have same issue.

–local work

remote build not work.

so sad.

Hey, can you paste that with request ID: ..... here. Text is generally more accessible and easier to search.

==> Verifying app config
Validating /Volumes/workspace/SWorkspace/onewheel-blog/fly.toml
Platform: machines
✓ Configuration is valid
→ Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01H47VMMPPTDA1MYNNCSDSHASC-nrt
Error: failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01H47VMNR6WEZ1QQCQG4C26ATX-nrt

I still have this issue. I just want to subscript the pro plan. But look like I will change my mind.