flyctl deploy: You hit a Fly API error with request ID...

savikko · October 28, 2022, 9:51am

I think this is now related to this “bigger” issue right now. For me it seems that almost every instance is down.

jaeyson · October 28, 2022, 10:33am

that seems the case, my dev instance is down and im still hitting fly api error. Tried all options mentioned above and still persists.

Omayma · October 28, 2022, 10:44am

We are running to the same problem, tried everything above but the issue persists.

jaeyson · October 28, 2022, 10:46am

maybe I should wait till it’s up? currently looking at status.flyio.net

Processing: Screen Shot 2022-10-28 at 18.44.16.png…

devjoda · October 28, 2022, 10:57am

Can’t launch no matter what region I try:

Error You hit a Fly API error with request ID: 01GGF3QM4EH3073GRDVPGKKQJQ-ams

scatterinfo · October 28, 2022, 11:09am

I’ve been getting the same error for the last three hours:

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GGF4E1NRZ4D2SQ81YJKQ7BYM-maa
Error failed to fetch an image or build from source: You hit a Fly API error with request ID: 01GGF4EFJSJEVSZYX6NM73EE8R-maa

fuma · October 28, 2022, 11:14am

Dev2 · October 28, 2022, 11:22am

Yesterday i was waiting for my deploy finished but i saw here in community that was some issues about deployment process. Then, now my application is PENDING status and my users cannot access my services in PRODUCTION.

Please, there’s any solution about this case? This status “PENDING” not changing by hours.

Thanks in advance

rugwiro · October 28, 2022, 11:23am

Yeah unfortunately there is an ongoing incident, but we’re deep into the weeds with it. Keep an eye on our status page for updates.

Dev2 · October 28, 2022, 11:24am

This is status of my current vm that status is PENDING

Dev2 · October 28, 2022, 1:57pm

There’s any way to redirect all traffic that hit my fly.dev application to another url ? My application remains offline. I need help!

kiru · October 28, 2022, 2:02pm

I am having the same issue. It’s frustrating not to be able to deploy hotfix. >8h downtime is insanely long.

Dev2 · October 28, 2022, 2:10pm

It’s alive!

scatterinfo · October 28, 2022, 6:59pm

I already switched back to Heroku. This kind of downtime is extremely damaging to my business.

quan · December 16, 2022, 5:36am

hi @eli ,

I got the error Error You hit a Fly API error with request ID: 01GMCPS24Y8RQDBR50E5BBZ9YW-sin when allocating new ipv4 flyctl ips allocate-v4 -r yul -a spread-s1
thanks

lc9900 · February 4, 2023, 5:45pm

Having the same issue right now. I’ve destroy/recreate app and still getting the same thing.

Corey_Alexander · February 4, 2023, 5:56pm

Also just hit this error both in CI and locally

Here is my local request ID in case its helpful to anyone:

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GRERYN8HZCFVS5RM6JNFKK6E-lga
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GRERYNVW86MJAC4N785NYVX4-lga

Sorry if this is causing weekend alarms :hugops:

kurt · March 19, 2023, 1:02pm

5 posts were merged into an existing topic: Fly API error allocating IPv6 address

dilraj · July 11, 2023, 2:11pm

I have been getting this issue since the last time I deployed.

I was having wierd issues, where my API would sometimes have the “OLD” code, instead of the new. This is unacceptable.

When I tried to redeploy, I see:

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: ABCDEFGHIJKLMNOPQRSTUVWXYZ-yyz

Error: failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: ABCDEFGHIJKLMNOPQRSTUVWXYZ-yyz

I then tried to destroy the machine and do another fly launch, but it gives back the same error! Now… MY APP IS COMPLETELY USELESS.

What should I do?!

Here is my fly.toml and my Dockerfile

dilraj · July 11, 2023, 3:56pm

Just a follow-up! I began look into Docker as a possible root-cause. When I was running:

docker build -t spg2 .

I started seeing an error:

Step 5/12 : RUN gem install bundler &&     bundle config set path 'vendor/bundle' &&     bundle config set deployment 'true' &&     bundle install --jobs=4 --without development test
 ---> Running in 110e7fa1fca8
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/local/bundle directory.
	/usr/local/lib/ruby/3.2.0/rubygems/installer.rb:712:in `verify_gem_home'
	/usr/local/lib/ruby/3.2.0/rubygems/installer.rb:902:in `pre_install_checks'
	/usr/local/lib/ruby/3.2.0/rubygems/installer.rb:302:in `install'
	/usr/local/lib/ruby/3.2.0/rubygems/resolver/specification.rb:104:in `install'
	/usr/local/lib/ruby/3.2.0/rubygems/request_set.rb:194:in `block in install'
	/usr/local/lib/ruby/3.2.0/rubygems/request_set.rb:182:in `each'
	/usr/local/lib/ruby/3.2.0/rubygems/request_set.rb:182:in `install'
	/usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:214:in `install_gem'
	/usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:230:in `block in install_gems'
	/usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:223:in `each'
	/usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:223:in `install_gems'
	/usr/local/lib/ruby/3.2.0/rubygems/commands/install_command.rb:169:in `execute'
	/usr/local/lib/ruby/3.2.0/rubygems/command.rb:323:in `invoke_with_build_args'
	/usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:251:in `invoke_command'
	/usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:191:in `process_args'
	/usr/local/lib/ruby/3.2.0/rubygems/command_manager.rb:149:in `run'
	/usr/local/lib/ruby/3.2.0/rubygems/gem_runner.rb:51:in `run'
	/usr/local/bin/gem:10:in `<main>'
The command '/bin/sh -c gem install bundler &&     bundle config set path 'vendor/bundle' &&     bundle config set deployment 'true' &&     bundle install --jobs=4 --without development test' returned a non-zero code: 1

I never seen this before; afterall, I’ve deployed and built the Docker container before.

Updating Docker

After debugging for hours, I tried ugprading Docker as suggested on Stackoverflow here.

For whatever reason, you can’t go to Docker Desktop > Check For Updates. That does NOT work.

Instead, I had to manually reinstall it from the Docker website. Before, I was on 19.x, now I am on Docker version 24.0.2, build cb74dfc

Running Docker Build

Now, I can run: docker build -t spg2 . and do docker run -p 3000:80 -e RAILS_ENV=development -e PORT=3000 spg2

I thought this would fix the:

Error: failed to fetch an image or build from source: error connecting to docker:
You hit a Fly API error with request ID: XYZ

But it DID not fix anything.

`fly doctor`

Everything checks out and looks good here:

fly doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for spg:
Checking that app has ip addresses allocated... Nope
	No ip addresses assigned to this app. If the app is not intended to receive traffic, this is fine.
	Otherwise, it likely means that the services configuration is not correctly setup to receive http, tls, tcp, or udp traffic.
	https://fly.io/docs/reference/configuration/#the-services-sections
No ipv4 or ipv6 ip addresses allocated to app spg
Build checks for spg:
Checking docker context size (this may take little bit)... PASSED (76 MB)
Checking for .dockerignore... PASSED

Final Thoughts

Nothing worked.

I am pretty sure it is something on Fly’s side from what I can see from other conversations:

Going to email billing@fly.io or get help on Twitter.