Is Fly having deployment issues?

1

https://status.flyio.net/ is all green but I’m seeing:

WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GM8HNNMKZK8QA2XPR8Z4FZ7S-iad
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GM8HPBNQQBNTSXQDMQ065DMM-iad

Searching for this type of error it seems to usually be deployment issues with the infrastructure.

Noticed there were issues yesterday, are there some new ones today?

2

We have a brief hiccup in IAD which should be resolved now. Can you try again?

3

I’ve been unable to deploy anything for going on 3 days:

Another user encountering what appears to be the same issue:

This has been before, during, and after the issues reported on the Status page.
I confirmed just now, I still can’t deploy.

1 Like
4

No change here.

5

@matthewp it looks like you deleted your builder a few days ago, and the new one came up in a weird state. We’re looking at that.

1 Like
6

It appears that things are happening now.

7

For me at least things are working again. Thanks!

8

We’re seeing deployment issues across multiple apps again. They’re getting stuck on “Starting instance” in the logs, with the instance’s status stuck on “pending”. Redeploying and restarting doesn’t help, and there’s no log output coming from the applications. (This is in ORD now, not IAD.)

1 Like
9

I’ve been unable to deploy anything for going on 2 days:
Stuck at [internal] load remote build context.

1 Like
10

What region are you trying to deploy to? I wonder if this is location-specific. I am stuck at the same step, in ORD. @abuznego reports the same issue, though I’m not sure what region.

11

I’m having a bunch of problems deploying (to an app in ewr) now as well. I’ve tried remote but am seeing this:

WARN Failed to start remote builder heartbeat: Couldn’t allocate volume, no disks available (region: atl, size_gb: 50)
Error failed to fetch an image or build from source: Couldn’t allocate volume, no disks available (region: atl, size_gb: 50)

With --local-only I get this:

Saving registry.fly.io/fafo-stripe-server:cache…
*** Images (e3ae97b0f801):
registry.fly.io/fafo-stripe-server:cache
registry.fly.io/fafo-stripe-server:deployment-01GMBYYRJXTAARXBVS8G4ZXNX6
ERROR: failed to write export report: write /layers/report.toml: no space left on device
Error failed to fetch an image or build from source: executing lifecycle: failed with status code: 62

12

That’s weird, it should not be trying to create a volume in atlanta for you. We’ll have a look.

The second error looks like an issue with local docker. If you’re on an m1 mac it might be QEMU related.

13

Looks like things are working again!

Thanks for all your help and whatever magic you all did!

Adam

14

@discoursebum yes all fixed! We brought a new Atlanta region online today, but it’s not fully enabled. We had an Atlanta region a few years ago and killed it off, but there were some DB records that suggested a person connection through Atlanta could get disks for a builder there. The DB was lying, so I deleted it.

1 Like
15

ya’ll are the best.

16

I am still experiencing deployment issues :frowning:

1 Like
17

I am still experiencing deployment issues.

18

I to am having deployment issues.

Error failed to fetch an image or build from source: error building: error during connect: Post "http://[fdaa:1:651a:a7b:c207:2f74:89a:2]:2375/v1.41/build?buildargs=null&buildid=upload-request%3A5ac16aa6c7e6bc4ca96191e48cda2dc6c3006289f1ffd10f9bf0bb422d2783ef&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&target=&ulimits=null&version=2": EOF