Machines won't start!

We are seeing increased latency on Amsterdam :thinking:
What region are you connecting from? you can find out at https://debug.fly.dev/

ams indeed.

=== Headers ===
Host: debug.fly.dev
X-Forwarded-Port: 443
Fly-Client-Ip: 2a02:a454:3d66:1:51a3:8b7e:9281:e4fa
Sec-Fetch-Dest: document
Fly-Region: ams
Fly-Tracestate:
Sec-Ch-Ua-Platform: “macOS”
X-Forwarded-For: 2a02:a454:3d66:1:51a3:8b7e:9281:e4fa, 2a09:8280:1:763f:8bdd:34d1:c624:78cd
X-Forwarded-Ssl: on
Fly-Forwarded-Port: 443
Sec-Fetch-Mode: navigate
Fly-Traceparent: 00-e384d05a5de7d5c3f5d1afa8a6b1eb79-0440df5344f5b7a3-01
Sec-Fetch-User: ?1
Sec-Ch-Ua: “Chromium”;v=“124”, “Google Chrome”;v=“124”, “Not-A.Brand”;v=“99”
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7
Referer: https://community.fly.io/
Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6
Fly-Forwarded-Proto: https
X-Request-Start: t=1715184813902391
Priority: u=0, i
Sec-Ch-Ua-Mobile: ?0
Accept-Encoding: gzip, deflate, br, zstd
X-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Request-Id: 01HXCFPHTE62YMRNZ666D1ZHBW-ams
Via: 2 fly.io
Sec-Fetch-Site: cross-site

2024-05-08 16:13:33.910237209 +0000 UTC m=+3600405.097979984

1 Like

I think you might be referencing a different test app, but I’ll dm you with those details @dangra.

I’m getting a similar problem to others in this thread on new deployments.

ams too

1 Like

I don’t think I can dm on this platform @dangra, so I have emailed the details to the email on your github account.

1 Like

@Yaeger @dominiks AMS latency is back to normal levels. Could I ask for a re-deploy? I can’t promise it will work but I will be looking on my side.

1 Like

Still the same for me, it stucks on different level every time, but always approximately 15MB

Same here, not looking so great yet. Kinda stuck on Preparing/waiting

This last execution was ran through GHA, I doubt that one runs through AMS :thinking:

Any update on this? @dangra I still can’t deploy anything. And there’s an app down which I can’t fix because I can’t deploy :cry:

Just tried again:

d1ecaaa6617e: Pushed
a073c805237b: Pushed
73a7d2f5f7f8: Pushed
925e1a7bb219: Layer already exists
8adf0e825511: Pushing [==================================================>]  41.33MB
a99dbc548466: Pushing [=============================>                     ]  51.32MB/85.53MB
dd7040dde505: Pushing [==================================================>]  49.98MB
44e24573f7cb: Layer already exists
5f70bf18a086: Layer already exists
734ee9eb249d: Layer already exists
50f407b42369: Layer already exists
a12586ed027f: Layer already exists

Stuck on this :person_shrugging:

2 Likes

By you have tried both: registry.fly.io and registry-iad.fly.io, right?

Yes

Gave that IAD registry another shot and this is how it looks. It progresses nicely and then boom, it’s stuck

I am encountering the same issue.

Exactly same situation

also from ams region?

I am encountering it from WAW region.

Same here with registry.fly.io on ams

An incident has been reported now

Is there a hotfix for this? I see according to the status page it’s only ams. Can we somehow use other regions? Is it about the regions of the machines? Or about where our API requests come in through?