Machines won't start!

21

We are seeing increased latency on Amsterdam :thinking:
What region are you connecting from? you can find out at https://debug.fly.dev/

22

ams indeed.

=== Headers ===
Host: debug.fly.dev
X-Forwarded-Port: 443
Fly-Client-Ip: 2a02:a454:3d66:1:51a3:8b7e:9281:e4fa
Sec-Fetch-Dest: document
Fly-Region: ams
Fly-Tracestate:
Sec-Ch-Ua-Platform: “macOS”
X-Forwarded-For: 2a02:a454:3d66:1:51a3:8b7e:9281:e4fa, 2a09:8280:1:763f:8bdd:34d1:c624:78cd
X-Forwarded-Ssl: on
Fly-Forwarded-Port: 443
Sec-Fetch-Mode: navigate
Fly-Traceparent: 00-e384d05a5de7d5c3f5d1afa8a6b1eb79-0440df5344f5b7a3-01
Sec-Fetch-User: ?1
Sec-Ch-Ua: “Chromium”;v=“124”, “Google Chrome”;v=“124”, “Not-A.Brand”;v=“99”
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7
Referer: https://community.fly.io/
Accept-Language: nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7,de;q=0.6
Fly-Forwarded-Proto: https
X-Request-Start: t=1715184813902391
Priority: u=0, i
Sec-Ch-Ua-Mobile: ?0
Accept-Encoding: gzip, deflate, br, zstd
X-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Request-Id: 01HXCFPHTE62YMRNZ666D1ZHBW-ams
Via: 2 fly.io
Sec-Fetch-Site: cross-site

2024-05-08 16:13:33.910237209 +0000 UTC m=+3600405.097979984

1 Like
23

I think you might be referencing a different test app, but I’ll dm you with those details @dangra.

I’m getting a similar problem to others in this thread on new deployments.

24

ams too

1 Like
25

I don’t think I can dm on this platform @dangra, so I have emailed the details to the email on your github account.

1 Like
26

@Yaeger @dominiks AMS latency is back to normal levels. Could I ask for a re-deploy? I can’t promise it will work but I will be looking on my side.

1 Like
27

Still the same for me, it stucks on different level every time, but always approximately 15MB

image
image795×48 2.4 KB

28

image
image1546×1004 154 KB

Same here, not looking so great yet. Kinda stuck on Preparing/waiting

29

This last execution was ran through GHA, I doubt that one runs through AMS :thinking:

30

Any update on this? @dangra I still can’t deploy anything. And there’s an app down which I can’t fix because I can’t deploy :cry:

Just tried again:

d1ecaaa6617e: Pushed
a073c805237b: Pushed
73a7d2f5f7f8: Pushed
925e1a7bb219: Layer already exists
8adf0e825511: Pushing [==================================================>]  41.33MB
a99dbc548466: Pushing [=============================>                     ]  51.32MB/85.53MB
dd7040dde505: Pushing [==================================================>]  49.98MB
44e24573f7cb: Layer already exists
5f70bf18a086: Layer already exists
734ee9eb249d: Layer already exists
50f407b42369: Layer already exists
a12586ed027f: Layer already exists

Stuck on this :person_shrugging:

2 Likes
31

By you have tried both: registry.fly.io and registry-iad.fly.io, right?

32

Yes

33

image
image1566×500 88.3 KB

Gave that IAD registry another shot and this is how it looks. It progresses nicely and then boom, it’s stuck

34

I am encountering the same issue.

35

Exactly same situation

36

also from ams region?

37

I am encountering it from WAW region.

38

Same here with registry.fly.io on ams

39

An incident has been reported now

40

Is there a hotfix for this? I see according to the status page it’s only ams. Can we somehow use other regions? Is it about the regions of the machines? Or about where our API requests come in through?