fly.io is offline - cannot proxy http request

Please show some logs. That error is generic, there are more details in the logs.

@jerome not sure if this helps:

2023-02-21T21:58:16.951 proxy[28b201e0] dfw [error] could not send HTTP request to instance: connection error: timed out
2023-02-21T21:58:17.041 proxy[28b201e0] dfw [error] could not send HTTP request to instance: connection error: timed out
2023-02-21T21:58:17.332 proxy[28b201e0] dfw [error] could not send HTTP request to instance: connection error: timed out
2023-02-21T21:58:17.332 proxy[28b201e0] dfw [error] could not send HTTP request to instance: connection error: timed out

Iā€™m storing my images in a Google Cloud Storage bucket. My rails app was trying to serve those images. Works on my development setup and it was working for at least 1 week :S

Your app has 2 seconds to accept a connection when we try to establish one from our hosts. Most apps accept a connection in ~1ms, so 2s doesnā€™t seem too crazy.

When connection timeouts happen like that, it usually means your appā€™s server is overloaded. Knowing Ruby / Rails, this is likely something blocking your serverā€™s accept loop. I see response times are very long, maybe some operations should happen in background jobs? Everything is, usually, synchronous in Ruby.

Is there any way to validate that the proxy allows, not blocking or has proper timeouts for that requests:

Iā€™m not sure about having something complicated in my app. Since Iā€™m using mostly boilerplate from ActiveStorage to retrieve images.

Failed to open TCP connection to oauth2.googleapis.com:443 (execution expired)

@jerome I just deployed my app to a competitor and works like a charm :S

Is there any way we can check this is not a proxy / routing error? google apis looks to be blocked or something.

Youā€™re using the same exact docker image?

We donā€™t proxy outgoing connections. It is possible that some APIs might rate-limit a whole host (containing multiple VMs) based on IP.

Can you try connecting via IPv6 only? These IPs are per VM and might will avoid rate limiting.

Just to add on, this is also happening to us (specifically our service cannot talk to GCS). We have a health check on our service which causes our service to fail to deploy when it cannot communicate with GCS. Our services which have been running for a while seem to be able to connect just fine to GCS, so this seems to be an issue which mostly manifests when the service is first starting, and resolves after a while.

@jerome I was not able to connect via IPv6. But I changed my IPv4 to a dedicated one. That change did not work and then I when back to a shared IPv4, it rotated and still the problem is visible. I would guess something changed in the network in background since my app was working just fine for at least 5 days and I did not changed anything related to networking or storage.

Regarding your question about the docker image. Yes, it is the same docket image and same exact code.

1 Like

@jerome I did nothing and everything is working again. Is there something I can do in order to avoid this proxy googleapis blocking?

I wonder if this is something I would expect in the future. Right now Iā€™m considering between hosting here or somewhere else.

The issue mightā€™ve been related to IPv6. @steveberryman found a host that was affected.

@jerome @steveberryman Awesome! Iā€™m glad it got fixed. Thanks so much!

2 Likes