Hi folks,
I have no clue whats going on, but just got alerted that all our machines our down/not reachable. machines seem to be up, several (1 Rails, 2 elixir apps) are not responding. Tried to restart, to no avail.
Deployment is also not working. According to the stats on the dashboard the app is working/running. But I am not getting any response, and deployment is stuck at
Running release task (pending)
Yeah my prod server is down right now getting these messages
2023-02-03T18:22:58.263 proxy[3889a33f] iad [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 40)
2023-02-03T18:23:09.035 proxy[3889a33f] iad [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 50)
2023-02-03T18:23:19.842 proxy[3889a33f] iad [warn] Could not proxy HTTP request. Retrying in 908 ms (attempt 60)
Started about 15 minutes ago for us across all apps, same errors:
2023-02-03T18:28:16.477 proxy[60ecf046] iad [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 30)
Also having a problem, starting at 1:11p Eastern time. Monitoring shows:
Could not proxy HTTP request. Retrying in 1000 ms (attempt 90)
could not find an instance to route to
Could not proxy HTTP request. Retrying in 20 ms (attempt 1)
Could not proxy HTTP request. Retrying in 1000 ms (attempt 10)
Could not proxy HTTP request. Retrying in 908 ms (attempt 20)
Could not proxy HTTP request. Retrying in 1000 ms (attempt 30)
Could not proxy HTTP request. Retrying in 1000 ms (attempt 40)
Re-starting doesn’t work, re-deploying doesn’t work.
Me too
2023-02-03T18:17:54.475 proxy[6460b70e] lhr [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 30)
2023-02-03T18:18:05.559 proxy[6460b70e] lhr [warn] Could not proxy HTTP request. Retrying in 985 ms (attempt 40)
2023-02-03T18:18:16.505 proxy[6460b70e] lhr [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 50)
@kurt tagging you incase you’re not aware of the issue.
We’re looking into this!
Same issue for us. It seems to be coming and going.
Follow the status page for updates: Fly.io Status - Edge routing issue
thanks @michael
I too am having issues with one of my projects: https://httpbin.dmuth.org/
I logged in and looked at my dashboards, and dashboards just aren’t being updated.
A suggestion for the future: a troubleshooting guide linked to from our dashboards so that when something like this happens, we can refer to a list of things to try, and places to raise concerns if self-troubleshooting doesn’t help.
Thank you!
As far as I can see now all that we had running is up, thanks folks!
Does this affect deploys as well?
A recurring problem this entire week has been “pending” apps stuck. They will not recover on their own.
I was able to get my app working by enabling dedicated CPU (some reason this worked), but my app is still stuck in the pending state for all but 1 server.
That’s probably unrelated, but we’ve been investigating that as well.
Just confirming that this was an issue for us as well via reverse proxy and is working again. IP was unreachable.
I lost access to another server at ~2:08p Eastern time.
This is actually happening to me right now. The app deployed successfully, I can ssh into it, but I can’t access its url
I haven’t been able to deploy all morning. The command keeps getting stuck at the Running release task (pending)... stage.
I haven´t been able to deploy this morning too. Every try I got this error message:
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GRED8E8X645881C38TMD74MZ-gru
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GRED8GCDEFZ18JW55PD3SDVX-gru
Any clues?
