Your first four worked for me without a specific egress facility, from a Fly machine in London.
I assume your released egress ID is still available internally, and someone at Fly could have a look at that. Does release remove the fixed egress address? What happens if you create another?
The most recent Infra Log entry mentioned a known bug with the static egress IPs, which they are (/ were) gradually fixing. (“For existing hosts, kernel upgrades will be applied as they are gradually rebooted. Of course, this is a somewhat slow process and unless we decided to reboot every existing host at once, we’ll likely continue to see these issues from time to time.”)
(Also, they have a more lasting fix in the works, .)
[A “host” in this context is the underlying physical server machine, so rebooting the VM wouldn’t have helped. Cloning it might have worked, though, since that can land you on a different physical machine.]
In general, the community forum here is the slowest and least reliable way of contacting Fly Support. People who find that they have a clearly broken static egress setup and who have a Support Plan should instead use their ticketing system in the dashboard. From what they were saying in the Log, there’s a reset procedure that they can do at their end…
Same bug here, same start time, app quarri in iad — and in our case it hit all three of our dedicated egress IPs at once, so it isn’t one unlucky host:
Since ~16:30 UTC 21 Aug: TCP SYNs to api.motherduck.com, api.openai.com, identity.xero.com, api.github.com and a customer MSSQL host on :1433 get no reply; DNS fine; MSS clamp to 1200/1000 changes nothing (fails at the SYN, so not PMTU); IPv6 sometimes handshakes then stalls at 0 bytes. Early on a couple of destinations still answered and google.com flapped 200/timeout; by 15:51 UTC today even ifconfig.me over v4 is dead. flyctl machine restart did nothing. Control: ephemeral machines in the same app/region on shared-pool egress (64.34.84.14, 216.246.40.69) reach everything instantly.
One more data point: flyctl machine egress-ip list -a quarri prints an empty table for us, although ifconfig.me (while it still answered) confirmed the machines were egressing from the IPs above.
We can’t take the egress-ip release route — these three IPs are allowlisted in customers’ firewalls for MSSQL sources, so releasing/re-allocating means customer-side changes. We’ve moved production onto cloned shared-pool machines for now, but the allowlisted sources are down until the dedicated path is repaired.
@flyio-support — could you run the reset procedure mentioned in the Infra Log for these three IPs (or migrate the machines to a patched host) while keeping the same IP assignments? Happy to run any diagnostics you need. We’re on pay-as-you-go so the dashboard ticketing isn’t available to us.
Just chucking ideas into the pool in the spirit of getting you going.
You mentioned DNS is fine; I was going to suggest adding in Google or Cloudflare’s dotted quads. Failing that, could you configure a proxy inside these machines, which point to a stable proxy somewhere else? I don’t know if that would be subject to the same networking problem you’re bumping into, but it might be worth a ponder.
Thanks @halfer. DNS is genuinely fine (resolves instantly, it’s the SYNs that never get answered), and a proxy doesn’t help here unfortunately: outbound from these machines is dead to any destination, so they can’t reach a proxy, and if traffic went via another box it would exit from that box’s IP rather than the allowlisted ones — which is the whole constraint. So it’s really down to the host-side tunnel reset on Fly’s end.
It’s weird that you and @Desprit had the same onset time, since it doesn’t sound like he is in iad, ..
The Machine-scoped egress IPs are deprecated now, so they might not have as thorough of monitoring in place.
According to the official docs, people can upgrade to app-scoped egress IPs without losing the specific addresses that they have already, although if fly m egress-ip list is reading empty then that might not be viable…
Aside: I’ve added the Questions / Help category to this thread, which will improve the odds of it getting noticed—as explained in the sticky post. (In contrast, the @ notifications don’t actually work with the flyio-support account, as far as I know.)
@mayailurus that was the answer — thank you. Outcome, for anyone else landing here:
The empty fly m egress-ip list was my own fault: flyctl was on v0.4.1. After fly version upgrade (v0.4.87) it lists all three machine-scoped IPs fine, and that version also has the promote subcommand.
fly machine egress-ip promote <machine-id> -a quarri converted one of the IPs to app-scoped, same address. Every machine in the app/region picked it up immediately — including fresh clones on healthy hosts — and outbound is fully working again: MotherDuck connects in 0.5s, and the customer MSSQL sources that allowlist that IP synced successfully (3M+ rows) within minutes. No customer-side change needed.
So for machine-scoped egress IPs caught by this bug: upgrade flyctl, promote to app-scoped, and make sure at least one machine is on a host other than the broken one (a clone will do). The two IPs still machine-scoped on the stopped originals remain dead, which is consistent with the host-tunnel explanation.
Hi! Sorry for this, it does seem like there was some issue with machine-scoped egress IPs in iad, though probably separate from the host tunnel issue mentioned in the Infra Log. In any case, moving to app-scoped egress IPs is the right move as they should be much more reliable compared to machine-scoped egress IPs; app-scoped ones were a redesign based on the limitations and unreliability with machine-scoped egress IPs. Machine-scoped ones are considered “soft-deprecated” at this point), though we should probably have been much louder in the messaging here to “nudge” the move to app-scoped egress IPs.
.)
..