Since about 12:12 UTC on 2026-10-26, a machine of ours in ord can’t complete TCP connections to several AWS EC2 instances (us-east-1 and eu-west-1). Our machine in cdg reaches the same instances normally.
• Affected: machine 28714d3b132798 (ord), egress IP 152.233.40.75
• Working: machine 2863e15b579d58 (cdg), egress IP 167.88.159.95
What we see:
1. From ord, connections to ports like 8443 and 8080 on these instances time out. The same ports connect from cdg and from other networks.
2. A packet capture on one instance shows the SYN from 152.233.40.75 arriving and the SYN-ACK going out. The ord machine never receives it and keeps retransmitting.
3. It depends on the source port. From ord, port 443 on the same instances returns ECONNREFUSED, so the RST gets back, while 8443 times out. Another EC2 instance replying from ports 80 and 443 works fine.
4. ICMP from these instances to 152.233.40.75 gets no reply, while 152.233.40.74, 152.233.40.76 and 167.88.159.95 all reply. 152.233.40.75 answers ping from non-AWS networks.
It looks like a filter on traffic into 152.233.40.75 from AWS that only lets source ports 80/443 through, possibly DDoS mitigation on that shared egress IP. Could someone at Fly check whether there’s a filter on that IP? I can send the instance IPs and packet captures privately.
outgoing connections from Fly machines go through a very basic routing stack/set of rules. Fly.io does nothing that would restrict outbound connections to only ports 443 or 80.
You mentioned ECONNREFUSED - that’s the destination refusing the connection, so I wonder if it’s worth inspecting the particular EC2 instance that’s having problems, or asking AWS support about this.
I’m surprised you get ICMP replies from AWS-hosted servers at all - AWS famously blocks ICMP, which makes diagnosing routing/reachability issues failry tricky. Are you sure those are AWS hosts? You can try mtr or traceroute in tcp mode, but those also tend to not be too reliable.
Possibly the same issue, in fra. From about 13:28 UTC our machine 286557e1c01e78 (outbound IP 152.233.61.15) couldn’t open TCP connections to anything we tried at AWS eu-central-1: PlanetScale Postgres on 5432 and HTTPS on 443 to s3.eu-central-1.amazonaws.com and api.planetscale.com. So for us 443 failed too. Other internet destinations worked from the same machine, and the same AWS endpoints were reachable from outside Fly. A restart didn’t help; cloning to a new machine fixed it at 13:40. The new machine’s outbound IP is 152.233.61.27, in the same /24, and it works fine.
Hi @wildsylvan - same as above, I wonder if it’s possible to reach out to AWS, as it looks like they are blocking traffic for some reason. As I’ve mentioned (and you cofirmed when saying “Other internet destinations worked from the same machine”) a Fly machine is not restricted when it comes to establishing outbound connections. As I said earlier, a traceroute might help determine what’s going on (though it might just confirm the route looks OK, further cementing the theory that some blocking is happening at AWS’s ingress layer).
Your issue is likely separate from the one reported in this thread. In this thread there seems to be an active ECONNREFUSED sent from AWS side, not just packets being loss. We don’t really do this on our side, so in this thread it is likely that some policy on AWS side got triggered for some reason. I don’t have an answer for your case in arn but we can keep the discussion in the other thread.
Thanks for looking. To clarify, it’s the return traffic to our egress IP that’s lost, not outbound filtering:
• A capture on the EC2 instance shows our SYN from 152.233.40.75 arriving and the SYN-ACK leaving the instance. The machine never receives it.
• The ECONNREFUSED on 443 is expected, since nothing listens on 443 there. It shows the RST from port 443 gets back to us, while the SYN-ACK from 8443 on the same instance doesn't. 8443 works from our cdg machine and from other networks.
• The pings went from the EC2 instances to 152.233.40.75 (no reply) and to 152.233.40.74, 152.233.40.76 and 167.88.159.95 (all replied). So ICMP works from those instances, just not to .75.
• All the instances are in AWS's published ip-ranges.json under EC2 (us-east-1 and eu-west-1).
It’s still failing at 14:03 UTC. The fra report above looks like the same thing: one egress IP can’t reach AWS, its neighbours are fine, and a new IP fixed it. Is it possible AWS is dropping traffic to certain Fly egress IPs? Have you seen other reports like this?
Hi. One of my two machines, in CDG, can not connect to supabase. It seems related to this thread.
Same signature in cdg since ~13:13 UTC today. Machine 8dd963be2d0e18 (app dito-api-staging), egress IP 167.88.159.81.
Not destination-specific: from the machine, TCP to portquiz.net (OVH, listens on all ports) works on 80/443 only; 22, 25, 3306, 5432, 5433, 6543, 8080, 9000, 27017 all time out. Same for every Supabase pooler region and for AWS endpoints. IPv6 egress is unaffected.
TCP traceroute: hop1 172.19.1.225, hop2 167.88.159.254, hop3 87.245.246.252 (RETN, AS9002), then no replies at all beyond hop 3 (ICMP, TCP) EXCEPT SYN-ACKs sourced from port 80/443, which arrive. `traceroute -T -p 5432 --sport=443` fails, `traceroute -T -p 443 --sport=5432` succeeds: the rule matches the source port of return traffic, not the local port. Hops 2 and 3 answer ping. The IP answers ping from a Cogent-routed network.
We spent some time digging into this, we believe this is likely some problem on AWS side unless all of our upstream transits started dropping packets to AWS at the same time at multiple locations specifically for non-80/443. It’s possible that we triggered some kind of firewall policy on their side but we would have no way of knowing. If you’re a customer of AWS, you could try raising this with them as well and see whether they would have any insight. In the mean time, switching machines definitely is a way to work around this kind of IP-based blockage, or you could try out static egress IPs that’s more easily swappable and gives you better predictability of which IP your connections appear to originate from on the other end.
portquiz.net seems to be served from AWS as well? Have you observed the same issue for non-AWS destinations?
One more data point for our IP: from the ord machine at 14:30 UTC, an OVH server on port 8080 connects fine, while the AWS instances on 8443/8080 still time out. So for 152.233.40.75 it’s specific to AWS sources, not all non-80/443 return traffic. Our cdg machine (167.88.159.95) still reaches the same AWS instances normally.
Hi, we have checked ourselves with an EC2 instance, and it looks like AWS is missing return path to some of our server IPs, interestingly other IPs in the same /24 subnet(s) route just fine. This really suggests something is funky on AWS’s side with return path routing. You may want to try out static egress IPs on our side and raising this issue with AWS as well as a customer.
Thanks @PeterCxy, that matches what we saw. One more data point from fra: it happened to us a second time. The machine we cloned to at 13:40 (outbound 152.233.61.27) lost new TCP connections to AWS eu-central-1 at about 15:56 UTC. Connections it had opened before that kept working; only new ones timed out. Cloning again at 15:59 gave us 152.233.61.1, same /24, which is still fine at 16:10.
So in one /24: .15 broke at about 13:28, .27 at about 15:56, and .1 works for now. We’ve asked PlanetScale, whose AWS account our database runs on, to raise it with AWS. We’re holding off on a static egress IP for now, since it looks like any address can be hit and a fixed one would need swapping the same way.