This might be similar to my issue, which turned out to be due to the recent release of shared IPs:
You can check whether your app currently has a dedicated IP or shared IP by running flyctl ips list. The TYPE column says “public” for a dedicated IP and “public (shared)” for a shared IP.