No. This only works if the app already has public IP address. Are you doing fly-replay response to an app without public IP? In this case this won’t work.
Another small update on the issue - I had to rollback the change for now. Looks like it caused issues with our registry which I don’t fully understand yet.