We do have a decent example of how to connect to an RDS instance, it runs a wireguard peer + pgbouncer within a VPC: GitHub - fly-apps/rds-connector: Trivial Terraform example for a WireGuard peer to RDS
The best way to do this is to create “gateways” in your Azure or AWS private networks. You make Fly wireguard peers work, or you can run something like Tailscale in your app instances and let it manage all that.
As much as I hate IP allow lists, we might have what you need to use those in a few months. We just need a whole bunch more IPs first.