Setting the upstream to <app>.fly.dev would work best, but you need to make sure it’s a variable for proxy_pass so nginx will resolve DNS more than once.
For what it’s worth, when we build docker-compose support in, it will launch multiple apps. So what you’re thinking is pretty close to “correct”.