Just figured this out, use directions from here to proxy your local connection.
I used this command to proxy since I had postico connected:
flyctl proxy 15432:5432 -a <postgres-app-name>
Then run:
pg_dump -h localhost -p 15432 -U postgres DATABASE_NAME > BACKUP_FILE_NAME
You’ll be prompted to enter your DB password and voila!, backup is saved to the current directory