Can I connect to remote database via wireguard, and if so, how?

You’re not far off!

You’d want something like a “bastion” host that you run in the AWS account that runs wireguard so you can make a connection to the database.

There is some prior art. You wouldn’t need to do exactly what these folks do but hopefully this illustrates some of the key parts.