Docker-in-Docker

Yes that would work if you add docker and docker compose you your app’s image. But I’d recommend you go even further separating your app code from the 3rd party code that you want to run with docker. Because docker is not really a secure sandbox.

Your particular applications sounds well suited for our machines API: Run User Code on Fly Machines · Fly Docs. Instead of shelling out a new process you would call the machine API with instructions on how to run the third party code.