Sounds like it’s being caused by concurrent remote builds. As found by other users e.g this thread:
(Note that we tried to run these concurrently but that threw an error)
As I replied there, if you want concurrent remote builds it seems you would need one of Fly’s larger paid plans. For example their Scale plan includes 10 concurrent remote builds. Check Plan Pricing · Fly