I didn’t realize that this was during the build step. Swap is not relevant. And delete the max-old-space-size option as build machines start at 4Gb.
If you run fly apps list, one of those will be your builder (it will have a name starting with fly-builder). Run fly apps destroy on that builder.
Then try deploying again with a bigger builder:
flyctl deploy --remote-only --vm-size shared-cpu-4x:8196