I think the problem is that Fly’s secrets are only available when the app is running, not during the build (edit: as @allison already pointed out).
You can create build-time secrets, but it’s a little complicated: Build Secrets · Fly Docs
Alternatively, if the values are not sensitive, you can set build-time environment variables in fly.toml using build arguments.