Secrets should work pretty much exactly like environment variables.*
A shot in the dark: Does your .env file contain quotes? And did you run flyctl secrets import directly on it?
flyctl secrets import does’t recognize normal quoting syntax (ref, code), so if your .env file has a line such as AUTH_TOKEN="abc" and you run flyctl secrets import < .env then the secret AUTH_TOKEN will be set to "abc" (literally, with the double quotes; which is bad). You can check if this happened by opening an SSH connection and running env, there shouldn’t be any quotes in the output.
*Secrets are not available at build time (notes), but I don’t think that’s the issue here.