`ring` Rust dependency build failure with Nixpacks

Questions / Help Build debugging
1

Hello, all. I’m trying to use Nixpacks to build and deploy a Rust app, but I am hitting an open issue in one of my transitive dependencies:

My error message from flyctl deploy:

failed to execute ["aarch64-linux-musl-gcc" <snip> "-c" "-o/app/target/aarch64-unknown-linux-musl/release/build/ring-424dcd4565ecf50b/out/aesv8-armx-linux64.o" "/root/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/pregenerated/aesv8-armx-linux64.S"]: No such file or directory (os error 2)', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:653:9

The recommended fix there is to set some environment variables to force the cc and ar tools, so I tried running:

flyctl launch --name buscemi \
    --region ewr  \
    --nixpacks \
    --build-arg TARGET_CC=aarch64-unknown-linux-musl-gcc \
    --build-arg TARGET_AR=aarch64-unknown-linux-musl-ar \
    --build-arg NIXPACKS_LIBS=boringssl

But I still get the same error. I’m not sure exactly what to expect the gcc executable to be called in the context of building on fly with Nixpacks, but if my error is indeed the same as the Github issue, I would guess that aarch64-linux-musl-gcc is incorrect?

It seems possible to me that the cross-compilation part is a red herring, but does anyone have any guesses how I could work around this error in the context of using flyctl?

1 Like
2

That seems like the wrong architecture for Fly, we can only run x86_64 images right now.

Not sure why it’s building it for aarch64, that seems like an issue.

Looks like there are a few issues in the nixpacks repo: Search · repo:railwayapp/nixpacks ring · GitHub

If you do try to fix it, flyctl will automatically pickup NIXPACKS_ env vars and pass them to nixpacks. One of the solution suggests setting NIXPACKS_NO_MUSL=1, so that would look like:

NIXPACKS_NO_MUSL=1 flyctl launch ...
3

This definitely seems like the right thing to address, but I can’t seem to get nixpacks-in-fly to take the option you recommended:

NIXPACKS_NO_MUSL=1 flyctl deploy --nixpacks --build-arg NIXPACKS_NO_MUSL=1 --env NIXPACKS_NO_MUSL=1

...

╔═══════════════════════════════ Nixpacks v1.4.0 ══════════════════════════════╗
║ setup      │ binutils, gcc, (rust-bin.stable.latest.default.override {       ║
║            │ targets = ["aarch64-unknown-linux-musl"]; }), musl, musl.dev    ║
║──────────────────────────────────────────────────────────────────────────────║
║ build      │ mkdir -p bin                                                    ║
║            │ cargo build --release --target aarch64-unknown-linux-musl       ║
║            │ cp target/aarch64-unknown-linux-musl/release/buscemi bin        ║
║──────────────────────────────────────────────────────────────────────────────║
║ start      │ ./bin/buscemi                                                   ║
╚══════════════════════════════════════════════════════════════════════════════╝
...

<same error as before>
4

Hi! Just to confirm, are you running flyctl deploy from a device with an ARM (aarch64) CPU? I tried this on an ARM MacBook and ran into the same problem that you did. If this is the case, you might try running flyctl deploy from an x86 device if you have one.

Here’s what I think is going on:

flyctl runs the nixpacks tool locally. By default, it will try to build Rust applications with musl, and when it does so, it explicitly configures the target CPU architecture for the build to that of the local machine. So if your computer has an aarch64 CPU, nixpacks incorrectly tells Fly’s remote builder to use aarch64 tooling! (It turns out that the ring dependency is a red herring here; when I tried building a Rust app without a dependency on ring, the build still failed, just at a different step.)

Setting NIXPACKS_NO_MUSL=1 should indeed fix the problem, since it will get nixpacks to stop overriding the target architecture. Unfortunately, it looks like flyctl isn’t passing along that variable in the way that nixpacks expects to receive it, which is why it’s not working.

2 Likes
5

That’s a little odd, but I’m not entirely surprised!

There’s this logic in flyctl:

	for _, kv := range os.Environ() {
		if strings.HasPrefix(kv, "NIXPACKS_") {
			cmd.Env = append(cmd.Env, kv)
		}
	}

This should pass along any NIXPACKS_ prefixed env vars. I’m fairly sure this worked. There’s slightly more info here: Build images with nixpacks - #18 by jerome

2 Likes
6

This definitely looks like it! Like @jerome pointed out, the logic in flyctl looks like it should handle this properly, but for some reason it won’t pass this through. I wonder if there is any bandwidth to implement support for configuring nixpacks in the flyctl CLI, or maybe via fly.toml?

In the meantime, I found a workaround by looking in the nixpacks source: I can add a rust-toolchain.toml with some boring content and use that to force the proper build target with nixpacks.

2 Likes
7

Nice! Glad it’s working for you now.

I also looked through the nixpacks source, and it seems like it doesn’t actually read build environment variables from the system(!?), but rather requires you to use --env on the command line. I just got a PR merged updating flyctl to pass along the variables this way, so hopefully running NIXPACKS_NO_MUSL=1 flyctl deploy --nixpacks ... will also work as documented in the next release.

1 Like
8

Did this fix go out already? It seems to be working as expected now :open_mouth:

9

Yes, it went out in 0.0.475 today!

1 Like