Illegal Instruction

When I deploy today I’m getting an “Illegal Instruction” error on a binary that used to run fine with the same Dockerfile a few days ago.

Did anything in the underlying infrastructure change? I am still using the same instance type (shared-cpu-1x).

Maybe the builder is using a slightly different architecture compared to the vms that run the app? Any pointers welcome.

Strange. We virtualize the builder and the VM architecture, so they should be the same. Could the binary have been rebuilt in some way?

I am using the Dockerfile to build it. I think I ran into this in the past once or twice and fixed it by destroying the fly builder and deploying again. This time this did not succeed. I also tried to scale the app to dedicated vms and deployed again using --no-cache to see if that makes a difference. So far always “Illegal Instruction”.

I’m experiencing a similar error, in my case, it’s a Next.js app that used to work before yesterday downtime and when I redeployed it, now it doesn’t work, I had to rollback to an old image to make it work. My first thought was that something changed on the underlying architecture after the Consul cluster rebuilt but I’m not sure to be honest.

It’s hard to tell without the raw logs. This sounds like the kind of error that can happen with local Docker builds on arm (on an m1 mac, for example). We’ve seen other edge cases that can cause similar errors, but they’re pretty rare. If you post the complete build log here you may get more useful information from the community.

I am using a M2 mac if that makes a difference.

I just tried Sign In · Fly in the browser and manually executed the build steps in the dockerfile and the binary works just fine this way.

But as far as I understand by default the build happens on fly.io builders and not locally right?

Yea, you can however build locally while deploying using the --local-only flag. What does flyctl version output out of curiosity?

flyctl v0.0.488 darwin/arm64 Commit: d56b165d BuildDate: 2023-03-15T12:39:07Z

but I just updated a few minutes ago, it was 0.0.475 (installed via brew, upgraded via brew)

I have now deployed using fly deploy --no-cache --local-only
(before that I did update Docker for Mac) and the deploy ran way slower (network I guess) but the deploy was successful and interestingly: the binary now runs on the fly server without “Illegal Instruction”.

Not sure what it means though. I would have expected the local build to cause even more issues (that’s why I did not try it before) because I expected that my M2 Mac is a different architecture to the one on fly.

That said: there are so many moving parts involved that I’m not certain whether me using --local-only flag fixed it or something else.

1 Like

--no-cache fixed my issue as well. :+1:

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.