Rails 7.1 app build fails on bootsnap or assets:precompile

Questions / Help Build debugging
1

I have an existing Rails app that’s been upgraded to Rails 7.1. I’ve created a new Fly app and have been trying to deploy for a full day with no success. I can successfully build the app and run it locally using Docker Desktop, so I know there are no simple typos or anything in the Dockerfile or other configs (I use docker compose locally in development).

When running fly deploy --no-cache I’m getting the following exception when hitting either the assets:precompile step or the bootsnap precompile step (I’ve played around with different Dockerfile configurations to see if anything is improved). It seems to be failing whenever the Rails env is loaded. I don’t have any solutions, but my hypotheses on the cause are the following:

  • The gems can’t be found for some reason (although it looks like bootsnap is found, while msgpack isn’t)
  • It’s trying to connect to the database for some reason, and the error is a red herring. I passed in the DATABASE_URL as a build arg, but still got the same error.
 => ERROR [build 6/6] RUN if [ "production" != "development" ]; then     SECRET  0.5s
------
 > [build 6/6] RUN if [ "production" != "development" ]; then     SECRET_KEY_BASE_DUMMY=1 DOCKER_BUILD=1 ./bin/rails assets:precompile; fi:
#0 0.526 <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require': /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack/msgpack.so: cannot open shared object file: No such file or directory - /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack/msgpack.so (LoadError)
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack.rb:7:in `<top (required)>'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/store.rb:5:in `block in <top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/explicit_require.rb:45:in `rescue in with_gems'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/explicit_require.rb:41:in `with_gems'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/store.rb:5:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache.rb:68:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache.rb:68:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:5:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:5:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/setup.rb:3:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/setup.rb:3:in `<top (required)>'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/config/boot.rb:6:in `<top (required)>'
#0 0.526        from ./bin/rails:3:in `require_relative'
#0 0.526        from ./bin/rails:3:in `<main>'
#0 0.526 <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require': /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack/msgpack.so: cannot open shared object file: No such file or directory - /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack/msgpack.so (LoadError)
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/msgpack-1.7.2/lib/msgpack.rb:7:in `<top (required)>'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/store.rb:5:in `block in <top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/explicit_require.rb:42:in `with_gems'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/store.rb:5:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache.rb:68:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache.rb:68:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:5:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:5:in `<top (required)>'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/setup.rb:3:in `require_relative'
#0 0.526        from /app/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.16.0/lib/bootsnap/setup.rb:3:in `<top (required)>'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from <internal:/usr/local/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:38:in `require'
#0 0.526        from /app/config/boot.rb:6:in `<top (required)>'
#0 0.526        from ./bin/rails:3:in `require_relative'
#0 0.526        from ./bin/rails:3:in `<main>'
------
Error: failed to fetch an image or build from source: error building: failed to solve: executor failed running [/bin/sh -c if [ "${RAILS_ENV}" != "development" ]; then     SECRET_KEY_BASE_DUMMY=1 DOCKER_BUILD=1 ./bin/rails assets:precompile; fi]: exit code: 1

fly.toml

# fly.toml app configuration file generated for rushline on 2023-11-13T13:33:59-05:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "rushline"
primary_region = "yyz"
console_command = "/app/bin/rails console"

[build]

[deploy]
  release_command = "/app/bin/rails db:prepare"

[env]
  # WEB_CONCURRENCY = "4"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 1
  processes = ["app"]
  [http_service.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "requests"

[[statics]]
  guest_path = "/app/public"
  url_prefix = "/"

Dockerfile (I’ve tried a few variations, some more complicated, this is the one that’s generated from the dockerfile-rails gem)

# syntax = docker/dockerfile:1

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.2.2
FROM ruby:$RUBY_VERSION-slim as base

# Rails app lives here
WORKDIR /app

ARG RAILS_ENV=production
ARG NODE_ENV=production

# Set production environment
ENV RAILS_ENV=${RAILS_ENV} \
    NODE_ENV="${NODE_ENV}" \
    BUNDLE_WITHOUT="development:test" \
    BUNDLE_DEPLOYMENT="1"

# Update gems and bundler
RUN gem update --system --no-document && \
    gem install -N bundler

# Install packages needed to install nodejs
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Install Node.js
ARG NODE_VERSION=18.16.1
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
    /tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
    rm -rf /tmp/node-build-master

###############################################################################

# Throw-away build stages to reduce size of final image
FROM base as prebuild

# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential libpq-dev libvips node-gyp pkg-config python-is-python3

###############################################################################

FROM prebuild as node

# Install yarn
ARG YARN_VERSION=3.6.1
RUN corepack enable && \
    corepack prepare yarn@${YARN_VERSION} --activate

# Install node modules
COPY --link package.json yarn.lock .yarnrc.yml ./
RUN yarn install --immutable

###############################################################################

FROM prebuild as build

# Build options
ENV PATH="/usr/local/node/bin:${PATH}"

# Install application gems
COPY --link Gemfile Gemfile.lock ./
RUN bundle install && \
    bundle exec bootsnap precompile --gemfile && \
    rm -rf ~/.bundle/ ${BUNDLE_PATH}/ruby/*/cache ${BUNDLE_PATH}/ruby/*/bundler/gems/*/.git

# Copy node modules
COPY --from=node /app/node_modules /app/node_modules
COPY --from=node /usr/local/node /usr/local/node
ENV PATH=/usr/local/node/bin:${PATH}

# Copy application code
COPY --link . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN if [ "${RAILS_ENV}" != "development" ]; then \
    SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile; fi

###############################################################################

# Final stage for app image
FROM base as app

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl imagemagick libvips postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Copy built artifacts: gems, application
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build /app /app

# Run and own only the runtime files as a non-root user for security
RUN useradd rails --create-home --shell /bin/bash && \
    chown -R rails:rails db log tmp
USER rails:rails

# Entrypoint prepares the database.
ENTRYPOINT ["/app/bin/docker-entrypoint"]

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/rails", "server"]

Gemfile

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby "3.2.2"

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem "rails", "~> 7.1"

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem "sprockets-rails"

# Use postgresql as the database for Active Record
gem "pg", "~> 1.1"

# Use the Puma web server [https://github.com/puma/puma]
gem "puma", "~> 6.0"

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem "turbo-rails"

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem "stimulus-rails"

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem "jbuilder"

# Use Active Model has_secure_password [https://guides.rubyonrails.org/active_model_basics.html#securepassword]
gem "bcrypt", "~> 3.1.7"

# Reduces boot times through caching; required in config/boot.rb
gem "bootsnap", require: false

# Use Redis adapter to run Action Cable in production
gem "redis", "~> 4.0"

# Use Kredis to get higher-level data types in Redis [https://github.com/rails/kredis]
# gem "kredis"

# Provides a fast, feature rich JS bundling system [https://vite-ruby.netlify.app/guide/rails.html]
gem "vite_rails"

gem "newrelic_rpm"
gem "bugsnag"

# gem "webpacker", "~> 5.4"

gem "cssbundling-rails", "~> 1.1"
gem "autoprefixer-rails"

gem "inline_svg"

gem "friendly_id", "~> 5.4"

gem "carrierwave"
gem "carrierwave-aws"
gem "mini_magick"

gem "icalendar", require: ["icalendar", 'icalendar/tzinfo']

gem "gon", "~> 6.4.0"
gem "kaminari", "~> 1.2.0"

gem "whenever", require: false

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw, :jruby]  # 2021: do we actually use this anywhere?

group :development, :test do
  # See https://guides.rubyonrails.org/debugging_rails_applications.html#debugging-with-the-debug-gem
  gem "debug", platforms: %i[ mri mingw x64_mingw ]
end

group :development do
  # Use console on exceptions pages [https://github.com/rails/web-console]
  gem "web-console"

  # Add speed badges [https://github.com/MiniProfiler/rack-mini-profiler]
  # gem "rack-mini-profiler"

  # Speed up commands on slow machines / big apps [https://github.com/rails/spring]
  # gem "spring"

  gem "solargraph"

  gem "letter_opener"

  gem "rubocop-shopify", require: false
end

group :test do
  # Use system testing [https://guides.rubyonrails.org/testing.html#system-testing]
  gem "capybara"
  gem "selenium-webdriver"
  gem "webdrivers"

  # Spec Setup
  gem "rspec-rails", "~> 6.0.3"
  gem "factory_bot_rails", "~> 6.2.0"

  # Additional Testing Gems
  gem "shoulda-matchers"

  gem "rack_session_access"  # why do we need this?
  gem "timecop"
  gem "faker"
end

Any help is greatly appreciated!

2

I’ve attempted to reproduce the problem, but so far have failed. Here’s what I tried:

rails new demo --js esbuild
cd demo
# <- replace Gemfile
# <- add to scripts section in package.json: "build:css": "echo build:css",
bundle install
bundle exec vite install
fly launch
fly deploy
3

Thanks for giving it a shot. I actually tried something similar last night with a barebones Rails app and didn’t have any issues deploying.

Do you have any thoughts on how I can debug further? I always struggle trying to debug Docker build processes. It would be good to see the ENV vars and the state of the file system when it fails, but I’m not sure that’s possible…

4

:man_facepalming: I figured it out. Just like all of these types of things, the resolution was simple once I was looking in the right place.

The issue was that when I ran fly launch on this app, it replaced my pre-existing .dockerignore file, and I missed out on ignoring the local vendor/bundle folder, so that was being sent as part of the build context (along with a bunch of other stuff that should also have been ignored). I believe because the gem files were found, they weren’t being installed, but the platform didn’t match between my M2 and the Linux system, so the right native extensions for msgpack weren’t found.

The tip off was that the build context that was being sent was over 1GB. It was taking a while to upload and I realized I didn’t fully understand what that step was doing, so after a bit of basic research, I dug into the .dockerignore more carefully, and noticed the missing entries.

A bit of feedback on fly launch

I’ve noticed that fly launch automatically overwrites Dockerfile, .dockerignore, and docker-entrypoint if they exist. I think I get why that approach was taken (ensures deployability, trying to make it simple for users), but it would be nice if there was a prompt to Accept all Fly defaults? or something so I could have the option of looking at the diffs. I know this can be done after (if using source control), but to me it felt a bit heavy handed to replace those files unilaterally.

5

That’s a bug, and a regression. I’m not sure what changed to cause that, but I will fix that. Thanks for the report!

By the way, the answer to your previous question is to try out

fly console --dockerfile Dockerfile.test -C bin/bash

Where Dockerfile.test is a copy of your Dockerfile where you ruthlessly strip out the parts that don’t work. What the above command will do is build that Dockerfile and shell into the result as root where you can explore. Update your Dockerfile based on what you found, and iterate.

1 Like