"Missing encryption key to decrypt file with" When deploying existing Rails app locally using `fly deploy`

Questions / Help Build debugging
1

I’m deploying an existing Rails app, I have an entirely fresh Fly organisation.
fly launch works fine. fly deploy fails on the assets:precomplie step.

Obviously, this is a common enough problem that it’s mentioned in the Getting Started Guide

I have my rails master key in the correct config file, and the error output even links to the file.
fly launch successfully created the RAILS_MASTER_KEY secret and I’ve confirmed that the key is correct.

Console Output:

 => [build 5/6] RUN bundle exec bootsnap precompile app/ lib/                                                                                            1.6s
 => ERROR [build 6/6] RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile                                                                            3.4s
------
 > [build 6/6] RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile:
#16 3.370 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile]: exit code: 1

That all being said, I don’t know why any credentials should be required during the build step. Any suggestions on how to avoid this for a “stock” rails setup?

2

When running with the suggested deferred precomplie step (generate dockerfile --precompile=defer)

I get the following error during deployment.

[info] Errno::EACCES: Permission denied @ dir_s_mkdir - /rails/tmp/cache/assets

This error is being reported to honeybadger, and the API key for honeybadger is stored in the credentials file. So access to the credentials is definitely possible.

3

Yeah, when you defer the build, it runs the compilation step during a time when secrets are available, which is why it’s being reported to Honey Badger. The only recommend deferring the build for troubleshooting, then switching back to running the compilation when secrets aren’t available.

If you are evaluating Fly.io for the first time there may be some value in setting precompile to defer initially for evaluation and then work over time to eliminate the issues that prevent you from running this step at build time. Once those issues are resolved, regenerate your Dockerfile using the following command:

bin/rails generate dockerfile --precompile=build

From Existing Rails Apps · Fly Docs

Could you paste the contents of $ bundle show in here so I can see the exact version of Rails you’re running?

4

Thanks for the reply Brad!
We’re on the latest release version of Rails - 7.0.4.3.

I have managed to get a deployment to work now, but it feels very hacky!

In the Dockerfile I had to add an arg and env var for the Rails Master Key

ARG RAILS_MASTER_KEY
# Set production environment
ENV RAILS_ENV="production" \
    BUNDLE_WITHOUT="development:test" \
    BUNDLE_DEPLOYMENT="1" \
    RAILS_MASTER_KEY=${RAILS_MASTER_KEY}

and then had to pass the master key as a build arg

fly deploy --build-arg RAILS_MASTER_KEY=$(cat config/master.key)

As far as I can tell, the build step shouldn’t need the master key. I know there are further improvements coming to this in 7.1 - @rubys has been hard at work (which is super exciting).

I’ve tried the backport approach suggested by DHH here but that didn’t seem to have any effect.
Arianf’s suggestion in the same thread also didn’t work.

$ bundle show

Gems included by the bundle:
  * actioncable (7.0.4.3)
  * actionmailbox (7.0.4.3)
  * actionmailer (7.0.4.3)
  * actionpack (7.0.4.3)
  * actiontext (7.0.4.3)
  * actionview (7.0.4.3)
  * active_storage_validations (1.0.3)
  * activejob (7.0.4.3)
  * activemodel (7.0.4.3)
  * activerecord (7.0.4.3)
  * activestorage (7.0.4.3)
  * activesupport (7.0.4.3)
  * addressable (2.8.4)
  * apnotic (1.7.0)
  * ast (2.4.2)
  * aws-eventstream (1.2.0)
  * aws-partitions (1.745.0)
  * aws-sdk-core (3.171.0)
  * aws-sdk-kms (1.63.0)
  * aws-sdk-s3 (1.120.1)
  * aws-sigv4 (1.5.2)
  * bcrypt (3.1.18)
  * bindex (0.8.1)
  * bootsnap (1.16.0)
  * builder (3.2.4)
  * bundler (2.4.10)
  * capybara (3.39.0)
  * concurrent-ruby (1.2.2)
  * connection_pool (2.4.0)
  * crass (1.0.6)
  * date (3.3.3)
  * debug (1.7.2)
  * devise (4.9.2)
  * devise-jwt (0.10.0)
  * docile (1.4.0)
  * domain_name (0.5.20190701)
  * dry-auto_inject (1.0.1)
  * dry-configurable (1.0.1)
  * dry-core (1.0.0)
  * erubi (1.12.0)
  * factory_bot (6.2.1)
  * factory_bot_rails (6.2.0)
  * faker (3.1.1)
  * faraday (2.7.4)
  * faraday-net_http (3.0.2)
  * ffi (1.15.5)
  * ffi-compiler (1.0.1)
  * globalid (1.1.0)
  * googleauth (1.5.1)
  * honeybadger (5.2.1)
  * http (5.1.1)
  * http-2 (0.11.0)
  * http-cookie (1.0.5)
  * http-form_data (2.3.0)
  * i18n (1.12.0)
  * image_processing (1.12.2)
  * importmap-rails (1.1.5)
  * inline_svg (1.9.0)
  * io-console (0.6.0)
  * irb (1.6.4)
  * jbuilder (2.11.5)
  * jmespath (1.6.2)
  * json (2.6.3)
  * jwt (2.7.0)
  * kwalify (0.7.2)
  * llhttp-ffi (0.4.0)
  * loofah (2.20.0)
  * mail (2.8.1)
  * marcel (1.0.2)
  * matrix (0.4.2)
  * memoist (0.16.2)
  * method_source (1.0.0)
  * mini_magick (4.12.0)
  * mini_mime (1.1.2)
  * minitest (5.18.0)
  * msgpack (1.7.0)
  * multi_json (1.15.0)
  * net-http2 (0.18.5)
  * net-imap (0.3.4)
  * net-pop (0.1.2)
  * net-protocol (0.2.1)
  * net-smtp (0.3.3)
  * nio4r (2.5.9)
  * nokogiri (1.14.3)
  * noticed (1.6.0)
  * orm_adapter (0.5.0)
  * os (1.1.4)
  * pagy (5.10.1)
  * parallel (1.22.1)
  * parser (3.2.2.0)
  * pg (1.4.6)
  * pg_search (2.3.6)
  * propshaft (0.7.0)
  * public_suffix (5.0.1)
  * puma (6.2.1)
  * racc (1.6.2)
  * rack (2.2.6.4)
  * rack-test (2.1.0)
  * rails (7.0.4.3)
  * rails-dom-testing (2.0.3)
  * rails-html-sanitizer (1.5.0)
  * rails_performance (1.0.5.2)
  * railties (7.0.4.3)
  * rainbow (3.1.1)
  * rake (13.0.6)
  * redis (5.0.6)
  * redis-client (0.14.1)
  * redis-namespace (1.10.0)
  * reek (6.1.4)
  * regexp_parser (2.7.0)
  * reline (0.3.3)
  * responders (3.1.0)
  * rexml (3.2.5)
  * rubocop (1.50.0)
  * rubocop-ast (1.28.0)
  * rubocop-md (1.2.0)
  * rubocop-minitest (0.30.0)
  * rubocop-performance (1.17.1)
  * rubocop-rails (2.19.0)
  * ruby-progressbar (1.13.0)
  * ruby-vips (2.1.4)
  * ruby2_keywords (0.0.5)
  * rubyzip (2.3.2)
  * sassc (2.4.0)
  * sassc-rails (2.1.2)
  * selenium-webdriver (4.8.6)
  * sidekiq (7.0.8)
  * signet (0.17.0)
  * simplecov (0.22.0)
  * simplecov-cobertura (2.1.0)
  * simplecov-html (0.12.3)
  * simplecov_json_formatter (0.1.4)
  * sprockets (4.2.0)
  * sprockets-rails (3.4.2)
  * stimulus-rails (1.2.1)
  * thor (1.2.1)
  * tilt (2.1.0)
  * timeout (0.3.2)
  * turbo-rails (1.4.0)
  * tzinfo (2.0.6)
  * unf (0.1.4)
  * unf_ext (0.0.8.2)
  * unicode-display_width (2.4.2)
  * warden (1.2.9)
  * warden-jwt_auth (0.8.0)
  * web-console (4.2.0)
  * webdrivers (5.2.0)
  * websocket (1.2.9)
  * websocket-driver (0.7.5)
  * websocket-extensions (0.1.5)
  * xpath (3.2.0)
  * zeitwerk (2.6.7)
5

Yeah I really dislike how Rails wants a “dummy” key for this step. That won’t fix your problem now though, so let’s try this instead.

$ bundle add dockerfile-rails
$ rails g dockerfile
$ fly deploy

In theory that will generate the correct Dockerfile for your version of rails. If it doesn’t then an issue can be opened at GitHub - rubys/dockerfile-rails: Provides a Rails generator to produce Dockerfiles and related files. so it fixes this problem for you and others.

6

Those are the steps I followed to start off with.

I’ll look to create an issue on GH tomorrow, thanks for your help Brad.

7

I’ve just confirmed, after deleting dockerfiles and rerunning

$ rails g dockerfile
$ fly deploy

I get this error.

------
 > [build 6/6] RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile:
#16 1.807 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile]: exit code: 1

It looks as though my setup still needs the master key at build time.

I’ve progressed to the “Different Error” stage by adding the RAILS_MASTER_KEY as an env var. ENV RAILS_MASTER_KEY="$(cat config/master.key)"

I’m not certain if this is progression or regression but I now get the same error with some extra output and this error in Honeybadger

8

Let’s try a vanilla application. You can run through the steps shown in the Getting Started guide. If it helps, I’ve condensed the important steps to the following:

rails new list; cd list; bin/rails generate scaffold Name name; fly launch; fly deploy

Feel free to accept all of the defaults, this app doesn’t need postgres or redis.

This app deploys for me, and I assume it will deploy for you. If not, something really weird is going on. If it does work for you, then something in your config is accessing a credential, and for some reason the stack traceback produced here isn’t being helpful.

1 Like
9

Okay! I think I’ve found the problem.

After a bit of trial and error with a vanilla application, deploys work with cssbundling-rails but fail with sassc-rails (which we’re currently using) and dartsass-rails (which we’re going to upgrade to at some point).

The only way I can get the build to work with sassc-rails or dartsass-rails is by passing the master key as a build arg.

Probably worth noting that we’re using propshaft and importmaps, but they don’t appear to cause the failure.

10

If I add bundle add sassc-rails to the one liner above I can still deploy. Is there anything more I need to do to reproduce the problem?

11

I’m getting some odd errors when I try and use sassc-rails on a clean install but this one-liner fails in the same way with dartsass-rails

rails new test-for-fly -a propshaft; 
cd test-for-fly;
bin/bundle add dartsass-rails;
bin/rails generate scaffold Coffee name:string description:text website:string --primary-key-type=string; 
fly launch; 
fly deploy;
12

Is there a way to delete all apps from my fly account… I have now created several while testing this!

13

Here’s what I use (Mac, Linux, WSL2):

fly apps list | grep -v ^fly-builder | cut -d ' ' -f 1 | xargs -n 1 fly apps destroy -y
1 Like
14

Thank you! :bowing_man:

15

When I try that, I get:

#16 1.243 Error reading app/assets/stylesheets/application.scss: Cannot open file.
#16 2.246 rails aborted!

That error goes away if I add:

./bin/rails dartsass:install

I’m still not seeing the original error.

16

Okay, I think this is progress…

I’m not longer able to replicate with a fresh build… which means its just a me-issue. Or more specifically a configuration problem in our current app.

Thank you very much for your time and efforts on this so far.

17

For what it’s worth, I’m trying to migrate from Heroku to Fly with a Rails 7.0.4.3 app that does not use Sass at all, but does have GitHub - rails/jsbundling-rails: Bundle and transpile JavaScript in Rails with esbuild, rollup.js, or Webpack., and I’m having the same issue (if I ignore the qemu macOS m1/m2 issues and just try to deploy…).

Your manually passing RAILS_MASTER_KEY got me past it and now onto other issues (like yarn missing despite being installed in the generated Dockerfile, but that’s for another day—edit: it’s because it’s in another stage, but asset precompilation with jsbundling-rails requires yarn, so keeping the executable around is needed).

18

This Rails PR might help:

If require_master_key is set to true in production.rb like this:

config.require_master_key = true

then setting dummy value like this:

RUN SECRET_KEY_BASE=DUMMY

won’t do anything. Rails will still go looking for a real key and will error out.

Comment from DHH:

“Yeah, I don’t think we want to make a liar about of that config. If you have a setup where you don’t want to inject the master key during build (and use dummy key base), then you need to turn that off.”

Based on that comment I turned off require_master_key and now precompile works without a real key:
config.require_master_key = false