Automatic sqlite3 backups for Rails applications using LiteStream and Tigris

Fresh Produce
1

litestream
litestream1280×640 27.2 KB
tigris
tigris1595×651 16 KB

When running SQLite in production, you need to have a solid backup mechanism setup. And I say this as someone who has accidentally deleted the production SQLite database. Trust me, resilience is something you should have setup from day one.

– Stephen Margheim - Supercharge the One Person Framework with SQLite: Rails World 2024

With flyctl v0.3.58 and dockerfile-rails release v1.7.0, all that is required is fly launch.

Demo

No special configuration is required. Create an application and deploy it. For example:

rails new demo --css tailwind
cd demo
./bin/rails generate scaffold Post name:string title:string content:text
./bin/rails db:migrate
echo 'Rails.application.routes.draw { root "posts#index" }' >> config/routes.rb
flyctl launch

Accept the defaults and your app is up and running.

To opt out, indicate that you want to tweak the settings before proceeding and unselect Tigris.

To demonstrate worry free backups, use the application to make a draft blog post:

  • Click on the “New post” button

  • Enter your name, a title, and some content; then click “Create Post”

Now destroy your machine and volume. If you have jq installed you can do so with the following commands:

fly machine list --json | jq -r ".[]|.id" | xargs fly machine destroy -f
fly volume list --json | jq -r ".[]|.id" | xargs fly volume destroy -y

(If you don’t have jq, omit --json and run the destroy commands directly. Or just brew install jq / apt-get install jq)

Once this is done, run fly deploy, visit your web page. Your post should be there.

9 Likes
2

Posted this on the twits at x.com before I saw the link to this post.

When I run fly console, it spins up a Fly Machine that doesn’t have access to the filesystem where the SQLite database resides. It’s surprising when I run something like User.all and I don’t see users or it tells me the database doesn’t exist. Eventually I remember that fly console runs a different machine and I can run fly ssh console -C "bin/rails" as a workaround.

Could fly console be modified for SQLite installations so that it works in a non-surprising way?

In the past I’ve proposed a fly task runner. For example, it would be useful if I could run fly exec rake db:migrate and have it do what I expect and fly exec console load up the machine I expect to run Rails tasks. Idealy I could customize this to boot a new machine or shell into an existing machine.

3

Another thing worth mentioning: this doesn’t work.

direnv: export +AUTHORITY +CANONICAL_HOST +PORT +STRIPE_PRICE +STRIPE_PUBLISHABLE_KEY +STRIPE_SECRET_KEY +WEB_ADDRESS_HOST ~PATH
tinyzap/server [main] → cd ~/Desktop
direnv: unloading
bradgessler/Desktop → rails new demo --css tailwind
cd demo
./bin/rails generate scaffold Post name:string title:string content:text
./bin/rails db:migrate
echo 'Rails.application.routes.draw { root "posts#index" }' >> config/routes.rb
flyctl launch

      create
      create  README.md
      create  Rakefile
      create  .ruby-version
      create  config.ru
      create  .gitignore
      create  .gitattributes
      create  Gemfile
         run  git init from "."
Initialized empty Git repository in /Users/bradgessler/Desktop/demo/.git/
      create  app
      create  app/assets/stylesheets/application.css
      create  app/controllers/application_controller.rb
      create  app/helpers/application_helper.rb
      create  app/jobs/application_job.rb
      create  app/mailers/application_mailer.rb
      create  app/models/application_record.rb
      create  app/views/layouts/application.html.erb
      create  app/views/layouts/mailer.html.erb
      create  app/views/layouts/mailer.text.erb
      create  app/views/pwa/manifest.json.erb
      create  app/views/pwa/service-worker.js
      create  app/assets/images
      create  app/assets/images/.keep
      create  app/controllers/concerns/.keep
      create  app/models/concerns/.keep
      create  bin
      create  bin/brakeman
      create  bin/dev
      create  bin/rails
      create  bin/rake
      create  bin/rubocop
      create  bin/setup
      create  bin/thrust
      create  Dockerfile
      create  .dockerignore
      create  bin/docker-entrypoint
      create  .rubocop.yml
      create  .github/workflows
      create  .github/workflows/ci.yml
      create  .github/dependabot.yml
      create  config
      create  config/routes.rb
      create  config/application.rb
      create  config/environment.rb
      create  config/cable.yml
      create  config/puma.rb
      create  config/storage.yml
      create  config/environments
      create  config/environments/development.rb
      create  config/environments/production.rb
      create  config/environments/test.rb
      create  config/initializers
      create  config/initializers/assets.rb
      create  config/initializers/content_security_policy.rb
      create  config/initializers/cors.rb
      create  config/initializers/filter_parameter_logging.rb
      create  config/initializers/inflections.rb
      create  config/initializers/new_framework_defaults_8_0.rb
      create  config/locales
      create  config/locales/en.yml
      create  config/master.key
      append  .gitignore
      create  config/boot.rb
      create  config/database.yml
      create  db
      create  db/seeds.rb
      create  lib
      create  lib/tasks
      create  lib/tasks/.keep
      create  log
      create  log/.keep
      create  public
      create  public/400.html
      create  public/404.html
      create  public/406-unsupported-browser.html
      create  public/422.html
      create  public/500.html
      create  public/icon.png
      create  public/icon.svg
      create  public/robots.txt
      create  script
      create  script/.keep
      create  tmp
      create  tmp/.keep
      create  tmp/pids
      create  tmp/pids/.keep
      create  vendor
      create  vendor/.keep
      create  test/fixtures/files
      create  test/fixtures/files/.keep
      create  test/controllers
      create  test/controllers/.keep
      create  test/mailers
      create  test/mailers/.keep
      create  test/models
      create  test/models/.keep
      create  test/helpers
      create  test/helpers/.keep
      create  test/integration
      create  test/integration/.keep
      create  test/test_helper.rb
      create  test/system
      create  test/system/.keep
      create  test/application_system_test_case.rb
      create  storage
      create  storage/.keep
      create  tmp/storage
      create  tmp/storage/.keep
      remove  config/initializers/cors.rb
      remove  config/initializers/new_framework_defaults_8_0.rb
         run  bundle install --quiet
         run  bundle lock --add-platform=x86_64-linux
Writing lockfile to /Users/bradgessler/Desktop/demo/Gemfile.lock
         run  bundle lock --add-platform=aarch64-linux
Writing lockfile to /Users/bradgessler/Desktop/demo/Gemfile.lock
         run  bundle binstubs bundler
       rails  importmap:install
       apply  /Users/bradgessler/.rbenv/versions/3.4.1/lib/ruby/gems/3.4.0/gems/importmap-rails-2.1.0/lib/install/install.rb
  Add Importmap include tags in application layout
      insert    app/views/layouts/application.html.erb
  Create application.js module as entrypoint
      create    app/javascript/application.js
  Use vendor/javascript for downloaded pins
      create    vendor/javascript
      create    vendor/javascript/.keep
  Configure importmap paths in config/importmap.rb
      create    config/importmap.rb
  Copying binstub
      create    bin/importmap
         run  bundle install --quiet
       rails  turbo:install stimulus:install
       apply  /Users/bradgessler/.rbenv/versions/3.4.1/lib/ruby/gems/3.4.0/gems/turbo-rails-2.0.11/lib/install/turbo_with_importmap.rb
  Import Turbo
      append    app/javascript/application.js
  Pin Turbo
      append    config/importmap.rb
         run  bundle install --quiet
       apply  /Users/bradgessler/.rbenv/versions/3.4.1/lib/ruby/gems/3.4.0/gems/stimulus-rails-1.3.4/lib/install/stimulus_with_importmap.rb
  Create controllers directory
      create    app/javascript/controllers
      create    app/javascript/controllers/index.js
      create    app/javascript/controllers/application.js
      create    app/javascript/controllers/hello_controller.js
  Import Stimulus controllers
      append    app/javascript/application.js
  Pin Stimulus
  Appending: pin "@hotwired/stimulus", to: "stimulus.min.js"
      append    config/importmap.rb
  Appending: pin "@hotwired/stimulus-loading", to: "stimulus-loading.js"
      append    config/importmap.rb
  Pin all controllers
  Appending: pin_all_from "app/javascript/controllers", under: "controllers"
      append    config/importmap.rb
         run  bundle install --quiet
       rails  tailwindcss:install
       apply  /Users/bradgessler/.rbenv/versions/3.4.1/lib/ruby/gems/3.4.0/gems/tailwindcss-rails-3.3.1/lib/install/tailwindcss.rb
  Add Tailwindcss include tags and container element in application layout
      insert    app/views/layouts/application.html.erb
      insert    app/views/layouts/application.html.erb
      insert    app/views/layouts/application.html.erb
  Build into app/assets/builds
      create    app/assets/builds
      create    app/assets/builds/.keep
      append    .gitignore
  Add default config/tailwindcss.config.js
      create    config/tailwind.config.js
  Add default app/assets/stylesheets/application.tailwind.css
      create    app/assets/stylesheets/application.tailwind.css
  Add default Procfile.dev
      create    Procfile.dev
  Ensure foreman is installed
         run    gem install foreman from "."
Fetching foreman-0.88.1.gem
Successfully installed foreman-0.88.1
Parsing documentation for foreman-0.88.1
Installing ri documentation for foreman-0.88.1
Done installing documentation for foreman after 0 seconds
Parsing documentation for foreman-0.88.1
Done installing documentation for foreman after 0 seconds
1 gem installed
  Add bin/dev to start foreman
       force    bin/dev
  Compile initial Tailwind build
         run    rails tailwindcss:build from "."

Rebuilding...

Done in 142ms.
         run  bundle install --quiet
         run  bundle binstubs kamal
         run  bundle exec kamal init
Created configuration file in config/deploy.yml
Created .kamal/secrets file
Created sample hooks in .kamal/hooks
       force  .kamal/secrets
       force  config/deploy.yml
       rails  solid_cache:install solid_queue:install solid_cable:install
      create  config/cache.yml
      create  db/cache_schema.rb
        gsub  config/environments/production.rb
      create  config/queue.yml
      create  config/recurring.yml
      create  db/queue_schema.rb
      create  bin/jobs
        gsub  config/environments/production.rb
      create  db/cable_schema.rb
       force  config/cable.yml
      invoke  active_record
      create    db/migrate/20250131213105_create_posts.rb
      create    app/models/post.rb
      invoke    test_unit
      create      test/models/post_test.rb
      create      test/fixtures/posts.yml
      invoke  resource_route
       route    resources :posts
      invoke  scaffold_controller
      create    app/controllers/posts_controller.rb
      invoke    tailwindcss
      create      app/views/posts
      create      app/views/posts/index.html.erb
      create      app/views/posts/edit.html.erb
      create      app/views/posts/show.html.erb
      create      app/views/posts/new.html.erb
      create      app/views/posts/_form.html.erb
      create      app/views/posts/_post.html.erb
      invoke    resource_route
      invoke    test_unit
      create      test/controllers/posts_controller_test.rb
      create      test/system/posts_test.rb
      invoke    helper
      create      app/helpers/posts_helper.rb
      invoke      test_unit
      invoke    jbuilder
      create      app/views/posts/index.json.jbuilder
      create      app/views/posts/show.json.jbuilder
      create      app/views/posts/_post.json.jbuilder
== 20250131213105 CreatePosts: migrating ======================================
-- create_table(:posts)
   -> 0.0018s
== 20250131213105 CreatePosts: migrated (0.0018s) =============================

Scanning source code
Detected a Rails app
Creating app in /Users/bradgessler/Desktop/demo
We're about to launch your Rails app on Fly.io. Here's what you're getting:

Organization: Brad Gessler               (fly launch defaults to the personal org)
Name:         demo-bitter-snowflake-6071 (generated)
Region:       San Jose, California (US)  (this is the fastest region for you)
App Machines: shared-cpu-1x, 1GB RAM     (most apps need about 1GB of RAM)
Postgres:     <none>                     (not requested)
Redis:        <none>                     (not requested)
Tigris:       private bucket             (determined from app source)

? Do you want to tweak these settings before proceeding? No
Created app 'demo-bitter-snowflake-6071' in organization 'personal'
Admin URL: https://fly.io/apps/demo-bitter-snowflake-6071
Hostname: demo-bitter-snowflake-6071.fly.dev
Set secrets on demo-bitter-snowflake-6071: RAILS_MASTER_KEY
Your Tigris project (wild-pine-2096) is ready. See details and next steps with: https://fly.io/docs/reference/tigris/

Setting the following secrets on demo-bitter-snowflake-6071:
AWS_ACCESS_KEY_ID: tid_UMXhdUWTTWDmqcXSoDlCuVcQiFzCllOSpHRMgiRrTVmANeRfec
AWS_ENDPOINT_URL_S3: https://fly.storage.tigris.dev
AWS_REGION: auto
AWS_SECRET_ACCESS_KEY: tsec_17cqXHOZGf+N5Gf27SYREDqLtL5Pw2rzBktob54wMG6c3MiyKfv1S2ug6GvPnFeClF0Cav
BUCKET_NAME: wild-pine-2096

Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Running: bin/rails generate dockerfile --label=fly_launch_runtime:rails --skip --tigris --litestream
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
        skip  Dockerfile
   identical  .dockerignore
        skip  bin/docker-entrypoint
       force  config/database.yml
      create  lib/tasks/litestream.rake
      create  config/dockerfile.yml
     updated  config/storage.yml
     updated  config/environments/production.rb

The following packages are missing from the Dockerfile: libyaml-dev

Wrote config file fly.toml
Validating /Users/bradgessler/Desktop/demo/fly.toml
✓ Configuration is valid

Correct the errors in your Dockerfile and run 'fly deploy' to
deploy your Rails app.

The following comand can be used to update your Dockerfile:

    bin/rails generate dockerfile

Desktop/demo [main] → bin/rails generate dockerfile
    conflict  Dockerfile
Overwrite /Users/bradgessler/Desktop/demo/Dockerfile? (enter "h" for help) [Ynaqdhm]
       force  Dockerfile
   identical  .dockerignore
    conflict  bin/docker-entrypoint
Overwrite /Users/bradgessler/Desktop/demo/bin/docker-entrypoint? (enter "h" for help) [Ynaqdhm]
       force  bin/docker-entrypoint
   identical  config/database.yml
   identical  lib/tasks/litestream.rake
    conflict  fly.toml
Overwrite /Users/bradgessler/Desktop/demo/fly.toml? (enter "h" for help) [Ynaqdhm]
       force  fly.toml
   identical  config/dockerfile.yml
   unchanged  config/storage.yml
   unchanged  config/environments/production.yml
Desktop/demo [main] → fly deploy
==> Verifying app config
Validating /Users/bradgessler/Desktop/demo/fly.toml
✓ Configuration is valid
--> Verified app config
WARN DATABASE_URL may be a potentially sensitive environment variable. Consider setting it as a secret, and removing it from the [env] section: https://fly.io/docs/apps/secrets/

==> Building image
==> Building image with Depot
--> build:
[+] Building 126.8s (20/21)
 => [internal] load build definition from Dockerfile                                                                                                         0.1s
 => => transferring dockerfile: 2.65kB                                                                                                                       0.1s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1                                                                                    0.8s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:93bfd3b68c109427185cd78b4779fc82b484b0b7618e36d0f104d4d801e66d25                              0.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:93bfd3b68c109427185cd78b4779fc82b484b0b7618e36d0f104d4d801e66d25                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                         0.0s
 => => Deduplicating step ID [internal] load build definition from Dockerfile, another build is calculating it                                               0.0s
 => [internal] load metadata for docker.io/library/ruby:3.4.1-slim                                                                                           1.1s
 => [internal] load .dockerignore                                                                                                                            0.1s
 => => transferring context: 916B                                                                                                                            0.1s
 => [base 1/4] FROM docker.io/library/ruby:3.4.1-slim@sha256:5007f131f506bdd87b73e0169999ebf7241f239909a7542634a8fecb33c55b4c                                2.3s
 => => resolve docker.io/library/ruby:3.4.1-slim@sha256:5007f131f506bdd87b73e0169999ebf7241f239909a7542634a8fecb33c55b4c                                     0.0s
 => => sha256:4acc32925b54cced057b69d61efab1b62118e11dfaf8258d04a60fe58d32464e 144B / 144B                                                                   0.1s
 => => sha256:1f33298c80b7e0ba396e3329714b07cee6d161b3dbd9c05530d562284dc6acdc 190B / 190B                                                                   0.1s
 => => sha256:36d5af13c6f28b26247cc36ea9bf498ef9699b4cd9d279a02e0b3efb40c48ec5 51.63MB / 51.63MB                                                             0.4s
 => => sha256:70eaa936dd769fd0f0c9129a887bb199b96a1e0f527edf5ffea9bfab4774c874 3.50MB / 3.50MB                                                               0.2s
 => => extracting sha256:70eaa936dd769fd0f0c9129a887bb199b96a1e0f527edf5ffea9bfab4774c874                                                                    0.2s
 => => extracting sha256:1f33298c80b7e0ba396e3329714b07cee6d161b3dbd9c05530d562284dc6acdc                                                                    0.1s
 => => extracting sha256:36d5af13c6f28b26247cc36ea9bf498ef9699b4cd9d279a02e0b3efb40c48ec5                                                                    1.8s
 => => extracting sha256:4acc32925b54cced057b69d61efab1b62118e11dfaf8258d04a60fe58d32464e                                                                    0.0s
 => [internal] load build context                                                                                                                          124.5s
 => => transferring context: 115.41kB                                                                                                                        0.3s
 => [base 2/4] WORKDIR /rails                                                                                                                                0.3s
 => [base 3/4] RUN gem update --system --no-document &&     gem install -N bundler                                                                          24.0s
 => [base 4/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl libjemalloc2 sqlite3 &&     rm -rf /var/lib/apt/lists /var/cac  6.2s
 => [build 1/6] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential libyaml-dev pkg-config &&     rm -rf /var/lib/apt  12.2s
 => [build 2/6] COPY Gemfile Gemfile.lock ./                                                                                                                 0.1s
 => [build 3/6] RUN bundle install &&     rm -rf ~/.bundle/ "/usr/local/bundle"/ruby/*/cache "/usr/local/bundle"/ruby/*/bundler/gems/*/.git &&     bundle   50.1s
 => [build 4/6] COPY . .                                                                                                                                     0.2s
 => [build 5/6] RUN bundle exec bootsnap precompile app/ lib/                                                                                                0.8s
 => [build 6/6] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile                                                                                    4.6s
 => [stage-2 1/3] COPY --from=build /usr/local/bundle /usr/local/bundle                                                                                      2.3s
 => [stage-2 2/3] COPY --from=build /rails /rails                                                                                                            1.4s
 => [stage-2 3/3] RUN groupadd --system --gid 1000 rails &&     useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash &&     mkdir /data &&    3.1s
 => exporting to image                                                                                                                                      14.9s
 => => exporting layers                                                                                                                                      1.3s
 => => exporting manifest sha256:c39db154f5d37ec2a070fb90ce8dbfa2a16fa7c423984e8a9ff04a0d898af6b1                                                            0.0s
 => => exporting config sha256:7538e3902d1942752ad591f3a53a34109781d3287eee49c4f1eb6d7331f57352                                                              0.0s
 => => pushing layers for registry.fly.io/demo-bitter-snowflake-6071:deployment-01JJZ4DQKMMT4J08PQ744QXBN3@sha256:c39db154f5d37ec2a070fb90ce8dbfa2a16fa7c4  11.9s
 => => pushing layer sha256:ecf5b416567dac8500dcf7183642b2891af9cbbbcd85fe5914b8d69b969e5bd6                                                                 8.7s
 => => pushing layer sha256:1cbe2cca39dd2ce29517f4eca4b6802375f36d99a3a6893f0b7f8306bbaa396e                                                                11.9s
 => => pushing layer sha256:4838a827e376f02aadd15bf7f353701af0e305b9b0a7e866a6c874373fb7c2f6                                                                 6.4s
 => => pushing layer sha256:e24016ad958597f6a5938cde438a790a69bca20cf689bd568ac34c18f0715ca2                                                                 9.4s
 => => pushing layer sha256:7538e3902d1942752ad591f3a53a34109781d3287eee49c4f1eb6d7331f57352                                                                 5.7s
 => => pushing layer sha256:af302e5c37e9dc1dbe2eadc8f5059d82a914066b541b0d1a6daa91d0cc55057d                                                                 1.8s
 => => pushing layer sha256:70eaa936dd769fd0f0c9129a887bb199b96a1e0f527edf5ffea9bfab4774c874                                                                 3.9s
 => => pushing layer sha256:1f33298c80b7e0ba396e3329714b07cee6d161b3dbd9c05530d562284dc6acdc                                                                 3.8s
 => => pushing layer sha256:36d5af13c6f28b26247cc36ea9bf498ef9699b4cd9d279a02e0b3efb40c48ec5                                                                 7.5s
 => => pushing layer sha256:4acc32925b54cced057b69d61efab1b62118e11dfaf8258d04a60fe58d32464e                                                                 3.6s
 => => pushing layer sha256:06a1759820837db255a7087d0cbd0d9855cc3ede6f601f0c0f4f9e4982d2a8f1                                                                 5.4s
 => => pushing layer sha256:b7a336aa6b03a390e9148029366ddb494bf30028b9b195a30c0da109838aaa0c                                                                 5.3s
 => => pushing manifest for registry.fly.io/demo-bitter-snowflake-6071:deployment-01JJZ4DQKMMT4J08PQ744QXBN3@sha256:c39db154f5d37ec2a070fb90ce8dbfa2a16fa7c  1.7s
--> Build Summary:
--> Building image done
image: registry.fly.io/demo-bitter-snowflake-6071:deployment-01JJZ4DQKMMT4J08PQ744QXBN3
image size: 197 MB

Watch your deployment at https://fly.io/apps/demo-bitter-snowflake-6071/monitoring

Provisioning ips for demo-bitter-snowflake-6071
  Dedicated ipv6: 2a09:8280:1::62:bd82:0
  Shared ipv4: 66.241.124.17
  Add a dedicated ipv4 with: fly ips allocate-v4

Creating a 1 GB volume named 'data' for process group 'app'. Use 'fly vol extend' to increase its size
This deployment will:
 * create 1 "app" machine

No machines in group app, launching a new machine
WARN failed to release lease for machine 3d8dd144b45548 [app]: lease not found

-------
 ✖ Failed: timeout reached waiting for health checks to pass for machine 3d8dd144b45548: failed to get VM 3d8dd144b45548: Get "https://api.machines.dev/v1/apps…
-------
Error: timeout reached waiting for health checks to pass for machine 3d8dd144b45548: failed to get VM 3d8dd144b45548: Get "https://api.machines.dev/v1/apps/demo-bitter-snowflake-6071/machines/3d8dd144b45548": net/http: request canceled
Desktop/demo [main] →

I tried this with another SQLite app today for a demo video and it also failed to launch/deploy.

4

Do the logs show anything interesting?

5

No, there’s no log because the app never deploys.

6

I tried this again today and it’s still broken; however I was able to get some logs:

Desktop/demo [main] → fly logs
WARN The running flyctl agent (v0.3.72) is older than the current flyctl (v0.3.74).
WARN The out-of-date agent will be shut down along with existing wireguard connections. The new agent will start automatically as needed.
2025-02-05T00:01:47Z runner[7811093f59e148] sjc [info]Pulling container image registry.fly.io/demo-bitter-butterfly-361:deployment-01JK9PENZE2W8F2410KETNQW2W
2025-02-05T00:01:57Z runner[7811093f59e148] sjc [info]Successfully prepared image registry.fly.io/demo-bitter-butterfly-361:deployment-01JK9PENZE2W8F2410KETNQW2W (9.662814673s)
2025-02-05T00:01:57Z runner[7811093f59e148] sjc [info]Setting up volume 'data'
2025-02-05T00:01:57Z runner[7811093f59e148] sjc [info]Uninitialized volume 'data', initializing...
2025-02-05T00:01:57Z runner[7811093f59e148] sjc [info]Encrypting volume
2025-02-05T00:02:02Z runner[7811093f59e148] sjc [info]Opening encrypted volume
2025-02-05T00:02:04Z runner[7811093f59e148] sjc [info]Formatting volume
2025-02-05T00:02:06Z runner[7811093f59e148] sjc [info]Configuring firecracker
2025-02-05T00:02:06Z health[7811093f59e148] sjc [warn]Health check on port 8080 is in a 'warning' state. Your app may not be responding properly. Services exposed on ports [80, 443] may have intermittent failures until the health check passes.
2025-02-05T00:02:06Z app[7811093f59e148] sjc [info]2025-02-05T00:02:06.569380628 [01JK9PKD4CJXKJ9899J7A9FZKE:main] Running Firecracker v1.7.0
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO Starting init (commit: 676c82a4)...
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO Checking filesystem on /data
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info]/dev/vdc: clean, 11/64512 files, 8785/258048 blocks
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO Mounting /dev/vdc at /data w/ uid: 1000, gid: 1000 and chmod 0755
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO Resized /data to 1056964608 bytes
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO Preparing to run: `/rails/bin/docker-entrypoint ./bin/rake litestream:run ./bin/rails server` as 1000
2025-02-05T00:02:07Z app[7811093f59e148] sjc [info] INFO [fly api proxy] listening at /.fly/api
2025-02-05T00:02:07Z runner[7811093f59e148] sjc [info]Machine created and started in 19.975s
2025-02-05T00:02:08Z app[7811093f59e148] sjc [info]2025/02/05 00:02:08 INFO SSH listening listen_address=[fdaa:0:793f:a7b:16d:309d:1001:2]:22 dns_server=[fdaa::3]:53
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info]2025/02/05 00:02:10 ERROR failed to run error="config file not found: /rails/tmp/litestream.yml"
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info] INFO Main child exited normally with code: 1
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info] INFO Starting clean up.
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info] INFO Umounting /dev/vdc from /data
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-02-05T00:02:10Z app[7811093f59e148] sjc [info][    4.087932] reboot: Restarting system
2025-02-05T00:02:13Z app[7811093f59e148] sjc [info]2025-02-05T00:02:13.542643652 [01JK9PKD4CJXKJ9899J7A9FZKE:main] Running Firecracker v1.7.0
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO Starting init (commit: 676c82a4)...
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO Checking filesystem on /data
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info]/dev/vdc: clean, 11/64512 files, 8785/258048 blocks
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO Mounting /dev/vdc at /data w/ uid: 1000, gid: 1000 and chmod 0755
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO Resized /data to 1056964608 bytes
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO Preparing to run: `/rails/bin/docker-entrypoint ./bin/rake litestream:run ./bin/rails server` as 1000
2025-02-05T00:02:14Z app[7811093f59e148] sjc [info] INFO [fly api proxy] listening at /.fly/api
2025-02-05T00:02:14Z runner[7811093f59e148] sjc [info]Machine started in 1.25s
2025-02-05T00:02:15Z app[7811093f59e148] sjc [info]2025/02/05 00:02:15 INFO SSH listening listen_address=[fdaa:0:793f:a7b:16d:309d:1001:2]:22 dns_server=[fdaa::3]:53
2025-02-05T00:02:16Z health[7811093f59e148] sjc [error]Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info]2025/02/05 00:02:17 ERROR failed to run error="config file not found: /rails/tmp/litestream.yml"
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info] INFO Main child exited normally with code: 1
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info] INFO Starting clean up.
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info] INFO Umounting /dev/vdc from /data
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info] WARN could not unmount /rootfs: EINVAL: Invalid argument
2025-02-05T00:02:17Z app[7811093f59e148] sjc [info][    4.123091] reboot: Restarting system
2025-02-05T00:02:20Z app[7811093f59e148] sjc [info]2025-02-05T00:02:20.472114649 [01JK9PKD4CJXKJ9899J7A9FZKE:main] Running Firecracker v1.7.0
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO Starting init (commit: 676c82a4)...
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO Checking filesystem on /data
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info]/dev/vdc: clean, 11/64512 files, 8785/258048 blocks
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO Mounting /dev/vdc at /data w/ uid: 1000, gid: 1000 and chmod 0755
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO Resized /data to 1056964608 bytes
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO Preparing to run: `/rails/bin/docker-entrypoint ./bin/rake litestream:run ./bin/rails server` as 1000
2025-02-05T00:02:21Z app[7811093f59e148] sjc [info] INFO [fly api proxy] listening at /.fly/api
2025-02-05T00:02:21Z runner[7811093f59e148] sjc [info]Machine started in 1.235s

The litestream.yml file is missing.

7

Make the following change:

--- a/lib/tasks/litestream.rake
+++ b/lib/tasks/litestream.rake
@@ -42,7 +42,7 @@ namespace :litestream do
     end
   end
 
-  task :run do
+  task :run => :prepare do
     require "shellwords"
 
     exec(*%w[bundle exec litestream replicate -config],

Note: at this point you may have a zero length sqlite3 database on your volume, and this will prevent db:prepare from being run. I’ll investigate tomorrow why the => :prepare is missing - it should be present in sqlite3 is detected and omitted if postgres is omitted as the prepare would be done on a deploy release machine.

8

This should now be fixed. As an added bonus, the change to the Ruby docker images removing libyaml has been reverted so once again fly launch will just work with the Rails provided Dockerfiles.

9

I was going to mention the libyaml issue next, but you beat me to it. I’ll give it a go within the next few days and report back if I run into issues.

Is the fly console issue above filed under wontfix or are there plans to address that?

10

Quick synopsis on the libyaml issue… Rails generated Dockerfiles didn’t include it, but depend on it. Ruby dockerhub images included it, but shouldn’t have, and stopped. This broke Rails Dockerfiles from being deployed (not just to fly.io, but everywhere). For now, the Ruby folks have reverted their change, and the Rails folks have updated their Dockerfile on main, but haven’t released the fix.

As to fly console, I’m not clear on what the issue is. Yes, fly console will create a new machine, and no, new machines won’t have access to volumes in production. What they will have, however, is the ability to restore a current snapshot from Tigris to play with. The rake task is currently named litestream:prepare, but I’m interested in your thoughts. In any case, the following will get you up and running:

bin/rails litestream:prepare

If you want to access the database in production, fly ssh console will do that for you.

11

Yeah, my point is the DX is “broken” from the PoV of a person deploying a SQLite database to production. The expectation is that fly console brings up bin/rails c in production and there’s a database there. As we know, in practice, that doesn’t happen.

12

I’m still confused. What I’m hearing is that a screwdriver is not a good tool for hammering in a nail. I’m suggesting a hammer instead.

Fly.io does not have an innate concept of an SQLite application.

I can describe the commands you can use to ssh into an existing machine and run rails console there. Or I can describe the commands to create a new machine, restore a snapshot of a database, and then run rails console against that snapshot.

What I can’t do is change the command that was explicitly designed to create a new machine to not do that.

13

I’m filing under “won’t fix” :rofl:

I get it, it’s just super hacky and will surprise people who don’t know about running fly ssh console -C "/bin/rails c" to bring up a sane console when they deploy to Fly.

I proposed the fly exec $TASK in flyctl as a way to make a SQLite deployment sane and more customizable (e.g. fly exec rake db:migrate, fly exec rails console), but I’ve never seen that go anywhere either. Said tasks would be configured in fly.toml, including if they’re run on another machine or the same machine.

14

lib/tasks/fly.rake:

namespace :fly do
  task :console do
    sh 'fly ssh console --pty -C "bin/rails console"'
  end
end
15

Yeah, I get how those scripts can be dropped into a project. I went with the ./bin/production executable:

#!/usr/bin/env sh
fly ssh console --pty -C "/rails/bin/rails console"

My point is Fly’s DX could reach “built for SQLite” levels if this stuff works out of the box from flyctl, but I don’t think that’s a priority.