My app just stopped working. It seems this has happened a few days ago and I was wondering if this is the same issue happening again.
Nothing has changed on the application side and I’ve tried all the suggestions that I could find here in other topics and couldn’t resolve this.
Any help, please?
Here is the fly.toml file:
# fly.toml file generated for mailpouch on 2023-03-03T17:39:39-03:00
app = "mailpouch"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
[experimental]
auto_rollback = true
[[services]]
http_checks = []
internal_port = 3000
processes = ["web"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
[[statics]]
guest_path = "/rails/public"
url_prefix = "/"
[processes]
web = "bundle exec puma -C config/puma.rb"
worker = "bundle exec sidekiq -e production -C config/sidekiq.yml"
[deploy]
release_command = "bin/rails db:migrate"
And Dockerfile:
# syntax = docker/dockerfile:1
# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.1.3
FROM ruby:$RUBY_VERSION-slim as base
LABEL fly_launch_runtime="rails"
# Rails app lives here
WORKDIR /rails
# Set production environment
ENV RAILS_ENV="production" \
BUNDLE_WITHOUT="development:test" \
BUNDLE_DEPLOYMENT="1"
# Update gems and bundler
RUN gem update --system --no-document && \
gem install -N bundler
# Throw-away build stage to reduce size of final image
FROM base as build
# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y build-essential curl libpq-dev node-gyp pkg-config python-is-python3
# Install JavaScript dependencies
ARG NODE_VERSION=19.7.0
ARG YARN_VERSION=1.22.19
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
/tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
npm install -g yarn@$YARN_VERSION && \
rm -rf /tmp/node-build-master
# Install application gems
COPY Gemfile Gemfile.lock .
RUN bundle install && \
bundle exec bootsnap precompile --gemfile
# Install node modules
COPY package.json yarn.lock .
RUN yarn install --frozen-lockfile
# Copy application code
COPY . .
# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/
# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile
# Final stage for app image
FROM base
# Install packages needed for deployment
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y postgresql-client && \
rm -rf /var/lib/apt/lists /var/cache/apt/archives
# Run and own the application files as a non-root user for security
RUN useradd rails
USER rails:rails
# Copy built artifacts: gems, application
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build --chown=rails:rails /rails /rails
# Deployment options
ENV RAILS_LOG_TO_STDOUT="1" \
RAILS_SERVE_STATIC_FILES="true"
# Entrypoint sets up the container.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]
# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/rails", "server"]
They are both stock from Rails / Fly tutorials so I would imagine they are good.
Hey there, looks like it might be related to the current upgrade we’ve been doing. It should’ve been smoother but some things went slightly wrong causing issues with new deployment.
Yours should be fixed now.
And it seems to be working now. This is major turn off for me. I just migrated from Heroku a few days ago 
Thank you for the quick reply!
Do you have an ETA on time to fix? Or can you at least share an understand of how to know which apps could be affected?
We have a few apps in our org that are experiencing this issue, which is bringing our site down. Extremely disappointing to get paged for this on a Saturday morning.
If it helps you debug, we’re seeing this behavior for our app offsyte-client-production without having triggered a new deploy.
I’m getting hit by a similar issue with my apps!
getting the same thing here Mar 4 at 12:15pm EST
It seems the rollback has fixed my issue. Thank you!
Still seeing this on LAX.
Why does fly do upgrades on weekends in the middle of the day?
Still seeing this in SEA region as well, but I’m sure that’s expected. Will check back on my project in a few hours.
Why does fly do upgrades on weekends in the middle of the day?
I unfortunately have to agree. I’m using Fly for personal projects primarily right now as a test for a larger Heroku migration for my employer, and unfortunately outages like this, as much as I 
fly and the fly.io team, make it pretty hard for me to recommend this for production usage for small teams.
I agree too.
I’m using fly.io by a few years for some production applications in my main job. Because it’s easier to deploy, maintain and configure than EKS/ECS and edge computing concept is pretty simples when your team does not have and SRE/DEVOPS squad dedicated to take care about infraestructure details.
So, that’s not the first issue that fly.io facing because some “updates”. And they should warn us by email, dashbard, wathever about this incoming updates.
I will release a hight traffic plataform for selling tickets in this monday, if they didn’t fix that and do not review about this situations, I think i must to get out here, sadly.
I just did a deploy that was impacted by this as well. 3 of my last 3 deploys suffered some kind of issue, which cements my view that fly is only suitable for personal projects where constant outages aren’t a big deal.
My app in the dfw region is still down.
Logs of the following in my logs:
2023-03-04T18:41:05Z proxy[93dcba29] lax [warn]Could not proxy HTTP request. Retrying in 960 ms (attempt 40)
I know folks are working on it 
I love fly, but it is pretty brutal to deploy some bug fixes to an app, just to have the deployment completely kill the app.
Moving regions doesn’t seem to fix anything.
This is incredibly concerning. Just rolled out two bug fixes to apps in FRA, just to be taken down by this. I do love lots the provided features, but I am deeply concerned that running real sites with Fly is a bit too risky… Only using the platform for a few months and had hours of downtime due to platform issues. Not great. At all.
I just moved from SEA To LAX and getting responses again. Monitoring for stability
Think I’ve just been hit by same problem. Did a deploy (trivial change) and now site won’t load at all. Nothing in logs. This is in LHR. Connection timeout error from browser.
I wish I’d checked the Fly status page before I deployed. Luckily this is just my little side project with basically zero traffic so no harm done. Glad I don’t have anything meaningful on here. It’s a real shame as everything else about Fly is so good.