Ruby on Rails App authentication works locally but fails after deployment

Need help,
I deployed my application to Fly.io and generated API documentation with Rswag. When I test the endpoints on localhost:3000, the user authentication and endpoints are functioning correctly. However, after deployment, the user authentication is failing. Although the user is able to create an account and log in, the authorization token generated during account creation is saved in the database but is still resulting in an unauthorized error even when passed as an API key in the headers.

This is the logs from the server

This is the request from remote host

This is the response from localhost

When you run your localhost, are you running with RAILS_ENV=production? Looking at the logs, I suspect that the problem isn’t localhost vs fly, but rather development vs production.

@rubys please are you talking about the docker file. In the dockerfile RAILS_ENV=production

It would be helpful to know if you can run your application in production mode on your development machine. If you can’t run in production mode on your machine, you likely won’t be able to run in production mode on fly.io.

On your development machine, try the following commands:

export RAILS_ENV=production
ruby bin/rails assets:precompile
ruby bin/rails db:prepare
ruby bin/rails server

If you are running on Windows with Powershell, substitute set for export.

@rubys I have followed the steps you suggested and it’s working perfectly without errors on the localhost. I have deployed it again but it’s still not working on the remote host.

If you can share a link to your codebase, if it’s open-source, that would be helpful. The JWT in your example of a failing request decodes as follows:

"{\"id\":4,\"exp\":1681219�43}"

whereas the JWT in your example of a successful local request decodes as follows:

{"id": 2, "exp": 1681203148}

There’s a difference here in the decoded results: In the former, the JSON has been double encoded, which is likely to cause the unexpected behavior you’ve observed with failing authentication. The token you’re sending to the API is invalid.

I am not sure why this would happen when deployed vs. when running locally, as I would typically expect double encoding like this to be caused by an error in the code – e.g: '{"x": "y"}'.to_json – so it would consistently fail across environments.

So, you need to track down why the input is double encoded when the application is deployed vs. when it’s running locally. A good starting point would be to consider whether you’re actually running the production build locally: when you run Fly deploy, Fly will build a container, and it is that container that Fly is running, not your Ruby on Rails application. If you haven’t tested the container locally, do that first: let Fly do a build, then do something like:

fly image show

Then from the result, extract the image in the format registry.fly.io/repository:tag. So, using the following example:

Image Details
  Registry   = registry.fly.io
  Repository = example
  Tag        = deployment-01GRMJ0NCKBPTVZPZ8QKSXT4DD
  Version    = N/A
  Digest     = sha256:146f5737def0242785cc2a2b1703ebb28e3486bd0e70e619d634d122aec6388b

The image would be:

registry.fly.io/example:deployment-01GRMJ0NCKBPTVZPZ8QKSXT4DD

Next, copy all of your Environment Variables from fly.toml into a file called fly.env and then run the image like so (where 8080 is a best guess at the port you’re using – swap it out if neccessary):

docker run -p 8080:8080 --env-file fly.env registry.fly.io/example:deployment-01GRMJ0NCKBPTVZPZ8QKSXT4DD

Now do your tests again, and see if you see the same behavior that you’re seeing on the Fly deployment. If you are able to reproduce it, you know there’s an issue with how your application is behaving when containerised.

Also, you can decode a JWT yourself using jwt.io. A great first step when debugging authentication issues is verifying that your JWT is exactly as you expect, because the encoded nature of a JWT can easily obscure simple issues that you’d catch easily otherwise :slight_smile:

(Also, just for future reference, when you share screenshots when asking for programming/software/infrastructure help, you should include a text version, because often the specifics of the error message are very important for people to copy and paste. I had to retype both JWTs out by hand in order to decode them!)

@shrink I’m looking at my code again but this is the link to the code on github

@shrink this is the results following the steps

(I wrote this comment, then changed my mind about the diagnosis, so I deleted it – ignore the edit history!)

I can’t figure out why this would happen, but it’s worth noting that you’re misusing JWTs. The purpose of a JWT is to encode information in a token that the client can use to interact with a stateless service: storing JWTs in the database is an anti-pattern.

That said, on to the actual issue:

  def generate_jwt
    JWT.encode({ id:, exp: 60.days.from_now.to_i }, Rails.application.secrets.secret_key_base)
  end

The issue seems to be that when this token is created, { id:, exp: 60.days.from_now.to_i } is encoded twice. I can’t identify why this would happen only when deployed to Fly – unless, perhaps, you haven’t created a secret for the value of SECRET_KEY_BASE, and that’s causing weird behaviour. Make sure you’ve created a SECRET_KEY_BASE secret for your fly app to rule that possibility out.

If the secret isn’t the issue, the next step would be to debug this by adding code to output the JWT when it’s created here:

Try to identify why generate_jwt is double encoding the input.

If you have docker installed locally, you can try the following:

bin/rails generate dockerfile --compose
export RAILS_MASTER_KEY=$(cat config/master.key)
docker compose build
docker compose up

This will run your dockerfile locally. Again, if you are running on Microsoft windows, adjust the commands as needed.