Rails turbo_stream_from does not connect in fly.io environment

A view in my rails application subscribes to a turbo stream channel with <%= turbo_stream_from session["_csrf_token"] %>. (I use the csrf token as channel name because the stream is meant to be per session.)
In my controller, there is an action which triggers Turbo::StreamsChannel.broadcast_replace_to session["_csrf_token"], partial: "devise/registrations/creating", target: "registration_new_full_page". That action replaces the registration form with a loading view before redirecting, given the registration was successful.

In my local development environment, which I access with http://127.0.0.1:3000, everything works just fine. When loading the registration page, the logs show Turbo::StreamsChannel is streaming from TrJhUDOIuzZEKUER3tuC6qsNRsKovL6XEI63bKSuXrk which is what I’d expect.

Unfortunately, after deploying to fly that exact log line is not shown any more. The broadcasted update is not received and nothing happens. Also, the HTML of the registration page in the browser shows <turbo-cable-stream-source channel="Turbo::StreamsChannel" signed-stream-name="..." connected=""></turbo-cable-stream-source> for the local development environment, but on fly-deployed environment, that same line is missing connected="".

My nginx config contains this:

map $http_host $rails {
  myapp.com myapp-rails.flycast;
  myapp-staging.com myapp-rails-staging.flycast;
}

...

  location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Ssl on; # Optional
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Real-IP $remote_addr;

    proxy_redirect off;

    proxy_read_timeout 120;
    proxy_connect_timeout 120;
    proxy_send_timeout 120;

    proxy_pass http://$rails;
  }

and I don’t see any messages in the browser console that do not appear in the local development environment.

What am I missing here? Any idea what I could be doing wrong?

It is rare to see people using nginx as a front end to rails these days (it used to be quite common). And this is the first I’ve seen using nginx to proxy to another machine running rails.

I’m not sure this is going to help, but I’m used to seeing the proxying of websocket done separately:

location / {
    ...
}

location /cable {
    ...
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
}

I’ll close by saying that I use nginx with rails and action cable; but I do it with https://phusionpassenger.com with both nginx and rails in the same virtual machine.

Thanks for the idea. I have tried what you mention and separated the /cable location but that did not change anything.

I know about passenger :slight_smile: and I might use that at some point but for now proxying to puma should be just fine. I am proxying to rails from another machine because requests hitting my fly application first need to go through a gateway and that gateway is nginx. More precisely, it is an openresty doing some lua-magic for requests aiming at subdomains and simply proxying requests aiming at the main domain to the fly application running rails.

I still can’t see what is preventing rails on fly from connecting the turbo streams broadcast channels. Another difference I have noticed between development and fly environment is that, in the browser network tab in the dev tools, a new keepalive get request is sent every 10 seconds to wss://myapp-staging.com/cable returning a 101 http status and showing up as Successfully upgraded to WebSocket in the rails log. That looks ok to me, but I don’t see that in the local development environment.

You shouldn’t see that. You should connect once and stay connected. That looks more like a reconnect after a failure.

What does config/cable.yml look like? It should be configured for redis.

config/cable.yml is configured for redis:

development:
  adapter: redis
  url: "redis://localhost:6379/1"

test:
  adapter: test

production:
  adapter: redis
  url: "redis://username:password@myapp-redis.flycast:6379/1"
  channel_prefix: rails_production

staging:
  adapter: redis
  url: "redis://username:password@myapp-redis-staging.flycast:6379/1"
  channel_prefix: rails_staging

I have set up a redis fly application, not using upstash for now. The redis logs don’t show anything unusual either, only the regular background saving messages.

Other than my use of passenger (which I use to launch multiple instances of a rails application) and my starting redis, nginx, and rails in the same vm, our applications are a lot alike. I see nothing obviously wrong with your configuration, except mine works and yours doesn’t :frowning:

My source is here: GitHub - rubys/showcase: Ballroom Dance Showcase Scheduler

If you go to https://smooth.fly.dev/, then click on demo, then Publish, then click the QRcode twice you can see it connect to cable - once. There won’t be any interesting data on the page unless you enter it in the demo, but you can observe the websocket behavior using the developer tools in your browser.

Good call. I just took a look at redis-cli monitor.
Working local development environment:

1701707705.107923 [1 [::1]:49232] "SELECT" "1"
1701707705.107960 [1 [::1]:49232] "CLIENT" "SETNAME" "ActionCable-PID-30748"
1701707705.108319 [1 [::1]:49232] "subscribe" "_action_cable_internal"
1701707705.108541 [1 [::1]:49232] "subscribe" "TrJhUDOIuzZEKUER3tuC6qsNRsKovL6XEI63bKSuXrk"

But on my fly environment, it looks like cable is not able to connect to redis, because I only see the authentication:

1701707326.735865 [0 172.16.144.130:57534] "AUTH" "(redacted)" "(redacted)"

and that authentication is probably not coming from cable, but from the redis client which I set up per config/initializers/redis.rb:

REDIS = if Rails.env.development?
  Redis.new(path: "/run/redis/redis.sock")
else
  if Rails.env.production?
    Redis.new(host: Rails.application.credentials.redis.host.production, username: Rails.application.credentials.redis.username, password: Rails.application.credentials.redis.password)
  else
    Redis.new(host: Rails.application.credentials.redis.host.staging, username: Rails.application.credentials.redis.username, password: Rails.application.credentials.redis.password)
  end unless ENV["SECRET_KEY_BASE_DUMMY"] # Rails secrets are not available at (docker image) build time
end

Unfortunately, I don’t see any errors in the rails logs, just a bunch of these: (taken from the fly app monitoring page)



2023-12-04T16:56:39.636 app[5683623a45978e] waw [info] I, [2023-12-04T16:56:39.634807 #305] INFO -- : [caf78c44-0c8b-4a9f-a9e5-0f3cbb3018e4] Started GET "/cable" for 2a09:8280:1::42:a57b at 2023-12-04 16:56:39 +0000

2023-12-04T16:56:39.636 app[5683623a45978e] waw [info] I, [2023-12-04T16:56:39.636100 #305] INFO -- : [caf78c44-0c8b-4a9f-a9e5-0f3cbb3018e4] Started GET "/cable" [WebSocket] for 2a09:8280:1::42:a57b at 2023-12-04 16:56:39 +0000

2023-12-04T16:56:39.636 app[5683623a45978e] waw [info] I, [2023-12-04T16:56:39.636154 #305] INFO -- : [caf78c44-0c8b-4a9f-a9e5-0f3cbb3018e4] Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)

2023-12-04T16:56:39.658 app[5683623a45978e] waw [info] I, [2023-12-04T16:56:39.657770 #305] INFO -- : Finished "/cable" [WebSocket] for 2a09:8280:1::42:a57b at 2023-12-04 16:56:39 +0000

2023-12-04T16:57:07.436 app[5683623a45978e] waw [info] I, [2023-12-04T16:57:07.436575 #305] INFO -- : [c4d572bb-541f-4f78-96cd-8f4b67290931] Started GET "/cable" for 2a09:8280:1::42:a57b at 2023-12-04 16:57:07 +0000

2023-12-04T16:57:07.437 app[5683623a45978e] waw [info] I, [2023-12-04T16:57:07.437404 #305] INFO -- : [c4d572bb-541f-4f78-96cd-8f4b67290931] Started GET "/cable" [WebSocket] for 2a09:8280:1::42:a57b at 2023-12-04 16:57:07 +0000

2023-12-04T16:57:07.437 app[5683623a45978e] waw [info] I, [2023-12-04T16:57:07.437487 #305] INFO -- : [c4d572bb-541f-4f78-96cd-8f4b67290931] Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)

2023-12-04T16:57:07.459 app[5683623a45978e] waw [info] I, [2023-12-04T16:57:07.459096 #305] INFO -- : Finished "/cable" [WebSocket] for 2a09:8280:1::42:a57b at 2023-12-04 16:57:07 +0000 

Any idea on how to investigate why cable is not connecting to my redis app on fly?

I don’t think flycast was meant for this purpose. Try an internal address, perhaps?

P.S. Since you are using different channel_prefix for each environment, you can share a redis instance.

Ok, looks like I’ve got it working now.

While setting the internal address instead of the flycast one, I also further played around with the nginx config and before deploying the rails update with the internal address, I deployed the following nginx config change and it seems to work now.

location /cable {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Ssl on; # Optional
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";

    proxy_pass http://$rails;
  }

  location / {
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Ssl on; # Optional
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Real-IP $remote_addr;

    proxy_redirect off;

    proxy_read_timeout 120;
    proxy_connect_timeout 120;
    proxy_send_timeout 120;

    proxy_pass http://$rails;
  }

Apparently, proxy_http_version 1.1; was key to making the cable connection work and it was missing in my config.

Interestingly though, I can’t leave out the lines before proxy_http_version 1.1; because if I do, I see a lot of [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: myapp-rails-staging.flycast errors in the rails logs.

Hey @rubys , thank you very much for helping me out with this!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.