Deploying Phoenix with the Bandit adapter on Fly.io

Phoenix
1

Wanted to put this out into the world in case someone runs into a similar scenario down-the-way.

I recently deployed a bit of Phoenix 1.7.0 boilerplate, and decided to replace the default HTTP server (Cowboy) with Bandit, just to see how that was.

… and it was pretty great! Everything seemed to work fine; except, when I went and checked the logs, I saw errors, at fifteen second intervals (there’s your clue!) in this form:

2022-12-12T13:15:00.000 app[abcdefgh] iad [info] 13:15:00.000 [error] GenServer #PID<0.1000.0> terminating
2022-12-12T13:15:00.000 app[abcdefgh] iad [info] ** (stop) "Could not determine a protocol"
2022-12-12T13:15:00.000 app[abcdefgh] iad [info] Last message: {:continue, :handle_connection}
2022-12-12T13:15:15.000 app[abcdefgh] iad [info] 13:15:15.000 [error] GenServer #PID<0.1001.0> terminating
2022-12-12T13:15:15.000 app[abcdefgh] iad [info] ** (stop) "Could not determine a protocol"
2022-12-12T13:15:15.000 app[abcdefgh] iad [info] Last message: {:continue, :handle_connection}

It took me a bit to make heads-or-tails, but it turns out, this is the response that Bandit is giving to the TCP checks (15s interval!) that are generated as part of the default fly.toml, similar to what was relayed in this GitHub issue.

I replaced the [[services.tcp_checks]] block in the fly.toml with:

[[services.http_checks]]
  grace_period = "1s"
  interval = "15s"
  method = "get"
  path = "/"
  protocol = "http"
  restart_limit = 0
  timeout = "2s"

… which got rid of that error. However, I started getting this in the log instead:

13:30:00.000 [info] Plug.SSL is redirecting GET / to https://127.0.0.1 with status 301
13:30:15.000 [info] Plug.SSL is redirecting GET / to https://127.0.0.1 with status 301

If you’ve spent time messing about with Plug.SSL before, I think you can quickly figure out what I forgot here: I need to send appropriate headers for Plug.RewriteOn. I appended this after the timeout = "2s" line:

  [services.http_checks.headers]
    X-Forwarded-Proto = "https"

… and I think everything’s clean now!

1 Like
2

Glad to hear things went smoothly!

Unlike Cowboy, Bandit explicitly logs all errors, including cases where clients connect but never actually make an HTTP request (that’s the case you’re seeing here). I suspect that I’ll likely be making an exception to the ‘log all errors’ rule for this, as it comes up pretty frequently.

2 Likes
3

Hi, we are having a similar error, but it appears constantly.

We can’t find why this problem occurs, would you have any clue why this error may appear?

Locally we cannot replicate it.

Sep 14 09:53:25 lhr vector error 12:53:25.624 [error] GenServer #PID<0.11319.9> terminating
Sep 14 09:53:25 lhr vector error ** (stop) "WebSocket upgrade failed: error in upgrade_header check: \"Did not find 'websocket' in ''\""
Sep 14 09:53:25 lhr vector Last message: {:continue, :handle_connection}
Sep 14 09:53:25 lhr vector State: {%ThousandIsland.Socket{socket: #Port<0.56338>, transport_module: ThousandIsland.Transports.TCP, read_timeout: 60000, span: %ThousandIsland.Telemetry{span_name: :connection, telemetry_span_context: #Reference<0.3332894219.134479873.61141>, start_time: -576392056899247330, start_metadata: %{parent_telemetry_span_context: #Reference<0.3332894219.114294786.27704>, remote_address: {172, 16, 189, 18}, remote_port: 39816, telemetry_span_context: #Reference<0.3332894219.134479873.61141>}}}, %{handler_module: Bandit.HTTP1.Handler, http_1_enabled: true, http_2_enabled: true, opts: %{http_1: [], http_2: [], websocket: []}, plug: {BrandkitWeb.Endpoint, []}, websocket_enabled: true}}
4

Hi @fedeotaran,

This is happening because your app is attempting to upgrade a non-web socket connection to a websocket (specifically, the handshake call is failing when trying to validate the upgrade header).

Assuming this is a Phoenix app, this is likely happening because you have an HTTP client hitting your websocket paths (as configured via socket directives in your Endpoint module).

Servers currently do not validate upgrade requests at the time they’re made; validation is deferred until after the originating c:Plug.call/2 call has completed. Improvements to this are tracked as Provide a mechanism to validate upgrade requests before issuing upgrades · Issue #11 · phoenixframework/websock_adapter · GitHub.

In the meantime, I’d suggest that you see if you potentially have a misconfigured link pointing to one of your socket endpoints as described above, or if you potentially have an automated health checker that may be hitting it as well.

2 Likes