Fly + Remix + Safari - Safari doesn't work, Chrome does

Questions / Help
1

I’m migrating a deployment over from Lambda + API Gateway to Fly. I’ve got it all deployed to Fly just fine, but super weird - our sign-in page works from Chrome, but not Safari. Same code for that route works fine on the Lambda deployment on both Safari and Chrome, and it works fine locally, too. Safari’s console shows: Failed to load resource: cannot parse response. The request shows - for the status of the response, no response headers, nothing. It’s like the request just died. However, the logs in Fly show a 204 being returned.

Again - I can run the exact same code locally, and it works in both Safari and Chrome as expected, and a very similar variant (only differing in package shape to get it running on Lambda) also works as expected there.

As noted in the title, this is a Remix app. Anyone have any idea why this one specific case is acting up?

2

That’s interesting. Can you crack open Safari Dev Tools and have a look at what the network tab shows for that page or request? Do you know if a 204 response includes any body content from Remix?

3

No status code even - it appears to just drop. No response body is shown.

FWIW, I deployed the same exact thing to AWS App Runner, no source changes besides some Terraform additions to get it up there. Same Dockerfile and everything. Runs just fine there.

4

HTTP 204 is supposed to return no content. If you are returning content it may be safari is much more strict and completely rejecting the response.

EDIT: safari does indeed reject data in a 204 response
204 No Content - HTTP | MDN (mozilla.org)

Futhermore, AWS API Gateway and AWS App Runner may be automatically stripping the bodies from 204 responses so that may explain why you don’t have a problem with them.

5

The Safari Dev tools details for that request would be super helpful. I don’t doubt there’s something weird about that response that Safari doesn’t like, and it’s possible our load balancer is the cause, but I don’t know how to replicate it!

You can usually “copy as cURL” from devtools, if you can do that and post the curl here we can investigate this.

6

Screenshot, + cURL below. Included full test credentials to that page.

It may actually be a Remix issue, and it just so happens that other hosts (ie App Runner) is more forgiving. I found this, which hasn’t been released yet: fix: null for body to conform to standards (#743) · remix-run/remix@d2dab19 · GitHub. I tried deploying that patch, but that didn’t work. It could be that there are other issues like that needing to be fixed.

CleanShot 2021-12-12 at 14.12.33@2x
CleanShot 2021-12-12 at 14.12.33@2x1920×701 49.1 KB

curl 'https://lively-pine-64.fly.dev/sign-in?_data=routes%2Fsign-in' \
-X 'POST' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Referer: https://lively-pine-64.fly.dev/sign-in' \
-H 'Accept: */*' \
-H 'Origin: https://lively-pine-64.fly.dev' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15' \
--data 'email=ben%40practicalbyte.com&password=password'
7

Hm - In Chrome, it appears that the response has a response body of 204, with a content-length header set with a value of 0. I wonder if Safari doesn’t like that, due to this: RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

A server MUST NOT send a Content-Length header field in any response
with a status code of 1xx (Informational) or 204 (No Content).

I don’t see where the remix server runtime could possibly be adding that header. Is that being done by your load balancer/proxy?

CleanShot 2021-12-12 at 14.51.01@2x
CleanShot 2021-12-12 at 14.51.01@2x1068×694 132 KB

8

@Ben_Kraus I’ve done some analysis and it seems that fly.io is incorrectly closing the HTTP2 stream.

Both cURL and Node.js were unable to successfully perform HTTP2 POST requests to your API endpoint with the provided information.

With cURL, the following error is received:

curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

With Node.js, the following error is received:

Error [ERR_HTTP2_STREAM_ERROR]: Stream closed with error code NGHTTP2_PROTOCOL_ERROR

However, when the password supplied is changed to be the incorrect password, the request successfully completes with a 200 response.

It seems that Chrome may be more forgiving in it’s HTTP2 implementation while Safari, cURL, and Node.js are much more strict.

I also tested these requests using HTTP1.1 (not in Safari) and the issue does not appear so this is specific to HTTP2.

One last thing to note is that in a successful login response, you are returning the headers supplied by the client in the server response headers. You should not do this and this might be causing problems with fly’s HTTP2 proxy which could possibly explain the issue.

You should instead change your code to return the client headers with X-From-Client- prepended to all the header names so they don’t cause interference, but ultimately this should be avoided and only done on debug endpoints or when in debug mode.

Give that a go and see if it helps.

Cheers,
Stefan

1 Like
9

Good job debugging this, it’s almost definitely a null body issue. We should handle that better. I made an internal issue to track this, we’ll update this topic if we get anywhere on it.

10

So if both an empty string and a null body don’t work, any other workaround ideas?

11

This appears to be an issue with a library we use. Investigating what kind of fix we can use with it.

1 Like
12

I believe your app returns a content-length: 47 header with its 204 response. Which is pretty weird.

We’re not messing with the response. I assume AppRunner and Lambda fix that for you. Chrome is also probably fixing that? Not entirely sure.

I’ve confirmed this by querying your VM directly (on port 8080). If you try the curl with --http1.1, you also get the actual response with the content-length: 47 header.

13

Strange. I have no idea how that’s happening - it has to be coming from Remix, or the underlying Node process.

14

That would be my guess, but I couldn’t find where exactly.

Can you run your application locally and confirm this behavior?

15

Yup, I’m seeing Content-Length = 47 when running locally.

16

OOOHHH! Interesting!

The request’s content length was 47…

I think it’s coming back around to what @charsleysa said - for some reason it’s including the request’s headers in the response! It could very well be a bug in our own code :man_facepalming:

Looking…

In that case, that makes me happy, because there’s a nice, proper, workaround. Just threw me for a loop as it worked everywhere else!

17

Yup! That was it. :man_facepalming: Found a tiny one-liner bug that was causing us to return the request’s headers along with the response headers :man_facepalming: :man_facepalming:

Thanks all for the help!

1 Like