Random bad request error when uploading file

Hi! I’m seeing an error in our production apps that I cannot reproduce locally which is making me believe that it might be related with Fly’s HTTP proxy.

We have an REST API endpoint that allows uploading files implemented with a controller that consumes the stream of the file from Plug.Upload. We then have a bash script that we are using for testing / bulk upload, that uploads file to an application using curl. The curl invocation is as follows:

curl -s -w "%{http_code}" -H "Authorization: Bearer $API_TOKEN" "$files_url" -F "file=@\"$file\"" -o "$response_body"

This works locally with no error. But when run against an app deployed in fly, randomly for some files sometimes we get a 400 bad request response and in the logs we see:

2024-12-05T16:21:46Z app[3d8d3dec1217d8] iad [info]16:21:46.103 [error] ** (Bandit.HTTPError) Request line HTTP error: "--------------------------N7amoT9mSemLDwAv5AK4sP\r\n"

Re-executing the curl command until it works fixes the issue

Reading through some GitHub issues and forum post I starting to believe that it might be related with how the HTTP request is parsed and my hypothesis is that maybe fly’s HTTP proxy is doing something that Bandit does not like.

Any ideas what the problem could be?

Also if any fly staff member reads this, I tried to pay for support but I was required to migrate to Pay as you go plan first. I did that but it becomes active next month, so I’m in a deadlock where I cannot pay to access to support :upside_down_face: :smiling_face_with_tear:

Hi… billing@fly.io should be able to resolve this part, actually.

(You don’t need an active support plan to email that particular address.)

[↑ I’m not staff, obviously, but a message directed there has quickly broken this kind of impasse in the past, :ring_buoy:.]