sometimes HTTP requests hang after downloading 156K of data

I’m seeing a weird issue where sometimes HTTP requests hang in the middle of the response.

For example, curl https://mess-with-dns.fly.dev/css/tailwind.min.css often hangs after receiving 156K of data. Some notes:

  • it’s not always 156K of data, sometimes it’s 2156k or 254k
  • it does seem to always be an even amount of kilobytes for some reason.
  • when I run the same curl command from localhost on the instance, I don’t see the same issue
  • I can reproduce the issue on two different machines (my home computer and a digitalocean machine)

It’s totally possible this is an issue on my end somehow, but I’m struggling to come up with an explanation so I’m posting it here

It does seem to be an issue. I’ve recreated it. I tried downloading it using that URL and found the same e.g

$ curl https://mess-with-dns.fly.dev/css/tailwind.min.css > tailwind.min.css
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  5 2865k    5  156k    0     0   2643      0  0:18:30  0:01:00  0:17:30     0
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

I wondered if it was the http version, but that also fails:

$ curl --http1.1 https://mess-with-dns.fly.dev/css/tailwind.min.css > tailwind.min.css
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 10 2865k   10  293k    0     0   4949      0  0:09:52  0:01:00  0:08:52     0
curl: (18) transfer closed with 2633885 bytes remaining to read

Is this file being served by your app (if so … it may be down to gzip, or something in-app)? Or are you using Fly’s static file functionality in the fly.toml? That may help narrow down where the issue is. This bit of the fly.toml if not sure:

When I run curl from flyctl ssh console , it works fine, which makes me think it’s not an issue with my app. It’s being served from my app.

$ flyctl ssh console
Connecting to top1.nearest.of.mess-with-dns.internal... complete
# bash
root@555c48a2:/# curl localhost:8080/css/tailwind.min.css
HTTP/1.1 200 OK

Can you tell us a bit more about how this is being served (language, framework, libraries, etc.)? It’s possible it’s interacting badly with our proxy.

It’s a Go program using net/http's static files feature

I finally noticed that curling the file on the instance takes a really long time – 1.6 seconds. I don’t know why that is, but it’s probably hanging because of that – sorry for the noise!

time curl localhost:8080/css/tailwind.min.css > /dev/null
real	0m1.678s

Interesting. Can you tell us which instance? I can look up our metrics on it.

I assume it’s getting those static files from the disk. I wonder if there’s something going on there.

I think it was 555c48a2, based on the logs – I just redeployed so it’s gone now. My guess it that it’s more to do with CPU usage inside the process though.

Yeah I was trying to reproduce and it does look fine now!