sometimes HTTP requests hang after downloading 156K of data

#1

I’m seeing a weird issue where sometimes HTTP requests hang in the middle of the response.

For example, curl https://mess-with-dns.fly.dev/css/tailwind.min.css often hangs after receiving 156K of data. Some notes:

  • it’s not always 156K of data, sometimes it’s 2156k or 254k
  • it does seem to always be an even amount of kilobytes for some reason.
  • when I run the same curl command from localhost on the instance, I don’t see the same issue
  • I can reproduce the issue on two different machines (my home computer and a digitalocean machine)

It’s totally possible this is an issue on my end somehow, but I’m struggling to come up with an explanation so I’m posting it here

#2

It does seem to be an issue. I’ve recreated it. I tried downloading it using that URL and found the same e.g

$ curl https://mess-with-dns.fly.dev/css/tailwind.min.css > tailwind.min.css
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  5 2865k    5  156k    0     0   2643      0  0:18:30  0:01:00  0:17:30     0
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

I wondered if it was the http version, but that also fails:

$ curl --http1.1 https://mess-with-dns.fly.dev/css/tailwind.min.css > tailwind.min.css
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 10 2865k   10  293k    0     0   4949      0  0:09:52  0:01:00  0:08:52     0
curl: (18) transfer closed with 2633885 bytes remaining to read

Is this file being served by your app (if so … it may be down to gzip, or something in-app)? Or are you using Fly’s static file functionality in the fly.toml? That may help narrow down where the issue is. This bit of the fly.toml if not sure:

#3

When I run curl from flyctl ssh console , it works fine, which makes me think it’s not an issue with my app. It’s being served from my app.

$ flyctl ssh console
Connecting to top1.nearest.of.mess-with-dns.internal... complete
# bash
root@555c48a2:/# curl localhost:8080/css/tailwind.min.css
HTTP/1.1 200 OK
#4

Can you tell us a bit more about how this is being served (language, framework, libraries, etc.)? It’s possible it’s interacting badly with our proxy.

#5

It’s a Go program using net/http's static files feature

#6

I finally noticed that curling the file on the instance takes a really long time – 1.6 seconds. I don’t know why that is, but it’s probably hanging because of that – sorry for the noise!

time curl localhost:8080/css/tailwind.min.css > /dev/null
real	0m1.678s
#7

Interesting. Can you tell us which instance? I can look up our metrics on it.

I assume it’s getting those static files from the disk. I wonder if there’s something going on there.

#8

I think it was 555c48a2, based on the logs – I just redeployed so it’s gone now. My guess it that it’s more to do with CPU usage inside the process though.

#9

Yeah I was trying to reproduce and it does look fine now!