I have an application that sometimes needs to write a 200 Mb text file to the machine’s attached volume. Until about the beginning of April, this process always took no more than 2 seconds. Now it takes at least 10 seconds. Sometimes more than 15 seconds.
Is this normal? Has anyone experienced the same drop in I/O performance?
Hm… I do see that same slowness on the low-budget shared vCPU Machines, —but not with Fly.io’s specifically performance set…
root@e82123abcdef98:/data# echo "$FLY_REGION"
ewr
root@e82123abcdef98:/data# dd if=/dev/zero of=test bs=1M count=200 conv=sync
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.425327 s, 493 MB/s
The existence of such a discrepancy isn’t super-surprising, since the shared class doesn’t consist of the fastest hares in the warren, overall. Even so, it’s possible that this is an unintended interaction of CPU throttling with virtualized I/O. (I seem to recall Fly.io mentioning that part of Firecracker’s own operations, on the host rather than the guest side, were also caught up in the throttle.) I’d suggest taking a look at the Grafana panel mentioned in that earlier thread.