Understanding FIRECRACKER LOAD AVERAGE (Part Deux) - shared CPU

[Whistler]

A variation on the theme in Understanding FIRECRACKER LOAD AVERAGE…

As well as monitoring from ‘outside’ my VMs (i.e. Fly-provided metrics) I also monitor from ‘within’ the VM, to try and ascertain where any CPU and/or memory issues may originate from.

Periodically I see bursts of high load averages that I can’t explain .

After further debugging (top -b -d 1 > /tmp/top.txt) I suspect that the reported load averages are of the host’s core(?), which in a shared CPU environment would explain the discrepancy I’m seeing. It’s also possible the data provided to me by top (and associated tools) isn’t a truthful reflection of reality.

I’ve included a sample output (egrep -A 3 "top - 00:4(2|3|4|5|6|7|8):" /tmp/top.txt | egrep "^top|^%Cpu" | awk 'NR%2{printf "%s ",$0;next;}1') from last night below. LHR host d88f, shared-cpu-1x, times in UTC (see section starting in bold, prior minute included to provide background re:spiking):

Note the load average spike to >1.00@~00:43:40 with no real associated increase in reported CPU usage from within the VM itself.

Would it be possible to confirm if my assumptions are correct re: firecracker load averages and shared CPUs?

top - 00:42:40 up 11 days, 1:21, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:41 up 11 days, 1:21, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:42 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.0 st
top - 00:42:43 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:44 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:45 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:42:46 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:47 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:42:48 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:49 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:50 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:42:51 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:52 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:53 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:54 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:42:55 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:56 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:57 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:42:58 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:42:59 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:43:00 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:01 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:02 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:03 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:04 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:05 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:06 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:07 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:08 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:09 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:10 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:11 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:12 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:13 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:14 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:15 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:16 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:17 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:18 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:43:19 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:20 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:21 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:22 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:23 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:24 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:25 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:43:26 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:27 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 96.1 id, 1.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:28 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:29 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:30 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:31 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:32 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:33 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:34 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:35 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:36 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:37 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 2.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:38 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:39 up 11 days, 1:22, 0 users, load average: 0.00, 0.00, 0.00 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:40 up 11 days, 1:22, 0 users, load average: 1.04, 0.22, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 2.0 st
top - 00:43:41 up 11 days, 1:22, 0 users, load average: 1.04, 0.22, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.0 si, 2.0 st
top - 00:43:42 up 11 days, 1:23, 0 users, load average: 1.04, 0.22, 0.07 %Cpu(s): 0.0 us, 2.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:43 up 11 days, 1:23, 0 users, load average: 1.04, 0.22, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:44 up 11 days, 1:23, 0 users, load average: 1.04, 0.22, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:45 up 11 days, 1:23, 0 users, load average: 0.96, 0.21, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 1.0 si, 1.0 st
top - 00:43:46 up 11 days, 1:23, 0 users, load average: 0.96, 0.21, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:47 up 11 days, 1:23, 0 users, load average: 0.96, 0.21, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:48 up 11 days, 1:23, 0 users, load average: 0.96, 0.21, 0.07 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:49 up 11 days, 1:23, 0 users, load average: 0.96, 0.21, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:50 up 11 days, 1:23, 0 users, load average: 0.88, 0.21, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:51 up 11 days, 1:23, 0 users, load average: 0.88, 0.21, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:52 up 11 days, 1:23, 0 users, load average: 0.88, 0.21, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:43:53 up 11 days, 1:23, 0 users, load average: 0.88, 0.21, 0.07 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:54 up 11 days, 1:23, 0 users, load average: 0.88, 0.21, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:55 up 11 days, 1:23, 0 users, load average: 0.81, 0.20, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:56 up 11 days, 1:23, 0 users, load average: 0.81, 0.20, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:43:57 up 11 days, 1:23, 0 users, load average: 0.81, 0.20, 0.07 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:58 up 11 days, 1:23, 0 users, load average: 0.81, 0.20, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:43:59 up 11 days, 1:23, 0 users, load average: 0.81, 0.20, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:00 up 11 days, 1:23, 0 users, load average: 0.74, 0.20, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:01 up 11 days, 1:23, 0 users, load average: 0.74, 0.20, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:02 up 11 days, 1:23, 0 users, load average: 0.74, 0.20, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:03 up 11 days, 1:23, 0 users, load average: 0.74, 0.20, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:04 up 11 days, 1:23, 0 users, load average: 0.74, 0.20, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:05 up 11 days, 1:23, 0 users, load average: 0.68, 0.20, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:44:06 up 11 days, 1:23, 0 users, load average: 0.68, 0.20, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:07 up 11 days, 1:23, 0 users, load average: 0.68, 0.20, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:08 up 11 days, 1:23, 0 users, load average: 0.68, 0.20, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:09 up 11 days, 1:23, 0 users, load average: 0.68, 0.20, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:10 up 11 days, 1:23, 0 users, load average: 0.63, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:11 up 11 days, 1:23, 0 users, load average: 0.63, 0.19, 0.07 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:12 up 11 days, 1:23, 0 users, load average: 0.63, 0.19, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:13 up 11 days, 1:23, 0 users, load average: 0.63, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:14 up 11 days, 1:23, 0 users, load average: 0.63, 0.19, 0.07 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:44:15 up 11 days, 1:23, 0 users, load average: 0.58, 0.19, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:16 up 11 days, 1:23, 0 users, load average: 0.58, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:17 up 11 days, 1:23, 0 users, load average: 0.58, 0.19, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:18 up 11 days, 1:23, 0 users, load average: 0.58, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:19 up 11 days, 1:23, 0 users, load average: 0.58, 0.19, 0.07 %Cpu(s): 2.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:20 up 11 days, 1:23, 0 users, load average: 0.53, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:21 up 11 days, 1:23, 0 users, load average: 0.53, 0.19, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:22 up 11 days, 1:23, 0 users, load average: 0.53, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:44:23 up 11 days, 1:23, 0 users, load average: 0.53, 0.19, 0.07 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st
top - 00:44:24 up 11 days, 1:23, 0 users, load average: 0.53, 0.19, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:25 up 11 days, 1:23, 0 users, load average: 0.49, 0.18, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:26 up 11 days, 1:23, 0 users, load average: 0.49, 0.18, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:27 up 11 days, 1:23, 0 users, load average: 0.49, 0.18, 0.07 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:28 up 11 days, 1:23, 0 users, load average: 0.49, 0.18, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:29 up 11 days, 1:23, 0 users, load average: 0.49, 0.18, 0.07 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:30 up 11 days, 1:23, 0 users, load average: 0.45, 0.18, 0.06 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:31 up 11 days, 1:23, 0 users, load average: 0.45, 0.18, 0.06 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 0.0 si, 1.0 st
top - 00:44:32 up 11 days, 1:23, 0 users, load average: 0.45, 0.18, 0.06 %Cpu(s): 3.0 us, 1.0 sy, 0.0 ni, 96.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:33 up 11 days, 1:23, 0 users, load average: 0.45, 0.18, 0.06 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:34 up 11 days, 1:23, 0 users, load average: 0.45, 0.18, 0.06 %Cpu(s): 2.0 us, 1.0 sy, 0.0 ni, 95.0 id, 0.0 wa, 0.0 hi, 1.0 si, 1.0 st
top - 00:44:35 up 11 days, 1:23, 0 users, load average: 0.41, 0.18, 0.06 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:36 up 11 days, 1:23, 0 users, load average: 0.41, 0.18, 0.06 %Cpu(s): 0.0 us, 1.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 0.0 si, 2.0 st
top - 00:44:37 up 11 days, 1:23, 0 users, load average: 0.41, 0.18, 0.06 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 98.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:38 up 11 days, 1:23, 0 users, load average: 0.41, 0.18, 0.06 %Cpu(s): 1.0 us, 0.0 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:39 up 11 days, 1:23, 0 users, load average: 0.41, 0.18, 0.06 %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
top - 00:44:40 up 11 days, 1:23, 0 users, load average: 0.38, 0.17, 0.06 %Cpu(s): 1.0 us, 1.0 sy, 0.0 ni, 97.1 id, 0.0 wa, 0.0 hi, 1.0 si, 0.0 st

Note: I’ve deleted anything outside of the 2 minute window shown above due to post character limits.

[eli]

Thanks for including all that background info! And it’s an interesting question, glad to come back to it.

So for vCPU behavior, I think you can treat a microVM pretty much like a regular KVM machine most of the time – as far as I can tell, firecracker-microvm wraps the KVM api for a lot of the behaviors you’d be curious about in this context.

A few hours ago, it sounds like you also observed an increase in overall resource utilization on your app, and that this happened at the same time that you saw performance issues.

If you’re interested in investigating this further, you might want to try some of the techniques detailed in this article on interpreting Linux loadavg at least as a jumping-off point.

Please feel free to respond with any additional questions or observations you want to share – we’re always happy to take a closer look

[Whistler]

The problem earlier, with my Postgres issue, is where Fly’s metric for load average was not reflected by the stats available within the VMs (either load average or CPU usage). It’s like the monitoring system was “stuck” (Fly’s load average being reported as pegged at 1).

The issue described above is where the load average being reported is the same within the VM and Fly’s metrics but this wasn’t reflected by the CPU usage (stats/idle/etc) shown within the VM itself. It’s possible the mismatches being seen are a reflection of the “noisy neighbour” problem (shared CPUs), or I just need to spend more time trying to debug the answer…

If/when I eventually move to dedicated CPUs I’ll see if the problem still occurs.