While validating a websocket fan-in/fan-out workload (single Machine holding 100K connections, exchanging small messages with all of them), I hit a throughput ceiling that turned out to be packet-rate limiting at the VM level. I measured it directly and want to share the numbers, and ask whether this is expected/documented behavior. The most surprising part is that the limit appears to be constant and doesn’t scale with Machine size.
Setup: iperf3 UDP floods between Machines in the same region (ams) over private networking (*.internal). The results are consistent with what I see with my websocket server.
Findings:
- Egress caps at ~55-58K packets/sec per Machine, and it’s packet-metered, not byte-metered: 32-byte and 1200-byte UDP datagrams both cap at ~52-55K pps (the latter running at ~490 Mbit/s, so bandwidth wasn’t the constraint).
- The limit does not scale with Machine size: performance-1x = 56.2K pps, performance-8x = 58.2K pps. Sender CPU was at ~25-30% of one core, so it’s not compute-bound. A shared-1x also hit ~56K pps in a 10s test, though burst credits likely covered the CPU cost there.
- Similar story for ingress, with lower limits (~30-45K pps). Inbound messages are dropped when the limit is exceeded.
- Ingress and egress budgets seem to be separate, with only mild degradation when flooding both ways simultaneously.
- Two concurrent sender-receiver pairs under the same app each sustained the full rate (~57–62K pps each), so the budget is per-Machine, not per-app or per-account.
Is this expected behavior? Assuming my measurements are correct, wouldn’t it make more sense for the limit to scale with Machine size? My Machine is mostly idle, but broadcast-style workloads are irreducibly one packet per connection, so this cap effectively bounds my fan-out at ~50K msgs/sec per Machine regardless of what I pay for. Is there any way to request a higher packet budget?