PSA: My database had 20 GB of its volume "used" when it shouldn't and yours might too

,
1

Hi, I was trying to track down some mystery file usage today for my pg db and I found where it was.

TLDR: I think there’s a bug in the way the dashboard shows “Used” storage for a volume. The default settings on my db’s volume’s filesystem had a 5% reserved block ratio applied and shown as part of the usage in the dashboard, when that space is actually usable.

In other words, for my 400 GB volume, 20 GB were “used” right off the rip according to the dashboard.

Before (with all default settings – I hadn’t fiddled with this database or volume at all):

root@080e337f007748:/# df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc        394G   32G  347G   9% /data

image
image775×424 43.8 KB

After applying tune2fs -m 0.5 /dev/vdc (DO NOT RUN THIS COMMAND):

root@9185e256a99d78:/# df -h /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc        394G   32G  346G   9% /data

image
image786×445 46.5 KB

I put the disclaimer in because you need to unmount the filesystem to not corrupt it with that command, and it’s not necessary anyway. The command changes the filesystem’s reserved blocks ratio from the linux-default 5% to 0.5%. Reserved blocks are filesystem blocks that are reserved for root so the OS remains usable if space runs out. Since fly machines only use root, all of the reserved blocks are usable by an app anyway.

You’ll note the screenshots are on two different volumes. That’s because I created a separate PG app and volume based on a snapshot of the same database a couple hours apart to validate this.

I sent an email to fly support now with this info, so we’ll see what they say.

2

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.