Blk_update_request error when trying to write to Postgres

Questions / Help
1

Hey! I have a node.js app with attached postgres deployed on free tier. It was working fine, until last night. Now the app itself is working fine (healthchecks are ok, and I can call some endpoints) but when I call endpoint to write some data to db I got this errors (these are logs from APP, not postgres):

2023-05-17T08:39:09.527 app[f53297ce] waw [info] [54075.732223] blk_update_request: I/O error, dev vdb, sector 8328 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.528 app[f53297ce] waw [info] [54075.733448] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 0, inode_bitmap = 1041

2023-05-17T08:39:09.530 app[f53297ce] waw [info] [54075.734841] blk_update_request: I/O error, dev vdb, sector 8336 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.531 app[f53297ce] waw [info] [54075.735980] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 1, inode_bitmap = 1042

2023-05-17T08:39:09.532 app[f53297ce] waw [info] [54075.737366] blk_update_request: I/O error, dev vdb, sector 8344 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.534 app[f53297ce] waw [info] [54075.738421] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 2, inode_bitmap = 1043

2023-05-17T08:39:09.535 app[f53297ce] waw [info] [54075.739929] blk_update_request: I/O error, dev vdb, sector 8352 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.536 app[f53297ce] waw [info] [54075.741108] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 3, inode_bitmap = 1044

2023-05-17T08:39:09.537 app[f53297ce] waw [info] [54075.742531] blk_update_request: I/O error, dev vdb, sector 8360 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.539 app[f53297ce] waw [info] [54075.743666] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 4, inode_bitmap = 1045

2023-05-17T08:39:09.540 app[f53297ce] waw [info] [54075.745038] blk_update_request: I/O error, dev vdb, sector 8368 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.541 app[f53297ce] waw [info] [54075.746166] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 5, inode_bitmap = 1046

2023-05-17T08:39:09.542 app[f53297ce] waw [info] [54075.747634] blk_update_request: I/O error, dev vdb, sector 8376 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.544 app[f53297ce] waw [info] [54075.748766] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 6, inode_bitmap = 1047

2023-05-17T08:39:09.545 app[f53297ce] waw [info] [54075.750018] blk_update_request: I/O error, dev vdb, sector 8384 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0

2023-05-17T08:39:09.546 app[f53297ce] waw [info] [54075.751039] EXT4-fs error (device vdb): ext4_read_inode_bitmap:201: comm sh: Cannot read inode bitmap - block_group = 7, inode_bitmap = 1048

2023-05-17T08:39:40.391 app[f53297ce] waw [info] [54106.595750] blk_update_request: I/O error, dev vdb, sector 8208 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.393 app[f53297ce] waw [info] [54106.596849] blk_update_request: I/O error, dev vdb, sector 8216 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.394 app[f53297ce] waw [info] [54106.597959] blk_update_request: I/O error, dev vdb, sector 8224 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.395 app[f53297ce] waw [info] [54106.599116] blk_update_request: I/O error, dev vdb, sector 8232 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.396 app[f53297ce] waw [info] [54106.600295] blk_update_request: I/O error, dev vdb, sector 8240 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.397 app[f53297ce] waw [info] [54106.601487] blk_update_request: I/O error, dev vdb, sector 8248 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.399 app[f53297ce] waw [info] [54106.602677] blk_update_request: I/O error, dev vdb, sector 8256 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.400 app[f53297ce] waw [info] [54106.604872] blk_update_request: I/O error, dev vdb, sector 8272 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.401 app[f53297ce] waw [info] [54106.604872] blk_update_request: I/O error, dev vdb, sector 8272 op 0x0:(READ) flags 0x87700 phys_seg 1 prio class 0

2023-05-17T08:39:40.402 app[f53297ce] waw [info] [54106.606012] blk_update_request: I/O error, dev vdb, sector 8280 op 0x0:(READ) flags 0x83700 phys_seg 1 prio class 0

2023-05-17T08:39:40.404 app[f53297ce] waw [info] [54106.607533] EXT4-fs error (device vdb): ext4_wait_block_bitmap:531: comm kworker/u2:2: Cannot read block bitmap - block_group = 9, block_bitmap = 1034

And also some nodejs error:

2023-05-17T08:42:09.005 app[f53297ce] waw [info] Error: EIO: i/o error, read

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Object.readSync (node:fs:749:3)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at tryReadSync (node:fs:449:20)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Object.readFileSync (node:fs:487:19)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Module._extensions..js (node:internal/modules/cjs/loader:1271:18)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Module.load (node:internal/modules/cjs/loader:1117:32)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Module._load (node:internal/modules/cjs/loader:958:12)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Module.require (node:internal/modules/cjs/loader:1141:19)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at require (node:internal/modules/cjs/helpers:110:18)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Object.<anonymous> (/node_modules/iconv-lite/encodings/index.js:6:5)

2023-05-17T08:42:09.005 app[f53297ce] waw [info] at Module._compile (node:internal/modules/cjs/loader:1254:14)

It was working before, I haven’t made any changes to configuration of machines/pg nor I have made any changes to postgres in app, any ideas what might be an issue here?

2

Hi there!

This is definetly a new one, here’s some tips for debugging this.

  • You mentioned health checks are fine for the app, can you verify health checks for your DB too? Is the disk full?
  • Can you double check your nodejs app is using your postgres? Sometimes node apps have a fallback to SQLite so if that’s the case you’re actually writing to the node app file system instead of PG. Try to connect to your postgres directly and see if data was written there.
  • The second logs block you sent seems like a NodeJS file system error (looking at readFileSync) so doesn’t seem postgres related, can you try verifying another endpoint of your app that it wasn’t just that specific endpoint that errored?

Hope these might help

3

Thanks for replying!

  • the db is fine (3/3 checks are okay). I am not sure how to exactly find if the disk is full but I am confident that it isn’t! The app and db was created 3/4 days ago and it shows that it is 8mb in size. Having 1gb volume attached I guess something else would have to take all the available space)
  • I could write/read when proxing DB, and I forgot to mention but I also could READ db data from app. Just coudn’t make Insert :frowning:

I fixed it by detaching and attaching DB to app, no idea what the issue was but I guess we can close this topic :slight_smile:

1 Like
4

Very interesting error indeed, glad its working now!

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.