Get constant input/output errors in ssh console

Questions / Help
,
1

When I run fly ssh console, I can connect, but I cannot do anything. My app is a simple api-only rails app with a SQLite database. The app is running, but I cannot connect to the machine to troubleshoot. I don’t think I have anything blatantly misconfigured.

When I ssh:

fly ssh console
Connecting to fdaa:7:7bac:a7b:254:bd34:e4cd:2... complete
-bash: /etc/profile: Input/output error
-bash: /root/.bash_profile: Input/output error
-bash-5.2# ls
-bash: /usr/bin/ls: Input/output error

My logs show the following:

2024-04-07T14:31:46.512 app[080624ea36e118] ord [info] 2024/04/07 14:31:46 New SSH Session - cameron@theaboutbox.com
2024-04-07T14:31:46.697 app[080624ea36e118] ord [info] [ 382.653932] blk_update_request: I/O error, dev vda, sector 12858080 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2024-04-07T14:31:46.708 app[080624ea36e118] ord [info] [ 382.665085] blk_update_request: I/O error, dev vda, sector 74008 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.709 app[080624ea36e118] ord [info] [ 382.666143] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #16385: comm bash: reading directory lblock 0
2024-04-07T14:31:46.721 app[080624ea36e118] ord [info] [ 382.677315] blk_update_request: I/O error, dev vda, sector 74008 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.722 app[080624ea36e118] ord [info] [ 382.678363] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #16385: comm bash: reading directory lblock 0
2024-04-07T14:31:46.732 app[080624ea36e118] ord [info] [ 382.688919] blk_update_request: I/O error, dev vda, sector 74008 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.733 app[080624ea36e118] ord [info] [ 382.689974] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #16385: comm bash: reading directory lblock 0
2024-04-07T14:31:46.745 app[080624ea36e118] ord [info] [ 382.701226] blk_update_request: I/O error, dev vda, sector 74008 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.746 app[080624ea36e118] ord [info] [ 382.702542] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #16385: comm bash: reading directory lblock 0
2024-04-07T14:31:46.757 app[080624ea36e118] ord [info] [ 382.713887] blk_update_request: I/O error, dev vda, sector 12649040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.758 app[080624ea36e118] ord [info] [ 382.714941] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #393387: comm bash: reading directory lblock 0
2024-04-07T14:31:46.768 app[080624ea36e118] ord [info] [ 382.725127] blk_update_request: I/O error, dev vda, sector 74264 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.769 app[080624ea36e118] ord [info] [ 382.726190] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #371: comm bash: reading directory lblock 0
2024-04-07T14:31:46.780 app[080624ea36e118] ord [info] [ 382.737156] blk_update_request: I/O error, dev vda, sector 76632 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.781 app[080624ea36e118] ord [info] [ 382.738200] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #2230: comm bash: reading directory lblock 0
2024-04-07T14:31:46.783 app[080624ea36e118] ord [info] [ 382.739399] blk_update_request: I/O error, dev vda, sector 12649040 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.784 app[080624ea36e118] ord [info] [ 382.740433] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #393387: comm bash: reading directory lblock 0
2024-04-07T14:31:46.785 app[080624ea36e118] ord [info] [ 382.741522] blk_update_request: I/O error, dev vda, sector 74264 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
2024-04-07T14:31:46.786 app[080624ea36e118] ord [info] [ 382.742550] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #371: comm bash: reading directory lblock 0
2024-04-07T14:31:46.787 app[080624ea36e118] ord [info] [ 382.743600] EXT4-fs error (device vda): __ext4_find_entry:1663: inode #2230: comm bash: reading directory lblock 0

Any idea how I can fix this?

2

Hi… These low-level block I/O errors seem to be rolling through the ord lately…

(I saw this recently there as well.)

Machine restart seems to fix it:

$ fly m restart 080624ea36e118
1 Like
3

That worked, thank you so much!

1 Like
4

Just wanted to add that I saw this as well in ord recently, but restarting the machine only helped temporarily

I then went to destroy and recreate it via fly scale, at the moment it seems like it’s fixed that…

1 Like
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.