I have a Python script running on Fly that records an audio stream and then periodically uploads the resulting MP3 file to Azure. It’s a shared-cpu-1x on the Apps v1 platform with an attached 5GB volume for storing the recordings before they’re uploaded. Apart from some network issues, it has been running fine.
Today I tried to SSH into the application and I got this error:
$ fly ssh console
Connecting to fdaa:2:4577:a7b:f3:7:ed72:2... complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
$ fly ssh console
Connecting to fdaa:2:4577:a7b:f3:7:ed72:2... complete
fork/exec /bin/bash: input/output error
The first failure is one I’ve gotten often and it goes away on its own, but the second one, the “input/output error”, I don’t know how to fix.
Looking in the application logs shows me this:
2023-06-02T03:26:23.312 app[f01c93ff] ord [info] 2023/06/01 22:26:23 unexpected error fetching cert: transient SSH server error: can't resolve _orgcert.internal
2023-06-02T03:26:23.396 app[f01c93ff] ord [info] 2023/06/01 22:26:23 unexpected error: [ssh: no auth passed yet, transient SSH server error: can't resolve _orgcert.internal]
2023-06-02T03:26:31.018 app[f01c93ff] ord [info] [286189.727799] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x84700 phys_seg 1 prio class 0
2023-06-02T03:26:31.019 app[f01c93ff] ord [info] [286189.729147] blk_update_request: I/O error, dev vdb, sector 4456464 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
2023-06-02T03:26:31.020 app[f01c93ff] ord [info] [286189.730699] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:26:31.022 app[f01c93ff] ord [info] 2023/06/01 22:26:31 unexpected error executing command: fork/exec /bin/bash: input/output error
2023-06-02T03:26:35.128 app[f01c93ff] ord [info] [286193.838532] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:26:35.129 app[f01c93ff] ord [info] 2023/06/01 22:26:35 unexpected error executing command: fork/exec /bin/bash: input/output error
2023-06-02T03:27:13.047 app[f01c93ff] ord [info] [286231.757015] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:27:13.048 app[f01c93ff] ord [info] 2023/06/01 22:27:13 unexpected error executing command: fork/exec /bin/bash: input/output error
2023-06-02T03:30:00.509 app[f01c93ff] ord [info] [286399.216622] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:30:00.510 app[f01c93ff] ord [info] 2023/06/01 22:30:00 unexpected error executing command: fork/exec /bin/bash: input/output error
2023-06-02T03:47:53.753 app[f01c93ff] ord [info] [287472.448618] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:47:53.754 app[f01c93ff] ord [info] 2023/06/01 22:47:53 unexpected error executing command: fork/exec /bin/bash: input/output error
2023-06-02T03:47:59.392 app[f01c93ff] ord [info] [287478.087826] blk_update_request: I/O error, dev vdb, sector 4456448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2023-06-02T03:47:59.393 app[f01c93ff] ord [info] 2023/06/01 22:47:59 unexpected error executing command: fork/exec /bin/bash: input/output error
Is the disk that my application is running on failing? If so, how can I fix that?