But accidentally at about 8 AM (Europe time) my app in ARN region started to throw error into logs: “No space left on device” .
I blindly did scale my app for better performance but that didn’t solve problem. At about 10:30 AM app started to work fine, but some images are missing.
Strange thing is when I refresh, image can appear.
I don’t know why but now everything work like before. Maybe someone from Fly.io acted.
I did create volume snapshot from a day ago but I didn’t attach it to a machine yet.
I’m scratching my head.
It is working now :). I think problem was on Fly.io side.
Are the machines in the app just round-robin copies of each other, with traffic being expected to be served roughly equally across the two?
If so, I assume that your two volumes contain an identical copy of the images you want to serve via HTTP. What are you using to keep the volumes in sync?
I think they are copies of each other. I don’t know answer to your question because it is outside my knowledge. Fly.io is not an easy platform. Sometimes I don’t know what I’m doing.
Second volume was newly created from a day-ago snapshot.
OK. You can get a shell on each machine using flyctl ssh console --machine d8dd599c19de58 (ditto for the other machine ID). Inside each shell you can do ls -l /data/storage to see what’s in there, and ls -lR /data/storage if you’re using subfolders.
Whoever built it may not know that volumes are not shared network drives, and if the data is dynamic, it will need to be synced on a regular basis. Talk to whoever built this software, to see if they need to add a sync system.
That does look promising… Try the following commands first, though, since if you have two or more of these Rails Machines then their volumes are probably all getting increasingly out of synch (as @halfer was suggesting):
$ fly m list -a cstillas
$ fly m list -a cstillas-db
Each of the volumes in the cstillas list (without the -db suffix) may have a different subset of your images…
