Most of our apps stopped streaming their logs a few days ago. I have been trying to find out what is going on but I guess there is not much left that I can do.
Analysis: The fly.io dashboard and flyctl logs both show the most recent messages, but do not print new messages while being open. When I close and reopen them, newer messages appear. Our log-shipper instance does not receive anything via NATS (confirmed via vector tap on the VM).
Can somebody internally take a look at it, please?
Did somebody have a chance to investigate this, yet?
Hi,
I’ve just been looking in to this and have restarted the nats proxy we run which filters log messages for customers. I have seen in its logs that your log shipper has resubscribed to the nats subjects you have configured - can you confirm if logs are shipping again?
Hi Steve,
unfortunately not. This issue is also not just limited to NATS and the log-shipper, but also to flyctl logs and the fly.io dashboard, which, as far as I understood through different posts here, do not use NATS but query your Elastic-Cluster directly.
Just to clarify: When I open the dasboard or run flyctl logs, the most recent messages are shown. Just new messages do not appear automatically (streaming does not work). Stopping flyctl logs and restarting it again shows the newest messages.
oh dear. how annoying! The dashboard and flyctl use elasticsearch for the initial pull of logs (to get the last logs historically) and then use nats for the live stream. I’ll continue the investigations!
Not sure if you did something, but it seems the log streams have returned after restarting the apps. I take this as solved 
Some of our nats servers weren’t sending messages across regions in our cluster (and not so helpfully not complaining about not being able to do it because that’s how nats works). I restarted the process everywhere to force all of the servers to rejoin with each other and it’s started the streams flowing again.
Now i’m looking in to ways to detect this automatically!
Restart all processes once per week 
