Hi, our grafana logs are showing no data since 6pm eastern yesterday. I can see logs in the fly dashboard for our apps, but not in grafana. We’ve tried restarting our apps, doing a new deploy on new machines, but the logs in grafana are still empty. Any tips on how to fix this?
Have the same issue, I’ve already raised a support request ~12 hours ago, still no responce.
It seems this is not only a Grafana issue, but also a problem with log collection on Fly.io’s side, as the fly logs command also returns no recent logs.
What region are you both in? One of my London apps has been streaming logs into Grafana for the last 12 hours straight.
my apps are in EWR
A good test you can do is to do flyctl ssh console, assuming you have a fly.toml in the same directory. This should get you a console onto one of your app’s machines, and it should put a line in your Grafana logs too.
What does your Grafana URL look like? Here is one of mine:
https://fly-metrics.net/d/fly-logs/fly-logs?from=now-6h&to=now&var-app=alpha-beta-delta&orgId=243639&var-query=&var-instance=All&refresh=5m
You’ll have to swap out the app name, of course.
the issue is not that we are not generating logs, there are plenty of them in the fly dashboard under logs and errors, but our grafana is empty for all our apps.
@aleksey-kiselev-691 Your issue is different. Daniel can see logs in the Fly dashboard, but you cannot. Are you also in EWR?
Same issue over here. I have apps running in IAD, GRU.
It stopped displaying logs in Grafana around 2 days ago
Just bumping. I have the same issue where I can see logs live in the dashboard (and in terminal with fly logs) but not in grafana. I started noticing the problem around 6 hours ago (morning 21. nov CET time). I have machines in CDG.
Our logs just appeared in Grafana. No changes or deployments, log entries just started flowing again. I imagine the history is gone though?
Similar issue since around 11 hours ago, I can see live logs in the fly dashboard, but logs aren’t kept in Grafana mostly. “mostly” because there were intermittent burst of logs (~19 minutes each time) being tracked correctly in Grafana since it started happening.
The history part of logs is rather beta, in general, and it’s common to have gaps like this.
People who need more reliable log retention are nudged toward setting up their own Vector instance, etc.
https://fly.io/docs/monitoring/exporting-logs/
There’s been talk for a while of Fly.io eventually offering a more solid and feature-rich extra-cost service—with convenient alerts, and so on—but that seems in limbo…
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.