I set up log shipping directly to grafana (logs.betterstack.com) about 9 months ago, and everything worked as expected. I set this up due to a blog post from fly.io and I followed the directions for a pre-built app specifically for log shipping. I’ve only generated a very tiny amount of logs (just a few kb).
Recently I received an email from grafana about exceeding my log storage space (1gb) which I thought would be impossible. I looked into my grafana logs and noticed tons of logs from other apps that don’t belong to me. I contacted grafana support and they said it was an issue with fly.io.
Hi @brad1, we’re grateful for the report, notifications like this give us heartburn and we’ve had a handful of folks investigating for the past few hours. We’re unable to identify anything that would indicate it’s possible for app logs to be delivered to another app or org by accident, nor can we detect any evidence that this has happened. We’ve reached out on Discourse, but if you don’t mind emailing support@fly.io we’d really appreciate additional information about what you’re seeing. You can also cc me directly on that email via scott@fly.io.
Our current theory is there is some sort of log amplification happening due to rate-limits or quotas being breached.