Hi,
We are seeing live application logs for auth-clams-tech in the Fly dashboard, but retained logs are completely empty in Grafana and in the app logs API. The app runs in IAD.
What works:
- The dashboard Live Logs view shows current stdout from the app.
- A direct NATS subscription to
logs.auth-clams-tech.>received a controlled, non-sensitive marker written to the running process’s stdout at 2026-07-18 21:04:31 UTC. - Our staging app, built from the same Dockerfile with the same tracing feature and stdout subscriber, has retained logs normally.
What does not work:
- Grafana reports no application logs over 48-hour and 7-day windows.
- The retained logs endpoint returns
{"data":[],"meta":{"next_token":""}}. It still returned zero entries when rechecked on 2026-07-19. - The controlled marker received over live NATS never appeared in retained search.
We have verified the deployed binary includes tracing, RUST_LOG is set, stdout/stderr are pipes, local logging works, and the running production process is producing logs. This appears to fail between Fly’s live NATS stream and the retained Grafana/VictoriaLogs indexing path.
This looks similar to:
Could someone from Fly check whether the retained-log ingestion or app metadata for this app is stuck? Is the missing history recoverable, and is there any app-side action worth taking? Previous reports suggest replacing the Machine does not help.
flyctl: v0.4.71
