How can I correlate activity between logs and grafana metrics

I recently set up some Grafana custom metrics, which is great. I want to try to correlate metrics there with data I’m logging from my application.

I set up google cloud logging as a logging alternative to the built in logging with fly. Google cloud logging has the ability to add labels to my logs. I would like to use them to identify which instance I’m logging from, so that I can look at Grafana metrics and correlate them with different versions of my app.

The grafana metrics have the ‘instance’ label, as described in Metrics on Fly.io · Fly Docs. I looked at the docs and found The Fly Runtime Environment · Fly Docs, which is the closest thing to the instance ID, but they seem to have a different format. Are these related? Can I get the instance ID in my application at runtime somehow?

Regarding correlating instance ID to an app, you can do a DNS TXT query to _instances.internal from within an instance, which should return every instance ID running inside your org, along with every app.

Have you checked out transformations - you can run calculations between any data in panels. Good starting point: Transform data | Grafana documentation

Also if you are using say Prometheus metrics, exemplars can become your common connection between logs, traces and metrics. More info here: Exemplars | Grafana documentation

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.