How can I correlate activity between logs and grafana metrics

[moowiz]

I recently set up some Grafana custom metrics, which is great. I want to try to correlate metrics there with data I’m logging from my application.

I set up google cloud logging as a logging alternative to the built in logging with fly. Google cloud logging has the ability to add labels to my logs. I would like to use them to identify which instance I’m logging from, so that I can look at Grafana metrics and correlate them with different versions of my app.

The grafana metrics have the ‘instance’ label, as described in Metrics on Fly.io · Fly Docs. I looked at the docs and found The Fly Runtime Environment · Fly Docs, which is the closest thing to the instance ID, but they seem to have a different format. Are these related? Can I get the instance ID in my application at runtime somehow?

[billy]

Regarding correlating instance ID to an app, you can do a DNS TXT query to _instances.internal from within an instance, which should return every instance ID running inside your org, along with every app.

[grafinger]

Have you checked out transformations - you can run calculations between any data in panels. Good starting point: Transform data | Grafana documentation

Also if you are using say Prometheus metrics, exemplars can become your common connection between logs, traces and metrics. More info here: Exemplars | Grafana documentation