Successful forwarding of metrics *and* logs to DataDog?

Hi there,

I’ve browsed various threads here around DataDog, and I’ve now successfully got log forwarding working via fly-log-shipper and metrics via having deployed a DD Agent that my backend service reports profiling/metrics to. This technically works, but is clunky for multiple reasons:

  1. It requires two fly applications, one for fly-log-shipper and one for the datadog agent. This could technically be solved by deploying both into a single VM, probably…
  2. Because the logs aren’t being read by a DD agent and shipped that way, the logs don’t naturally carry any metadata about e.g., service name. While you can inject this yourself (I use logrus and Go and so can use WithFields({}) for that) for logs you emit, other logs to stdout/stderr from the software or on the host in general will not carry this metadata and so arrive untagged.

The main issue with forwarding logs via fly-log-shipper is that it doesn’t augment the logs with the metadata needed in DataDog to e.g., tie APM together with log lines and provide additional features: ddsource, hostname, service and environment fields specifically. You can tweak the log_json transform to add hostname, environment and service as you can rely on .fly.host and .fly.app.name and an env variable for environment as such:

  type = "remap"
  inputs = ["fly_socket"]
  source = '''
  . = parse_json!(.message)
  .hostname = .fly.host
  .environment = get_env_var!("ENV")
  .service  = .fly.app.name
  '''

I’m struggling even with the above to find a way to e.g., define specific metadata on a per-app basis, since the logs are being sourced from all apps in the org. If I wanted to, for example, have both a Go app and a Python app, there is no real way to e.g., set .ddsource appropriately in Vector.

Has anyone found a more adequate way to do this? Has Fly been in touch with DataDog perhaps, about an integration? I’m quite keen to see this work, as DataDog is hugely valuable and its a bit of a blocker for me in terms of Fly to be stuck with Prometheus/Grafana :confused:

Hello!

  1. To use just a single app to send both logs and metrics, we could try also sending metrics through fly-log-shipper by adding a datadog_metrics sink.
  2. Augmenting the datadog sink with the relevant metadata in datadog’s standard reserved tags makes sense- we could add a datadog_json transform that takes fly_socket as its input, and set the tags there as you described.

I’m struggling even with the above to find a way to e.g., define specific metadata on a per-app basis, since the logs are being sourced from all apps in the org. If I wanted to, for example, have both a Go app and a Python app, there is no real way to e.g., set .ddsource appropriately in Vector.

To inject specific per-app metadata, you could try adding extra VRL to the remap transform to set ddsource based on fly.app.name:

if (.fly.app.name == "my_go_app") {
  .ddsource = "go"
} else if (.fly.app.name == "my_python_app") {
  .ddsource = "python"
}

A bunch of conditionals like this is a bit clunky, maybe we could define the .fly.app.name → metadata mappings using an enrichment table CSV or something, but it will need to be configured somewhere.