Ship logs from multiple apps

I’m using fly-log-shipper to ship logs to Datadog. Seems to be working quite nicely, especially after I figured out how to get my Spring Boot app to output JSON logs; Datadog now detects the log level correctly.

I want to ship logs from my production and staging apps. So, at the moment, I have two fly-log-shipper apps. One has subject logs.<production-app-name>.*.* and the other has subject logs.<staging-app-name>.*.*

Is there a way to set the subject, or some other way, so that I can achieve this with only one fly-log-shipper app? I would particularly like to exclude logs from the free builder, which is a little noisy.

If you don’t set the subject, logs from every app in the organization will ship by default. Could you exclude specific log patterns in Datadog?

I could do that. Datadog (and other services?) charge based on how much data they receive. So that’s why I’ve chosen to filter at source.

Some services (like LogDNA/Mezmo) will discount logs that you explicitly filter out. Is there a different price for logs that don’t get indexed?

1 Like

Hi @wrightak

If I understand you correctly, this vector filter should work. Unset your $SUBJECT and modify 18-23 with something like:

[transforms.remap]
  type = "remap"
  inputs = ["nats"]
  source = '''
  . = parse_json!(.message)
  '''

[transforms.log_json]
  type = "filter"
  inputs = ["remap"]
  condition = '.fly.app.name == "production-app-name" || .fly.app.name == "staging-app-name"'
2 Likes

Yeah, I’d like to know the answer to this. It wasn’t clear in the docs I read so I have contacted their support. Thanks!

This was exactly what I was looking for. I’ve just confirmed it works and I’ve also learned more about vector transforms. Thank you!