Log Shipper - Filter Logs through Vector config

Questions / Help
1

I’m using the log-shipper to ship out our logs. There are some logs that we would like to be able to filter, and we have a property (customlevel=ignore) that is meant to signify that the log should be ignored. I’m attempting to filter them in the vector.toml file using a Filter Transform, but it’s not working. I’ve tried lots of different options for the conditions, different fields, etc and have not been able to get any logs to filter.

I’m new to Vector so it could very well be something I’m doing wrong. Anything I’m missing here, or would there be anything with the log-shipper that would prevent log filtering in general?

Here’s the relevant portion of the vector.toml, with a bit of custom logic in the log_json remap section and then the added Filter transform (side note the remap logic adding the “env” property works fine):

[transforms.log_json]
  type = "remap"
  inputs = ["fly_socket"]
  source = """
    . = parse_json!(.message)
    if (ends_with!(.fly.app.name, "-staging")) {
      .env = "staging"
    } else if (starts_with!(.fly.app.name, "fly-builder")) {
      .env = "fly"
    } else {
      .env = "prod"
    }
  """

[transforms.filter_ignores]
  type = "filter"
  inputs = ["log_json"]
  condition = '.customlevel != "ignore"'

Thanks in advance for any thoughts or advice!

2 Likes