Fly.io's Log Shipper attaching unwanted fields on logs

1

I’m using fly.io’s log shipper to ship to Datadog, and while it is working, its adding a bunch of fields to my audit logs which I don’t want.

Its also adding an extra field that is
{
logs: {
level: “info”
}
}
regardless of actual log level.

Is there a way to configure the log shipper so that it only sends what it sees from stdout and no other fields.

3

Hi… The Vector configuration file mentioned in the docs can apparently do pretty extensive transformations (VRL). Here’s a fairly complicated example, from a fellow user:

https://community.fly.io/t/structured-logging-with-fly/10489/4

That del operation in particular might be what you’re looking for, :thought_balloon:.

Hope this helps a little!

4

I can’t seem to get this to work as some guest_path is erroring out when specifying a custom vector file:

Failed to update machines: failed to update machine e286712ae26d28: failed to update VM e286712ae26d28: invalid_argument: guest_path must be absolute

5

Error: failed to read file-local: guest path, /etc/vector/vector.toml, must be absolute

6

@MarSan031 please share the exact command you’re using when that error gets thrown.

  • Daniel
7

Failed to update machines: failed to update machine e286712ae26d28: failed to update VM e286712ae26d28: invalid_argument: guest_path must be absolute

Occurs with fly deploy --file-local="/etc/vector/vector.toml=./vector.toml"

Error: failed to read file-local: guest path, /etc/vector/vector.toml, must be absolute

Occurs with
fly deploy --file-local="/etc/vector/vector.toml=C:\Users\sugar\GolandProjects\noted\cloud\logshipper\vector.toml"
which is my absolute path for the vector (on windows)

8

a similar error happens if I git clone the repo and use the dockerfile in the [build] section of the fly.toml instead of the fly-log-shipper:latest image.

9

fly deploy --file-local="/etc/vector/vector.toml=./vector.toml" works for me.

$ fly deploy -a ta-shipper-test --file-local="/etc/vector/vector.toml=./vector.toml" --detach
==> Verifying app config
Validating /src/roadmr-log-shipper/fly.toml
--> Verified app config
==> Building image
Searching for image 'ghcr.io/superfly/fly-log-shipper:latest' remotely...
image found: img_dozgpe15q1g4096y

Watch your deployment at https://fly.io/apps/ta-shipper-test/monitoring

-------
Updating existing machines in 'ta-shipper-test' with rolling strategy

-------
 ✔ [1/2] Cleared lease for d8d99e4be44058
 ✔ [2/2] Cleared lease for 6833ee9cd66248
-------

I wonder whether Windows is doing something weird with the quotes, path separator (try using \ instead of / on the guest path) or = separator (I’m on Linux). Also make sure you have the latest version of flyctl. Other than that, this should work and it does for me with the exact command you ran.

  • Daniel
10

Totally is a windows problem because with the same codebase,
fly deploy --file-local="/etc/vector/vector.toml=./vector.toml"
gets that guest_path error, but when I just ran it on my Macbook it worked.

As an FYI, I was using Git Bash which attempts to be Unix like, but it also fails in powershell.

11

Also your buddies at Datadog recommend doing this:

Which appears to break because a single quote is considered illegal or something in the .ddsource.

I got it to work with this:

  type = "remap"
  inputs = ["nats"]
  source = '''
  . = parse_json!(.message)
  .ddsource = "fly-io"
  .host = .fly.app.instance
  .env = "production"
  '''
12

Also, the original problem of fly.io attaching excess fields to my audit logs, which I need for compliance, is still happening, which is why I needed to manipulate the vector.toml in the first place.

I have no idea how to work a vector.toml, so if anyone can lead me in the right direction or provide me documentation on how to remove these fields from the logs:

event / event.provider
fly / fly.region / fly.app / fly.app.instance / fly.app.name
hostname
log / log.level

Thanks for everything.

13

I don’t use Vector myself, but from their docs (the “VRL” link above), it looks like del(.event) would be worth trying.

del(.user_info)

This program changes the contents of each event that passes through this transform, deleting the user_info field […]

Something analogous appears in the older forum example, up there, as well.

Possibly you need del(.event.provider) first, to empty things out…

14 
[transforms.log_json]
  type = "remap"
  inputs = ["nats"]
  source = '''
  . = parse_json!(.message)
  .ddsource = "fly-io"
  .host = .fly.app.instance
  .env = "production"

  del(.log.level)
  del(.event.provider)
  del(.fly.app.instance)
  del(.fly.app.name)
  del(.fly.region)
  del(.hostname)
  del(.env)
  '''

Is the solution. Thanks for everything everyone.