Cleaning up logs (Log Shipper, Postgres)

Questions / Help
1

Hey guys,
We are going live with Fly with an analytics app. We have about 13 services so it would be nice to get logging system going. We are using Log Shipper an have that working as designed.

However, there is an issue with noise and a lack of options for dealing with it…

For example, the standard Fly Postgres instance pushes uninteresting logs multiple times a second:

2022-07-26T15:01:29.549 app[1a3edf2a] lax [info] keeper | 2022-07-26T15:01:29.549Z INFO cmd/keeper.go:1675 postgres parameters not changed

2022-07-26T15:01:29.549 app[1a3edf2a] lax [info] keeper | 2022-07-26T15:01:29.549Z INFO cmd/keeper.go:1702 postgres hba entries not changed

2022-07-26T15:01:34.737 app[1a3edf2a] lax [info] keeper | 2022-07-26T15:01:34.736Z INFO cmd/keeper.go:1504 our db requested role is master

2022-07-26T15:01:34.738 app[1a3edf2a] lax [info] keeper | 2022-07-26T15:01:34.738Z INFO cmd/keeper.go:1542 already master

Is there a way to filter these types of logs before they get pushed?

Possible to silence the logging on postgres or a single app within the org?

2

As a tip, logging is going to be an important issue for 99% of your paying customers with real-world apps.

As good a job as fly is doing in everything else, this area seems to be underserved and under documented.

@jsierles

3

The easiest way to filter things with the log shipper is by adding a Filter Transform to the log shipper vector config to filter on either the .message or .fly.app.name fields and exclude the logs you don’t want shipped.

We’re actively working on improving our log services in many ways so feedback like this is really useful!

4

thanks @steveberryman …

There is a problem with that. Right now we are just using the Docker container you guys posted for Log Shipper.

So to customize we’d need to: fork the whole thing, identify files for editing (there are no docs), learn about vector (reading all the vector docs), and spend ~20 man hours on trial and error to get things working…

So for now we just have to take what we’re given!

If it were easy for you guys, I’d highly suggest a quick docs update/rewrite and possibly a few options for the docker container for common needs like filtering and refining output.

5

Yes, sorry it’s definitely not ideal right now! We’re a small team and working on many different projects in various areas. Improving the log tooling is high on the list and we should get to it soon!

Until we do, if you did want to deploy a customised version of our log shipper, all of our config including the Dockerfile and fly.toml are here: GitHub - superfly/fly-log-shipper: Ship logs from fly to other providers.

adding a filter should be fairly straight forward. I think this should work for you to exclude logs from specific apps:

[transforms.log_filter]
type = "filter"
inputs = [ "log_json" ]
condition = 'includes(["postgres-app-name"], .fly.app.name)'

You’d also need to change the inputs in the corrisponding config in the sinks/ directory for whatever sink you were using to log_filter rather than log_json.

6

@arpowers not directly related to log shipping, but the latest Postgres images we use are less chatty with logs. If you run fly image update you should get that version.

7

thanks @kurt that might get the job done.

also thank you @steveberryman I appreciate the details.

If it’s any consolation, really impressive work nearly every where else guys. I’ve definitely been evangelizing the company as “disruptive” and “break through” tech at least for me…

now, if only it could handle logs :wink: