Fly logs to papertrail/logdna

Questions / Help
1

What is the recommended way of piping logs to services like papertrail/logdna?

I read this post which links to https://github.com/superfly/fly-log-shipper but TBH I think that assumes some knowledge that I don’t think I have. I wasn’t sure where to start with that.

In my case I’m running an Elixir app, so is it best to just use a Elixir package like https://github.com/larskrantz/logger_papertrail_backend, or is there a more “Fly” way of doing things that doesn’t require changes to the app?

2

That repository does require some deeper-than-we’d-like knowledge.

The flow looks a bit like this:

  • Clone the fly-log-shipper repository
  • Create an app w/ flyctl apps create --no-config --name <your-app-name>
  • Replace the app name in fly.toml with the one you just created
  • Set ORG as an env variable in [env] in the fly.toml, it corresponds to the organization from which you’d like to pull logs.
  • flyctl secrets set ACCESS_TOKEN=$(flyctl auth token) LOGDNA_API_KEY=<your LogDNA API key>
  • flyctl deploy

You should now be set.

We should make that a “launcher” (a bit like our postgres setup).

3

Thank you that’s helpful.

Once I’ve created this app, does it simply share the logs from all other apps in the organisation to Logdna?

4

It does, yes :slight_smile:

You can specify which specific app you’d like by setting a different SUBJECT env var. This takes a NATS subject.

The format is: logs.{app-name}.{region}.{instance-id}. Using logs.> means you want logs from all regions for all apps and instances. If you just want 1 app, you can use logs.{app-name}.>.

Crash course:

  • Subject “parts” are delimited by dots (.).
  • * in a NATS subject means “match everything for this part of the subject”
  • > means “match everything from any number of subject parts”
5

Thanks. I’ll give it a go :slight_smile:

6

@jerome I was able to get this running, and delivering to S3. Thanks for working on this. Can I deploy another instance to send the same org logs to a different sink or do I just add other secrets to the existing one?

7

You can use multiple sinks with one instance. You can also deploy a second instance, totally up to you.

8

Thanks @Kurt. Got it hooked up to aws and erasearch. Thank you and the team for putting that together!

9

Is --no-config still valid, or am I doing something wrong?

$ flyctl apps create --no-config --name enaia-logs
Error: unknown flag: --no-config
Usage:
  flyctl apps create [APPNAME] [flags]
$ flyctl apps create enaia-logs --no-config
Error: unknown flag: --no-config
Usage:
  flyctl apps create [APPNAME] [flags]

A launcher would be great, but as an interim step, up-to-date instructions in the README for fly-log-shipper would be helpful.

10

Thanks a lot for this, I got it up and running too and it works like a charm.

I had to remove the --no-config flag from the flyctl command otherwise the instructions were all good :slight_smile:

11

I am keen on starting to ship app logs to Honeycomb. Is this the recommended way today? This looks useful: GitHub - superfly/fly-log-shipper: Ship logs from fly to other providers

Thanks!

12

Yes, using fly-log-shipper is the least friction way to ship logs out of Fly (see this post for 2 other approaches), though using it isn’t without issues, it’d seem: Fly log shipper not shipping logs

Looks like fly-log-shipper indeed supports honeycomb.

13

Thanks for all of the help on this thread. I thought that I was successfully sending logs from all apps to Papertrail, but logs from some of my apps aren’t getting through. I checked the logs for my fly-log-shipper instance and saw this:

2022-08-20T07:02:09.645 app[645be35a] nrt [info] 2022-08-20T07:02:09.644563Z ERROR sink{component_kind="sink" component_id=papertrail component_type=papertrail component_name=papertrail}: vector::internal_events::tcp: TCP socket error. error=Connection timed out (os error 110) error_code="socket_failed" error_type="writer_failed" stage="sending" internal_log_rate_secs=10

Any ideas on what caused this and how to fix?

Ironically, this log went through to Papertrail, but logs from my other apps don’t appear. I don’t have anything set for the SUBJECT env variable.

14

Is there a way to control the app name that is sent to LogDNA? I see vector being used for all the applications, which is not ideal…

EDIT: yes, it can be done by modifying the vector.toml config as follows:

[transforms.log_json]
  type = "remap"
  inputs = ["nats"]
  source = '''
  . = parse_json!(.message)
  .app = .fly.app.name
  '''

The new part is the second to last line, the rest is unchanged.