Fly Logs over NATS

41

Just saw the PR and it’s now merged. Thank you!

1 Like
43

For those of you discovering this thread here is a transform and sink for Google Stackdriver

[transforms.gcp_json]
type = "remap"
inputs = ["filter_json"]
source = '''

    . |= parse_regex!(.message, r'^(\[(?P<service>\w+)\]\s+)?(?P<message>.*)$')

    structured =
        parse_json(.message) ??
        parse_apache_log(.message, format: "combined") ??
        parse_common_log(.message) ?? {}

    ., err = merge(., structured)

    if .service == null {
        .service = "system"
    }

    if ends_with(.fly.app.name, "-staging") ?? false {
        .env = "staging"
    } else {
        .env = "production"
    }

    del(.log)
    del(.host)
    del(.event)

    '''

[sinks.gcp]
type = "gcp_stackdriver_logs"
inputs = ["gcp_json"]
credentials_path = "..."
log_id = "{{fly.app.name}}.{{service}}"
project_id = "infrastructure-fly"
severity_key = "level"

# https://medium.com/google-cloud/writing-developer-logs-with-google-cloud-logging-484016c05e16
[sinks.gcp.resource]
type = "generic_node"
location =  "{{fly.region}}"
namespace = "{{fly.app.name}}"
node_id = "{{fly.app.instance}}"

Setup

I am running an S6 multi-process setup with multiple services in a single APP. The logs for each service are prepended with the service name in between brackets (in the S6 log script) and the message is send as json by the internal app.

I’m applying following parsing rules:

  • service: Parsing out the service name from the log message prefix
  • env: Parsing out from app environment based on convention (app name ends with either -staging or -production)

Specific Google Log explorer filters

Google provides 3 log filters:

  • Resource
  • Log Name
  • Severity

Resource

Resource filter is configured by provide resource metadata. I’m configuring the log entry as a generic node: A generic node identifies a machine or other computational resource for which no more specific resource type is applicable. The label values must uniquely identify the node.

Log Name

The Log Name filter is populated with the log_id: A combination of the app name and the service name.

Severity

The Severity filter is configured by defining theseverity_key. In my case Google stackdriver severity is set based on the level field which is parsed from the message string itself, supporting json, apache and common log formats.

Happy logging!

1 Like
44

@Donald, sorry not been able to get back to this for quite some time, glad to hear you resolved it.

For everyone else interested in this issue. Depending on your logs you might need to do additional parsing to get correct levels.

1 Like
45

Sorry, I am completely new to this.

Do I just create a separate application only for this log shipper?
If so, how does it communicate with my actual app then?

Here we have some vector configs and a nats client (fly-logs), along side a wrapper script to run it all, that will subscribe to a log stream of your organisations logs, and ship it to various providers.

Oh okay I should’ve read more carefully. :man_facepalming: Sorry.

46

No problem at all. Getting started is always a bit hard. It’s a separate app indeed. You can easily run it as a 256mb instance.

1 Like
47

Hi,
we’ve been trying to use your NATS gateway in Frankfurt region to no availa. The logs are just not coming in. Any idea how to debug it further? I’ve exec-ed into the container and tried to run fly-logs manually to see if anything is coming through (no output). I know that the api token I’m using is correct since I’m not getting auth error.
We’ve got two organizations and actually the fly-logs works for one but not for the other.

48

Check the dashboard monitoring tab for your app, do you have logs there? If not then this would be a fly issue and you will need to call in re-inforcements :slight_smile: