Send NGINX access.log logs through fly-log-shipper

Hey guys,

I was reading the “Shipping Logs” (Shipping Logs · Fly) blog post and decided to try out the service. I followed the instructions to setup fly-log-shipper and it seems to work great with JSON-formatted logs out of the box.

However, some of my services output NGINX logs in a combined format, which ingests as is, without further destructuring of the log message. I looked through fly-log-shipper code. I’m not very familiar with NATS and Vector, but as far as I can understand fly-log-shipper expects logs from apps in JSON format and otherwise treats them as plain text.

NGINX log format seems pretty common to me, is there an easy way to process it via fly-log-shipper somehow? I suppose I can make NGINX spew out JSON logs instead, but I’m wondering whether I can do that centrally on the fly-log-shipper.



Good question, I also am not extremely familiar with Vector (I wrote that article tho :sweat_smile:).

:one:Based on some research, I think getting Nginx to output JSON logs is your best bet. Here are two examples of how to do that:

  1. Tech Blog: How to configure JSON logging in nginx? - Velebit AI
  2. How to View and Configure NGINX Access & Error Logs | Better Stack Community

:two: The other way to go about this is to customize the Vector configuration, which would mean forking the log shipper repository and editing the Vector toml config. This is likely a bunch more work than the previous option, mostly because it involves googling Vector configs and it seems like there’s a bunch of opinions/ways to go about that. (for example, figuring out wtf is going on here).

1 Like

Thanks for your input!

I ended up going the route #1, thankfully it turned out to be pretty simple: I use Kong and all I needed to do is to add new env var (KONG_NGINX_HTTP_LOG_FORMAT="json_format '{…}'") and update a couple of other env vars to pass the new log format name to them. This still took me half a day to figure out though because documentation is not great.

I tried looking at route #2 and yeah, you’re right that it looks like way more work so I bailed out.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.