Streaming app logs from one Fly app to another for analysis

Hello! I am running a personal website on Fly.io using Caddy to serve a generated static site. I wanted to do a bit of analysis over the server logs, using something like Goatcounter or Goaccess.

My initial attempt with Goatcounter ran into issues: I wanted to self-host Goatcounter, so I had to run it as its own app. I attempted to use Caddy’s net log handler to write to goatcounter-app.internal:9090, while adding ncat -l 9090 > /data/caddy.log & to the launch script of my Goatcounter instance to try and read that stream to a file. I could see logs from Caddy on stdout, but none showed up in the caddy.log file on the Goatcounter host, so I imagine I must have done something incorrect there.

My next plan is to try running Goaccess locally on the same VM as my Caddy instance, so that it can inspect a logfile directly.

I am curious if anyone has attempted anything similar, and whether or not there are any recommendations for sharing log data between Fly apps.

You can subscribe to logs from another app in the same organization using NATs: Fly Logs over NATS

@dusty built an example app that subscribes to NATs and stores logs with Loki, which sounds pretty close to what you’re trying: GitHub - fly-apps/fly-log-local: Store Fly app logs locally.

2 Likes

Neat! Too bad we already impl a custom log-sink with AppRunner and Firehose + S3.

Btw, there needs to be “one-click” deploy button for (such uber useful) fly-apps!


edit: Just read the source – a vector.toml, a dockerfile, a loki.yml and that’s pretty much it; so succinct. The pieces just seem to fit together like lego blocks!