Local Logging Setup

1

Hey everyone,

I’ve put together a simple app to store Fly logs on an attached volume. This uses Vector to read the Fly NATS endpoint and Loki as the datastore. It offers configurable log retention and should help with troubleshooting app issues ( think OOM alerts ). I know many of you have already come up with your own log solution at Fly but this might help others looking for a quick solution. One thing to note is that this will not backfill any prior logs.

The README guides you through a quick setup and how to query but further customization is possible. I believe the default settings should work for most but adjustments may be required for heavy log flows.

Let me know if you run into issues or have any additional thoughts.

7 Likes
2

Will this do anything about the main problem I have with Fly.io logging, which is the fact that it doesn’t log everything? There’s a big delay between when a message is printed to the console, and most are not printed, for some reason.

3

Hi @Aimjock

What types of messages are being dropped? Also, any details about your app, etc would help us troubleshoot.

4

I run a Twitch bot which connects to Twitch using the twitchio library. Most print statements aren’t in the fly logs logs/monitoring on the website. If I send a message to the bot, it should print to the console, which it does when running locally.

5

How do you actually connect the fly-log-local app to the main app I would like to store logs for?

6

@titocosta I believe you just run it (as per the README) and it will collect the logs from all your apps.

Edit: To only collect logs from a specific app, you can set the SUBJECT environment variable (docs).

1 Like
7

I keep getting Tailing logs failed: read tcp 127.0.0.1:53703->127.0.0.1:3100: read: connection reset by peer errors when I run the command logcli query -z UTC -q '{forwarder="vector"}' -t