Fly Log Shipper Duplicate Logs on Apps v2

[jfrydman]

Hello,

I was getting duplicate log entries when using the Fly Log Shipper app (shipping to Logtail) and couldn’t figure out why until I realized that starting a new app on Apps v2 always creates two instances.

For now I fixed it by running fly scale count 1 under my logshipper app, so there was just one instance running.

The current docs don’t mention anything about this “gotcha”:

1. Exporting Logs · Fly Docs
2. Shipping Logs · The Fly Blog

Given Fly recommends having two instances via the warning message when viewing the app that says

Your app is running on 1 machine. We’d strongly recommend running at least 2 instances to ensure high availability. Check out the documentation for more details on scaling.

What’s the ideal way this should be configured without shipping duplicate logs?

[francoab]

@jfrydman

SUBJECT = logs.<app_name>..<instance_id>

You add add the specific instance ID to only get logs from there.

[env]
  SUBJECT = "logs.sandwich.dfw.00bb33ff"

More info:

Export logs

Stream logs to local volume

You’re welcome. For B, I meant to start fly logs in the background from your startup command, for example by setting the command to (untested) fly logs -i "$FLY_ALLOC_ID" >>/path/to/log 2>&1 & ORIG_CMD. So you don’t have to manually ssh in, and when the VM restarts the logs process will be started automatically. For A, here is a quick demo of using ts to add timestamps: $ apt-get install -y moreutils $ ./app Waiting for request... Processing request... Request complete $ ./app 2>&1 | ts Apr 15…

[jfrydman]

Hey @francoab,

This doesn’t solve the problem I explained. I want logs from all my app production instances, I just want them once.

The issue is that Fly recommends having two instances of the same app for high availability and that following the doc instructions for the fly logshipper app does not mention that by running fly launch you’d spin up two instances of the logshipper app, which in turn would ship the logs twice.

Either the docs are outdated and they actually recommend only having one instance of the logshipper app, or that app needs to somehow be updated to know if other instance had already shipped the log.

[charsleysa]

Hi @jfrydman

Yea the change to deploy two instances by default in my opinion was a bandaid for users who don’t understand what high availability requires. Fly have also dropped the ball on updating all previous docs that were impacted by this change.

Luckily a fix for you should be pretty simple, you just need to set another environment variable. Set QUEUE to pretty much any value and you should be good to go.

E.g.

[env]
  QUEUE = "org-logs"

[francoab]

Just submitted a PR to add this to the documentation, thanks so much for pointing it out!

PR: Update Fly Log Shipper Queue documentation by francoabaroa · Pull Request #820 · superfly/docs · GitHub

[tom93]

I think @charsleysa’s suggestion (QUEUE) is the proper solution for the Log Shipper, but I just want to mention that instead of fly scale count 1 you can deploy using fly launch --ha=false (which I think is cleaner).