Is it possible to send logs elsewhere, or to grep logs back in time?

Questions / Help
1

Hi,

Is it possible to send logs elsewhere for filtering, querying, etc? Or is it possible to query logs, and not just tail them, with flyctl in a way I don’t see?

In the flyctl help I see:

Logs can be filtered to a specific instance using the --instance/-i flag or
to all instances running in a specific region using the --region/-r flag.

But I don’t see a way to filter back in time, to say query logs that happened a few days ago.

I assume that’s because logs are being streamed to flyctl as they’re being collected.

Thanks,
Leigh

1 Like
2

Here are some threads that might help you out.

and

3 Likes
3

Hi @danwetherald,

I have exactly the same question: fly logs offers to have a real-time, on-the-fly logging, but can it go back in history?

From the links you mentioned, I guess it’s possible to record these logs somehow, but that means they’re not persisted by default anywhere?

Thanks,
Félix

4

The logs are stored, but for a limited time and mostly for diagnostic purposes at this point. Each of our customers has different log retention needs — some don’t need their logs at all and some need to store them forever for compliance reasons — so we make it easy to access the logs and help customers send them over to whichever service they’re most comfortable with for analysis and/or archival.

Depending on what you’d like to do with the logs, I can try suggesting a few options. Either hosted ones like Datadog, Sumologic, Elastic, etc, or self-hosted options on Fly, or archival on S3.

1 Like
5

Thanks!
Where can I see these logs stored by default? Is it possible to access them through flyctl?

6

The stored logs aren’t customer accessible at this point - I’d suggest configuring a destination in log shipper to access your logs.

fly logs is a great way to stream your logs, but we don’t currently have support for history in that command.

7

We’ve had some luck with integrating LogFlare into our Elixir application.
The advantage of this is that you do not need to run a separate node for the e.g. fly-log-shipper.

The disadvantage is that it will only log things that happen while your application is running: Problems during the build/release process, or logs w.r.t. cluster management will not show up.

2 Likes
8

Thanks @sudhir.j for the answer, and thanks @qqwy for the advice!

1 Like
9

Thanks for using Logflare :blush:

Okay we have a pull request in to add Logflare to the fly-log-shipper but you can use our fork until that happens!

I would suggest keeping the Logflare logger going as is, and not using the console Logger backend in your prod Elixir config. The log shipper should pickup everything until Logger gets started, and then after that the Logflare Logger backend will take over.

I would also suggest using two different Logflare sources. One for the Fly logs, and one for the Elixir logs. They will have different payload shapes and it will be easier so as to not clash schemas.

I’ll work on cleaning up the Fly logs a bit with an example fork at some point. Just need to remap things a bit more.

5 Likes
10

I want to be able to grep the last 30 days of logs for 404 responses. Is fly-log-shipper still the right way to do that? I looked at the logging docs but didn’t see it mentioned there.

11

Yes, or to avoid egress costs, you can run a log sink on Fly: Streaming app logs from one Fly app to another for analysis - #2 by kurt

2 Likes