Viewing logs older than 100 lines ago

1

I’m trying to debug something that happened about 14 hours ago where it seems like one of our services died and did not start back up until about 4 hours ago.

Running fly status --all gives me this:

Instances
ID       VERSION REGION DESIRED STATUS  HEALTH CHECKS RESTARTS CREATED
23524f06 10      sin    run     running               13       4h24m ago
048e3cf1 10      sin    stop    failed                1        2021-01-31T22:29:40Z

while running logs on 23524f06 with fly -i 23524f06 gives me only the last 100 lines, which since this is a particularly chatty application, only covers the past 30 minutes :sweat_smile:

Is there any way to see logs further back than that?

P.s. not sure if it’s related to the issue I’m debugging, but the metrics page in the dashboard is also not showing full stats even when I have it set to 1d prior. Only data transfer is showing for the full 1d, while firecracker load and memory is only showing from 4 hours ago.

Screenshot 2021-02-04 at 12.22.21 PM
Screenshot 2021-02-04 at 12.22.21 PM1154×779 73.2 KB

1 Like
2

There isn’t yet. Logs are considered “best effort” from us until we figure out a way to make them more reliable and probably charge for retention or allow 3rd party apps to ingest them.

That said, I might be able to allow more logs to be fetched via flyctl.

Looks like a bug in our UI. Thanks for reporting this :slight_smile:

1 Like
3

+1 for accessing logs from further back. Currently I’ve been experimenting with pushing to AWS Cloudwatch. Which works but isn’t ideal. And it only covers app-made logs, not anything related to fly itself.

Even 24 hours worth would be handy. Then you could auto-delete them. Since an issue should be spotted within 24 hours and that’s when the logs are most useful. For me at least.

2 Likes
4

Has there been any progress made in exporting logs to a third-party? I was thinking of something like a vector.dev vm (or anything really) listening for log events through the the fly api( which means customers would be able to roll their solution) for the whole org, enrich them and ship them off to a service like datadog. I used vector as an example since you are already using and it also does metrics withy to many movings parts.

5

Where do you want to send them? Building the plumbing for this is more complex than I’d like, but we have some simple ideas to try.

6

It’d be dope to be able to subscribe to logs on a dedicated NATS topic. We can hook that up to any storage solution afterwards.

7

That’s my pipe dream! Most people just want to forward logs to LogDNA and DataDog though. :smiley:

8

I thought maybe you could stream through the API and have and since vector supports http sources I could deploy it like just another app and have it forward to datadog and co. But the NATS idea sounds even better.

9

Is there any progress on the side of exporting logs to a central source? I there a way I could use the method described here: Logging | Uptrace using vector? Could I perhaps deploy a vector container and have it query the fly.io api for all my apps logs as described in https://vector.dev/docs/reference/configuration/sources/http/

10

uptrac.dev is where I want to send my logs

11

We’re working on this now! We have some experimental infrastructure for publishing logs through NATs. We’ll get you access as soon as we can.

1 Like
12

Is there any update on this? We’re bumping into this issue and would like to retain logs further out.

13

you can subscribe to logs now (via NATS)
I use slightly modified GitHub - superfly/fly-log-shipper: Ship logs from fly to other providers but it should work “out of the box” now
(shipping and storing logs is still on you tho, but there are lots of options)

1 Like
14

+1 for a little more as the default. I just had a crash right now and fly logs isn’t enough lines to show me the top of the stack trace.

3 Likes