Debugging fly-log-shipper

I’ve deployed fly-log-shipper and fiddled with settings until I managed to get it starting without dying.

Now it just sits there doing nothing. Logs are being generated in another app in my org, but nothing is showing in the log-shipper instance’s logs, and nothing is being saved to my S3 bucket.

Any suggestions how to debug this? It might be handy to have a stdout option for fly-log-shipper so it’s possible to check if anything is being received…

You could connect to your log-shipper instance via ssh (flyctl ssh console) and run vector tap (see Introducing `vector tap` | Vector).

Thanks @tosie

It looks like the correct command would be:

vector tap fly_socket

However, nothing’s coming through.

It seems to have randomly started working :person_shrugging:

Logs are now visible with vector tap fly_socket, and also appearing in the bucket configure in sinks.aws_s3 in the vector.toml file :pray:

Some of our nats servers weren’t sending messages across regions in our cluster (and not so helpfully not complaining about not being able to do it because that’s how nats works). I restarted the process everywhere to force all of the servers to rejoin with each other and it’s started the streams flowing again.

Now i’m looking in to ways to detect this automatically!

@steveberryman thanks for the info! I guess I was unlucky trying to get this working at exactly that moment :face_holding_back_tears:

Does fly have any plans for offering log aggregation/storage/search as a core service?

1 Like

does fly-log-shipper have any documentation anywhere?

I can’t seem to get this working for datadog or logflare at all.

Hi @adam1

The best resource for information on fly-log-shipper would probably be the github README or community posts.

@adam1 I found I had to build from source to get proper support for the features I needed. If you’re using docker, double-check which version of fly-log-shipper is actually being loaded, and compare that with the tagged versions in git, and master.

Yeah, the readme tells me which env vars to set but that’s about it. I’ve set the vars but nothing is happening.

I’ve made 6 separate apps now. One of them sends metrics to datadog, none of them send logs. I’ve followed datadogs examples, spoken to their support etc but nothing works. I guess the fact that there is no actual documentation .and that the issue reporting is switched off for this repo is a bit of a sign.

tbh, I’ve wasted weeks on this now. made several attempts with no luck. I wondered if there was any actual documentation for this that might give me some clues, but as there isn’t I think I’ll just stick with the logflare elixir package instead.