fly-log-shipper not working as expected. Have I set it up correctly?

Hey, I’m trying to get the fly-log-shipper working but I no logs seem to be coming through. I’m using logflare. Here’s my fly.toml:

app = "log-shipper"

[build]
  image = "ghcr.io/superfly/fly-log-shipper:latest"

[env]
  ORG = "my-org-name"
  SUBJECT = "logs.>"

I also set the following secrets:

$ fly secrets list                                                                                                                                                                              
NAME                  DIGEST                           DATE
ACCESS_TOKEN          e89b9b37c695f6d480a5e50axxxxxxxx 25m0s ago
LOGFLARE_API_KEY      0723d6cdbb9383d5f1e9b936xxxxxxxx 36m2s ago
LOGFLARE_SOURCE_TOKEN d41d8cd98f00b204e9800998xxxxxxxx 27m33s ago

as per the readme.

My log-shipper app has deployed successfully but no logs seem to be getting shipped to my logflare source.

the logs for the log shipper itself are as follows:

$ fly logs                                                                                                                                                                                       [17:49:29]
2021-12-02T17:25:57.339 runner[12b0d24d] lhr [info] Starting instance
2021-12-02T17:25:57.370 runner[12b0d24d] lhr [info] Configuring virtual machine
2021-12-02T17:25:57.372 runner[12b0d24d] lhr [info] Pulling container image
2021-12-02T17:25:59.472 runner[12b0d24d] lhr [info] Unpacking image
2021-12-02T17:26:02.494 runner[12b0d24d] lhr [info] Preparing kernel init
2021-12-02T17:26:02.944 runner[12b0d24d] lhr [info] Configuring firecracker
2021-12-02T17:26:03.055 runner[12b0d24d] lhr [info] Starting virtual machine
2021-12-02T17:26:03.192 app[12b0d24d] lhr [info] Starting init (commit: 7943db6)...
2021-12-02T17:26:03.216 app[12b0d24d] lhr [info] Preparing to run: `bash start-fly-log-transporter.sh` as root
2021-12-02T17:26:03.228 app[12b0d24d] lhr [info] 2021/12/02 17:26:03 listening on [fdaa:0:3610:a7b:2809:12b0:d24d:2]:22 (DNS: [fdaa::3]:53)
2021-12-02T17:26:03.434 app[12b0d24d] lhr [info] Dec 02 17:26:03.431  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info"
2021-12-02T17:26:03.436 app[12b0d24d] lhr [info] Dec 02 17:26:03.436  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2021-12-02T17:26:03.514 app[12b0d24d] lhr [info] Dec 02 17:26:03.513  INFO vector::topology::running: Running healthchecks.
2021-12-02T17:26:03.515 app[12b0d24d] lhr [info] Dec 02 17:26:03.515  INFO vector::topology::builder: Healthcheck: Passed.
2021-12-02T17:26:03.515 app[12b0d24d] lhr [info] Dec 02 17:26:03.515  INFO vector::topology::builder: Healthcheck: Passed.
2021-12-02T17:26:03.518 app[12b0d24d] lhr [info] Dec 02 17:26:03.517  INFO vector::topology::running: Starting source. key=fly_socket
2021-12-02T17:26:03.518 app[12b0d24d] lhr [info] Dec 02 17:26:03.518  INFO vector::topology::running: Starting source. key=fly_log_metrics
2021-12-02T17:26:03.518 app[12b0d24d] lhr [info] Dec 02 17:26:03.518  INFO vector::topology::running: Starting transform. key=log_json
2021-12-02T17:26:03.519 app[12b0d24d] lhr [info] Dec 02 17:26:03.518  INFO vector::topology::running: Starting sink. key=blackhole
2021-12-02T17:26:03.519 app[12b0d24d] lhr [info] Dec 02 17:26:03.518  INFO vector::topology::running: Starting sink. key=fly_log_metrics_prometheus
2021-12-02T17:26:03.523 app[12b0d24d] lhr [info] Dec 02 17:26:03.519  INFO source{component_kind="source" component_id=fly_socket component_type=socket component_name=fly_socket}: vector::sources::util::unix_stream: Listening. path="/var/run/vector.sock" type="unix"
2021-12-02T17:26:03.523 app[12b0d24d] lhr [info] Dec 02 17:26:03.521  INFO vector: Vector has started. debug="false" version="0.17.2" arch="x86_64" build_id="cc07286 2021-10-19"
2021-12-02T17:26:03.523 app[12b0d24d] lhr [info] Dec 02 17:26:03.521  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
2021-12-02T17:26:03.531 app[12b0d24d] lhr [info] Dec 02 17:26:03.531  INFO vector::sinks::blackhole: Total events collected events=0 raw_bytes_collected=0

Have I done everything correctly? How might I debug? Is there a step-by-step guide for setting this up? Any examples?

Can you try with the latest image flyio/log-shipper? We’ve moved to Docker Hub.

https://hub.docker.com/r/flyio/log-shipper/

It’s possible the ghcr image did not have Logflare, or concatenating the config was busted.

Nice. Yeah using the image from docker hub seems to have sorted it.

Might be worth updating the readme as there’s no obvious instruction of where to get the image from. I got it from looking at the packages page.

Good idea :smiley: