logs: prometheus grafana loki log-shipper set up

Questions / Help
,
1

Hey there, I’m trying to replicate a docker-compose setup in our fly.io instances.
I’ve installed grafana and it’s pointing to prometheus (fly-config) [working-communicating properly].
Then installed and instance of loki and log-shipper(from fly, tried both from github repo and docker image).
Everything looks good, but I’m not receiving anything (no logs found) when I explore loki.
I’m assuming there is a configuration issue but I’m not sure where.
Can anyone give me a hand?

dockerfile for grafana

FROM grafana/grafana:8.2.2
USER root
ENTRYPOINT ["/run.sh"]

toml file for grafana

app = "grafana"

[mount]
  source = "grafana_storage"
  destination = "/var/lib/grafana"

[env]
  GF_INSTALL_PLUGINS = "grafana-worldmap-panel,grafana-clock-panel"

[[services]]
  internal_port = 3000
  protocol = "tcp"

[services.concurrency]
  hard_limit = 25
  soft_limit = 20

[[services.ports]]
  handlers = ["http"]
  port = 80

[[services.ports]]
  handlers = ["tls", "http"]
  port = 443

[[services.tcp_checks]]
  interval = "10000"
  timeout = "2000"

dockerfile for loki

FROM grafana/loki:2.5.0
COPY loki-config.yaml /etc/loki/local-config.yaml
USER loki
ENTRYPOINT [ "/usr/bin/loki" ]
CMD ["-config.file=/etc/loki/local-config.yaml"]

toml file for loki

app = "loki"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[mount]
  source = "loki_storage"
  destination = "/data/loki"

[env]

[[services]]
  internal_port = 3100
  protocol = "tcp"

[services.concurrency]
  hard_limit = 25
  soft_limit = 20

[[services.ports]]
  handlers = ["http"]
  port = 80

[[services.ports]]
  handlers = ["tls", "http"]
  port = 443

[[services.tcp_checks]]
  interval = "10000"
  timeout = "2000"

toml file for log-shipper

app = "log-shipper"

[build]
  image = "flyio/log-shipper"

[metrics]
  port = 9598
  path = "/metrics"

Possible issues ?
The loki instance does not have any user/password (not a cloud instance)
Loki config file ?

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 15m
  wal:
    enabled: true
    dir: /data/loki/wal
    flush_on_shutdown: true
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2022-06-01
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /data/loki/index
  filesystem:
    directory: /data/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 14d

table_manager:
  retention_deletes_enabled: true
  retention_period: 14d

ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://alertmanager:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

Thank you so much!

2

In case it’s helpful: I believe I had a similar situation. In my case, the solution involved properly setting the LOKI_URL env var in fly-log-shipper.

Here’s a link to the thread for reference: