fly-log-shipper sending to Loki (Grafana cloud) - Internal log [Events dropped] is being rate limited

Questions / Help
1

Hi,

I deployed fly-log-shipper using the following fly.toml:

app = 'flyio-loki-shipper'
primary_region = 'sin'

[build]
  image = 'ghcr.io/superfly/fly-log-shipper:latest'

[services]
  http_checks = []
  internal_port = 8686

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

I set the secrets:

$ fly secrets list
NAME         	DIGEST          	CREATED AT 
ACCESS_TOKEN 	xxx	                27m18s ago	
LOKI_PASSWORD	xxx	                27m7s ago 	
LOKI_URL     	xxx	                27m7s ago 	
LOKI_USERNAME	xxx              	27m7s ago 	
ORG          	xxx	                27m20s ago	
PORT         	xxx	                27m7s ago

I confirmed that these credentials+URL work for sending to Grafana cloud separately.

However I see errors about vector (?) being rate limited in fly logs and no logs are being sent to Grafana:

2024-05-01T11:15:14Z app[3d8ddd5fed9d78] sin [info]2024-05-01T11:15:14.731421Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=17}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2024-05-01T11:15:14Z app[3d8ddd5fed9d78] sin [info]2024-05-01T11:15:14.731527Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=17}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being rate limited.
2024-05-01T11:15:14Z app[3d8ddd5fed9d78] sin [info]2024-05-01T11:15:14.731617Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=17}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being rate limited.
2024-05-01T11:15:14Z app[d8d9567ce12998] sin [info]2024-05-01T11:15:14.733196Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=11}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being rate limited.
2024-05-01T11:15:14Z app[d8d9567ce12998] sin [info]2024-05-01T11:15:14.733282Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=11}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being rate limited.
2024-05-01T11:15:14Z app[d8d9567ce12998] sin [info]2024-05-01T11:15:14.733328Z ERROR sink{component_kind="sink" component_id=loki component_type=loki component_name=loki}:request{request_id=11}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being rate limited.

My app only produced a handful of log lines in this time so I don’t think my app is the issue.

Any ideas?

Thanks.

2

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.