Logs not shipping for v2 apps?

Questions / Help
1

I have deployed the log shipper using the guidelines here. However, the only logs being “shipped” are from the log shipper service itself.

The service does not seem to be picking up logs from any other service in my org even though I have set the ORG env var. I do have multiple orgs within my account. When I run fly logs for my other services, I do see logs coming through the console.

I also tried just forwarding all raw logs to stderr by deploying my own version of the shipper as follows:

[sinks.my_sink_id]
type = "console"
inputs = [ "nats" ]
encoding.codec = "raw_message"
target = "stderr"

Still no dice - the only logs coming through the shipper are from the shipper service itself.

Is there something I am missing, or a step I can take to debug? I have enabled the public graphql port for the shipper, though I’m not sure what to look for (vector top connects but only gives a blank screen as output).

2

We’re seeing the same thing over here too. None of the apps that have been updated to v2 apps are shipping logs any longer.

3

Can you both report which region your shipper is deployed in?

4

bos

5

I figured it out! The error was on my end.

My endpoint (HTTP_URL) was returning a 400 status code. I did not realize, however, that this failure would prevent vector from sending additional logs. I’d assumed that vector would send logs in spite of the failure condition. Once I returned a 200 status code, logs started to flow through again. The key for me was to better understand vector healthchecks.

Really appreciate your taking a look!

2 Likes
6

@jsierles sorry for the delay!

our shipper is deployed in EWR. The logs on that shipper app all look like this:

2023-08-23T06:36:53.047 app[5683d704ce708e] ewr [info] 2023-08-23T06:36:53.047360Z WARN sink{component_kind="sink" component_id=logtail component_type=http component_name=logtail}:request{request_id=96223}:http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true

2023-08-23T06:36:53.047 app[5683d704ce708e] ewr [info] 2023-08-23T06:36:53.047524Z WARN sink{component_kind="sink" component_id=logtail component_type=http component_name=logtail}:request{request_id=96223}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: connection closed before message completed internal_log_rate_limit=true
7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.