Logs stopped shipping AND unable to tail logs in flyctl or web interface

1

Hi,

There are two issues here.

Firstly the logs have stopped being shipped from my shipper battletabs-logs-shipper, im shipping them to axiom. The last logs that Axiom received were on the 1st of July:

image
image1749×499 28.7 KB

Secoond issue:

When I run fly logs -a battletabs (or any other app), then it outputs the logs, but then fails to tail.

The same goes for the logs shown in https://fly.io/apps/battletabs/monitoring it is not tailing the logs, it only shows whatever was there when I refreshed.

Can you please advise?

2

What region?

3

iad…

It looks like the logs started flowing again at 10:22am yesterday …

Did you guys do something?

4

Yes. Bouncing nats-server on all iad workers seems to have resolved the issue. Others also noticed this issue, I just asked which region to see if nats-servers needed to be kicked elsewhere too.

5

okay thanks, is there something I should do in the future if it happens again, or just message here?

6

It is not something you can fix (it isn’t even something I can directly fix). Definitely message here.

I’m in Raleigh, and this affected me too. What’s weird is that I was occasionally seeing a small number of log messages (not many, but non-zero).

7

Okay no worries, thanks keep up the great work, I really want Fly to be the best :slight_smile:

8

Same here. No logs since yesterday. Shipper logs it’s active but my principal app log doesn’t write anything into monitoring or send it back to my sink. :frowning:

spring-water-4208

9

regiom?

10

All its in Mad.

In case extra info:
App with no logs written since yesterday.
2865531ce94138 spring-water-4208 mad app

Fly logs shipper:
784ee71fee3728 red-flower-4186 mad app
e82dd73c0dd668 small-brook-6805 mad app

11

I’ve created an app and deployed it to mad, and see logs.

I’ve reported your problem internally to try to find someone who can look into it.

12

Thanks, really kind. Sometimes it’s really hard to get answers or support.:sweat_smile:

13

In mad things look to have cleared up. I can see logs from my app. Can you check?

14

Yes, now I can. It’s happening from time to time. Nothing it’s printed and days later something comes with bad timing. The only way o discovered to trigger logs for a while again it’s doing a new deploy again but after a few hours, same thing happen.

15

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.