is `processes` inferfering with debugging filesystem-based issues?

Questions / Help
1

I’m running a Rails app using the experimental processes feature. Relevant configuration:

[processes]
web = "bundle exec puma"
job-runner = "bundle exec que"
postfix = "postfix start-fg"
newrelic = "newrelic-infra"

Today I was just working on adding the last entry, a New Relic logging agent, and hit a snag. In short: I’ve configured Postfix to dump logs into somewhere public (/var/log/postfix.log, to be specific), and New Relic to tail that file and ship it over to newrelic.com. But it doesn’t work.

In attempting to SSH debug (just a simple fly ssh console), I don’t see /var/log/postfix.log on disk at all. It should be there – I know that Postfix is otherwise properly configured (when I have it log to stdout, I see that come through on the fly.io logs page), and when tooling around in the SSH session I can get it to log to the file. I also noticed that Postfix didn’t think it was running in the SSH session… in fact, I was only able to locate the process for puma in the session, even though que was up and working. (There is no [relevant] persistent storage involved here – I don’t need these logs to live forever, I just need the disk to mediate between these two processes.)

I’m pretty fuzzy on the differences between apps/processes/etc., but it seems like I’ve misunderstood where the lines are drawn. My new understanding is that, if I have two processes, they don’t share (non-persistent) disk and in fact they can’t see each other at all, which would explain why both my SSH session and the New Relic agent can’t see the log file from Postfix.

Am I now understanding it correctly? Assuming so, how would I go about getting Postfix to dump logs into a place where the New Relic agent can tail them? Secondarily, is there a way I can SSH into a specific process’s, uh, namespace (?) and poke around its disk and process tree?

2

Your understanding is right (see).

I haven’t done it, but I imagine:

  1. Either running the New Relic agent as another process along side the main process of a regular Fly app (docs, also) (Fly’s multiprocess apps aren’t really meant for sidecars like observability daemons).
  2. Or, running a separate Fly app that reads from Fly-managed NATS-based log shipper and feeds it into New Relic (ref announcement, also / see).

One can ssh from a running set of app instances (VMs) via fly ssh console -s -a <app-name> (ref).

3

Thanks, this helped point me in the right direction.

I had a suspicion that I was pushing processes beyond what it was intended for and probably doing it wrong. Ditching the whole tail-log-file-from-disk thing and switching to the log-shipper simplified things dramatically and improved the completeness of the logs, to boot, so thanks for the suggestion.

1 Like
4

If you have the bandwidth to switch some more, see the recently announced Multi-process Machines, which are meant for apps that need sidecars (like observability agents) to run alongside the main process. You can then altogether avoid running fly-log-shipper.