Process group name should be part of metrics and logs as labels or something

[endersonmaia]

I have a Fly App that’s composed of 5 different process groups, and although it’s easier to maintain the code of a single fly.toml and Dockerfile, it’s hard to manage metrics collection and logs filtering when I can’t know from which process group that information (metrics, log event) came from.

For logs, the process group name could even have its name injected as a prefix for the log message, or at least as a label where we can filter it somehow. Like fly.app.process_group:

For metrics, it could be a label, just like {app=}, we could have a {process_group=} to also filter metrics by this.

Related: Add the process group as a label on metrics

—

Another issues that comes with it, is controlling the metrics endpoint individually by process group.

Related: Metrics & Process Groups - #3 by MatthewIngwersen

[mayailurus]

@endersonmaia

Another issues that comes with it, is controlling the metrics endpoint individually by process group.

This part seems to exist now, if you use double-brackets (i.e., [[metrics]] rather than [metrics]):

Custom metrics can be also configured for specific processes, using double brackets to define multiple [[metrics]] tables:

[[metrics]]
  port = 9394
  path = "/metrics"
  processes = ["web"]

[[metrics]]
  port = 9113
  path = "/metrics"
  processes = ["proxy"]

[endersonmaia]

Having the process group name as part of metrics would allow us to have better dashboards.

Using the instance ID is useless since I can’t really (easily) know which process I’m looking at.

[dusty]

@endersonmaia a bit late on the reply but fly_instance_info contains the machine process group information. You can add a join to pull these into your dashboard/viz, ie:

* on(instance, region) group_left(process_group)
  (
    fly_instance_info{app="FLY_APP_NAME"}
  )

Full example:

(
  sum(
    fly_instance_load_average{app="FLY_APP_NAME", minutes="5"}
  ) by (instance, region)
/
  sum(
    count(fly_instance_cpu{app="FLY_APP_NAME", mode="idle"}) without (cpu)
  ) by (instance, region)
)
* on(instance, region) group_left(process_group)
  (
    fly_instance_info{app="FLY_APP_NAME"}
  )

Let me know if you run into issues.

[martiserra]

Thanks @dusty, that’s great for the metrics side of things, but what about the logs? I’m using the official log shipper to send logs to BetterStack and they are missing the process group information in the fly object. It would be great if the process group was also included in there. Is that viable?

[dusty]

Good call, I’m looking into this!

[dusty]

This has been rolled out. Let us know if you notice any issues. Note, you may need to bounce your machines.