VictoriaMetrics remote_write

[adminorcail]

Hi!

We have multiple apps on fly.io and we would like to monitor their metrics and logs. fly.io already provides this (and this is awesome by the way!), but there’s an itch: some services expose their metrics endpoint and require authentication. Unfortunately, this is not something doable right now in the fly.io config deployment config file. We cannot really change the port or disable authentication because those are external services.

On top of that, we have a local observability stack composed of multiple services (Alloy, Loki, Prometheus, Grafana) that we wish to deploy. To avoid duplicating things, we had the idea to reuse the fly.io metrics/logs as they already work nicely, but just add an additional service (Alloy) that would scrape our internal metrics that needs authentication and then write them to VictoriaMetrics. It seems that fly.io already use this endpoint internally, but can an app use it as well?

I also saw that there are other Prometheus exporters (like Fluent Bit) that would allows us to directly expose a /metrics endpoint that can be scraped by fly.io, which is not the case for Alloy at the moment, but we would prefer to have our deployment stack to match with our local one (thus, using Grafana services).

Overall, our stack would go from (only for metrics):

• Alloy (scrapes our internal services and write metrics to Prometheus)
• Prometheus (federates with fly.io VictoriaMetrics)

to:

• Alloy (scrapes our internal services and write metrics to VictoriaMetrics)

Thanks!

[halfer]

I don’t know if this will affect you, but there is some talk of metrics/logs changing in the near future. It may be worth getting some info on this before doing non-trivial building work:

https://community.fly.io/t/major-outage/23914/58

[adminorcail]

Thanks! Indeed, that helped me a lot, I will consider using only our stack and not rely on fly.io for that part.

[halfer]

It sounds like Fly could indeed continue their current offering. The issue may be that it’s not certain. I do hope they continue it, but of course they need enough people to pay for it to make it worth maintaining.

[adminorcail]

Yes, I hope that this will stay because I really like that feature. However, I’ve come to the conclusion that I needed a custom Grafana instance for our purposes because we can’t add data sources on the one provided by fly.io. Also, I need to plan ahead I won’t take the risk to have to change everything in 3 (or so) months. But in other projects, I would definitely use that feature!