`fly-replay` header affecting metrics

1

Hi all,

I’ve been playing around with using the fly-replay header to build an API gateway.

Currently I’ve got a public app configured (proxy-test-gateway) and an internal app configured (proxy-test-internal) thats been assigned a FlyCast private ipv6 address so it’s not publicly visible. Both apps have two instances each, one in yyz and one in syd.

Everything seems to be working fine except the metrics all appear to come from proxy-test-gateway. No metrics are recorded for proxy-test-internal.

Is this a bug or is it intended? I would have thought that since proxy-test-internal is handling the request then the metrics would be assigned to that app.

If you want to give it a go you can hit https://proxy-test-gateway.fly.dev/

1 Like
2

I don’t have an answer for you (other than be the rubber duck)…

  1. When you say metrics, you mean the one Fly pushes out through NATS (and on to fly-metrics.net)? Or, the default metrics they show on the app (fly.io) dashboard?

  2. Without the fly-replay header, these metrics for proxy-test-internal start appearing just fine?

  3. Is it known whether Fly publishes metrics of requests sent over 6pn (in this case, over the Flycast address)?

  4. I’d wager proxy-test-internal is missing a [[services]] section. A wild guess: Fly perhaps doesn’t setup/publish metrics for such (non-public) apps, by default?

If I may, I do have a query of my own: Is this test code open source? If not, will/can it be?

3

Hi @ignoramous

Thanks for your thoughts, here’s the answers to your questions:

  1. The ones in the fly dashboard, I haven’t checked the grafana site yet for these apps.

  2. I haven’t tested hitting the internal app directly, mainly because it’s not publicly accessible

  3. That’s a good question, it might be that they don’t publish any metrics for private FlyCast proxys

  4. The internal app does have a services section as I’m pretty sure that FlyCast needs it (though I could be wrong) and the fly-replay header doesn’t seem to work without it if you’re only specifying the app option

The code isn’t currently published anywhere, just sitting on a folder on my computer. But currently it’s really simple. The gateway app always responds with a 200 and the fly-replay header with the app option pointing to the internal app. The internal app always responds with a 200 and a json payload containing hello world and the region of the instance that responded.

I also tested using fly-prefer-region from the client and it seems to work but its really slow, the latency is doubled which wounds like the edge that you hit initially is doing all the work.

This means that if you are in syd have the prefer header set to yyz, your request will be initially routed to the gateway app in yyz from the syd edge. The yyz gateway app will respond to the syd edge with the replay header. The syd edge will then replay the request to the yyz internal app. And finally the yyz internal app will respond to the syd edge. So that adds up to a lot of round trips.

1 Like