I in the logs that the agent appears to be bind correctly:
2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/forwarder/forwarder.go:356 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-37-1-app.agent.datadoghq.eu" (1 api key(s))
2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/dogstatsd/listeners/uds_common.go:142 in Listen) | dogstatsd-uds: starting to listen on fly-global-services
2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/dogstatsd/listeners/udp.go:95 in Listen) | dogstatsd-udp: starting to listen on 127.0.0.1:8125
2022-07-24T13:15:44.686 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/tagger/collectors/workloadmeta_main.go:115 in stream) | workloadmeta tagger collector started
2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/runner/runner.go:95 in ensureMinWorkers) | Runner 1 added 4 workers (total: 4)
2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/python/init.go:322 in resolvePythonExecPath) | Using '/opt/datadog-agent/embedded' as Python home
2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/python/init.go:389 in Initialize) | Initializing rtloader with Python 3 /opt/datadog-agent/embedded
I can send traces correctly to DD (mostly because the agent listens to it via TCP) but metrics (StatsD) are sent to the agent via UDP and I can’t for the life of me make it work
I try play around with DD_AGENT_HOST env var but nothing seems to work. I tried black-bush-2163.internal, black-bush-2163.fly.dev, but metrics via UDP are simply not being sent
So far I got:
Sending traces via this agent
Sending logs via fly-log-shipper
but metrics are a mystery yet.
Has anyone got any luck with a dedicated DD agent via UDP?
UDP services need to listen on the fly-global-services address, but your config is setting DD_DOGSTATSD_SOCKET to set fly-global-services as a Unix Domain Socket (uds) path which won’t work (dogstatsd-udp is still binding to 127.0.0.1:8125).
I read the agent docs thousand of times and missed the DD_BIND_HOST
anyway, still didn’t work. tried different address for the client (fly-global-services, fly-global-services.internal, black-bush-2163.internal, black-bush-2163.fly.dev, the IP address), none of them sent the metrics as expected and as weird as it gets, using black-bush-2163.internal resulted in failure to resolve the address…
I just gave and will probably use a different app for metrics
You’d need to connect over IPv6 (nc -4u is v4, I believe? ref).
If the agent is running on port 8125 in a Fly app named dd007 which may be running multiple instances (VM), say. You’d connect to it (from another Fly app / Fly maching running in the same 6pn network of the same org) either allocating and using a Flycast IP (over port 8125); or by explicitly connecting to any running instance of dd007 using its 6pn DNS name (on port 8125 as well) (1, 2), which should look like <alloc-id>.vm.appname.internal (a specific instance of that app in any region but same 6pn network and org), or region.appname.internal (all instances of the app in that region).
I ended up giving up on datadog, I’m using newrelic for logs and stuffs and for metrics, I instrumented the app with prometheus and I’m sending them (or actually Fly is sending it hahaha) to Fly’s dedicated grafana (see this)
Working great tho I miss datadog
I think it will only work properly if Fly creates some kind of integration out of the box or something
@ignoramous - thanks for this. That was helpful, I actually finally got metrics to work from a NodeJs
server (Fly app) to the datadog-agent running on Fly from your comment! Rereading this Running Fly.io Apps On UDP and TCP · Fly Docs made it finally click. I had to use the external port, so this now works when ssh’d into a Fly app in the same org.
Nice! I am glad my rather incohesive drivel helped.
I must point out though, my comment was about connecting to a datadog-agent over Fly’s internal network (aka 6pn), while the glimpses of your solution suggests that you’re connecting to it over the public internet (<datadog-agent-app>.fly.dev:8125 instead of top1.nearest.of.<datadog-agent-app>.internal:8125).