sending metrics to datadog agent via UDP

I have the following datadog agent config:

app = "black-bush-2163"

kill_signal = "SIGINT"
kill_timeout = 5

[experimental]
  auto_rollback = true

[env]
  DD_SITE="datadoghq.eu"
  DD_APM_NON_LOCAL_TRAFFIC = "true"
  DD_LOGS_ENABLED="true"
  DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL="true"
  DD_CONTAINER_EXCLUDE_LOGS="name:datadog-agent"
  DD_API_KEY="xxx"
  DD_APM_ENABLED="true"
  DD_PROCESS_AGENT_ENABLED="true"
  DD_DOGSTATSD_SOCKET="fly-global-services"

[build]
  image = "datadog/agent:7"

[[services]]
  internal_port = 8125
  processes = ["app"]
  protocol = "udp"

  [[services.ports]]
    port = 8125

[[services]]
  internal_port = 8126
  processes = ["app"]
  protocol = "tcp"

  [[services.ports]]
    port = "8126"

# [[services]]
#   internal_port = 8126
#   processes = ["app"]
#   protocol = "tcp"

# [[services.ports]]
#   handlers = ["udp"]
#   port = 8125

I in the logs that the agent appears to be bind correctly:

2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/forwarder/forwarder.go:356 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-37-1-app.agent.datadoghq.eu" (1 api key(s))

2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/dogstatsd/listeners/uds_common.go:142 in Listen) | dogstatsd-uds: starting to listen on fly-global-services

2022-07-24T13:15:44.685 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/dogstatsd/listeners/udp.go:95 in Listen) | dogstatsd-udp: starting to listen on 127.0.0.1:8125

2022-07-24T13:15:44.686 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/tagger/collectors/workloadmeta_main.go:115 in stream) | workloadmeta tagger collector started

2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/runner/runner.go:95 in ensureMinWorkers) | Runner 1 added 4 workers (total: 4)

2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/python/init.go:322 in resolvePythonExecPath) | Using '/opt/datadog-agent/embedded' as Python home

2022-07-24T13:15:44.711 app[8e14cce6] ams [info] 2022-07-24 13:15:44 UTC | CORE | INFO | (pkg/collector/python/init.go:389 in Initialize) | Initializing rtloader with Python 3 /opt/datadog-agent/embedded

I can send traces correctly to DD (mostly because the agent listens to it via TCP) but metrics (StatsD) are sent to the agent via UDP and I can’t for the life of me make it work

I try play around with DD_AGENT_HOST env var but nothing seems to work. I tried black-bush-2163.internal, black-bush-2163.fly.dev, but metrics via UDP are simply not being sent

So far I got:

  • Sending traces via this agent
  • Sending logs via fly-log-shipper

but metrics are a mystery yet.

Has anyone got any luck with a dedicated DD agent via UDP?

UDP services need to listen on the fly-global-services address, but your config is setting DD_DOGSTATSD_SOCKET to set fly-global-services as a Unix Domain Socket (uds) path which won’t work (dogstatsd-udp is still binding to 127.0.0.1:8125).

Maybe try setting DD_BIND_HOST = "fly-global-services" instead?

I read the agent docs thousand of times and missed the DD_BIND_HOST :man_facepalming:

anyway, still didn’t work. tried different address for the client (fly-global-services, fly-global-services.internal, black-bush-2163.internal, black-bush-2163.fly.dev, the IP address), none of them sent the metrics as expected and as weird as it gets, using black-bush-2163.internal resulted in failure to resolve the address…

I just gave and will probably use a different app for metrics

thanks a lot @wjordan