Grafana Tempo/Agent

Hello,

Has anybody had any experience with setting up a Grafana Agent on fly.io? I’m looking to start collecting traces in Grafana Tempo.

Best regards,
Alex

Hi @alex2

We are using the grafana agent for tracing and metrics (we also used to use it for logging but have since moved to fly logs shipper to capture log messages that are generated by fly infrastructure as well).

Our current setup is every node has its own instance of the agent running alongside our app and this has worked fine for us in the past. As we scale up we’re finding that having the agent run on every node is starting to hit rate limiting issues for tracing from Grafana Cloud.

That setup should work for you too if you’re running a small number of servers. We’re looking at running dedicated servers for the agents for tracing and investigating what to do for metrics as fly does offer its own metrics collector but currently there are tradeoffs.

If you have any specific questions let me know and I’ll try my best to answer them.

Just curious what your Dockerfile and agent yaml look like? Trying unsuccessfully to get env variables working in the agent yaml.

Hi @chasers

We have since moved to running dedicated apps for the grafana agent.

The following is for the app that runs our grafana agent for tempo.

Dockerfile


FROM grafana/agent:latest

# Add our Grafana Cloud Agent config file
COPY ./src/agent.yaml /etc/agent/agent.yaml

CMD ["--config.file=/etc/agent/agent.yaml", "--metrics.wal-directory=/etc/agent/data", "--config.expand-env=true"]

agent.yaml

server:
    log_level: warn
    http_listen_port: 12345

traces:
    configs:
        - name: default
          receivers:
              otlp:
                  protocols:
                      http:
          remote_write:
              - endpoint: tempo-us-central1.grafana.net:443
                basic_auth:
                    username: ${TEMPO_USERNAME}
                    password: ${TEMPO_PASSWORD}
          batch:
              timeout: 15s
              send_batch_size: 10000
              send_batch_max_size: 256

Make note of the extra param --config.expand-env=true thats required for using environment variables inside the agent.yaml file

Thanks!!

Yeah I was doing this and getting some error message that it couldn’t find the config.file argument. Not working a ton with Docker I’m sure I was doing something wrong.

The run.sh hack ended up working for me though.

I think this will be helpful for others.

Also I had to delete everything in the [[services]] fly.toml before the deployment would go green. It’s not serving any traffic anyways. Instance would stay up, it just didn’t look healthy.