Grafana cannot write to mounted volume

Questions / Help Build debugging
1

Hello! I’m trying out Fly with the goal of deploy a fly grafana instance and connect it to fly postgres instance. I’ve followed the grafana example but got app[d09eebf6] ewr [info] GF_PATHS_DATA='/var/lib/grafana' is not writable.

I saw this thread and the logs pointed me to this migration guide which indicates grafana is using the root group now instead of grafana for >=7.3. The example specifies 8.0.0 though so is there something else I should be doing here?

fly.toml 
# fly.toml file generated for speenah-grafana on 2021-10-18T20:22:43-04:00

app = "speenah-grafana"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[build]
  image = "grafana/grafana"

[env]
  GF_INSTALL_PLUGINS = "grafana-worldmap-panel,grafana-clock-panel"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[mounts]
  source = "grafana_storage"
  destination = "/var/lib/grafana"

[[services]]
  http_checks = []
  internal_port = 3000
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Thanks!

2

That’s a pain! The Grafana Dockerfile sets the user to grafana, and we do change ownership on the volume to match.

Can you try adjusting your fly.toml to:

[build]
image = "grafana/grafana:main"

And see if that helps?

There’s a workaround here where you can create something like flyrun.sh:

#!/bin/bash

chown -R grafana:grafana /var/lib/grafana

exec "/run.sh"

Then create a Dockerfile like this:

FROM grafana/grafana:main
ADD flyrun.sh /
RUN chmod +x flyrun.sh
CMD['/flyrun.sh']

It’s kind of gross but it’s worth a try!

3

Hi Kurt, thanks for the suggestion! Unfortunately these did not work.

For my Dockerfile I did this:

FROM grafana/grafana:main
COPY ./flyrun.sh /
ENTRYPOINT ["/flyrun.sh"]

Running chmod +x /flyrun.sh in the dockerfile was giving me Operation not permitted. I ensured the file had 755 perms before building.

flyrun.sh:

#!/usr/bin/env bash

whoami
ls -lah /var/lib/grafana
chown -R grafana:root /var/lib/grafana

exec "/run.sh"

And during deployment I get these errors :frowning2:

[info] grafana
[info] total 24K
[info] drwxr-xr-x    3 root     root        4.0K Oct 19 00:34 .
[info] drwxr-xr-x    6 root     root        4.0K Oct 20 22:11 ..
[info] drwx------    2 root     root       16.0K Oct 19 00:34 lost+found
[info] chown: /var/lib/grafana/lost+found: Permission denied
[info] chown: /var/lib/grafana: Operation not permitted
[info] chown: /var/lib/grafana: Operation not permitted
[info] GF_PATHS_DATA='/var/lib/grafana' is not writable.
4

Change your Dockerfile to

FROM grafana/grafana:8.2.2

USER root 

ENTRYPOINT ["/run.sh"]
1 Like
5

This works, thanks! I’d prefer a way without using root but this unblocks me for now

6

I just want to confirm that setting USER root worked. There’s already a PR open here that makes the suggested changes: Build our own image running under root by cschmatzler · Pull Request #1 · fly-apps/grafana · GitHub