Docker build failing to resolve rubygems.org DNS

Hello,

We are deploying Coder in Fly to run ephimeral VS Code workspaces. Within the fly machines in fly we are using docker to deploy dev containers.

For some reason, it seems that in docker build time, docker is no able to resolve DNS.

coder@217811931fd589:~/rails-interview/.devcontainer$ docker build .
[+] Building 3.3s (3/5)                                                                           
[+] Building 3.5s (3/5)                                                                             
 => [internal] load build definition from Dockerfile                                           0.0s
 => => transferring dockerfile: 1.29kB                                                         0.0s
 => [internal] load .dockerignore                                                              0.0s
[+] Building 3.6s (3/5)                                                                                                           
 => [internal] load build definition from Dockerfile                                                                         0.0s
 => => transferring dockerfile: 1.29kB                                                                                       0.0s
[+] Building 26.5s (5/5) FINISHED                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                 0.0s
 => => transferring dockerfile: 1.29kB                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                    0.0s
 => => transferring context: 2B                                                                                                      0.0s
 => [internal] load metadata for mcr.microsoft.com/devcontainers/ruby:3.1                                                            1.5s
 => [1/2] FROM mcr.microsoft.com/devcontainers/ruby:3.1@sha256:12ba6eb66dc8a1af8e2c0e84623d9501f2a725bca8ba18874c19f22d1d746939     24.4s
 => => resolve mcr.microsoft.com/devcontainers/ruby:3.1@sha256:12ba6eb66dc8a1af8e2c0e84623d9501f2a725bca8ba18874c19f22d1d746939      0.0s
 => => sha256:f808801a3f0184ac8050f02728ba2dd47df9c1ed3fa41b1980eaf2e7516898bb 19.54kB / 19.54kB                                     0.0s
 => => sha256:b0248cf3e63c73d0e496a67807d056ca41d5e968b61087e8eca2cf4b9b4d7b99 55.05MB / 55.05MB                                     1.0s
 => => sha256:127e97b4daf784e08840a21765f0d4f251192ef2994d0e4a253490f81e63955b 5.17MB / 5.17MB                                       0.8s
 => => sha256:12ba6eb66dc8a1af8e2c0e84623d9501f2a725bca8ba18874c19f22d1d746939 1.61kB / 1.61kB                                       0.0s
 => => sha256:be935d53e75e23eb549d1e0918707cd19876e64255d4959d3864012ccfc1432a 3.54kB / 3.54kB                                       0.0s
 => => sha256:0336c50c9f6942b660e433b1086238eec37057c34b14c4e3b28bd7bf05bd84ba 10.88MB / 10.88MB                                     0.9s
 => => sha256:1b89f3c7f7da8adf032a33a75d1b659cee33179ecb88ea0ba75e4fc58ebe63a6 54.58MB / 54.58MB                                     1.8s
 => => sha256:2d62772179761f8fddfaed03ed2bdf7078b103b193d18d79a9b49364830d56cf 196.81MB / 196.81MB                                   4.1s
 => => sha256:4647bcaf3a5f4710eed11d4e71ccbf52d564c9f6f015bafc5bf758f94f8fa6eb 199B / 199B                                           1.4s
 => => extracting sha256:b0248cf3e63c73d0e496a67807d056ca41d5e968b61087e8eca2cf4b9b4d7b99                                            4.4s
 => => sha256:b19b5b35f1b9ae8b8355502f75566c31ed755a36431748ab2fe6b069369cbc9c 32.61MB / 32.61MB                                     3.8s
 => => sha256:3f2ffd2da6af29f0381cd7f53f1503d72a886c04852a70e47e0c60db8fe15c2c 174B / 174B                                           2.0s
 => => sha256:7c6b77d9323603ffb36ebec2525811f60d8e308d3ddc304364b0b1a0129f1478 14.16MB / 14.16MB                                     2.4s
 => => sha256:70764ebf99364922367333bb53c429cdf03a14b0c9b66127a222014a4ffc9cef 421B / 421B                                           2.7s
 => => sha256:76aa829e9b7aabe68ff14b4c7b99879d6dee9fa1a67e0a43a3eb74f7d4f46a9b 141B / 141B                                           2.9s
 => => sha256:59c044af77684a66352a4b65f0868a1d66585c4dd1faa9e1df447736745fb62c 227B / 227B                                           3.2s
 => => sha256:bb353a4e28392b66cf4e910b2dc86724e72686333b069f3409d4605798fcc228 238B / 238B                                           3.4s
 => => sha256:116d64e6df7dfe022b73fbc89dbda331498947e2fe1b6b392e310be655f9325b 45.68MB / 45.68MB                                     4.4s
 => => sha256:a55e38561316d692f19d139ebda656303ba47fdf28ead7261d6c58ec4c68c355 63.92MB / 63.92MB                                     8.4s
 => => sha256:f9b992bf34b350003c8013c53bccc9beda0957a3340b6a9a227ad8e169a03504 2.09MB / 2.09MB                                       4.7s
 => => sha256:b905b1e4d81ca50710693c2eb97cab16cbb052db5625f756e286073278c191f1 15.21MB / 15.21MB                                     4.8s
 => => extracting sha256:127e97b4daf784e08840a21765f0d4f251192ef2994d0e4a253490f81e63955b                                            0.3s
 => => extracting sha256:0336c50c9f6942b660e433b1086238eec37057c34b14c4e3b28bd7bf05bd84ba                                            0.3s
 => => extracting sha256:1b89f3c7f7da8adf032a33a75d1b659cee33179ecb88ea0ba75e4fc58ebe63a6                                            2.8s
 => => extracting sha256:2d62772179761f8fddfaed03ed2bdf7078b103b193d18d79a9b49364830d56cf                                            7.5s
 => => extracting sha256:4647bcaf3a5f4710eed11d4e71ccbf52d564c9f6f015bafc5bf758f94f8fa6eb                                            0.0s
 => => extracting sha256:b19b5b35f1b9ae8b8355502f75566c31ed755a36431748ab2fe6b069369cbc9c                                            1.0s
 => => extracting sha256:3f2ffd2da6af29f0381cd7f53f1503d72a886c04852a70e47e0c60db8fe15c2c                                            0.0s
 => => extracting sha256:7c6b77d9323603ffb36ebec2525811f60d8e308d3ddc304364b0b1a0129f1478                                            0.2s
 => => extracting sha256:70764ebf99364922367333bb53c429cdf03a14b0c9b66127a222014a4ffc9cef                                            0.0s
 => => extracting sha256:76aa829e9b7aabe68ff14b4c7b99879d6dee9fa1a67e0a43a3eb74f7d4f46a9b                                            0.0s
 => => extracting sha256:59c044af77684a66352a4b65f0868a1d66585c4dd1faa9e1df447736745fb62c                                            0.0s
 => => extracting sha256:bb353a4e28392b66cf4e910b2dc86724e72686333b069f3409d4605798fcc228                                            0.0s
 => => extracting sha256:116d64e6df7dfe022b73fbc89dbda331498947e2fe1b6b392e310be655f9325b                                            2.4s
 => => extracting sha256:a55e38561316d692f19d139ebda656303ba47fdf28ead7261d6c58ec4c68c355                                            2.1s
 => => extracting sha256:f9b992bf34b350003c8013c53bccc9beda0957a3340b6a9a227ad8e169a03504                                            0.2s
 => => extracting sha256:b905b1e4d81ca50710693c2eb97cab16cbb052db5625f756e286073278c191f1                                            1.0s
 => ERROR [2/2] RUN gem install rails webdrivers                                                                                     0.5s
------
 > [2/2] RUN gem install rails webdrivers:
#0 0.379 ERROR:  Could not find a valid gem 'rails' (>= 0), here is why:
#0 0.379           Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz)
#0 0.385 ERROR:  Could not find a valid gem 'webdrivers' (>= 0), here is why:
#0 0.385           Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz)
------
Dockerfile:6
--------------------
   4 |     
   5 |     # Install Rails
   6 | >>> RUN gem install rails webdrivers
   7 |     
   8 |     # Default value to allow debug server to serve content over GitHub Codespace's port forwarding service
--------------------
ERROR: failed to solve: process "/bin/sh -c gem install rails webdrivers" did not complete successfully: exit code: 2

This seems to be problematic only when building the image. If we spin up a container and run the same command it works correctly. I suspect this might be related to some of the ipv6 stuff from Fly.

Any ideas?

Changing /etc/resolve.conf to google DNS seems to solve the issue. @kurt any ideas ?

What template are you using to run docker on fly?

@charsleysa I’m currently using this template coder/main.tf at main · coder/coder · GitHub

The fly_machine is using the following docker image (this is the Dockerfile):

FROM codercom/code-server:4.12.0-ubuntu

USER root

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && \ 
    apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

RUN apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io

USER coder

The entry point for the container has been modified in order to start the docker daemon. The entry point looks like this:

set -e

    sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
    sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

    # Add coder to the docker group
    sudo usermod -aG docker $USER

    # Start code-server
    code-server --auth none >/tmp/code-server.log 2>&1 &

    # Set the hostname to the workspace name
    # sudo hostname -b "${data.coder_workspace.me.name}-fly"

    # Install the Fly CLI and add it to the PATH
    curl -L https://fly.io/install.sh | sh
    echo "export PATH=$PATH:/home/coder/.fly/bin" >> ~/.bashrc

    # TODO: ...
    newgrp docker

    source ~/.bashrc

    # Start the docker daemon
    sudo service docker start

Try adding the following to your entrypoint script:

    echo '{"data-root": "/home/coder/docker", "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64"}' | sudo tee -a /etc/docker/daemon.json > /dev/null

    sudo sysctl net.ipv6.conf.default.forwarding=1
    sudo sysctl net.ipv6.conf.all.forwarding=1

    sudo ip6tables -t nat -A POSTROUTING -s 2001:db8:1::/64 -j MASQUERADE

@charsleysa Thanks so much! That seems to have done the trick. I did try changing the forwarding parameters with sysctl but the was missing the other pieces.

The other thing that I haven’t figured out yet is how to make the coder user part of the docker group so that I don’t need to use sudo when interacting with the docker daemon. Adding newgrp docker to the entry point doesn’t seem to work. It does work when I start a bash session in the fly machine but it doesn’t persist across sessions.

BTW, is there any blog post/documentation on how to run docker in fly ?

Thanks so much for the quick help! You saved me a lot of time!

1 Like

Theres a lot of old resources that are somewhat out of date so you’ll have to do a bit of reading to get your head around it.

fly-apps/docker-daemon: A Docker daemon to run in Fly and access via a WireGuard peer. (github.com)

VS Code - Remote Server over SSH · Fly Docs

Here’s what my entrypoint script looks like, it’s a bit hacky:


    set -e
    # Download and install Docker
    getent group docker || sudo groupmod -n docker coder
    curl -fsSL https://get.docker.com -o get-docker.sh
    sudo sh get-docker.sh
    sudo apt-get update -y
    sudo apt-get upgrade -y
    sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
    sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
    sudo mkdir -p /home/coder/docker /etc/docker
    echo '{"data-root": "/home/coder/docker", "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64"}' | sudo tee -a /etc/docker/daemon.json > /dev/null
    sudo service docker start
    sleep 2

    sudo sysctl net.ipv6.conf.default.forwarding=1
    sudo sysctl net.ipv6.conf.all.forwarding=1

    sudo ip6tables -t nat -A POSTROUTING -s 2001:db8:1::/64 -j MASQUERADE

    # Start code-server
    code-server --auth none >/tmp/code-server.log 2>&1 &
    # Set the hostname to the workspace name
    sudo hostname -b "${data.coder_workspace.me.name}-fly"
    # Install the Fly CLI and add it to the PATH
    curl -L https://fly.io/install.sh | sh
    echo "export PATH=$PATH:/home/coder/.fly/bin" >> ~/.bashrc
    source ~/.bashrc

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.