I want to know if fly.io supports the following use case.
Let’s say I want to deploy a golang binary/app in fly.io that needs access to an external resource (let’s say a database) that is not publicly available.
How would you accomplish this?
Can I join the external server (or another host on that network) to the fly.io wireguard net?
Alternatively, I could open a tls/443 port on my end. If I do that, can I run a wireguard peer over websockets and connect both peers. Can I then expose that access to other fly.io apps? I wouldn’t mind creating a new peer on each app.
Anyway, maybe it is crazy but I want to know if anyone has a new for that use case and how they have solved it.
As a reminder, what I want to be able to access external resources (at tcp level) and also being able to access my fly apps from my external network (mostly routing http/s traffic).
Then I start wireguard locally with the configuration generated.
I can access my golang server that runs in fly from my machine. After starting the wg tunnel, fly is resolving dns for .internal. So I run (drio-fly is my app name):
➜ dig +noall +answer _apps.internal txt
_apps.internal. 5 IN TXT "drio-fly,fly-builder-shy-smoke-5052"
fly-test/app master [!?]
➜ host drio-fly.internal
drio-fly.internal has IPv6 address fdaa:1:2918:a7b:7a:12b1:7f16:2
# NOTE: sadly, this is the first time I am going to start using v6 IPs.
fly-test/app master [!?]
➜ curl http://drio-fly.internal:8080
<!DOCTYPE html>
.... more html stuff
Now I want to be able to test that I can connect to a port in my local machine from the fly app.
For that purpose, I add some http client code in the root handler in the fly webserver. Instead of directly serving the template, I first make a request to the local machine and then I pass the result to the template data.
I believe I can just add the ipv6 address assigned to the network interface in the local machine that fly created for me when I setup the tunnel.
Is there any subcommand I can use from flyctl to test if I can connect/access the local machine from the fly app? Deploying a new version of the app takes time.
I have decided that the best way to proceed with this is to not use wireguard directly and use tailscale instead.
Tailscale has instructions on how to run a flyio app that has access to your tailnet.
It basically works by using a multistage Dockerfile that pulls the necessary tailscale bits plus whatever else you need for your app. You also want to treat the node as ephemeral and pass your ephemeral key as a variable to your docker container.
At the end of the day we end up with a docker container that uses the tailscale client to connect to your tailscale network before starting your app/service.
My Dockerfile looks like this:
FROM golang:1.19.2 as golang
WORKDIR /app
COPY . ./
RUN unset GOPATH; go mod download
RUN unset GOPATH; go build -o ./main
FROM alpine:latest as builder
WORKDIR /app
COPY . ./
FROM alpine:latest as tailscale
WORKDIR /app
ENV TSFILE=tailscale_1.36.0_amd64.tgz
RUN wget https://pkgs.tailscale.com/stable/${TSFILE} && \
tar xzf ${TSFILE} --strip-components=1
# https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds
FROM alpine:latest
RUN apk update && apk add bash ca-certificates iptables ip6tables && rm -rf /var/cache/apk/*
# Tailscale directories
RUN mkdir -p /var/run/tailscale /var/cache/tailscale /var/lib/tailscale
#RUN modprobe tun
RUN mkdir -p /dev/net && mknod /dev/net/tun c 10 200 && chmod 600 /dev/net/tun
# Copy binary to production image
COPY --from=builder /app/start.sh /app/start.sh
COPY --from=tailscale /app/tailscaled /app/tailscaled
COPY --from=tailscale /app/tailscale /app/tailscale
COPY --from=golang /app/main /app/main
RUN mkdir -p /var/run/tailscale /var/cache/tailscale /var/lib/tailscale
EXPOSE 8080
# Run on container startup.
CMD ["/app/start.sh"]
Unfortunately I am running into an issue:
docker run \
-e TAILSCALE_AUTHKEY=xxxxx \
-p 8080:8080 \
--name drio-fly \
drio-fly:alpha
2023/01/31 15:56:29 logtail started
2023/01/31 15:56:29 Program starting: v1.36.0-t96f658038-g49bd44543, Go 1.19.4-tsdc0ce6324d: []string{"/app/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/var/run/tailscale/tailscaled.sock"}
2023/01/31 15:56:29 LogID: 9e14d394a25cf1b5030ae96d3c1f1a5997d2419757b4ef06d4b029b0761e963b
2023/01/31 15:56:29 logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
2023/01/31 15:56:29 wgengine.NewUserspaceEngine(tun "tailscale0") ...
2023/01/31 15:56:29 Linux kernel version: 5.15.49-linuxkit
2023/01/31 15:56:29 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory
2023/01/31 15:56:29 wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): CreateTUN("tailscale0") failed; /dev/net/tun does not exist
2023/01/31 15:56:29 flushing log.
2023/01/31 15:56:29 logger closing down
2023/01/31 15:56:29 getLocalBackend error: createEngine: tstun.New("tailscale0"): CreateTUN("tailscale0") failed; /dev/net/tun does not exist
failed to connect to local tailscaled; it doesn't appear to be running
/app/start.sh: line 11: /app/main: not found
make: *** [run] Error 127
The error seems to be that the tun capabilities are not enable on that linux kernel that docker is running. When the tailscale daemon tries to load the tun modules it fails because we don’t have the /lib/modules directory available.
Anyway, I am asking the fly community, are you running any app in this mode, if so, did you encounter this issue?
First of all, the problem from my previous post was that I did not enable --cap-add and --device so that functionality was not enabled to the running container. My full docker cmd is:
Notice that I was trying to run the container in my local machine. When you deploy to fly, they take care of that so your container will have access to the necessary linux functionality to run.
Keep also in mind that if you are using tailscale ephemeral keys, you will have to regenerate a key on each deployment.
I am going to modify my code to make usage of the new tailscale access so it is a more complete example. But we are getting there!
