Accessing an external non public resource from your app

I want to know if supports the following use case.

Let’s say I want to deploy a golang binary/app in that needs access to an external resource (let’s say a database) that is not publicly available.

How would you accomplish this?

Can I join the external server (or another host on that network) to the wireguard net?

Alternatively, I could open a tls/443 port on my end. If I do that, can I run a wireguard peer over websockets and connect both peers. Can I then expose that access to other apps? I wouldn’t mind creating a new peer on each app.

Anyway, maybe it is crazy but I want to know if anyone has a new for that use case and how they have solved it.

Thank you,

Yes, you can. It might be a bit of work, though. See also: IPv6 WireGuard Peering · Fly (esp, the part about RDS connectors).

If you desire fancy routing stuff, Kurt recommends tailscale: WireGuard peering without proxy - #2 by kurt | Edit wireguard allowed ips of an app - #2 by kurt

flyctl proxy is another handy little command, if you have the liberty to install flyctl on the remote host: wireguard tunnels from userland - #2 by kurt

Prior art: Can I use fly + WireGuard to expose an internal web interface?

1 Like

Thank you for the reply @ignoramous,

I am trying a little experiment to test this.

As a reminder, what I want to be able to access external resources (at tcp level) and also being able to access my fly apps from my external network (mostly routing http/s traffic).

To that end, I create a fly app (golang server) and a local (runs in my laptop) golang server.

  1. Created a fly app that runs a golang http server. All works well.
  2. I then create a wg tunnel as described here.
  3. Then I start wireguard locally with the configuration generated.
  4. I can access my golang server that runs in fly from my machine. After starting the wg tunnel, fly is resolving dns for .internal. So I run (drio-fly is my app name):
➜ dig +noall +answer _apps.internal txt
_apps.internal.         5       IN      TXT     "drio-fly,fly-builder-shy-smoke-5052"
fly-test/app master [!?]
➜ host drio-fly.internal
drio-fly.internal has IPv6 address fdaa:1:2918:a7b:7a:12b1:7f16:2
# NOTE: sadly,  this is the first time I am going to start using v6 IPs. 

fly-test/app master [!?]
➜ curl http://drio-fly.internal:8080
<!DOCTYPE html>
.... more html stuff

Now I want to be able to test that I can connect to a port in my local machine from the fly app.
For that purpose, I add some http client code in the root handler in the fly webserver. Instead of directly serving the template, I first make a request to the local machine and then I pass the result to the template data.

I believe I can just add the ipv6 address assigned to the network interface in the local machine that fly created for me when I setup the tunnel.

Is there any subcommand I can use from flyctl to test if I can connect/access the local machine from the fly app? Deploying a new version of the app takes time.

Thank you,

Code: fly-test · GitHub

1 Like

Incredible stuff; living on the literal cutting edge there :wink:

There’s flyctl ping (announcement, docs). So you may try pinging the 6pn (IPv6) address of your local machine’s WireGuard peer from the VM?

Btw, if it works for your usecase, you can consider running flyctl proxy on your local machine instead of creating a WireGuard peer by hand.

Thank you, @ignoramous.

I did not know that it was possible to ssh into an app!
That is great.
Unfortunately I hit an error that I have documented here.

I will continue pushing.


I have decided that the best way to proceed with this is to not use wireguard directly and use tailscale instead.

Tailscale has instructions on how to run a flyio app that has access to your tailnet.

It basically works by using a multistage Dockerfile that pulls the necessary tailscale bits plus whatever else you need for your app. You also want to treat the node as ephemeral and pass your ephemeral key as a variable to your docker container.

At the end of the day we end up with a docker container that uses the tailscale client to connect to your tailscale network before starting your app/service.

My Dockerfile looks like this:

FROM golang:1.19.2 as golang
COPY . ./
RUN unset GOPATH; go mod download
RUN unset GOPATH; go build -o ./main

FROM alpine:latest as builder
COPY . ./

FROM alpine:latest as tailscale
ENV TSFILE=tailscale_1.36.0_amd64.tgz
RUN wget${TSFILE} && \
  tar xzf ${TSFILE} --strip-components=1

FROM alpine:latest
RUN apk update && apk add bash ca-certificates iptables ip6tables && rm -rf /var/cache/apk/*

# Tailscale directories
RUN mkdir -p /var/run/tailscale /var/cache/tailscale /var/lib/tailscale

#RUN modprobe tun
RUN mkdir -p /dev/net && mknod /dev/net/tun c 10 200 &&  chmod 600 /dev/net/tun

# Copy binary to production image
COPY --from=builder /app/ /app/
COPY --from=tailscale /app/tailscaled /app/tailscaled
COPY --from=tailscale /app/tailscale /app/tailscale
COPY --from=golang /app/main /app/main
RUN mkdir -p /var/run/tailscale /var/cache/tailscale /var/lib/tailscale

# Run on container startup.
CMD ["/app/"]

Unfortunately I am running into an issue:

docker run \
                -e TAILSCALE_AUTHKEY=xxxxx \
                -p 8080:8080 \
                --name drio-fly \
2023/01/31 15:56:29 logtail started
2023/01/31 15:56:29 Program starting: v1.36.0-t96f658038-g49bd44543, Go 1.19.4-tsdc0ce6324d: []string{"/app/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/var/run/tailscale/tailscaled.sock"}
2023/01/31 15:56:29 LogID: 9e14d394a25cf1b5030ae96d3c1f1a5997d2419757b4ef06d4b029b0761e963b
2023/01/31 15:56:29 logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
2023/01/31 15:56:29 wgengine.NewUserspaceEngine(tun "tailscale0") ...
2023/01/31 15:56:29 Linux kernel version: 5.15.49-linuxkit
2023/01/31 15:56:29 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory
2023/01/31 15:56:29 wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): CreateTUN("tailscale0") failed; /dev/net/tun does not exist
2023/01/31 15:56:29 flushing log.
2023/01/31 15:56:29 logger closing down
2023/01/31 15:56:29 getLocalBackend error: createEngine: tstun.New("tailscale0"): CreateTUN("tailscale0") failed; /dev/net/tun does not exist
failed to connect to local tailscaled; it doesn't appear to be running
/app/ line 11: /app/main: not found
make: *** [run] Error 127

The error seems to be that the tun capabilities are not enable on that linux kernel that docker is running. When the tailscale daemon tries to load the tun modules it fails because we don’t have the /lib/modules directory available.

Anyway, I am asking the fly community, are you running any app in this mode, if so, did you encounter this issue?

Thank you,

1 Like

Ok, I think I have this working now.

First of all, the problem from my previous post was that I did not enable --cap-add and --device so that functionality was not enabled to the running container. My full docker cmd is:

	docker run \
		--cap-add=NET_ADMIN \
		--device=/dev/net/tun:/dev/net/tun \
		-p $(PORT):$(PORT) \
		--name $(CONT_NAME) \

Notice that I was trying to run the container in my local machine. When you deploy to fly, they take care of that so your container will have access to the necessary linux functionality to run.

Keep also in mind that if you are using tailscale ephemeral keys, you will have to regenerate a key on each deployment.

I am going to modify my code to make usage of the new tailscale access so it is a more complete example. But we are getting there!

1 Like

I am going to close this not without thanking everybody for their support and the fly team for building such a cool project.

Also, here is proof of success. The quote (the wire in case you want to know)

The game is out there, and it's either play or get played.

is coming from a server running in another machine in the tailnet.

Bye for now,

1 Like