Edit: False alarm. Once I got certification working I started getting the 404 Not Found: no match for IP:port error again. But only got to test this once and then started having issues with certification again. Going to let some time pass in case it’s rate limits and then test some more. The build does at least work and boots Tailscale and Caddy without error.
This is a little early for me to be sure because I haven’t tested this with my whole set up but did get an isolated test fully working. I am expecting it to work for me but I believe I ran into this issue where I have tested this too many times and issued too many Let’s Encrypt certificates for the same domains. I have done this before, just gotta wait it out 
Anyways, here we go. If Caddy is the entry to all your other apps, then this will let you authenticate all traffic on Caddy with Tailscale, only allowing authorized traffic to access any apps. Each app behind Caddy does not need to set up Tailscale this way, fly private networking takes over after that.
At least for now you need to do a manual build of Caddy to build it with the caddy-tailscale plugin because of conflicts with dependencies.
First follow the step 1 instructions here and then instead of using the TAILSCALE_AUTHKEY environment variable we’re going to use TS_AUTH because that’s what caddy-tailscale plugin uses.
Copy main.go as instructed in the comment in main.go. Also run go mod init caddy before building the dockerfile. You need to be able to copy the files this command creates to the dockerfile or use a volume if using docker compose.
main.go
// Copyright 2015 Matthew Holt and The Caddy Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Package main is the entry point of the Caddy application.
// Most of Caddy's functionality is provided through modules,
// which can be plugged in by adding their import below.
//
// There is no need to modify the Caddy source code to customize your
// builds. You can easily build a custom Caddy with these simple steps:
//
// 1. Copy this file (main.go) into a new folder
// 2. Edit the imports below to include the modules you want plugged in
// 3. Run `go mod init caddy`
// 4. Run `go install` or `go build` - you now have a custom binary!
//
// Or you can use xcaddy which does it all for you as a command:
// https://github.com/caddyserver/xcaddy
package main
import (
caddycmd "github.com/caddyserver/caddy/v2/cmd"
// plug in Caddy modules here
_ "github.com/caddyserver/caddy/v2/modules/standard"
_ "github.com/tailscale/caddy-tailscale"
)
func main() {
caddycmd.Main()
}
This is all you need in go.mod. The rest of the require statements will be set by the go get commands in the Dockerfile. go mod init caddy should create this file but you can double check that it’s set to go 1.19. I used go version 1.19 because 1.20 ran into issues with the gvisor dependency from caddy-tailscale.
go.mod
module caddy
go 1.19
Dockerfile
FROM golang:1.19-bullseye as caddy
# https://github.com/caddyserver/caddy/releases
ENV CADDY_VERSION v2.6.4
WORKDIR /app/caddy
COPY ./where/you/copied/main.go/on/your/local /app/caddy
RUN go get github.com/caddyserver/caddy/v2/cmd
RUN go get github.com/caddyserver/caddy/v2/modules/standard
RUN go get github.com/tailscale/caddy-tailscale
RUN go build
RUN ls -al
FROM bitnami/minideb:bullseye as tailscale
ENV TSFILE=tailscale_1.36.2_amd64.tgz
WORKDIR /app
RUN install_packages wget ca-certificates
RUN wget https://pkgs.tailscale.com/stable/${TSFILE} && \
tar xzf ${TSFILE} --strip-components=1
FROM bitnami/minideb:bullseye
RUN install_packages ca-certificates iptables iproute2
# iptables-legacy is needed to fix this error with Tailscale:
# "health("router"): error: setting up filter/ts-input: running [/usr/sbin/iptables -t filter -N ts-input --wait]: exit status 4: iptables v1.8.7 (nf_tables): Could not fetch rule set generation id: Invalid argument"
RUN update-alternatives --set iptables /usr/sbin/iptables-legacy
RUN update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
RUN mkdir --parents \
/config/caddy \
/data/caddy/caddy \
/etc/caddy \
/usr/share/caddy \
/var/run/tailscale \
/var/cache/tailscale \
/var/lib/tailscale \
;
COPY --from=tailscale /app/tailscaled /usr/bin/tailscaled
COPY --from=tailscale /app/tailscale /usr/bin/tailscale
COPY --from=builder /app/caddy/caddy /usr/bin/caddy
COPY ./path/to/local/caddy.json /config/caddy/caddy.json
COPY ./path/to/start.sh /start.sh
RUN chmod +x /start.sh
# https://github.com/caddyserver/caddy/releases
ENV CADDY_VERSION v2.6.4
# See https://caddyserver.com/docs/conventions#file-locations for details
ENV XDG_CONFIG_HOME /config
ENV XDG_DATA_HOME /data/caddy
EXPOSE 8080 8443 2019
ENTRYPOINT ["/start.sh"]
And then if you used go mod init caddy you’ll see the “caddy” binary in the docker build logs.
#14 [8/8] RUN ls -al
#14 0.472 total 74348
#14 0.472 drwxr-xr-x 1 root root 4096 Mar 4 07:03 .
#14 0.472 drwxr-xr-x 1 root root 4096 Mar 4 06:34 ..
#14 0.472 -rwxr-xr-x 1 root root 75976074 Mar 4 07:03 caddy
#14 0.472 -rw-r--r-- 1 root root 9619 Mar 4 07:02 go.mod
#14 0.472 -rw-r--r-- 1 root root 126558 Mar 4 06:13 go.sum
#14 0.472 -rw-rw-r-- 1 root root 1507 Mar 4 06:26 main.go
The entrypoint needs to boot up tailscale since we’re using a local client with the caddy-tailscale plugin.
start.sh
#!/usr/bin/env bash
tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock &
tailscale up --authkey=${TS_AUTHKEY} --hostname=your-fly-caddy-app-name
caddy run --config /config/caddy/caddy.json
And then the Caddy file something like this. This has some access logging set up so it’s easier to debug and see what’s going on in Fly logs.
{
"admin": {
"disabled": true
},
"logging": {
"logs": {
"platform": {
"level": "DEBUG",
"writer": {
"output": "stdout"
},
"include": [
"http.log.access.platform"
]
}
}
},
"apps": {
"http": {
"http_port": 80,
"https_port": 443,
"servers": {
"services": {
"listen": [
":8080",
":8443"
],
"logs": {
"default_logger_name": "platform"
},
"automatic_https": {
"disable_certificates": true,
"disable_redirects": false
},
"routes": [
{
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"handler": "authentication",
"providers": {
"tailscale": {}
}
}
]
},
{
"match": [
{
"host": [
"your.real.host.com"
]
}
],
"handle": [
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "fly-app.internal:7777"
}
]
}
]
}
// ... Your other apps here
]
}
],
"terminal": true
}
]
}
}
}
}
}
Then on the Fly configuration side your services just need to look like this. And make sure to set the environment variable for TS_AUTH here depending how you’re managing that. You can also set TS_VERBOSE=1 for additional Tailscale logging.
services = [
{
ports = [
{
port = 443
handlers = ["tls", "http"]
},
{
port = 80
handlers = ["http"]
}
]
"protocol" : "tcp",
"internal_port" : 8080
}
]
This Caddy app should also have an IPv4, IPv6, and SSL certification for the domain you are going to test this with. Now you’ll be able to change the DNS records for your domain to point to the IPs and set up the Let’s Encrypt challenge.
Aaaand that’s it. In monitoring you should be able to see Tailscale and Caddy boot up. Then should be able to test your domain and see in the fly logs if you’re authenticating correctly as well as seeing the bots get rejected.