Running TCP DNS server?

I’ve successfully been able to run a UDP DNS server on port 53. However, when I try to run a DNS server on TCP port 53 I get the following error:

$ flyctl deploy
Deploying <myapp>
==> Validating App Configuration
***    ✘ services.0.ports.0.port: must be one of: [80, 443, 5000, 10000..10100]
Error App configuration is not valid

According to App Configuration (fly.toml) · Fly it looks like 53 is indeed not a valid public TCP port, but can that limitation be lifted so that I can run a TCP DNS server?

The use case for using TCP for DNS is to e.g. return TXT or DNSSEC responses that can’t fit into a UDP packet.

FWIW my config file is:

app = "<myapp>"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 53
  protocol = "tcp"

  [[services.ports]]
    port = "53"

We don’t yet allow port 53 publicly on your anycast IP. We’re working on allowing all / most / many ports in the coming months!

The only ways to expose any port from fly right now is through the private network or via the experimental allow_public_port config (this will open the port on every instance of your app, but only for its specific public IPv6). They’re much harder to access though.

OK, good to know that opening up port 53 is on the roadmap. Unfortunately until public TCP IPv4 port 53 access is available it means I can’t use Fly for my DNS service, so if the rollout timeframe could be reduced to “weeks” from “months” I’d be a happy customer.

The Fly networking gnomes are discussing.

Don’t quote me on this (my new motto is “y’all get my stream of consciousness thoughts on this message board and sometimes I will be wrong”) but:

I think the issue is that we have a static filter for allowable ports right now, both in IP filters and in our front-end proxy. We can add ports to it (the discussion we’re having started Friday when I tried to outdo Tailscale’s Minecraft post), but the thing we’d really like to do is get rid of it altogether.

Doing that will likely involve a bit of eBPF coding and with it a fleetwide kernel update, which is probably weeks away.

But we can probably just get over our purity objections and add 53/tcp (and 25565/tcp) to the static filter; that’s I think a tiny change, though Steve may yell at me for saying that.

I’m psyched to see people considering using us for DNS hosting. Can you say more (here or privately) about what you’re doing?

I can already see loads of potential awesome projects coming if you open 25565/tcp. The Minecraft community is full of original ideas.

One idea that I would think of:
A project where you host a minimal Minecraft server that waits for players to join it then launch the real Minecraft server and forward every new request to the real server.
The real server could be on whatever cloud, even on fly.io with the persistent volumes and communicate with the real server using the private networking :blush:.

Then when there are no players the minimal Minecraft server shutdown the real Minecraft server and waits again for players.

All of this for saving some money and avoid running a big empty server all the time. I did a project like this a while ago, but it worked only with DigitalOcean if you want to get a better idea: https://github.com/Alteiria/gamocosmautostart.

Sorry for being off-topic but expanding the public allowed ports is something that a lot of people would dream of and would bring a lot of more projects on fly.

It’s not really off-topic at all. I snuck 25565 into the PR I wrote last night that opens 53/tcp. Like I said, my kids demand of me that we make Fly suitable for Minecraft servers.

We’re certainly going to expand public ports. There’s no product reason that we limit the ports, it was just an engineering compromise. In the short term, if there are super-important ports for us to open up, we can do that; within a month or two, hopefully, we’ll just be serving all ports, all protocols (there will be a small amount of goofy eBPF to make that work, but it’s not a huge project).

Well if you don’t mind I would really love to have the port 6443/tcp.

It’s not the super important port, but it’s the de facto alternative port for 443 and also the default port for kube-apiserver which is a key component for hosting a Kubernetes API. The latter one interest me because I plan to build a highly available Kubernetes API on fly.

Thank you in advance for considering my request.

QUICK NOTE.

Somehow, for reasons I promise you don’t want us to get into, getting DNS open was more complicated than getting 25565/tcp-udp open for Minecraft. If you check this evening, you should be able to run a Minecraft server, though don’t quote me, because I’m not going to try to actually run one until this evening.

There will probably be some wonkiness: 25565/udp will only work on your fly-global-services address (all edge UDP gets directed there). What I’m saying here is: don’t bang your head against the wall too much on it; if you have problems, let us know, and we’ll try to be the ones who spend time debugging, not you.

53/tcp is still not open. In the meantime I recommend running DNS over 25565/tcp.

1 Like

I KID. 53/tcp should work sometime early next week.

Getting every port working is on our docket. The simple way for us to that is with socket-steering eBPF code, but to make that work we’ll need to do a fleet kernel update, and since we have the luxury, it’ll probably be a couple weeks out.

I can knock out 6443 early next week too, and I’m happy to take requests for other ports if it’s likely that more than 1 person wants it. (I say “I’m happy” but really it’s Steve doing the heavy lifting here and just me taking the credit. Except for DNS-over-Minecraft, an idea I expect full credit for.l)

Having port 6443 for kubernetes would be really awesome.

Also 8443 for dendrite (Matrix.org home server) would be also really awesome. Matrix uses 8443 for federation with other servers.

The matrix port is a good idea!

What are you doing with kubernetes? It weirds me out a little to expose the k8s API publicly. I think for k8s, it’d be better to setup a wireguard peer and connect through the private network (and all ports work over the private network).

1 Like

3 posts were split to a new topic: kubernetes on fly

You should no longer get this error! I’m able to launch 53/tcp things now.

We also added 8443/tcp (and, of course, the Minecraft port).

(Correction: I spoke too soon on 8443/tcp, that may be a couple hours; it’ll go in with the next fly-proxy deploy. 53/tcp and 8443/tcp are, so far, the weird ones; everything else is a one-liner.)

1 Like

Confirmed that 53/tcp is live at my end. Fantastic!

4 Likes

Could you also open up port 853/TCP?

Trying to deploy a DNS over TLS app. Already tested it to be working on 53/TCP (over TLS), but the required port is 853.

2 Likes

Yep, this shouldn’t be a problem. I’ll ping back when it’s done (if I take too long, please feel free to yell at me).

2 Likes

Hi thomas: Is this still in fly’s backlog?

Curious: What’s weird about those? Something specific to fly’s load-balancer? :smiley:

It’s still in our backlog! The way we want to fix this is with a BPF program to redirect all anycast traffic on any port to our proxy; to get that working, we need to get all our machines on a specific kernel. We’re currently working on fixing an incompatibility between that kernel and our existing BPF code.

We spent a good amount of time last week working on this; it’s definitely being actively worked on, and is annoying the hell out of us.

1 Like

@thomas We have a request for 7777 as well: Deploying on unsupported port

Yep, on it!