UDP packets being dropped if the third buffer byte is less than 8

I have a simple UDP server running in Node.js on Fly.io, bound to the special address fly-global-services and port 53.
It look likes that packets are being dropped just because they have some specific bytes in them. From my tests, if the third byte of the buffer is less than 8, then the packet will be dropped.

Node.js server code:

const server = require('dgram').createSocket('udp4');
server.on("message", console.log);
server.bind(53, "fly-global-services");

This is the client Node.js code:

const server = require('dgram').createSocket('udp4');
server.connect(53, <flyio-app-address>, () => {
  const buffer = Buffer.alloc(3);
  buffer.writeUInt8(0, 0);
  buffer.writeUInt8(0, 1);
  buffer.writeUInt8(7, 2); // <-- packet dropped because of this byte
  server.send(buffer);
});

fly.toml UDP configuration

[[services]]
  internal_port = 53
  protocol = "udp"

  [[services.ports]]
    port = "53"

Dockerfile:

FROM node:16
WORKDIR /usr/src/app
COPY . .
EXPOSE 53
CMD [ "node", "index.js" ]
USER root

App is running in a single region (fra) with backup regions ams, cdg.
Any idea for this weird bug ?

Whoah this is an amazing bug report. We will look into this, thank you for investigating. :smiley:

1 Like

Our UDP forwarding path is completely oblivious to the contents of packets, so it’s hard to imagine what we could be doing to cause this… but here you are with a minimized test case. We’ll dig in. Thank you!

@Emmanuel_Di_Iorio Lemme ask you this: I have a UDP echo service running on crimson.sockpuppet.prg on 5000/udp; any UDP packet you send to that host/port should just bounce back at you.

Would it be easy for you to try to send a packet there and see if you get a response?

I’m not a node person, but if you can give me the bash steps it’d take to get your code running locally, I can test myself too.

Later update

Turned out all I needed to do was type node. Here’s what I see:

> server.connect(5000, "crimson.sockpuppet.org", () => {
...   const buffer = Buffer.alloc(3);
...   buffer.writeUInt8(0, 0);
...   buffer.writeUInt8(0, 1);
...   buffer.writeUInt8(7, 2); // <-- packet dropped because of this byte
...   server.send(buffer);
... });

Then, in a flyctl ssh console window on the app:

root@c718c543:/# tcpdump -vvnX -i any udp port 5000
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
18:48:18.732215 IP (tos 0x0, ttl 64, id 7174, offset 0, flags [DF], proto UDP (17), length 31)
    172.19.0.107.5000 > 12.216.156.90.62353: [udp sum ok] UDP, length 3
	0x0000:  4500 001f 1c06 4000 4011 c917 ac13 006b  E.....@.@......k
	0x0010:  0cd8 9c5a 1388 f391 000b 9c0d 0000 07    ...Z...........

That’s the reply to the packet being sent.

I’ll try moving it to fra real quick though.

Works on fra too.

I tried sending the same packet to crimson.sockpuppet.org and it indeed worked

So, I changed the server binding port to 5000 and app region from cdg/fra → ams (only ams, without backup regions) and still didn’t worked, and discovered that actually the same packet (regardless of the content) sometimes is getting delivered and sometimes not. Maybe there is a firewall (on my side or other sides) or maybe some Node.js weirdness, since your server is actually 100% reliable.

I will make more tests in the next days. For now, you can consider the ticket closed tho.
Really appreciate the time for the analysis @thomas

If you can’t figure out what’s going on here, please do keep making noise. A couple people are doing somewhat intense DNS stuff on Fly.io, so I have some reason to believe this stuff is working, but you might be spotting some corner case we need to know about.

I can’t reproduce this on my side, though.