I have been looking into running a UDP syslog server but would like to limit access to it to only certain IPs, naturally.
Is there a best approach to doing this on fly?
Since fly does not have the equivalent of an AWS security group (at the platform level) would it mean installing something like iptables in my Dockerfile? I have read bits about it which apparently gets complicated when running inside docker with the host having its own rules.
For people that have IP-limited access before with fly, or with a Dockerfile in general, how did you do it?
We don’t have a great way to limit things to IPs, unfortunately. What you can do is:
If you trust the clients, connect them to wireguard and have them speak UDP over the private network
If you don’t, you could run some basic auth on Fly. You can actually run your own wireguard server if you want, or look at servers doing something like DTLS.
One thing about syslog: most people end up just running it over TCP with TLS, especially since log messages frequently span packets. There’s not a huge advantage to exposing UDP syslog in most cases.
For some background what I was hoping to try out was ingesting CDN logs. They want a hostname and port number to send data to. And they only output UDP. Not TCP. I checked (Else, yep, I’d just use Datadog or some such rival. None of those logging services seem to provide a hostname and port which supports UDP. Probably because, as you say, using TCP would indeed make more sense).
So … my plan was to run vector on the fly vm to act as a middleman. It would receive logs from the CDN over syslog UDP. Then simply output them to [wherever … S3, Datadog etc]. Which in theory should work.
But of course it would then be exposed to anyone. Not ideal. I’d want to limit syslog to IPs used by e.g the CDN or me to prevent anybody sending data to it.
I suspect wireguard is a bit beyond my capabilities I mean, I got it working locally to connect to my database, but for anything else, like running my own one, eek. Not sure about that. That wouldn’t work for other people as the domain wouldn’t resolve, I’m guessing. I would assume I need an IPv4 or hostname which resolves over the public internet.
I searched the usual places and do you know of any reason why this answer (using iptables) would/wouldn’t work with fly’s setup? (Scroll down to the bottom one where they say that’s how they managed it):
I can try it out myself, but just wondered if it was even worth pursuing.
Oh, while here: a feature request. A firewall at the platform level. Stop requests before they even get to the app
Guessing it’s on your roadmap at some point. You can probably do this stuff with wireguard like you say. But that’s maybe a bit intimidating for the casual migrating-from-X user.
The problem with iptables is while it should work, ip source lists, especially with cdn’s, can change frequently and without much notice. Which CDN are you using? If they have a fairly standard message format with a good identifier for log messages you could filter on that with vector. For example, if the syslog tag is set to mycdn-1234 you could just have vector only forward messages that contain that. It won’t stop anyone sending messages to vector, but it will stop them going anywhere else (unless they work out the magic string and really want to mess you up). This is similar to how Heroku suggest people deal with this issue with their log sinks.
It’s a little ‘security through obscurity’ but should also work reasonably well depending on what the messages are like.
However, Logplex does not support authentication for syslog drains. Therefore, it is possible for third parties to send log messages to your server if they know its address. As a security measure, you can filter out messages with unknown (or missing) drain tokens.
I don’t think you can put firewall rules on a DigitalOcean load balancer, so they’re roughly equivalent. It’s not a bad feature, just complicated to implement. Usually there’s a better way to lock down access. This particular time (because CDNs are silly), there’s not.
Go-MMProxy is a Go reimplementation of mmproxy, which was originally created by CloudFlare. Go-MMProxy improves on mmproxy’s runtime stability while providing potentially greater performance in terms of connection and packet throughput. Cloudflare’s blogpost on mmproxy serves as a great write-up on how go-mmproxy works under the hood.
I tested this with SSH and it works great, go-mmproxy also offers support for UDP so it shouldn’t be too hard to make it work for your use-case. Setting -p udp is all you need.
Here is my setup for SSH.
FLY
Forwarding port 10022 (as suggested by @kurt) to internal port 2222 where I have MMProxy listening on.
ip rule add from 127.0.0.1/8 iif lo table 123
ip route add local 0.0.0.0/0 dev lo table 123
#ip -6 rule add from ::1/128 iif lo table 123
#ip -6 route add local ::/0 dev lo table 123
I’m only routing IPv4. Docker doesn’t have IPv6 enabled by default. If you need it you could enable it when running in fly context only.
You need to enable --cap-add NET_ADMIN running in Docker, or you will get errors when running ip rule add. If you don’t need/want to do local testing, you can get around this by only setting up your ip rules in the fly context.
(Else, yep, I’d just use Datadog or some such rival. None of those logging services seem to provide a hostname and port which supports UDP. Probably because, as you say, using TCP would indeed make more sense).
I mean, I got it working locally to connect to my database, but for anything else, like running my own one, eek. Not sure about that. That wouldn’t work for other people as the domain wouldn’t resolve, I’m guessing. I would assume I need an IPv4 or hostname which resolves over the public internet.