HTTP/2 Continuation Flood Protection

HTTP/2 Continuation Floods are a recently-disclosed Denial of Service vector that impacts HTTP/2 servers. Remember, every Fly.io app that exposes an HTTP service is transparently offering HTTP/2 through our Anycast network.

The TL;DR on this post is: we’ve upgraded our Anycast network to automatically break Continuation Floods. One less thing to worry about!

The rest of this is just fun detail.

HTTP/2, the successor to HTTP/1.1, is a binary protocol. Like in HTTP/1.1, clients make TCP connections to servers. But after that, it’s all different. HTTP/2 muxes many requests over a single connection. Each request is a stream of binary frames. Frames carry things like request data, settings, and, importantly, HTTP headers.

The HTTP headers of an HTTP/2 request are delivered in a sequence of HEADER and CONTINUATION frames. The last CONTINUATION frame has a flag indicating all headers have been sent. Headers are themselves compressed, and chunked across multiple frames. It’s a whole thing.

The attack is simple: open up an HTTP/2 connection, kick off the headers with a HEADERS frame, and then send a never-ending stream of CONTINUATION frames. Naive servers will keep unpacking and allocating memory for headers until they explode. Hee hee.

What’s fun about this attack is that it’s heavily leveraged. It targets server memory (and maybe compute), and so isn’t really volumetric; you don’t need a botnet to make it potent.

On the other hand, what’s nice about this attack is that it’s easily fixed. Just don’t accept unbounded numbers of CONTINUATION frames. So, we don’t. In fact: we didn’t even before this disclosure (we had a static maximum header list size), but we’ve also applied a patch to the h2 Rust crate we rely on to decisively fix this.

The long-term comprehensive fix for this problem is to make sure you’re using up-to-date versions of your web framework or HTTP libraries. Patches are rolling out now. But in the immediacy, if you’re deploying a web application on Fly.io, chances are the only way to speak HTTP/2 to it is through our proxies. And they now block this attack.

10 Likes