fly-proxy now supports ZSTD content encoding

Fresh Produce
1

If you use http handler for your service, which is the default setting for [http_service] fly.toml configuration, fly-proxy compresses HTTP responses sent by your app.

Brotli had been the preferred compression algorithm used by the proxy until now, as it provides good compression rate and fast compression/decompression speed.

Recently, Chrome added support for ZSTD compression algorithm (Chrome Platform Status), the other major browsers are expected to follow. fly-proxy now supports it too. Compared to Brotli, ZSTD is slightly faster to compress on the server side and much faster to decompress on the client side.

You don’t need to do anything to enable it for your app. If the client indicates support for ZSTD via Accept-Encoding HTTP header, the proxy will use it as the new preferred compression algorithm.

12 Likes
2

How to use gzip instead of zstd?

3

Hey @sarat

If you control the client you can just omit zstd and br from Accept-Encoding.
If you don’t control the client, you can gzip compress the responses in your app. The proxy won’t touch the response if it’s already compressed by an app.

1 Like
4

Got it. Thanks.

5

This default caused me some minor inconveniences. Firefox appears to support this now, but the “Response” tab in the DevTools doesn’t decode it correctly so you get a garbled result:

CleanShot 2024-06-21 at 11.19.11@2x
CleanShot 2024-06-21 at 11.19.11@2x1218Ă—942 249 KB

Trying to debug this using the “Copy as curl” option gave me a curl line that included this:

... -H 'Accept-Encoding: gzip, deflate, br, zstd'

But when I ran the curl command I got this error:

Unrecognized content encoding type. libcurl understands deflate, gzip content encodings.

I managed to figure it out (that’s how I learned that fly-proxy can do ZSTD now) but I imagine this trap will catch other people out too.

6

Hi Simon,

This comes down to the version of curl/libcurl you have. I have two computers here: the one with Ubuntu 20.04 does what you showed, but the one with 22.04 is happy. The command I tried:

curl --compressed -H "accept-encoding: gzip, deflate, br, zstd" https://debug.fly.dev

The key here is the --compressed flag which Firefox’s save as curl gives you. It instructs curl to try to decompress the response to show the final content. If curl doesn’t understand the compression then it’ll barf as you showed.

Anyway, on Ubuntu 22.04:

$ curl --compressed -H "accept-encoding: gzip, deflate, br, zstd" https://debug.fly.dev
=== Headers ===
Host: debug.fly.dev
Via: 2 fly.io
Fly-Forwarded-Proto: https
( other boring headers)
Accept: */*
X-Forwarded-Proto: https

2024-06-21 20:21:10.928443711 +0000 UTC m=+700475.696653173

You can check if your curl groks zstd:

$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

The one on Ubuntu 20.04 does not have zstd while the one on 22.04 above does (last line).

On 20.04 it’s Curl 7.68.0 (same version for libcurl). zstd support was introduced in curl/libcurl 7.70.

As for Firefox - zstd support was supposedly added in Firefox 126, the gibberish in the response payload might be a bug on their side or it could be “by design” depending on whether the idea is for “response payload” to show the verbatim binary blob that was received :slight_smile: (in your case though, since it’s ostensibly trying to decode json, I would expect it not to choke on the blob, but decompress and json-decode it!).

Cheers,

  • Daniel
1 Like
7

How can I get this to work? Here’s my app https://withered-silence-1670.fly.dev/

The server already compressed to gzip, I assume fly proxy will convert from gzip to zstd or brotli since the client specified it accepts those header.

8

@pmbanugo it was mentioned that compressed responses from the server aren’t modified, so you’ll have to change your server to send uncompressed responses.

9

Thanks!