This is development notes from engineers at Fly. It is an experiment in transparency.
- Reject requests when backhauling for an instance that’s reached its hard limit, allowing our edge proxy to retry with a different, less loaded, instance.
- This appears to have improved performances by spreading the load better
- Now using adaptive http/2 window resizing
- Enable retrying canceled requests (canceled at connection time)
- Added a timeout where it was missing (e.g: creating backhaul connections)
- This could lead to needless waiting on bad connections
- Allow a much longer timeout for connections to end when restarting / deploying our proxy
- This resulted in less connection breakage upon reload
expose-fd listenerson our haproxy stats endpoint for hitless reloads
- We had upgraded to a more recent version of haproxy, but didn’t have the exact configuration required for “hitless” reloads (keeping connections alive on reload). More information here: https://www.haproxy.com/blog/hitless-reloads-with-haproxy-howto/
- Upgraded multiple dependencies
- Reduced the amount of reloading of our proxy configuration
- Started using Honeycomb for tracing slow requests
- We built our own Honeycomb client for Rust, it’s far from perfect, but it works
(this isn’t a particularly slow requests, but we also submit requests where there was an error)
- Upgraded the default kernel to 4.19.146
- This fixes the “firmware bug” kernel messages on some of our newer AMD Epyc hosts"