Fixing EOF errors on api.fly.io

1

Hey folks. We made a little change to our API that should reduce the number of intermittent EOF, 503, and 502 errors you’ve been seeing lately. If you’re still encountering them, please let us know!

Here’s some details if you’re curious…

The Fly API is made up of several apps – a regular 'old Rails app and a few go services – all running on Fly. (In fact, our stack is almost entirely normal Fly apps running right alongside yours. Turtles all the way down!) In front of these apps is an haproxy cluster at api.fly.io which proxies requests to the right backend.

haproxy used *.fly.dev addresses for backends which caused connections to go through fly-proxy twice:

ingress -> fly-proxy -> haproxy -> fly-proxy -> rails/etc

This extra layer isn’t great for performance and occasionally confused haproxy into closing in-flight connections. This is where the EOF errors come from.

After today, haproxy is connecting directly to backends over the private network (app-name.internal), which skips the second fly-proxy hop. So far metrics look promising and hopefully EOFs are under control.

3 Likes
2

Hi @michael,

Glad to hear things are getting better. I just ran into some EOF errors.

> fly ssh console
Error get app: Post "https://api.fly.io/graphql": write tcp [2601:280:c400:5c9:5b2e:3014:ccb1:4330]:37078->[2a09:8280:1:f28:246e:d6a:949:dbbf]:443: write: connection reset by peer

> fly ssh console                                                                                                                                    1 ↵
Error ssh: can't build tunnel for personal: error fetching dialer: establish failed: err err handling establish: can't load organizations from config: Post "https://api.fly.io/graphql": EOF

> fly ssh console 
Error get app: Post "https://api.fly.io/graphql": EOF

> fly ssh console                                                                                                                                    1 ↵
Error get app: Post "https://api.fly.io/graphql": EOF

> fly version
flyctl v0.0.260 linux/amd64 Commit: 250dc89 BuildDate: 2021-12-02T20:30:36Z