# Response body timing out in IAD & EWR (all good in 13 other regions)

**URL:** <https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727>\
**Category:** General\
**Tags:** proxy, login-only\
**Created:** [January 2, 2026, 7:46pm UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727 "2026-01-02T19:46:11Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 2, 2026, 7:46pm UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/1 "2026-01-02T19:46:12Z")

</div>

I have been tracking down some gnarly response body timeouts in IAD & EWR. The app is deployed in 15 regions, and IAD + EWR are the only regions where this issue is happening on a consistent basis. I dug deep in the app that sits behind [Fly.io](http://Fly.io) Proxy and everything looks correct.

To reproduce the issue (it may take a few requests to hit it):

```bash
curl --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: ewr' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'

* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
* Trying 137.66.16.250:443...
* Connected to changelog.com (137.66.16.250) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nightly.changelog.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: ewr]
> GET / HTTP/2
> Host: nightly.changelog.com
> User-Agent: curl/8.7.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: ewr
>
* Request completely sent off
< HTTP/2 200
< date: Fri, 02 Jan 2026 19:10:01 GMT
< fly-request-id: 01KE01ZTQ5D40PRR7CRECPXQ77-lhr
< last-modified: Fri, 02 Jan 2026 03:59:21 GMT
< server: Fly/fbde0e6c3 (2025-12-17)
< content-type: text/html
< vary: Accept-Encoding
< x-varnish: 295985 624063
< age: 132
< via: 2 fly.io, 2 fly.io, 1.1 080eee0c702098 (Varnish/7.7), 2 fly.io, 2 fly.io
< etag: W/"69574299-caa3c"
< accept-ranges: bytes
< x-request-id: 01KE01ZTQ5D40PRR7CRECPXQ77-lhr
< access-control-allow-origin: *
< cache-status: region=ewr; origin=nightly(localhost:5030),changelog-nightly-2023-10-10.fly.dev; ttl=-72.593; grace=86400.000; keep=604800.000; storage=storage.memory; hit; stale; hits=1
< content-length: 830012
< flyio-debug: {"n":"edge-cf-lon1-a6eb","nr":"lhr","ra":"195.144.8.28","rf":"Verbatim","sr":"ewr","sdc":"ewr1","sid":"080eee0c702098","st":0,"nrtt":8,"bn":"worker-cf-ewr1-7a70","mhn":"edge-cf-ewr1-b989","mrtt":69}
<
* Operation timed out after 20006 milliseconds with 0 out of 830012 bytes received
* Connection #0 to host changelog.com left intact
curl: (28) Operation timed out after 20006 milliseconds with 0 out of 830012 bytes received

```

Notice that the response headers are received, but the body remains stuck at `0 out of 830012 bytes`

I am able to reproduce the issue if I run the command on the LHR instance:

```bash
root@2873321a446d58:/# curl --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: ewr' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'
* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [1568 bytes data]
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to changelog.com (137.66.16.250) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nightly.changelog.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: ewr]
} [5 bytes data]
> GET / HTTP/2
> Host: nightly.changelog.com
> User-Agent: curl/8.14.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: ewr
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* Request completely sent off
{ [5 bytes data]
< HTTP/2 200
< date: Fri, 02 Jan 2026 19:29:11 GMT
< fly-request-id: 01KE02YY42T1SCBVQAGZEC2PXG-lhr
< last-modified: Fri, 02 Jan 2026 03:59:21 GMT
< server: Fly/fbde0e6c3 (2025-12-17)
< content-type: text/html
< vary: Accept-Encoding
< x-varnish: 2359342 1966767
< age: 2
< via: 2 fly.io, 2 fly.io, 1.1 080eee0c702098 (Varnish/7.7), 2 fly.io, 2 fly.io
< etag: W/"69574299-caa3c"
< accept-ranges: bytes
< x-request-id: 01KE02YY42T1SCBVQAGZEC2PXG-lhr
< access-control-allow-origin: *
< cache-status: region=ewr; origin=nightly(localhost:5030),changelog-nightly-2023-10-10.fly.dev; ttl=57.830; grace=86400.000; keep=604800.000; storage=storage.memory; hit; hits=1
< content-length: 830012
< flyio-debug: {"n":"worker-cf-lon1-6b6a","nr":"lhr","ra":"172.19.29.50","rf":"Verbatim","sr":"ewr","sdc":"ewr1","sid":"080eee0c702098","st":0,"nrtt":0,"bn":"worker-cf-ewr1-7a70","mhn":"edge-cf-ewr1-7292","mrtt":69}
<
} [5 bytes data]
  0 810k 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0* Operation timed out after 20002 milliseconds with 0 out of 830012 bytes received
  0 810k 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0
* Connection #0 to host changelog.com left intact
curl: (28) Operation timed out after 20002 milliseconds with 0 out of 830012 bytes received

```

If I run the command directly on the EWR instance I am NOT able to reproduce:

```bash
curl --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: ewr' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'
* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [1568 bytes data]
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to changelog.com (137.66.16.250) port 443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nightly.changelog.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.14.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: ewr]
} [5 bytes data]
> GET / HTTP/2
> Host: nightly.changelog.com
> User-Agent: curl/8.14.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: ewr
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* Request completely sent off
{ [5 bytes data]
< HTTP/2 200
< date: Fri, 02 Jan 2026 19:26:56 GMT
< fly-request-id: 01KE02V1QRPBDDWNDZAT5NZ7J3-ewr
< last-modified: Fri, 02 Jan 2026 03:59:21 GMT
< server: Fly/fbde0e6c3 (2025-12-17)
< content-type: text/html
< vary: Accept-Encoding
< x-varnish: 624196 1966746
< age: 9
< via: 2 fly.io, 2 fly.io, 1.1 080eee0c702098 (Varnish/7.7), 2 fly.io
< etag: W/"69574299-caa3c"
< accept-ranges: bytes
< x-request-id: 01KE02V1QRPBDDWNDZAT5NZ7J3-ewr
< access-control-allow-origin: *
< cache-status: region=ewr; origin=nightly(localhost:5030),changelog-nightly-2023-10-10.fly.dev; ttl=50.966; grace=86400.000; keep=604800.000; storage=storage.memory; hit; hits=12
< content-length: 830012
< flyio-debug: {"n":"worker-cf-ewr1-7a70","nr":"ewr","ra":"172.19.2.162","rf":"Verbatim","sr":"ewr","sdc":"ewr1","sid":"080eee0c702098","st":0,"nrtt":0,"bn":null,"mhn":null,"mrtt":null}
<
{ [441 bytes data]
100 810k 100 810k 0 0 29.6M 0 --:--:-- --:--:-- --:--:-- 30.4M
* Connection #0 to host changelog.com left intact

```

I have similar issues with the IAD instance, but this one is failing even before returning any headers:

```bash
curl --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: iad' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'
* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* Connected to changelog.com (137.66.16.250) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nightly.changelog.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: iad]
> GET / HTTP/2
> Host: nightly.changelog.com
> User-Agent: curl/8.7.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: iad
>
* Request completely sent off
  0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0* Operation timed out after 20006 milliseconds with 0 bytes received
  0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0
* Connection #0 to host changelog.com left intact
curl: (28) Operation timed out after 20006 milliseconds with 0 bytes received

```

I can see that 2700 connections are currently opened so maybe that has something to do with it:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/e/5/e5f8fca5815b8633c9964a6b7cb64c492688cdbf.png)

I am not sure why the [Fly.io](http://Fly.io) Proxy is not closing these connections (idle timeout is configured to 60 & concurrency is set to requests (not connections). FTR, here is the [fly.toml](https://github.com/thechangelog/pipely/blob/a76a0e1133a10128a3250e647286fe3dd0660ec3/fly.io/cdn-2025-12-06/fly.toml).

Also, [these checks are running every hour](https://github.com/thechangelog/pipely/actions/workflows/check_all.yml), and currently failing due to timeouts in these two regions - IAD & EWR. As a stop-gap, I am going to restart the instances, but the fix is temporary (the issue comes back after a few days at most, usually a few hours).

Is there anything else that I can provide to help debug this?

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 2, 2026, 8:53pm UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/2 "2026-01-02T20:53:32Z")

</div>

There is another significant piece of info that I missed: forcing the request to HTTP/1.1 always works. In other words, HTTP/2 triggers this issue:

```bash
curl --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: ewr' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'

* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* Connected to changelog.com (137.66.16.250) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: nightly.changelog.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: ewr]
> GET / HTTP/2
> Host: nightly.changelog.com
> User-Agent: curl/8.7.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: ewr
>
* Request completely sent off
< HTTP/2 200
< date: Fri, 02 Jan 2026 20:50:39 GMT
< fly-request-id: 01KE07M4EGWYHYH81HHJHEK4H7-lhr
< last-modified: Fri, 02 Jan 2026 03:59:21 GMT
< server: Fly/fbde0e6c3 (2025-12-17)
< content-type: text/html
< vary: Accept-Encoding
< x-varnish: 557308 557306
< age: 2
< via: 2 fly.io, 2 fly.io, 1.1 080eee0c702098 (Varnish/7.7), 2 fly.io, 2 fly.io
< etag: W/"69574299-caa3c"
< accept-ranges: bytes
< x-request-id: 01KE07M4EGWYHYH81HHJHEK4H7-lhr
< access-control-allow-origin: *
< cache-status: region=ewr; origin=nightly(localhost:5030),changelog-nightly-2023-10-10.fly.dev; ttl=57.198; grace=86400.000; keep=604800.000; storage=storage.memory; hit; hits=1
< content-length: 830012
< flyio-debug: {"n":"edge-cf-lon1-8e62","nr":"lhr","ra":"31.222.222.66","rf":"Verbatim","sr":"ewr","sdc":"ewr1","sid":"080eee0c702098","st":0,"nrtt":0,"bn":"worker-cf-ewr1-7a70","mhn":"edge-cf-ewr1-b989","mrtt":69}
<
  0 810k 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0* Operation timed out after 20006 milliseconds with 0 out of 830012 bytes received
  0 810k 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0
* Connection #0 to host changelog.com left intact
curl: (28) Operation timed out after 20006 milliseconds with 0 out of 830012 bytes received

```

I cannot reproduce the failure when forcing HTTP/1.1:

```bash
curl --http1.1 --verbose --output /dev/null --header 'Host: nightly.changelog.com' --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: ewr' --connect-timeout 10 --max-time 20 --resolve changelog.com:443:137.66.16.250 'https://changelog.com/'
* Added changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* Connected to changelog.com (137.66.16.250) port 443
* ALPN: curl offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [315 bytes data]
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [25 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted http/1.1
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: nightly.changelog.com
> User-Agent: curl/8.7.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: ewr
>
* Request completely sent off
< HTTP/1.1 200 OK
< date: Fri, 02 Jan 2026 20:50:39 GMT
< fly-request-id: 01KE07PM559RXKGZ2ERA98XVYG-lhr
< last-modified: Fri, 02 Jan 2026 03:59:21 GMT
< server: Fly/fbde0e6c3 (2025-12-17)
< content-type: text/html
< vary: Accept-Encoding
< x-varnish: 360493 557306
< age: 84
< via: 2 fly.io, 2 fly.io, 1.1 080eee0c702098 (Varnish/7.7), 1.1 fly.io, 1.1 fly.io
< etag: W/"69574299-caa3c"
< accept-ranges: bytes
< x-request-id: 01KE07PM559RXKGZ2ERA98XVYG-lhr
< access-control-allow-origin: *
< cache-status: region=ewr; origin=nightly(localhost:5030),changelog-nightly-2023-10-10.fly.dev; ttl=-24.516; grace=86400.000; keep=604800.000; storage=storage.memory; hit; stale; hits=2
< content-length: 830012
< connection: keep-alive
< flyio-debug: {"n":"edge-cf-lon1-a6eb","nr":"lhr","ra":"31.222.222.66","rf":"Verbatim","sr":"ewr","sdc":"ewr1","sid":"080eee0c702098","st":0,"nrtt":0,"bn":null,"mhn":"worker-cf-ewr1-7a70","mrtt":69}
<
{ [1450 bytes data]
100 810k 100 810k 0 0 1188k 0 --:--:-- --:--:-- --:--:-- 1186k
* Connection #0 to host changelog.com left intact

```

---

<div class="post-metadata">

**Author:** ![mayailurus](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/mayailurus/32/17593_2.png) [@mayailurus](https://community.fly.io/u/mayailurus)\
**Post date:** [January 2, 2026, 9:08pm UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/3 "2026-01-02T21:08:26Z")

</div>

> [@gerhard](#):
>
> (idle timeout is configured to 60 & concurrency is set to requests (not connections). FTR, here is the [fly.toml](https://github.com/thechangelog/pipely/blob/a76a0e1133a10128a3250e647286fe3dd0660ec3/fly.io/cdn-2025-12-06/fly.toml).

Hm… I don’t know whether this is the source of _all_ your problems, but the configuration of the idle timeout is actually malformed:

```auto
[http_service.http_options]
  idle_timeout = 60

[[services]]
  internal_port = 9000

# ...

[[services.ports]]
  handlers = ["tls", "http"]
  port = 443

```

There is no `[http_service]` block above the `[http_service.options]` one, so I believe this is getting interpreted as an HTTP service declaration that’s _missing_ almost all of its required keys.

(TOML syntax is rather strange about this kind of thing.)

Moreover, the Fly Proxy is generally unhappy when you have both `[http_service]` and `[[services]]` simultaneously. It’s really common to see strange behavior in that situation, 🐉.

Hope this helps a little!

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 3, 2026, 11:42am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/4 "2026-01-03T11:42:20Z")

</div>

That was a good catch!

I made the change in [Right-size cdn-2025-12-06 + cleanup by gerhard · Pull Request #49 · thechangelog/pipely · GitHub](https://github.com/thechangelog/pipely/pull/49) , captured all the context (especially the machine configs before & after), and rolled it out.

Even before the manifest fix got applied, the issue fixed itself:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/1/4/1444a46167582e2f7abdd682545fe868723e4ae3.png)

All instances have been healthy for the last 6 hours, since Jan 3, 5:33 AM GMT: [Check all instances · Workflow runs · thechangelog/pipely · GitHub](https://github.com/thechangelog/pipely/actions/workflows/check_all.yml)

Will keep an eye on this & ensure that everything remains healthy for the next 48 hours.

Thank you all 💪

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 6, 2026, 7:45am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/5 "2026-01-06T07:45:44Z")

</div>

Apart from a few slower regions (NRT → IAD & GRU → IAD) everything has been working well since:

- [8 runs timed out](https://github.com/thechangelog/pipely/actions/workflows/check_all.yml?query=is%3Acancelled+branch%3Amain) (the responses took too long)
- [9 runs failed](https://github.com/thechangelog/pipely/actions/workflows/check_all.yml?query=is%3Afailure+branch%3Amain)
- [80 succeeded](https://github.com/thechangelog/pipely/actions/workflows/check_all.yml?query=is%3Asuccess+branch%3Amain)

That gives us an 82% success ratio across all regions which is not great, but a significant improvement over what we had before (0%).

Will check in again in a week’s time. FWIW: [Kaizen 22 - Let it Crash - January, 2026](https://github.com/thechangelog/changelog.com/discussions/554)

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 10, 2026, 12:29pm UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/6 "2026-01-10T12:29:03Z")

</div>

The issue is back in DFW:

```bash
curl --output /dev/null --verbose --header 'Flyio-Debug: doit' --header 'Fly-Force-Region: dfw' --connect-timeout 10 --max-time 30 --resolve changelog.com:443:137.66.16.250 --resolve cdn.changelog.com:443:137.66.16.250 --resolve nightly.changelog.com:443:137.66.16.250 'https://changelog.com/master/feed'
* Added changelog.com:443:137.66.16.250 to DNS cache
* Added cdn.changelog.com:443:137.66.16.250 to DNS cache
* Added nightly.changelog.com:443:137.66.16.250 to DNS cache
* Hostname changelog.com was found in DNS cache
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 137.66.16.250:443...
* Connected to changelog.com (137.66.16.250) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [318 bytes data]
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [2039 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
* subject: CN=changelog.com
* start date: Dec 31 16:52:47 2025 GMT
* expire date: Mar 31 16:52:46 2026 GMT
* subjectAltName: host "changelog.com" matched cert's "changelog.com"
* issuer: C=US; O=Let's Encrypt; CN=E8
* SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://changelog.com/master/feed
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: changelog.com]
* [HTTP/2] [1] [:path: /master/feed]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [flyio-debug: doit]
* [HTTP/2] [1] [fly-force-region: dfw]
> GET /master/feed HTTP/2
> Host: changelog.com
> User-Agent: curl/8.7.1
> Accept: */*
> Flyio-Debug: doit
> Fly-Force-Region: dfw
>
* Request completely sent off
< HTTP/2 200
< cache-control: max-age=3600, s-maxage=604800, stale-while-revalidate=604800, stale-if-error=604800
< cf-cache-status: DYNAMIC
< cf-ray: 9bbaddeca9b8f06e-DFW
< content-type: application/xml
< date: Sat, 10 Jan 2026 08:39:58 GMT
< etag: W/"92c0acaacc67a56f1989a83fd5b5aa12"
< last-modified: Wed, 07 Jan 2026 21:01:31 GMT
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=M%2FF7F5C62GP%2FCpYdZKg%2ForC7iBtnFlnjzavKiJkyg5I9dKjhewRGOvZjbm19sZd3hjUvp%2BG4nBE5U1iMgmj6SvhWEi8KSJ4SbYqeE4AAHcrLq%2FpXIUFqTz2n%2F%2B54jhLZJw%3D%3D"}]}
< server: Fly/f93a653b1 (2026-01-08)
< vary: Accept-Encoding
< x-varnish: 100541 98306
< age: 13456
< via: 1.1 1857001cd725e8 (Varnish/7.7), 2 fly.io, 2 fly.io
< accept-ranges: bytes
< x-request-id: 01KEKXTHCFYA5SH5EPB98H0V1X-lhr
< cache-status: region=dfw; origin=feeds(localhost:5020),feeds.changelog.place; ttl=29743.231; grace=86400.000; keep=604800.000; storage=storage.memory; hit; hits=143
< content-length: 14010021
< fly-request-id: 01KEKXTHCFYA5SH5EPB98H0V1X-lhr
< flyio-debug: {"n":"edge-cf-lon1-a6eb","nr":"lhr","ra":"195.144.8.28","rf":"Verbatim","sr":"dfw","sdc":"dfw1","sid":"1857001cd725e8","st":0,"nrtt":0,"bn":"worker-cf-dfw1-5eac","mhn":"edge-cf-dfw1-9a88","mrtt":103}
<
  0 13.3M 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0* Operation timed out after 30002 milliseconds with 0 out of 14010021 bytes received
  0 13.3M 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0
* Connection #0 to host changelog.com left intact
curl: (28) Operation timed out after 30002 milliseconds with 0 out of 14010021 bytes received

```

The instance is receiving little traffic compared to all the other instances, and my interpretation of this graph is that the [Fly.io](http://Fly.io) Proxy is not terminating idle connections after 60s, as configured:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/9/2/92d6a23181b6c6eda310f3ae10a23ad4a9a101db.png)

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 11, 2026, 7:49am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/7 "2026-01-11T07:49:23Z")

</div>

I removed DFW, and the connections went away after 8h of not running anything in that region:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/8/6/865521fe369fbcaa3d64b105352656777cff49cb.png)

Since then, ORD has been failing to transfer response bodies consistently, and AMS had brief periods too.

To summarise the problem:

- HTTP/1.1 works
- Running the same command on the instance works
- The issue is limited to HTTP/2 through the [Fly.io](http://Fly.io) Proxy

This is the current fly.toml:

```ini
# Full app config reference: https://fly.io/docs/reference/configuration/
app = "cdn-2025-12-06"
# Closest to the APP origin
primary_region = "iad"

kill_signal = "SIGTERM"
kill_timeout = 30

[deploy]
  strategy = "rolling"
  max_unavailable = 1

[http_service]
  internal_port = 9000
  force_https = false

[http_service.http_options]
  idle_timeout = 60

[http_service.concurrency]
  # This matches Varnish's thread_pools * thread_pool_max - 10%
  hard_limit = 2700
  soft_limit = 2000
  type = "requests"

[[http_service.checks]]
  grace_period = "5s"
  interval = "5s"
  method = "get"
  path = "/health"
  timeout = "3s"

[mounts]
  source = "varnish_file_cache"
  destination = "/var/lib/varnish/cache"

```

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 11, 2026, 9:19am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/8 "2026-01-11T09:19:30Z")

</div>

I have removed ORD, and added DWF back.

[I have also configured periodic HTTP/1.1 checks](https://github.com/thechangelog/pipely/pull/53), alongside the existing HTTP/2 checks.

[The first run succeeded across all 10 regions](https://github.com/thechangelog/pipely/actions/runs/20892048223/job/60024597520?pr=53) - SJC, LAX, LHR, CDG, IAD, DFW, FRA, AMS, SIN, NRT - let’s see how it behaves over the next few days:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/d/2/d211c344852465e2c671ced0ee36d9d4ef8dc547.jpeg)

Current app concurrency state - we switched away from ORD, notice the lingering connections:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/5/9/5922d41df16fd937aff2816587598402c7a1aed4.png)

---

<div class="post-metadata">

**Author:** ![gerhard](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/gerhard/32/40165_2.png) [@gerhard](https://community.fly.io/u/gerhard)\
**Post date:** [January 11, 2026, 10:47am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/9 "2026-01-11T10:47:33Z")

</div>

DFW got back in the same state within an hour of running, had to remove it:

 ![image](https://global.discourse-cdn.com/fly1/original/3X/9/d/9d22d92a7b0655581aad98162cca5cf0fcff0564.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/fly1/original/3X/6/e/6e8955d00cb218e186ebbe3a6ab06fefdce15290.svg) [@system](https://community.fly.io/u/system)\
**Post date:** [January 18, 2026, 10:48am UTC](https://community.fly.io/t/response-body-timing-out-in-iad-ewr-all-good-in-13-other-regions/26727/10 "2026-01-18T10:48:05Z")

</div>

Closed on January 18, 2026, 10:48am UTC

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.
