server sent GOAWAY and closed the connection

Questions / Help
1

I can’t fly launch. It creates a db cluster then throws this error (app name changed to app-name):

Error Failed attaching app-name-db to the Postgres cluster app-name: ssh: can't build tunnel for personal: server returned a non-200 status code: 504.\nTry attaching manually with 'fly postgres attach --app app-name app-name-db'

❯ fly postgres attach --app app-name app-name-db
Error ssh: can't build tunnel for personal: Post "https://api.fly.io/graphql": http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""

❯ fly doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: pinger: Post "https://api.fly.io/graphql": http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug="")

We can't establish connectivity with WireGuard for your personal organization.

WireGuard runs on 51820/udp, which your local network may block.

If this is the first time you've ever used 'flyctl' on this machine, you
can try running 'flyctl doctor' again.

udp port 51820 works fine, there’s nothing blocked on my home network.

2 Likes
2

I’ve been getting the exact same error since yesterday. Trying fly ping and fly wireguard reset also return the same result.

1 Like
3

Yeah, i’m not sure how to proceed. There doesn’t seem like there’s any way out of this error, so i’m blocked from using fly.io completely.

4

Same issue - though fly ping works, and fly wireguard reset gives me a timeout error.

Tried with the fra and ams regions, in case this has anything to do with it.

Edit: Tried a few more times and now it worked for me in the ams region.

5

I got a similar error on sin regions.

6

I am getting the same exact issue trying to upload a 125mb docker image.

EDIT: After a few retries, it now works.

7

Been getting this for 5 days in the syd region

1 Like
8

I’ve just tried to launch a Remix app as my initial Fly project and am getting this constantly. Also trying on the Syd region.

9

I just created a new Remix app and tried deploying to LAX region and am getting the same error.

Error error connecting to docker: failed building options: Post "https://api.fly.io/graphql": http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""

10

If some of you are getting this constantly, can you run fly like this:

env GODEBUG=http2debug=2 fly ...

This will output a lot of logs, but it might help us get to the bottom of it if you encounter a GOAWAY error again.

We haven’t been able to reproduce on our end, for now.

1 Like
11

@jerome This is happening fairly often (~50% or more of deploys) for us right now during near the end of a fly deploy command. Here’s the output right before it fails with GODEBUG added:

==> Creating release
2022/09/12 13:41:50 http2: Transport encoding header ":authority" = "api.fly.io"
2022/09/12 13:41:50 http2: Transport encoding header ":method" = "POST"
2022/09/12 13:41:50 http2: Transport encoding header ":path" = "/graphql"
2022/09/12 13:41:50 http2: Transport encoding header ":scheme" = "https"
2022/09/12 13:41:50 http2: Transport encoding header "authorization" = "Bearer <FILTERED>"
2022/09/12 13:41:50 http2: Transport encoding header "user-agent" = "fly/0.0.388"
2022/09/12 13:41:50 http2: Transport encoding header "content-type" = "application/json; charset=utf-8"
2022/09/12 13:41:50 http2: Transport encoding header "accept" = "application/json; charset=utf-8"
2022/09/12 13:41:50 http2: Transport encoding header "content-length" = "1824"
2022/09/12 13:41:50 http2: Transport encoding header "accept-encoding" = "gzip"
2022/09/12 13:41:50 http2: Framer 0x14000ac7a40: wrote HEADERS flags=END_HEADERS stream=9 len=14
2022/09/12 13:41:50 http2: Framer 0x14000ac7a40: wrote DATA flags=END_STREAM stream=9 len=1824 data="{\"query\":\"mutation($input: DeployImageInput!) { deployImage(input: $input) { release { id version reason description deploymentStrategy user { id email name } evaluationId createdAt } releaseCommand { id command evaluationId } } }\",\"variables\":{\"input\":{\"a" (1568 bytes omitted)
2022/09/12 13:42:50 http2: Framer 0x14000ac7a40: read GOAWAY len=8 LastStreamID=2147483647 ErrCode=NO_ERROR Debug=""
2022/09/12 13:42:50 http2: Transport received GOAWAY len=8 LastStreamID=2147483647 ErrCode=NO_ERROR Debug=""
2022/09/12 13:42:50 http2: Framer 0x14000ac7a40: read PING len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 13:42:50 http2: Transport received PING len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 13:42:50 http2: Framer 0x14000ac7a40: wrote PING flags=ACK len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 13:42:50 http2: Transport readFrame error on conn 0x140003e1c80: (*errors.errorString) EOF
2022/09/12 13:42:50 http2: Framer 0x14000ac7a40: wrote RST_STREAM stream=9 len=4 ErrCode=CANCEL
2022/09/12 13:42:50 RoundTrip failure: http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""
Error Post "https://api.fly.io/graphql": http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""
1 Like
12

@jerome And this is mine:

==> Creating release
2022/09/12 21:43:29 http2: Transport encoding header ":authority" = "api.fly.io"
2022/09/12 21:43:29 http2: Transport encoding header ":method" = "POST"
2022/09/12 21:43:29 http2: Transport encoding header ":path" = "/graphql"
2022/09/12 21:43:29 http2: Transport encoding header ":scheme" = "https"
2022/09/12 21:43:29 http2: Transport encoding header "user-agent" = "flyctl/0.0.389"
2022/09/12 21:43:29 http2: Transport encoding header "content-type" = "application/json; charset=utf-8"
2022/09/12 21:43:29 http2: Transport encoding header "accept" = "application/json; charset=utf-8"
2022/09/12 21:43:29 http2: Transport encoding header "authorization" = "***"
2022/09/12 21:43:29 http2: Transport encoding header "content-length" = "1073"
2022/09/12 21:43:29 http2: Transport encoding header "accept-encoding" = "gzip"
2022/09/12 21:43:29 http2: Framer 0xc000b5ae00: wrote HEADERS flags=END_HEADERS stream=9 len=14
2022/09/12 21:43:29 http2: Framer 0xc000b5ae00: wrote DATA flags=END_STREAM stream=9 len=1073 data="{\"query\":\"mutation($input: DeployImageInput!) { deployImage(input: $input) { release { id version reason description deploymentStrategy user { id email name } evaluationId createdAt } releaseCommand { id command evaluationId } } }\",\"variables\":{\"input\":{\"a" (817 bytes omitted)
2022/09/12 21:44:29 http2: Framer 0xc000b5ae00: read GOAWAY len=8 LastStreamID=2147483647 ErrCode=NO_ERROR Debug=""
2022/09/12 21:44:29 http2: Transport received GOAWAY len=8 LastStreamID=2147483647 ErrCode=NO_ERROR Debug=""
2022/09/12 21:44:29 http2: Framer 0xc000b5ae00: read PING len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 21:44:29 http2: Transport received PING len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 21:44:29 http2: Framer 0xc000b5ae00: wrote PING flags=ACK len=8 ping="\v{\xa2\xf0\x8b\x9b\xfeT"
2022/09/12 21:44:29 http2: Transport readFrame error on conn 0xc0002b3080: (*errors.errorString) EOF
2022/09/12 21:44:29 http2: Framer 0xc000b5ae00: wrote RST_STREAM stream=9 len=4 ErrCode=CANCEL
2022/09/12 21:44:29 RoundTrip failure: http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""
Error Post "https://api.fly.io/graphql": http2: server sent GOAWAY and closed the connection; LastStreamID=2147483647, ErrCode=NO_ERROR, debug=""

which looks pretty much the same.

1 Like
13

Thank you!

We’ve tweaked a few settings and now you should get a less cryptic error (504 gateway timeout) when this issue happens.

Such timeouts shouldn’t be happening in the first place though. We’re working on that.

1 Like