"upstream service is unavailable" error?

I’m getting the same error running basically any flyctl command: “upstream service is unavailable”

When running fly doctor:

❯ flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: pinger: upstream service is unavailable)

I ran LOG_LEVEL=debug fly agent daemon-start, looks like the error is being returned from Fly’s GraphQL API:

Creating WireGuard peer "[...]" in region "dfw" for organization personal
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: AddWireGuardPeerInput!) { addWireGuardPeer(input: $input) { peerip endpointip pubkey } }","variables":{"input":{"name":"[...]","nats":false,"organizationId":"[...]","pubkey":"[...]","region":"dfw"}}}
DEBUG <-- 200 https://api.fly.io/graphql (618.43ms) {"data":{"addWireGuardPeer":null},"errors":[{"message":"upstream service is unavailable","locations":[{"line":1,"column":44}],"path":["addWireGuardPeer"],"extensions":{"code":"SERVICE_UNAVAILABLE"}}]}

Any idea what this issue is or how I could start debugging it?

Fly seem to be on the case. Hopefully fixed soon.

Awesome, thanks! I checked the status page but didn’t see anything… looks like they updated it since :slight_smile:

1 Like

We’ve deployed a fix to most of the gateways (that error message means that the message our API is sending to the gateway to create a new WireGuard peer isn’t being ack’d) and I’ve tested them, and we should be OK now. If you’re still seeing the error, can you tell me what region you’re hitting?

(We’re still diagnosing!)

1 Like

Thanks, Thomas! It looks like I’m still getting the same error on the DFW region.

Thanks! Give me one sec.

1 Like

Give it another shot? I thought DFW dodged this problem, applied the fix, and tested a new peer there myself.

Works now! Appreciate the help, you guys are fast.

Don’t give us too much credit for this! I should have caught DFW. Be ruthless in letting us know if things aren’t working the way you expect them to. Thank YOU for following up on this.