"upstream service is unavailable" error?

Questions / Help
1

I’m getting the same error running basically any flyctl command: “upstream service is unavailable”

When running fly doctor:

❯ flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: pinger: upstream service is unavailable)

I ran LOG_LEVEL=debug fly agent daemon-start, looks like the error is being returned from Fly’s GraphQL API:

Creating WireGuard peer "[...]" in region "dfw" for organization personal
DEBUG --> POST https://api.fly.io/graphql {{"query":"mutation($input: AddWireGuardPeerInput!) { addWireGuardPeer(input: $input) { peerip endpointip pubkey } }","variables":{"input":{"name":"[...]","nats":false,"organizationId":"[...]","pubkey":"[...]","region":"dfw"}}}
}
DEBUG <-- 200 https://api.fly.io/graphql (618.43ms) {"data":{"addWireGuardPeer":null},"errors":[{"message":"upstream service is unavailable","locations":[{"line":1,"column":44}],"path":["addWireGuardPeer"],"extensions":{"code":"SERVICE_UNAVAILABLE"}}]}

Any idea what this issue is or how I could start debugging it?

2

Fly seem to be on the case. Hopefully fixed soon.

3

Awesome, thanks! I checked the status page but didn’t see anything… looks like they updated it since :slight_smile:

1 Like
4

We’ve deployed a fix to most of the gateways (that error message means that the message our API is sending to the gateway to create a new WireGuard peer isn’t being ack’d) and I’ve tested them, and we should be OK now. If you’re still seeing the error, can you tell me what region you’re hitting?

(We’re still diagnosing!)

1 Like
5

Thanks, Thomas! It looks like I’m still getting the same error on the DFW region.

6

Thanks! Give me one sec.

1 Like
7

Give it another shot? I thought DFW dodged this problem, applied the fix, and tested a new peer there myself.

8

Works now! Appreciate the help, you guys are fast.

1 Like
9

Don’t give us too much credit for this! I should have caught DFW. Be ruthless in letting us know if things aren’t working the way you expect them to. Thank YOU for following up on this.

3 Likes
10

I am facing the same error today. What needs to be done on my end?

I ran fly wireguard create

11

@abeeshake

Could you give this another try? We resolved an issue today with one of our gateways that likely fixed this. Could you also confirm which region you were attempting to create the wireguard peer in?

12

it works now. I think this was temporary.

13

I have been facing the same error which says, upstream is unavailable

14

Platform: machines
✓ Configuration is valid
→ Verified app config
WARN DATABASE_URL may be a potentially sensitive environment variable. Consider setting it as a secret, and removing it from the [env] section: Secrets and Fly Apps · Fly Docs

==> Building image
WARN Failed to start remote builder heartbeat: Not authorized to access this firecrackerapp

Error: failed to fetch an image or build from source: error connecting to docker: failed building options: upstream service is unavailable

15

We have fixed upstream service is unavailable issue last week. Let me know if you still have the issue.

16

Hi, am having this issue

flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: pinger: upstream service is unavailable)

Trying from Illinois

17

I’m in the same situation.

$ fly doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: pinger: upstream service is unavailable)

I need ssh console but…

$ fly ssh console
Error: ssh: can't build tunnel for persona-chat: upstream service is unavailable