App can’t deploy: WireGuard 403 + remote builder 500

level11 · March 3, 2026, 8:25am

App: examphoria / Org: prod-examphoria

App is suspended despite good billing standing. Can’t deploy:

fly doctor shows WireGuard 403 (websocket: expected 101 but got 403), persists after wireguard websockets enable
Stuck on “Waiting for Depot Builder” while deploying, already tried to reset the builder from settings
Remote builders (both depot and legacy) fail with unable to upgrade to h2c, received 500
Destroyed and recreated builder — same result
Local Docker build works but needs WireGuard to push the image

Please unsuspend the app and check WireGuard/builder infra for prod-examphoria org. flyctl v0.4.15, region: iad.

lillian · March 3, 2026, 11:27am

403 on wireguard is quite odd, there is no path in the code that could return that status code. can you provide the complete output of fly doctor with the LOG_LEVEL=debug env variable added?

note that “suspended” is an informative and not administrative app state, it just means there are no machines running.

it does not. flyctl is probably doing something wrong here. can you try fly auth docker and pushing the image manually with docker cli, and then fly deploy --image <the image you just pushed>?

level11 · March 9, 2026, 8:46am

Hello, in the last days I manually scaled the app to “1” instance because I was unable to deploy and guarantee service for my client. Running fly doctor with debug log level, as you kindly requested, currently leads to this:

(base) alelisi@Mac resurgence-duos % fly doctor -c ExamphorIA/fly/production/fly.toml
DEBUG Loaded flyctl config from/Users/alelisi/.fly/config.yml
DEBUG determined hostname: "Mac.home"
DEBUG determined working directory: "/Users/alelisi/Documents/LVL11_2024/10_14_simone_gigantino/Untitled/resurgence-duos"
DEBUG determined config directory: "/Users/alelisi/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG checking for updates...
DEBUG started querying for statuspage incidents
DEBUG client initialized.
DEBUG app config loaded from ExamphorIA/fly/production/fly.toml
DEBUG started querying for host issues
DEBUG --> POST https://api.fly.io/graphql

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query($appName: String!) { apphostissues:app(name: $appName) { hostIssues { nodes { internalId message createdAt updatedAt } } } }",
  "variables": {
    "appName": "examphoria"
  }
}


DEBUG {
  "query": "query($admin: Boolean!) { organizations(admin: $admin) { nodes { id slug rawSlug name type paidPlan billable viewerRole internalNumericId } } }",
  "variables": {
    "admin": false
  }
}


DEBUG {}
DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (345.83ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"apphostissues":{"hostIssues":{"nodes":[]}}}}
DEBUG <-- 200 https://api.fly.io/graphql (362.25ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"organizations":{"nodes":[{"id":"896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ","slug":"personal","rawSlug":"alessandro-lisi-level11","name":"Alessandro Lisi (Level11)","type":"PERSONAL","paidPlan":false,"billable":true,"viewerRole":"admin","internalNumericId":"745351"},{"id":"zQpLKVXNGapwgH61jXMLyk5zbLcok2DRV","slug":"prod-examphoria","rawSlug":"prod-examphoria","name":"PROD-ExamphorIA","type":"SHARED","paidPlan":true,"billable":true,"viewerRole":"admin","internalNumericId":"886064"},{"id":"yw2NK09lG62eytyo75
DEBUG   <-- https://api.fly.io/graphql: jV2OwzbVT3PjjvN","slug":"prod-setti-fireworks","rawSlug":"prod-setti-fireworks","name":"PROD-Setti-Fireworks","type":"SHARED","paidPlan":true,"billable":true,"viewerRole":"admin","internalNumericId":"1379086"}]}}}
DEBUG app config loaded from ExamphorIA/fly/production/fly.toml
Testing authentication token... DEBUG Starting task manager
DEBUG --> POST https://api.fly.io/graphql

DEBUG monitoring tokens at /Users/alelisi/.fly/config.yml
DEBUG {
  "query": "query { viewer { ... on User { id email enablePaidHobby } ... on Macaroon { email } } }",
  "variables": null
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (218.83ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"viewer":{"id":"8xKyJZAnyRxvgTB0Dv79","email":"level11.software@gmail.com","enablePaidHobby":true}}}
PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }",
  "variables": {
    "slug": "personal"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (217.43ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"organization":{"id":"896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ","internalNumericId":"745351","slug":"personal","rawSlug":"alessandro-lisi-level11","name":"Alessandro Lisi (Level11)","type":"PERSONAL","billable":true,"limitedAccessTokens":{"nodes":[]}}}}
PASSED
Testing WireGuard DNS... DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }",
  "variables": {
    "slug": "personal"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (207.3ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"organization":{"id":"896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ","internalNumericId":"745351","slug":"personal","rawSlug":"alessandro-lisi-level11","name":"Alessandro Lisi (Level11)","type":"PERSONAL","billable":true,"limitedAccessTokens":{"nodes":[]}}}}
PASSED
Testing WireGuard Flaps... DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }",
  "variables": {
    "slug": "personal"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (212.15ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"organization":{"id":"896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ","internalNumericId":"745351","slug":"personal","rawSlug":"alessandro-lisi-level11","name":"Alessandro Lisi (Level11)","type":"PERSONAL","billable":true,"limitedAccessTokens":{"nodes":[]}}}}
PASSED
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { appcompact:app(name: $appName) { id internalNumericId name hostname cnameTarget deployed network status appUrl platformVersion organization { id internalNumericId slug paidPlan } postgresAppRole: role { name } } }",
  "variables": {
    "appName": "examphoria"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (248.13ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"appcompact":{"id":"examphoria","internalNumericId":6610152,"name":"examphoria","hostname":"examphoria.fly.dev","cnameTarget":"eoz5x1.examphoria.fly.dev","deployed":true,"network":"","status":"suspended","appUrl":"https://2a09:8280:1::64:dce8:0","platformVersion":"machines","postgresAppRole":null,"organization":{"id":"zQpLKVXNGapwgH61jXMLyk5zbLcok2DRV","internalNumericId":"886064","slug":"prod-examphoria","paidPlan":true}}}}
DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }",
  "variables": {
    "input": {
      "peerIps": [
        "fdaa:b:4e45:a7b:8dd7:0:a:102",
        "fdaa:f:e291:a7b:163a:0:a:102"
      ]
    }
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (194.75ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"validateWireGuardPeers":{"invalidPeerIps":[]}}}
WARN The running flyctl agent (v0.4.16) is older than the current flyctl (v0.4.19).
WARN The out-of-date agent will be shut down along with existing wireguard connections. The new agent will start automatically as needed.
DEBUG started agent process (pid: 43788, log: /Users/alelisi/.fly/agent-logs/1276621549.log)

App specific checks for examphoria:
Checking that app has ip addresses allocated... DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "query ($appName: String!) { app(name: $appName) { ipAddresses { nodes { id address type region createdAt network { name organization { slug } } serviceName } } sharedIpAddress } }",
  "variables": {
    "appName": "examphoria"
  }
}


DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (221.2ms)

DEBUG   <-- https://api.fly.io/graphql: {"data":{"app":{"sharedIpAddress":"66.241.125.89","ipAddresses":{"nodes":[{"id":"ip_704327k34p595r6k","address":"2a09:8280:1::64:dce8:0","type":"v6","region":"global","createdAt":"2025-02-11T11:17:57Z","network":null,"serviceName":null}]}}}}
PASSED
Checking A record for examphoria.fly.dev... PASSED
Checking AAAA record for examphoria.fly.dev... PASSED
DEBUG Task manager done
DEBUG done monitoring tokens
DEBUG querying for host issues resulted to []
DEBUG querying for statuspage incidents resulted to &{[]}
(base) alelisi@Mac resurgence-duos %

level11 · March 9, 2026, 8:50am

Here you go! Remember that in the last days, prior to running this command, I’ve scaled the app manually to 1 with fly scale to guarantee service to my client:

(base) alelisi@Mac resurgence-duos % fly doctor -c ExamphorIA/fly/production/fly.toml
DEBUG Loaded flyctl config from/Users/alelisi/.fly/config.yml
DEBUG determined hostname: “Mac.home”
DEBUG determined working directory: “/Users/alelisi/Documents/LVL11_2024/10_14_simone_gigantino/Untitled/resurgence-duos”
DEBUG determined config directory: “/Users/alelisi/.fly”
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG checking for updates…
DEBUG started querying for statuspage incidents
DEBUG client initialized.
DEBUG app config loaded from ExamphorIA/fly/production/fly.toml
DEBUG started querying for host issues
DEBUG → POST 


DEBUG → POST 


DEBUG {
“query”: “query($appName: String!) { apphostissues:app(name: $appName) { hostIssues { nodes { internalId message createdAt updatedAt } } } }”,
“variables”: {
“appName”: “examphoria”
}
}

DEBUG {
“query”: “query($admin: Boolean!) { organizations(admin: $admin) { nodes { id slug rawSlug name type paidPlan billable viewerRole internalNumericId } } }”,
“variables”: {
“admin”: false
}
}

DEBUG {}
DEBUG {}
DEBUG ← 200 
 (345.83ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“apphostissues”:{“hostIssues”:{“nodes”:
}}}}
DEBUG ← 200 
 (362.25ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“organizations”:{“nodes”:[{“id”:“896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ”,“slug”:“personal”,“rawSlug”:“alessandro-lisi-level11”,“name”:“Alessandro Lisi (Level11)”,“type”:“PERSONAL”,“paidPlan”:false,“billable”:true,“viewerRole”:“admin”,“internalNumericId”:“745351”},{“id”:“zQpLKVXNGapwgH61jXMLyk5zbLcok2DRV”,“slug”:“prod-examphoria”,“rawSlug”:“prod-examphoria”,“name”:“PROD-ExamphorIA”,“type”:“SHARED”,“paidPlan”:true,“billable”:true,“viewerRole”:“admin”,“internalNumericId”:“886064”},{“id”:“yw2NK09lG62eytyo75
DEBUG   ← https://api.fly.io/graphql: jV2OwzbVT3PjjvN”,“slug”:“prod-setti-fireworks”,“rawSlug”:“prod-setti-fireworks”,“name”:“PROD-Setti-Fireworks”,“type”:“SHARED”,“paidPlan”:true,“billable”:true,“viewerRole”:“admin”,“internalNumericId”:“1379086”}]}}}
DEBUG app config loaded from ExamphorIA/fly/production/fly.toml
Testing authentication token… DEBUG Starting task manager
DEBUG → POST 


DEBUG monitoring tokens at /Users/alelisi/.fly/config.yml
DEBUG {
“query”: “query { viewer { … on User { id email enablePaidHobby } … on Macaroon { email } } }”,
“variables”: null
}

DEBUG {}
DEBUG ← 200 
 (218.83ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“viewer”:{“id":“8xKyJZAnyRxvgTB0Dv79”,“email”:"level11.software@gmail.com”,“enablePaidHobby”:true}}}
PASSED
Testing flyctl agent… PASSED
Testing local Docker instance… Nope
Pinging WireGuard gateway (give us a sec)… DEBUG → POST 


DEBUG {
“query”: “query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }”,
“variables”: {
“slug”: “personal”
}
}

DEBUG {}
DEBUG ← 200 
 (217.43ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“organization”:{“id”:“896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ”,“internalNumericId”:“745351”,“slug”:“personal”,“rawSlug”:“alessandro-lisi-level11”,“name”:“Alessandro Lisi (Level11)”,“type”:“PERSONAL”,“billable”:true,“limitedAccessTokens”:{“nodes”:
}}}}
PASSED
Testing WireGuard DNS… DEBUG → POST 


DEBUG {
“query”: “query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }”,
“variables”: {
“slug”: “personal”
}
}

DEBUG {}
DEBUG ← 200 
 (207.3ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“organization”:{“id”:“896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ”,“internalNumericId”:“745351”,“slug”:“personal”,“rawSlug”:“alessandro-lisi-level11”,“name”:“Alessandro Lisi (Level11)”,“type”:“PERSONAL”,“billable”:true,“limitedAccessTokens”:{“nodes”:
}}}}
PASSED
Testing WireGuard Flaps… DEBUG → POST 


DEBUG {
“query”: “query($slug: String!) { organization(slug: $slug) { id internalNumericId slug rawSlug name type billable limitedAccessTokens { nodes { id name expiresAt revokedAt user { email } } } } }”,
“variables”: {
“slug”: “personal”
}
}

DEBUG {}
DEBUG ← 200 
 (212.15ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“organization”:{“id”:“896mQD3oYX6ANTBln3aOD7GN5OcpAMbYQ”,“internalNumericId”:“745351”,“slug”:“personal”,“rawSlug”:“alessandro-lisi-level11”,“name”:“Alessandro Lisi (Level11)”,“type”:“PERSONAL”,“billable”:true,“limitedAccessTokens”:{“nodes”:
}}}}
PASSED
DEBUG → POST 


DEBUG {
“query”: “query ($appName: String!) { appcompact:app(name: $appName) { id internalNumericId name hostname cnameTarget deployed network status appUrl platformVersion organization { id internalNumericId slug paidPlan } postgresAppRole: role { name } } }”,
“variables”: {
“appName”: “examphoria”
}
}

DEBUG {}
DEBUG ← 200 
 (248.13ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“appcompact”:{“id”:“examphoria”,“internalNumericId”:6610152,“name”:“examphoria”,“hostname”:“examphoria.fly.dev”,“cnameTarget”:“eoz5x1.examphoria.fly.dev”,“deployed”:true,“network”:“”,“status”:“suspended”,“appUrl”:“https://2a09:8280:1::64:dce8:0”,“platformVersion”:“machines”,“postgresAppRole”:null,“organization”:{“id”:“zQpLKVXNGapwgH61jXMLyk5zbLcok2DRV”,“internalNumericId”:“886064”,“slug”:“prod-examphoria”,“paidPlan”:true}}}}
DEBUG → POST 


DEBUG {
“query”: “mutation($input: ValidateWireGuardPeersInput!) { validateWireGuardPeers(input: $input) { invalidPeerIps } }”,
“variables”: {
“input”: {
“peerIps”: [
“fdaa:b:4e45:a7b:8dd7:0:a:102”,
“fdaa:f:e291:a7b:163a:0:a:102”
]
}
}
}

DEBUG {}
DEBUG ← 200 
 (194.75ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“validateWireGuardPeers”:{“invalidPeerIps”:
}}}
WARN The running flyctl agent (v0.4.16) is older than the current flyctl (v0.4.19).
WARN The out-of-date agent will be shut down along with existing wireguard connections. The new agent will start automatically as needed.
DEBUG started agent process (pid: 43788, log: /Users/alelisi/.fly/agent-logs/1276621549.log)

App specific checks for examphoria:
Checking that app has ip addresses allocated… DEBUG → POST 


DEBUG {
“query”: “query ($appName: String!) { app(name: $appName) { ipAddresses { nodes { id address type region createdAt network { name organization { slug } } serviceName } } sharedIpAddress } }”,
“variables”: {
“appName”: “examphoria”
}
}

DEBUG {}
DEBUG ← 200 
 (221.2ms)

DEBUG   ← https://api.fly.io/graphql: {“data”:{“app”:{“sharedIpAddress”:“66.241.125.89”,“ipAddresses”:{“nodes”:[{“id”:“ip_704327k34p595r6k”,“address”:“2a09:8280:1::64:dce8:0”,“type”:“v6”,“region”:“global”,“createdAt”:“2025-02-11T11:17:57Z”,“network”:null,“serviceName”:null}]}}}}
PASSED
Checking A record for examphoria.fly.dev… PASSED
Checking AAAA record for examphoria.fly.dev… PASSED
DEBUG Task manager done
DEBUG done monitoring tokens
DEBUG querying for host issues resulted to 

DEBUG querying for statuspage incidents resulted to &{
}
(base) alelisi@Mac resurgence-duos %

system · March 17, 2026, 7:51am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.