Deploying failing - "malformed authorization header"

[p-christ]

Hey,

Most of the time when we deploy from Github actions it fails with errors like below - any idea what this is about?

> [09/21] Waiting for 4d891331c67518 [app] to have state: started
> [10/21] Machine 5683dd94a6958e [app] has state: started
> [10/21] Checking that 5683dd94a6958e [app] is up and running
✔ [10/21] Machine 5683dd94a6958e [app] update succeeded
> [12/21] Machine 9080e015a139e8 [app] has state: started
> [12/21] Checking that 9080e015a139e8 [app] is up and running
✔ [12/21] Machine 9080e015a139e8 [app] update succeeded
✖ [11/21] Machine 6e82dd35f7d658 [app] update failed: failed to update VM 6e82dd35f7d658: request returned non-2xx status, 504
✖ [06/21] Machine 3287490eb3e685 [app] update failed: timed out waiting for machine to reach started state: failed to wait for VM 3287490eb3e685 in started state: Get "https://api.machines.dev/v1/apps/gizmo-frontend-production/machines/3287490eb3e685/wait?instance_id=01HETA2GXY90RGH08S59B8237S&state=started&timeout=60": net/http: request canceled
✖ [09/21] Machine 4d891331c67518 [app] update failed: timed out waiting for machine to reach started state: failed to wait for VM 4d891331c67518 in started state: Get "https://api.machines.dev/v1/apps/gizmo-frontend-production/machines/4d891331c67518/wait?instance_id=01HETA3SWB6MPANX4Q92BDFXGX&state=started&timeout=60": net/http: request canceled

and i click on the link it provides it says

{"error":"malformed authorization header"}

[pi3ch]

Same error. Deploying via Terraform.

Create request failed: 401 Unauthorized, {"error":"malformed authorization header"}

[pi3ch]

For the context this error started to appear today. Was the an update or outage?

[mootari]

The most recent update was likely Fly deploy changes for process groups and auto-scaled services which may have introduced some other changes under the hood.

Looking through the Terrform provider output (via DEBUG=1 TF_LOG=debug terraform apply) I can for example see the following redirect chain:

1. http://api.machines.dev
2. https://api.machines.dev/
3. https://docs.machines.dev/

which then returns the Swagger docs. I don’t know the significance of this initial request for the provider though.

Unfortunately the (unsupported) Terraform provider seems have become more of a liability than a helpful shortcut.

[pi3ch]

@mootari

Unfortunately the (unsupported) Terraform provider seems have become more of a liability than a helpful shortcut.

Sadly seems to be the case

[ben-io]

Why is the terraform provider requesting https://api.machines.dev/? It’s not an API endpoint, the docs redirect is expected.

[pi3ch]

It is here https://github.com/pi3ch/terraform-provider-fly/blob/main/internal/provider/provider.go#L22

const FLY_MACHINES_ENDPOINT string = "api.machines.dev"

[pi3ch]

BTW app is being made successfully. During machine creation it errs.

Request

2023-11-11T14:18:44.228+1100 [WARN]  unexpected data: terraform.local/local/fly:stdout="2023/11/11 14:18:44.227790 DEBUG [req] HTTP/1.1 POST http://api.machines.dev/v1/apps/my-machine/machines"
2023-11-11T14:18:44.228+1100 [WARN]  unexpected data:
  terraform.local/local/fly:stdout=
  | POST /v1/apps/my-machine/machines HTTP/1.1\r
  | Host: api.machines.dev\r
  | User-Agent: req/v3 (https://github.com/imroc/req)

Response

HTTP/1.1 301 Moved Permanently\r
  | location: https://api.machines.dev/v1/apps/my-machine/machines\r
  | server: Fly/442f90d3 (2023-11-07)\r
  | via: 1.1 fly.io\r
  | fly-request-id: 01HEY674SNC1T1ERKM6QSBYQV5-sin\r
  | content-length: 0\r

It looks like to be http to https redirect and it doesn’t carry the token.

Investigating…

[mootari]

FWIW we’re no longer using the fly_machine resource. Here is a simplified example of our workaround (which still needs to get battle-tested):

terraform {
  required_providers {
    restapi = { source = "Mastercard/restapi", version = "~> 1.18" }
  }
}

provider "restapi" {
  uri                  = "https://api.machines.dev/v1"
  update_method        = "POST"
  write_returns_object = true

  headers = {
    Authorization = "Bearer ${var.FLY_API_TOKEN}"
  }
}

resource "restapi_object" "my_machine" {
  path         = "/apps/${fly_app.my_app.name}/machines"
  destroy_path = "/apps/${fly_app.my_app.name}/machines/{id}?kill=true"

  data = jsonencode({
    // See https://fly.io/docs/machines/working-with-machines/#create-a-machine
  })
}

@ben-io

Why is the terraform provider requesting https://api.machines.dev/ ?

The provider specifies api.machines.dev as endpoint instead of api.machines.dev/v1 (source). My best guess is that it’s doing an initial request to check access (somewhere in here maybe).

@pi3ch

BTW app is being made successfully. During machine creation it errs.

The provider manages apps through the GraphQL API.

[pi3ch]

Ok, got it fixed: fixed malformed auth issue due to http to https redirect · fly-apps/terraform-provider-fly@c566744 · GitHub

Given upstream doesn’t release new PRs, I published it under my account: Terraform Registry (NOTE: this release only has a fix for fly machine api)