Machines API deployment failing — kernel panic in NRT and SIN regions

Questions / Help Build debugging
,
1

Hi Fly.io team,

I’m deploying machines using the Fly Machines API, and I’m currently experiencing repeated failures when launching machines in the Tokyo (nrt) and Singapore (sin) regions.

Here’s what I’m seeing:

  • During machine startup, the init process panics with:

    thread 'main' panicked at init/src/main.rs:790:25: removal index (is 0) should be < len (is 0)
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00006500
...
Virtual machine exited abruptly

    Then the VM reboots and exits abruptly.

  • After several retries, the proxy starts reporting:

    [PM01] machines API returned an error: "rate limit exceeded"
[PR03] could not find a good candidate within 1 attempts at load balancing.

Machines created via the Fly.io CLI or GitHub deployment workflow can start successfully in the NRT region and run normally.
However, machines created through the Machines API consistently fail to start in NRT and SIN.
I’d like to resolve this issue because my project relies on automated daily deployments that switch regions based on timezone requirements.

Could you please help check if there’s a platform-level issue affecting these regions or the Machines API?

Thank you!

2

Hi… Actually, you checked already, without realizing it: the Fly.io CLI (flyctl) also uses the Machines API.

You can see this in action yourself by putting it into debug mode:

$ LOG_LEVEL=debug fly m run --region nrt debian:bookworm sleep inf
DEBUG --> POST https://api.machines.dev/v1/apps/multiverse/machines
                       ^^^^^^^^^^^^^^^^
DEBUG {
  "config": {
    "init": {
      "cmd": [
        "sleep",
        "inf"
      ]
    },
    "guest": {
      "cpu_kind": "shared",
      "cpus": 1,
      "memory_mb": 256
    },
    "image": "docker-hub-mirror.fly.io/library/debian:bookworm@sha256:<hex-string>",
    "dns": {}
  },
  "region": "nrt",
  "skip_secrets": false
}

(Assumes Bash syntax—and many of the output lines were elided.)

Odds are good that it’s instead a glitch in the JSON config object that you were passing in your other attempt; a missing cmd, perhaps, :thought_balloon:.

If you want, you can try posting the output of fly m status -d, applied to one of the Machines that exits abruptly, and perhaps one of us fellow users here in the community forum will spot the source of the trouble…

1 Like
3

this looks like a bug in our init process (it shouldn’t be crashing!), I’m not quite sure what is going on here but I’ve poked the team about it.

1 Like
4

Hi @jedi, does it fail for every machine created through the machine API? Can you share the json payload or try a different image?

1 Like
5

Hi, thanks for the follow-up!

Yes — it fails for every machine created through the Machines API under this account, but works fine when I create machines via the CLI or GitHub deployment workflow (using a different image, but in the same region).

Here’s the JSON payload I’m sending in my API request (from my script that runs daily to switch regions based on timezone):

{
  "region": "nrt",
  "config": {
    "image": "registry.fly.io/app_name:ci-latest",
    "guest": {
      "cpus": 2,
      "memory_mb": 512,
      "cpu_kind": "shared"
    },
    "services": [
      {
        "protocol": "tcp",
        "internal_port": 3000,
        "autostop": false,
        "autostart": true,
        "ports": [
          { "port": 80, "handlers": ["http"], "force_https": true },
          { "port": 443, "handlers": ["http", "tls"] }
        ]
      }
    ]
  }
}

This is the same payload used daily by my automation script (running in Google Apps Script).

The same Google Apps Script works perfectly on another account, which can deploy machines without any issues.

Thank you very much for your assistance.

6

Hi @dangra

Following your suggestion, I just changed the image name. The deployment is now successful, and everything is running smoothly.
Thank you very much for your help!

1 Like