Creating machines via Machines API is very slow compared to "fly scale count"

1

Assuming an app with currently 8 machines, doing fly scale count 10 will add 2 new machines.

The logs show


2024-10-08T17:57:45.841 runner[90801650b37ee8] iad [info] Pulling container image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT
2024-10-08T17:57:46.003 runner[90801650b37ee8] iad [info] Successfully prepared image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT (162.684737ms)
2024-10-08T17:57:46.685 runner[784326df27e458] iad [info] Pulling container image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT
2024-10-08T17:57:47.213 runner[784326df27e458] iad [info] Successfully prepared image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT (528.122802ms)
2024-10-08T17:57:47.240 runner[90801650b37ee8] iad [info] Configuring firecracker
2024-10-08T17:57:47.299 app[90801650b37ee8] iad [info] 2024-10-08T17:57:47.299432424 [01J9PMFA5W6GTTMBPQ7W89MGAH:main] Running Firecracker v1.7.0
2024-10-08T17:57:47.809 app[90801650b37ee8] iad [info] [ 0.425528] PCI: Fatal: No config space access function found
2024-10-08T17:57:48.071 app[90801650b37ee8] iad [info] INFO Starting init (commit: 04656915)...
2024-10-08T17:57:48.203 app[90801650b37ee8] iad [info] INFO Preparing to run: `docker-entrypoint.sh bun run src/worker.ts` as root
2024-10-08T17:57:48.205 app[90801650b37ee8] iad [info] INFO [fly api proxy] listening at /.fly/api
2024-10-08T17:57:48.251 runner[90801650b37ee8] iad [info] Machine created and started in 2.414s
2024-10-08T17:57:48.278 runner[784326df27e458] iad [info] Configuring firecracker
2024-10-08T17:57:48.351 app[90801650b37ee8] iad [info] 2024/10/08 17:57:48 INFO SSH listening listen_address=[fdaa:1:1205:a7b:2ec:4c40:21d9:2]:22 dns_server=[fdaa::3]:53
2024-10-08T17:57:48.355 app[784326df27e458] iad [info] 2024-10-08T17:57:48.355361658 [01J9PMFAZDSYR7EFDDPMBAWE5R:main] Running Firecracker v1.7.0
2024-10-08T17:57:48.819 app[784326df27e458] iad [info] [ 0.389731] PCI: Fatal: No config space access function found
2024-10-08T17:57:49.062 app[784326df27e458] iad [info] INFO Starting init (commit: 04656915)...
2024-10-08T17:57:49.182 app[784326df27e458] iad [info] INFO Preparing to run: `docker-entrypoint.sh bun run src/worker.ts` as root
2024-10-08T17:57:49.183 app[784326df27e458] iad [info] INFO [fly api proxy] listening at /.fly/api
2024-10-08T17:57:49.234 runner[784326df27e458] iad [info] Machine created and started in 2.567s

About 2.5s to create each machine.

Now, sending two consecutive POST requests to /v1/apps/{app-name}/machine with the same Docker image ref. The logs:

2024-10-08T18:01:58.531 runner[56830372c37128] iad [info] Pulling container image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT
2024-10-08T18:02:00.108 runner[e825479c737118] iad [info] Pulling container image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT
2024-10-08T18:02:10.798 runner[e825479c737118] iad [info] Successfully prepared image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT (10.689347091s)
2024-10-08T18:02:12.125 runner[e825479c737118] iad [info] Configuring firecracker
2024-10-08T18:02:12.199 app[e825479c737118] iad [info] 2024-10-08T18:02:12.199618404 [01J9PMQ2EMH6TB8F19M925P035:main] Running Firecracker v1.7.0
2024-10-08T18:02:12.732 app[e825479c737118] iad [info] [ 0.450109] PCI: Fatal: No config space access function found
2024-10-08T18:02:13.045 app[e825479c737118] iad [info] INFO Starting init (commit: 04656915)...
2024-10-08T18:02:13.188 app[e825479c737118] iad [info] INFO Preparing to run: `docker-entrypoint.sh bun run src/worker.ts` as root
2024-10-08T18:02:13.191 app[e825479c737118] iad [info] INFO [fly api proxy] listening at /.fly/api
2024-10-08T18:02:13.213 runner[e825479c737118] iad [info] Machine created and started in 13.12s
2024-10-08T18:02:13.363 app[e825479c737118] iad [info] 2024/10/08 18:02:13 INFO SSH listening listen_address=[fdaa:1:1205:a7b:2fb:17f2:f667:2]:22 dns_server=[fdaa::3]:53
2024-10-08T18:02:21.345 runner[56830372c37128] iad [info] Successfully prepared image registry.fly.io/app-name:deployment-01J9PM125XMFYWZAG2HAY9DDDT (22.813909663s)
2024-10-08T18:02:22.943 app[56830372c37128] iad [info] 2024-10-08T18:02:22.943168708 [01J9PMQ0XP9RWK8MXMA34HW2R4:main] Running Firecracker v1.7.0
2024-10-08T18:02:23.638 app[56830372c37128] iad [info] [ 0.571832] PCI: Fatal: No config space access function found
2024-10-08T18:02:24.358 app[56830372c37128] iad [info] INFO Preparing to run: `docker-entrypoint.sh bun run src/worker.ts` as root
2024-10-08T18:02:24.390 app[56830372c37128] iad [info] INFO [fly api proxy] listening at /.fly/api
2024-10-08T18:02:24.423 runner[56830372c37128] iad [info] Machine created and started in 25.899s

First one took 13s, second one took 25s. Pulling the same container image seems to take way longer via the Machines API for some reason.

Are there any plans to improve Machine creation speed via the Machines API? Or any plans to expose fly scale through the API? I’d like a quick and easy way to add X machines (just like fly scale) programmatically.
I know there’s autoscaler, but that’s based on metrics. There should be a simple way to achieve the same imperatively and programmatically…

2

fly scale count just uses the Machines API from the CLI, exactly like you. You can see the code here: flyctl/internal/command/scale/count_machines.go at master · superfly/flyctl · GitHub

I think the difference in creates is more likely due to worker placement. If you land on a host that already has the image pulled, it will be very fast. A host that needs to pull the image fresh is going to be slower.

1 Like
3

Yeah, I figured that, but for some reason my experience is fly scale count is usually faster than creating machines directly. Dunno…

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.