Can't run command in background with Machines API exec

1

I run some long-running commands on a machine of mine using the machines API exec endpoint. Apparently something changed a few weeks ago because the same API calls now fail. My analysis showed that commands which should fork (run in the background and return immediately where called) do not fork any longer. I was able to confirm that passing the following command arrays to that endpoint actually waits for 4 seconds instead of returning immediately:

["setsid", "--fork", "sleep", "4"]
["bash", "-c", "sleep 4 &"]
["bash", "-c", "sleep 4", "&"]

In a terminal, they do return immediately.

Did anything change on the fly side of things? Or am I doing something wrong?

I have stumbled upon this while searching for “Client.Timeout exceeded while awaiting headers”, but I don’t know if it’s related.

1 Like
2

Could you post the curl console command, a response body, and response headers for an example call?

3

Sure, but there is nothing spectacular there:

curl --request POST   --url https://api.machines.dev/v1/apps/myapp/machines/d8xxxxxx/exec   --header 'Content-Type: application/json' --header 'Authorization: Bearer mytoken'   --data '{
  "command": [
    "setsid", "--fork", "sleep", "4"
  ],
  "timeout": 10
}'
{"stdout":"","stderr":"","exit_code":0,"exit_signal":0}
4

I, forgot the response headers:

HTTP/1.1 200 OK
content-type: application/json; charset=utf-8
fly-span-id: d26626d230e0eece
fly-trace-id: 526399400be28ab39256ed85a1acaf99
date: Mon, 03 Mar 2025 21:57:24 GMT
content-length: 55
x-envoy-upstream-service-time: 4031
server: Fly/d763a6058 (2025-03-03)
via: 1.1 fly.io
fly-request-id: 01JNF06YCV4SR3MTX7NWK0YQ97-fra
1 Like
5

Fair enough. Could you put a shell script in your Dockerfile which contains the & / --fork background device, so that you don’t have to specify it in your exec? Maybe the command doesn’t like to do the fork, but the container should be fine with it.

6

funny enough, in my exec I specify [run-script-in-bg.sh] with

# run-script-in-bg.sh

/path/to/script.sh &

and the behavior is the same. It blocks and times out.

7

Oh, that’s very odd. OK, what happens if you use docker exec <container> run-script-in-bg.sh locally on a running container?

8

Thank you for your help with this.

The container I am using is drakkan/sftpgo:v2.6. If I do podman exec sftpgo /var/lib/sftpgo/sleep-in-bg.sh with

# /var/lib/sftpgo/sleep-in-bg.sh
sleep 4 &

then it actually returns immediately and the sleep happens in the background (which I can observe with htop inside the container).

1 Like
9

Ooh, intriguing…

Next I would try having this script baked into your container, launch the machine in Fly in the usual way. Now if you were to use exec then it will block per your original report. But if you shell into the machine, and run it from the console, what happens in that case?

I was wondering if your exec endpoint was opening up an interactive shell for the lifetime of the command, but I can’t see any such option in the JSON request body.

Finally I would suggest launching a very common Docker image (e.g. Slim Ubuntu) in Fly in case there is something weird with the kernel of the image you’re using.

10

Same problem with fly machine exec, but it is not blocking when calling the script after I shell into the machine.

Reproduce:

  • fly machine create drakkan/sftpgo:v2.6 --name testing --rm --vm-cpu-kind shared --vm-cpus 1 --vm-memory 256
  • start machine, shell into it, create a script and chmod +x it. Call that script and watch it actually run in the background, not blocking. Script: 
    • #!/bin/bash
sleep 6 &
  • fly machine exec 32873d3f453e98 /var/lib/sftpgo/sleep-in-bg.sh → blocking.

The upstream docker image is based on debian bookworm.

I also tried it with the default nginx:latest image and I can observe the same behavior there.

1 Like
11

Ooh, righto, well I am quite out of ideas. That’s weird enough that I’d raise a call/bug for it, IMO.

Maybe flyctl exec is designed to watch the process and exit when the process exits, rather than behaving like docker exec?

12

Okay, thank you for looking into this with me.
Are you from the fly staff? Do you raise a bug internally? Or should I shoot an email to support@?

13

No, I’m just a customer. There’s support information here, though it looks like staff hereabouts are pretty flexible about diving into anything. I can’t speak for them though :zany_face:

14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.