machine creation stuck at 'started' state

[joprice]

I’m seeing behavior similar to Machines not deploying, stuck at preparing image , but when creating machines via the api. I have a similar setup already working in another app and was preparing a second environment. It uses the same coordination code for listing machines and creating missing ones. However, in this new app, machines never come up. They loop with Pulling container image / Successfully prepared image, then seem to be auto-destroyed.

Update: I was using a file secret and suspected that could be the issue. I removed it and the app started. I only added this file secret as a workaround, because a few weeks ago creating the machine and relying on the secret was not working. That is, the secret was never found. I’m confused now what state the two apps are in and wonder now if the first will fail on re-deploy.

[Sam-Fly]

@joprice If you want to verify the current state of the secrets on your machines, here’s what I’d do:

1. Run flyctl secrets list to get the list of secret names present on each app
2. use fly ssh console to ssh into a working machine in each app, and run env to get the values of the secrets present in the app.
3. Check for any mismatches or missing between the two apps
4. While ssh-ed in, check if the file secret is present, and if so what value it has.

If anything is missing or mismatched, you can update the secrets with fly secrets set.

[joprice]

Thank you for the tips. I do use this from time to time to verify things, but the reason I originally went with the file based secret was because I added a secret but it wasn’t “visible” from the manually created machine in the app using the api. After some thought, my only suspicion is that originally I had the secret staged, since the app never uses the deploy command. It simply hosts different machines with tags different env vars that correspond to the ai model it should load. In the new app, I noticed the secret was still in staged mode and did a deploy, which failed since it doesn’t have the same configuration as the api-generated machine, and then destroyed the machine. From that point on, the file based secret stopped working, so I removed it and now the secret it visible. So it seems I might have been misusing the system in the original deploy. I could avoid this setup if there was a way to provide a secret via the machine api in a secure way without it residing in the app (making the initial failed deploy unnecessary).