Fly Apps on machines: prerelease of fly deploy

We’re working to migrate the Fly Apps platform to Machines. We’re calling this Apps v2. Apps v1 are the apps that use Nomad.

The first big step will be making new apps use the machines platform instead of nomad. There’s a bunch of small steps to get there. Today we’re sharing a prerelease and PR that improves fly deploy for apps with machines.

Install the prerelease or build the PR to try out the new fly deploy behavior:

To install the prerelease:

curl -L https://fly.io/install.sh | sh -s -- prerelease
fly version
fly v0.0.451-pre-1 darwin/arm64 Commit: 8a0f2ae8 BuildDate: 2023-01-13T23:46:04Z

Try it out and let us know what you think! We’re looking for bugs we can squash, missing features, things you’d like see. Let us know in this thread.

Getting started

You’ll need an app that has machines in it and hasn’t been launch on nomad. If you don’t have one lying around, you can make one with:

fly launch --image nginx
...
? Would you like to set up a Postgresql database now? No
? Would you like to set up an Upstash Redis database now? No
? Would you like to deploy now? No

fly machine run nginx
fly ips allocate-v6

The ip isn’t necessary. It helps if you want to see the nginx service running at app_name.fly.dev. Make sure to change the internal port in fly.toml to port 80.

First deploy

Then run: fly deploy

% fly deploy
...
? Deploying machines with `fly deploy` is highly experimental and may produce unexpected results. Proceed? Yes
...
WARN Found 1 machines that are not part of the Fly Apps Platform:
winter-grass-7821 machines
ID              NAME                    STATE   REGION  IMAGE                   IP ADDRESS                      VOLUME  CREATED                     LAST UPDATED
39080ddec59e87  snowy-grass-4279        started sea     library/nginx:latest    fdaa:0:bfcd:a7b:a6:9b78:fc9c:2          2023-01-13T21:53:14Z        2023-01-13T21:53:15Z

? Migrate 1 existing machines into Fly Apps Platform? Yes
INFO Migrating 1 machines to the Fly Apps Platform
Deploying winter-grass-7821 app with rolling strategy
  Waiting for 39080ddec59e87 to have state started, currently: started
  Finished deploying

You’ll be prompted to move the existing machine(s) into the Fly Apps Platform. This is a one time step.

The Fly Apps Platform is all the things it takes to run and orchestrate Fly Apps—restarts, deploys, scale, secrets, etc. Machines that are part of the Fly Apps Platform get deployed to as a group, and will be updated based on the fly.toml app configuration. We use some metadata on the machine’s config to indicate it’s part of the Fly Apps Platform.

Subsequent deploys

Subsequent deploys will update all of the machines that have the metadata indicating they are part of the Fly Apps Platform.

Other machines are simply ignored. Other machines are ones creates using fly machines run or with the Machines API. Developer will need to manage these other machines themselves using the fly machines commands or the Machines API.

Scaling

We don’t yet support using fly scale to create/destroy machines in Apps v2. Right now, you’ll need add new machines to the App using fly machines clone or the Machines API and ensure the metadata fields are included in the config of the new machines. For example, we can start 3 new machines using:

for i in $(seq 1 3) ; do
  fly machine clone 39080ddec59e87
done

The next deployment will apply to all 4 machines in the app:

fly deploy
--> Verified app config
==> Building image
Searching for image 'nginx' remotely...
image found: img_wd57v5nge95v38o0
? Deploying machines with `fly deploy` is highly experimental and may produce unexpected results. Proceed? Yes
...
Deploying winter-grass-7821 app with rolling strategy
  Machine e148eee9bd6789 has state: started
  Machine 9185905b4e6383 has state: started
  Machine 0e286039f61d86 has state: started
  Machine 39080ddec59e87 has state: started
  Finished deploying
9 Likes

fly m clone would continue to be part of the ‘apps platform’?

But flyctl deploy won’t touch Machines API machines… so, this guidance is potentially misleading?

Also, I firmly believe treating machines created with fly m run with distinction is going to be a major source of confusion. I’d reckon, either don’t allow fly m run for ‘apps platform’ (for ex, show 'did you mean to flyctl m clone instead), or treat it just the same as other machines of the app.

I have used flyctl deploy for machines since August. And it worked with fly m run machines just fine. Not sure what changed in the intervening time period that prompted this decision. It’d be nice if it continued to: That’d be very Fly-like-- straight-forward and un-complect.

1 Like

We’re using the machine metadata to tag machines as “flyctl managed”. Most people who create machines with the API or fly machine run don’t want a single command to wipe out all their machine configs.

fly machine clone copies the metadata, so anything created via clone will continue to work.

It would make sense to “adopt” fly machine run machines into apps somehow. Possibly a flag to the command, or a config option in fly.toml. We’ll think about that.

2 Likes

How can I invoke a machine directly from another Fly app running within the same organization? I have an app with an endpoint and port (my-app.fly.dev:443) that I added some machines to; I tried invoking the machine from within the fly VPC using url $MACHINE_ID.vm.my-app.internal:443, but I get a “no such host” exception.

Not sure you can send request over any (Fly VPC) 6pn IP and wake up the Machines. The only way I know of is to assign a Flycast IP, which is like an Anycast IP but for 6pn. The only downside is that you can’t target any one particular Machine with a Flycast IP.

Flycast IPs are free to use (for now), but need to be assigned explicitly.

Bug: release_command is not being executed.

1 Like

If your machine is started and listening on IPv6, you should be able to get to it with the internal port. My apps run on 8080, so I’d use: $MACHINE_ID.vm.my-app.internal:8080.

1 Like

release_command has not been implemented for machine deploys yet.

Does it always work? It wasn’t some months ago: Machines app without public ports - #2 by losfair

Thanks for trying out the new fly deploy! Will you please share the error (and output) you’re seeing?

Also, will you run fly version and share that output?

1 Like

I’ve been using the prerelease version since it was announced, and I have had no significant problems so far.

I found two issues:

  1. release_command is not being executed. No error; it’s just not executed.
  2. It looks like it doesn’t support zero downtime deployments.

Deployments:

flyctl v0.0.451-pre-2 linux/amd64 Commit: 93197062 BuildDate: 2023-01-14T10:10:55Z
1 Like

Can’t deploy after updating to pre-3. It’s working fine after going back to pre-2.

Infinite loop:

2023-01-19T01:34:03.691 proxy[<id>] iad [info] Starting machine
2023-01-19T01:34:03.692 proxy[<id>] iad [info] Starting machine
2023-01-19T01:34:03.692 proxy[<id>] iad [error] machines API returned an error: "machine ID <id> lease currently held by <email>, expires at 2023-01-19T02:03:37Z"
2023-01-19T01:34:03.693 proxy[<id>] iad [error] machines API returned an error: "machine ID <id> lease currently held by <email>, expires at 2023-01-19T02:03:37Z"

Do you see output like this when you run fly deploy?

...
==> Verifying app config
--> Verified app config
Running APP_NAME release_command: /release_cmd.sh ...
  release_command 39080ddec24187 completed successfully
Deploying APP_NAME app with rolling strategy
...

If so, the release_command is running and then stopping. You can see the stdout/stderr from the reelease_command machine using fly logs or in the web dashboard.

If not, then probably a bug. Please share your fly.toml if that’s the case so I can repro the bug.

fly deploy for Apps v2 defaults to a rolling deployment strategy. It updates each machine using the Machines API and waits for the machine to return to a started state. It will also wait for the health checks to pass for each machine. The --detach flag will skip waiting on the health checks.

fly deploy for Apps v2 also supports the immediate strategy. It does the same thing as rolling strategy, except it doesn’t wait for the machines to start or become healthy.

canary and bluegreen deployments are not supported, yet. I’ll update flyctl to throw an error if one of those strategies is attempted to avoid any confusion.

Thanks for continuing to test this! I appreciate it.

Will you share the app name and fly.toml? I’ll use those to create a separate repro of this issue. Let me know if you prefer to share those in this thread, and we can figure something else out.

1 Like