Where are build releases run?

When I run fly launch or fly deploy, and I have some build steps on my Dockerfile. Where are these build steps run? In my understanding, it depends on the --remote-only and --local-only flags.

  1. --remote-only runs on a fly thing with access to .flycast and .internal domain names.
  2. --local-only runs on wherever the command is being called. In example, github actions and my PC, and these build don’t have access to .internal and .flycast domain names.

Is this correct?
I need to know this but I can’t find it clearly on your docs anywhere.

My Dockerfile in case it’s relevant to the answer.

# Set Bun and Node version
ARG BUN_VERSION=1.1.13
ARG NODE_VERSION=20.12.2
FROM imbios/bun-node:${BUN_VERSION}-${NODE_VERSION}-slim

# Set production environment
ENV NODE_ENV="production"
ARG DATABASE_URL
ENV DATABASE_URL=${DATABASE_URL} 

# Bun app lives here
WORKDIR /usr/src/app

# Copy app files to app directory
COPY / .

# CD into the root directory
RUN cd ../../

# Install node modules
RUN bun install

# Build next js
RUN bunx turbo run --filter @diet-it/web build

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD [ "bunx", "turbo", "run", "--filter", "@diet-it/web", "start" ]

The bunx turbo run --filter @diet-it/web build invocation triggers prisma migrate deploy. This command needs access to my postgres database.

@rubys is the best writer here, by a wide margin, and I was pleased to see him posting again.

(There used to be several drifting down each day. Like the orange leaves in the fall up here.)

Perhaps you glossed over his earlier response’s most important part, though, two days ago?

I.e., migrations are factored out of the main build process and execute apart, in their own special context.

Cantankerous individuals like myself go further and construct our own builders—in part to have complete control over the exact nitty details of what network access is and [usually more importantly] is not allowed. (I’m using Guix, but he gave the easier RCHAB example.)

That’s a project in its own right, though, so stick with the Fly.io platform feature that was designed specifically for performing migrations, unless there really is a compelling reason to do otherwise…

Added builders, duplicated, postgres

Maybe it’s my poor knowledge of English or my lack of interpretation skills, but I couldn’t understand the migrations part out of his comment. Not complaining, just thought he had focused on other aspects of my questions and though it would be a good idea to make an objective question.
Thank you for your answer.

First, I’ll start by answering the question you pose in the title: deploy release commands are run on ephemeral machines in your app’s private network, and with access to all of your secrets. And by ephemeral machines, I mean a virtual machine that is created for the sole purpose of running that one command and then destroyed immediately thereafter.

I’ve written and deployed Next.js apps and apps that use prisma, but never a Next.js app using prisma; what’s different (and new to me) here is turbo, which looks cool.

It looks like you define workflows in turbo, in a file called turbo.json. In your application, you have defined two workflows: build and start. Someplace in build you are triggering a prisma migration, I gather that is during a db:push task.

The smallest change you could make that likely get you up and running is to edit those workflows so that the db:push task is run as a part of the start workflow rather than the build workflow. If you are deploying to exactly one machine, that is fine. If you are deploying to multiple machines it is worth splitting that workflow out to a separate step, and running that as a deploy release command.

1 Like

This may be helpful, a turborepo example using nextjs and prisma:

Awesome. I’d like to skip directly to the deploy_release strategy so I don’t have to revisit this topic in a close future.
So I tried following the referenced docs:

  1. I removed the prisma scripts from my bunx turbo --filter @diet-it/web build pipeline and moved it to its own command. bunx turbo run --filter @diet-it/db db:build which now calls the db:build script:
"db:build": "bun run db:deploy && bun run db:generate",
"db:deploy": "prisma migrate deploy",
"db:generate": "prisma generate",
  1. Added a release_command to my web fly.toml file:
[deploy]
  release_command = "bunx turbo run --filter @diet-it/db db:build"

But now github returns a empty error that, if I look into the logs of the referenced machine, I can see the release_command failed because it didn’t have access to the DATABASE_URL environment variable again.

# https://fly-metrics.net/d/fly-logs/fly-logs?instance=6e82570ea76d08&orgId=799116&var-	
@diet-it/db:db:build: error: script "db:build" exited with code 1
@diet-it/db:db:build: error: script "db:deploy" exited with code 1
@diet-it/db:db:build: error: "prisma" exited with code 1
@diet-it/db:db:build: Prisma CLI Version : 5.21.1
@diet-it/db:db:build:
@diet-it/db:db:build: [Context: getConfig]
@diet-it/db:db:build: Validation Error Count: 1
@diet-it/db:db:build:
@diet-it/db:db:build:    |
@diet-it/db:db:build:  8 |   url      = env("DATABASE_URL")
@diet-it/db:db:build:  7 |   provider = "postgresql"
@diet-it/db:db:build:   -->  prisma/schema/schema.prisma:8
@diet-it/db:db:build: error: Environment variable not found: DATABASE_URL.
@diet-it/db:db:build: Error code: P1012
@diet-it/db:db:build: Error: Prisma schema validation - (get-config wasm)
@diet-it/db:db:build:
@diet-it/db:db:build: Datasource "db": PostgreSQL database
@diet-it/db:db:build: Prisma schema loaded from prisma/schema
@diet-it/db:db:build: $ bun prisma migrate deploy
@diet-it/db:db:build: $ bun run db:deploy && bun run db:generate
@diet-it/db:db:build: cache bypass, force executing 07a484ce0b1dcd7a
• Remote caching disabled
• Running db:build in 1 packages
• Packages in scope: @diet-it/db
2024/10/28 14:16:54 INFO SSH listening listen_address=[fdaa:9:d488:a7b:342:a100:d647:2]:22 dns_server=[fdaa::3]:53
Machine created and started in 4.667s
 INFO [fly api proxy] listening at /.fly/api
 INFO Preparing to run: `/usr/local/bin/docker-entrypoint.sh bunx turbo run --filter @diet-it/db db:build` as root
2024-10-28T14:16:53.422744546 [01JB9QS4GKG9D0FXTWMGRJ2153:main] Running Firecracker v1.7.0
Configuring firecracker
Successfully prepared image registry.fly.io/diet-it-backend:deployment-01JB9QRN2KZYA6DYX607STBZG9 (2.646417274s)
Pulling container image registry.fly.io/diet-it-backend:deployment-01JB9QRN2KZYA6DYX607STBZG9

It seems like even on a release_command machine environment, I’m still having trouble to access my DATABASE_URL.
Sorry if I’m overextending this thread and asking too much of you all. Butt I really don’t know how to proceed.

The example github is nice but I think he does run his prisma commands on the build step and not on a release command as seen on his Dockerfile:

What I’m seeing is that build is running deploy, and deploy is running migrate. Without seeing the full logs, my assumption is that the build is failing, which would make sense as it is running on your build machine which does not have access to secrets.

Perhaps you meant something like?

"db:build": "bun run db:generate",
"db:deploy": "prisma migrate deploy",
"db:generate": "prisma generate",

On this command, you probably want db:deploy rather than db:build.

It’s a bit confusing because I named these command poorly.
Basically, the command bunx trubo run --filter @diet-it/db db:build calls both db:generate and db:deploy. Both these commands needs access to the DATABASE_URL. My fly.toml

[deploy]
  release_command = "bunx turbo run --filter @diet-it/db db:build"

calls them both on the release_command.
So, I did just tried my code with your suggestion just to make sure, and the result is the same. I get an error for missing DATABASE_URL on the ephemeral machine fly spins up to run the release_command step.
The complete logs are here just in case you want to look them yourself:
Github action logs
Logs from the machine fly spins up just for the release_command step

The logs after making yours changes. (Running only prisma migrate deploy on the release_command and prisma generate together with the next build command):

I don’t understand it completely. But it errors when it tries to build my database status page, so I removed my status page and then the same issue happened.
Github actions logs
Logs from Prometheus filtering by the ephemeral machine referenced on the github action (instance=e7843754b219d8)

I think the secret is available on the --app.

fly console --app diet-it-web
env
SHELL=/bin/bash
FLY_PRIVATE_IP=254235:9:132:a7b:2345werf:253:3e32:2
FLY_MACHINE_ID=123453253463
DATABASE_URL=postgres://AAAAAAA:BBBBBBBBBBBBB@diet-it-main-db.flycast:5432/diet_it_web?sslmode=disable
WEB_URL=https://diet-it-web.fly.dev
FLY_ALLOC_ID=1111111111111
FLY_VM_MEMORY_MB=222
YARN_VERSION=1.22.22
PWD=/usr/src/app
BACKEND_INTERNAL_URL=http://diet-it-backend.flycast:3049/

I just censored the values.
Unfortunately I can’t force start the ephemeral machine to check its environment variables. After I try, it hangs:

fly machine start 908017d0b6d428 --app diet-it-web
Waiting on lease for machine 908017d0b6d428...

I’m lost. I don’t know how to proceed. There’s either a bug on fly.io where ephemeral machines generated by the release_command are not setting the environment correctly (like on this issue), or your documentation is wrong here.
Or I’m doing something completely wrong and I can’t figure it out.

From what I see in your logs, you never got to the release step, it is the build step that failed. fly console is indeed the way to explore what an ephemeral machine looks like - it is created when you run that command, destroyed when you exit, and has access to your network and secrets.

Here is the error:

2024-10-28T16:10:55.5514290Z #10 55.47 @diet-it/web:build:    Generating static pages (0/12) ...
2024-10-28T16:10:56.5387062Z #10 56.46 @diet-it/web:build:    Generating static pages (3/12) 
2024-10-28T16:10:56.5388468Z #10 56.46 @diet-it/web:build:    Generating static pages (6/12) 
2024-10-28T16:10:56.9862661Z #10 56.91 @diet-it/web:build: Error occurred prerendering page "/status". Read more: https://nextjs.org/docs/messages/prerender-error
2024-10-28T16:10:56.9864628Z #10 56.91 @diet-it/web:build: PrismaClientInitializationError: 
2024-10-28T16:10:56.9866228Z #10 56.91 @diet-it/web:build: Invalid `prisma.$queryRaw()` invocation:
2024-10-28T16:10:56.9867363Z #10 56.91 @diet-it/web:build: 
2024-10-28T16:10:56.9868210Z #10 56.91 @diet-it/web:build: 
2024-10-28T16:10:56.9869449Z #10 56.91 @diet-it/web:build: error: Environment variable not found: DATABASE_URL.
2024-10-28T16:10:56.9870824Z #10 56.91 @diet-it/web:build:   -->  schema.prisma:601
2024-10-28T16:10:56.9871878Z #10 56.91 @diet-it/web:build:    | 
2024-10-28T16:10:56.9872915Z #10 56.91 @diet-it/web:build: 600 |   provider = "postgresql"
2024-10-28T16:10:56.9874686Z #10 56.91 @diet-it/web:build: 601 |   url      = env("DATABASE_URL")
2024-10-28T16:10:56.9875703Z #10 56.91 @diet-it/web:build:    | 

Rendering static pages will need access to the database, so this step too needs to be moved to the release machine. Unfortunately this is a bit more painful as you will also need to capture the prerendered output and load it on each server. The easiest way to do that is with Tigris. If this is of interest, I can talk you through this.

(Technically, this too could be run on the deployed machine, but that kinda defeats the purpose of prerendering).

It may be possible to serve static files directly from Tigris which would make things even easier. I can check on the status of this effort.

But look at the second message logs, isn’t this run after the build step is finished?
I removed all the static pages from this project, and it still fails. Also, on these logs, prisma migrate deploy does not get called until the release_command is triggered.

Release commands are only run after a successful build. Do you have a log where a build succeeded?

The second message logs contains the following:

1970-01-20 21:35:30.555 @diet-it/db:db:build: error: script "db:build" exited with code 1

Even tho I named my command db:build not thinking on the implications. It was called by the release_command specification.
I will re run everything to be sure. But these logs were supposed to happen on the release_command.

I think the build command completed successfully.
If you look at this part:


On these logs:

The second logs are from the ephemeral machine created on the exact moment this like was called:

Running diet-it-backend release_command: bunx turbo run --filter @diet-it/db db:build

You can confirm this if you look at the timestamps. Even tho they are on different timezones, they match:

Github action release command timestamp -> 2024-10-28T14:16:48.4844668Z
Fly ephemeral machine start             -> 2024-10-28 11:16:49.525

I assume the github action is on GMT. My servers are on gru GMT-3:

Github action release command timestamp -> 2024-10-28 11:16:48.484
Fly ephemeral machine start             -> 2024-10-28 11:16:49.525

Those logs do indeed show a build succeeding and a release command failing:

2024-10-28T14:16:42.1619150Z ==> Building image with Depot
2024-10-28T14:16:45.0135790Z #10 exporting to image
2024-10-28T14:16:46.1959916Z --> Build Summary:  (​)
2024-10-28T14:16:46.6085121Z e[38;5;252m--> Building image donee[0m
2024-10-28T14:16:48.4844668Z Running diet-it-backend release_command: bunx turbo run --filter @diet-it/db db:build
2024-10-28T14:16:59.8689885Z Error release_command failed running on machine 4d890457c65768 with exit code 1.

Do you have logs from the release command? What you posted previous has different timestamps.

By the way, while having fly deploy run as a github action is clearly a desirable place to end up, it might be easier/faster to debug using fly deploy locally. Once you get it working, you can move on to a github action.