Playwright Failing

My Next.js web app is working locally; however, when I deploy to Fly it fails with the following server error:

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] browserType.launch: Executable doesn't exist at /root/.cache/ms-playwright/chromium-1076/chrome-linux/chrome

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ╔═════════════════════════════════════════════════════════════════════════╗

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ Looks like Playwright Test or Playwright was just installed or updated. ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ Please run the following command to download new browsers: ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ npx playwright install ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ║ <3 Playwright Team ║

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] ╚═════════════════════════════════════════════════════════════════════════╝

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] at PlaywrightWebBaseLoader._scrape (/app/.next/server/chunks/959.js:55452:40)

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] at async PlaywrightWebBaseLoader.load (/app/.next/server/chunks/959.js:55480:22)

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] at async clean (/app/.next/server/chunks/425.js:44:25)

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] at async PreviewPage (/app/.next/server/app/preview/[objectId]/page.js:699:18) {

2023-09-04T19:55:09.417 app[4d891447f00698] dfw [info] name: 'Error'

Here is my Dockerfile. As you can see, I am attempting to manually install Chromium as well as make it executable.

# syntax = docker/dockerfile:1

# Adjust NODE_VERSION as desired
ARG NODE_VERSION=18.17.1
FROM node:${NODE_VERSION}-slim as base

LABEL fly_launch_runtime="Next.js"

# Next.js app lives here
WORKDIR /app

# Set production environment
ENV NODE_ENV="production"

# Throw-away build stage to reduce size of final image
FROM base as build

# Install packages needed to build node modules
RUN apt-get update -qq && \
    apt-get install -y build-essential pkg-config python-is-python3

# Install node modules
COPY --link package-lock.json package.json ./
RUN npm ci --include=dev

# Handle Playwright error
ENV PLAYWRIGHT_BROWSERS_PATH="~/.cache/ms-playwright"

RUN npx playwright install-deps chromium

RUN chmod +x ~/.cache/ms-playwright/chromium-1076/chrome-linux/chrome

# Copy application code
COPY --link . .

# Build application
RUN npm run build

# Remove development dependencies
RUN npm prune --omit=dev

# Final stage for app image
FROM base

# Copy built application
COPY --from=build /app /app

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD [ "npm", "run", "start" ]

But when I include a RUN ls -l ~/.cache/ms-playwright command, nothing is returned, which makes me think there is nothing in the directory.

I have come across two relevant posts (1, 2); however, I still have no success.

Some of the solutions recommend using other base images such as FROM zenika/alpine-chrome:with-playwright or FROM zenika/alpine-chrome:with-node.

Although I don’t understand how to preserve my current Node/Next environment AND get Playwright in the same Dockerfile.

Thanks so much!

Since the chmod succeeded earlier, I suspect that there is something in that directory. The RUN command is run on the build machine and when sent to the terminal the output scrolls by in a small window which then collapses and is overwritten immediately unless the command fails. I suspect that’s what is happening here.

I suspect that the problem here is that the PLAYWRIGHT_BROWSERS_PATH environment variable is only set during the build stage, and not in the final stage. I don’t know enough about playwright, but if that is needed during install-deps then move that statement to before the 'FROM base as build line, that will make it a part of the base image and available to both the build and final stages.

It may be worth looking at the dockerfile I produce if puppeteer is used: https://github.com/fly-apps/dockerfile-node/blob/main/test/base/puppeteer/Dockerfile . Ideally if we can get playwright working, I’d like to see dockerfile-node updated so that others with similar needs get a Dockerfile that ideally just works, or failing that provides a better starting point.

Unfortunately, none of this worked.

I tried another couple hours worth of permutations.

1: Your apt-get method of installing Chromium to “/usr/bin/chromium” from Puppeteer Dockerfile… didn’t work for Playwright. I think because I cannot get Playwright to search for Chromium in usr/bin/chromium

I realized PLAYWRIGHT_BROWSERS_PATH only changes the first part of where Playwright searches for Chromium executable. According to server error, Playwright always appends /chromium-1076/chrome-linux/chrome

PLAYWRIGHT_BROWSERS_PATH="usr/bin/chromium"

...

browserType.launch: Executable doesn't exist at /usr/bin/chromium/chromium-1076/chrome-linux/chrome

So, since I couldn’t control Playwright Search Path, I tried linking the path of Chromium after apt-get install (“usr/bin/chromium”) to the default search path (/root/.cache/ms-playwright/chromium-1076/chrome-linux/chrome)… using the code below.

RUN ln -s /usr/bin/chromium/chromium-1076/chrome-linux/chrome /usr/bin/chromium

This ran, but was unsuccessful. I also tried installing Chromium with apt-get, then moving the file to Playwright’s default Search Path

mv /usr/bin/chromium /root/.cache/ms-playwright/chromium-1076/chrome-linux/chrome

This failed saying that “No file or directory exists”.

2: I found something in Playwright docs called Hermetic install. This is where Playwright installs browser binaries in node_modules/playwright-core/.local-browsers with the following code:

RUN PLAYWRIGHT_BROWSERS_PATH=0 npx playwright install --with-deps chromium

However, this changed nothing as server error points out that Playwright is still searching for Chrome in /root/.cache/ms-playwright/chromium-1076/chrome-linux/chrome, as described above


The RUN command is run on the build machine and when sent to the terminal the output scrolls by in a small window which then collapses and is overwritten immediately unless the command fails.

Is there any way to get verbose log output? It would be helpful to intersperse debugging statements to confirm things like where files are downloaded. For example:

RUN ls -l /root/.cache/ms-playwright/chromium-1076/chrome-linux/chrome

RUN which chromium

I’ll play with this some more myself tomorrow; but to answer your question, what I generally do if the build succeeds but the app doesn’t work as I suspect:

fly console

This will create a “ephemeral” machine with your image, log you in as root, and you can run whatever command you like.

I’m making progress with:

# Install browsers
RUN npx playwright install

# Remove development dependencies
RUN npm prune --omit=dev


# Final stage for app image
FROM base

# Copy built application
COPY --from=build /app /app
COPY --from=build /root/.cache /root/.cache

Excellent, I made further progress!

Using your Dockerfile, I received an error about missing dependencies

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] browserType.launch:

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ╔══════════════════════════════════════════════════════╗

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ Host system is missing dependencies to run browsers. ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ Missing libraries: ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libgobject-2.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libglib-2.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libnss3.so ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libnssutil3.so ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libsmime3.so ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libnspr4.so ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libatk-1.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libatk-bridge-2.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libcups.so.2 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libgio-2.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libdrm.so.2 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libdbus-1.so.3 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libexpat.so.1 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libxcb.so.1 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libxkbcommon.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libatspi.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libX11.so.6 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libXcomposite.so.1 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libXdamage.so.1 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libXext.so.6 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libXfixes.so.3 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libXrandr.so.2 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libgbm.so.1 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libpango-1.0.so.0 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libcairo.so.2 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ║ libasound.so.2 ║

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] ╚══════════════════════════════════════════════════════╝

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] at PlaywrightWebBaseLoader._scrape (/app/.next/server/chunks/959.js:55452:40)

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] at async PlaywrightWebBaseLoader.load (/app/.next/server/chunks/959.js:55480:22)

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] at async clean (/app/.next/server/chunks/425.js:44:25)

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] at async PreviewPage (/app/.next/server/app/preview/[objectId]/page.js:699:18) {

2023-09-05T17:10:27.617 app[4d891447f00698] dfw [info] name: 'Error'

So, I changed Dockerfile to the following

#Install browsers
RUN npx playwright install --with-deps chromium

# Remove development dependencies
RUN npm prune --omit=dev

# Final stage for app image
FROM base

# Copy built application
COPY --from=build /app /app
COPY --from=build /root/.cache /root/.cache
COPY --from=build /usr/lib /usr/lib

Now, the npx command installs the dependencies, and COPY --from=build /usr/lib /usr/lib is necessary because this, presumably, is where the dependencies are stored.

Finally, looks like Chrome is able to Navigate to the page I want to scrape; however, I get the following error for page crashing.

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] page.goto: Page crashed

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] =========================== logs ===========================

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] navigating to "https://a16z.com/announcement/asimov/", waiting until "domcontentloaded"

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] ============================================================

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] at PlaywrightWebBaseLoader._scrape (/app/.next/server/chunks/959.js:55457:37)

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] at async PlaywrightWebBaseLoader.load (/app/.next/server/chunks/959.js:55480:22)

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] at async clean (/app/.next/server/chunks/425.js:44:25)

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] at async PreviewPage (/app/.next/server/app/preview/[objectId]/page.js:699:18) {

2023-09-05T17:11:48.275 app[4d891447f00698] dfw [info] name: 'Error'

I was reminded from an earlier post of yours, that Chrome is memory hog and requires 1GB RAM. So I upgraded from shared-1x-cpu@512MB to shared-1x-cpu@1024MB using the following command in terminal: fly scale memory 1024

However, I receive the same error. The page still crashes with machine memory 1024MiB. Any thoughts?

Update: I don’t think “Page Crash” is related to VM memory. I’m all the way up to 2GB RAM, and error still occurring.

I’ll admit upfront that this is just a guess, but it should be quick to test. I know that Chrome doesn’t like being run as root, and requires special options be passed when you do so. A quick scan of google turns up nothing, but that could be because nobody else has tried running playwright as root. In any case, the lines you would need to add look like the following:

Put them after the COPY --from=build statements. You don’t need the mkdir. The chown is only needed if there are directories that your application needs to write to.

At first I got errors during build, and changed Dockerfile to this

# Copy built application
COPY --from=build /usr/lib /usr/lib
COPY --from=build /app /app

# Run and own only the runtime files as a non-root user for security
RUN useradd rails --create-home --shell /bin/bash && \
    mkdir /data && \
    chown -R rails:rails /data

USER rails:rails

COPY --from=build /root/.cache /home/rails/.cache

It was important to move the COPY --from=build /root/.cache /home/rails/.cache underneath user change, to first create /home/rails directory, and then to copy the Browsers to that path.

Still, I received the server error. The more I read, the more that “Page Crash” appears to be a memory issue. I don’t see the docker run command that Fly uses on backend, and wonder if it’s allocating proper memory, despite me upgrading the machine.

Here’s an excerpt from why Page Crashes with Pupeteer on Docker:

By default, Docker runs a container with a /dev/shm shared memory space 64MB. This is typically too small for Chrome and will cause Chrome to crash when rendering large pages. To fix, run the container with docker run --shm-size=1gb to increase the size of /dev/shm .

You won’t find one, because it doesn’t exist. See: Docker without Docker · The Fly Blog

The TL;DR version: you get a true VM. The image that docker builds is just used to seed the filesystem.

I went ahead and switched to using Puppeteer, changed my Dockerfile to match Puppeteer Dockerfile, and now my app is fully functioning.

I’ll leave this thread open for continued discussion. Thanks @rubys for the back and forth!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.