Chromedriver not working on python bot

Python
1

I’ve migrated my bot from heroku to fly.io because of heroku removing free plan, but after i tested chromedriver to scrap one site it fails to do so:

discord.ext.commands.errors.CommandInvokeError: Command raised an exception: WebDriverException: Message: unknown error: session deleted because of page crash

My chromedriver options (I’m using undetected_chromedriver - optimized version of selenium webdriver):

opts = uc.ChromeOptions()
opts.add_argument("--no-sandbox")
opts.add_argument("--headless")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--enable-javascript")

driver = uc.Chrome(use_subprocess=True, options=opts)

My dockerfile:

# using ubuntu LTS version
FROM ubuntu:20.04 AS builder-image

# avoid stuck build due to user prompt
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install --no-install-recommends -y python3.9 python3.9-dev python3.9-venv python3-pip python3-wheel build-essential && \
	apt-get clean && rm -rf /var/lib/apt/lists/*

# create and activate virtual environment
# using final folder name to avoid path issues with packages
RUN python3.9 -m venv /home/kexobot/venv
ENV PATH="/home/kexobot/venv/bin:$PATH"

# install requirements
COPY requirements.txt .
RUN pip3 install --no-cache-dir wheel
RUN pip3 install --no-cache-dir -r requirements.txt

FROM ubuntu:20.04 AS runner-image
RUN apt-get update && apt-get install --no-install-recommends -y python3.9 python3-venv && \
    # Install ffmpeg and opus
    apt-get install -y ffmpeg && apt-get install libopus0 && \
    # We need wget to set up the PPA and xvfb and gnupg to have a virtual screen and unzip to install the Chromedriver
    apt-get install -y wget xvfb unzip && apt-get install gnupg -y  && \
    # Set up the Chrome PPA
    wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
    echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
    # Set up the Chrome PPA
    apt-get update && apt-get install -y google-chrome-stable && \
	apt-get clean && rm -rf /var/lib/apt/lists/*


# Set up Chromedriver Environment variables
ENV CHROMEDRIVER_VERSION 2.19
ENV CHROMEDRIVER_DIR /chromedriver
RUN mkdir $CHROMEDRIVER_DIR

# Download and install Chromedriver
RUN wget -q --continue -P $CHROMEDRIVER_DIR "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip"
RUN unzip $CHROMEDRIVER_DIR/chromedriver* -d $CHROMEDRIVER_DIR

# Put Chromedriver into the PATH
ENV PATH $CHROMEDRIVER_DIR:$PATH

RUN useradd --create-home kexobot
COPY --from=builder-image /home/kexobot/venv /home/kexobot/venv

USER kexobot
RUN mkdir /home/kexobot/code
WORKDIR /home/kexobot/code
COPY . .

EXPOSE 5000

# make sure all messages always reach console
ENV PYTHONUNBUFFERED=1

# activate virtual environment
ENV VIRTUAL_ENV=/home/kexobot/venv
ENV PATH="/home/kexobot/venv/bin:$PATH"


# /dev/shm is mapped to shared memory and should be used for gunicorn heartbeat
# this will improve performance and avoid random freezes
CMD ["gunicorn","-b", "0.0.0.0:5000", "-w", "4", "-k", "gevent", "--worker-tmp-dir", "/dev/shm", "app:app", "python KexoBOT.py", "python KexoBOTNews.py"]
2

Hey, did you get this to work? I’m having a similar issue.

3

I discovered that 256 MB ram is not enough for chromedriver, that’s why i moved my chromedriver bot to mogenius

2 Likes
4

@kexo pls how were you able to host your bot on mogenius as I’m facing some difficulty setting mine up.

5

I’ve had some issues with docker i posted above, so i simplified it and it works on mogenius, you can remove chromedriver installation if you don’t need it. If you can’t setup your bot you can ask on their discord server

FROM python:3.9-slim-buster

RUN mkdir -p /usr/src/app/
WORKDIR /usr/src/app/

COPY . /usr/src/app/

COPY requirements.txt .
RUN pip3 install --no-cache-dir wheel
RUN pip3 install --no-cache-dir -r requirements.txt

RUN apt-get update && \
    apt-get install -y gnupg wget curl unzip --no-install-recommends && \
    wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
    echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
    apt-get update -y && \
    apt-get install -y google-chrome-stable && \
    CHROME_VERSION=$(google-chrome --product-version | grep -o "[^\.]*\.[^\.]*\.[^\.]*") && \
    CHROMEDRIVER_VERSION=$(curl -s "https://chromedriver.storage.googleapis.com/LATEST_RELEASE_$CHROME_VERSION") && \
    wget -q --continue -P /chromedriver "http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip" && \
    unzip /chromedriver/chromedriver* -d /usr/src/app/chromedriver/

# make sure all messages always reach console
ENV PYTHONUNBUFFERED=1

CMD python your_bot.py
2 Likes