Ghost VMs keep running in the background?

[Yaeger]

I am running into a weird issue:

These VMs don’t exist anymore. They don’t show up in fly status. The code that’s showing in the stacktraces isn’t even in our codebase anymore. In a recent deploy we removed predis/predis and also laravel/horizon, because we don’t want to use Redis anymore. I also removed the Redis app that it’s trying to connect to. Somehow there are still VMs in the background that are trying to connect to Redis?? Would love some help on this, we’re burning through our Sentry limits!

If a dev could at least kill them, that would be great.

I can still see the logs of the “correct” VM by selecting it from the dropdown in the dashboard, but other than that the logs are flooded by “ghost” VMs.

The IDs of the ghost VMs are:

• 8523c5e8
• 3608ca3c
• fe676750

Maybe there’s more, it’s hard to see.

[eli]

Hi! I can confirm those VMs don’t really exist. Those loglines were likely emitted by release jobs which never exited.

You’ve probably already seen this, but there’s some additional context in a recent similar thread that might be interesting.

We’re working on rolling out a permanent fix for this problem soon, and we can take care of the ones you listed by hand

[Yaeger]

Hi Eli! This must be it. I added a release task today and noticed they were hanging, after that I made the following changes so that it worked again:

Thank you!

[fideloper-fly]

Hi!!

Can you share your Dockerfile? Specially the entrypoint and command.

Bonus points if you can share the entrypoint script too (docker/entrypoint.sh IIRC)

[Yaeger]

infra/docker/run.sh:

#!/usr/bin/env sh

if [ $# -gt 0 ];then
     # If we passed a command, run it as root
     exec "$@"
else
    /usr/bin/php /var/www/html/artisan config:cache --no-ansi -q
    /usr/bin/php /var/www/html/artisan route:cache --no-ansi -q
    /usr/bin/php /var/www/html/artisan view:cache --no-ansi -q

    exec supervisord -c /etc/supervisord.conf
fi

infra/Dockerfile:

FROM alpine:3.16 as base

LABEL fly_launch_runtime="laravel"

RUN apk update \
    && apk add curl zip unzip tzdata supervisor nginx htop vim ca-certificates rsync \
           php81           php81-cli        php81-intl \
           php81-soap      php81-openssl    php81-gmp \
           php81-pdo_odbc  php81-json       php81-dom \
           php81-pdo       php81-zip        php81-pdo_mysql \
           php81-sqlite3   php81-pdo_pgsql  php81-bcmath \
           php81-gd        php81-odbc       php81-pdo_sqlite \
           php81-gettext   php81-xmlreader  php81-bz2 \
           php81-iconv     php81-pdo_dblib  php81-curl \
           php81-ctype     php81-phar       php81-xml \
           php81-common    php81-mbstring   php81-tokenizer \
           php81-xmlwriter php81-fileinfo   php81-opcache \
           php81-simplexml php81-pecl-redis php81-sockets \
           php81-pcntl     php81-posix      php81-fpm \
    && ln -sf /usr/bin/php81 /usr/bin/php \
    && cp /etc/nginx/nginx.conf /etc/nginx/nginx.old.conf \
    && rm -rf /etc/nginx/http.d/default.conf \
    && curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer \
    && adduser -D -u 1000 -g 'app' app \
    && addgroup nginx app \
    && mkdir -p /var/run/php \
    && chown -R app:app /var/run/php \
    && mkdir -p /var/www/html

WORKDIR /var/www/html
# copy application code, skipping files based on .dockerignore
COPY .. /var/www/html

RUN composer install --optimize-autoloader --no-dev \
    && mkdir -p storage/logs \
    && chown -R app:app /var/www/html \
    && echo "* * * * * /usr/bin/php /var/www/html/artisan schedule:run" > /etc/crontabs/app \
    && mv infra/docker/supervisor.conf /etc/supervisord.conf \
    && mv infra/docker/nginx.conf /etc/nginx/nginx.conf \
    && mv infra/docker/server.conf /etc/nginx/server.conf \
    && mv infra/docker/php.ini /etc/php81/conf.d/php.ini \
    && sed -i 's/protected \$proxies/protected \$proxies = "*"/g' app/Http/Middleware/TrustProxies.php

RUN rm -rf /etc/php81/php-fpm.conf; \
    rm -rf /etc/php81/php-fpm.d/www.conf; \
    mv infra/docker/php-fpm.conf /etc/php81/php-fpm.conf; \
    mv infra/docker/app.conf /etc/php81/php-fpm.d/app.conf;

RUN composer dump-autoload \
    && php artisan optimize:clear \
    && chmod -R ug+w /var/www/html/storage \
    && chmod -R 755 /var/www/html

# Multi-stage build: Build static assets
FROM node:14 as build_assets

RUN mkdir /app

RUN mkdir -p  /app
WORKDIR /app
COPY .. .
RUN npm ci && npm run build

# Create final container, adding in static assets
FROM base

COPY --from=build_assets /app/public/build /var/www/html/public/build
COPY --from=build_assets /app/public/build/assets/widget-loader.*.js /var/www/html/public/js/widget-loader.js

EXPOSE 8080

ENTRYPOINT ["/var/www/html/infra/docker/run.sh"]

It’s basically the old default Laravel configuration that Fly gives when running fly launch, with adjustments for our specific use case.

[Yaeger]

Btw, this is the code after fixing the issue that was causing my release command to hang

[fideloper-fly]

interesting, gotcha!

The newer stuff isn’t using supervisord anymore, I didn’t catch that you were using it

[Yaeger]

Yea off-topic but I still have issues with the new Laravel config:

• My app responds significantly slower in the new config
• It makes it hard to run scheduler and worker without using multiple processes. I don’t mind paying for multiple VMs but I’ve been running into a loooooot of issues when using multiple processes, so I’m not going to use them until they are more stable