Failed to deploy a large 3rd party docker image

foodiebuddy · November 28, 2020, 3:50am

Hey guys, I am new to fly.io and I ran into some problems when deploying a large docker image.

I am trying to deploy an Angular app that I wrote. It is almost 700MB (ok I might have written a terrible Dockerfile). So here is the log after I ran command “flyctl deploy”:

$ flyctl deploy
Deploying frosty-tree-6510
Validating App Configuration
Validating App Configuration done
Services
TCP 80/443 ⇢ 4200

Deploying image: XXXX/YYYY:0.0.1

Image ID: sha256:5725611306d167ca880e7a48d52d931b6a1d93c1ac25273330c5248e73b33abf
Image size: 693 MB
Image resolving done
Creating deployment tag
→ registry,fly,io/frosty-tree-6510:deployment-1606533657
The push refers to repository [registry,fly,io/frosty-tree-6510]

Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
deployment-1606533657: digest: sha256:f8a172e18c8ebcd9ab747465f969381ead82a8d4ae3b6f6d6694388f303082be size: 2618
Optimizing Image
Done Optimizing Image
Creating Release
Release v0 created
Deploying to : frosty-tree-6510.fly.dev

Monitoring Deployment
You can detach the terminal anytime without stopping the deployment

v0 is being deployed
[1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total]
[1 desired, 1 placed, 0 healthy, 0 unhealthy [health checks: 1 total, 1 critical]
[1 desired, 1 placed, 0 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total, 1 critical]
[1 desired, 1 placed, 0 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total]
[1 desired, 1 placed, 0 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total, 1 critical]
[1 desired, 1 placed, 0 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total, 1 passing]
[1 desired, 1 placed, 1 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total, 1 passing]
[1 desired, 1 placed, 1 healthy, 0 unhealthy [restarts: 1] [health checks: 1 total, 1 passing]
v0 deployed successfully

But when I run “flyctl status”, it shows “App Status” is “dead”, while “Deployment Status” is “successful”. Could someone help me understand the following questions?

Is this problem caused by lack of memory? If so will fly pick the appropriate cpu and memory based on the docker image?
Sometimes after a deployment fails, if I try to deploy again, or run “flyctl apps create” with the same docker image, it shows “An unknown error occured”. Is there a way to view verbose logs?
Where do I check how many containers (and their specs) I have launched apps with, so that I know how much will I be billed for?
I understand that I need to bring up multiple containers If I need to bring up multiple containers (like using docker-compose), how do I config some inbound rules (like backend container only accept http request from frontend container at a specific port)?

Thanks in advance.

michael · November 28, 2020, 6:34pm

A 700mb docker image should be just fine. Unfortunately we’ve seen much crazier

It could be caused by OOMs. We don’t do anything to choose the vm size for you. Checkout the docs for more info. To diagnose the failure, try running flyctl logs in another terminal while you deploy to see log output. You can also see failed vms with flyctl status --all then fetch logs and info for specific vms with flyctl status instance {instance-id} and flyctl logs -i {instance-id}

Where are you seeing that error? It sounds like it’s on our end.

You can see some usage by going to the usage tab for your organization: https://fly.io/organizations/personal/usage. We’re going to expose more actual usage metrics for billing soon.

We have some news on private networking coming very soon.

foodiebuddy · November 28, 2020, 10:00pm

Thank you @michael, glad to hear that 700MB is not so crazy
Command “flyctl logs” shows the normal logs that I see when I build docker image,

WARNING: This is a simple server for use in testing or debugging Angular applications
…
Compiling @angular/core : es2015 as esm2015
Compiling @angular/platform-browser : es2015 as esm2015
…

but the logs repeated multiple times, as if the they showed up every time I see a log about health check update from command “flyctl deploy”.

I could not reproduce the “An unknown error occured” for now, I will provide an update once I run into it again.

I am able to tell which vm am I deployed on with “flyctl scale vm”, but the price of scaling up scares me. I mean, it costs $35 for 1GB of memory, I suspect I need at least 2 vms with this spec (one for frontend, one for backend) to run my apps, and it is already more expensive than a ec2 t3.large (2 vCPU and 8 GB memory) instance or a similar instance from DigitalOcean already and I can deploy my whole stack with docker-compose in it.

kurt · November 28, 2020, 11:18pm

Is this an Angular app? You probably don’t need more RAM to run it, it probably just needs to be compiled before it’s deployed. Most people run Angular apps on our smallest instances.

You can build it locally and follow this guide to do that: Run a Static Website · Fly Docs

There are also decent guides for building multi-stage Docker images for Angular, like this one: Containerizing Angular application for production using Docker - DEV Community

We actually deploy our static + backend code into the same Fly VM. If you get the static files built, it’s pretty easy to copy them into a “backend” image. We’re happy to help with example Dockerfile if you give us a few more details. Is your backend a Node app?

foodiebuddy · November 29, 2020, 12:44am

@kurt
Wow I did not expect to get a reply today
The frontend is angular and backend is spring boot. I also have redis and mysql. I will look into the two guides, thanks.
Btw I have a question about redis: the doc says

There is no configuration to be done to enable Redis. The environment variable FLY_REDIS_CACHE_URL contains a URL for Redis and should be retrieved by your application so they can use a standard Redis driver to connect.

Does it mean there will always be a dedicated redis container for each of my application, and I can simply change the redis host in my application to FLY_REDIS_CACHE_URL with default port of 6379? Do I get to use this redis for free?

foodiebuddy · November 29, 2020, 4:46am

@kurt
I am able to reduce the docker image to be less than 60 MB following that instruction, that’s awesome!
However when I test it locally, the frontend could not connect to backend (got net::ERR_EMPTY_RESPONSE). I think the problem is in the filter of the backend path in the nginx.conf file. I know this might not be related to fly, but do you mind taking a look at my nginx.conf when you get a chance? Thanks!
Here is my nginx.conf file:

events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
server {
listen 80;
server_name host.docker.internal;
location / {
root /usr/share/nginx/html/;
index index.html index.html;
try_files $uri $uri/ /index.html =404;
}
location /business {
proxy_pass http;//foodybuddy-backend:8443/business; # sorry I have to replace http: with http; because I am new user and I can only have at most 2 urls in a post
}
location /users {
proxy_pass http;//foodybuddy-backend:8443/users; # same
}
location /votes {
proxy_pass http;//foodybuddy-backend:8443/votes; # same
}
}
}

and here is my dockerfile for the backend:

FROM adoptopenjdk/openjdk11:jre-11.0.8_10
RUN addgroup --system foodybuddygroup && adduser --system foodybuddy && adduser foodybuddy foodybuddygroup
USER foodybuddy:foodybuddygroup
ARG JAR_FILE=target/*.jar
ADD ${JAR_FILE} app.jar
EXPOSE 8443
ENTRYPOINT [“java”,“-jar”,“app.jar”]

dockerfile for the frontend:

FROM node:12.16.1-alpine As builder
WORKDIR /usr/src/app
COPY ./package*.json ./
RUN npm install
COPY . .
RUN npm run build --prod
FROM nginx:1.17.1-alpine
COPY ./nginx.conf /etc/nginx/nginx.conf
RUN rm /etc/nginx/conf.d/default.conf
COPY --from=builder /usr/src/app/dist/frontend-web/ /usr/share/nginx/html

and in the environment.ts and environment.prod.ts I defined the following:

serverUrl: ‘http://host.docker.internal:8443’,

and here is my docker-compose file:

version: ‘2’
services:
foodybuddy-frontend:
container_name: foodybuddy-frontend
image: ‘my-name/foodybuddy-frontend:0.0.1’
ports:
- ‘80:80’
depends_on:
- foodybuddy-backend
foodybuddy-backend:
container_name: foodybuddy-backend
image: ‘my-name/foodybuddy-backend:0.0.1’
ports:
- ‘8443:8443’
volumes:
- ‘/PATH-TO-LOGS/foodybuddyLogs:/usr/src/app/appBackEnd/logs’
env_file:
- ./dockerConfig/.env
depends_on:
- mysql
- redis
redis:
container_name: redis
image: ‘redis:alpine’
ports:
- ‘6379:6379’
volumes:
- $PWD/redis/redis-data:/var/lib/redis
- $PWD/redis/redis.conf:/usr/local/etc/redis/redis.conf
mysql:
container_name: mysql
image: ‘mysql:8.0.21’
ports:
- ‘3306:3306’
environment:
MYSQL_DATABASE: foodybuddy
MYSQL_USER: foodybuddy
MYSQL_PASSWORD: password
MYSQL_ROOT_PASSWORD: root-password
volumes:
- $PWD/mysql/data:/var/lib/mysql

kurt · November 30, 2020, 3:37pm

Try removing this line:

server_name host.docker.internal;

That makes it only respond to requests for host.docker.internal.

Topic		Replies	Views
2.2 GB docker image is not uploading. Questions / Help	19	2608	April 2, 2021
Large image failing to deploy	6	854	February 13, 2021
Deployment freezes forever Build debugging rails	13	744	February 6, 2023
Failing to deploy Questions / Help registry	1	87	April 23, 2024
is there a limit on image size for the registry.fly.io? registry	1	209	September 29, 2023

Failed to deploy a large 3rd party docker image

Related topics