Dart + Sqlite on fly machines - server crashes

Hello

I am running a dart based server (compiled binary) on fly machines. My server needs sqlite. I use the dart package sqlite3 to connect. It works fine on Mac and Linux on my local env. For deploying on fly.io I am using a Dockerfile similar to this…

FROM dart:stable AS build

WORKDIR /app
COPY ./proj/pubspec.* .
RUN dart pub get

# Copy app source code and AOT compile it
COPY ./proj .
# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe ./bin/main.dart -o ./bin/server

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage
FROM alpine:3.14 as sqlite
RUN apk add --no-cache sqlite sqlite-dev sqlite-libs
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/

# Start server.
EXPOSE 8080
ENTRYPOINT ["/app/bin/server"]

The server starts up fine. But as soon as it tries to access sqlite, it seems to crash, with no logs either. Fly.io’s orchestrator restarts the machine. Logs look something like this

2023-02-02T14:51:34.257 app[6e82512c736787] maa [info] (I) GameHandler <redacted app log>
2023-02-02T14:51:35.740 app[6e82512c736787] maa [info] Starting clean up.
2023-02-02T14:51:36.099 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.103 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.110 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.117 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.131 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.147 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.147 proxy[6e82512c736787] maa [error] failed to connect to instance after 6 attempts
2023-02-02T14:51:36.177 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.181 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.187 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.193 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.202 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.218 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.218 proxy[6e82512c736787] maa [error] failed to connect to instance after 6 attempts
2023-02-02T14:51:36.270 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.274 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.279 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.285 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.296 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.312 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.312 proxy[6e82512c736787] maa [error] failed to connect to instance after 6 attempts
2023-02-02T14:51:36.457 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.471 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.476 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.481 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.493 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.515 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.515 proxy[6e82512c736787] maa [error] failed to connect to instance after 6 attempts
2023-02-02T14:51:36.642 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.646 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.650 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.656 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.669 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.687 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:36.687 proxy[6e82512c736787] maa [error] failed to connect to instance after 6 attempts
2023-02-02T14:51:36.740 app[6e82512c736787] maa [info] [ 15.126433] reboot: Restarting system
2023-02-02T14:51:37.666 runner[6e82512c736787] maa [info] machine did not have a restart policy, defaulting to restart
2023-02-02T14:51:38.039 app[6e82512c736787] maa [info] Starting init (commit: e3cff9e)...
2023-02-02T14:51:38.058 app[6e82512c736787] maa [info] Preparing to run: `/app/bin/server` as root
2023-02-02T14:51:38.067 app[6e82512c736787] maa [info] 2023/02/02 14:51:38 listening on [fdaa:1:19cd:a7b:1449:d2c4:dff3:2]:22 (DNS: [fdaa::3]:53)
2023-02-02T14:51:38.094 proxy[6e82512c736787] maa [error] failed to connect to machine: could not connect to already-started machine on 172.19.128.26:8081 (is your app listening on 0.0.0.0:8081?), error: connection refused by remote server
2023-02-02T14:51:38.097 proxy[6e82512c736787] maa [info] Starting machine

Any ideas on how I could figure out what’s going on. In the logs, the second line above says Starting clean up, that is not from my application’s logs, but coming from the application container. No idea what that is about either.

Thanks!

Can you share your fly.toml (or the relevant [[services]] section)?

It looks like Fly proxy isn’t able to talk to app’s main process on 8081.


Edit: And if your app is unservice-able, make sure there isn’t a lingering main process. This can happen for a variety of reasons; one that I saw in our code was: The server process stopped listening for incoming connections up on receiving a SIGINT, but some sockets (outgoing connections) were still open and wouldn’t close, and so the main server process would linger around yet not accept any incoming connections (as it had stopped listening for those). This resulted in a very similar slew of errors you’ve shared. In such cases, a forced exit(0) or equivalent might be necessary by a watchdog or supervisor or something.

Commit: node: forcefully exit on system-down event · serverless-dns/serverless-dns@1d73b2e · GitHub

Health checks for Machines would be nice as it’d (on paper) handle such scenarios with a auto-restart / forced-shutdown… Health checks on Machines

Actually it’s not that. My app works fine till I hit an API that needs sqlite. Then it crashes. I’m trying to figure out how to get sqlite on to the docker image, so that the dart executable is able to dynamically link to it.

Let me get back with some more info if I can. Most likely it seems it doesn’t have much to do with fly.io. I’ll post an update here.

1 Like

If you look into the dart image on dockerhub and click on stable, you will see that it is debian based. Debian and alpine have different libc implementations, so it is best not to mix them.

Try replacing:

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage
FROM alpine:3.14 as sqlite
RUN apk add --no-cache sqlite sqlite-dev sqlite-libs
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/

With:

FROM debian:bullseye-slim
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y libsqlite3-0 && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/

Feel free to replace libsqlite3-0 with libsqlite3-dev if you need more, or add sqlite3 if you also want the command line interface.

2 Likes

Thanks for the hint @rubys

I finally got dart+sqlite to work on a local docker container. I was trying to keep the image size small, which is why I was attempting what I was. So now I did this for a minimal image

FROM dart:stable AS build

WORKDIR /app
COPY ./proj/pubspec.* .
RUN dart pub get

# Copy app source code and AOT compile it
COPY ./proj .
# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe ./bin/main.dart -o ./bin/server

# Get libsqlite3.so
FROM debian:bullseye-slim as sqlite
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM"
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y libsqlite3-dev && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives && \
    find /usr/lib/aarch64-linux-gnu -type f ! -name 'libsql*' -delete


# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/
COPY --from=sqlite /usr/lib/aarch64-linux-gnu /usr/lib

# Start server.
EXPOSE 8080
ENTRYPOINT ["/app/bin/server"]

I am manually copying the libsqlite.so files, and this works fine now. I am able to use sqlite from the dart app.

However, I am now running into an issue which is more of a Dockerfile related issue, and I’m guessing there might be a cleaner way to do this. When I try to push this build to fly.io remote build server, the build fails because the dir /usr/lib/aarch64-linux-gnu does not exist (the location that I have hardcoded from where to copy the libsqlite3.so file).

Instead, the .so files are found at /usr/lib/x86_64-linux-gnu when the container is built on fly, because of the difference in underlying architecture (my local is an m1 mac). Is there a good way to handle this, so that I don’t have to hardcode this in the Dockerfile?

I think I did not explain the problem fully… adding to it:

I can see from the dart Dockerfile (dart-docker/Dockerfile at 2c4fd37c2a7352f7533ad1283fabf42c6dc3e3c6 · dart-lang/dart-docker · GitHub) how to get the arch.

What I am wondering is, when I build from scratch then how do I know from where to copy the .so files. In the scratch image uname / dpkg etc are not available.

A hacky way I can think of is to write a file in the sqlite image which contains the folder location. And then read that file in scratch to get the source location from where I could copy these .so files.

Does that make sense?

Couldn’t get the file trick to work because there’s no cat in the scratch image. So can’t read the file to get the location of the .so files.

Anyway, this is not something for the fly.io forums. Things are working now with some hard-coding. Here’s the Dockerfile for reference for someone who might be trying Dart + Sqlite

# Specify the Dart SDK base image version using dart:<version> (ex: dart:2.12)
FROM dart:stable AS build

# Resolve app dependencies
WORKDIR /app
COPY ./pubspec.* .
RUN dart pub get

# Copy app source code and AOT compile it
COPY ./ .
# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe ./bin/main.dart -o ./bin/server

# Get libsqlite3.so 
# found in /usr/lib/x86_64-linux-gnu, as it seems fly machines are x86_64
FROM debian:bullseye-slim as sqlite
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM"
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y libsqlite3-dev && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives && \
    find /usr/lib/x86_64-linux-gnu -type f ! -name 'libsql*' -delete


# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/
COPY --from=sqlite /usr/lib/x86_64-linux-gnu /usr/lib

# Start server.
EXPOSE 8080
ENTRYPOINT ["/app/bin/server"]
1 Like

The approach the rails Dockerfiles use is to install whatever is needed during the build stage, but then only install libsqlite3-0 on the final stage.

Looking at what is installed: Debian -- File list of package libsqlite3-0/bullseye/amd64 ; perhaps add a rm -rf /usr/share/doc afterwards?

Final update. Removed the hard-coding now. It seems to work fine with the dart package sqlite3. Basiclally it picks up libsqlite3.so if it is placed in the /usr/lib dir. Dockerfile looks like the following in case it helps someone in future

# Specify the Dart SDK base image version using dart:<version> (ex: dart:2.12)
FROM dart:stable AS build

# Resolve app dependencies
WORKDIR /app
COPY ./pubspec.* .
RUN dart pub get

# Copy app source code and AOT compile it
COPY ./ .
# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe ./bin/main.dart -o ./bin/server

# Get libsqlite3.so
FROM debian:bullseye-slim as sqlite

RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y libsqlite3-dev && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives && \
    cp /usr/lib/$(uname -m)-linux-gnu/libsqlite3.so /tmp/libsqlite3.so

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage
# libsqlite3.so file stored in `/tmp/` from the sqlite stage
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/
COPY --from=sqlite /tmp/libsqlite3.so /usr/lib/libsqlite3.so

# Start server.
EXPOSE 8080
ENTRYPOINT ["/app/bin/server"]
2 Likes