Build timeout 10 minutes, large sqlite db

Questions / Help Build debugging
1

Hi, I have a large sqlite db, ~2GB.

Deployment used to work fine. But now stops just before the 10 minute mark:

==> Verifying app config
→ Verified app config
==> Building image
Remote builder fly-builder-white-wood-1586 ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.12 linux x86_64
[+] Building 596.1s (0/1)
=> [internal] load remote build context 596.1s
*ERRO[0622] Can’t add file /private/var/folders/7g/qc77bj6j63q0gr0mrzvbmxkc0000gn/T/tmph1n8z2gw/test/x.db to tar: io: read/write on closed pipe *
*ERRO[0622] Can’t close tar writer: io: read/write on closed pipe *
Error failed to fetch an image or build from source: error building: unexpected EOF

Could this be related to the remote builder detecting inactivity in 10 minutes?

2

That might be the case, but it looks like here that you’re uploading your entire sqlite database on every deployment. That’s going to eventually hit Docker image limits. Is that intentional? You could place the database on a persistent volume to speed things up.

1 Like
3

You could also try to build that locally (using --local-only), if you really want that db file to sit insider container image.

But as @jsierles mentioned, it might be better to handle that sqlite db file some other way.

Something I would consider there would be using persistent volumes and then before app starts, restoring sqlite db with litestream. That would probably make things little better structured. Probably.

1 Like
4

Thank you for the quick replies.

Re: intentional upload of sqlite database every deployment, yes @jsierles . I rebuild the read-only database after reconstructing this locally. Will explore the creation of a persistent volume though and run --local-only as suggested by @savikko.

Instead of building via the remote builder option via datasette-publish-fly, I tried doing the more manual process of constructing the dockerfile then running:

  1. fly create APP
  2. fly create volumes APP_DATA
  3. fly deploy on fly.toml

This seemed to work but keen on trying out the litestream replication suggestion as well.