adventures in static file hosting

1

I developed a fork of the word guessing game Semantle that runs entirely in browser using HTTP Range requests against a sqlite database, using GitHub - phiresky/sql.js-httpvfs. I was curious about whether it might be faster when the backend gets bogged down. Each guess requires a handful of 4kb page reads against a ~900MB database file. The file is stored as a LFS object in git.

I tried gitlab pages using the example .gitlab-ci.yml. Daniel Holth / semantleless · GitLab. Gitlab does deploy the site including the LFS object but their CDN does not appear to support Range requests against the file. A request for 4kb will fetch the entire file.

I tried Github Pages. The content has to go into the / or /docs directory. Github Pages does not expand LFS objects, and limits regular objects to 100MB each. When you try to fetch the database you just get the internal LFS reference, not the whole file.

I tried the Create a static website on Fly example, but thought I could get clever by using a statics section e.g.

[[statics]]
  guest_path = "/srv/http"
  url_prefix = "/"

plus ADD https://dholth.gitlab.io/semantleless/word2vec.db /srv/http/word2vec.db
or ADD --chown=1000:1000 https://dholth.gitlab.io/semantleless/word2vec.db /srv/http/word2vec.db which avoids having to send as much data from my local computer to fly.io

It seems like the fly.io [[ statics ]] section doesn’t supportRange or HEAD, or it didn’t like the file permissions. The base goStatic image runs under a user appuser but ADD with no additional arguments creates a file that is only rw by root.

I removed [[ statics ]] from my fly.toml . fly.io semed to remember or cache the static routes for this application.

What Worked

I wound up deleting and recreating my app to make sure everything was clean, and created a Dockerfile including alpine:latest so that flyctl ssh console would also work. fly’s remote builder and ADD from a URL lets me include a large amount of data in the image while sending a tiny amount of build context.

FROM docker.io/pierrezemb/gostatic:latest as gostatic-plus
ENTRYPOINT ["/goStatic"]

# plus some tools so we can see what's going on
FROM alpine:latest
COPY --from=gostatic-plus /goStatic /goStatic
ADD --chown=1000:1000 https://dholth.gitlab.io/semantleless/word2vec.db /srv/http/word2vec.db
USER 1000
COPY . /srv/http/
ENTRYPOINT ["/goStatic"]
CMD ["-port","8080","-https-promote", "-enable-logging"]

In the future the database probably just goes on a CORS-friendly file hoster.

2 Likes
2

Oh it’s the first time I hear about [[statics]]. Awesome feature!

1 Like
3

The tl/dr for me is what are the limits on [[ static ]]; does it support HEAD and bytes range requests? Does the file inside the container have to have certain permissions to be copied correctly? Maybe I made a simple mistake with the filename.

4

Ok this is super cool experimentation. I think you landed on the most-correct way to do this on Fly, but our statics feature should support this someday! It doesn’t support range requests, I don’t think.

We have a lot planned with statics. I think we can make this better.

Did you use the same Dockerfile when you tried the [[statics]] block?

This is super cool.

2 Likes
5

Now that you have a caching layer for static files… are you also considering some way of caching HTTP responses?