How to deploy a Deno app with data from Deno KV and JSON files?

Background

My app was built specifically for Deno Deploy so it’s tightly coupled with Deno KV. Also, there are JSON files that are constantly modified on the local to provide another set of data. Before Deno Deploy Classic was closed, all I need was to update those JSON files locally, git push --force the whole project to GitHub and the data from JSON would be updated, while the data from the KV was unchanged. I didn’t have to care about Docker and database, and didn’t think I would need to understand them.

Now Deno Deploy requires me to update the app to Deno 2, which looks not trivial to me, as my app depends on some old packages. As I intend to rewrite it in a different language anyway, I want to keep the app as is, with little modification as possible. However I’m willing to learn new DevOp knowledge.

Problem

Is it possible to achieve that workflow in Fly? It seems to me that I have to launch the app and the database separately. Is it possible to launch both as one, especially when git push --force is an easy way for me to update the JSON files. I think with doing the volume right this can be done, as Deno KV is just an sqlite3 file. But I’m not sure how to config the fly.toml file.

Currently here is how I setup:

dockerfile

FROM denoland/deno:1.46.3 AS builder

RUN apt-get update && apt-get install -y procps && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY deno.json ./
COPY . .

EXPOSE 8000
CMD ["deno", "task", "start"]

docker-compose.yaml

services:
  main:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - ./Cấu hình và dữ liệu:/app/Cấu hình và dữ liệu
    environment:
      - DENO_DIR=/app/.deno_dir
    restart: unless-stopped

volumes:
  deno_kv:
    driver: local

fly.toml

app = "doi-thoai"
build = { }
primary_region = "sin"

[env]
CORS_API = "https://doi-thoai.fly.dev"
DATABASE_URL = "https://denokv-test.fly.dev"
DENO_KV_ACCESS_TOKEN = "xxxxxxxxxxx"
ORIGIN = "http://doi-thoai.fly.dev"

[http_service]
auto_start_machines = true
auto_stop_machines = true
force_https = true
internal_port = 8_000
min_machines_running = 0
processes = [ "app" ]

[[vm]]
cpu_kind = "shared"
cpus = 1
memory = "1gb"
memory_mb = 1_024

When I run the container on my local, it looks working. But on Fly the app only shows up the UI but no data. The log says:

Machines are not listening on 0.0.0.0:8000 so our proxy is having a hard time finding a candidate to route requests to. This is an issue with your application code.

  • Your app could be missing environment variables. Some frameworks require specific environment variables to be set otherwise they stop your app.
  • The app could be listening to the wrong port, you can either change your fly.toml internal_port or modify your application code to listen on the correct port.
  • Make sure your app is listening to 0.0.0.0 and not localhost or 127.0.0.1.
  • Your app could not be listening at all, check your logs to make sure it’s running correctly.

Checking the browser console it says:

Loading module from “https://doi-thoai.fly.dev/_frsh/js/5515824eaf1af6b89dfc73010a29137508b5de18/chunk-2EHLMYDL.js” was blocked because of a disallowed MIME type (“text/html”).

It also looks like that the deps are only downloaded on the first load, making it’s super slow.

I’m not sure how to fix this. In theory, even when the database is failed, the JSON data should still work, as it is a part of the code.

If it works locally, it should work remotely.

A quick comment on your local config - ./Cấu hình và dữ liệu:/app/Cấu hình và dữ liệu is asking for trouble from a path perspective. Remove spaces and UTF-characters, e.g. something like ./config-data:/app/config-data.

From there you can configure your app to look into /app/config-data/ plus whatever files/subfolders you need. You can also configure a Fly volume to map to /app/config-data so that, instead of writing to this ephemeral path in your throwaway machine, the volume catches it instead.

Your fault report and error seem to be in conflict - you say that the UI is operational, but you are getting a proxy error saying that port 8000 is not responding in your app. Is your UI not being served from port 8000? I wonder if you are getting that error because your listener is slow to start, but it does start serving requests eventually. If so, your app should save data already, though if you don’t have a Fly volume, the data will be lost when you redeploy.

This is a web server problem that your browser is highlighting for you. This file should have a response Content-Type header of application/js. I am not familiar with Deno, but it should be configured in your listener or your web server.

It is not impossible that the Fly proxy is breaking this though. Could you find the same file in a local run of the app to see what the same header is on the same file?

Well, as you said, if it works locally, it should work remotely. But anyway, I’ve changed the path to your suggestion. Nothing changes.

I haven’t set Fly volume yet. Currently I just want to make sure that it works.

Is your UI not being served from port 8000?

How can I check? Also, what is my listener? My browser?

Could you find the same file in a local run of the app to see what the same header is on the same file?

In my local, there is no folder _frsh, only _fresh. I don’t know why the letter e is dropped. Even assuming that is fine, then that folder is dockerignore’d. So it’s generated only on Fly’s machine.

My wording was a bit careless. I should have said: if it doesn’t work locally, it definitely won’t work remotely. There are a few additional bits that need to work for the Fly bit to work.

That said, your Docker Compose YAML won’t affect Fly - by default, Fly doesn’t use Docker.

That is OK for now. You won’t have any data persistence remotely, but getting it to work at all is a good first step.

It does not look like you have declared any static folders that are served directly by the proxy, so unless you are using a frontend proxy like Cloudflare, I would guess that your UI is indeed coming from your machine on port 8000.

No, a listener is another name for a web/API server. I assume you are using one built into Deno.

I think I can see the problem(s):

The first three JS files are loaded in parallel, but they take 10 seconds; that is bad enough. Then the body is loaded, which is fully 10MB in size, which is a liability. The favicon is nearly half a megabyte!

I would say I didn’t get anything rendered for 60 seconds, which likely explains why the proxy thinks the app is not operational; it is kinda up, but it is like a snail on sleeping pills. :snail:

All the items in red are missing or corrupted. At a guess, the app doesn’t have enough memory, but as previously discussed, the server response headers are wrongly labelling the type of JS chunks too. I would be curious to know why there are so many of them.

In summary, I hope this is a personal vibe-coding effort, and not something you paid for. While I don’t know Deno, I would be surprised if this was not a mess under the covers - from the outside, I would not expect this to scale at all.

This is probably the cause of the 404s. I don’t know if your app will work if you fix it, but it would be a good place to start. Would you share where the string “_fresh” is configured? Make sure, again, that the “e” is standard ASCII and not some accented or umlauted variant.

Update

_frsh is fine here. My research indicates that this string is correct. The app is using a web framework called Deno Fresh.

Ah, interestingly some _frsh resources do actually succeed, and they have the right type:

https://doi-thoai.fly.dev/_frsh/js/90ca1dff35d26ec5e4b9700e27991c8036ed049d/chunk-V2DJWQXH.js

So the wrong Content-Type is a red herring; if the resource actually existed then the internal server would resolve the type correctly. I wonder if a build process or cache warmer is failing?

Right, your Dockerfile is missing a RUN deno task build line. Add that before the EXPOSE, commit, and redeploy. (AI gave me this answer, but it seems good to me. You can always shell into your running machine and try deno task build manually if you wish).

In the first load after my last deploy it says:

instance refused connection. is your app listening on 0.0.0.0:8000? make sure it is not only listening on 127.0.0.1 (hint: look at your startup logs, servers often print the address they are listening on)

But somehow after a couple reload it works now. It seems that letting it idle for a while and the problem reappears. My hypothesis is that as the build take some times, Fly identifies it as not listening. But it seems that adding RUN deno task build in dockerfile is a guaranteed way to have those chunk files in red. That means, the UI is rendered, but it’s not functional. In my local container it’s even worse. So I guess it’s best to stick to what works: not having a build step.

FWIW, here are the full commands defined as tasks:

"start": "deno run -A --unstable-kv --watch=static/,routes/ dev.ts",
"build": "deno run -A --unstable-kv dev.ts build",

No, this is what I code by myself without any LLM. LLMs were sucked back then. Plus, I prefer real learning than vibe-coding. Pretty sure the framework wasn’t vide-coding as well.

What do you mean by scale? Serving a lot of user, serving a lot of time, or maintaining and expanding features?

You have auto-stop enabled in fly.toml, so the Machine probably just shut down after that idle period. Changing the setting to "off" or "suspend" would help a little, although it wouldn’t solve the fundamental problem…

Frameworks that want to download and build stuff on first boot are generally a pain on the Fly.io Machines platform, which is instead geared more toward ones where you can compile everything ahead of time in your Dockerfile. The newer Sprites concept may eventually be easier for such uses, but it’s not yet at the “no surprises” implementation level, from what I’ve heard.

Moreover, the Machines platform is also not a great fit if you just want a single VM. It’s more for people who want to spread multiple nodes around the world and/or do clever things with the Machines API. Single-Machine apps are prone to weird availability gaps like you’re seeing and even to permanent data loss, if you’re running your own database, for example.

(Neither halfer nor I work for Fly.io, incidentally, and these are just our own opinions.)

I’d suggest setting this environment variable in fly.toml, within the [env] stanza. (The docker-compose.yaml file is ignored by default, and even with the compatibility mode enabled, there are rough edges. It’s best to learn the Fly.io idioms, in my view.)

More generally, try looking at the logs (fly logs) and metrics. For the former, it’s best to start it in a separate terminal window and then leave it running there while you perform your tests, etc.

If you see a lot of CPU activity at first boot, then you can switch to a performance CPU for a while and see if that helps things. (Network and disk can also be bottlenecks.) This is more expensive, but it can be a useful troubleshooting step.

As one last tip, fly ssh console will give a command prompt within the Machine (as long as there is at least one running). Then you can inspect the environment variables (printenv), see which files really exist (find /app), and so on.


Addendum: The free trial has an auto-stop that you can’t opt out of, so it may be hard to evaluate that aspect (as a strict freebie). You can try manually stopping with fly m stop and see if response times are equally bad on the next request.

I think it’s the other way around - you don’t have a build step, so Deno is compiling unavailable assets on-the-fly (though apparently not all of them). If you make it part of your image, it will have the full complement available when it is first booted. It may also avoid the problem @mayailurus alludes to, which is a high CPU requirement stemming from initial asset compilation on first boot.

Thank both of you for your help. I’ve figured out what happened. The task “start” I define is calling dev.ts as well, which will build again regardless of the build already made by the task “build”. Somehow they are conflicted. To have the build step in the dockerfile, in the command step I need to use another task that make use of the build, which calls main.ts.

For now it seems working (except a dependency caching problem which is a Deno’s bug). Now I have two other questions.

How does machine work?

I notice that when a deploy is made, 2 machines will be created. This is explained as a way to avoid downtime in case one machine has problem. And it seems to me that when the app is called from idle, a random machine will be called. Is that correct? If downtime is not really a concern, would destroy one save more money? And how frequent does downtime happen?

Also, is there a way to stop all machine at once?

How does volume work?

This is my current fly.toml

[env]
  CORS_API = 'https://doi-thoai.fly.dev'
  DENO_KV_ACCESS_TOKEN = 'xxxx'
  ORIGIN = 'http://doi-thoai.fly.dev'
  DATABASE_URL = '/deno_dir'
  DENO_DIR = '/deno_dir'

[mounts]
  source = "deno_dir"
  destination = "/deno_dir"
  processes = ["disk"] # optional - attach volumes to Machines that belong to one or more process groups

The database should be located in root/deno_dir. But I’m unable to mount it correctly. It is destroyed when all machines go idle.

Mostly correct… Under high load, both will be running simultaneously. (This is the topic of the auto-scaling/auto-stop link up above.)

Very little. A stopped Machine costs you only the RootFS fee, which is typically less than $0.15/month. (Last I checked, the dashboard lists the RootFS size that’s used in this calculation, but it’s not much larger than your uncompressed Docker image.)

It’s pretty frequent. The Fly.io platform will migrate your Machine to a different underlying physical host, without you asking, which involves a shutdown for a while, and each deploy also stops the Machine for varying amounts of time.

Moreover, congestion can cause a Machine to not be able to start until an auto-migration can happen, and in the past some people have found their (single-Machine) app offline for multiple hours. Volumes can exacerbate this, although the details aren’t really documented. (Last I heard, volumes prevented auto-migration entirely, but I get the impression that’s been relaxed recently.)

A non-obvious shortcut for this is to use the right-arrow key (→ cursor key on the keyboard) in fly m stop’s interactive list to select all.

You don’t want that final line, as there are is no "disk" process. (That would appear in a [processes] stanza.)

Volume attachment can be verified with fly m list externally and via df -h from within SSH.

Volumes are an advanced feature of Fly.io, in my opinion, and it’s best to avoid them if you can. I know you wanted to minimize changes within the app, but Managed Postgres (or another managed database externally) is really the only beginner-friendly database option on the Machines platform.

(Or, if all you need is a key-value store, then maybe Tigris would work? I don’t know Deno KV at all, :sweat_smile:.)

Volumes are not shared between Machines, and they’re tied to the underlying physical host, so when that hardware dies, which it will someday, then you will lose all that data.

(In some niches, people find a loosely acceptable middle ground in using streaming backups like Litestream. Then permanent data loss is avoided, but there’s still a manual app-rescue emergency every so often.)

The volume looks working. Many thanks :grin:

As my app is mostly for my need, not having any user in my awareness, and many data are outdated quickly, I guess I can accept the risk of downtime to reduce the cost. As for the volume, shouldn’t that be the most easiest option as all it takes is a config in fly.toml? I’ll explore other databases later, but at least I have to read the docs and use correct APIs. I don’t have money to pay for LLMs, and frankly I still prefer learning. I guess it’s better to take a look at them when it’s time to rewrite the app.

Also, when I deploy the new version, I see this warning:

Updating existing machines in 'doi-thoai' with rolling strategy

WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
  - 0.0.0.0:8000
Found these processes inside the machine with open listening sockets:   
 PROCESS        │ ADDRESSES         

────────────────┼───────────────────────────────────────
 /.fly/hallpass │ [fdaa:aa:66a5:a7b:186:91f3:716a:2]:22


-------
 ✖ Failed to clear lease for 784e…  
-------
Checking DNS configuration for doi-thoai.fly.dev
✓ DNS configuration verified

And in the monitoring page, I still see this error:

Something in your code or configuration is preventing your app from listening on 0.0.0.0 at the port specified by your fly.toml internal_port 8000. Your users will see error 502 accessing your app. This is an issue with your application code.

Do you know why they happen? And can you explain what that processes = ["disk"] does? I don’t get what the docs says.

This is for the unusual case in which you have multiple process groups and, furthermore, want only one of those process groups to have volumes. An example would be if you had a tiny cron Machine in your app, which wouldn’t need persistent storage.

(In this example, "disk" would instead be the "web" process group defined in the cron doc.)

A rarer situation is when people have expensive Machines to handle heavyweight tasks and cheaper ones to handle the common fare. The bigger workers can need a 50G volume for intermediate scene files, and the like…

@halfer’s guess of a slow startup is the standard one, really. I see a delay of around 12 seconds when waking it up with a request from out here.

A more typical boot time is around a tenth of that.

Excellent! The Fly.io platform is good for this, in general, since it’s small enough that a person can realistically read the entire set of official docs, yet it’s definitely not a toy or curtailed system…