Image using libsodium runs locally but not on fly

I have an Docker-built app that runs fine locally, but on the libsodium crypto calls I make fail (specifically crypto_pwhash_str, but I imagine other calls fail, too).

I even downloaded the image with fly auth docker and docker pull ... and the image works fine locally, but not on So it’s not a difference in how the image is built. Something about the way runs it is different. What’s the difference?

I did check errno and it returns 12, which is an out of memory error. But my app is using about 24MB memory out of 221MB available, so maybe a red herring?

EDIT: adding this memory usage chart for the past 12 hours if it’s helpful

I’ve learned now that fly runs things in Firecracker VMs, not Docker containers. So that’s the reason it’s different. Now I need to figure out what about the Firecracker VM doesn’t like my libsodium calls.

Here is a simple example that reproduces the problem: example demonstrating libsodium use failure. Launch with `fly launch` and see the logs for the error. · GitHub I’ll reproduce the code below so you don’t have to follow the link.

If you want to try it:

git clone flysodiumtest
cd flysodiumtest
fly launch

Then watch the logs and see this error:

Attempting crypto_pwhash_str...
Failed crypto_pwhash_str: return code: -1

Building and running the container locally works fine and produces this:

Attempting crypto_pwhash_str...



# -- Stage 1 -- #
FROM nimlang/nim:1.6.10-alpine@sha256:408ebac99ad2d170a59e7a09c10e82e7336cf71fa38c7a1322aaa598a54d32c2 as builder
RUN apk update && apk add libsodium-static libsodium musl-dev
RUN nimble install -y libsodium
COPY . .
RUN nim c -o:testapp main.nim
RUN ls -al

# -- Stage 2 -- #
FROM alpine:3.13.12@sha256:16fd981ddc557fd3b38209d15e7ee8e3e6d9d4d579655e8e47243e2c8525b503
WORKDIR /root/
RUN apk update && apk add libressl3.1-libcrypto
COPY --from=builder /app/testapp /usr/local/bin/
CMD ["/usr/local/bin/testapp"]


import std/asynchttpserver
import std/asyncdispatch
import libsodium/sodium

proc main {.async.} =
  # libsodium test
    echo "Attempting crypto_pwhash_str..."
    echo crypto_pwhash_str("some text")
    echo "ok"
    echo "Failed crypto_pwhash_str: ", getCurrentExceptionMsg()

  # webserver to keep fly happy
  var server = newAsyncHttpServer()
  proc cb(req: Request) {.async.} =
    let headers = {"Content-type": "text/plain; charset=utf-8"}
    await req.respond(Http200, "Hello World", headers.newHttpHeaders())
  while true:
    if server.shouldAcceptRequest():
      await server.acceptRequest(cb)
waitFor main()


switch("gc", "orc")
switch("d", "nimDebugDlOpen")
when defined(linux):
  import os
  switch("dynlibOverride", "libsodium")
  switch("cincludes", "/usr/include")
  switch("clibdir", "/usr/lib")
  switch("passL", "-lsodium")

Hi, @iffy! A 12-hour memory-usage plot is low resolution. You might need really high resolution to catch a spike in memory usage that crashes an app. So I still find that out-of-memory error intriguing. You could try upping the RAM on your VM, or adding swap.

1 Like

I don’t think crypto_pwhash_str needs to allocate 200MB of memory. The app is doing nothing (except responding to health checks), sitting at 24MB. No one is using it. It immediately fails on a fresh start.

Anyway, I upped it to 512MB and it worked. But if I need 512MB just to run crypto_pwhash_str, I think fly isn’t up to what I want to do with it. For contrast, running it locally with docker, I see an initial memory-usage spike of ~100MiB (that also seems excessive), which then settles down to 420KiB.

Whelp, nope, I was wrong. crypto_pwhash_str is intentionally memory intensive: [Question] Current expectations of ram usage · Issue #128 · jedisct1/libsodium.js · GitHub :slight_smile: So I need to adjust my expectations.

Thanks for prodding me to up the RAM.


Oh you know, we had this exact same problem with a hashing library Phoenix was using. It’s incredibly surprising to run into.