Image using libsodium runs locally but not on fly

Questions / Help
1

I have an Docker-built app that runs fine locally, but on fly.io the libsodium crypto calls I make fail (specifically crypto_pwhash_str, but I imagine other calls fail, too).

I even downloaded the image with fly auth docker and docker pull ... and the image works fine locally, but not on fly.io. So it’s not a difference in how the image is built. Something about the way fly.io runs it is different. What’s the difference?

I did check errno and it returns 12, which is an out of memory error. But my app is using about 24MB memory out of 221MB available, so maybe a red herring?

EDIT: adding this memory usage chart for the past 12 hours if it’s helpful

Screen Shot 2023-01-19 at 10.51.31 PM
Screen Shot 2023-01-19 at 10.51.31 PM2724×1208 210 KB

2

I’ve learned now that fly runs things in Firecracker VMs, not Docker containers. So that’s the reason it’s different. Now I need to figure out what about the Firecracker VM doesn’t like my libsodium calls.

3

Here is a simple example that reproduces the problem: Fly.io example demonstrating libsodium use failure. Launch with `fly launch` and see the logs for the error. · GitHub I’ll reproduce the code below so you don’t have to follow the link.

If you want to try it:

git clone https://gist.github.com/bfecbfad83e5dfabd7dc242025841975.git flysodiumtest
cd flysodiumtest
fly launch

Then watch the logs and see this error:

Attempting crypto_pwhash_str...
Failed crypto_pwhash_str: return code: -1

Building and running the container locally works fine and produces this:

Attempting crypto_pwhash_str...
$argon2id$v=19$m=262144,t=3,p=1$5ox30Dg+Ier4oMTiAFx0Pw$ialpiGxpj2FeqBwAnG4ayYAOD5g/hilman80lWJvmtQ
ok

Code

Dockerfile

# -- Stage 1 -- #
FROM nimlang/nim:1.6.10-alpine@sha256:408ebac99ad2d170a59e7a09c10e82e7336cf71fa38c7a1322aaa598a54d32c2 as builder
WORKDIR /app
RUN apk update && apk add libsodium-static libsodium musl-dev
RUN nimble install -y libsodium
COPY . .
RUN nim c -o:testapp main.nim
RUN ls -al

# -- Stage 2 -- #
FROM alpine:3.13.12@sha256:16fd981ddc557fd3b38209d15e7ee8e3e6d9d4d579655e8e47243e2c8525b503
WORKDIR /root/
RUN apk update && apk add libressl3.1-libcrypto
COPY --from=builder /app/testapp /usr/local/bin/
CMD ["/usr/local/bin/testapp"]

main.nim

import std/asynchttpserver
import std/asyncdispatch
import libsodium/sodium

proc main {.async.} =
  # libsodium test
  try:
    echo "Attempting crypto_pwhash_str..."
    echo crypto_pwhash_str("some text")
    echo "ok"
  except:
    echo "Failed crypto_pwhash_str: ", getCurrentExceptionMsg()

  # webserver to keep fly happy
  var server = newAsyncHttpServer()
  proc cb(req: Request) {.async.} =
    let headers = {"Content-type": "text/plain; charset=utf-8"}
    await req.respond(Http200, "Hello World", headers.newHttpHeaders())
  server.listen(Port(8080))
  while true:
    if server.shouldAcceptRequest():
      await server.acceptRequest(cb)
waitFor main()

config.nims

switch("gc", "orc")
switch("d", "nimDebugDlOpen")
when defined(linux):
  import os
  switch("dynlibOverride", "libsodium")
  switch("cincludes", "/usr/include")
  switch("clibdir", "/usr/lib")
  switch("passL", "-lsodium")
4

Hi, @iffy! A 12-hour memory-usage plot is low resolution. You might need really high resolution to catch a spike in memory usage that crashes an app. So I still find that out-of-memory error intriguing. You could try upping the RAM on your VM, or adding swap.

1 Like
5

I don’t think crypto_pwhash_str needs to allocate 200MB of memory. The app is doing nothing (except responding to health checks), sitting at 24MB. No one is using it. It immediately fails on a fresh start.

Anyway, I upped it to 512MB and it worked. But if I need 512MB just to run crypto_pwhash_str, I think fly isn’t up to what I want to do with it. For contrast, running it locally with docker, I see an initial memory-usage spike of ~100MiB (that also seems excessive), which then settles down to 420KiB.

6

Whelp, nope, I was wrong. crypto_pwhash_str is intentionally memory intensive: [Question] Current expectations of ram usage · Issue #128 · jedisct1/libsodium.js · GitHub :slight_smile: So I need to adjust my expectations.

Thanks for prodding me to up the RAM.

3 Likes
7

Oh you know, we had this exact same problem with a hashing library Phoenix was using. It’s incredibly surprising to run into.