HOWTO: Use Podman instead of Docker for local builds and uploading container images

If for any reason including but not limited to:

  • not needing root or privileged group (docker) access to do local builds and uploads.
  • being diametrically opposed to running a privileged daemon just so you can build some glorified tarballs.
  • thinking: how is this a client-server architecture thing for just building a friggin image!?
    …you may want to use Podman for your Fly.io related container image building needs.

So I wondered, can I just use Podman without installing anything from Docker and not requiring any root permissions?

Let’s start with just the ability to being able to deploy a locally built image.
Good folks at Fly allowed direct uploads to their registry at registry.fly.io.
Neat! Let’s build the image locally and upload it to Fly’s registry.

% podman build -t example-app .
% podman push example-app docker://registry.fly.io/example-app:latest
Error: error copying image to [...] unauthorized: authentication required.

Oh. Duh. Okay. How do we do that? I recall seeing docker mention under flyctl auth command help. Let’s do that.

% flyctl auth docker
Error unknown command "docker" for "flyctl"

Huh. So flyctl looks for Docker CLI but we don’t have it.

Update (2021-01-09):
As @michael pointed in the thread required registry credentials are just the auth token and username is ignored. We can ask for our token from flyctl instead of getting it do the credential passing for us.

flyctl auth token | podman login -u x –password-stdin

You can ignore this part from the original post:

Remember I said podman is CLI compatible with docker? Let’s see if we can get flyctl to not worry too much.

% ln -s $(which podman) ~/.local/bin/docker
% export PATH=$HOME/.local/bin:$PATH
% flyctl auth docker
Authentication successful. You can now tag and push images to registry.fly.io/{your-app}

Nice! Let’s try pushing again.

% podman push example-app docker://registry.fly.io/example-app:latest
Getting image source signatures
[...]
Storing signatures

Awesome. Can we deploy now?

% flyctl deploy -i registry.fly.io/example-app:latest
==> Validating App Configuration
==> Validating App Configuration done
Services
TCP 80/443 ⇢ 8080
[...]
==> Creating Release
Error not a valid image: registry.fly.io/example-app:latest

“Not a valid image”? I’m certain it’s valid. After all "it works on my laptop"™. :nerd_face:

Podman defaults to building OCI format container images but supports Docker image manifest v2 and schemas v1 and v2 as well. Let’s try something more Dockery; maybe deployment portion expects squarely-Docker image manifest and schema.

% podman push --format v2s2 example-app docker://registry.fly.io/example-app:latest
Getting image source signatures
[...]
Writing manifest to image destination
Storing signatures

% flyctl deploy -i registry.fly.io/example-app:latest
==> Validating App Configuration
[...]
1 desired, 1 placed, 1 healthy, 0 unhealthy [health checks: 1 total, 1 passing]
--> v13 deployed successfully

:tada:

tl;dr – just give me the commands

Fine…

1. Pass your registry credentials to Podman

% flyctl auth token | podman login -u x –password-stdin

2. Build Locally

% podman build -t example-app .

3. Upload

podman push --format v2s2 example-app docker://registry.fly.io/example-app:<ver>

Either use a specific version tag for <ver> or you can use customary latest.

4. Deploy

Either edit fly.toml to change the builder to image:

[build]
  image = "registry.fly.io/example-app:latest"

…or you can pass the image to flyctl directly.

% flyctl deploy -i registry.fly.io/example-app:latest

Can I get flyctl do local a build with Podman too?

I don’t know, yet. For this functionality flyctl wants to connect to local Docker daemon. Although Podman supports a client-server model like Docker over a Unix domain socket, I don’t know if it speaks a compatible protocol so flyctl can get its bidding done. I will likely make a follow up to this post about this.

9 Likes

That’s really cool, nice detective work :slight_smile: A few people asked about Podman, now we have something to send them to. Thanks for sharing!

It’s surprisingly simple, but the symlink and export step is gross. Thankfully you can login to podman directly using the token from flyctl:

flyctl auth token | podman login -u x –password-stdin
3 Likes

I’m using podman for local builds by running a podman system service.

One way to use it is like this:

podman system service unix:///tmp/podman.sock &
env DOCKER_HOST=unix:///tmp/podman.sock flyctl ...

Arch Linux has systemd units for a podman user service and a podman system service. It’s a matter of running the podman system service and passing that socket to flyctl via DOCKER_HOST.

4 Likes

Oh that’s cool, I run a Docker Daemon on Fly. Maybe I’ll try Podman the same way.

1 Like

I run a Podman Daemon on fly! It runs as root right now, but I’m trying to make it run as a regular user.

3 Likes

whoah neat. That’ll save me some time!

This is really cool! Thanks for sharing. I read about the Podman daemon yesterday and started thinking how to use that instead of dockerd in our remote builders

1 Like

You haven’t seen the coolest part: here’s my build file for deploying podman-daemon directly from git: .build.yml. On the last line of that file you can see that I’m using podman with flyctl to build the image locally and deploy to fly.

It’s podman all the way down.

Edit:
I’m running into a strange error. flyctl builds my image using my local podman socket successfully. It even pushes it to the correct fly registry. After it says “push done”, I get an error saying the built image could not be found.

You could take it a step further and just use libpod directly in the builder. No need to run any daemons.

1 Like

Looks like using podman via the DOCKER_HOST env var ends up with the same problem @bdd uncovered while trying to push a podman built image. Because flyctl calls podman via the API to push the image, I’m not sure where we can specify that the image to be pushed must be v2s2 and not OCI format.

@michael @kurt I dug into this issue a bit more and it gets murky real quick. So the issue is that podman offers a docker compatible REST API which flyctl uses when pointed to by the DOCKER_HOST env variable. This API has a push endpoint which flyctl calls after the image is built. It should just be a case of telling podman to push the v2s2 formatted image instead of the OCI one when calling this API. Unfortunately because this API is docker compatible, and docker doesn’t deal with multiple image formats, the API has no parameter for image format. So the issue is that when this API is called with podman, podman pushes the OCI image instead of the v2s2 image.

So that leaves us with three unappealing ways to fix podman support:

  1. Have flyctl detect when its talking to podman, and use podman specific APIs to ensure that the correct image format is used.
  2. Fix podman so that when using the docker API, it pushes the v2s2 image exactly as docker would.
  3. Fix the component on Fly’s end that only accepts v2s2 images to also accept OCI images.

I’d like to file an issue with podman to fix no. 2, but would you guys be able to confirm from your end that this is the problem? I’d be happy to build and push with podman again so that we can reproduce this.

2 Likes

You had me at “glorified tarballs” and “friggin image” :ok_hand:

Note that the login doesn’t work that way for me (MacOS and maybe newer version of podman?). Reading from stdin won’t work (it produces an error “reading password: inappropriate ioctl for device”) and it expects a registry, so my method is:

podman login -u x -p `flyctl auth token` registry.fly.io/example-app-latest```
2 Likes

I have that command on mac…

apple@x-MacBook-Pro-2 podman system        
Manage podman

Description:
  Manage podman

Usage:
  podman system [command]

Available Commands:
  connection  Manage remote API service destinations
  df          Show podman disk usage
  events      Show podman system events
  info        Display podman system information
  prune       Remove unused data

Error: missing command 'podman system COMMAND'

On NixOS setting the following options allowed me to use Podman for local builds:

virtualisation.podman = {
  enable = true;
  dockerCompat = true;
};

users.extraUsers.nameless.extraGroups = [
  # ...
  "podman"
];

Replace nameless in users.extraUsers.nameless.extraGroups with your user name. Important: you need to log out and log in for the change in your user groups to take effect.

I saw this almost a year later.
The culprit is probably the editor of Discourse. It converted one -- (0x2d, 0x2d) to UTF-8 0xe28093 en-dash. That’s why podman CLI’s flag parser was treating it as an argument, trying to display “Password:” but before that attempting to figure out with ioctl(TCGETS, ... if it can suppress echo.