Introduction

I’ve been meaning to take more of a look at podman for a while. This is a Redhat backed project which provides an alternative to Docker for local operation of containers. One of it’s big selling points over the current versions of Docker is the ability to run containers as an ordinary user, without needing access to a daemon process running as root (it’s worth noting that Docker are working on ‘rootless’ mode too)

What I thought would be useful would be to compare the two with common use-cases and see some of what’s going on under the hood. Both projects use runc as the underlying tool for launching containers, but the higher level components are quite different.

Installation

For these tests I installed Docker-CE on Ubuntu 18.04.3, following the Docker install process here and installed Docker 19.03.5. For podman I made use of the Ubuntu installation instructions here using the packages provided by the kubic project I installed Podman version 1.7.0.

I was pleasantly surprised to find that it’s possible to have both podman and Docker installed on the same host using this method, the deb’s provided didn’t clash at all with regards to dependencies.

Initial Post Installation

After installation, with Docker there are two daemon processes running, containerd and dockerd. With podman, as expected, there are no long running processes initially. One thing I did notice, which was unexpected was that after running the podman info command, a podman process was left running on the host.

Image storage

The Docker daemon stores all the files related to it in /var/lib/docker which obviously won’t make sense for a container tool that’s running for the individual user. Podman stores its files at /home/$USER/.local/share/containers/storage . One point that could be relevant here for some use cases is that this will mean that if you have multiple users on a host running containers, each user will pull and store their own copy of all images, which would take some additional storage

Basic commands

Running a basic interactive container with podman run -it ubuntu:18.04 /bin/bash works as expected and lands you in a “root” shell inside the container. One interesting point is that, on Ubuntu, podman defaults to requesting images from Docker Hub first, although it does support a registry search order. This is an important point for security as having a different search order for container images could result in unexpected behaviour (more details on this here and here)

Processes behind the scenes

What’s happening behind the scenes for this command is quite interesting. There’s a couple of processes being used to manage our ubuntu container. A slirp4netns process is running. This is a tool which helps networking work in unprivileged containers.

There’s also a conmon process running, which is another helper process.

These two processes are used for every container, so if you run 10 containers, you’ll get 20 supporting processes.

Comparing this to Docker, conmon seems to be the equivalent of the containerd-shim process that runs with every container and there’s no slirp4netns equivalent needed as Docker is running with root privileges.

Namespaces

Docker uses Linux namespaces to provide an isolated environment for contained processes, and as they both use runc under the covers, it’s not too surprising to see that podman is similar. There are some differences though that are worth noting. Running lsns on a host running an ubuntu container via podman we can see the following

4026532621 user        4  1352 rorym            podman
4026532622 mnt         2  1352 rorym            podman
4026532624 net         1  1479 rorym            /bin/bash
4026532679 mnt         1  1479 rorym            /bin/bash
4026532680 mnt         1  1465 rorym            /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --e
4026532681 uts         1  1479 rorym            /bin/bash
4026532682 ipc         1  1479 rorym            /bin/bash
4026532683 pid         1  1479 rorym            /bin/bash

Running lsns on a Docker container doing using the same image and command we get

4026532626 mnt         1  1932 root             /bin/bash
4026532627 uts         1  1932 root             /bin/bash
4026532628 ipc         1  1932 root             /bin/bash
4026532629 pid         1  1932 root             /bin/bash
4026532631 net         1  1932 root             /bin/bash

The podman list includes a user namespace which isn’t too surprising as we’re running as an ordinary user, but appear to be the root user inside the container. What is interesting is that there’s a single user namespace which is attached to the podman process, rather than it being directly attached to the container. Also we can see that there’s a mnt namespace for slirp4netns.

Capabilities

By default, Docker containers get a set of capabilities, which can allow them to execute operations which require root privileges. While podman is running as an ordinary user and making use of user namespaces, it does still use capabilities, but unlike Docker they only allow privileges within the user namespace and not over the entire host.

Running pscap under podman shows the following lines relevant to podman

1     1352  rorym       podman pause      full
1     1465  rorym       slirp4netns       net_bind_service
1     1468  rorym       conmon            full
1468  1479  rorym       bash              chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

For Docker we get :-

978   1913  root        containerd-shim   full
1913  1932  root        bash              chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap

The key difference being the 4 processes running for podman are all in a user namespace.

AppArmor

Docker provides a default AppArmor policy which restricts the contained process. Looking at the podman setup, there doesn’t appear to be an apparmor policy getting enabled by default. Running aa-status shows 0 processes in enforce mode.

Seccomp

Docker also uses a seccomp-bpf filter to restrict calls to specific syscalls. Looking at the bash process running under Podman, we can see that there is also a Seccomp profile attached there

cat /proc/1479/status | grep Seccomp
Seccomp:	2

Limitations of rootless containers

With the use of rootless containers, there does come some limitations/complications. There are some things that won’t work in rootless mode. Things like not being able to bind ports < 1024 make sense as this is a feature generally restricted for the root user. However, some of the other items that are root only might be a surprise like network management.

As Dan Walsh points out podman can also run containers as root, and that’s something I’ll explore more in the next post

Conclusion

It’s clear that the podman ecosystem is coming along quite well and that, for a lot of use cases, it could be used by developers on local machines to avoid the overhead and security risk of running a root daemon to allow for container use. There are some limitations, but those seem to be more about the inherent limits of running as an unprivileged user than anything else.