I’ve been meaning to take more of a look at podman for a while. This is a Redhat backed project which provides an alternative to Docker for local operation of containers. One of it’s big selling points over the current versions of Docker is the ability to run containers as an ordinary user, without needing access to a daemon process running as root (it’s worth noting that Docker are working on ‘rootless’ mode too)
What I thought would be useful would be to compare the two with common use-cases and see some of what’s going on under the hood. Both projects use runc as the underlying tool for launching containers, but the higher level components are quite different.
For these tests I installed Docker-CE on Ubuntu 18.04.3, following the Docker install process here and installed Docker 19.03.5. For podman I made use of the Ubuntu installation instructions here using the packages provided by the kubic project I installed Podman version 1.7.0.
I was pleasantly surprised to find that it’s possible to have both podman and Docker installed on the same host using this method, the deb’s provided didn’t clash at all with regards to dependencies.
Initial Post Installation
After installation, with Docker there are two daemon processes running, containerd and dockerd. With podman, as expected, there are no long running processes initially. One thing I did notice, which was unexpected was that after running the
podman info command, a podman process was left running on the host.
The Docker daemon stores all the files related to it in
/var/lib/docker which obviously won’t make sense for a container tool that’s running for the individual user. Podman stores its files at
/home/$USER/.local/share/containers/storage . One point that could be relevant here for some use cases is that this will mean that if you have multiple users on a host running containers, each user will pull and store their own copy of all images, which would take some additional storage
Running a basic interactive container with
podman run -it ubuntu:18.04 /bin/bash works as expected and lands you in a “root” shell inside the container. One interesting point is that, on Ubuntu, podman defaults to requesting images from Docker Hub first, although it does support a registry search order. This is an important point for security as having a different search order for container images could result in unexpected behaviour (more details on this here and here)
Processes behind the scenes
What’s happening behind the scenes for this command is quite interesting. There’s a couple of processes being used to manage our ubuntu container. A slirp4netns process is running. This is a tool which helps networking work in unprivileged containers.
There’s also a conmon process running, which is another helper process.
These two processes are used for every container, so if you run 10 containers, you’ll get 20 supporting processes.
Comparing this to Docker, conmon seems to be the equivalent of the
containerd-shim process that runs with every container and there’s no
slirp4netns equivalent needed as Docker is running with
Docker uses Linux namespaces to provide an isolated environment for contained processes, and as they both use
runc under the covers, it’s not too surprising to see that
podman is similar. There are some differences though that are worth noting. Running
lsns on a host running an ubuntu container via podman we can see the following
4026532621 user 4 1352 rorym podman 4026532622 mnt 2 1352 rorym podman 4026532624 net 1 1479 rorym /bin/bash 4026532679 mnt 1 1479 rorym /bin/bash 4026532680 mnt 1 1465 rorym /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --e 4026532681 uts 1 1479 rorym /bin/bash 4026532682 ipc 1 1479 rorym /bin/bash 4026532683 pid 1 1479 rorym /bin/bash
lsns on a Docker container doing using the same image and command we get
4026532626 mnt 1 1932 root /bin/bash 4026532627 uts 1 1932 root /bin/bash 4026532628 ipc 1 1932 root /bin/bash 4026532629 pid 1 1932 root /bin/bash 4026532631 net 1 1932 root /bin/bash
The podman list includes a user namespace which isn’t too surprising as we’re running as an ordinary user, but appear to be the
root user inside the container. What is interesting is that there’s a single user namespace which is attached to the podman process, rather than it being directly attached to the container. Also we can see that there’s a mnt namespace for slirp4netns.
By default, Docker containers get a set of capabilities, which can allow them to execute operations which require root privileges. While podman is running as an ordinary user and making use of user namespaces, it does still use capabilities, but unlike Docker they only allow privileges within the user namespace and not over the entire host.
pscap under podman shows the following lines relevant to podman
1 1352 rorym podman pause full 1 1465 rorym slirp4netns net_bind_service 1 1468 rorym conmon full 1468 1479 rorym bash chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap
For Docker we get :-
978 1913 root containerd-shim full 1913 1932 root bash chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap
The key difference being the 4 processes running for podman are all in a user namespace.
Docker provides a default AppArmor policy which restricts the contained process. Looking at the podman setup, there doesn’t appear to be an apparmor policy getting enabled by default. Running
aa-status shows 0 processes in enforce mode.
Docker also uses a seccomp-bpf filter to restrict calls to specific syscalls. Looking at the
bash process running under Podman, we can see that there is also a Seccomp profile attached there
cat /proc/1479/status | grep Seccomp Seccomp: 2
Limitations of rootless containers
With the use of rootless containers, there does come some limitations/complications. There are some things that won’t work in rootless mode. Things like not being able to bind ports < 1024 make sense as this is a feature generally restricted for the root user. However, some of the other items that are root only might be a surprise like network management.
As Dan Walsh points out podman can also run containers as root, and that’s something I’ll explore more in the next post
It’s clear that the podman ecosystem is coming along quite well and that, for a lot of use cases, it could be used by developers on local machines to avoid the overhead and security risk of running a root daemon to allow for container use. There are some limitations, but those seem to be more about the inherent limits of running as an unprivileged user than anything else.