Moving to rootless Podman: cannot expose privileged port 80 and volume Permission denied

We run a small internal status dashboard on a single RHEL VM: an Nginx reverse proxy, a Python app and PostgreSQL, defined in a Compose file. As part of removing the Docker daemon (and the root-equivalent docker group) from our hosts, we moved it to rootless Podman under a dedicated svc-status user. The first podman compose up failed on the proxy:

Error: rootlessport cannot expose privileged port 80, you can add 'net.ipv4.ip_unprivileged_port_start=80' to /etc/sysctl.conf (currently 1024), or choose a larger port number (>= 1024): listen tcp 0.0.0.0:80: bind: permission denied

After we temporarily moved the proxy to port 8080 to get further, PostgreSQL crashed on start:

chmod: changing permissions of '/var/lib/postgresql/data': Operation not permitted
initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted

Exactly the same Compose file had run under Docker for two years.

Reproducing it

Both failures were deterministic and reproducible with plain podman run, which ruled out the Compose layer:

podman run --rm -p 80:80 docker.io/library/nginx:1.28
mkdir -p ~/status/pgdata
podman run --rm --user 999:999 -e POSTGRES_PASSWORD=devpass \
  -v ~/status/pgdata:/var/lib/postgresql/data docker.io/library/postgres:17

What we ruled out

Port 80 already in use. ss -ltnp 'sport = :80' showed nothing listening. The error says permission denied, not address already in use, and the hint in the message itself pointed to the real cause.

Plain Unix permissions only. The data directory was owned by svc-status with mode 0700, and the user could write to it from the shell. Yet the container could not change it.

Firewall. firewalld rules would reject traffic after the bind, not stop the bind.

Root cause

Two separate consequences of running without root:

1. Linux only lets privileged processes bind ports below net.ipv4.ip_unprivileged_port_start, which defaults to 1024. Docker's daemon runs as root and binds on the container's behalf. Rootless Podman's port forwarder, rootlessport, runs as our unprivileged user, so the kernel refuses port 80.
2. Rootless containers run in a user namespace. Container UID 0 maps to our host user, and container UIDs 1 and up map to the subordinate range in /etc/subuid. Our Compose file runs postgres with user: "999:999", and under Docker someone had long ago run chown 999 on the host directory to match. In the rootless namespace, container UID 999 is host UID 525286, while the copied directory belonged to svc-status, which appears as root inside the container. UID 999 could not chmod a directory owned by root, hence Operation not permitted. On top of that, SELinux on RHEL labels files in the home directory user_home_t, which container processes may not access unless the mount is relabelled. Docker on this host had run without selinux-enabled, so labels had never mattered; once the postgres issue was fixed, Nginx failed with open() "/etc/nginx/conf.d/default.conf" failed (13: Permission denied) on its bind-mounted config.

What is happening under the hood

Ports. ip_unprivileged_port_start is a per network namespace sysctl. Rootless Podman publishes ports from the host's network namespace, as the invoking user, through rootlessport or the pasta network backend (the default from Podman 5.0). Any port below the threshold fails the bind regardless of what happens inside the container.

UID mapping. You can see the mapping directly:

podman unshare cat /proc/self/uid_map

         0       1001          1
         1     524288      65536

Container UID 0 is host UID 1001 (our user). Container UID 999 is host UID 524288 + 999 - 1 = 525286. Files created by postgres inside the volume will show that number on the host, and files owned by host UID 1001 appear owned by root inside the container.

SELinux. With SELinux enforcing, container processes run as container_t. They can read and write files labelled container_file_t, but not user_home_t. The :z (shared) and :Z (private) volume options tell Podman to relabel the host directory before mounting.

Environment

Minimal reproduction

sudo useradd -m podtest && sudo -iu podtest
podman run -d --name web -p 80:80 docker.io/library/nginx:1.28
mkdir data && podman run --rm -v ./data:/data docker.io/library/alpine:3.21 \
  sh -c 'adduser -D -u 999 pg && su pg -c "touch /data/x"'

The first command fails with the rootlessport error. The second fails with touch: /data/x: Permission denied.

The fix

Port 80. We had three options and picked the narrowest: lower the unprivileged port threshold to exactly 80 on this host, so only ports 80 to 1023 become bindable by any user, rather than all of them.

echo 'net.ipv4.ip_unprivileged_port_start=80' | sudo tee /etc/sysctl.d/90-unprivileged-port.conf
sudo sysctl --system
sysctl net.ipv4.ip_unprivileged_port_start

This host has no other services and no untrusted local users, which is why we accepted that any local user could now bind 80 to 1023. On a shared machine we would instead keep the container on 8080 and redirect with a firewall rule:

sudo firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=8080
sudo firewall-cmd --reload

PostgreSQL volume. We used the :U option, which makes Podman chown the bind-mounted directory to the container's user mapping, together with :Z to relabel it for SELinux:

 services:
   db:
     image: docker.io/library/postgres:17
     user: "999:999"
     volumes:
-      - ./pgdata:/var/lib/postgresql/data
+      - ./pgdata:/var/lib/postgresql/data:Z,U
   proxy:
     image: docker.io/library/nginx:1.28
     ports:
       - "80:80"
     volumes:
-      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro,Z

After the first start, the directory on the host is owned by a subordinate UID. To inspect or back it up as the service user, enter the user namespace:

podman unshare ls -ln ~/status/pgdata | head -3
podman unshare tar czf /tmp/pgdata.tgz -C ~/status pgdata

Surviving logout. One more rootless surprise: when we closed the SSH session, systemd stopped the user's processes and the stack went down. We enabled lingering so the user's services keep running without a login session, and converted the stack to Quadlet units managed by systemctl --user:

sudo loginctl enable-linger svc-status
systemctl --user daemon-reload
systemctl --user enable --now status-db.service status-proxy.service

Alternatives and trade-offs

• --userns=keep-id. Maps your host UID to the same UID inside the container, which is ideal for development containers editing your source tree. It does not help postgres, which insists on running as its own UID 999.
• podman unshare chown 999:999 pgdata. Same result as :U, done once by hand. Fine, but easy to forget when the directory is recreated.
• Named volumes. podman volume create pgdata avoids host-path ownership and SELinux labels entirely, because Podman creates and labels the storage. We kept a bind mount only because existing backup scripts read that path.
• setsebool or disabling SELinux. Never. The :Z relabel is the supported fix; turning SELinux off removes a real layer of container isolation.
• Avoid :Z on shared paths. Relabelling a system directory like /etc or your whole home directory can break the host. Only apply it to directories dedicated to the container.

How to test the fix

podman compose up -d
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost/
podman exec status-db-1 psql -U postgres -c "select version();" | head -3
ls -lZ ~/status | grep pgdata

Expected: 200 from the proxy, a PostgreSQL 17 version string, and the directory labelled container_file_t. We also rebooted the VM to prove the units came back without anyone logging in.

Confirming in production

After a week, we checked that nothing on the host ran containers as root any more and that the stack survived two unattended reboots:

sudo ps -eo user,comm | grep -E "conmon|postgres|nginx" | sort | uniq -c
systemctl --user --machine=svc-status@ list-units 'status-*'

Every container process ran as svc-status or one of its subordinate UIDs. The external uptime check on port 80 had no gaps apart from the planned reboots.

Regression guard

• The Quadlet files live in Git and are deployed by our configuration management, together with the sysctl drop-in and the linger setting, so a rebuilt VM gets all three.
• A post-deploy check curls port 80 and runs psql -c 'select 1' through podman exec.
• podman auto-update is enabled for the images, with a health check, so a broken image rolls back instead of leaving the stack down.

Security implications

Moving to rootless removed the biggest risk on this host: membership in the docker group, which is effectively root. A container escape now lands in an unprivileged user with no sudo rights. The port fix trades some of that back: lowering ip_unprivileged_port_start lets any local user bind ports 80 to 1023, so a compromised local account could impersonate a service on a well-known port if it starts first. Keep the threshold as high as you can (80, not 0), use the firewall redirect on shared hosts, and keep SELinux enforcing with :Z labels rather than disabling it.

Prevention checklist

• Check every published port below 1024 before migrating to rootless.
• Map out which container UIDs need to write each bind mount; use :U or named volumes.
• Use :Z for container-private bind mounts on SELinux hosts, never on system paths.
• Enable linger and use Quadlet or systemd user units for long-running rootless services.
• Use podman unshare for backups and inspection of volume data.

References

• Podman documentation, podman run (volume options and userns): https://docs.podman.io/en/latest/markdown/podman-run.1.html
• Podman documentation, rootless tutorial: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md
• Podman documentation, Quadlet (podman-systemd.unit): https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html
• Linux kernel documentation, IP sysctl: https://docs.kernel.org/networking/ip-sysctl.html
• Red Hat documentation, Building, running, and managing containers on RHEL 9: https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/building_running_and_managing_containers/