podman cannot run [conmon:e] Include journald in compilation path to log to systemd journal

[b3n3w]

Hi there,

on one of our machines were not able to run any podman container due to the following error:

[conmon:e] Include journald in compilation path to log to systemd journal
Error: write child: broken pipe

I already reinstalled podman, conmon and checked the permissions of /var/log/journal.
When running as sudo the operation works without problem.

The user is already in the systemd-journal group

Debug output of podman:
DEBU[0000] mounted container "fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0" at "/home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged" DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-f4ef9681-4344-7949-fa74-162c74242d8f tap0 DEBU[0000] Created root filesystem for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 at /home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription DEBU[0000] Setting CGroups for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 to user.slice:libpod:fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d DEBU[0000] Workdir "/" resolved to host path "/home/user/.local/share/containers/storage/overlay/4d5225f292fbc66dff495c31621fcb05193e294a5b81d9188687d441ff88a140/merged" DEBU[0000] Created OCI spec for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 at /home/user/.local/share/containers/storage/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/config.json DEBU[0000] /usr/local/libexec/podman/conmon messages will be logged to syslog DEBU[0000] running conmon: /usr/local/libexec/podman/conmon args="[--api-version 1 -c fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 -u fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 -r /usr/bin/crun -b /home/user/.local/share/containers/storage/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata -p /run/user/1000/containers/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/pidfile -n admiring_dewdney --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l journald --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/user/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0]" INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0.scope [conmon:e] Include journald in compilation path to log to systemd journal DEBU[0000] Cleaning up container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-f4ef9681-4344-7949-fa74-162c74242d8f for container fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0 DEBU[0000] unmounted container "fb8f8ce087fb8f8d9fc8ef4aedb53db3e0fb8f98084a85525898743f83a1aef0" DEBU[0000] ExitCode msg: "write child: broken pipe" Error: write child: broken pipe

[jklaiho]

I'm seeing the same thing. While deploying a Ubuntu 22.04 server, I downloaded https://github.com/containers/conmon/releases/download/v2.1.3/conmon.amd64, and am unable to use log_driver="journald" in containers.conf with it.

Seems related to containers/podman#5467

[haircommander]

@saschagrunert it seems our static builds are missing a journald library can you PTAL

[saschagrunert]

I guess we did exclude systemd (libsystemd-journald and libsystemd) on purpose because it includes a huge amount of dependencies. Also statically building systemd incorporated some troubles in the past. I see that crun does statically link systemd via nix, so we may wanna copy over the bits from there.

[b3n3w]

I guess we did exclude systemd (libsystemd-journald and libsystemd) on purpose because it includes a huge amount of dependencies. Also statically building systemd incorporated some troubles in the past. I see that crun does statically link systemd via nix, so we may wanna copy over the bits from there.

Do you have any recommendations how to get conmon and podman running again ? I already tried to uninstall every related service and reinstall them.

[mullerch]

I had the same issue. My workaround is to drop logs with podman run --log-driver none.

[jklaiho]

With no recent developments on this issue, I thought I'd share what I did to install conmon from source in a way that supports journald logging on Ubuntu 22.04:

apt install -y libglib2.0-dev libseccomp-dev libsystemd-dev
<download and extract https://github.com/containers/conmon/archive/refs/tags/v<VERSION_HERE>.tar.gz somewhere like /tmp, then in the extracted directory:>
GOCACHE="$(mktemp -d)" make PREFIX=/usr
GOCACHE="$(mktemp -d)" make PREFIX=/usr podman

After this, /usr/libexec/podman/conmon exists. Podman itself doesn't need to be installed at this stage yet. Note: PREFIX=/usr might not be strictly necessary, I think it defaults to /usr/local which may be fine for you, but I recall having some issues down the line when installing/configuring other Podman dependencies without it. I use the same prefix later when building Podman itself from source. YMMV.

[haircommander]

I have attempted to add systemd to the nix builds in #431

It looks like the upstream debian packages add systemd libraries for the builds, so we should be good there. Are folks out there still using OBS builds/does support need to be added there?

[stellarpower]

I am using the Kubic PPA and this suddenly surfaced for me. Interestingly, I didn't do anything, so guess unattended-upgrades maybe kicked in. If it's possible to add to those builds too, that would be appreciated; otherwise will just turn logging off for the time being as a workaround.

[AdamWyzgol]

Hi, any news in that issue? I see PR that could fix, anyone is working on that?