Podman's default logging can make podman crash when sending large amounts of data on stdout - consider disabling logging?

[trantp]

I recently managed to trigger the following podman issue while using distrobox:
containers/podman#13779

Per default the stdout and stderr of a podman container are logged via journald and this can apparently trigger rate-limiting and cause an interactive distrobox session to terminate.

This can be reproduced by following the steps in the linked podman issue like this:

distrobox create debian:12 --name deb12
distrobox enter -a '--log-level debug' deb12
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 20 | head -n 9999

This triggers the error:

DEBU[0006] Container <id> exec session <id> completed with exit code -1 
Error: timed out waiting for file <some-path>: internal libpod error

Running distrobox create with the parameter -a '--log-driver none' prevents this issue.

This seems like it might be a reasonable thing to set by default for an interactive session as was also discussed in the podman issue.

Alternatively it may be useful to document this behavior and / or offer a shortcut command line parameter to deactivate logging.

I'm a bit unsure if this can be fixed in podman itself unless they decide to default to disabling logging in interactive sessions because this appears to be triggered by an interaction between podman and journald that may not be too simple to circumvent.

[k11i]

Hello @trantp !

I ran into the same problem using neovim within a distrobox and scrolling around in a large file.

Sadly your suggested solution is not without problems.

After creating the "distrobox" with the suggested parameter -a '--log-driver none' and then entering it with distrobox enter <boxname> the script hangs.
Usually the box will be setup at first entry with the necessary packages which is shown in the terminal like this:

Starting container...                            [ OK ]
Installing basic packages...                     [ OK ]
Setting up devpts mounts...                      [ OK ]
Setting up read-only mounts...                   [ OK ]
Setting up read-write mounts...                  [ OK ]
Setting up host's sockets integration...         [ OK ]
Integrating host's themes, icons, fonts...       [ OK ]
Setting up package manager exceptions...         [ OK ]
Setting up distrobox profile...                  [ OK ]
Setting up sudo...                               [ OK ]
Setting up groups...                             [ OK ]
Setting up users...                              [ OK ]
Setting up skel...                               [ OK ]

Using the suggested parameter the enter command hangs after

Starting container...

A quick glance at the code of distrobox-enter shows the following code:

        ....
	printf >&2 "%-40s\t" "Starting container..."
	mkdir -p "${HOME}/.cache/"
	rm -f "${HOME}/.cache/.${container_name}.fifo"
	mkfifo "${HOME}/.cache/.${container_name}.fifo"
	while true; do
		# save starting loop timestamp in temp variable, we'll use it
		# after to let logs command minimize possible holes
		${container_manager} logs -f "${container_name}" 2> /dev/null \
			> "${HOME}/.cache/.${container_name}.fifo" &
		logs_pid="$!"
          .....

Trying to take a peek into the container during the hang shows:

$> podman logs -f <containerID>
Error: this container is using the 'none' log driver, cannot read logs: this container is not logging output
$> 

So at least distrobox-enter needs the logging during startup.

On the other side I think that a solution for the described problem should be found. ;-)

Greetings!