Python worker hung forever on a subprocess call: pipe deadlocks and missing timeouts

Our media pipeline runs Celery workers that shell out to ImageMagick and ffmpeg to build thumbnails and previews. One afternoon the queue depth for media.previews started climbing and never came back down. Each worker host had eight worker processes; one by one they stopped acknowledging tasks. Celery's inspect command showed the same picture on every host:

celery -A media inspect active

-> celery@media-worker-3: OK
    * {'id': '6f1d...', 'name': 'media.tasks.build_preview', 'time_start': 1773921411.2, 'worker_pid': 4187}
    * {'id': '0a92...', 'name': 'media.tasks.build_preview', 'time_start': 1773921588.9, 'worker_pid': 4190}

Those tasks had been "active" for more than two hours. Nothing was logged after the line announcing the ffmpeg command. No exception, no timeout, no OOM kill.

Reproducing it

The stuck tasks all involved videos from one partner upload, all longer than about 40 minutes. Running the task by hand on one of those files hung locally too. Shorter files from the same partner finished fine.

What we ruled out

• ffmpeg itself hanging. Running the exact command in a terminal completed in 90 seconds.
• Disk or network stalls. iostat was idle and the source files were local copies.
• Celery prefetch or broker issues. The tasks had started and were holding their worker processes; the broker was healthy.

Root cause

py-spy showed where the Python side was waiting:

py-spy dump --pid 4187

Thread 4187 (idle): "MainThread"
    _wait (subprocess.py:2011)
    wait (subprocess.py:1277)
    build_preview (media/tasks.py:88)

And the kernel showed where the child was waiting:

ps -o pid,stat,wchan:20,cmd --ppid 4187

    PID STAT WCHAN                CMD
  41622 S    pipe_write           ffmpeg -i /data/in/7c1e.mp4 -vf fps=1/10,scale=320:-1 ...

The task code was:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()
out, err = proc.stdout.read(), proc.stderr.read()

ffmpeg writes progress and stream information to stderr. For long videos, that output exceeded the pipe buffer. ffmpeg blocked in write() waiting for someone to drain the pipe. Python blocked in wait() waiting for ffmpeg to exit. Neither could make progress, and there was no timeout to break the tie.

What is happening under the hood

A pipe is a fixed-size kernel buffer. On Linux the default capacity is 64 KiB (16 pages). When the writer fills it, write() blocks until the reader consumes some data. If the reader is waiting for the writer to exit before reading, you have a classic deadlock.

The Python documentation warns about exactly this for Popen.wait() with stdout=PIPE or stderr=PIPE, and recommends communicate(). communicate() reads both pipes concurrently (with select/poll on POSIX) until EOF, then waits for the process, so the child can always write.

Short videos never produced 64 KiB of stderr, so the bug only appeared with long inputs. That is why it went unnoticed for months.

We also found two related traps while fixing it.

No timeout means forever. subprocess.run() and communicate() default to timeout=None. Any child that waits for input, a lock, or a stalled network share will hold the worker indefinitely. Tools that prompt on stdin are a common case; git asking for credentials or ssh asking to confirm a host key will sit there if stdin is inherited.

Timeout does not kill grandchildren. With shell=True, or a wrapper script, the direct child is /bin/sh, and the real program is a grandchild. On timeout, run() sends SIGKILL to the shell only. Current CPython then just reaps the shell and raises TimeoutExpired (very old 3.x releases also tried to read the pipes again and could hang a second time). The grandchild keeps running as an orphan, still burning CPU and still holding the write end of the pipe. In a worker that retries, every timeout leaves another ffmpeg behind.

Environment

Minimal reproduction

import subprocess, sys

child = "import sys; sys.stderr.write('x' * 100_000); sys.stderr.flush()"

proc = subprocess.Popen([sys.executable, "-c", child],
                        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
proc.wait()          # hangs: child blocked writing the 65,537th byte
print("never reached")

Change 100_000 to 10_000 and it finishes, which is the intermittent behaviour we saw in production.

The grandchild variant:

import subprocess
subprocess.run("sleep 300; echo done", shell=True,
               stdout=subprocess.PIPE, timeout=2)
# raises TimeoutExpired after 2 s, but `sleep 300` is still running
# check with: pgrep -af 'sleep 300'

Who is affected

Any code using Popen(..., stdout=PIPE or stderr=PIPE) followed by wait(), or reading one pipe to completion before the other. Anything that calls external tools without a timeout. Anything that uses shell=True or wrapper scripts with a timeout and assumes the timeout stops all the work.

The fix

We replaced the hand-rolled Popen code with one helper that every task uses:

import os
import signal
import subprocess


class CommandFailed(RuntimeError):
    pass


def run_tool(cmd: list[str], timeout: float) -> subprocess.CompletedProcess:
    proc = subprocess.Popen(
        cmd,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        start_new_session=True,
        env={**os.environ, "GIT_TERMINAL_PROMPT": "0"},
    )
    try:
        out, err = proc.communicate(timeout=timeout)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGKILL)
        out, err = proc.communicate()
        raise CommandFailed(f"timed out after {timeout}s: {cmd[0]}")
    if proc.returncode != 0:
        raise CommandFailed(f"{cmd[0]} exited {proc.returncode}: {err[-2000:]!r}")
    return subprocess.CompletedProcess(cmd, proc.returncode, out, err)

What each part does:

• communicate() drains both pipes, removing the deadlock.
• timeout= puts an upper bound on the whole call.
• start_new_session=True puts the child in its own session and process group, so os.killpg kills it and every descendant. After the group is dead, the pipes close and the second communicate() returns promptly.
• stdin=subprocess.DEVNULL means a tool that tries to prompt gets EOF and fails fast instead of waiting.
• No shell=True; arguments are passed as a list.

For ffmpeg specifically we also cut the noise at the source with -hide_banner -loglevel error -nostdin, so stderr only carries real errors.

The task change:

-proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-proc.wait()
-out, err = proc.stdout.read(), proc.stderr.read()
+result = run_tool(cmd, timeout=600)

Alternatives and trade-offs

• subprocess.run(cmd, capture_output=True, timeout=...) is fine when there is no shell and the tool does not spawn children. It uses communicate() internally. We still prefer the helper because it handles the process group.
• Redirect output to a file (stderr=open(log_path, "wb")) avoids pipes completely and keeps full logs for large outputs. Good for tools that write megabytes.
• stderr=subprocess.DEVNULL when you truly do not need the output. Cheap, but you lose error messages.
• Celery task_time_limit as a backstop. It kills the worker process on expiry, which also leaves the child orphaned if it is in another session, so it complements the helper rather than replacing it.

How to test the fix

Tests that would hang without the fix, protected by pytest-timeout so CI fails instead of stalling:

import subprocess
import sys
import time
import pytest
from media.proc import run_tool, CommandFailed


@pytest.mark.timeout(10)
def test_large_stderr_does_not_deadlock():
    child = "import sys; sys.stderr.write('x' * 1_000_000)"
    result = run_tool([sys.executable, "-c", child], timeout=5)
    assert len(result.stderr) == 1_000_000


@pytest.mark.timeout(10)
def test_timeout_kills_grandchildren():
    marker = "sleep 299.5"
    start = time.monotonic()
    with pytest.raises(CommandFailed):
        run_tool(["sh", "-c", f"{marker}; echo done"], timeout=1)
    assert time.monotonic() - start < 3
    leftover = subprocess.run(["pgrep", "-f", marker], capture_output=True)
    assert leftover.returncode == 1  # no matching process left

pip install pytest-timeout
pytest tests/test_proc.py -q

Confirming in production

After deploying we requeued the stuck tasks and checked three things.

The long partner videos completed in about 90 seconds each. No worker showed tasks older than the configured timeout:

celery -A media inspect active --json | jq '[.[][] | select(.time_start < (now - 900))] | length'

0

And no orphaned ffmpeg processes were left behind after forced timeouts, which we tested by sending a deliberately corrupted file:

pgrep -af ffmpeg | wc -l

The count returned to zero within a second of the task failing with timed out after 600s.

Regression guard

• A Semgrep rule in CI flags .wait() on a Popen created with stdout=PIPE or stderr=PIPE, and any subprocess call without timeout.
• All external tool calls go through run_tool; a code review note points to it.
• A Prometheus alert fires when the oldest active Celery task is older than twice the longest task timeout.

Security implications

The original code passed arguments as a list, so there was no shell injection. The fix keeps it that way and removes the one place where a wrapper used shell=True with a filename, which was an injection risk for partner-supplied file names. Hung workers were also an availability problem: a crafted long input could stall the queue, so the timeout doubles as denial-of-service protection. Killing with SIGKILL means tools cannot clean up temp files, so we run each job in its own temp directory and remove it in a finally block.

Prevention checklist

• Never call wait() on a process whose output goes to a pipe; use communicate().
• Every subprocess call gets a timeout.
• Use start_new_session=True and os.killpg when the tool may spawn children.
• Set stdin=subprocess.DEVNULL for non-interactive tools.
• Avoid shell=True; pass a list of arguments.

References

• Python subprocess documentation (Popen.wait warning, communicate, timeouts): https://docs.python.org/3/library/subprocess.html
• os.killpg: https://docs.python.org/3/library/os.html
• Linux pipe(7), pipe capacity: https://man7.org/linux/man-pages/man7/pipe.7.html
• Celery task time limits: https://docs.celeryq.dev/en/stable/userguide/workers.html
• py-spy: https://github.com/benfred/py-spy