1,567 zombie processes in a container

I was hunting a memory leak and found something else instead: a container hosting 1,567 zombie processes. They had been sitting there for sixteen days. Nobody had ever seen them.

Here is why, and how one word fixes it.

What a zombie actually is

When a program terminates on Linux, the kernel releases its memory right away. But it keeps a small record in the program's name, holding its exit code, in case whatever launched it wants to know how things went.

The parent comes and reads the record, and it disappears. That takes a few microseconds.

A zombie is simply a record nobody came to read. There is no program left behind it, no memory, nothing running. Just a line that stays.

PID: the unique number the kernel assigns to every process. When I talk about “process number 1” in this article, PID 1 is what I mean. The parent's number is called the PPID.
fork: a process duplicates itself to create a new one. The original is the parent, the copy is the child. This is the basic mechanism for creating a process on Linux: the call actually used these days is clone, and fork is just one particular case of it.
exec: the copy swaps its program for another one, keeping the same number. A fork followed by an exec is what your shell does for every command you type.
prefork: creating your children up front at startup rather than one per task. There are two upsides. Creating a process costs time, so you pay that bill once at startup and the children only have to wait for work. And if one of them crashes or starts leaking, the parent replaces it without the service going down. This is the shape of a web server or a worker: one parent handing out work, children doing it.
signal: a one-word message sent to a process, either by the kernel or by another process. It carries no data, only its own name. Three of them matter here:
- SIGTERM: “please terminate”. The process can catch it to finish what it started and close its files cleanly before leaving. This is what docker stop sends first.
- SIGKILL: “you die now”. Impossible to catch or ignore, the kernel carries it out itself. No cleanup is possible, which is where the leftover temp files and half-finished writes come from.
- SIGCHLD, no I in it: “one of your children has just terminated”. The kernel sends it to the parent, and this is the signal that tells it there is a record to go and read. A parent that never goes and reads it produces a zombie. Watch out for the wording trap here: “ignoring” SIGCHLD in the strict sense, meaning explicitly setting it to SIG_IGN, does the opposite — the kernel then reaps the children on its own and no zombie appears at all. The zombie comes from the missing wait() call, not from the signal.
reaping: the parent reads the exit code of its finished child through the wait() call, and the record disappears.
orphan: a child whose parent has died. The kernel hands it over to process number 1, which therefore inherits the reaping.

Why they pile up inside a container

On Linux, when a program dies leaving children behind, those children are handed over to process number 1. Reading their records is now its job.

On a regular server, number 1 is systemd, and this is precisely what it is there for.

Inside a container there is no systemd. Number 1 is your application, simply because it is the first program launched.

And your application has no idea it inherited that chore. Nobody wrote it for that. So it ignores the records, and they stack up.

In my case, the application opened a browser in the background for every task. Each shutdown left two records behind. Sixteen days later, 1,567.

Why nobody sees them

Because a zombie consumes almost nothing. Let's be precise: the record is a real kernel structure, the task_struct, and it weighs about 7 KB — 7,104 bytes on my kernel, a figure you can read straight out of /proc/slabinfo. Across my 1,567 zombies that comes to 11 MB, and that is a floor: the kernel also keeps a few satellite structures around each record. It is real, but it is lost in the noise.

Two things need to be told apart. On one side, the memory the program was using to do its work: its data, its open files. The kernel handed that back to the system the instant the program terminated. On the other side, the record, which stays, and which the kernel keeps on its own books.

Your monitoring tools measure the first one. For each process, they show how much memory it occupies. For a zombie that number is zero, because there is nothing left to measure.

So every reflex you have misses it. You sort processes by descending memory, the zombies sit at the bottom of the list and scroll off the screen. You look at the RAM graphs, nothing moves. You restart the container, everything comes back up, and you conclude it was nothing.

One detail made me pause. Those records do not belong to your application, they belong to process number 1. And in that container, number 1 was not the application: it was a small supervisor that restarts it automatically whenever the code changes.

So the application could restart as many times as it liked, number 1 stayed in place, and the records with it. When I took the measurement, the application had restarted eight minutes earlier. The oldest records were sixteen days old.

Restarting the container, on the other hand, wipes everything: number 1 dies, and the records die with it. That is exactly what lets the problem go unnoticed for months, because a restart makes it look like everything is fine.

If your application is number 1 directly, with no supervisor above it, the question does not arise: restarting it amounts to restarting the container.

Does it actually matter

Honestly, in my case, no. A hundred records a day against a ceiling of 38,193: it would have taken roughly a year before it became a problem. The memory leak I was chasing in the first place had a completely different cause.

That ceiling is the pids.max of my container's cgroup. Check yours rather than reusing my number, because it varies wildly:

docker exec MY_CONTAINER cat /sys/fs/cgroup/pids.max

Docker sets no limit by default, a --pids-limit 100 fills up in a day, and if there is no cgroup limit at all then the kernel's pid_max is the backstop — and that one is shared with the rest of the machine.

But the ceiling exists. Once it is reached, no program can start inside the container any more, and that includes the processes you would need to work out what is going on. Better not to get there.

Counting them

docker exec MY_CONTAINER sh -c "grep -l '^State:.*Z' /proc/[0-9]*/status | wc -l"

And to find out what they are:

docker exec MY_CONTAINER sh -c "grep -h '^Name:' \$(grep -l '^State:.*Z' /proc/[0-9]*/status) | cut -f2 | sort | uniq -c | sort -rn | head"

Which gave me:

   1499 headless_shell
     25 Socket Process
     25 RDD Process
     13 Utility Process
      5 celery

If the number climbs day after day and never comes back down, you have the same problem I had.

Fixing it

This is not about changing your application. It is about handing number 1 back to a program that knows how to clean up, and letting your application sit one rung lower.

Docker ships what you need, behind a single flag:

docker run --init my-image

In Docker Compose:

services:
  my-worker:
    init: true

That's it. Docker puts tini at number 1, a tiny program whose only job is exactly this: launch your application, then read the records as they come in.

One limit worth knowing: tini only reaps what is handed to it, meaning the orphans the kernel reassigns to it. If your application itself launches children and stays alive without ever calling wait(), the records stay attached to it and --init will change nothing. The test fits in one command: if the PPID of your zombies is 1, the flag is enough. That was my case.

Where that flag does not exist, because neither docker run nor Compose is what starts the container, you install tini straight into the image:

RUN apt-get update && apt-get install -y --no-install-recommends tini \
    && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["my-app"]

Pleasant side effect: your docker stop becomes instant. It used to take ten seconds, and here is why: the kernel does not apply the default signal actions to process number 1. With no SIGTERM handler, number 1 simply drops the polite shutdown request Docker sends first, and you have to wait out the ten seconds after which Docker loses patience and sends SIGKILL. Moved one rung down under tini, that same application is no longer covered by the exception and stops immediately.

What I take away from it

A container can accumulate sixteen days of garbage without a single dashboard flagging anything at all, simply because that garbage weighs nothing.

If your containers launch sub-processes — browsers, converters, command-line tools — the question is worth asking once and for all: who is number 1 in your setup? If the answer is your application, add --init and move on.

Sources

wait(2) — the definition of a zombie and the note on SIG_IGN: “children that terminate do not become zombies”. It is also the page that points out an unreaped zombie holds a slot in the kernel process table.
pid_namespaces(7) — orphans being reparented to number 1 of the namespace, the PR_SET_CHILD_SUBREAPER exception, and the rule behind the ten seconds of docker stop: “only signals for which the init process has established a signal handler can be sent to the init process”.
proc_pid_status(5) — the fields the counting commands read: State, whose values include Z (zombie), and Name, truncated to 16 characters.
Control Group v2 — the PID controller and pids.max, the ceiling discussed above, whose default value is max.
krallin/tini — the program Docker puts at number 1, and what it does exactly: “reaping zombies and performing signal forwarding”.
docker run and the Compose file reference — --init and init: true, in Docker's own words: “run an init inside the container that forwards signals and reaps processes”. The same reference notes that --pids-limit takes -1 to lift the limit.

Questions, or other technical details to share? Get in touch or find me on LinkedIn.