Profile a Leaking Worker

Never ask anyone to upload a heap snapshot. It contains connection tokens, API keys, and customer step data verbatim — a customer will refuse, and they are right to. This procedure keeps the snapshot on their box and moves only an aggregate histogram off it. That is the difference between an investigation that stalls and one that closes.

Every command below runs over SSH against the worker host.

1. Confirm it is actually OOM, and find what is dying

A worker under pm2 dies and restarts without the container ever leaving Up, so docker ps and docker stats both look calm. The real tells:

docker inspect <container> --format '{{.State.OOMKilled}}|{{.RestartCount}}'
docker exec <container> pm2 list          # the ↺ column is the crash count
dmesg -T | grep -aiE "Memory cgroup out of memory|Killed process"

OOMKilled=true on a running container means the cgroup OOM-killed a child while PID 1 survived. In dmesg, the process name is truncated to 15 characters: node /usr/src/a is the worker (…/worker/dist/src/bootstrap.js), not the engine — the engine runs as /usr/local/bin/node. Getting this backwards sends you profiling the wrong process.

2. JS heap or native? Decide before you snapshot