python-cpu-profiling
Installation
SKILL.md
Python CPU and Runtime Profiling
Use this skill when you need to explain, investigate, or improve Python execution time in this repository. It covers the four profilers that are relevant here: cProfile, pyinstrument, py-spy, and yappi. The target program is python/program.py; commands below assume they are run from the python/ directory unless stated otherwise.
Scope and selection
Choose the clock and profiler based on the question, not on the smallest reported number. This program performs an HTTP request, parses HTML with Beautiful Soup, and counts words, so network wait, imports, and parsing can show up differently in each profile.
| Tool | Method and clock | Use it when you need | Primary output | Cost and important limits |
|---|---|---|---|---|
cProfile |
Deterministic function-call instrumentation; the CLI's default timer is elapsed real time (the profiler's built-in real-time clock) | Exact call counts and inclusive/self timing for Python functions | profile.out, inspected with pstats |
More overhead for call-heavy code; it changes the execution it measures and is not a benchmark. It profiles the interpreter process that starts the script, not child interpreters automatically. |
pyinstrument |
Statistical stack sampling at an interval; its normal report represents wall-clock/elapsed activity | A readable call tree showing where user-perceived time, including waits, is spent | Terminal tree, or HTML/JSON/etc. report | Usually less intrusive than deterministic tracing, but samples can miss very short functions; a smaller interval costs more CPU and memory. |
py-spy |
External OS-level sampling profiler; samples the running process without importing code into it and normally focuses on active/on-CPU threads | A low-overhead view of a running or production-like process, including stacks without changing the target | Live top, one-shot dump, or SVG flame graph from record |
Requires OS permission to read another process (especially on macOS and for PID attach on Linux). Sampling is approximate, and a short script may finish before enough samples are collected. |
yappi |
Deterministic in-process function/thread profiling; clock is explicitly selectable (cpu or wall) |
CPU-vs-wall comparisons, per-function totals, and thread-aware reports from Python code | Function and thread tables printed by print_all() |
Instrumentation can be substantial, especially with many calls or threads; child processes need their own profiler setup. |