RuleFlow: Generating Reusable Pandas Program Optimizations

This skill enables Claude to optimize Pandas programs using the RuleFlow methodology — a 3-stage hybrid approach that (1) discovers per-program optimizations via LLM analysis, (2) generalizes them into abstract rewrite rules with pattern variables and runtime preconditions, and (3) applies those rules deterministically as a compiler pass. Rather than ad-hoc one-off suggestions, this produces principled, validated transformations that can be reused across codebases. The technique achieves up to 1770x speedups on individual patterns and consistently outperforms both systems-based (Modin) and compiler-based (Dias) Pandas optimization frameworks.

When to Use

• When the user asks to optimize Pandas DataFrame operations in a Python script or Jupyter notebook
• When profiling reveals slow Pandas calls (iterrows, chained indexing, apply with lambdas, copy-heavy operations like drop or rename)
• When the user wants to generate reusable optimization rules for a codebase with many similar Pandas patterns
• When migrating a data pipeline and wanting to systematically replace slow idioms with fast equivalents
• When the user has a notebook collection and wants to batch-apply performance improvements
• When building a CI linter or pre-commit hook that flags slow Pandas patterns and suggests rewrites

Key Technique

The core insight is decoupling discovery from deployment. Traditional LLM-based optimization asks the model to optimize each program individually — this is expensive, unreliable (3.79% yield), and non-reproducible. RuleFlow instead uses the LLM only twice: once to discover optimized code variants (SnippetGen), and once to generalize those variants into abstract rewrite rules (RuleGen). After that, a deterministic compiler (CodeGen) applies rules without any LLM involvement.

Rewrite rules use a typed pattern language. Each rule has three components: a Left-Hand Side (LHS) pattern with typed abstract variables like @{Name: v1} for identifiers or @{Const(str): c1} for string literals; a Right-Hand Side (RHS) replacement using those same variables; and runtime preconditions that must hold for the rewrite to be safe (e.g., isinstance(@{v1}, pd.DataFrame)). This makes rules both general enough to match many programs and safe enough to apply automatically.