FeatureBench: Test-Driven Feature Task Extraction and Agentic Coding Evaluation

This skill enables Claude to extract feature-level coding tasks from real code repositories by tracing unit test dependencies through a call graph, separating feature code from the codebase, and constructing executable evaluation environments. The core technique — derived from the FeatureBench paper (ICLR 2026) — produces tasks that span multiple files, commits, and PRs, yielding far more realistic feature-development challenges than single-PR bug-fix benchmarks.

When to Use

• When the user wants to create coding benchmarks from an existing open-source repository
• When evaluating an LLM coding agent's ability to implement multi-file features end-to-end
• When the user needs to automatically identify which code constitutes a "feature" by tracing from its tests
• When constructing sandboxed Docker environments for reproducible agent evaluation
• When the user wants to separate a feature from a codebase without breaking other functionality
• When building training data with verifiable correctness for agentic coding systems
• When assessing whether an agent can implement features from scratch vs. extend existing code

Key Technique

FeatureBench's insight is that unit tests already encode feature boundaries. By executing tests under a dynamic tracer, you capture every function call a test triggers. This produces a directed dependency graph where nodes are functions (with source locations) and edges are call relationships. Functions touched only by the target feature's tests — and never by other passing tests — are the feature's extractable code. Removing that code should make the feature's tests fail while all other tests continue to pass. This "fail-to-pass / pass-to-pass" (F2P/P2P) invariant is the correctness guarantee.

The method uses an LLM classifier to distinguish test targets (functions under test) from test utilities (helpers, fixtures, mocks) among a test file's imports. Targets become the entry points for a breadth-first traversal of the dependency graph. Nodes encountered during P2P test execution are marked "remained" (must not be removed); nodes absent from P2P runs are marked "extracted" (safe to remove). The traversal is bounded to produce patches of 3,000–5,000 lines, yielding tasks of realistic feature complexity.