CodeCircuit: Inferring LLM-Generated Code Correctness via Attribution Graph Analysis

This skill enables Claude to assess LLM-generated code correctness by applying the attribution graph reasoning framework from the CodeCircuit paper. Instead of relying solely on test execution or surface-level pattern matching, you simulate the mechanistic interpretability approach: decompose code into a line-level dependency graph, extract topological features (density, centrality, clustering), and use structural signatures to identify where reasoning breaks down. This technique is particularly powerful for catching "structural near misses"--code that looks almost right but has subtle logical flaws in control flow, boundary conditions, or state transitions.

When to Use

• When a user asks you to review LLM-generated code (from ChatGPT, Copilot, or any model) for correctness without running it
• When debugging a function that passes some tests but fails edge cases, and the user wants to understand why the logic is wrong
• When a user asks "is this algorithm implementation correct?" for classic algorithms (binary search, sorting, graph traversal, dynamic programming)
• When verifying code translated between languages (Python to C++, Java to Python) where subtle semantic differences cause bugs
• When a user wants a deeper analysis than linting--structural reasoning about whether the code's logic actually solves the stated problem
• When identifying which specific line or expression in a function is the root cause of incorrect behavior

Key Technique: Line-Level Attribution Graph Analysis

CodeCircuit's core insight is that correct code and incorrect code have measurably different internal structure when you trace how each line's output depends on prior computations. The paper constructs attribution graphs where nodes are code lines (or sub-expressions) and directed edges represent data/control flow dependencies weighted by their influence on the final output. Correct implementations form dense, well-connected subgraphs with balanced centrality--meaning no single line dominates the reasoning flow unnaturally. Incorrect code shows characteristic anomalies: fragmented components, abnormally high betweenness centrality on wrong operations (e.g., a greedy heuristic dominating where careful state tracking should), and high error-to-feature influence ratios where the model "shortcuts" past necessary logic.

The practical takeaway for code review is a structured diagnostic: build the dependency graph of a function, compute topological features, and look for specific failure signatures. High graph fragmentation suggests missing logic connections. Centrality spikes on simple expressions (like high = mid instead of high = mid - 1) indicate the code is taking a shortcut that skips a necessary boundary adjustment. Low clustering around loop update logic signals that iteration state is not being properly maintained across iterations.