Backtesting & Bias Avoidance

Use this skill to build a correct, look-ahead-free backtest of a trading strategy and audit it for the biases that make backtests lie — look-ahead bias, survivorship bias, overfitting and data-snooping — with realistic transaction costs, out-of-sample and walk-forward validation, and a full performance profile.

The single most important job of this skill is deciding whether a backtest's edge is real or an artifact. The most undervalued and error-prone step in quant research is exactly this: a strategy can look spectacular in-sample purely because of a forgotten lag, a survivorship-cleaned universe, or a parameter mined over hundreds of trials. Default to assuming the edge is fake until it survives a clean, out-of-sample test net of costs.

Core Workflow

1. Normalize the setup. Capture the universe/asset, the signal rule, the study period, the train/test split, and the cost assumptions. If the user has no strategy in mind, default to a momentum demonstration on controlled synthetic data so the "true" edge is known and bias detection is unambiguous. Ask only when a required element is missing.
2. Confirm assumptions. Default to next-bar execution (signals act on the bar after they are known), a chronological train/test split with the most recent ~30% held out, realistic round-trip costs, and a 252-day annualization unless the user specifies otherwise. State every assumption.
3. Read references/backtest-guide.md before the first dossier in a session. Use it for the engine-construction rules, the bias taxonomy and how each is tested, the default audit thresholds, the report blueprint, and the appendix requirements.
4. Compute with the bundled scripts/run_backtest.py. It is offline by default (synthetic data) and can also read real prices via yfinance: it builds a clean engine with a linear-plus-market-impact cost model; demonstrates and quantifies look-ahead bias; attaches Newey-West HAC t-statistics and 95% CIs to every Sharpe (reporting the variance-inflation factor, effective sample size N_eff = T/factor, and a per-bet/per-segment significance check); measures overfitting with CSCV's Probability of Backtest Overfitting (PBO) plus a bootstrap CI; deflates the Sharpe for multiple testing (Deflated Sharpe Ratio, reported as an optimistic upper bound since the trial universe is a lower bound on researcher degrees of freedom); runs hundreds of independent synthetic AR(1) paths to turn PBO/DSR/out-of-sample-Sharpe point values into distributions and a false-discovery rate; draws an equity-and-drawdown chart; and reports the full metric set. Install dependencies with pip install numpy pandas scipy matplotlib (add yfinance only for real data).
5. Separate the clean result from the biased one. Always report the look-ahead-free, out-of-sample, net-of-cost number as the headline, and present any in-sample or gross figure only as a contrast that shows how much the bias inflates it.
6. Produce Markdown by default. If the user asks for Word, PDF, or a polished deliverable, generate the analytical content here first, then use the relevant document skill for final layout.

Analysis Rules