How to use the backtest overfitting calculator
- Export the returns of every variant you tested. Each column is one configuration (a parameter set, a filter, a universe); each row is one period, in time order. All columns must cover the same periods.
- Paste the table. The first line can hold the variant names. Columns can be separated by commas, tabs or spaces, so a block copied from a spreadsheet works. The default is a simulated example: 10 variants over 120 months.
- Choose the number of blocks (S). The history is cut into S contiguous blocks of equal length, and every way of choosing half of them for the in-sample set is tested. Eight blocks give 70 splits.
- Choose the ranking metric and the frequency. The Sharpe ratio is the usual choice; average return ignores risk. The frequency only affects how results are annualized.
How CSCV works
Combinatorially symmetric cross-validation, from Bailey, Borwein, López de Prado and Zhu (2014, published in the Journal of Computational Finance in 2017), turns the question “did I overfit?” into a repeated experiment:
- Split the T × N matrix of returns into S blocks of consecutive rows.
- For every combination of S/2 blocks, use those rows as the in-sample set and the remaining rows as the out-of-sample set. There are C(S, S/2) such splits.
- In each split, find the variant with the best in-sample performance, then see where that same variant ranks out of sample.
- Count how often it ranks at or below the out-of-sample median.
The design is symmetric: every block appears in the in-sample set in exactly half of the splits, and in-sample and out-of-sample sets always have the same length. Keeping blocks contiguous preserves the serial structure of returns within each block.
A worked example
The default data simulate 10 variants of a fictional strategy over 120 months. Every month has a common market shock plus noise specific to each variant, and the true average returns differ only slightly, from 0.20% a month for V1 to 0.65% for V10. In other words, the variants are mostly noise with a little real skill. The numbers are generated by a seeded random number generator, so they are the same every time.
With 8 blocks the calculator tests 70 splits. In each, the in-sample winner had an average annualized Sharpe ratio of 0.53 on the data it was picked from. On the other half, the same variants averaged only 0.06, keeping 11% of their in-sample performance.
The probability of backtest overfitting is 35.7%: in that share of splits, the winner did no better than the median variant out of sample. In 42.9% of splits it lost money out of sample, and it was never the best variant out of sample (0% of splits).
The table shows that V9 won 43% of the in-sample contests. It also has the best full-sample Sharpe ratio (0.36), but by construction it is not the variant with the highest true return: noise decided the ranking.
Reading the results
PBO. The headline number estimates how likely the best backtest is to be a below-median strategy in practice. Around 50%, ranking by backtest is no better than drawing a variant at random. A low PBO says the ranking carries real information, not that the winning strategy is profitable.
The logit histogram. The chart shows the distribution of λ across all splits. Mass on the right means the in-sample winner tends to stay near the top; mass on the left means it tends to sink. The red bars add up to the PBO.
Performance degradation. The gap between the in-sample and out-of-sample Sharpe ratios of the selected variant measures how much of its backtest performance was luck. The degradation slope, from a regression of out-of-sample on in-sample performance across splits, is -0.88 in the example: a higher in-sample figure went with a lower out-of-sample one. Part of this negative relation is built into the method, because a variant’s in-sample and out-of-sample halves together make up its full history, so an unusually good first half leaves less for the second. A clearly negative slope is still the signature of selection on noise.
Probability of loss. The share of splits in which the selected variant lost money out of sample. It speaks directly to the risk of deploying the backtest winner.
What drives overfitting
- Number of variants. The more configurations you try, the more likely the best one is best by luck. Ten variants is a small search; parameter sweeps often test thousands.
- Signal-to-noise ratio. When true differences between variants are small relative to the noise in returns, as in the example, the ranking is mostly random. Larger, persistent differences in skill lower the PBO.
- Sample length. Longer histories shrink the noise in each block’s performance estimate.
- Similar variants. Many near-identical parameter sets behave like a few independent trials. The method handles correlation automatically, because it works with the actual return paths.
- Metric. Ranking by average return instead of Sharpe ratio gives a PBO of 32.9% here. Use the metric you would actually choose by, because the PBO describes that choice.
How PBO fits with other checks
PBO evaluates the research process that produced a strategy, not the strategy itself. The deflated Sharpe ratio asks whether the winner’s Sharpe ratio beats what the best of N lucky trials would show, using only summary statistics. Walk-forward efficiency compares in-sample and out-of-sample returns in time order, the way a strategy would actually be re-optimized and traded. A strategy that passes all three is far more credible than one that passes any single test.
What this calculator does not include
- Missing trials. The result is only as honest as the matrix. Variants you tried but did not export, and earlier ideas that shaped the search, are invisible to it.
- Time order across blocks. CSCV uses future blocks as in-sample data for past ones. That makes efficient use of a short history but ignores regime changes in a way a walk-forward test does not.
- Costs. Paste returns after commissions, spreads and slippage; otherwise high-turnover variants look better than they are.
- Small samples. With few variants or short blocks, the PBO can take only a few values and is itself very uncertain.
- Statistical power. A low PBO does not prove the best strategy will make money. Check its out-of-sample returns and risk with the Sharpe ratio calculator.
- Unequal blocks. When the number of rows is not a multiple of S, the earliest blocks get one extra row each.