4.6 Selection bias and multiple testing

4.6.1The problem in one line#

If you test $m$ independent rules with no edge at level $\alpha=5\%$, the probability of at least one false "discovery" is $1-0.95^m$: 40% for $m=10$, 87% for $m=40$, 99.4% for $m=100$.

4.6.2The expected maximum Sharpe ratio of noise#

Proofsketch at the level of Leadbetter, Lindgren and Rootzén, 1983, Ch. 1

First, $P(M_N\le x)=\Phi(x)^N$. Choose $a_N$ with $N(1-\Phi(a_N))=1$. The normal tail satisfies $1-\Phi(x)\sim\phi(x)/x$ as $x\to\infty$ (Mills' ratio), so for fixed $y$ and $b=1/a_N$,

$$\frac{1-\Phi(a_N+by)}{1-\Phi(a_N)}\sim\frac{\phi(a_N+y/a_N)}{\phi(a_N)}=\exp\!\Big(-y-\frac{y^2}{2a_N^2}\Big)\to e^{-y}.$$

Hence $N(1-\Phi(a_N+b_Ny))\to e^{-y}$ and $P(M_N\le a_N+b_Ny)=\big(1-\tfrac{e^{-y}+o(1)}{N}\big)^N\to\exp(-e^{-y})$. Any $b_N$ asymptotically equivalent to $1/a_N$ gives the same limit. The choice $b_N=\Phi^{-1}(1-1/(Ne))-a_N$ makes the approximation exact at $y=1$, since then $N(1-\Phi(a_N+b_N))=e^{-1}$; by the same Mills-ratio argument it is $\sim1/a_N$. The mean of the standard Gumbel law is $\gamma$: with $u=e^{-y}$,

$$\int_{-\infty}^{\infty}y\,dG(y)=\int_0^\infty(-\ln u)\,e^{-u}\,du=-\Gamma'(1)=\gamma.$$

So $\mathbb E[M_N]\approx a_N+\gamma b_N=(1-\gamma)a_N+\gamma\Phi^{-1}(1-1/(Ne))$. (Convergence of the mean, not only of the distribution, follows from uniform integrability of $(M_N-a_N)/b_N$, which holds for Gaussian tails.)∎

The right-hand side of (4.17) is a pure number: $\mathbb E[\max\hat S]$ in units of the cross-trial standard deviation $\sqrt V$ of the Sharpe estimates. With $V=1$, (4.17) gives 1.574598 for $N=10$, 2.530603 for $N=100$ and 3.255122 for $N=1000$, and the module's code reproduces these values from (4.17) directly. The Gumbel approximation slightly overstates the exact mean $\int xN\phi(x)\Phi(x)^{N-1}dx$:

$N$	Gumbel approximation (4.17)	Exact $\mathbb E[M_N]$
2	0.5198	0.5642 ($=1/\sqrt\pi$)
10	1.5746	1.5388
100	2.5306	2.5076
1000	3.2551	3.2414

For $N=2$ the approximation is poor; it is designed for large $N$.

From units to Sharpe ratios Under the null that no trial has skill, each per-period $\hat S$ has standard deviation $\approx1/\sqrt T$ by (4.13). So $\sqrt V=1/\sqrt T$ per period and $\sqrt{q/T}$ annualised.

4.6.3The deflated Sharpe ratio#

Bailey and López de Prado (2014) replace the benchmark $S^*$ in the PSR by the expected maximum under the null:

where $N$ is the number of independent trials and $V$ the variance of the Sharpe estimates across trials. In practice $N$ is estimated by clustering correlated trials (strategies that differ only slightly are not independent), and $V$ is the sample variance of the trial Sharpe ratios.

4.6.4Family-wise error and false discovery rate#

Test $m$ hypotheses $H_1,\dots,H_m$ with $p$-values $p_1,\dots,p_m$. Let $I_0$ be the set of true nulls, $m_0=|I_0|$, $V$ the number of true nulls rejected, and $R$ the total number rejected. A valid $p$-value for a true null satisfies $P(p_i\le u)\le u$.

• Family-wise error rate: $\mathrm{FWER}=P(V\ge1)$.
• False discovery rate (Benjamini and Hochberg 1995): $\mathrm{FDR}=\mathbb E[V/\max(R,1)]$.

FWER control is right when a single false positive is costly, such as deploying capital on one strategy. FDR control is right when screening many candidates for further work, such as a list of factors to investigate.

Proof

$P(V\ge1)=P\big(\bigcup_{i\in I_0}\{p_i\le\alpha/m\}\big)\le\sum_{i\in I_0}\alpha/m=m_0\alpha/m\le\alpha$.∎

Proof

Suppose some true null is rejected, and let $j$ be the step at which the first true null is rejected. All hypotheses rejected before step $j$ are false nulls, so $j-1\le m-m_0$, that is $m-j+1\ge m_0$. The rejected true null has $p_{(j)}\le\alpha/(m-j+1)\le\alpha/m_0$. Therefore $\{V\ge1\}\subseteq\{\min_{i\in I_0}p_i\le\alpha/m_0\}$, and by the union bound $P(V\ge1)\le m_0\cdot\alpha/m_0=\alpha$. Holm's thresholds $\alpha/(m-k+1)$ are all at least $\alpha/m$, so it rejects at least what Bonferroni rejects.∎

Proof

Write $\mathrm{FDR}=\sum_{i\in I_0}\mathbb E\big[\mathbf 1\{H_i\text{ rejected}\}/\max(R,1)\big]$. Fix $i\in I_0$ and let $R^{(i)}$ be the number of BH rejections when $p_i$ is replaced by 0; $R^{(i)}$ is a function of the other $p$-values only.

Claim: for each $k\ge1$, $\{p_i\le k\alpha/m,\ R=k\}=\{p_i\le k\alpha/m,\ R^{(i)}=k\}$. If $p_i\le k\alpha/m$ and $R=k$, lowering $p_i$ to 0 only shifts order statistics at or below $p_i$'s position. The values above position $k$ do not change, so the conditions $p_{(j)}>j\alpha/m$ for $j>k$ still hold, and $p_{(k)}$ can only fall. Hence $R^{(i)}=k$. Conversely, if $R^{(i)}=k$ and $p_i\le k\alpha/m$, then at least $k$ of the original $p$-values are $\le k\alpha/m$ (the $k-1$ others rejected with $p_i=0$, and $p_i$ itself), so $R\ge k$. If $R=j>k$, then lowering $p_i$ to 0 keeps $p_{(j)}\le j\alpha/m$, giving $R^{(i)}\ge j>k$, a contradiction. So $R=k$.

On $\{H_i \text{ rejected}, R=k\}$ we have $p_i\le k\alpha/m$. Using the claim and independence,

$$\begin{aligned} \mathbb E\Big[\frac{\mathbf 1\{H_i\text{ rej.}\}}{\max(R,1)}\Big]&=\sum_{k=1}^m\frac1kP\Big(p_i\le\frac{k\alpha}m,\,R^{(i)}=k\Big)=\sum_{k=1}^m\frac1kP\Big(p_i\le\frac{k\alpha}m\Big)P(R^{(i)}=k)\\ &\le\sum_{k=1}^m\frac{\alpha}{m}P(R^{(i)}=k)=\frac\alpha m. \end{aligned}$$

Summing over the $m_0$ true nulls gives $\mathrm{FDR}\le m_0\alpha/m$.∎

Under arbitrary dependence, Benjamini and Yekutieli (2001) show that replacing $\alpha$ by $\alpha/\sum_{i=1}^m i^{-1}$ restores control. Backtests of related rules on the same market are strongly dependent, so this matters.

Harvey, Liu and Zhu (2016) They catalogue 316 factors proposed to explain the cross-section of US expected returns, apply Bonferroni, Holm and Benjamini–Hochberg–Yekutieli adjustments to the growing set of tests over time, and conclude that a newly proposed factor should clear a $t$-ratio of about 3.0 rather than 2.0, and that many published findings are likely false. For orientation, the Bonferroni two-sided cut-off $\Phi^{-1}(1-\alpha/2m)$ at $\alpha=5\%$ is 1.96 for $m=1$, 2.81 for $m=10$, 3.48 for $m=100$ and 3.76 for $m=300$. Their figure of about 3.0 comes from less conservative methods applied to the time path of discoveries. For a private researcher the lesson is concrete: if you have tried 100 things, your $t>2$ is not evidence.

4.6.5White's Reality Check#

Bonferroni and Holm ignore the correlation between rules, which makes them conservative for families of similar strategies. White (2000) tests the best rule directly against the distribution of the best rule under the null.

Let $f_{k,t}$ be the excess return of rule $k$ over a benchmark (for example buy-and-hold) at $t$, $k=1..K$, and $\bar f_k$ its mean. The null is $H_0:\max_k\mathbb E[f_k]\le0$: no rule beats the benchmark. The statistic is $V_K=\sqrt T\max_k\bar f_k$. Its null distribution is approximated by resampling time periods with the stationary bootstrap of Politis and Romano (1994), which draws blocks of random geometric length to preserve serial dependence, and computing

Recentring by $\bar f_k$ imposes the null in the least favourable configuration (all means equal to zero). The $p$-value is the fraction of $V^*_{K,b}\ge V_K$. Because the maximum is taken jointly over all $K$ rules in every bootstrap draw, the correlation between rules is handled automatically.

Hansen (2005) showed the test loses power when the family contains many very poor rules, and proposed the Superior Predictive Ability (SPA) test, which studentises and drops clearly inferior rules. Romano and Wolf (2005) extended the approach to a step-down procedure (StepM) that identifies which rules beat the benchmark. The arch Python package implements SPA and StepM. The code reality_check implements White's version; its test checks that 30 noise rules give a $p$-value above 0.05 and that adding a genuine edge to one rule gives a $p$-value below 0.05.

The essential requirement is that $K$ must be every rule you tried, not only the ones you kept. A Reality Check over the 26 survivors of an unrecorded search is meaningless.

4.6.6The probability of backtest overfitting#

Bailey, Borwein, López de Prado and Zhu (2017) ask a different question: if I choose the best variant in-sample, how often is it below median out-of-sample?

If the variants are all noise, the IS winner's OOS rank is uniform and $\mathrm{PBO}\approx0.5$. If one variant has a genuine edge, it wins IS and stays above median OOS, and PBO is near 0. A high PBO means your selection procedure is choosing noise. With $S=16$ there are 12,870 splits. The symmetric design means every block is IS in half the splits and OOS in the other half, which avoids the arbitrariness of a single hold-out.

The method assumes that the $N$ variants are the whole search, and that blocks are long enough for serial dependence not to leak across block boundaries. It says nothing about whether the family of variants has an edge: a family of uniformly good strategies can have a high PBO simply because their OOS ranks are random. Read it together with DSR.

4.6.7A research protocol#

1. Keep a research log. Every backtest run, including failed ideas, is a line with date, hypothesis, parameters and result. $N$ in the DSR is the number of lines, adjusted for correlation.
2. State the hypothesis and the economic reason before testing. Momentum (4.3) has a documented mechanism; "RSI 37 crossing on Tuesdays" does not.
3. Reserve a final hold-out period that you touch once.
4. Report: the full distribution of trial Sharpe ratios, DSR, PBO, the walk-forward curve, turnover and breakeven cost.
5. Require a margin. Post-publication evidence (Case study 4.7.1) says that even published, peer-reviewed effects lose about half their return. Assume yours will lose more.