13.6 Machine-learning pitfalls: leakage, purged cross-validation, labels

Leakage taxonomy#

1. Look-ahead features: any feature at $t$ that uses data after $t$: smoothed regime probabilities (13.5), full-sample GARCH parameters, centred moving averages, fundamental data stamped with the period end rather than the publication date, index membership as of today (survivorship, M04 · 4.4).
2. Preprocessing on the full sample: scaling, PCA, winsorising thresholds or feature selection fitted on all data and then cross-validated.
3. Overlapping labels: a label that spans several periods shares information with neighbouring labels.
4. Selection on the test set: tuning hyperparameters until the test score is good. This turns the test set into a training set (M04 · 4.6).

Overlapping labels#

Suppose the label for sample $i$ is the forward return $y_i = \sum_{k=1}^h r_{i+k}$ over $h$ days, with i.i.d. daily returns of variance $\sigma^2$, and samples are taken every day.

Proof

$y_i$ and $y_j$ (say $j>i$) share the returns $r_{j+1},\dots,r_{i+h}$, of which there are $\max(0, h - (j-i))$. Covariance is $\sigma^2$ times that count; each variance is $h\sigma^2$.∎

With $h = 20$, a sample and its neighbour have label correlation 0.95. Ordinary k-fold puts these near-duplicates on both sides of a fold boundary, and a flexible model (a forest, a boosted tree, nearest neighbours) memorises the training neighbour and "predicts" the test sample. The effective number of independent labels is roughly $N/h$, not $N$.

Purged k-fold with embargo#

Each sample $i$ carries an interval $[t_{0,i}, t_{1,i}]$: the time its features are observed and the time its label is resolved. Samples are sorted by $t_0$. Split positions into $K$ contiguous test blocks. For a test block $B$ let $T_{\text{start}} = t_{0,\min B}$ and $T_{\text{end}} = \max_{j\in B} t_{1,j}$.

• Purge: a sample $i\notin B$ may be used for training only if
  $$t_{1,i} < T_{\text{start}} \quad\text{or}\quad t_{0,i} > T_{\text{end}}. \tag{13.28}$$
  Then no training label interval intersects any test label interval.
• Embargo: also drop the $\lceil \epsilon N\rceil$ positions immediately after the block. Training samples just after the test block have features (trailing windows, serially correlated inputs) that overlap the test period; the embargo removes them. A typical $\epsilon$ is 0.01.

Walk-forward evaluation (train on the past, test on the next block, roll) avoids training on the future altogether but uses each observation for testing only once and in one historical path. Purged k-fold uses all data for testing, at the cost of training on data from after the test period, which is acceptable only if the data-generating process is stable. Combinatorial purged CV (López de Prado 2018, ch. 12) generates many backtest paths and feeds the probability-of-backtest-overfitting analysis in M04 · 4.6. sklearn.model_selection.TimeSeriesSplit does not purge or embargo.

Triple-barrier labels#

Fixed-horizon labels ("sign of the 10-day return") ignore the path: a trade that hit its stop on day 2 and recovered by day 10 is labelled a win. The triple-barrier method labels event $e$ by the first of three barriers touched by $R_{e,t} = \ln(P_t/P_e)$:

In practice pt and sl are multiples of a volatility estimate known at $e$ (for example the GARCH forecast of 13.4), so barriers widen in stressed markets. The resolution time $t_{1,e}$ is exactly the label end time that purging needs.

Meta-labelling#

A primary model (a rule from M04 · 4.2, or a discretionary view) decides the side: long or short. A secondary model is trained on the meta-label

that is, "was acting on this signal profitable?". Its predicted probability decides whether and how much to trade. This separates direction (where domain knowledge lives) from sizing (where statistical learning helps), and it raises precision at the cost of recall.

Example in R-multiples (M03): a breakout rule with precision 40%, average win 2R and average loss 1R has expectancy $0.4\times2 - 0.6\times1 = 0.2R$ per trade. If a meta-model discards half the signals and the kept half has precision 52% with the same payoff sizes, expectancy per kept trade becomes $0.52\times2 - 0.48 = 0.56R$, and expectancy per original signal is $0.28R$. Whether that survives depends on the meta-model's own out-of-sample validity: it is another model fitted on the same history, and the multiple-testing penalty of M04 · 4.6 applies to it too.