4.4 Backtesting methodology · M04 Systematic research

Motivation

A backtest is a simulation of what a rule would have earned. It is a program, and like any program it has bugs. Unlike most bugs, backtest bugs are biased: almost all of them make the result look better. This section defines what a correct backtest is, lists the ways it fails, and gives tests that catch the failures.

4.4.1Event-driven and vectorised designs#

Fix the timing convention once. The strategy observes bar $t$ after its close, computes target weight $w_t\in\mathcal F_t$ , and the order fills at the open of $t+1$ . (Filling at the close of $t$ is possible in India through the closing auction session introduced in 2026, see M01, but a close fill must then use a signal computed before the close.)

Algorithm 4.1event-driven backtest; run_backtest in the code

State: cash $K$ , shares $q$ , pending target $w$ . For $t=0,1,\dots,T-1$ :

At the open: equity $E^o_t=K+qO_t$ . Desired shares $q^*=wE^o_t/O_t$ . Trade $\delta=q^*-q$ at fill price $O_t(1+s\operatorname{sign}\delta)$ , where $s$ is slippage. Pay commission $c|\delta|\cdot$ fill. Update $K$ and $q$ .
At the close: equity $E_t=K+qC_t$ .
Call the strategy on bars $0..t$ only, and store its target $w$ for the next open.

Because step 3 receives a truncated history, look-ahead is impossible by construction, whatever the strategy code does. That is the main argument for event-driven design. Its cost is speed: a Python loop over bars, calling the strategy each time.

A vectorised backtest computes all signals at once on full arrays and multiplies by returns. It is fast and convenient for parameter sweeps, and it is correct if and only if the weights are lagged properly.

Proposition 4.8when vectorised equals event-driven

Suppose the event-driven engine rebalances to the target weight at every open and costs are zero. Let $a_t=w_{t-1}$ (with $a_0=0$ ) be the weight held over bar $t$ . Then

E^o_t=E^o_{t-1}\Big(1+a_{t-1}\Big(\frac{O_t}{O_{t-1}}-1\Big)\Big),\qquad E_t=E^o_t\Big(1+a_t\Big(\frac{C_t}{O_t}-1\Big)\Big). \tag{4.8}

With proportional cost $k$ per side, the open equity is multiplied in addition by $(1-k|a_t-\tilde a_t|)$ , where $\tilde a_t=a_{t-1}g_t/(1+a_{t-1}(g_t-1))$ , $g_t=O_t/O_{t-1}$ , is the weight to which the previous position has drifted. This is exact to first order in $k$ .

Proof

With zero costs, after rebalancing at open $t-1$ the engine holds $q=a_{t-1}E^o_{t-1}/O_{t-1}$ shares and cash $E^o_{t-1}-qO_{t-1}$ . At the next open, $E^o_t=E^o_{t-1}-qO_{t-1}+qO_t=E^o_{t-1}+a_{t-1}E^o_{t-1}(g_t-1)$ . Rebalancing at open $t$ is self-financing, so $E^o_t$ is unchanged and the new holding is $a_tE^o_t/O_t$ shares. Then $E_t=E^o_t+a_tE^o_t(C_t/O_t-1)$ . With costs, the traded value at open $t$ is $|a_tE^o_t-qO_t|=E^o_t|a_t-\tilde a_t|$ , since $qO_t/E^o_t=a_{t-1}E^o_{t-1}g_t/E^o_t=\tilde a_t$ . The cost $kE^o_t|a_t-\tilde a_t|$ is deducted. The engine sizes the new position from pre-cost equity, so the post-cost weight is $a_t/(1-k|a_t-\tilde a_t|)$ ; that discrepancy is $O(k)$ times the next return.∎

The code verifies (4.8) to $10^{-10}$ with zero costs and to $2\times10^{-3}$ relative with costs. The vectorised formula is correct because of the lag $a_t=w_{t-1}$ . Drop that lag and you earn the return of the bar on which the signal was computed. The test test_unshifted_vectorised_is_lookahead shows a rule "long if today's close is above today's open" turning a random walk into a fortune when the shift is omitted.

The engine also has a second mode, rebalance="on_change": it trades only when the target changes and otherwise lets the weight drift. That is how a swing trader actually holds a position, and it has much lower turnover than constant-weight rebalancing.

4.4.2Look-ahead bias: a catalogue#

Look-ahead bias is using information at time $t$ that was not available at $t$ . Ways it enters:

Same-bar execution. The signal uses the close of $t$ and the position earns the return from close $t-1$ to close $t$ . Fix: lag the position by one bar, as in (4.8).
Centred or forward windows. rolling(..., center=True), shift(-1), or a swing high confirmed $k$ bars later but marked on the day it occurred (4.1.2, 4.2.4).
Full-sample normalisation. Z-scoring a feature with the full-sample mean and standard deviation, or choosing a threshold such as "the top decile of RVOL" from the whole history. Use expanding or rolling statistics.
Restated or late fundamentals. A quarterly result is known only after its announcement date and time. Under SEBI's listing regulations (LODR Regulation 33), Indian listed companies file quarterly results within 45 days of quarter end and annual results within 60 days; check the current regulation before relying on these deadlines. Databases often store the restated number against the quarter-end date. Use announcement-dated, as-first-reported data.
Index membership. Using today's Nifty 500 list for a 2015 backtest. Use point-in-time constituents from NSE index-change press releases.
Corporate-action adjustment of price levels. Adjusted prices are correct for returns. A rule on price levels (a minimum price of ₹100, or "₹1,000 is resistance") changes when a later split rescales the past.
Time zones. The US cash market closes around 01:30–02:30 IST on the next Indian calendar day. Using the S&P 500 close dated $t$ to trade Nifty at its close on day $t$ uses information from 10–11 hours in the future.
Smoothed state estimates. Regime probabilities from a Kalman or Hamilton smoother use the whole sample. Use filtered probabilities (M13 · 13.5).
Parameter selection on the full sample. Choosing the best moving-average length using all data and then reporting the backtest over the same data. This is look-ahead in the choice of model, and 4.6 treats it.

A necessary test If a strategy is causal, perturbing any data after date $k$ leaves all targets and equity up to $k$ unchanged. The test test_no_lookahead_future_perturbation multiplies all prices after $k$ by a random path and asserts that targets and equity through $k$ are bit-identical. Run this test on every new strategy. It is necessary, not sufficient: it cannot detect look-ahead hidden inside your data, such as restated fundamentals or survivor lists, because perturbing prices does not change those inputs.

4.4.3Survivorship bias#

A survivor-biased universe contains only assets that still exist (or are still in an index) at the end of the sample. Firms that were delisted, merged or dropped from the index are missing, and they are disproportionately the poor performers.

Proposition 4.9survivorship bias in a simple model

Let a stock's annual return be $R\sim N(\mu,\sigma^2)$ and suppose it survives the year iff $R>-d$ . Then

\mathbb E[R\mid\text{survive}]=\mu+\sigma\,\frac{\phi(a)}{1-\Phi(a)},\qquad a=\frac{-d-\mu}{\sigma}. \tag{4.9}

Proof

$\mathbb E[R\mid R>-d]=\mu+\sigma\,\mathbb E[Z\mid Z>a]$ with $Z$ standard normal. Now $\mathbb E[Z\mathbf 1\{Z>a\}]=\int_a^\infty z\phi(z)dz=[-\phi(z)]_a^\infty=\phi(a)$ , and $P(Z>a)=1-\Phi(a)$ .∎

Worked example 4.4

Small caps with $\mu=12\%$ , $\sigma=40\%$ , and removal from the universe after a fall of more than 50% in a year ( $d=0.5$ ). Then $a=-1.55$ , survival probability 93.9%, and the survivors' mean return is 17.1%: a bias of 5.1 percentage points a year from a 6% annual attrition rate. Real delisting is not a clean threshold, and the bias compounds across years. For breakout systems the bias is worse, because a stock that broke out and then collapsed out of the index is exactly the losing trade the backtest omits.

4.4.4Costs and slippage#

M01 built the full cost algebra. For research, collapse it into a per-side rate $c$ and an assumed slippage $s$ .

Proposition 4.10cost drag and breakeven cost

Let a strategy have gross mean return $\mu_g$ and volatility $\sigma$ per year, and turnover $\mathrm{TO}$ per year (total traded value, buys plus sells, divided by average capital; this is twice the one-way turnover, the lesser of purchases or sales over average assets, that fund factsheets usually report). With per-side cost $c$ (including slippage), the net mean is approximately $\mu_n=\mu_g-c\,\mathrm{TO}$ , the net Sharpe ratio is $(\mu_g-c\,\mathrm{TO})/\sigma$ , and the breakeven cost is

c^*=\mu_g/\mathrm{TO}. \tag{4.10}

Proof

Each period's net return is the gross return minus $c$ times traded value over equity (first order in $c$ , as in Proposition 4.8). Summing over a year gives $\mu_g-c\,\mathrm{TO}$ . Costs are deterministic given turnover, so to first order they do not change $\sigma$ . Set $\mu_n=0$ for $c^*$ .∎

Worked example 4.5

A breakout system makes 25 round trips a year, each with the full portfolio: $\mathrm{TO}=50$ . At 11.9 bp per side (Worked example 4.3) the drag is $50\times0.119\%=5.95\%$ a year. Add 10 bp per side of slippage and it is 10.95%. A gross 18% a year becomes about 7%. The breakeven cost is $18\%/50=36$ bp per side, a margin that mid-cap slippage alone can consume. Measure turnover in every backtest and report $c^*$ alongside the Sharpe ratio.

Slippage has two parts: half the bid–ask spread, and market impact, which grows roughly with the square root of order size relative to daily volume (M05 derives the Almgren–Chriss model). For stocks with price bands, also model the no-fill day: a sell stop on a stock locked at its lower band does not execute.

4.4.5Walk-forward analysis#

Algorithm 4.2walk-forward

Choose a training length $L$ and test length $\ell$ . For $k=0,1,\dots$ : fit or select parameters on $[k\ell,\,k\ell+L)$ ; freeze them; run the strategy on $[k\ell+L,\,k\ell+L+\ell)$ and record its out-of-sample returns. Concatenate the test segments into one out-of-sample equity curve. Rolling walk-forward keeps $L$ fixed. Anchored walk-forward starts every training window at the beginning of the data.

Example: daily Nifty 500 data from January 2010 to December 2025, $L=3$ years and $\ell=1$ year, gives 13 test years (2013–2025), each traded with parameters chosen only from the three preceding years.

Practitioner notes on walk-forward:

It reduces, but does not remove, selection bias. If you try 50 versions of the walk-forward procedure (window lengths, parameter grids, rules) and report the best stitched curve, you are back to 4.6.
Labels that span time (a 20-day forward return) overlap the boundary between train and test. Purge training samples whose label window overlaps the test window, and embargo a gap after it. M13 · 13.6 gives the purged k-fold construction.
The short test windows make the out-of-sample Sharpe ratio noisy. Use 4.5 for error bars on the stitched curve, not on each window.

4.4.6A backtest checklist#

Point-in-time universe (including delisted names) and point-in-time fundamentals.
Signals computed on bars $\le t$ ; fills at the next open, or at a modelled price within the next bar; stops filled at $\min(O,\text{stop})$ on gaps.
Indian costs per side from M01 (STT, exchange, stamp, GST, DP) plus slippage, with a sensitivity table at 2× costs.
Turnover, breakeven cost, exposure (fraction of time invested), and number of trades reported.
The future-perturbation test passes.
The full count of trials is recorded in a research log (4.6.7).
Error bars on the Sharpe ratio (4.5) and a deflated Sharpe ratio (4.6.3).