4.1 Technical analysis as statistical hypotheses

Motivation

Technical analysis is the study of price and volume history to time trades. Most of it is stated in pictures and folklore. A researcher's first job is to turn each picture into a function of data and each piece of folklore into a hypothesis that data can reject. Once you do this, two things become obvious: most indicators are reweightings of past returns, and an indicator "works" only if returns are autocorrelated or if the signal sorts assets with different mean returns.

4.1.1Setup#

Let $P_t$ be the close of a stock or index on trading day $t$ , adjusted for splits, bonuses and dividends (a corporate-action adjustment rescales past prices so that returns across the event are correct). Write $p_t=\ln P_t$ and the log return $r_t=p_t-p_{t-1}$ . Each day also has an open $O_t$ , high $H_t$ , low $L_t$ and traded volume $V_t$ (shares). Let $\mathcal F_t$ be the information available after the close of day $t$ : all bars up to and including $t$ .

Definition 4.1signal

A signal is a process $s_t$ that is $\mathcal F_t$ -measurable: its value on day $t$ is a function of bars dated $\le t$ only. A position rule maps the signal to a target weight $w_t$ (fraction of equity), which is first held from the open of day $t+1$ .

Definition 4.2trading hypothesis

For horizon $h\ge 1$ let $R_{t,t+h}=\sum_{j=1}^{h} r_{t+j}$ be the forward return. A rule built on a binary signal $s_t\in\{0,1\}$ has predictive content if

\Delta_h \;=\; \mathbb E[R_{t,t+h}\mid s_t=1]-\mathbb E[R_{t,t+h}\mid s_t=0]\neq 0 .

It is economically useful only if $\Delta_h$ exceeds the round-trip cost of acting on the signal (4.4.4). The null hypothesis of the weak-form efficient market (prices follow a martingale with constant drift; see M13 · 13.1) implies $\Delta_h=0$ for every signal.

Two points follow at once. First, the conditioning must be on $\mathcal F_t$ , never on anything dated later; the most common bug in retail backtests is to compute $s_t$ with the close of day $t$ and also earn the return of day $t$ . Second, the hypothesis is about a conditional mean. A chart pattern that "looks bullish" is not a hypothesis until it says which mean, over which horizon, compared with what.

4.1.2Indicators as functions of the price series#

Definition 4.3core indicators

For a series $x$ and window $n$ :

Simple moving average: $\mathrm{SMA}_n(x)_t=\frac1n\sum_{k=0}^{n-1}x_{t-k}$ .
Exponential moving average: $\mathrm{EMA}_n(x)_t=\alpha x_t+(1-\alpha)\mathrm{EMA}_n(x)_{t-1}$ with $\alpha=2/(n+1)$ .
Trend: the sign of a smoothed slope. Common choices are $\operatorname{sign}(p_t-p_{t-n})$ (time-series momentum), $\operatorname{sign}(P_t-\mathrm{SMA}_n(P)_t)$ , or $\operatorname{sign}(\mathrm{SMA}_m-\mathrm{SMA}_n)$ for $m<n$ .
Resistance and support (Donchian channel): $U^n_t=\max(H_{t-n},\dots,H_{t-1})$ and $D^n_t=\min(L_{t-n},\dots,L_{t-1})$ . A resistance level is a price where selling has previously stopped advances; the Donchian channel is the simplest objective version. Note that the window excludes day $t$ .
Breakout: $B^n_t=\mathbf 1\{P_t>U^n_t\}$ , the close exceeds the highest high of the previous $n$ days.
Relative volume: $\mathrm{RVOL}^n_t=V_t\big/\frac1n\sum_{k=1}^{n}V_{t-k}$ . Again the average excludes today.
Relative strength (RS) against a benchmark index $I_t$ (for example Nifty 500): $\mathrm{RS}_t=P_t/I_t$ . The Mansfield version is $100\,(\mathrm{RS}_t/\mathrm{SMA}_n(\mathrm{RS})_t-1)$ , positive when the stock is outperforming its own recent relative trend. A cross-sectional RS rank is the percentile of a stock's trailing 12-month return within a universe; IBD's proprietary "RS rating" is of this type.

Swing traders also use pivot support and resistance: a swing high at day $u$ is a bar with $H_u>H_{u\pm j}$ for $j=1..k$ . A swing high at $u$ is only known at day $u+k$ . Code that marks it at $u$ is using the future. This is the single most common source of look-ahead in pattern-recognition code.

Definition 4.4Wilder's RSI

The relative strength index (unrelated to relative strength above) with period $n$ (usually 14) is built from price changes $d_t=P_t-P_{t-1}$ . Let $W$ denote Wilder's smoother: $W_n=\frac1n\sum_{t=1}^{n}x_t$ , then $W_t=\frac{(n-1)W_{t-1}+x_t}{n}$ , an EMA with $\alpha=1/n$ . Define $U_t=W(d^+)_t$ , $D_t=W(d^-)_t$ with $d^+=\max(d,0)$ , $d^-=\max(-d,0)$ , and

\mathrm{RSI}_t=100-\frac{100}{1+U_t/D_t}=100\,\frac{U_t}{U_t+D_t}.

4.1.3What moving averages and RSI really are#

Proposition 4.1moving-average rules are momentum rules

Apply moving averages to log prices. For $1\le m<n$ ,

\mathrm{SMA}_m(p)_t-\mathrm{SMA}_n(p)_t=\sum_{i=0}^{n-2} w_i\, r_{t-i},\qquad w_i=\frac{n-1-i}{n}-\frac{(m-1-i)^+}{m}\;\ge 0, \tag{4.1}

with $\sum_i w_i=(n-m)/2$ . The weights rise linearly for $i<m-1$ , peak at $i=m-1$ , and fall linearly to zero at $i=n-1$ . The rule "price above its $n$ -day average" is the case $m=1$ , with linearly declining weights $w_i=(n-1-i)/n$ .

Proof

Write $p_{t-j}=p_t-\sum_{i=0}^{j-1}r_{t-i}$ . Then

\begin{aligned} \mathrm{SMA}_n(p)_t&=\frac1n\sum_{j=0}^{n-1}\Big(p_t-\sum_{i=0}^{j-1}r_{t-i}\Big)=p_t-\frac1n\sum_{i=0}^{n-2}\#\{j: i<j\le n-1\}\,r_{t-i}\\ &=p_t-\sum_{i=0}^{n-2}\frac{n-1-i}{n}r_{t-i}. \end{aligned}

The same with $m$ gives $p_t-\sum_{i=0}^{m-2}\frac{m-1-i}{m}r_{t-i}$ . Subtract: $p_t$ cancels and (4.1) follows. For $i\le m-2$ , $w_i=\big(1-\frac{i+1}{n}\big)-\big(1-\frac{i+1}{m}\big)=(i+1)\big(\frac1m-\frac1n\big)>0$ , increasing in $i$ . For $m-1\le i\le n-2$ , $w_i=(n-1-i)/n>0$ , decreasing. Finally $\sum_{i=0}^{n-2}\frac{n-1-i}{n}=\frac{(n-1)n/2}{n}=\frac{n-1}{2}$ , and likewise $\frac{m-1}{2}$ , so $\sum w_i=\frac{n-m}{2}$ .∎

So the 50/200-day "golden cross" is a bet on a triangular-weighted average of the last 199 daily returns, with peak weight on the return 49 days ago. Every moving-average rule (SMA, EMA, MACD) is a linear filter of past returns followed by a nonlinearity (the sign). Its value depends entirely on the autocorrelation structure of returns.

Proposition 4.2expected return of a linear trend rule

Let $r_t$ be covariance-stationary with mean $\mu$ and autocovariances $\gamma_k=\operatorname{Cov}(r_t,r_{t-k})$ . Take the position $x_t=\sum_{i=0}^{L-1}w_i r_{t-i}$ , held over day $t+1$ . Then

\mathbb E[x_t r_{t+1}]=\mu^2\sum_{i=0}^{L-1}w_i+\sum_{i=0}^{L-1}w_i\gamma_{i+1}. \tag{4.2}

Proof

$\mathbb E[r_{t-i}r_{t+1}]=\operatorname{Cov}(r_{t-i},r_{t+1})+\mu^2=\gamma_{i+1}+\mu^2$ . Sum with weights.∎

The first term is not timing skill: if $\mu>0$ , a rule with positive weights is long on average and collects part of the drift. The second term is the only source of timing profit, and it is a weighted sum of autocovariances at lags $1..L$ . Under a random walk ( $\gamma_k=0$ ) every MA rule earns only a diluted buy-and-hold return, before costs. The sign function in practical rules changes the constants but not the conclusion; Problem 4.3 asks you to show that for Gaussian returns the sign rule's timing term is still a linear combination of the $\gamma_k$ .

Worked example 4.1a one-day momentum rule on an index

Suppose Nifty 50 daily returns have $\sigma=1\%$ , $\mu=0.05\%$ and first-order autocorrelation $\rho_1=0.05$ , with $\gamma_k=0$ for $k\ge2$ . These are illustrative numbers of the right order, not estimates; M02 shows how to estimate them. Use the unit-volatility position $x_t=r_t/\sigma$ (in multiples of capital). By (4.2) with $L=1$ , $w_0=1/\sigma$ :

\mathbb E[x_tr_{t+1}]=\frac{\mu^2+\rho_1\sigma^2}{\sigma}=\frac{2.5\times10^{-7}+5\times10^{-6}}{0.01}=5.25\times10^{-4}\ \text{per day}.

Its daily volatility is about $\sigma\cdot 1=1\%$ , so the gross daily Sharpe ratio is $0.0525$ , about $0.83$ annualised. Now the cost. The position changes each day by $|x_t-x_{t-1}|\approx|Z_t-Z_{t-1}|$ with $Z$ iid standard normal, whose mean is $\mathbb E|N(0,2)|=2/\sqrt\pi\approx1.128$ times capital. Delivery-equity costs at a discount broker are about 11.9 bp per side (4.4.4). Daily cost $\approx1.128\times11.9=13.4$ bp, against a gross edge of 5.25 bp. The net expected return is about $-8$ bp a day. The predictability is real in this example and the strategy still loses money. Index futures are cheaper (M01), which is why fast trend rules are run on futures, not on delivery equity.

Proposition 4.3RSI is a normalised momentum oscillator

With $W$ Wilder's smoother,

\mathrm{RSI}_t=50+50\,\frac{W(d)_t}{W(|d|)_t}. \tag{4.3}

Proof

$W$ is linear, $d=d^+-d^-$ and $|d|=d^++d^-$ . So $W(d)=U-D$ and $W(|d|)=U+D$ . Then $50+50\frac{U-D}{U+D}=\frac{50(U+D)+50(U-D)}{U+D}=100\frac{U}{U+D}$ .∎

RSI is therefore an exponentially weighted mean price change divided by an exponentially weighted mean absolute change, mapped to $[0,100]$ . "RSI above 70 is overbought, sell" is a short-term reversal hypothesis: it says that a large normalised recent gain predicts a negative return. "RSI above 50 and rising" is a momentum hypothesis. The same indicator supports opposite rules, and both cannot be right on the same horizon. The data must decide, and 4.3 summarises what the data say. The code checks (4.3) to machine precision.

4.1.4Testing one rule honestly#

The simplest test of Definition 4.2 is an event study (M12 builds this in full). Collect the $N$ days with $s_t=1$ , compute forward returns $R_{t,t+h}$ in excess of the index, and form $t=\bar R/(\hat\sigma_R/\sqrt N)$ . Two traps make this $t$ -statistic too large.

Proposition 4.4overlapping horizons inflate $t$ -statistics

Let daily excess returns be iid with variance $\sigma^2$ , and let $Y_t=\sum_{j=1}^{h}r_{t+j}$ be sampled on $N$ consecutive days ( $N\gg h$ ). Then

\operatorname{Var}(\bar Y)\approx\frac{h^2\sigma^2}{N},\qquad\text{while the naive formula gives }\frac{\operatorname{Var}(Y_t)}{N}=\frac{h\sigma^2}{N}. \tag{4.4}

The naive standard error is too small by a factor $\sqrt h$ .

Proof

$\operatorname{Cov}(Y_t,Y_{t+k})=(h-|k|)\sigma^2$ for $|k|<h$ and zero otherwise, since the two windows share $h-|k|$ days. Hence $\operatorname{Var}(\bar Y)=\frac1{N^2}\sum_{t,s}\operatorname{Cov}(Y_t,Y_s)\approx\frac1N\sum_{k=-(h-1)}^{h-1}(h-|k|)\sigma^2=\frac{h^2\sigma^2}{N}$ , using $\sum_{k=-(h-1)}^{h-1}(h-|k|)=h^2$ .∎

Use non-overlapping events, or Newey–West (heteroskedasticity- and autocorrelation-consistent) standard errors with at least $h-1$ lags. The second trap is cross-sectional: breakouts cluster in time (many stocks break out in the same strong week), so events are not independent across stocks. Cluster standard errors by date, or test the rule as a calendar-time portfolio.

Worked example 4.2a 20-day breakout in Nifty 500 stocks

Suppose a screen finds 60 non-overlapping 20-day breakouts on above-average volume, with mean 20-day return in excess of the Nifty 500 of 1.2% and standard deviation 8%. Then $t=0.012/(0.08/\sqrt{60})=1.16$ . The evidence is consistent with no edge. To reach $t=2$ at the same mean and dispersion you need $N=(2\times0.08/0.012)^2\approx178$ independent events. If the 60 events had come from overlapping daily samples, the naive $t$ could easily exceed 3, which would be spurious.

Practitioner notes

The canonical academic test of chart rules is Brock, Lakonishok and LeBaron (1992). They tested moving-average and trading-range-break rules on the Dow Jones index from 1897 to 1986, reported strong support for the rules, and found returns after buy signals higher and less volatile than after sell signals. They used bootstraps under random-walk, AR(1), GARCH-M and EGARCH nulls. Case study 4.7.3 discusses what happened when Sullivan, Timmermann and White (1999) asked how many rules had been tried before those 26 were chosen.
Indicator parameters (14 for RSI, 20/50/200 for moving averages) are conventions, not optimal values. Every parameter you vary is another trial, and 4.6 charges for it.
Prefer rules on log prices or returns. A rule on price levels (a fixed ₹ stop, "round number" resistance at ₹1,000) changes meaning when the series is back-adjusted for a later bonus issue. That is a look-ahead in disguise (4.4.2).