13.1 What "prediction" can mean in a near-efficient market

Motivation

Most market "predictions" you will read are point forecasts without a probability, a horizon or a scoring rule, so they cannot be wrong in any measurable sense. A trader needs the opposite: forecasts with stated probabilities that can be scored, aggregated and sized. The first question is what can be forecast at all when thousands of well-funded participants are trying to do the same.

Setup#

Let $P_t$ be a price, $r_{t+1} = \ln(P_{t+1}/P_t)$ the log return, and $\mathcal F_t$ an information set (a σ-algebra) available at time $t$ . Write $\mathbb E_t[\cdot] = \mathbb E[\cdot \mid \mathcal F_t]$ . Fama (1970) classified market efficiency by the information set:

Weak form: $\mathcal F_t$ = past prices and volumes. Technical rules (M04 · 4.1) should not earn abnormal returns.
Semi-strong form: $\mathcal F_t$ = all public information (filings, macro data, news). Fundamental analysis (M06) should not earn abnormal returns after costs.
Strong form: $\mathcal F_t$ includes private information. Even insiders cannot earn abnormal returns. This form is rejected by the existence of profitable insider trading, which is why insider trading is illegal (SEBI's Prohibition of Insider Trading Regulations, 2015).

"Abnormal" needs a model of normal returns $r^e_{t+1}$ (CAPM, a factor model, or simply a constant). Efficiency then says

\mathbb E_t\!\left[r_{t+1} - r^e_{t+1}\right] = 0, \tag{13.1}

so the abnormal return $\varepsilon_{t+1} = r_{t+1} - r^e_{t+1}$ is a martingale difference sequence (MDS) with respect to $\{\mathcal F_t\}$ . Any test of efficiency is a joint test of (13.1) and of the model for $r^e$ : this is Fama's joint-hypothesis problem. A "predictable" return may be a risk premium the model omitted, not an inefficiency.

Proposition 13.1an MDS cannot be forecast by $\mathcal F_t$

Let $\varepsilon_{t+1}$ satisfy $\mathbb E_t[\varepsilon_{t+1}] = 0$ and $\mathbb E[\varepsilon_{t+1}^2] < \infty$ . For every $\mathcal F_t$ -measurable forecast $f_t$ with $\mathbb E[f_t^2] < \infty$ : (a) $\operatorname{Cov}(f_t, \varepsilon_{t+1}) = 0$ ; (b) $\mathbb E[(\varepsilon_{t+1} - f_t)^2] = \mathbb E[\varepsilon_{t+1}^2] + \mathbb E[f_t^2] \ge \mathbb E[\varepsilon_{t+1}^2]$ , with equality iff $f_t = 0$ a.s.

Proof

(a) By the tower property, $\mathbb E[f_t \varepsilon_{t+1}] = \mathbb E[f_t\, \mathbb E_t[\varepsilon_{t+1}]] = 0$ , and $\mathbb E[\varepsilon_{t+1}] = 0$ , so the covariance is zero. (b) Expand: $\mathbb E[(\varepsilon - f)^2] = \mathbb E[\varepsilon^2] - 2\mathbb E[f\varepsilon] + \mathbb E[f^2]$ and use (a).∎

So in a world satisfying (13.1) every forecast of the abnormal return that is not identically zero has a larger mean squared error than the trivial forecast. Out-of-sample $R^2$ (defined in 13.3) is then negative in population. Note what the proposition does not say. It says nothing about $\mathbb E_t[\varepsilon_{t+1}^2]$ : volatility can be highly predictable in an efficient market. It says nothing about the predictability of $r^e_{t+1}$ itself, so time-varying risk premia are allowed. And it says nothing about events that are not returns (whether RBI cuts rates, whether Brent closes above $90).

The Grossman–Stiglitz paradox#

If prices reflected all information perfectly, nobody would pay to collect information, and then prices could not reflect it. Grossman and Stiglitz (1980) formalised this in a model with a risky asset whose payoff is $u = \theta + \epsilon$ , a costly signal $\theta$ (cost $c$ ), CARA investors with risk aversion $a$ , and noisy supply. A fraction $\lambda$ of investors buy the signal. The price is a noisy function of $\theta$ , so uninformed traders learn part of $\theta$ from it.

Proposition 13.2equilibrium information, Grossman–Stiglitz 1980, sketch

In the CARA-normal model, the ratio of an informed to an uninformed trader's expected utility is

\frac{\mathbb E[V_I]}{\mathbb E[V_U]} = e^{a c}\sqrt{\frac{\operatorname{Var}(u \mid \theta)}{\operatorname{Var}(u \mid P_\lambda)}}, \tag{13.2}

where $P_\lambda$ is the equilibrium price when a fraction $\lambda$ is informed. An interior equilibrium $\lambda^\* \in (0,1)$ sets this ratio to 1.

Sketch

With CARA utility and normal payoffs, a trader's maximised expected utility is $-\exp(-a W_0)\,\exp\!\big(-\tfrac{1}{2}\,\mathrm{SR}^2\big)$ in conditional terms, where SR is the conditional Sharpe ratio of the asset. Taking expectations over the conditioning variable gives a factor proportional to $\sqrt{\operatorname{Var}(u\mid\cdot)}$ . The informed trader pays $c$ , which multiplies utility by $e^{ac}$ (utilities are negative). Since $\operatorname{Var}(u \mid P_\lambda)$ falls as $\lambda$ rises (prices become more informative), the ratio (13.2) is monotone in $\lambda$ , and the equilibrium condition pins down $\lambda^\*$ . The full derivation is in Grossman and Stiglitz (1980, §I–II).∎

The trading lesson is the equilibrium condition itself: the gross return to being informed equals the cost of becoming informed. Markets are efficient to within the cost of information, execution and risk-bearing. Edges exist but are small, costly to find, and shrink when the cost of finding them falls (see M04 · 4.7 on post-publication decay).

Three different forecasting problems#

Target	Typical predictability	Why	Economic use
Conditional mean of returns, $\mathbb E_t[r_{t+1}]$	Out-of-sample $R^2$ around 0 to 1% (13.3)	Competition; Proposition 13.1	Market timing, signals, factor tilts
Conditional variance, $\mathbb E_t[r_{t+1}^2]$	High: volatility clustering (M02) gives persistence near 1	Volatility is not traded away by directional traders	Position sizing, VaR, option pricing (M09)
Probabilities of discrete events	Varies; scored with proper scoring rules (13.7)	Many events are not prices	Scenario trees (M12), hedging decisions

Practitioner notes

"The market is efficient" and "I can forecast volatility well" are compatible. Most of the forecasting a working desk does is of second moments, not first moments.
A stated edge should come with the cost of information that generated it. If the cost is near zero (a moving-average rule anyone can code), Proposition 13.2 says the gross edge should be near zero too.
The joint-hypothesis problem means a predictable return is not automatically a free lunch. Momentum crashes (M04 · 4.3) are a reminder that some predictable returns are compensation for rare, large losses.