Part VI · Prediction and synthesis · M13 · week 7 · 16 h

Forecasting and market prediction

What prediction can mean in near-efficient markets, ARMA, out-of-sample R², GARCH by maximum likelihood, regime switching, machine-learning pitfalls, proper scoring rules and the forecast journal.

Start reading · Learning outcomes

Builds on M02 Statistics of returnsM04 Systematic research

Prerequisites M02 (returns, autocorrelation, volatility clustering), M04 (backtesting, Sharpe-ratio statistics, multiple testing). M03 and M09 help for the economic value of forecasts. Maths: conditional expectation and the tower property, maximum likelihood, finite Markov chains, convexity and Jensen's inequality. Time: ~16 hours (notes 9 h, problems 4 h, code 3 h). Primary texts: J. D. Hamilton, Time Series Analysis (Princeton, 1994), ch. 3–5, 21, 22; R. Tsay, Analysis of Financial Time Series (3rd ed., Wiley, 2010), ch. 2–4; M. López de Prado, Advances in Financial Machine Learning (Wiley, 2018), ch. 3, 4, 7; P. Tetlock and D. Gardner, Superforecasting (Crown, 2015); T. Gneiting and A. Raftery, "Strictly proper scoring rules, prediction, and estimation", JASA (2007).

Learning outcomes

State the three forms of market efficiency and the Grossman–Stiglitz argument precisely, and say which kind of forecast (mean, variance, event probability) each one leaves room for.
Derive the stationarity conditions, autocorrelations, Yule–Walker equations and $h$ -step forecasts of AR( $p$ ) models, and compute the population $R^2$ of an AR forecast.
Define the out-of-sample $R^2$ of Campbell and Thompson, derive the Sharpe ratio a mean-variance investor gains from a predictor with a given $R^2$ , and derive the Stambaugh bias of predictive regressions.
Derive the unconditional variance, ARMA representation, kurtosis, half-life, multi-step forecasts, log-likelihood and score of GARCH(1,1), and estimate it by maximum likelihood from scratch.
Derive the Hamilton filter and the Kim smoother for a two-state Markov-switching model, and explain why smoothed probabilities are look-ahead in a backtest.
Build purged k-fold cross-validation with an embargo, triple-barrier labels and meta-labels, and quantify the leakage that ordinary k-fold creates with overlapping labels.
Prove that the Brier and logarithmic scores are strictly proper, prove the Murphy decomposition of the Brier score, and score your own probabilistic forecasts in a journal.

Sections ≈ 2 h 45 min of study

Foundation 3 sections · ≈ 35 min
Core 7 sections · ≈ 1 h 30 min
Advanced 3 sections · ≈ 40 min

Learning outcomes Foundation 5′
13.1 What "prediction" can mean in a near-efficient market Core 15′
13.2 Linear time-series models: ARMA Advanced 15′
13.3 Return predictability and out-of-sample $R^2$ Core 20′
Figure Out-of-sample R² when predictability is real but small
13.4 GARCH(1,1): forecasting volatility Core 20′
Figure GARCH(1,1) forecast term structure and cone
Lab GARCH(1,1) on the Nifty: fit, conditional volatility and forecast cone
13.5 Regime switching: the Hamilton filter Advanced 10′
Figure Hamilton filter on simulated regimes: filtered versus smoothed
13.6 Machine-learning pitfalls: leakage, purged cross-validation, labels Advanced 15′
13.7 Proper scoring rules and calibration Core 15′
Figure Reliability diagram and the Murphy decomposition
13.8 Judgmental forecasting and the forecast journal Foundation 15′
13.9 Case studies Core 10′
Summary of key results Foundation 15′
Readings and resources Core 5′
Facts that change Core 5′