14.3 Shrinkage covariance (Ledoit–Wolf) · M14 Portfolio and going live

Motivation

With $n$ assets and $T$ observations the sample covariance $S$ has $n(n+1)/2$ parameters. When $n$ is not small relative to $T$ , $S$ is noisy: its large eigenvalues are too large and its small eigenvalues too small, and if $n > T$ it is singular. Inverting it, as (14.1)–(14.3) do, magnifies the noise.

14.3.1The optimal shrinkage intensity#

Setup

Let $F$ be a structured target with few parameters (for example $mI$ with $m = \operatorname{tr}S/n$ , a constant-correlation matrix, or a one-factor model). Consider $\hat\Sigma(\delta) = \delta F + (1-\delta)S$ , $\delta \in [0,1]$ , with loss $\mathbb E\|\hat\Sigma(\delta) - \Sigma\|_F^2$ (Frobenius norm).

Proposition 14.6Optimal shrinkage intensity

If $F$ is non-random and $\mathbb ES = \Sigma$ , the loss is minimised at

\delta^* = \frac{\mathbb E\|S - \Sigma\|_F^2}{\mathbb E\|S - \Sigma\|_F^2 + \|\Sigma - F\|_F^2}, \tag{14.4}

the ratio of the estimation variance of $S$ to that variance plus the squared bias of $F$ .

Proof

The intuition is the bias–variance trade-off: shrink a lot when $S$ is noisy (few observations, many assets) or the target is close to the truth.

14.3.2The Ledoit–Wolf estimator#

Ledoit–Wolf $\delta^*$ involves the unknown $\Sigma$ . Ledoit & Wolf ("A Well-Conditioned Estimator for Large-Dimensional Covariance Matrices", Journal of Multivariate Analysis 88, 2004, 365–411) give consistent estimators of the numerator and denominator for the target $mI$ . With demeaned data $x_t$ and the normalised norm $\|M\|^2 = \operatorname{tr}(MM^\top)/n$ :

d^2 = \|S - mI\|^2, \quad \bar b^2 = \frac1{T^2}\sum_{t=1}^{T}\|x_tx_t^\top - S\|^2, \quad b^2 = \min(\bar b^2, d^2), \quad \hat\delta = b^2/d^2. \tag{14.5}

$\bar b^2$ estimates the variance of $S$ (the average squared deviation of single-observation outer products from their mean, divided by $T$ ), and $d^2$ estimates the total dispersion of $S$ around the target. Their companion paper for portfolios, "Honey, I Shrunk the Sample Covariance Matrix" (Journal of Portfolio Management 30(4), 2004), uses a constant-correlation target.

Worked examplecode test

True $\Sigma$ diagonal with variances from 0.5 to 2.0, $n = 30$ , $T = 60$ . Over 15 samples, the Ledoit–Wolf estimate has less than 70% of the Frobenius loss of $S$ , always has $\hat\delta \in [0, 1]$ , and always has a smaller condition number. A smaller condition number means the inverse in (14.1)–(14.3) amplifies errors less.

Practitioner notes

For an Indian equity portfolio of 20–40 stocks estimated on 2–3 years of daily data, shrinkage changes GMV weights materially. Shrinking $\mu$ is even more valuable: the James–Stein idea of pulling each sample mean toward a common mean is the same bias–variance argument applied to the first moment.