13.7 Proper scoring rules and calibration

Setup#

A binary event $Y\in\{0,1\}$; a forecast $p\in[0,1]$; a scoring rule $S(p,y)$, here a loss (lower is better). If the forecaster's true belief is $q = \Pr(Y=1)$, the expected score of reporting $p$ is

$S$ is proper if $\bar S(q;q)\le\bar S(p;q)$ for all $p,q$, and strictly proper if equality implies $p=q$ (Gneiting and Raftery 2007).

The Brier score (Brier 1950) over $N$ forecasts and the log score are

Proof

Expand: $q(1-2p+p^2) + (1-q)p^2 = q - 2pq + p^2 = (p-q)^2 + q - q^2$. The second term does not depend on $p$, and the first is uniquely minimised at $p=q$.∎

Proof

$-q\ln p - (1-q)\ln(1-p) = [-q\ln q - (1-q)\ln(1-q)] + [q\ln(q/p) + (1-q)\ln((1-q)/(1-p))]$, which is (13.35). Gibbs' inequality: since $\ln$ is strictly concave, Jensen gives $-D_{KL}(q\|p) = \sum_y q_y\ln(p_y/q_y) \le \ln\sum_y q_y(p_y/q_y) = \ln 1 = 0$, with equality iff $p_y/q_y$ is constant, i.e. $p=q$.∎

Counter-example (an improper rule) The absolute error $|p-y|$ has expected value $q(1-p) + (1-q)p = q + p(1-2q)$, which is linear in $p$. For $q>1/2$ it is minimised at $p=1$, for $q<1/2$ at $p=0$. A forecaster scored by absolute error should report only 0 or 1, destroying all probabilistic information. The test test_strict_propriety_on_a_grid checks all three behaviours numerically.

The two proper rules differ in their tails. Brier is bounded: the worst possible single score is 1. The log score is unbounded: a forecast of $p=0$ for an event that happens scores $+\infty$. That is a feature. It encodes the view that certainty about an uncertain event is an unbounded error, and it is why the 100% forecast in case study 13.9.1 is indefensible regardless of how the rest of a track record looks.

Calibration, resolution and the Murphy decomposition#

A forecaster is calibrated if $\Pr(Y=1\mid p = f) = f$ for every forecast value $f$ they use. Calibration alone is cheap: always forecasting the base rate $\bar y$ is perfectly calibrated and useless. The other ingredient is resolution: how much the outcome frequency varies across the forecaster's categories.

Group forecasts into $K$ bins with $n_k$ forecasts each, all equal to $f_k$ within bin $k$. Let $\bar y_k$ be the observed frequency in bin $k$ and $\bar y$ the overall frequency.

Proof

Within bin $k$, add and subtract $\bar y_k$: $\sum_{i\in k}(f_k - y_i)^2 = n_k(f_k-\bar y_k)^2 + 2(f_k-\bar y_k)\sum_{i\in k}(\bar y_k - y_i) + \sum_{i\in k}(y_i-\bar y_k)^2$. The middle sum is zero by definition of $\bar y_k$. Because $y_i\in\{0,1\}$, $y_i^2 = y_i$ and $\sum_{i\in k}(y_i-\bar y_k)^2 = n_k\bar y_k - n_k\bar y_k^2 = n_k\bar y_k(1-\bar y_k)$. So

$$\mathrm{BS} = \frac1N\sum_k n_k(f_k-\bar y_k)^2 + \frac1N\sum_k n_k\bar y_k(1-\bar y_k).$$

For the second term, use $\sum_k n_k\bar y_k = N\bar y$ and $\sum_k n_k\bar y_k^2 = \sum_k n_k(\bar y_k-\bar y)^2 + N\bar y^2$ (the cross term $2\bar y\sum_k n_k(\bar y_k - \bar y)$ vanishes). Then $\frac1N\sum_k n_k\bar y_k(1-\bar y_k) = \bar y - \frac1N\sum_k n_k(\bar y_k - \bar y)^2 - \bar y^2 = \mathrm{UNC} - \mathrm{RES}$.∎

Uncertainty depends only on the events, not the forecaster. A good forecaster has small REL and large RES. If forecasts within a bin are not all equal (equal-width bins on continuous forecasts), two extra within-bin terms appear and (13.36) holds only approximately; the code's bins=None option bins by distinct forecast value so the identity is exact.

The Brier skill score against a reference forecast is $\mathrm{BSS} = 1 - \mathrm{BS}/\mathrm{BS}_{\text{ref}}$. Use two references: the coin flip (0.5, $\mathrm{BS}=0.25$) and climatology (the base rate, $\mathrm{BS} = \bar y(1-\bar y)$).

Worked example 1: five forecasts#

Forecasts $p = (0.9, 0.2, 0.6, 0.7, 0.4)$, outcomes $y = (1,0,1,1,0)$. Squared errors: $0.01, 0.04, 0.16, 0.09, 0.16$; sum 0.46; $\mathrm{BS} = 0.092$. Log score: $-\frac15[\ln0.9 + \ln0.8 + \ln0.6 + \ln0.7 + \ln0.6] = -\frac15[-0.10536 - 0.22314 - 0.51083 - 0.35667 - 0.51083] = 0.341366$. Murphy with one bin per distinct forecast: each bin has one forecast and $\bar y_k\in\{0,1\}$, so REL = BS = 0.092, RES = $\frac15\sum(\bar y_k - 0.6)^2 = \frac15(3\times0.16 + 2\times0.36) = 0.24$, UNC = $0.6\times0.4 = 0.24$. Check: $0.092 - 0.24 + 0.24 = 0.092$. With one forecast per bin, resolution equals uncertainty mechanically and reliability carries the whole score. The decomposition only becomes informative with many forecasts per bin, which is why the forecast journal (13.8) uses coarse probability levels.

Worked example 2: a forecast journal#

Ten weekly questions answered with probabilities from a five-level scale:

$p$	0.6	0.7	0.3	0.6	0.8	0.3	0.7	0.6	0.3	0.8
$y$	1	1	0	0	1	0	1	1	1	1

$\mathrm{BS} = 0.161$, $\mathrm{LS} = 0.501$. Base rate $\bar y = 0.7$, UNC $= 0.21$. Bins: $f=0.3$ ($n=3$, $\bar y = 1/3$), $f=0.6$ ($n=3$, $\bar y = 2/3$), $f=0.7$ ($n=2$, $\bar y=1$), $f=0.8$ ($n=2$, $\bar y=1$). REL $= \frac1{10}[3(0.0333)^2 + 3(0.0667)^2 + 2(0.3)^2 + 2(0.2)^2] = 0.0277$. RES $= \frac1{10}[3(0.3667)^2 + 3(0.0333)^2 + 2(0.3)^2 + 2(0.3)^2] = 0.0767$. Check: $0.0277 - 0.0767 + 0.21 = 0.161$. Skill: $\mathrm{BSS} = 0.356$ against a coin flip and $0.233$ against climatology. The diagnosis: the 0.7 and 0.8 forecasts were underconfident (all happened). Ten forecasts are, of course, far too few to conclude anything (13.8).