13.8 Judgmental forecasting and the forecast journal

What the tournaments found#

Tetlock's Expert Political Judgment (2005) scored a large set of long-range expert political forecasts and found that most experts did little better than simple baselines. IARPA's Aggregative Contingent Estimation (ACE) programme then ran a geopolitical forecasting tournament, which the Good Judgment Project (GJP), led by Tetlock and Mellers, entered. Mellers et al. (2014, Psychological Science 25(5):1106–1115) report on the first two years: the first year began with 2,246 participants, who forecast 199 geopolitical outcomes, scored by Brier score. Reported findings (APS summary of the paper): probability training beat scenario training, which beat no training; forecasters in teams beat those who saw crowd beliefs, who beat independent forecasters; and the best forecasters of year 1, placed in elite teams ("superforecasters"), were far more accurate than the other groups in year 2. A figure that superforecasters were about 30% more accurate than intelligence-community analysts with access to classified information is widely repeated (reported by press coverage and Good Judgment Inc.); I could not trace it to a primary peer-reviewed table, so treat it as indicative.

Practices that the evidence supports#

1. Start from a base rate (the outside view). Before thinking about this RBI meeting, ask how often the RBI has moved rates at a meeting in a comparable inflation and growth setting. Before forecasting "Nifty up this week", note what a simple model implies: with 12% annual drift and 15% annual volatility in a lognormal model, $\Pr(\text{up over a week}) = \Phi\big((0.12 - 0.15^2/2)/52 \,/\, (0.15/\sqrt{52})\big) = \Phi(0.1005) = 0.540$; over a month 0.583; over a year 0.766. A weekly direction forecast far from 0.54 needs strong evidence.
2. Decompose (Fermi). "Will MCX gold close more than 3% higher in a month?" MCX gold in rupees is roughly international gold in dollars times USD/INR (plus import duty and premia, M10). Decompose into the two drivers or use a volatility estimate: with 16% annual volatility and 8% drift, a lognormal model gives $\Pr(> +3\%) = 0.30$ and $\Pr(< -3\%) = 0.22$. Then adjust for what the model misses (scheduled events, duty changes).
3. Update in small steps, in odds.

Proof

Write Bayes' rule for $H$ and for $\neg H$; divide; $\Pr(E)$ cancels.∎

Example: prior $\Pr(\text{rate cut at the next RBI meeting}) = 0.30$ (odds 0.429). A CPI print you judge twice as likely under "cut" as under "no cut" (likelihood ratio 2) moves the odds to 0.857, probability 0.46. Evidence with likelihood ratio 0.5 moves it to 0.18. Superforecasters made many small updates rather than a few large ones. Thinking in likelihood ratios prevents both over-reaction to vivid news and anchoring on the prior.

4. Be granular. Distinguish 60% from 70%. Tetlock and Gardner (2015) report that rounding the best forecasters' probabilities to coarser categories made them less accurate.
5. Aggregate and track. Averaging independent forecasts reduces noise; keeping score is what makes improvement possible.

Designing your forecast journal#

Each entry is a row with: question ID, date asked, exact question, resolution criterion and data source, resolution date, probability, base rate used, one-paragraph rationale, update log (date, new probability, reason), outcome, Brier and log score. Rules:

• Questions must resolve unambiguously from a named source. Good: "Will the Nifty 50 close (NSE official close) on Tuesday 2026-12-01 be above its close on Tuesday 2026-11-24?" Bad: "Will markets be bullish next week?"
• Mix question types:
  • Direction: Nifty weekly close up or down; MCX gold monthly contract up or down over a month.
  • Range/volatility: "Will India VIX close above 15 on Friday?"; "Will Brent's front-month settlement move more than 10% over the next 21 trading days?" (A lognormal model with 35% volatility gives about 0.16 for each tail, so 0.32 in total, as a base rate.)
  • Events: RBI policy decision; a scheduled data release above or below consensus.
• Use a fixed, coarse probability scale (for example 0.1, 0.2, …, 0.9 plus 0.02 and 0.98, eleven levels; fewer if you forecast rarely) so that each bin fills up and the Murphy decomposition is exact.
• Score against two baselines: 0.5 and your own running base rate for that question type.
• Review every 50 resolved questions: calibration table, REL, RES, and a list of the worst log-score misses with what you knew at the time.

How many questions do you need? If the per-question difference between your Brier score and the baseline's has standard deviation $s$, then to detect a mean difference $d$ at the 5% level you need roughly

With $s = 0.15$ and $d = 0.02$ that is 216 questions; with $d = 0.05$, 35. A year of weekly questions on three markets gives about 150. Forecasting skill, like trading skill (M04 · 4.5), takes longer to establish than intuition suggests.