Research note

Superforecasting for traders

What Tetlock and Gardner's Superforecasting teaches traders: calibration, Brier scores, base rates, small updates, Fermi estimates, journals and teams.

13 min read

Short answer: Superforecasting by Philip Tetlock and Dan Gardner (Crown, 2015) argues that forecasting is a skill you can measure and improve. The people who did it best put numbers on their beliefs, started from base rates, updated often, broke hard questions into smaller ones and kept score. None of that gives a trader an edge by itself. What it gives you is a way to find out whether you have one.

The book reports on the Good Judgment Project, a research team led by Tetlock and Barbara Mellers that competed in a forecasting tournament run by the US intelligence research agency IARPA. This post uses the book's ideas, the published research behind them, and an unpublished summary of the book we worked from. The commentary on trading is ours.

What did the Good Judgment Project find?

IARPA's ACE program (Aggregative Contingent Estimation) set out to improve intelligence forecasts by eliciting, weighting and combining the judgments of many analysts, and testing those forecasts against real events. Five university-based research groups competed. Tetlock and Mellers' group recruited volunteers, asked them for numeric probabilities on geopolitical questions, and let them update as often as they wanted before each question closed.

Their 2014 paper in Psychological Science found three things that improved accuracy:

  • Training. A probability-reasoning module of about 45 minutes corrected some common biases and taught forecasters to use reference classes and to average estimates when several were available.
  • Teaming. Forecasters who worked in teams and shared rationales were more accurate than forecasters who worked alone.
  • Tracking. The top 2% of performers from the first year were placed together in elite teams.

All three improved both calibration and resolution, two terms that come up again below. A statistical aggregation method helped further, including a step that pushed the combined forecast further from 50% ("extremizing").

The follow-up in 2015, in Perspectives on Psychological Science, looked at those top performers, the superforecasters. People expected them to fall back toward average, since one good year can be luck. They didn't, two years in a row. The authors concluded that superforecasters are "partly discovered and partly created." They found support for four explanations working together: cognitive ability and style, task-specific skill, motivation and commitment, and the richer environment of elite teams.

Tetlock's earlier book, Expert Political Judgment (2005), is the bleak background. It collected expert political and economic forecasts over about 20 years and found that, on average, they were not much better than chance. It also found that eclectic, flexible thinkers ("foxes") predicted better than those committed to one big theory ("hedgehogs"). Anyone who reads market commentary will recognize the hedgehog.

What is calibration, and why should traders care?

You are calibrated when your stated probabilities match how often things happen. If you say 70% on a hundred different calls, about seventy should come true. If only fifty do, you are overconfident. If ninety do, you are underconfident and leaving information on the table.

The 2014 GJP paper measured calibration by sorting forecasts into narrow probability bins and comparing each bin with how often those events actually happened. Plot that as a curve and a perfectly calibrated forecaster sits on the diagonal. For forecasts above 50%, points below the line mean overconfidence.

Calibration matters for trading because position size is a statement of confidence. A trader who sizes every idea as if it were a 70% call, while being right about half the time, is overbetting on every trade. The book's commandment on this asks forecasters to balance prudence and decisiveness, to avoid both rushing to judgment and hanging around "maybe" for too long. In trading terms: don't size like a blowhard, and don't hedge every thesis down to nothing.

Resolution is the other half. A forecaster who says 50% on everything can be perfectly calibrated and useless. Resolution measures how far your forecasts move away from the base rate in the right direction. The best forecasters have both, and traders need both too: the confidence to size up when the evidence is strong, and the honesty to stay small when it isn't.

How does a Brier score work?

The Brier score comes from a 1950 paper by Glenn Brier of the US Weather Bureau in Monthly Weather Review. The tournament used the original version: for each possible outcome, take the squared difference between your forecast and what happened (1 if it happened, 0 if not), then add them up. A score of 0 is perfect and 2 is the worst possible.

The arithmetic on a yes-or-no question:

  • Say 70% and the event happens: (1 − 0.7)² + (0 − 0.3)² = 0.18.
  • Say 70% and it doesn't: (0 − 0.7)² + (1 − 0.3)² = 0.98.
  • Say 50% on everything: 0.5 every time, whatever happens.
  • Say 100% and be wrong: 2, the worst score.

Two things follow. First, a confident wrong call costs far more than a cautious one, because the penalty is squared. Second, the score is proper, which means you get the best expected score by reporting what you actually believe. You can't game it by shading toward 50% or by exaggerating.

One Brier score tells you little. The 2014 paper has a good example. Early in the tournament, a question asked whether there would be a deadly clash involving government forces in the South or East China Sea within four months. The best forecasters started near 20%, which the authors note reflects the base rate for such events over four-month periods, and lowered their estimates as time passed. Then one happened on December 11, 2011. Their score on that question was bad, but their reasoning was sound. Traders know the pattern well: a well-reasoned trade can lose and a careless one can win. Only many scored decisions separate skill from luck.

Why start with the base rate?

Kahneman and Lovallo's 1993 paper "Timid Choices and Bold Forecasts" describes two ways to forecast. The inside view focuses on the case at hand: the plan, the obstacles, the story. The outside view mostly ignores those details and asks how a class of similar cases has turned out. They argue that people tend to treat each problem as unique, and that the inside view produces overly optimistic forecasts.

The book turns this into a habit. Start from the outside view, then adjust. The question Tetlock and Gardner suggest asking is "How often do things of this sort happen in situations of this sort?"

For a trader, the base rate is the historical record of the setup you are about to take, not the story you've built around this particular chart. MRT's own research gives a few examples:

  • In our Darvas Box audit of S&P 500 stocks from 2010 to 2026, 72% of trades were stopped out by day 60 (/blog/darvas-box).
  • In our exit-model sweep on post-earnings-beat stocks with a 2×ATR stop and a 20-day hold, the per-trade win rate was between 37.4% and 43.1% depending on the profit-target setting, including no target at all. That panel is current S&P 500 members, so survivorship bias pushes the numbers up.
  • Our internal negative-findings log lists ideas that looked convincing and failed when tested. Read it as a base rate for how often a plausible signal turns out to be noise.

If a setup historically works about four times in ten, your starting point on the next one should be about four in ten. The chart in front of you has to earn any move above that.

Prediction markets give you a different kind of outside view for event questions. MRT's /predictions page shows curated Polymarket odds on crypto, the Fed and the economy, market-moving politics, and company and market questions, refreshed about every two minutes. A market price is not a fact. It reflects whoever is trading that contract. But it's a useful number to write your own forecast next to before you look at the news, and a useful check when your view is far from it.

How do you update without overreacting?

The book's guidance is to avoid both extremes: don't cling to a prior when the evidence turns, and don't swing wildly on one headline. Tetlock and Gardner compare updating to brushing and flossing. It's dull and it pays off over time. The 2015 paper backs this up: in the first year superforecasters averaged 2.77 forecasts per question against 1.47 for the comparison groups, and, citing earlier GJP work, the authors note that how often people updated was the strongest single behavioral predictor of accuracy.

For a trader this means stating ahead of time what would change your mind. Before entering, write down the evidence that would make you cut the position, and the evidence that would make you add to it. When a new data point arrives, ask how much more likely it is under your thesis than under the alternative. Most news is weakly informative and deserves a small adjustment. A few events really do break a thesis and deserve a big one. The skill is telling them apart, and a written plan makes it much harder to pretend afterward that a broken thesis was only "noise."

A stop-loss is a pre-committed update. It decides in advance at what point the evidence has gone against you, so you don't have to decide it while you're losing money.

What is Fermi-izing?

Enrico Fermi was known for rough estimates of things that seemed impossible to know, built by breaking the question into parts you can estimate. The book uses his name for the habit of splitting a hard forecast into smaller ones, separating what you can know from what you can't, and putting your assumptions where you can see them.

A trading thesis is usually several forecasts bundled together. "This stock will rerate after earnings" depends on whether results beat expectations, whether guidance holds, whether the market rewards beats in this sector right now, and whether the broader market cooperates over your holding period. Each piece can be estimated and checked on its own. Writing them down does two things. It shows which part of the thesis is doing most of the work, and it shows how quickly the joint probability shrinks when several things all have to go right.

Why keep a forecasting journal?

Every commandment in the book depends on feedback. Tetlock and Gardner compare it to learning to ride a bike: you can't get good by reading, and you need feedback clear enough to tell whether you're rolling along or crashing. Markets give fast feedback on price and slow, noisy feedback on whether you were right for the right reasons. A journal helps close that gap.

A useful trading journal records four things for every trade, written before the result is known:

  • the thesis in one or two sentences, with the base rate you started from
  • your probability that the thesis plays out over the stated horizon
  • the stop, which defines how much you are risking
  • what would make you exit early or add

MRT's trade journal (/journal) covers the risk and outcome side. You record entry, stop and exit, and each closed trade's R-multiple is measured against the stop as it stood at entry, so moving a stop later can't rewrite your history. The journal's expectancy panel then shows average R by trade source, so results are measured in units of planned risk. Its Notes field ("why you took it") is the place to write the thesis and your probability. MRT doesn't calculate a Brier score or a calibration curve from journal entries, so that part is a spreadsheet job for now. After 30 or 50 trades, group your probability calls into bins and check how often each bin came true. That's your own calibration curve.

The book also warns against learning too much from one result. Do a post-mortem on wins as well as losses, because a win can come from offsetting mistakes.

Do teams beat lone forecasters?

In the 2014 paper, forecasters working in teams were more accurate than those working independently, and more comments within a team went with better accuracy. The same paper notes that teams can also go wrong, through poor dynamics, social loafing or groupthink. The book's commandment on teams names three skills: stating the other side's case so well that they'd agree with it, asking precise questions until a vague claim becomes clear, and "learning to disagree without being disagreeable."

Most retail traders work alone. You can still borrow the parts that matter:

  • Before a trade, write the strongest case against it, as its best advocate would put it.
  • Ask a trading partner to grade your written thesis, not your conviction.
  • Average your own estimate with an independent one (a prediction market, a peer, a model) when you have it. The GJP training taught averaging for this reason.

Testing a strategy against honest benchmarks plays a similar role. In MRT's Strategy Lab (/strategy-lab), the honesty panel checks a backtest against buy-and-hold and against a shuffled version of the strategy's own signals with a fixed seed, then gives a Go or No-Go based on fixed checks. It's in-sample by default. Its optional walk-forward switch re-scores the chosen settings on calendar-year slices (or two halves for shorter histories), and only the first slice counts as in-sample. That's the backtest version of keeping score on questions you didn't get to see when you made up your mind.

What doesn't carry over from the tournament?

  • Tournament questions were well defined and resolved within a set time. Markets rarely give you a clean yes-or-no, and your trading questions overlap.
  • Superforecasters showed skill on mostly geopolitical questions. That is evidence that forecasting skill exists, not evidence that anyone can forecast prices well.
  • Calibration doesn't make money on its own. A perfectly calibrated forecast is worthless as a trade if the market price already reflects it. Edge is the gap between your well-calibrated probability and the price.
  • Rare, extreme events are the hardest to score. A handful of trades can't tell you whether your 5% calls are really 5% calls.

FAQ

What is a good Brier score?

It depends on how hard the questions are, so compare scores only on the same set of questions. On the original 0-to-2 scale, always saying 50% on yes-or-no questions scores 0.5, so a forecaster who can't beat that has shown nothing. A tougher benchmark is always forecasting the base rate for that kind of question.

Can I compute a Brier score for my trades?

Yes, if you write a probability before each trade and define clearly what counts as the thesis being right. Keep the outcome separate from profit and loss. A trade can make money while the thesis is wrong, and lose money while it's right.

What does "outside view first" mean in trading?

Start from how often your type of setup has worked historically, in a similar market, before you weigh the details of this particular chart or story. Then adjust, and keep the adjustment modest unless you have evidence your own record supports.

Are superforecasters just lucky?

The research says no, at least not mostly. Mellers and colleagues found that the top performers didn't fall back toward average in the following two years, which is what luck would predict. They pointed to a mix of ability and thinking style, task skill, motivation, and the effect of working in elite teams.

Sources

  • Philip E. Tetlock and Dan Gardner, Superforecasting: The Art and Science of Prediction, Crown, 2015 (publisher page, accessed 2026-10-17): https://www.penguinrandomhouse.com/books/227815/superforecasting-by-philip-e-tetlock-and-dan-gardner/
  • Good Judgment Inc., "Ten Commandments for Aspiring Superforecasters" (from the book's appendix), accessed 2026-10-17: https://goodjudgment.com/philip-tetlocks-10-commandments-of-superforecasting/
  • Glenn W. Brier, "Verification of Forecasts Expressed in Terms of Probability", Monthly Weather Review 78(1), 1950, pp. 1–3: https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_0001_vofeit_2_0_co_2.xml
  • Mellers et al., "Psychological Strategies for Winning a Geopolitical Forecasting Tournament", Psychological Science 25(5), May 2014: https://escholarship.org/uc/item/4rg4n9vr
  • Mellers et al., "Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions", Perspectives on Psychological Science 10(3), 2015, pp. 267–281: https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015---superforecasters.pdf
  • IARPA, ACE (Aggregative Contingent Estimation) program page, accessed 2026-10-17: https://www.iarpa.gov/research-programs/ace
  • Daniel Kahneman and Dan Lovallo, "Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking", Management Science 39(1), January 1993, pp. 17–31: https://doi.org/10.1287/mnsc.39.1.17
  • Philip E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know?, Princeton University Press, 2005 (new edition 2017): https://press.princeton.edu/books/paperback/9780691175973/expert-political-judgment
  • Superforecasting book breakdown (unpublished study summary, supplied to MRT and not publicly linked)
  • MacroMomentum Terminal, Darvas Box, backtested: /blog/darvas-box
  • MacroMomentum Terminal, exit-model sweep and negative-findings log (internal research, not publicly linked)
Educational only, not financial advice. Past and backtested results do not guarantee future returns. Do your own research or speak with a licensed professional before making investment decisions.

Not financial advice

Built for education. Backtests are historical; they do not predict future returns and are not a recommendation to buy or sell any security.

connecting…