The Politics of Prediction

Nate Silver is a Wunderkind. Not yet 35, Silver has already developed the leading statistical tool for assessing prospective professional baseball players, been consulted by the 2008 Obama presidential campaign to help assess the implications of opinion polls for the election’s outcome, writes a widely read New York Times blog and in 2009 was named by Time magazine as one of the 100 most influential people in the world. Along the way, and apparently in his spare time, Silver made more than $400,000 playing professional poker (before losing about a third of it and moving on). Silver’s initial foray into professional poker turned a $100 initial stake into $15,000, which was apparently enough to convince Silver to quit his day job, which has been to the benefit of the rest of us. Silver adds to this impressive list of accomplishments with the publication next week of his first book The Signal and the Noise: Why Most Predictions Fail – But Some Don't (Penguin USA).

This generally well-researched and ambitious book covers a lot of ground. It describes Silver’s evolution as a Fox (to use Philip Tetlock’s terminology favored by Silver) who is “tolerant of nuance, uncertainty, complexity, and dissenting opinion.” Silver discusses prediction in sports, geosciences, economics, politics and health. He clearly has done lots of reading in the academic literature. In most of these chapters he has tracked down and spoken to the characters that show up in the book -- a nice touch for a work that combines analysis with a bit of reporting. I’m not one of those characters, but I did engage in a long phone conversation with Silver while he was researching the book to discuss various aspects of prediction. In our discussion, I was impressed by his thoroughness and attention to detail, and have been looking forward to the final product.

Silver’s book is at its best when he writes about the things he knows best. That is to say, when he is writing about sports, political predictions and poker, Silver’s book is original and insightful. However, when he writes about weather, climate, earthquakes, the stock market and terrorism, the book doesn’t quite reach this high standard, and sometimes the discussion of these complex topics seems a bit cursory or hurried, which is decidedly contrary to Silver’s approach to analysis for which he has become well known.

Consequently, there is a brilliant book here, but it is a subset of the volume that was produced. The brilliant book shares Silver’s personal experiences in building the PECOTA system to evaluate baseball prospects a la Moneyball, his remarkable successes (and lessons learned) as a professional poker player and his experiences developing the FiveThirtyEight election forecasting system now hosted at the New York Times. I got my PhD in political science, and have long been a critic of the silly election forecasting methods proposed by academics. Silver’s work is much more solid than that of the academics, and probably the best of its kind. Silver also writes insightfully about chess, conditional probabilities (and the importance of Bayes theorem) and the difficulty of distinguishing predictive skill from pure chance.

The book’s thirteen chapters appear to have been written such that with few edits each might appear as stand-alone essays in places like the New York Times magazine (indeed, the chapter on weather forecasting recently was the basis for exactly such an essay). This makes the book easy to read but also makes it difficult for Silver to sustain a consistent argument throughout. For instance, after introducing the reader early on to Frank Knight’s definitions of “risk” (where you can readily quantify probabilities, such as in the roll of a die) and “uncertainty” (where you can’t readily quantify probabilities, such as the expected price of copper in 2057), Silver then leaves this distinction behind and subsequently throughout the book uses the word “uncertainty” to mean what Knight called “risk.”

Silver strikes a balanced tone throughout but he explicitly expresses a “fealty” to the integrity of forecasting as forecasting. He thinks that forecasters should report the numbers and not shade their work in one direction or another according to factors external to the forecasting. That’s because he is interested in finding the “signal” in messy and complex data, where the “signal” refers to the “truth.” This is an admirable objective, of course, but there are many times where the “signal” to be found in data is not simply about “truth.” In other words, forecasters do more than just forecast, they serve as political advocates, entertainers, reasons for rationalization and means of avoiding accountability, among other roles.

For instance, Silver explains that The Weather Channel introduces a bias into its forecasts such that low probabilities of rain are routinely increased, explaining, “If it rains when it isn’t supposed to, they curse the weatherman for ruining their picnic, whereas an unexpectedly sunny day is taken as a serendipitous bonus.” Silver critiques this as “not good science” but doesn’t really follow through on the implications – The Weather Channel is providing not just science, but entertainment and even an opportunity for satisfaction, a fact readily acknowledged by The Weather Channel executive interviewed by Silver. Of course, Silver is absolutely correct that good decision making depends upon resolving the signal from the noise, but sometimes, what constitutes the signal is not science alone. That is where prediction gets complicated in ways numbers cannot begin to describe.

Consider as well the lawsuit that was brought against government scientists and officials in L’Aquila, Italy following the devastating 2009 earthquake. Silver calls the trial “ridiculous,” even though the government scientists and officials had no scientific basis for their pronouncements. It turned out that they were engaged in a very public battle with an amateur earthquake forecaster and overstated the amount of certainty in their belief that an earthquake would not be forthcoming – never a good idea in an earthquake-prone region. How the lawsuit turns out will be based on Italian law, but the government officials clearly erred in their statements. As Silver explains, “Politics, broadly defined, can get in the way.”

Probably wisely for an up and coming public intellectual, Silver does not follow through on the various ways that politics actually does get in the way of good forecasts and good decisions. He engages some of these issues in his discussion of the housing bubble but avoids essentially all of them in his discussion of climate change.

I would have liked to see Silver engage more on the issue of prediction evaluation. Though the notion comes up a few times in the book, Silver does not really engage the technical concept of “skill” in forecasting as a quantitative measurement of a prediction’s improvement over a naïve baseline expectation. The phenomena that he is most interested in are perfect subjects for such evaluations. Another issue given less attention than I expected is ignorance or “unknown unknowns.” Silver explains that “in general our predictive errors come in thinking that there is more certainty in the world than there really is” but such Knightian uncertainty isn’t as systematically dealt with as it might have been.

All that said, Silver has written an enjoyable and ambitious book. His fans, me included, have come to expect rigor and detail, and this volume does not disappoint. Silver’s first book has a lot of signal in it – it’s a welcome tonic to the often over-exalted role of prediction in modern society.