Prediction markets have to occasionally “get it wrong” to be calibrated
Subscribe to Decision Science News by Email (one email per week, easy unsubscribe)
PREDICTION MARKETS NOT AS BAD AS THEY APPEAR
Two recent events in the UK made it look like prediction markets’ predictions aren’t worth much.
The soccer team Leicester City won the premiere league title despite the markets putting the odds of them doing so at 5,000 to 1 (.02%).
Last week, people in the UK, voted to leave the European Union. A few hours before it was sure they would exit, a prediction market put their probability of leaving at 10%. See the figure above from PredictIt. X axis is roughly time before the outcome was certain. Y axis can be interpreted as probability of exit (70 cents = 70%). It jumped from 10% to 90% in just five hours.
Analysts like to “explain” market results, coming up with a reason why an event was a failure of the prediction market. For instance, in the two events above, the Wall Street Journal, perhaps correctly, claims the bets were unduly influenced by London bettors. Through big London bets the odds moved to reflect what Londoners believe instead of the sentiment of the crowd. In predicting a Brexit, the sentiment of the crowd is exactly what you want.
Whenever the prediction market is far on the wrong side of 50%, explanations will arise as to why the prediction market was wrong. Let’s take a step back here.
A desirable property of a prediction market is that it is calibrated. To be calibrated, events that it predicts to be 90% likely should occur 90% of the time. Events that it predicts to be 10% likely should occur 10% of the time.
If events that it predicts to be 10% likely (e.g. Brexit) occur 0% of the time, the prediction market has a problem. It is over-estimating the chances.
Looking at the calibration of prediction markets across many events, one will see that they are typically very well calibrated. Take, for instance that Hypermind prediction market. The figure below shows close to 500 events that it predicted. If the market were perfectly calibrated, the points would fall along the diagonal line (i.e., 10% likely events would happen 10% of the time, 20% likely events would happen 20% of the time, and so on).
It’s very well calibrated. Here’s a similar chart for U.S. primaries at PredictWise.
So the next time a market “misses” a 10% prediction, let us keep in mind that it needs to miss 10% of those predictions to stay calibrated. As the Émile Servan-Schreiber mentions in this post, “It is perhaps a bit ironic to note that the data from the Brexit question slightly improved [the prediction market’s] overall calibration. It is as if the occurence of an unlikely event was long overdue in order to better match predicted probabilities to observed outcomes!”
You may wish that that prediction markets had prefect calibration and predicted only 0% or 100%. We do, too. But the world is an imperfect place.
Dan,
Sure, but the flip side of this reasoning is that betting markets aren’t necessarily calibrated. There are lots of reasons to think that those Leicester City odds were pretty arbitrary and that, even prospectively, the 5000-1 odds were too strong, and that the odds moved too slowly as information came in. (I had a couple posts on this recently.)
So, sure, a single failed prediction doesn’t mean the odds were uncalibratred. But, conversely, I think you should be wary of using the difficulty of evaluating single predictions as a rationale for assuming the odds were correct.
June 29, 2016 @ 10:54 am
OK, but note that my point is not that the odds on a single event were correct. I am wagging my finger at those who are saying that a missed prediction here or there means that prediction markets don’t make accurate predictions. On the whole, they do, and for every value of p.
June 29, 2016 @ 11:12 am