The other important thing I’ll say is, if the Comey quote is true, then actually he needed to listen to good election forecasts that showed the number was more like 70 percent. So that becomes an argument for further forecasts.
Well, what is a “good” forecast? If we go back to 2016, as you say, Nate Silver’s forecast gave Trump a 30 percent chance of winning. Other models pegged Trump’s chances at more like 1 percent or low single digits. The sense is that, because Trump won, Nate Silver was, therefore, “right.” But of course, we can’t really say that. If you say something has a 1-in-100 chance of happening, and it happens, that could mean you underrated it, or it could just mean the 1-in-100 chance hit.
This is the problem with figuring out whether election forecasting models are tuned correctly to real-world events. Going back to 1940, we have only 20 presidential elections in our sample size. So there’s no real statistical justification for a precise probability here. 97 versus 96—it’s insanely hard with our limited test size to know whether these things are being calibrated correctly to 1 percent. This entire exercise is much more uncertain than the press, I think, leads the consumers of polls and forecasts to believe.
In your book, you talk about Franklin Roosevelt’s pollster, who was an early genius of polling—but even his career, eventually, went up in flames later on, right?
This guy, Emil Hurja, was Franklin Roosevelt’s pollster and election forecaster extraordinaire. He devised the first kind of aggregate of polls, the first tracking poll. A really fascinating character in the story of polling. He’s crazy accurate at first. In 1932 he predicts that Franklin Roosevelt is going to win by 7.5 million votes, even though other people are forecasting that Roosevelt’s going to lose. He wins by 7.1 million votes. So Hurja is better calibrated than the other pollsters at the time. But then he flops in 1940, and then later he’s basically as accurate as your average pollster.
In investing, it’s hard to beat the market over a long period of time. Similarly, with polling, you have to rethink your methods and your assumptions constantly. Even though early on Emil Hurja is getting called “the Wizard of Washington” and “the Crystal Gazer of Crystal Falls, Michigan,” his record slips over time. Or maybe he just got lucky early on. It’s hard after the fact to know whether he was really this genius predictor.
I bring this up because—well, I’m not trying to scare you, but it may be that your biggest screw-up is somewhere in the future, yet to come.
That’s sort of the lesson here. What I want people to think about is, just because the polls were biased in one direction for the past couple of elections doesn’t mean they’re going to be biased the same way for the same reasons in the next election. The smartest thing we can do is read every single poll with an eye toward how that data was generated. Are these questions worded properly? Is this poll reflective of Americans across their demographic and political trends? Is this outlet a reputable outlet? Is there something going on in the political environment that could be causing Democrats or Republicans to answer the phone or answer online surveys at higher or lower rates than the other party? You have to think through all these possible outcomes before you accept the data. And so that is an argument for treating polls with more uncertainty than the way we’ve treated them in the past. I think that’s a pretty self-evident conclusion from the past couple of elections. But more importantly, it’s truer to how pollsters arrive at their estimates. They are uncertain estimates at the end of the day; they’re not ground truth about public opinion. And that’s how I want people to think about it.