538 analyzed the various polls--and corrected for their methodologies and predictive power--just fine!
A sibling comment brings up a batter, and I think that's a great analogy. A good batter has a batting average of, say 0.273. Nobody is shocked when they hit the ball!
538 gave a similar chance to Trump. He hit the ball.
Yes and no. 538 published an article after the election explaining that there were in fact systemic errors in the models; the result wasn't just random probability within models that were otherwise correct: https://fivethirtyeight.com/features/the-polls-missed-trump-...
> While the errors were nationwide, they were spread unevenly. The more whites without college degrees were in a state, the more Trump outperformed his FiveThirtyEight polls-only adjusted polling average,1 suggesting the polls underestimated his support with that group. And the bigger the lead we forecast for Trump, the more he outperformed his polls.2 In the average state won by Trump, the polls missed by an average of 7.4 percentage points (in either direction); in Clinton states, they missed by an average of 3.7 points. It’s typical for polls to miss in states that aren’t close, though. The most important concentration of polling errors was regional: Polls understated Trump’s margin by 4 points or more in a group of Midwestern states that he was expected to mostly lose but mostly won: Iowa, Ohio, Pennsylvania, Michigan, Wisconsin and Minnesota.
If the errors had been random rather than systematic, it would have been statistically quite unlikely for Trump to have swept all of those midwestern states.
AFAICT the polls had a systemic error AND the 538 models included the possibility of a systemic error. It's really the only way that they could have given Trump a 28% chance in the first place.
Of course, the polls themselves also acknowledge the chance of a systemic error. They're reported similar to +/- 2% 19 times out of 20. They make no claim that the 19/20 of multiple polls are not correlated.
> AFAICT the polls had a systemic error AND the 538 models included the possibility of a systemic error.
Well, 538 model took into account historical evidence that election differences from polling averages between states are strongly correlated rather than independent; arguably, that's an effect that exists in part because such deviations are in part due to systemic errors (though the particular systemic errors may differ from cycle to cycle) in the polling, rather than pure sampling error.
Right, but 538 did account for such an error in coming up with their ~75/25% outcome chances. Had polling errors just been random, the overall prediction would have been more like 95/5%.
But their key insight is that polling errors are often correlated, and not just statistical sampling errors that can randomly offset in either direction.
> The +/- in a given poll is measurement error derived from the sample size. They don’t account for systemic error.
Sure, but even the worst poll-based prediction outfit isn't just aggregating poll results and using the uncertainty based on sample size to determine the probabilities of different outcomes.
Pretty much all of them are using some model derived from the past relationships of polls to election results, which will, to a greater or lesser extent, capture systemic polling error that is not unique to the current cycle in the model.
A sibling comment brings up a batter, and I think that's a great analogy. A good batter has a batting average of, say 0.273. Nobody is shocked when they hit the ball!
538 gave a similar chance to Trump. He hit the ball.