Bias and Variance in Multiparty Election Polls

Abstract

Recent polling failures highlight that election polls are prone to biases that the margin of error customarily reported with polls does not capture. However, such systematic errors are difficult to assess against the background noise of sampling variance. Shirani-Mehr et al. (2018) developed a hierarchical Bayesian model to disentangle random and systematic errors in poll estimates of two-party vote shares at the election level. The method can inform realistic assessments of poll accuracy. We adapt the model to multiparty elections and improve its temporal flexibility. We then estimate bias and variance in 5,240 German national election polls, 1994–2021. Our analysis suggests that the average absolute election-day bias per party was about 1.5 percentage points, ranging from 0.9 for the Greens to 3.2 for the Christian Democrats. The estimated variance is, on average, about twice as large as that implied by usual margins of error. We find little evidence of house or mode effects. Common biases indicate industry effects due to similar methodological problems. The Supplementary Material provides additional results for 1,751 regional election polls.