The Art of Polling

It’s the last few days of the election, and the race (as usual) has been tightening between President Barack Obama and Mitt Romney. As such, a flurry of polls will be released, especially those from the battleground swing states, detailing how close the candidates are to winning.

While polling is an important part of the political process, it’s important to understand that while polling is a science, it's just as much an art.

Now, that seems counter-intuitive to many people; statistics are all numbers, right? And numbers are pretty much black and white, right? Well--welcome to the world of statistics, where numbers are anything but.

I won’t go into the philosophy of statistics here (and, yes, there is such a thing), but I’m going to focus specifically on political polling. 

Let’s take a look at a regular poll: the race is 48% Romney, 49% Obama. The poll has a +/- margin of error of 3.5%, and there were 1080 respondents. 

The results are just that: of the people they polled, 48% answered "Romney" and 49% answered "Obama." (Except they totally didn’t, but we’ll get back to that in a moment.) They probably sent out thousands of surveys (most likely robocalls—automated phone calls—but could be gathered in other methods as well) and got 1080 responses. This response rate determines the margin of error—which means those numbers can each be off by 3.5%. If a poll is “within the margin of error,” that means the difference between the two answers (in this case, 1%) is less than what the answers could be (in this case, 3.5%). With varying degrees of likelihood, this poll could easily be 49% Romney and 48% Obama, or (much less likely) 51.5% Romney and 45.5% Obama. There is also a chance (pretty small, but somewhere in the range of 5% depending on the statistical models used) that the entire results are just plain wrong. So basically there is a 95% chance that the 49%-48% results are correct, allowing a range of 3.5% for each result.

(Note to statisticians and econometricists who may be reading this: Yes, I realize I’m glossing over and criminally simplifying a large part of your profession. Deal with it.)

So those are the numbers. But how are those numbers created?

First, we need to look at how the questions are asked. For presidential polling, there are two major components of this. First is the “screen,” which pollsters use to determine if you are a likely voter and/or how devoted you are to a candidate. Usually these seem like mundane questions (“How much attention have you paid to election coverage?”) but they convey a lot of information to pollsters. Most pollsters will have a set of questions they use (and their own formulas to determine a respondent’s likelihood of voting) to screen a person first. 

Then, they have to determine how to ask the question. While this may seem straightforward, it’s actually very, very tricky. Studies have shown that making even token neutral changes in the wording of a question can cause wide swings in results. For example, asking “If the election were held today, who would you vote for?” and “Who would you vote for if the election were held today?” can actually yield different results, even though it’s the exact same question. Most pollsters will account for this, but it’s still a very difficult thing to keep neutral.

Once they have the results, however, they have to be weighted. What happens when you get all 1080 back, but it’s clear that 60% of the respondents identify as Democrat and 40% as Republican, yet the state you are focused on is a true 50/50 split? Your results are clearly going to skew Democrat because you didn’t take a “true” random sample. Pollsters, therefore, will take the results and adjust them to match the demographics of the area they are polling in. So in this case those 60% of Democrats would be weighted down while the answers of the Republicans would be weighted higher. (Note that this would still correctly reflect the correct proportion of Democrats who vote Romney or Republicans who vote Obama.) These demographics aren’t just about party affiliation—they could be about age, income, race, etc.--although they are limited to the questions asked during the polling.

Of course, things can get more creative (and trickier) with that. For example, in 2008 there were a ton of first-time voters who voted for Obama, a portion of which probably won’t vote again (or may vote for Romney). The pollsters may have to take this into consideration when they determine whether someone is a likely voter or not. However, no one really knows exactly what this number will be. Will Obama retain 80% of it or just 75%? That alone might make a different of a half a percentage point in your final results; and if the results are drastically different (50%? 95%?) then all poll calculations are way off.

After all this, throw in voters who give inaccurate information (either intentionally or accidentally) or true undecideds who change their mind after answering to add sort of a built-in, unmeasurable margin of error.

So, when you see any sort of poll, you’re looking at numbers that:

•  Have been filtered through a screening process to determine how likely the person is to vote;
• Might be skewed based on the wording of the question;
• Has been adjusted due to the pollster’s demographic weighting formula
• The normal built-in fuzziness of undecideds or fickle voters;
• Has the normal margin of error present in any statistical result
• Still has a small chance to be completely wrong.

Pretty much every single step in this process requires either a judgment call from someone or some sort of expected variance in the results.  It’s not hard to wonder how accurate the results are.

The good news, of course, is that pollsters are professionals. They try and account for each of these problems the best they can. Most pollsters pride themselves in being accurate and neutral, and so they iron out to the best of their abilities any biases that may arise in the results. In addition, there is sort of a “natural” tendency for some of these factors to balance out—for example, most likely inaccurate/accidental answers are evenly split between the parties so it is not a net gain for either side.

Still, there is a fear that the polls are more art than science. What if pollsters assume that the cell phone numbers they are not calling are 80% Democrats but in reality it’s diminished to 70%? Obviously for non-close states this matters little—who cares if the Obama vote is understated by 4% if he’s going to win by 25% anyway? However, if all the major pollsters are pulling from similar information, they may all be making the same mistakes in their formulas. This is also true in successive polls—if every single poll in Ohio shows Obama with a 2% lead, that either means he has a legitimate 2% lead, or that the pollsters keep repeatedly making the same mistake every time they poll in Ohio. Unfortunately, we won’t know which until after the election.

Since pollsters keep their formulas secret—for good reason—we won’t know if there are any particular biases. However, the main point is that if Romney is winning by 1% in Florida or Obama is winning by 2% in Nevada, it actually means that pollsters--or any of us, really--have no idea who is winning.