Science In Action: Bush Poll Numbers -- Margin Of Error

"Bush Approval Rating Hazy . . . Try Again." No headline ever says that. They always give a specific approval rating, say "40%", based on some recent poll. They ignore the fact that such a result is not so precise, but is rather hazy. The "margin of error" figure buried somewhere in the article is the key to understanding just how hazy that "40%" number really is.

Why Are These Poll Results So Variable?

A number of polling organizations regularly ask Americans about how the President's performance is perceived (see table below for recent results). The question is along the lines of "Do you approve or disapprove of the way George W. Bush is handling his job as president?" Note that although these polls all asked almost exactly the same question over the same few days between 6 and 9 September 2005, yet they give a range of results.

Bush Approval Rating, Various Polls
Poll	Date, Sept. '05	Approve, %	Disapprove, %	Margin of Error, percentage points
AP/Ipsos	6-8	39	59	3.1
Time	7-8	42	52	3
Newsweek	8-9	38	55	3
CBS	6-7	42	52	4
Pew	6-7	40	52	3.5
Zogby	6-7	40	59	2.9
Source: Pollingreport.com

The results of such polls always refer to something called "margin of error". What does that mean, and is it important for interpreting poll results?

What is the "Margin of Error"?

Understanding the concept referred to by pollsters as "margin of error" is vital to understanding poll results. Unfortunately, many journalists don't understand this concept or its importance, and thus news stories often mislead based on misinterpretation of polls. (See earlier post on this subject.)

"Margin of error" is the term the media use for what statisticians call the "confidence interval". It would be more properly called "margin of accuracy", or even "margin of inaccuracy".

Polls purport to tell us the opinions of some group of people. In the case of the political polls discussed here that group is all the adults in the U.S.A. (Sometimes polls focus on other groups, such as registered voters, likely voters, men, women, Republicans, Democrats or independents, or people in a particular state.)

There are about 220,000,000 people in the U.S.A. over 18 years old. It would be impossible to ask each and every one of them his or her opinion of the President's performance. Therefore we only ask a sample. Polling is sampling. How well the results from this sample represent the results you would get if you actually could poll the entire adult population is described by the "margin of error", "sampling error", or "confidence interval" statistic.

The key is to use a random sample. As discussed in another previous post, we can't really get a totally random sample, but pollsters, epidemiologists, and other researchers have established methods for at least getting a quasi-random sample of a population.

If the sample is random (anyone in the population has an equal chance of being included in the sample), the math of statistical sampling tells us that if you sample only one thousand people from that universe of 220 million, you will get a fairly good representation of the true opinion of the whole population.

In fact, you can be sure that the result from the sample (for example, 40% say they approve the job the President is doing), plus or minus the confidence interval (say 3 percentage points), will include the actual result you would have gotten from the whole universe from which the sample was drawn 95% of the time. There is only a 5% chance (one time in 20) that the true result is outside that "95% confidence interval". So in this case the pollster would be saying that it is pretty likely that between 37% and 43% of U.S. adults approve the President's job performance.

Note: The math of statistical sampling means that the pollsters don't actually know the exact approval rating the whole population would give. They only know that there is a 19-in-20 chance that the approval rating for the whole adult population is between 37% and 43%, based on the results from the one thousand people they actually asked. They don't really know where within that range the rating from the whole population would be, only that there is a 95% chance that it is in that range.

If exactly the same poll were to be performed twice, with the same sample size drawn from the same population at the same time, the results of those two samples would almost certainly be different. In fact there is a 1-in-20 chance that their confidence intervals wouldn't even overlap.

So polls are like the old Magic 8 Ball®: They always answer "Reply hazy, try again". The "margin of error" is supposed to show just how hazy that result is.

Here is that table of poll results again, restated to take into account their confidence intervals.

Bush Approval Rating, Various Polls,
showing confidence ranges
Poll	Approve, %	Disapprove, %
AP/Ipsos	35.9-42.1	55.9-62.1
Time	39-45	49-55
Newsweek	35-41	52-58
CBS	38-46	48-56
Pew	36.5-43.5	48.5-55.5
Zogby	37.1-42.9	56.1-61.9
Source: Pollingreport.com

Notice that now instead of all of the results being different, they are all the same. All of the "approve" ranges overlap. There is no significant difference between their results. (the "disapprove" ranges don't overlap. The AP/Ipsos and Zogby polls report significantly higher disapproval numbers than the others, which all overlap. This may be explained by the details of the way the question was phrased, how the "not sure" responses were handled, and the particular sampling methods used.)

A graphical display would show this better. I'll try to add one later.