Be Careful with Polls

Cross-posted from Roger Pielke Jr.'s Blog

Mark Blumenthal of The National Journal has an insightful blog post about the perils of public opinion polls. Here is an excerpt:

How do Americans feel about cap-and-trade legislation?

In recent weeks, two media pollsters reported results on the point. "Six in 10 Americans support a 'cap-and-trade' proposal to cut pollution," said the CNN/Opinion Research Corporation poll. Despite "growing public skepticism about global warming," the Pew Research Center found "more support than opposition for a policy to set limits on carbon emissions."

How accurately do these questions measure public opinion on cap-and-trade legislation?

To answer that question, you may want to consider how Americans answered another: "Some people say the 1975 Public Affairs Act should be repealed. Do you agree or disagree with this idea?"

As a well-informed reader of NationalJournal.com, you are probably inclined to wrinkle your brow and ask, "What's that?" For good reason: It never existed. But its fictitious nature didn't stop 34 percent from expressing an opinion when University of Cincinnati political scientist George Bishop and his colleagues asked a sample of Cincinnati adults that question in 1978. Bishop and other scholars have consistently replicated that finding using national samples and similarly fictitious or unknown legislation. As summarized in Bishop's book, The Illusion of Public Opinion, between 30 and 40 percent of Americans will offer opinions on legislation they have never heard of.

Blumenthal asks "what are we to make of responses to questions that use possibly unfamiliar terms like "greenhouse gases" and "carbon dioxide emissions?""

He put that question to George Bishop, author of the 1975 study referenced above, here is Professor Bishop's response:

"'Cap-and-trade' legislation is so obscure and so little-known by the vast majority of Americans," he concluded via e-mail, that questions about it generate the same sort of "pseudo-opinions" as the fictitious 1975 Public Affairs Act. "Reliable and valid measures of public opinion on such a complex policy issue," he writes, "cannot be so simply simulated by merely telling respondents what it's about and then asking them to react to it on the spot. Down that road lie misleading illusions and the manufacturing of public opinion -- a disservice to the Congress, the president and the press that covers them."

My view is that public opinion is plenty strong enough for action to occur, in other words, there is nothing politically intrinsic about the issue that stands out as being a barrier to action. By contrast, legislation to make abortion illegal might face such an intrinsic political barrier. That means that the issue is about the specifics of policy, and the political implications of specific policies -- who wins and who loses in specific bills. Consequently, at this point in the debate public matters very little. What matters are the perceptions of various decision makers in Congress. Crafting policy that can be effective, be seen to be effective and provide parochial as well as national benefits is the political challenge facing the Congress. From what I read, they are not doing so well in meeting these criteria.