CRA Responds to Article on Polling

Don Mills, head of Corporate Research Associates, has provided an energetic response to the May 24th article about polling. 

In case you missed it in the Herald, it is reproduced below.

Don is correct to say that polls are often correct—his own record is a good one. Likewise, Nate Silver in the US has an extraordinarily good track record in state-by-state predictions for Presidential elections.

Don is probably also right that telephone surveys are better than some of the other methodologies, although the evidence is by no means conclusive. The poll that came closest to getting BC right was done by Interactive Voice Response (IVR).

The Herald prints my articles as written but, as Don would know, they make their own choice about headlines. In this case the choice was not, in my view, a good one.

“Polls no better than crystal ball” could easily mislead the reader as to the essential message of the article and seems to have had that impact on Don.

The point was that a poll a long way from an election is not necessarily a good predictor of what will  happen when people actually vote. For example, the average of polls in Canada for April 2012 showed the Conservatives and NDP tied at 33% while the Liberals trailed badly at 21%.

Twelve months later, the Liberals were ahead with 33% followed by the Conservatives at 30% and NDP trailing at 24%. Clearly at least one of these averages is wrong as a predictor of the next election.

CRA’s February 2012 poll in Newfoundland and Labrador had the Progressive Conservatives leading with 54%, followed by the NDP at 28% and the Liberals at 18%. Twelve months later, it was NDP 39%, Progressive Conservatives 38%, and Liberals 22%. Again, at least one of these is wrong as a prediction of what will happen in the next election.

In September of 2009, CRA reported the Liberals leading the Progressive conservatives by 6% in New Brunswick, a lead that was still reported as 5% based on polling in August of  2010. By the time of the election, CRA’s daily tracking poll was correctly predicting a PC win.

In fact, the PCs improved  enough to beat the Liberals by 14%, a 19% difference from  CRA’s  poll numbers of just a month earlier.

We can all look forward to seeing Don’s numbers early next month, but electorates can be volatile, especially when close to half are undecided. We should not be surprised if the actual election turns out differently than June’s numbers. If so, we will probably see Don predicting it.

Poll-faulting

Bill Black appears to take great pleasure in disparaging both our industry and our company in his May 25 column on polling. I suspect his deep-rooted disdain for the market research industry may help explain the rather narrow perspective presented.

Certainly, the polling industry deserves criticism for its recent performance during elections in B.C. and Alberta. Mr. Black has ignored the issue of new research methodologies that have increasingly been used in elections in an effort to reduce costs, such as online panels and Interactive Voice Response (IVR), as well as the ongoing debate within the industry itself related to the statistical reliability of such methods.

These considerations, I believe, are central to the recent performance of the polling industry and should be examined. Certainly, the record for these new methods of data collection have been, at best, mixed.

To be sure, there are increasing challenges to ensure the collection of representative samples that are reliable. CRA continues to rely on random telephone surveys, the most costly form of data collection, for our political polling. I believe this methodology continues to be the most statistically reliable method for this type of work. It certainly was in the last two provincial elections in Atlantic Canada when compared with competing online surveys.

Corporate Research Associates has had an amazing “crystal ball” over the past 20 years, correctly predicting every election we have covered during that period, including the last four provincial elections CRA conducted on behalf of the media, each well within the stated margin of error (visit www.cra.ca for details).

Mr. Black apparently didn’t think that that record was worth mentioning, so intent he appears to have been in impugning our reputation. CRA will continue to maintain our high standards in the conduct of all our work, while working within our industry to ensure that the reliability, or lack thereof, of all data collection methodologies are both well recognized and understood.

Don Mills, Chairman & CEO, Corporate Research Associates Inc.

Share

Related Articles

Who Represents You + Show all articles

Reference Material