In which Chris Trotter tries to make sense out of a poll he does not like, and a bunch of people get confused about online polls

On Friday, Chris Trotter opined in the Dominion Post about recent poll results. He was not happy about the estimate in the latest Colmar Brunton poll of 27% support for Labour, and asked this question:

“Are opinion polls being used, quite deliberately, to demoralise the Government's opponents?”

The answer to this question is, of course, “No.”

Polls are regularly scheduled, and adhere to strict, unchanging guidelines about how to collect their information. There are times when the political left does not like the results, and there are times the political right does not like them, too (see 2002). There is no evidence that New Zealand political polls have been tampered with for partisan gain.

Trotter’s rationale for asking the question is that the revelations about Rupert Murdoch’s influence in British politics might clue us in to similar shenanigans over here, in particular that the recent poll might be some manner of trick by shadowy right-leaning forces. He then quickly says that normally this kind of question would only occur to conspiracy theorist nuts, but says these times are different.

No, these times are not different. Chris Trotter is peddling a quite ridiculous line that relies on grand, lengthy conspiracies, and has absolutely no evidence to support it. His thinking amounts to: “one right-winger is a crook about something, so they all must be crooks about everything.”

His delusional thinking, published in an influential venue, may also cloud some people’s perceptions of the left’s intellect in general. As a person who, like Trotter, prefers policies of the left and hopes they will win, that makes me upset.

Trotter also cites my own academic work as tangential support for his Tinfoil Hat theory:

“…by conducting its polling mostly on weeknights, Colmar Brunton's academic critics (Rob Salmond, Keith Rankin) argue that the company is much more likely to make contact with higher income-earners  men and women whose electoral choices traditionally favour the more conservative political parties.”

Trotter misstates my argument. Polling “mostly on weeknights” is not really a problem with a polling method. After all, most nights are weeknights! My critique was that Colmar Brunton (circa 2002-2004) polled only on weeknights, meaning they have a low chance of contacting certain low-wage groups like office cleaners, restaurant staff, shift-workers, or bartenders, even with an aggressive call-back procedure.

And at that time I showed that Colmar Brunton did have a right-leaning bias of somewhere around 5% (see chart above).

Colmar Brunton has now changed its practice. Now they poll for two weekend days and three weekdays. As I mentioned over at 08wire in 2008, there remains some potential for residual bias with this new method. The extent of any remaining bias depends on how many of their initial calls are made on the first two (weekend) days. I do not know that proportion. Aggressive call-backs, which are the practice at Colmar Brunton, make any day-of-the-week bias smaller still. Their revised method it is a massive improvement on what came before.

The proof, of course, is in the data. Below are charts showing poll ratings for both National and Labour from July 2009 to July 2011 across three firms (Colmar Brunton/TVNZ, Reid Research/TV3, and Roy Morgan). Colmar Brunton no longer stands out in this crowd as it did in 2002 – 2004. And this crowd has a good track record, too. Shaun McGirr and I found, for example, that the TV3 poll outperformed ipredict and all other individual polls as a predictor for the 2008 election. That poll was also very accurate in 2005. The Morgan poll also has a good track record predicting results, both in New Zealand and overseas.

Trotter argues that, despite their track record, his unproven suspicions of Murdoch-esque skullduggery should lead us to reject all those polls wholesale and instead rely more on the new Horizon poll, which features an online sample and some statistical corrections Trotter approves of (and, entirely coincidentally, shows a virtual tie between the left and the right).

Horizon has no track record to speak of. The only election it has actually predicted was the Auckland Mayoral race, which its final poll got wrong by 17%. (To be fair, almost all the pollsters got that one badly wrong.)

In terms of nationwide polling, all we know so far is that it gives consistently different results to two polls that have a good track record of predicting nationwide results.

I am interested in the general idea of online polling, partly because of the good track record that other online pollsters such as YouGov have developed overseas. Some say online polling is inherently flawed. Those people are short-sighted and wrong. The internet is just another broad sampling tool, like the telephone. If you can understand the population at large, and also understand the sample that your sampling tool can reach, then you can provide population estimates from the sample. The identity of the tool doesn’t matter too much.

So, I am open to being convinced about Horizon. Having said that, until we see how Horizon’s particular estimation method stacks up against multiple election results, I’ll stick with the firms with the track records.

Comments (18)

by DeepRed on July 26, 2011

Part of the problem is that pollsters have struggled to adapt to the mobile age, and we're not alone. Maybe they could take a leaf out of Finland's book, where mobile phones are the norm rather than the exception.

by alexb on July 27, 2011

I'm not for a second suggesting there is some sort of grand conspiricy, but there have been some howlers of polls recently which have favoured the more right leaning candidate, such as the Auckland mayoralty race, where the polls gave Banks much more of a shot, or the TTT byelection which suggested it was neck and neck, Harawira subsequently won by 8%, a fairly significant margin and much higher than the margin of error. I think Trotter's real point was that media coverage of polls is self reinforcing, Fairfax has been hammering home polls that suggest National is winning, and other polls are quietly ignored. Again, I don't think there is a conspiricy, its just that there is definately a dominant media narrative about the popularity of Key, and the polls that reinforce that narrative are promoted.

by Draco T Bastard on July 27, 2011
Draco T Bastard

The internet is just another broad sampling tool, like the telephone.

More people have access to the internet than a telephone these days and more and more people are dropping the telephone for the far more versatile internet. This means that telephone polling is becoming less and less accurate.

According to it's methodology Horizon should be the most accurate but, as you point out, it's not there yet. This will probably change as more people sign up to its panel giving them a greater pool from which to select the people that they survey.

by Tim Watkin on July 27, 2011
Tim Watkin

Alex, the Auckland mayoralty polling did seem to go wrong. Can you, or anyone, remember the pollsters? As to your other point, which ignored polls are telling a different story?

Draco, quite right, seems to me growing a large and unbiased enough panel is the key. So long as you're not getting people to offer an opinion purely cos they get a free takeaway voucher or somesuch, then you should have access to quite a cross-section. Although having said that, you're still leaning towards the wealthier end of society. And I'm told online polls do screw slightly young (for a change).

by alexb on July 28, 2011

Tim - The Horizon Poll was all over The Standard, and barely rated a mention on Stuff. Now, I'm not suggesting The Standard is balanced or unbiased, but I am saying the Horizon poll deserved more attention, even if it is a new and somewhat untested poll.

by Tim Watkin on July 29, 2011
Tim Watkin

Alex, the Sunday Star-Times, which is owned by the same folk as Stuff, have tossed at least two front page stories at the Horizon polls. That's pretty significant coverage for an untested poll.

But most media rely on their own polling and give very little coverage to others'. So hardly surprising that a poll with no media affiliation hasn't been sprinkled across other media.

But more pointedly, the fact remains it's at odds with all the others, and one of the points Rob's making is that the left can wish all it likes, but if ColmarBrunton, Reid, DigiPoll etc are all saying much the same thing, maybe it's just because they're right. 

by tussock on August 02, 2011

How is it TV3's polling always underrates the small parties in favour of the larger ones (and thus, at the moment, giving even more to National)? That's not a red vs blue thing, so maybe they weight the ages of respondants differently? Assuming the big parties have an older voting block.

Maybe it's the "are you likely to vote" question, and how it's worded, if they have one.

by Frank Macskasy on August 07, 2011
Frank Macskasy

"...Having said that, until we see how Horizon’s particular estimation method stacks up against multiple election results..."

But surely, polling becomes more and more accurate as we get nearer to an Election.

As the number of Undecideds choose actual preferences, the polling percetaages will "firm up" and (theoretically)  should mirror the actual result on Election Day.

The greatest criticism I have with the mainstream polls is that when they release their figures, they make no account of Undecideds. So a figure of 50% "support" for National is derived from those (eg; 80% of respondents) who stated a preference. Not from the 100% of respondents who may or may not have a preference.

That is a considerable difference. 50% of 100% = 50. 50% of 80% = 40.

And Rule #1 in politics is learning to count. National can indeed govern alone with 50% of the Party Vote. But not on 40.

(And I hope I've done my sums right here... )


by Ross Calverley on August 14, 2011
Ross Calverley

The only poll, Frank, that makes any allowance for undecideds is Horizon!

by on April 03, 2012

How is it TV3's polling always underrates the small parties in favour of the larger ones (and thus, at the moment, giving even more to National)? That's not a red vs blue thing, so maybe they weight the ages of respondants differently? Assuming the big parties have an older voting block.

SEO Analysis
by Sharon Gillam on December 29, 2012
Sharon Gillam

The article is clearly written and every point is factual and is no-nonsense. I have surf the internet looking for topics such as these and it is here where I find it written and stated well.

thesis writing

by Sharon Gillam on January 31, 2013
Sharon Gillam

A fantastic webpage always comes-up with new and exciting information and while analyzing I have think this webpage is really have all those fantastic that are qualified a webpage to be a fantastic one.

editing and proofreading

by Sharon Gillam on March 02, 2013
Sharon Gillam

Really impressed! Everything is very open and very clear explanation of issues. It contains truly information. Your website is very useful. Thanks for sharing.

editing and proofreading

by Sharon Gillam on April 27, 2013
Sharon Gillam

I am preparing my assignment paper and gathering information on this topic. Your post is one of the better that I have read.

academic writing

by Sharon Gillam on May 28, 2013
Sharon Gillam

I truly appreciate this post.The design of the home is very beautiful. I am very lucky to be able to come to see your weblog.Keep up the good work.

writing a thesis

by Sharon Gillam on June 20, 2013
Sharon Gillam

Very nice post. I just stumbled upon your weblog and wished to say that I have truly enjoyed surfing around your blog posts. After all I’ll be subscribing to your rss feed and I hope you write again soon!

custom writing

by Sharon Gillam on August 06, 2013
Sharon Gillam

Thanks for this read mate. Well, this is my first visit to your blog! But I admire the precious time and effort you put into it, especially into interesting articles you share here!

academic writing

by Sharon Gillam on November 28, 2013
Sharon Gillam

I really enjoyed the quality information you offer to your visitors for this blog. I will bookmark your blog and have my friends check up here often.

writing a personal statement

Post new comment

You must be logged in to post a comment.