In which Chris Trotter tries to make sense out of a poll he does not like, and a bunch of people get confused about online polls

On Friday, Chris Trotter opined in the Dominion Post about recent poll results. He was not happy about the estimate in the latest Colmar Brunton poll of 27% support for Labour, and asked this question:

“Are opinion polls being used, quite deliberately, to demoralise the Government's opponents?”

The answer to this question is, of course, “No.”

Polls are regularly scheduled, and adhere to strict, unchanging guidelines about how to collect their information. There are times when the political left does not like the results, and there are times the political right does not like them, too (see 2002). There is no evidence that New Zealand political polls have been tampered with for partisan gain.

Trotter’s rationale for asking the question is that the revelations about Rupert Murdoch’s influence in British politics might clue us in to similar shenanigans over here, in particular that the recent poll might be some manner of trick by shadowy right-leaning forces. He then quickly says that normally this kind of question would only occur to conspiracy theorist nuts, but says these times are different.

No, these times are not different. Chris Trotter is peddling a quite ridiculous line that relies on grand, lengthy conspiracies, and has absolutely no evidence to support it. His thinking amounts to: “one right-winger is a crook about something, so they all must be crooks about everything.”

His delusional thinking, published in an influential venue, may also cloud some people’s perceptions of the left’s intellect in general. As a person who, like Trotter, prefers policies of the left and hopes they will win, that makes me upset.

Trotter also cites my own academic work as tangential support for his Tinfoil Hat theory:

“…by conducting its polling mostly on weeknights, Colmar Brunton's academic critics (Rob Salmond, Keith Rankin) argue that the company is much more likely to make contact with higher income-earners  men and women whose electoral choices traditionally favour the more conservative political parties.”

Trotter misstates my argument. Polling “mostly on weeknights” is not really a problem with a polling method. After all, most nights are weeknights! My critique was that Colmar Brunton (circa 2002-2004) polled only on weeknights, meaning they have a low chance of contacting certain low-wage groups like office cleaners, restaurant staff, shift-workers, or bartenders, even with an aggressive call-back procedure.

And at that time I showed that Colmar Brunton did have a right-leaning bias of somewhere around 5% (see chart above).

Colmar Brunton has now changed its practice. Now they poll for two weekend days and three weekdays. As I mentioned over at 08wire in 2008, there remains some potential for residual bias with this new method. The extent of any remaining bias depends on how many of their initial calls are made on the first two (weekend) days. I do not know that proportion. Aggressive call-backs, which are the practice at Colmar Brunton, make any day-of-the-week bias smaller still. Their revised method it is a massive improvement on what came before.

The proof, of course, is in the data. Below are charts showing poll ratings for both National and Labour from July 2009 to July 2011 across three firms (Colmar Brunton/TVNZ, Reid Research/TV3, and Roy Morgan). Colmar Brunton no longer stands out in this crowd as it did in 2002 – 2004. And this crowd has a good track record, too. Shaun McGirr and I found, for example, that the TV3 poll outperformed ipredict and all other individual polls as a predictor for the 2008 election. That poll was also very accurate in 2005. The Morgan poll also has a good track record predicting results, both in New Zealand and overseas.

Trotter argues that, despite their track record, his unproven suspicions of Murdoch-esque skullduggery should lead us to reject all those polls wholesale and instead rely more on the new Horizon poll, which features an online sample and some statistical corrections Trotter approves of (and, entirely coincidentally, shows a virtual tie between the left and the right).

Horizon has no track record to speak of. The only election it has actually predicted was the Auckland Mayoral race, which its final poll got wrong by 17%. (To be fair, almost all the pollsters got that one badly wrong.)

In terms of nationwide polling, all we know so far is that it gives consistently different results to two polls that have a good track record of predicting nationwide results.

I am interested in the general idea of online polling, partly because of the good track record that other online pollsters such as YouGov have developed overseas. Some say online polling is inherently flawed. Those people are short-sighted and wrong. The internet is just another broad sampling tool, like the telephone. If you can understand the population at large, and also understand the sample that your sampling tool can reach, then you can provide population estimates from the sample. The identity of the tool doesn’t matter too much.

So, I am open to being convinced about Horizon. Having said that, until we see how Horizon’s particular estimation method stacks up against multiple election results, I’ll stick with the firms with the track records.