We have known since last May that the March 2018 Population Census is badly flawed. Does it matter? What can be done?

A modern state depends on a regular population census. It is usual at this stage to mention a host of administrative services – such as the funding of District Health Boards – which require accurate population counts. But the use of the Census is wider than that.

Suppose you saw a survey which said that 64 percent of the respondents were female and 36 percent were male. You would suspect that the survey was unreliable. How would you know? The Population Census reports that about half of us are female (51 percent) and half are male. But what if the survey was of people over 85? As it happens, the 2013 Population Census reported that 64/36 ratio, so you would have some confidence in the survey results.

(I got the ratio off the Statistics New Zealand website. It was not particularly user-friendly. Government departments seem to compete for who has the worst website.)

We have not got the population figures for 2018. Unfortunately the response rate for the recent census was exceptionally low. That means that key groups, the young, ethnic minorities, the old, the computer illiterate, may be seriously undercounted. We do not know which and we cannot be sure because we have no reliable population census to benchmark the outcomes against.

The public has known this since about May 2018 and Statistics New Zealand has since been trying to fill the gaps by imputing the missing figures using data from administrative sources, in effect trying to create the forms that people never filled in.

They cannot do this on all the variables. For instance, the longest personal income series goes back to the 1926 census. The 2013 census results showed some anomalies which, rather jumping to conclusions, I was hoping to sort out by using the 2018 census results. That will not be possible. Fortunately there are other data sources, but none of them gives income by ethnicity. So I will not be able to tell whether in the last decade there has been a major break in the personal income distribution nor why, if it has happened.

An even bigger concern is the census question about Maori descent, which is used for determining the allocation of Maori electoral seats. It may not be possible to impute the numbers with sufficient accuracy for electoral purposes. I shall not be surprised if the issue ends up in court.

One opinion I have heard is that the courts may conclude that the 2018 Census does not meet the standards set by statute and throw out any electoral redistribution based on the 2018 figures. Even if the courts do not, litigation would delay things. It will be a muddle and I shant be surprised if the next election is fought on boundaries determined by the 2013 Census with no allowance for population shifts since then.

Indeed, we may not have the imputed census results in time. The current promise is August 2019, a year later than usual. But it is noticeable that the promised data has already suffered slippage so it may be missed again, because the statisticians in SNZ will not release data until they are absolutely satisfied with it. On the other hand any later than next August may mean that it may not be possible to redraw the electoral boundaries before the 2020 election (even if there are no court challenges).

Why has this happened? The superficial reason is that the 2018 Census switched to online questionnaires with the paper ones used in previous censuses as a backup for those who did not go online. Insufficient people filled in either. But other countries have used the method without the same disaster. What went wrong here?

A review is yet to be published, but a standard conjecture is that, as with so much other government spending, the previous National Government underfunded the 2018 operation. Instead of saying ‘we cannot do the job for the amount you are providing’, Statistics New Zealand cut back the number of enumerators. The shortage of census enumerators meant each had to cover too many dwellings so they did not get to enough people.

(Some enumerators have told stories about what went wrong. Such things have gone wrong in the past and were silently fixed up. This time the complaints went public. Another concern is that there was less pre-census publicity than was usual in the past. I hardly noticed any. I am told that the effort was put into social media to boost the youth turnout.)

The conclusion from the research on electronic voting is that

            - if you have an online process you need a paper one too;

            - you have to resource both properly or turnout (participation) will drop;

            - if you put obstacles in the way, participation can drop.

Like so many other underfunded government activities of the era, the outcome has been below quality and we are, and public policy is, suffering. As in the case of mine, building and WOF inspections and so many other cases, the cost of remedying the failure from the lack of resources is extremely expensive.

One worry is that the SNZ’s senior management had no experience of running a census, which meant they were not alert to the possibilities of things going wrong. In the past the census management team had often been broadly the same group over a number of censuses. This time we had neophytes tackling a really hard change in practice.

One of the strangest things in the whole story is that an OIA request elicited

            ‘The Executive Leadership Team (ELT) [of SNZ] did not receive any 2018 related papers within the period .... 1 June to 24 October. This is because the ELT is not involved in operational decision making regarding the 2018 Census. The ELT does, however, receive oral updates regarding the 2018 Census at their meetings.’

Huh? Here is the greatest crisis that Statistics New Zealand has faced in living memory and the leadership eschews reports (which enables them to prepare for discussion and to reflect after the meeting). Instead it depended upon ‘oral updates’. Can you imagine a Treasury leadership team confessing that when the budget is being prepared it had nothing in writing and depended upon what it was told? This is generic management gone mad.

 To be clear, I have the greatest confidence in the professional statisticians in Statistics New Zealand. Their imputed census will be as good as it can get. It will not be as comprehensive as an orthodox census omitting some important variables, it will probably be even later than promised and it probably wont stand up in a court of law.

So what has to be done? There has been considerable discussion in the statistics community about the options. The consensus is that there should be another census in 2021 (which would realign us with the international census cycle) and that until then we should depend on the 2013 census for statutory purposes.