We have known since last May that the March 2018 Population Census is badly flawed. Does it matter? What can be done?

A modern state depends on a regular population census. It is usual at this stage to mention a host of administrative services – such as the funding of District Health Boards – which require accurate population counts. But the use of the Census is wider than that.

Suppose you saw a survey which said that 64 percent of the respondents were female and 36 percent were male. You would suspect that the survey was unreliable. How would you know? The Population Census reports that about half of us are female (51 percent) and half are male. But what if the survey was of people over 85? As it happens, the 2013 Population Census reported that 64/36 ratio, so you would have some confidence in the survey results.

(I got the ratio off the Statistics New Zealand website. It was not particularly user-friendly. Government departments seem to compete for who has the worst website.)

We have not got the population figures for 2018. Unfortunately the response rate for the recent census was exceptionally low. That means that key groups, the young, ethnic minorities, the old, the computer illiterate, may be seriously undercounted. We do not know which and we cannot be sure because we have no reliable population census to benchmark the outcomes against.

The public has known this since about May 2018 and Statistics New Zealand has since been trying to fill the gaps by imputing the missing figures using data from administrative sources, in effect trying to create the forms that people never filled in.

They cannot do this on all the variables. For instance, the longest personal income series goes back to the 1926 census. The 2013 census results showed some anomalies which, rather jumping to conclusions, I was hoping to sort out by using the 2018 census results. That will not be possible. Fortunately there are other data sources, but none of them gives income by ethnicity. So I will not be able to tell whether in the last decade there has been a major break in the personal income distribution nor why, if it has happened.

An even bigger concern is the census question about Maori descent, which is used for determining the allocation of Maori electoral seats. It may not be possible to impute the numbers with sufficient accuracy for electoral purposes. I shall not be surprised if the issue ends up in court.

One opinion I have heard is that the courts may conclude that the 2018 Census does not meet the standards set by statute and throw out any electoral redistribution based on the 2018 figures. Even if the courts do not, litigation would delay things. It will be a muddle and I shant be surprised if the next election is fought on boundaries determined by the 2013 Census with no allowance for population shifts since then.

Indeed, we may not have the imputed census results in time. The current promise is August 2019, a year later than usual. But it is noticeable that the promised data has already suffered slippage so it may be missed again, because the statisticians in SNZ will not release data until they are absolutely satisfied with it. On the other hand any later than next August may mean that it may not be possible to redraw the electoral boundaries before the 2020 election (even if there are no court challenges).

Why has this happened? The superficial reason is that the 2018 Census switched to online questionnaires with the paper ones used in previous censuses as a backup for those who did not go online. Insufficient people filled in either. But other countries have used the method without the same disaster. What went wrong here?

A review is yet to be published, but a standard conjecture is that, as with so much other government spending, the previous National Government underfunded the 2018 operation. Instead of saying ‘we cannot do the job for the amount you are providing’, Statistics New Zealand cut back the number of enumerators. The shortage of census enumerators meant each had to cover too many dwellings so they did not get to enough people.

(Some enumerators have told stories about what went wrong. Such things have gone wrong in the past and were silently fixed up. This time the complaints went public. Another concern is that there was less pre-census publicity than was usual in the past. I hardly noticed any. I am told that the effort was put into social media to boost the youth turnout.)

The conclusion from the research on electronic voting is that

            - if you have an online process you need a paper one too;

            - you have to resource both properly or turnout (participation) will drop;

            - if you put obstacles in the way, participation can drop.

Like so many other underfunded government activities of the era, the outcome has been below quality and we are, and public policy is, suffering. As in the case of mine, building and WOF inspections and so many other cases, the cost of remedying the failure from the lack of resources is extremely expensive.

One worry is that the SNZ’s senior management had no experience of running a census, which meant they were not alert to the possibilities of things going wrong. In the past the census management team had often been broadly the same group over a number of censuses. This time we had neophytes tackling a really hard change in practice.

One of the strangest things in the whole story is that an OIA request elicited

            ‘The Executive Leadership Team (ELT) [of SNZ] did not receive any 2018 related papers within the period .... 1 June to 24 October. This is because the ELT is not involved in operational decision making regarding the 2018 Census. The ELT does, however, receive oral updates regarding the 2018 Census at their meetings.’

Huh? Here is the greatest crisis that Statistics New Zealand has faced in living memory and the leadership eschews reports (which enables them to prepare for discussion and to reflect after the meeting). Instead it depended upon ‘oral updates’. Can you imagine a Treasury leadership team confessing that when the budget is being prepared it had nothing in writing and depended upon what it was told? This is generic management gone mad.

 To be clear, I have the greatest confidence in the professional statisticians in Statistics New Zealand. Their imputed census will be as good as it can get. It will not be as comprehensive as an orthodox census omitting some important variables, it will probably be even later than promised and it probably wont stand up in a court of law.

So what has to be done? There has been considerable discussion in the statistics community about the options. The consensus is that there should be another census in 2021 (which would realign us with the international census cycle) and that until then we should depend on the 2013 census for statutory purposes.

Comments (8)

by James Green on December 18, 2018
James Green

The institutional memory for conducting censuses goes back 1000 years to the Domesday Book. That such a mess was made of it is indicative of serious problems.

by Pat on December 18, 2018
Pat

Again, good luck with that.

Since we determined almost two generations ago that what we knew/did was wrong we have regressed to the point where we dont know what we dont know....and as that long aquired knowledge (expertise/experience) was abandoned many years ago its redevelopment will take at least as many to recover, if at all.

Yet another display of short-termism

 

by Moz on December 19, 2018
Moz

James Green, I vaguely recall mention of a census in a somewhat older book, although that is hearsay written some time after the events in question. Now is a particularly apt time of year to discuss that, as the alleged census apparently took place on the 25th December...

by Draco T Bastard on December 26, 2018
Draco T Bastard

I got the ratio off the Statistics New Zealand website. It was not particularly user-friendly. Government departments seem to compete for who has the worst website.

There's plenty of private websites that do the same thing.

Unfortunately the response rate for the recent census was exceptionally low.

[Citation needed]

The public has known this since about May 2018

No they haven't. They didn't have access to the actual stats and so they knew nothing at all. All they had were complaints.

The shortage of census enumerators meant each had to cover too many dwellings so they did not get to enough people.

I was on the Census helpdesk at the time. Many of the enumerators simply weren't doing their job at all.

Wouldn't surprise me to find that National had underfunded the process. IIRC, They had suggested, back during the Canterbury earthquakes, actually dropping the census altogether.

by Brian Easton on December 27, 2018
Brian Easton

James Green and Moz. Actually the first known censuses go back 4000 odd years ago to Egypt. They and the ones your refer to were all an exercise for working out the tax base. Nowadays that has a low significance in our censuses.

An interesting contribution Draco T Bastard. First of all about the criticisms. I wrote ‘The public has known this since about May 2018' You wrote ‘No they haven't. They didn't have access to the actual stats and so they knew nothing at all. All they had were complaints.’ Signals were appearing in April and May.

That they were keeping the census responses open till an unprecedented May said something was wrong. Perhaps, though, I should have said ‘the facts were in the public domain’ rather than ‘the public knew’.

You asked for a citation to ‘[u]nfortunately the response rate for the recent census was exceptionally low.’ See the May item above. There are many more.

Your following observations, from someone on the ground, are helpful 

‘The shortage of census enumerators meant each had to cover too many dwellings so they did not get to enough people.’

‘I was on the Census helpdesk at the time. Many of the enumerators simply weren't doing their job at all.’

Just a caution about your last sentence. It seems likely that the enumerators were badly trained and/or badly managed.

by James Green on January 11, 2019
James Green

Sorry for the late reply. I said the "institutional memory", by which I meant that New Zealand and England were institutionally connected and that a line of censuses can be traced back as far as 1000 years (approx.). I didn't check if the line of censuses was actually unbroken though (at least 1 per lifetime since the first book). I doubt Roman and Egyptian censuses had any affect on how last years NZ census was run other than the general concept of a census itself.

by Brian Easton on January 14, 2019
Brian Easton

The first modern British census, James, was in 1801. I dont recall, in any histroical account I have read it anchoring back the the Doomesday Census. Incidentally, there was a public debate about whether the Bible allowed them. 

by Brian Easton on March 06, 2019
Brian Easton

Post new comment

You must be logged in to post a comment.