How and what we remember is complicated but crucial. So when we consider the Maori Party's criticism of Helen Clark, shouldn't we ask if New Zealand is a better or worse place to be Maori given her three terms in government?

Well, this is a cat amongst Helen Clark's United Nation's pigeons. In the midst of a parliamentary recess when political news is thin on the ground, the Maori Party has told the world – and it's the world that matters in this case – that it doesn't support Clark's bid for the Secretary-General's job.

Marama Fox came out this week declaring Clark's record with New Zealand's indigenous people poor and that her party could not support her UN bid as a result. According to RNZ this morning, that's a reversal of her position earlier this year, when she told Radio Waatea the party did back Clark. But her switch is likely a response to concern from party members and, to my understanding, has the backing of the board.

It's a great talking point and all sorts are lining up to take a stance – Winston Peters says it's "treachery in the extreme" and Paul Henry, "treason". Curiously, and unhelpfully for Fox, Maori Party founder Tariana Turia has said she understands the party's position, but also backs Clark.

Yet Fox is, reasonably enough, sticking to her guns, pointing to the Foreshore and Seabed Act of 2004, Labour's failure to sign up New Zealand to the UN's Declaration on the Rights of Indigenous Peoples and the Tuhoe raid that happened on her watch.

And she's had backing from some of the sharpest Maori political observers around, such as Morgan Godfrey and Maiki Sherman, who point to the same events as Fox.

They are all fair points, especially as Clark has asked to be judged on her record. But while some are asking why anyone should care when it's ultimately the UN Security Council that decide the next UN chief, they're missing the point that this gives her opponents a reason to sideline her in what is an intensely political contest.

Despite the enthusiastic coverage in New Zealand, the odds have always been against Clark. It's the turn of the east Europeans for the job and Russia will have huge sway in deciding who wins. And she's up against well-connected Europeans, some who are women, some who speak Russian and at least one woman who grew up in a community country. So this is undoubtedly a blow.

But what interests me is how this again exposes the distinctly different world views in this country. Godfrey and Sherman make many good arguments against Clark and it's clear, and to my mind understandable, that many Maori are slow to forgive being used as political pawns in the wider game. If you're the victim of the game-playing, you feel it much more intensely.

Godfrey, for example, told me on Twitter yesterday the foreshore and seabed wasn't just one issue amongst many to consider when you looked at Clark's record with Maori, it was "the issue".

And it's clear that the Foreshore and Seabed Act was deeply flawed. But I'm torn, and I stop short of endorsing the critics because I think you can't judge history and people's acts without acknowledging context. Issues are never as black and white in the moment, as they appear years later. And we risk forgetting the detail as years go by.

For example, Sherman says in her Newshub blog that Clark "signed off" on the Tuhoe raids. That's a blunt interpretation.

When the police finally apologised for the raids in 2012, Annette King, the Police Minister at the time of the raids, spoke for the first time about how they were initiated. She told Paul Holmes on TVNZ's Q+A that she, Clark and other senior cabinet members were only told about the raids the night before. They were told the raids were based on investigations into terrorism and the resulting charges would be laid under the new Terrorism Suppression Act.

King says both she and Clark challenged the police as to whether action under that new act was appropriate and were told yes. But they were not show evidence and were, rightly, reluctant to over-rule an operational decision by the Police Commissioner. The police briefing, she said, was "a courtesy". Not a request for sign-off.

Further, she says that she and her cabinet colleagues were "very surprised" to see the brutal way the police carried out the mission the next day.

So I think we have to be very careful how, and what, we remember. The same can be said, to some degree, for the UN Declaration (which as I wrote at the time came full of fish hooks) Foreshore and Seabed Act.

Andrew wrote this excellent Pundit post back in 2009 when a ministerial review found the foreshore and seabed law wanting. But it's interesting to note Chris Trotter's counter-factual in the comments.

Andrew's argument is that Clark could have played the hand she was dealt much better – and without making terrible law – but Chris argues that the political context made that nigh on impossible.

I have sympathy for both views, but what I do feel strongly is that you can't judge a Prime Minister without looking at the environment she was operating in and some counter-factuals. It's undeniable Clark was working within very tight political boundaries when it came to race relations and being seen as too sympathetic to Maori would have been political suicide. Don Brash's Orewa speech was only months away when the foreshore and seabed decision was made, and the political upheaval that caused didn't come out of nothing. Much of the Pakeha electorate's mood was not sympathetic to Maori concerns, which was clear by her earlier decision to ditch the Closing the Gaps policy (in name at least), another compromise that sits as a black mark on her race relations record.

So what? That's certainly what Godfrey says. Politics is no excuse to use Maori as pawns; just do the right thing. Except people and groups are used as pawns in politics every day. Politicians often make choices to sate the majority at the expense of the minority. In some ways it's essentially democratic, even if sometimes unjust.

So while I think history will recall that Clark's record with Maori was one of her failings in office, I understand that she would argue she did the best she could in those times. The Closing the Gaps intentions suggests she would have done more if she thought she could. So while we question her record, should we not also ask if New Zealand would be a better or worse place to be Maori had she done differently?

That is a much harder question to answer.

For a start, take a wider look at her political impact on the lives of Maori. Her strong stance on anti-smoking laid the ground work for Turia, Hone Harawira and the Maori Party's efforts today. She ultimately backed the anti-smacking bill. Consider her government's record on social spending, KiwiSaver and more.

Could not a firmer stance on behalf of Maori lost Labour the whisker-close 2005 election? And how would Maori have suffered under a Brash-led National government? Maori seats gone (and with them the Maori Party – that's especially ironic, given Fox's stance), "special treatment" gone, Treaty of Waitangi references in law gone. You could argue, as Clark probably would, that given the court's ruling, Maori were always going to lose some rights at that moment in history, be it under her or the next government. So did her stance ensure the loss was less than it might have been under Brash? Did she cut off a leg to save the poison spreading?

These are the soul-crunching calculations of politics. Context is vital. And, whatever your final judgement, fairness surely demands these memories are accurately recalled and remain part of the conversation.