Rough Magic: the Role of Evaluation in Overcoming Indigenous Disadvantage

Last week we saw the publication of the Productivity Commission’s Overcoming Indigenous Disadvantage (OID) 2016 Report. Predictably, it reported mixed results. I don’t propose to summarise the contents, but focus instead on the narrative around its release.

The Minister’s media release emphasised the positives, but stated that

“a great deal more needed to be done to address Indigenous disadvantage, including building the evidence of what worked”….

…. Minister Scullion also acknowledged that in the areas of incarceration, domestic violence, mental health and substance misuse, increased effort was required to improve outcomes – and better evidence was needed to drive this progress….

…. “Until recently, there has not been sufficient investment in evidence to drive Indigenous-specific mental health and suicide prevention responses.”….

….Minister Scullion said it was also important that individual programmes within the Indigenous Affairs portfolio were properly evaluated to determine their effectiveness….

… “The Coalition Government is working hard to build a better evidence base than there has been previously for Indigenous Affairs. We are increasing the use of quantitative data and using a variety of mechanisms to evaluate the success of individual Indigenous Affairs programmes”.

Coincidentally, the Secretary of the Prime Minister’s Department, Martin Parkinson in a recent speech noted:

The current Closing the Gap framework is coming up to 10 years old and many of the targets about to expire. We should not be fearful of where we go from here but rather seize the enormous opportunity this presents. This is our chance— Government and Indigenous people—to take the lessons of the past decade and work together to reset the agenda, to focus our efforts to truly close the gap in the outcomes for Indigenous and non-Indigenous Australians.

 A high proportion of what we fund has, at best, a weak evidence base of how it affects Indigenous peoples. We must gather evidence which shows we are improving the lives of Indigenous Australians. And if that evidence tells us otherwise, we must change our approach. 

We need to put our minds to many questions—what did we get right and what did we get wrong? And why? We need to commit to the economic development for Aboriginal and Torres Strait Islander peoples based on accumulated knowledge. We know that the keys to self-reliance, independence and improved social outcomes are: higher levels of employment; Indigenous business ownership; and the opportunity to use and develop culture, knowledge and land assets to generate wealth.

Joining the chorus, the Deputy Chair of the Productivity Commission, Karen Chester, made a robust case for more evaluation of Government programs on ABC Radio National’s AM program (audio here) and in The Australian.

Digressing briefly, The Australian’s story was headlined “Lack of account in $5.9bn spend ‘beggars belief’”. The misleading headline managed to simultaneously imply that the report found that there is a widespread lack of accountability for public funding, and that the problem lies with funding recipients rather than the governments who are responsible for the effective management and oversight of funding programs. This inaccurate message was reinforced by the apparently misleading text under the photo of Ms Chester (in the online story) which quotes her as saying she is “staggered by the programs’ lack of success” whereas she is quoted in the article as saying “she was staggered at the lack of attention paid to assessin­g what works, ….Evaluation is missing in action­ and it beggars belief that it’s missing in action…”

So what to make of the narrative that the problem we face with Indigenous policy is fundamentally about insufficient evaluation?

There are at least three potential interpretations. 

The first is that the explanations offered by the Government and bureaucracy should be taken at face value, and that once we manage to successfully evaluate the key programs, and adjust our policy and program settings in response, we will be back on track.

A second is that those at the centre of devising, developing, implementing policy in relation to Indigenous affairs are working from a set of deeply entrenched and extremely technocratic assumptions both as to how the public policy system works and how Indigenous communities respond to policy initiatives. In this rational and technocratic framework, policies, laws and programs are devised to structure and if necessary change behaviour so as to accord with a set of values and expectations largely determined by mainstream interests (eg cutting benefits to parents whose children don’t attend school, or income management of welfare payments to ensure expenditure is not directed to gambling or alcohol consumption) through the imposition of incentives both positive and negative. These types of technocratic approaches can often miss the mark as, apart from the inevitable implementation challenges and shortcomings, they are often perceived by Indigenous citizens as irrelevant to Indigenous concerns, poorly targeted, culturally insensitive, externally imposed, inappropriately coercive, or even discriminatory.

In this technocratic world of unidimensional causation described by a process of inputs, outputs and outcomes, evaluation and remediation is the final link in the circular process which closes the circle. We should not be surprised then that evaluation will work from within the technocratic frame and assumptions adopted by mainstream policymakers, and will be unlikely to identify underlying and fundamental issues driving policy failure which lie outside the technocratic mindset.

A third more cynical interpretation is that the Government knows that its policies, laws and programs related to Indigenous citizens are not working, but doesn’t understand why, or is not prepared to resource programs effectively, or is not prepared to drive the states and territories to perform effectively, or is focussed on politics to the exclusion of policy, or plays favorites, or won’t take advice, or tries to be all things to all people, or actually isn’t prepared to give the Indigenous policy sector the priority it needs to cut through the inertia which surrounds every policy domain to a greater or lesser extent, and so on. In this interpretation, the calls for more evaluation are in effect a cynical distraction aimed at buying time and political cover, along with a hope that by the time the next Overcoming Indigenous Disadvantage Report is issued, the electorate and stakeholders will have forgotten what was said two years previously, and in any case, life will have moved on.

These potential interpretations raise a number of general issues. If evaluation is so important now, why has it been neglected for the past three years, or indeed the past ten years? What work is proposed to understand why the focus on evaluation (particularly in Indigenous programs) has been allowed to fade and lapse? And what about the various audits, coronial inquests, reviews, and the like which have been undertaken over the past decade? Did we learn nothing from them? Was the Hope Coronial Inquiry into 22 Kimberley suicides a waste of time? Did we fail to learn anything from the Mullighan Royal Commission into the abuse of children in the APY Lands? How is it that the Office of Evaluation and Audit established under the ATSIC legislation, with an independent statutory remit, has been dismantled, and absorbed into the ANAO?  And where has the ANAO been while this systemic ‘lack of attention’ to evaluation has emerged? And how might the electorate and Indigenous citizens be assured that the same deterioration in focus won’t occur in the future?

Whichever of the three interpretations one accepts, we can expect more attention to be directed to evaluation issues by Government. Under the first two interpretations, there will likely be an increase in the number of evaluations, reviews, and effectiveness audits commissioned and undertaken. Under the third, policy failure will continue, periodic disasters or crises will emerge, and questions about policy effectiveness will continue to be raised, if not by governments, then certainly by others.

Contrary to most accepted wisdom, I am not entirely persuaded that more evaluation would be an unalloyed positive. Like Prospero in The Tempest, “this rough magic/ I here abjure”. Too often, in my perhaps jaded experience, evaluative work (especially when undertaken within the executive arm of government) is an exercise in going through the motions, designed to provide confirmation for policies and decisions which are based on political imperatives or ideology rather than evidence.

In my experience, effective evaluations inevitably revolve as much around value judgments as objective analysis, but few policymakers are prepared to justify their decisions on values over ostensible evidence. Yet statistical analysis can be, and often is, shaped to support (or at least not contradict) a predetermined policy or political narrative. Moreover, apart from the substantial cost of effective and independent evaluations, there is a real opportunity cost both in terms of what those resources might have been directed towards, and in terms of policymakers inwards focus into what the evaluation or review will say rather than towards making programs and policy initiatives more effective.

Implicit in this critique is a view that in most areas of policy, we actually know what will deliver positive outcomes, and the challenge is to deliver the basics, not the optimal. We know that provision of police services will improve community safety. We know that provision of adequate housing and reducing overcrowding will improve health and education outcomes. This is not an argument in favour of zero evaluation, but rather is an argument in favour of targeted and risk based evaluation activity with an eye on the expected net benefits of the proposed evaluation. It is an argument in favour of short and sharp evaluations designed to identify, say, the three most significant changes that might be made to a policy, not the list of fifty (or more) recommendations which comprise all conceivable changes. It is an argument for a focus on policy effectiveness, and against mechanistic evaluation processes which claim to be comprehensive, but are in reality drivers of yet more complexity and process. It is an argument for doing good, not achieving perfection.

One of the reasons governments call for more evaluation activity is the fact that evaluations rarely lead to the fundamental questioning of the government initiative under review. Indeed, in my more cynical moments, it seems to me that an anthropologist from Mars studying our evaluation culture might easily describe it as akin to a ceremonial ritual activity, designed to reassure all who participate that the world as we know it will continue, that the future though uncertain will not be disastrous, and that if we participate in the ritual, we will all receive our just rewards, if not in this life, then in the next. In other words, don’t rock the boat now; things will improve in the future if only we participate in the ritual ceremony of evaluation.

Clearly I am conflicted about the merits of evaluation. Nevertheless, given the general acceptance of the ritual of evaluation in our public policy life, there is a case for establishing some benchmarks. In particular, it seems incontrovertible that the quality of evaluations will increase to the extent that there is greater transparency around its commissioning, greater independence in the process of reaching conclusions, and increased accountability in monitoring the implementation of the responses to evaluations by governments.

One further point worth noting regarding the recent Overcoming Indigenous Disadvantage reports are that they are technically not owned by the Productivity Commission, but by a Steering Committee made up of bureaucrats from across the states and territories as well as major federal agencies. The same is the case for the Indigenous Expenditure Report produced by the Commission. While the undoubted technical capacity of the Productivity Commission is brought to bear in producing these reports, it does mean that the bureaucracy effectively controls and determines the way in which the reports are framed and completed. Secondly it has the consequence that the Indigenous portfolio is likely to receive less access to the Productivity Commission’s independent research capacity. This is because the Commission when allocating its finite research resources, will usually take the view that it is already allocating substantial resources to the Indigenous sector. The recent Commission research paper on Primary School Achievement is the exception which proves the rule. Thus paradoxically, while the Productivity Commission plays a crucial independent role in devising and formulating innovative policy solutions to many of the most challenging public policy issues the nation faces, it does not often do this in relation to Indigenous affairs. This is an issue which requires reconsideration if we are to commit to a stronger evaluation culture in the Indigenous policy sector.

On its face, if the Government is serious in its view about the need for greater focus on evaluation of Government programs impacting Indigenous people, it should commit to action designed to establish a new and more robust evaluation framework for the Indigenous policy sector.

Breaking my rule about limiting recommendations to three, here is my list of ten recommended actions for a more robust evaluation framework in Indigenous affairs:

1.       Initiate an urgent review to take stock of all evaluations relevant to Indigenous affairs over the past five years, identify outstanding recommendations, and publish the results;

2.       Develop and publish a regular (say biannual) Evaluation Plan for  the Indigenous Affairs portfolio;

3.       Commit to publishing the terms of reference and expected timeframe for completion of all reviews and evaluations as they are initiated.

4.       Commit to establishing an Evaluation Oversight Committee which includes external members including evaluation experts and representatives of key Indigenous peak bodies, with functions which ensure it sees and comments on all evaluation Terms of Reference, receives copies of and comments on all draft reports provided to Government, and provides formal comment on the Government’s response to all evaluations.

5.       Commit to publish all evaluations within two weeks of their receipt and to publishing a formal response to the recommendations of all reviews within six weeks of receipt.

6.       Request the ANAO to undertake a regular (say every three years) meta-evaluation of the state of evaluation in the Indigenous policy sector.

7.       Commit to publishing the formal advice from the Prime Minister’s Indigenous Advisory Council to the Government.

8.       Review the regular reports oversighted by the Steering Committee for the Review of Government Service Provision and determine if the Productivity Commission ought to be granted a more autonomous role in developing the reports and in reviewing the Indigenous policy sector.

9.       Encourage the states and territories to adopt similar approaches to evaluation given that most of the programs which impact on Indigenous lives are delivered by state and local governments.

10.   Convene a national evaluation conference each year (similar to the annual native title conference) where Indigenous groups can present their perspectives on the successes and failures in government programs, and identify issues which require further evaluative attention.

I am not holding my breath on seeing these recommendations implemented. But it does seem to me that the preparedness of Government to develop a much more transparent and robust framework for the evaluation of Indigenous programs will provide a much clearer indication of which of the three interpretations discussed above best describes the reality of Indigenous public policy in Australia today.