Last week we saw the
publication of the Productivity
Commission’s Overcoming Indigenous Disadvantage (OID) 2016 Report. Predictably,
it reported mixed results. I don’t propose to summarise the contents, but focus
instead on the narrative around its release.
The Minister’s
media release emphasised the
positives, but stated that
“a great deal more needed to be done to address Indigenous
disadvantage, including building the evidence of what worked”….
…. Minister Scullion also acknowledged that in the areas
of incarceration, domestic violence, mental health and substance misuse,
increased effort was required to improve outcomes – and better evidence was
needed to drive this progress….
…. “Until recently, there has not been sufficient
investment in evidence to drive Indigenous-specific mental health and suicide
prevention responses.”….
….Minister Scullion said it was also important that
individual programmes within the Indigenous Affairs portfolio were properly
evaluated to determine their effectiveness….
… “The Coalition Government is working hard to build a
better evidence base than there has been previously for Indigenous Affairs. We
are increasing the use of quantitative data and using a variety of mechanisms
to evaluate the success of individual Indigenous Affairs programmes”.
Coincidentally, the Secretary of the Prime Minister’s
Department, Martin Parkinson in a recent
speech noted:
The current Closing the Gap framework is coming up to 10 years old and
many of the targets about to expire. We should not be fearful of where we go
from here but rather seize the enormous opportunity this presents. This is our
chance— Government and Indigenous people—to take the lessons of the past decade
and work together to reset the agenda, to focus our efforts to truly close the
gap in the outcomes for Indigenous and non-Indigenous Australians.
A high proportion of what we
fund has, at best, a weak evidence base of how it affects Indigenous peoples.
We must gather evidence which shows we are improving the lives of Indigenous
Australians. And if that evidence tells us otherwise, we must change our
approach.
We need to put our minds to many questions—what did we get right and
what did we get wrong? And why? We need to commit to the economic development
for Aboriginal and Torres Strait Islander peoples based on accumulated knowledge.
We know that the keys to self-reliance, independence and improved social
outcomes are: higher levels of employment; Indigenous business ownership; and
the opportunity to use and develop culture, knowledge and land assets to
generate wealth.
Joining the chorus, the Deputy Chair of the Productivity
Commission, Karen Chester, made a robust case for more evaluation of Government
programs on ABC Radio National’s AM program (audio
here) and in
The Australian.
Digressing briefly
, The
Australian’s
story was headlined “
Lack of account in $5.9bn spend ‘beggars
belief’”. The misleading headline managed to simultaneously imply that the
report found that there is a widespread lack of accountability for public
funding, and that the problem lies with funding recipients rather than the
governments who are responsible for the effective management and oversight of
funding programs. This inaccurate message was reinforced by the apparently misleading
text under the photo of Ms Chester (in the online story) which quotes her as
saying she is “staggered by the programs’ lack of success” whereas she is
quoted in the article as saying “
she was staggered
at the lack of attention paid to assessing what works, ….Evaluation is missing in action and it beggars belief that it’s missing in action…”
So what to make of the narrative that the problem we face
with Indigenous policy is fundamentally about insufficient evaluation?
There are at least three potential interpretations.
The
first is that the explanations offered by the Government and bureaucracy should
be taken at face value, and that once we manage to successfully evaluate the
key programs, and adjust our policy and program settings in response, we will
be back on track.
A second is that those at the centre of devising,
developing, implementing policy in relation to Indigenous affairs are working
from a set of deeply entrenched and extremely technocratic assumptions both as
to how the public policy system works and how Indigenous communities respond to
policy initiatives. In this rational and technocratic framework, policies, laws
and programs are devised to structure and if necessary change behaviour so as
to accord with a set of values and expectations largely determined by
mainstream interests (eg cutting benefits to parents whose children don’t attend
school, or income management of welfare payments to ensure expenditure is not
directed to gambling or alcohol consumption) through the imposition of
incentives both positive and negative. These types of technocratic approaches
can often miss the mark as, apart from the inevitable implementation challenges
and shortcomings, they are often perceived by Indigenous citizens as irrelevant
to Indigenous concerns, poorly targeted, culturally insensitive, externally
imposed, inappropriately coercive, or even discriminatory.
In this technocratic world of unidimensional causation
described by a process of inputs, outputs and outcomes, evaluation and remediation
is the final link in the circular process which closes the circle. We should not
be surprised then that evaluation will work from within the technocratic frame
and assumptions adopted by mainstream policymakers, and will be unlikely to
identify underlying and fundamental issues driving policy failure which lie
outside the technocratic mindset.
A third more cynical interpretation is that the Government
knows that its policies, laws and programs related to Indigenous citizens are
not working, but doesn’t understand why, or is not prepared to resource programs
effectively, or is not prepared to drive the states and territories to perform
effectively, or is focussed on politics to the exclusion of policy, or plays
favorites, or won’t take advice, or tries to be all things to all people, or
actually isn’t prepared to give the Indigenous policy sector the priority it
needs to cut through the inertia which surrounds every policy domain to a
greater or lesser extent, and so on. In this interpretation, the calls for more
evaluation are in effect a cynical distraction aimed at buying time and
political cover, along with a hope that by the time the next Overcoming
Indigenous Disadvantage Report is issued, the electorate and stakeholders will
have forgotten what was said two years previously, and in any case, life will
have moved on.
These potential interpretations raise a number of general
issues. If evaluation is so important now, why has it been neglected for the
past three years, or indeed the past ten years? What work is proposed to
understand why the focus on evaluation (particularly in Indigenous programs)
has been allowed to fade and lapse? And what about the various audits, coronial
inquests, reviews, and the like which have been undertaken over the past
decade? Did we learn nothing from them? Was the Hope
Coronial Inquiry into 22 Kimberley
suicides a waste of time? Did we fail to learn anything from the
Mullighan Royal Commission into the
abuse of children in the APY Lands? How is it that the Office of Evaluation and
Audit established under the ATSIC legislation, with an independent statutory
remit, has been dismantled, and absorbed into the ANAO? And where has the ANAO been while this
systemic ‘lack of attention’ to evaluation has emerged? And how might the
electorate and Indigenous citizens be assured that the same deterioration in
focus won’t occur in the future?
Whichever of the three interpretations one accepts, we can
expect more attention to be directed to evaluation issues by Government. Under
the first two interpretations, there will likely be an increase in the number
of evaluations, reviews, and effectiveness audits commissioned and undertaken.
Under the third, policy failure will continue, periodic disasters or crises will
emerge, and questions about policy effectiveness will continue to be raised, if
not by governments, then certainly by others.
Contrary to most accepted wisdom, I am not entirely persuaded
that more evaluation would be an unalloyed positive. Like Prospero in The Tempest, “this rough magic/ I here
abjure”. Too often, in my perhaps jaded experience, evaluative work (especially
when undertaken within the executive arm of government) is an exercise in going
through the motions, designed to provide confirmation for policies and
decisions which are based on political imperatives or ideology rather than
evidence.
In my experience, effective evaluations inevitably revolve
as much around value judgments as objective analysis, but few policymakers are
prepared to justify their decisions on values over ostensible evidence. Yet statistical
analysis can be, and often is, shaped to support (or at least not contradict) a
predetermined policy or political narrative. Moreover, apart from the
substantial cost of effective and independent evaluations, there is a real
opportunity cost both in terms of what those resources might have been directed
towards, and in terms of policymakers inwards focus into what the evaluation or
review will say rather than towards making programs and policy initiatives more
effective.
Implicit in this critique is a view that in most areas of
policy, we actually know what will deliver positive outcomes, and the challenge
is to deliver the basics, not the optimal. We know that provision of police
services will improve community safety. We know that provision of adequate housing
and reducing overcrowding will improve health and education outcomes. This is
not an argument in favour of zero evaluation, but rather is an argument in
favour of targeted and risk based evaluation activity with an eye on the
expected net benefits of the proposed evaluation. It is an argument in favour
of short and sharp evaluations designed to identify, say, the three most
significant changes that might be made to a policy, not the list of fifty (or
more) recommendations which comprise all conceivable changes. It is an argument
for a focus on policy effectiveness, and against mechanistic evaluation processes
which claim to be comprehensive, but are in reality drivers of yet more
complexity and process. It is an argument for doing good, not achieving
perfection.
One of the reasons governments call for more evaluation activity
is the fact that evaluations rarely lead to the fundamental questioning of the government
initiative under review. Indeed, in my more cynical moments, it seems to me
that an anthropologist from Mars studying our evaluation culture might easily
describe it as akin to a ceremonial ritual activity, designed to reassure all
who participate that the world as we know it will continue, that the future
though uncertain will not be disastrous, and that if we participate in the
ritual, we will all receive our just rewards, if not in this life, then in the
next. In other words, don’t rock the boat now; things will improve in the
future if only we participate in the ritual ceremony of evaluation.
Clearly I am conflicted about the merits of evaluation.
Nevertheless, given the general acceptance of the ritual of evaluation in our
public policy life, there is a case for establishing some benchmarks. In
particular, it seems incontrovertible that the quality of evaluations will
increase to the extent that there is greater transparency around its
commissioning, greater independence in the process of reaching conclusions, and
increased accountability in monitoring the implementation of the responses to
evaluations by governments.
One further point worth noting regarding the recent Overcoming
Indigenous Disadvantage reports are that they are technically not owned by the
Productivity Commission, but by a Steering Committee made up of bureaucrats
from across the states and territories as well as major federal agencies. The
same is the case for the
Indigenous Expenditure Report
produced by the Commission. While the undoubted technical capacity of the Productivity
Commission is brought to bear in producing these reports, it does mean that the
bureaucracy effectively controls and determines the way in which the reports
are framed and completed. Secondly it has the consequence that the Indigenous portfolio
is likely to receive less access to the Productivity Commission’s independent
research capacity. This is because the Commission when allocating its finite research
resources, will usually take the view that it is already allocating substantial
resources to the Indigenous sector. The recent Commission research paper on
Primary School Achievement is the
exception which proves the rule. Thus paradoxically, while the Productivity Commission
plays a crucial independent role in devising and formulating innovative policy
solutions to many of the most challenging public policy issues the nation
faces, it does not often do this in relation to Indigenous affairs. This is an
issue which requires reconsideration if we are to commit to a stronger
evaluation culture in the Indigenous policy sector.
On its face, if the Government is serious in its view about
the need for greater focus on evaluation of Government programs impacting
Indigenous people, it should commit to action designed to establish a new and
more robust evaluation framework for the Indigenous policy sector.
Breaking my rule about limiting recommendations to three, here
is my list of ten recommended actions for a more robust evaluation framework in
Indigenous affairs:
1.
Initiate an urgent review to take stock of all
evaluations relevant to Indigenous affairs over the past five years, identify
outstanding recommendations, and publish the results;
2.
Develop and publish a regular (say biannual) Evaluation
Plan for the Indigenous Affairs
portfolio;
3.
Commit to publishing the terms of reference and
expected timeframe for completion of all reviews and evaluations as they are
initiated.
4.
Commit to establishing an Evaluation Oversight
Committee which includes external members including evaluation experts and
representatives of key Indigenous peak bodies, with functions which ensure it
sees and comments on all evaluation Terms of Reference, receives copies of and
comments on all draft reports provided to Government, and provides formal
comment on the Government’s response to all evaluations.
5.
Commit to publish all evaluations within two
weeks of their receipt and to publishing a formal response to the
recommendations of all reviews within six weeks of receipt.
6.
Request the ANAO to undertake a regular (say every
three years) meta-evaluation of the state of evaluation in the Indigenous
policy sector.
7.
Commit to publishing the formal advice from the
Prime Minister’s Indigenous Advisory Council to the Government.
8.
Review the regular reports oversighted by the Steering
Committee for the Review of Government Service Provision and determine if the
Productivity Commission ought to be granted a more autonomous role in
developing the reports and in reviewing the Indigenous policy sector.
9.
Encourage the states and territories to adopt
similar approaches to evaluation given that most of the programs which impact
on Indigenous lives are delivered by state and local governments.
10.
Convene a national evaluation conference each
year (similar to the annual native title conference) where Indigenous groups
can present their perspectives on the successes and failures in government
programs, and identify issues which require further evaluative attention.
I am not holding my breath on seeing these recommendations
implemented. But it does seem to me that the preparedness of Government to
develop a much more transparent and robust framework for the evaluation of Indigenous
programs will provide a much clearer indication of which of the three interpretations
discussed above best describes the reality of Indigenous public policy in Australia
today.