I want to cover three issues in this post. First, provide a
very brief and selective account of the history of evaluation in relation to
Commonwealth policies and programs. Second, to unpick some of the issues raised
in the recent ANAO performance audit on ‘Evaluating Aboriginal and Torres
Strait Islander Programs’ (link
here). Third, to make some preliminary or provisional comments in relation
to the Productivity Commission’s project to develop a whole of government
Evaluation Strategy (link
here). Paragraph numbers in square brackets refer to the ANAO report.
The Indigenous policy domain has a long history of concerns
over accountability, waste and effectiveness that reaches back to the very
beginnings of the Commonwealth’s decision to take on an active role in
Indigenous affairs in the late 1960s. This in turn led to periodic bouts of
intense media and political scrutiny into the propriety and effectiveness of
public funding for Indigenous affairs. For example, a series of dramatic
allegations (never confirmed) were aired in the Senate in 1988 alleging the
existence of a ‘black mafia’ which in turn led to a Special Audit Reports into
the Department of Aboriginal Affairs and the Aboriginal Development Commission
by the Auditor General in March and October 1989. In turn, a desire to pre-empt
and counter this sort of political pressure led to the establishment within the
ATSIC legislation of the Office of Evaluation and Audit (OEA) headed by a
ministerially appointed Director to oversight ATSIC’s programs and report to
both the Commission and the Minister. After the abolition of ATSIC, OEA was shifted
to, and eventually subsumed within, the ANAO.
The establishment of OEA broadly coincided with the
adoption of program performance budgeting by the Commonwealth, which in turn
led to an increased focus on performance monitoring and evaluation across the
public sector. Since the abolition of ATSIC around 2005, the Indigenous Affairs
portfolio has lodged in a number of key portfolios (DIMIA; FACS; FaHCSIA) each
with their own program management and evaluation cultures, before finally
coming to rest in the Prime Minster & Cabinet portfolio since 2013.
While it is arguable that more onerous standards of
accountability have been applied to Indigenous programs over the past 30 years,
it is certainly the case that concepts such as accountability, value for money,
efficiency and effectiveness have been ongoing benchmarks (if only internally
to government) in the administration of Indigenous affairs programs and
policies. It is however almost 15 years since the last major accountability
controversy in the Indigenous affairs portfolio which related to the activities
of a number of senior ATSIC leaders, and which created the opportunity for
ATSIC to be abolished by the Howard Government. For those interested in this
history, I refer you to pages 27 – 30 in the recent CAEPR Policy Insights paper
I co-authored with Neil Westbury (link
here).Consequently, it is not surprising that the substantive (as opposed
to rhetorical ) focus on accountability has diminished over time. In turn, the
impetus for effective and substantive evaluation has diminished, even as the
prevalence of rhetoric advocating greater use of evaluation has increased.
At the rhetorical level, there have been numerous reports
and speeches by public sector leaders advocating greater use of evaluation as
the solution to ongoing policy failure in Indigenous affairs. Thus in 2009, the
Department of Finance and Deregulation’s Strategic Review of Indigenous
Expenditure ‘stressed the need for a more rigorous approach to program
evaluation at a whole of government level’ [para 1.4, p14].
In August 2016, in the Dungala Kaiela Oration, the
Secretary of PMC, Martin Parkinson (link
here and link
here) argued
A high proportion of what we
fund has, at best, a weak evidence base of how it affects Indigenous peoples.
We must gather evidence which shows we are improving the lives of Indigenous
Australians. And if that evidence tells us otherwise, we must change our
approach.
In December 2016 in a speech titled ‘Public policy and
whats missing in action?’ (link
here), the Deputy Chair of the Productivity Commission, Karen Chester,
argued that:
While there is extensive
reporting on the extent of
Indigenous disadvantage, there is a lack of evidence about what works (and what
does not) in bridging outcome gaps. And while evaluating the impacts of
policies on Indigenous outcomes can be challenging, the challenges should not
be seen as insurmountable….Stop looking for silver bullets and policy sound
bites. And just get back to the dirt under the fingernail work of building
evidence-based policy and building a much stronger evaluation culture (we need
to know more about what works and why).
These views, which have an attractive internal logic, implicitly
assume that Government programs have straightforward objectives and that if only
we find appropriate or effective policy incentives, Indigenous behaviours will
change in ways which lead to improved socio-economic outcomes. Both these
assumptions are problematic in cross cultural contexts and certainly not necessarily the case in the
context of the Indigenous domain. Moreover, even if policies were technically
able to improve outcomes, robust policy evaluations inevitably come into conflict
with the deep-seated political imperatives to frame policies and programs to
advance political objectives and also to avoid any admission of failure that
might create political embarrassment for Ministers or the Government.
Thus notwithstanding the continuous stream of
pro-evaluation rhetoric from public service leaders, the recent ANAO
performance audit ‘Evaluating Aboriginal and Torres Strait Islander Programs’
outlines a sorry history of desultory performance by the Department of Prime
Minister and Cabinet (DPMC) and its Indigenous Affairs Group (IAG). These
results appear to derive from policymakers prioritisation of the perception of
action over substantive action. Below, I selectively point to a number of
issues raised by the ANAO audit which reinforce this conclusion. I recommend
those interested read the whole report (link
here).
The
ANAO performance audit
The ANAO performance audit was aimed not at assessing the
effectiveness of DPMC’s evaluations, but at assessing the effectiveness of the
design and implementation of the department’s evaluation framework for
the IAS [para. 6]. This more limited objective was arguably a mistake,
since even a wholly effective design and implementation of the framework is
merely a necessary but not sufficient prerequisite for successful
evaluation of the IAS. Nevertheless, the audit does uncover and reveal a wide
range of useful information about the DPMC approach and commitment to
comprehensive evaluation of not only the IAS (Its most significant program) but
to Indigenous related policies and programs more generally.
In chapter one, the ANAO outlined the background to the
audit. In 2017, the ANAO performance audit of the Indigenous Advancement
Strategy (IAS) (the largest program in the Indigenous affairs portfolio
comprising five program streams and $4.8bn over four years) ‘found the Department
did not effectively implement the strategy’ [Para 1.9]. See my previous blog
post on the IAS audit (link
here). In response, in February 2017 the then Minister announced
funding of $40m over four years to strengthen IAS evaluation in the portfolio [para
1.10]. In February 2018, the Department released an IAS Evaluation
Framework [para.1.12]. IAG program areas are responsible for conducting
the majority of program evaluations under the framework.
In chapter two, the performance audit outlines significant
delays in establishing the evaluation framework for the IAS, extending from
March 2014 through to February 2018 [Figure 2.1, para 2.1]. The ANAO
note that the IAS evaluation framework does not include references to the Government’s
higher level objectives including the Closing the Gap framework [paras 2.26,
2.29]. The ANAO also note that the IAS evaluation framework ‘is intended to
align with the wider role of the Productivity Commission in overseeing the
development and implementation of a whole of government evaluation strategy of
policies and programs that effect [sic] Indigenous Australians’. However
because of delays in passing legislation and appointing a Commissioner, the
Productivity Commission did not begin work until April 2019 and it won’t be
complete until July 2020 [para. 2.31].
In chapter three, the ANAO assess implementation and management
fo the IAS Framework. The ANAO note that the framework includes various
commitments for activities designed to strengthen evaluation quality, but
without any timeframes specified for implementation [para.3.3].The ANAO
concluded that the performance targets for the IAS evaluation framework were
focussed on the delivery of short term outputs (ie not longer term outcomes).
The audit’s first recommendation [para 3.12] was that the Department
ensure its performance information for evaluations are supported by a reliable
methodology for measuring longer term outcomes. The audit’s second
recommendation is largely process related [para 3.22]. The rest of the
chapter is concerned with management oversight arrangements for evaluations. In
a surprising and rather damning assessment, the ANAO found that there was
limited evidence that the IAG Executive Board were involved in the oversight of
the evaluation activities [para.3.31]. The Indigenous Evaluation
Committee (which included some external members) raised concerns in late 2018
that a number of completed evaluations had not been published [para. 3.33].
The ANAO also noted delays in the Department developing effective mechanisms to
track evaluation recommendations and management responses [paras.3.36-3.40].
In chapter four, which considers the application of the
framework, the ANAO conclude that ‘As the department is still developing
procedures to support the application of the IAS evaluation framework, it is
too early to assess whether evaluations are being conducted in accordance with
the framework’ [Box, page 37]. After assessing the Department’s
processes for prioritising and selecting areas for evaluation, the ANAO
recommended that the department ‘formalise its evaluation prioritisation
process by developing structured criteria for assessing significance,
contribution and risk and conducting a strategic analysis of gaps in evaluation
coverage’ [para 4.10]. While the Department indicated it agreed with the
recommendation, its response is so heavily caveated as to be meaningless.
Moreover, it made no reference or commitment to undertaking a strategic gaps
analysis as explicitly recommended [para 4.11].
Chapter four also included discussion of the desirability
of independence within evaluation teams [paras 4.25 to 4.28]. The
discussion is extraordinarily hedged and timid, and failed to make a
substantive assessment of the actual independence of the evaluations
undertaken. There is also a discussion of the issue of publication of completed
evaluations and management responses [paras 4.37 to 4.41. See also appendix
4]. Of 35 evaluations on the Department’s 2018-19 workplan, 15 had not
commenced. Of the remaining 20, 8 had been published and 12 withheld from
publication. The ANAO discuss the reasons for non-publication. In at least four
cases (involving very significant and sensitive program evaluations) the
Department was waiting to brief the Minister or awaiting his noting of a brief
[para 4.38; appendix 4]. In plain language, the Minister was preventing
timely publication of the evaluations.
I found this performance audit disappointing for two
reasons. First, it was clear from the inexorable delays and bureaucratic
fudging – much focussed merely on the development of an evaluation framework
against which actual evaluations might be assessed – that the Department (and
the Minister – see para 2.7) were not prioritising evaluation activities.
Primary responsibility rests with senior management, as there are indications
throughout the audit that middle level officers were seeking to advance the
agenda, but kept hitting blockages above. If one assumes that the senior management
of the DPMC are competent (Which I do), then it is difficult to explain the
disjunction between the rhetoric of the Secretary in 2016 and the appalling
delays in making substantive progress between 2014 and 2019 as anything but a
process of bureaucratic game playing. In particular, the bureaucratic
imperative appears to be to contrive circumstances where the department is
always appearing to be moving forward, but never reaches a destination.
The second reason the audit is disappointing is that the
ANAO itself appears to have gone soft. It describes the audit as a performance
audit, but it fails to robustly assess poor performance. Its focus on the
framework rather than evaluations themselves is highly process oriented. There
are only three recommendations, all of which are process related; this is a
major flaw in the performance audit as it effectively leaves the DPMC off the
hook notwithstanding the documented lack of substantive commitment to
progressing an effective and substantive evaluation agenda. The discussion on
independence of evaluations has all the robustness of a marshmallow. The
discussion on publication and management responses is slightly better, more
akin to fudge. Where are the robust recommendations on publishing evaluations
undertaken with taxpayers’ funds, and ensuring that evaluations are not subject
to being diluted through political interference, second-guessing, concern about
vested interests and the like. We are told in para 3.10 of an instance where
the Department agreed to an internal audit recommendation designed to
strengthen evaluation activity, but then did not implement the recommendation.
Given this behaviour, how are we believe that the Department will take any of
the recommendations or less formal suggestions included in this report
seriously?
Our scepticism is reinforced by the fact that notwithstanding
the consistent rhetoric about the merits of evaluation of Indigenous programs
for over a decade, the senior management of the nation’s premier department of
state has been shown to be incapable of implementing an effective evaluation
program. How are we to respond to the disjunction between the ANAO conclusion [in
para.10]: ‘The department’s
implementation and management of the IAS evaluation framework is partially
effective’ (a generous finding given the actual content of the audit report and
the limited results to date) and the Secretary’s letter in response in Appendix
1 which fails to acknowledge or address in any way the negative content of
the audit. This deliberate blindness ought to be a matter of concern to the Minister,
to the Government, and to First Nations peoples.
The Productivity Commission Indigenous
Evaluation Project
In April 2019, the Treasurer requested the Productivity
Commission (PC) to ‘develop a whole of Government Evaluation Strategy for
policies and programs affecting Indigenous Australians’. The scope of the
strategy includes a principles based framework for the evaluation of policies
and programs affecting Indigenous Australians, identification of priorities for
evaluation, and a description of the Productivity Commission’s approach for
reviewing agencies conduct of evaluations against the strategy. The Treasurer’s
Letter of Direction plus a recently released Issues Paper are available on the
Productivity Commission’s website (link here).
The Issues Paper appears comprehensive and well structured.
I don’t propose to summarise or discuss it at this point in time.
There are however a number of interesting issues embedded
in the these developments. These include:
·
While the PC undoubtedly has substantial
evaluation expertise, it is unclear how its proposed role in monitoring the
effectiveness of programs will relate to the virtually identical role recently
allocated to the new NIAA (see my previous post link
here);
·
It is unclear whether the Treasurer was
requesting advice (which is allowed under the PC’s legislation) or directing
the PC to undertake an executive action (which does not appear to be within its
statutory remit). To the extent that it is the latter, there are potential
conflict of interest issues in the PC subsequently assessing and evaluation
agency programs and policies utilising an evaluation framework it has developed
and signed off on without any external approval;
·
It is also unclear what this new whole of government
evaluation strategy/framework will mean for existing evaluation frameworks
within agencies (including DPMC). There appears to be scope for substantial
duplication, and there is also the risk or likelihood that the DPMC evaluation
framework for the IAS, which took for excruciatingly slow years to be
developed, and was audited by the ANAO at a cost of $335k, will be set aside in
favour of the new whole of government evaluation framework to be promulgated in
2020;
·
Finally, there is a larger risk embedded in the
establishment of an Indigenous Commissioner within the PC with a substantial
role in oversighting the evaluation efforts of Commonwealth agencies related to
Indigenous peoples. This larger risk is that the PC will feel disinclined in
its mainstream inquiries and review to seriously consider policy issues related
to the impact of mainstream programs on Indigenous peoples.
While the involvement of the Productivity Commission in
Indigenous evaluation is in my view very positive, it is not without risk and
challenges. To date, there has been virtually no public discussion of the
benefits, risks, and challenges of the expanded role for the Productivity
Commission in the Indigenous policy domain. For example, the proposed new
arrangements do not fundamentally address the lack of independence inherent in
program and policy agencies commissioning evaluations. The Commission’s current
project is constrained by the remit contained in the Treasurer’s direction.
There is a case for a broader assessment of the structure of evaluation in the Indigenous
policy domain (and probably beyond). For example, there appears to be a strong
case for the re-establishment of an independent statutory agency dedicated to undertaking
key Indigenous evaluations such as previously existed with OEA.
In conclusion, the Government’s
decision to bring the Productivity Commission into the policy and program
evaluation space is welcome. It raises a number of issues, risks and challenges,
including potential overlaps with the newly promulgated role of the National
Indigenous Australians Agency. While a PC oversight role is welcome, it may not
go far enough towards ensuring evaluations are consistently independent and
free from political and other bias.
In relation to the DPMC (and from 1 July 2019 the NIAA),
the ANAO ought to have recommended a major upgrade to the resourcing and
management commitment to evaluation of policies and programs. There appears to have
been a significant capability deficit in relation to building a robust and
independent culture of evaluation. This will need to be a significant focus of
the Minister and NIAA senior management going forward.
In relation to the ANAO, it seems to me it is time for an
independent evaluation of the ANAO’s performance audit capabilities and
performance. Perhaps that is something the Productivity Commission might take
on as part of its priority setting for the new Indigenous Evaluation Strategy!
After all, mainstream programs are increasingly impacting on First Nations, and
the effectiveness of those programs and their oversight arrangements is a
matter highly relevant to vulnerable people, including First Nations.
No comments:
Post a Comment