Monday, 1 July 2019

Marshmallow and fudge: evaluation and the Indigenous policy domain





I want to cover three issues in this post. First, provide a very brief and selective account of the history of evaluation in relation to Commonwealth policies and programs. Second, to unpick some of the issues raised in the recent ANAO performance audit on ‘Evaluating Aboriginal and Torres Strait Islander Programs’ (link here). Third, to make some preliminary or provisional comments in relation to the Productivity Commission’s project to develop a whole of government Evaluation Strategy (link here). Paragraph numbers in square brackets refer to the ANAO report.

The Indigenous policy domain has a long history of concerns over accountability, waste and effectiveness that reaches back to the very beginnings of the Commonwealth’s decision to take on an active role in Indigenous affairs in the late 1960s. This in turn led to periodic bouts of intense media and political scrutiny into the propriety and effectiveness of public funding for Indigenous affairs. For example, a series of dramatic allegations (never confirmed) were aired in the Senate in 1988 alleging the existence of a ‘black mafia’ which in turn led to a Special Audit Reports into the Department of Aboriginal Affairs and the Aboriginal Development Commission by the Auditor General in March and October 1989. In turn, a desire to pre-empt and counter this sort of political pressure led to the establishment within the ATSIC legislation of the Office of Evaluation and Audit (OEA) headed by a ministerially appointed Director to oversight ATSIC’s programs and report to both the Commission and the Minister. After the abolition of ATSIC, OEA was shifted to, and eventually subsumed within, the ANAO.

The establishment of OEA broadly coincided with the adoption of program performance budgeting by the Commonwealth, which in turn led to an increased focus on performance monitoring and evaluation across the public sector. Since the abolition of ATSIC around 2005, the Indigenous Affairs portfolio has lodged in a number of key portfolios (DIMIA; FACS; FaHCSIA) each with their own program management and evaluation cultures, before finally coming to rest in the Prime Minster & Cabinet portfolio since 2013.

While it is arguable that more onerous standards of accountability have been applied to Indigenous programs over the past 30 years, it is certainly the case that concepts such as accountability, value for money, efficiency and effectiveness have been ongoing benchmarks (if only internally to government) in the administration of Indigenous affairs programs and policies. It is however almost 15 years since the last major accountability controversy in the Indigenous affairs portfolio which related to the activities of a number of senior ATSIC leaders, and which created the opportunity for ATSIC to be abolished by the Howard Government. For those interested in this history, I refer you to pages 27 – 30 in the recent CAEPR Policy Insights paper I co-authored with Neil Westbury (link here).Consequently, it is not surprising that the substantive (as opposed to rhetorical ) focus on accountability has diminished over time. In turn, the impetus for effective and substantive evaluation has diminished, even as the prevalence of rhetoric advocating greater use of evaluation has increased.

At the rhetorical level, there have been numerous reports and speeches by public sector leaders advocating greater use of evaluation as the solution to ongoing policy failure in Indigenous affairs. Thus in 2009, the Department of Finance and Deregulation’s Strategic Review of Indigenous Expenditure ‘stressed the need for a more rigorous approach to program evaluation at a whole of government level’ [para 1.4, p14].

In August 2016, in the Dungala Kaiela Oration, the Secretary of PMC, Martin Parkinson (link here and link here)  argued
A high proportion of what we fund has, at best, a weak evidence base of how it affects Indigenous peoples. We must gather evidence which shows we are improving the lives of Indigenous Australians. And if that evidence tells us otherwise, we must change our approach.

In December 2016 in a speech titled ‘Public policy and whats missing in action?’ (link here), the Deputy Chair of the Productivity Commission, Karen Chester, argued that:
While there is extensive reporting on the extent of Indigenous disadvantage, there is a lack of evidence about what works (and what does not) in bridging outcome gaps. And while evaluating the impacts of policies on Indigenous outcomes can be challenging, the challenges should not be seen as insurmountable….Stop looking for silver bullets and policy sound bites. And just get back to the dirt under the fingernail work of building evidence-based policy and building a much stronger evaluation culture (we need to know more about what works and why).

These views, which have an attractive internal logic, implicitly assume that Government programs have straightforward objectives and that if only we find appropriate or effective policy incentives, Indigenous behaviours will change in ways which lead to improved socio-economic outcomes. Both these assumptions are problematic in cross cultural contexts  and certainly not necessarily the case in the context of the Indigenous domain. Moreover, even if policies were technically able to improve outcomes, robust policy evaluations inevitably come into conflict with the deep-seated political imperatives to frame policies and programs to advance political objectives and also to avoid any admission of failure that might create political embarrassment for Ministers or the Government.

Thus notwithstanding the continuous stream of pro-evaluation rhetoric from public service leaders, the recent ANAO performance audit ‘Evaluating Aboriginal and Torres Strait Islander Programs’ outlines a sorry history of desultory performance by the Department of Prime Minister and Cabinet (DPMC) and its Indigenous Affairs Group (IAG). These results appear to derive from policymakers prioritisation of the perception of action over substantive action. Below, I selectively point to a number of issues raised by the ANAO audit which reinforce this conclusion. I recommend those interested read the whole report (link here).

The ANAO performance audit

The ANAO performance audit was aimed not at assessing the effectiveness of DPMC’s evaluations, but at assessing the effectiveness of the design and implementation of the department’s evaluation framework for the IAS [para. 6]. This more limited objective was arguably a mistake, since even a wholly effective design and implementation of the framework is merely a necessary but not sufficient prerequisite for successful evaluation of the IAS. Nevertheless, the audit does uncover and reveal a wide range of useful information about the DPMC approach and commitment to comprehensive evaluation of not only the IAS (Its most significant program) but to Indigenous related policies and programs more generally.

In chapter one, the ANAO outlined the background to the audit. In 2017, the ANAO performance audit of the Indigenous Advancement Strategy (IAS) (the largest program in the Indigenous affairs portfolio comprising five program streams and $4.8bn over four years) ‘found the Department did not effectively implement the strategy’ [Para 1.9]. See my previous blog post on the IAS audit (link here). In response, in February 2017 the then Minister announced funding of $40m over four years to strengthen IAS evaluation in the portfolio [para 1.10]. In February 2018, the Department released an IAS Evaluation Framework [para.1.12]. IAG program areas are responsible for conducting the majority of program evaluations under the framework.

In chapter two, the performance audit outlines significant delays in establishing the evaluation framework for the IAS, extending from March 2014 through to February 2018 [Figure 2.1, para 2.1]. The ANAO note that the IAS evaluation framework does not include references to the Government’s higher level objectives including the Closing the Gap framework [paras 2.26, 2.29]. The ANAO also note that the IAS evaluation framework ‘is intended to align with the wider role of the Productivity Commission in overseeing the development and implementation of a whole of government evaluation strategy of policies and programs that effect [sic] Indigenous Australians’. However because of delays in passing legislation and appointing a Commissioner, the Productivity Commission did not begin work until April 2019 and it won’t be complete until July 2020 [para. 2.31].

In chapter three, the ANAO assess implementation and management fo the IAS Framework. The ANAO note that the framework includes various commitments for activities designed to strengthen evaluation quality, but without any timeframes specified for implementation [para.3.3].The ANAO concluded that the performance targets for the IAS evaluation framework were focussed on the delivery of short term outputs (ie not longer term outcomes). The audit’s first recommendation [para 3.12] was that the Department ensure its performance information for evaluations are supported by a reliable methodology for measuring longer term outcomes. The audit’s second recommendation is largely process related [para 3.22]. The rest of the chapter is concerned with management oversight arrangements for evaluations. In a surprising and rather damning assessment, the ANAO found that there was limited evidence that the IAG Executive Board were involved in the oversight of the evaluation activities [para.3.31]. The Indigenous Evaluation Committee (which included some external members) raised concerns in late 2018 that a number of completed evaluations had not been published [para. 3.33]. The ANAO also noted delays in the Department developing effective mechanisms to track evaluation recommendations and management responses [paras.3.36-3.40].

In chapter four, which considers the application of the framework, the ANAO conclude that ‘As the department is still developing procedures to support the application of the IAS evaluation framework, it is too early to assess whether evaluations are being conducted in accordance with the framework’ [Box, page 37]. After assessing the Department’s processes for prioritising and selecting areas for evaluation, the ANAO recommended that the department ‘formalise its evaluation prioritisation process by developing structured criteria for assessing significance, contribution and risk and conducting a strategic analysis of gaps in evaluation coverage’ [para 4.10]. While the Department indicated it agreed with the recommendation, its response is so heavily caveated as to be meaningless. Moreover, it made no reference or commitment to undertaking a strategic gaps analysis as explicitly recommended [para 4.11].

Chapter four also included discussion of the desirability of independence within evaluation teams [paras 4.25 to 4.28]. The discussion is extraordinarily hedged and timid, and failed to make a substantive assessment of the actual independence of the evaluations undertaken. There is also a discussion of the issue of publication of completed evaluations and management responses [paras 4.37 to 4.41. See also appendix 4]. Of 35 evaluations on the Department’s 2018-19 workplan, 15 had not commenced. Of the remaining 20, 8 had been published and 12 withheld from publication. The ANAO discuss the reasons for non-publication. In at least four cases (involving very significant and sensitive program evaluations) the Department was waiting to brief the Minister or awaiting his noting of a brief [para 4.38; appendix 4]. In plain language, the Minister was preventing timely publication of the evaluations.

I found this performance audit disappointing for two reasons. First, it was clear from the inexorable delays and bureaucratic fudging – much focussed merely on the development of an evaluation framework against which actual evaluations might be assessed – that the Department (and the Minister – see para 2.7) were not prioritising evaluation activities. Primary responsibility rests with senior management, as there are indications throughout the audit that middle level officers were seeking to advance the agenda, but kept hitting blockages above. If one assumes that the senior management of the DPMC are competent (Which I do), then it is difficult to explain the disjunction between the rhetoric of the Secretary in 2016 and the appalling delays in making substantive progress between 2014 and 2019 as anything but a process of bureaucratic game playing. In particular, the bureaucratic imperative appears to be to contrive circumstances where the department is always appearing to be moving forward, but never reaches a destination.

The second reason the audit is disappointing is that the ANAO itself appears to have gone soft. It describes the audit as a performance audit, but it fails to robustly assess poor performance. Its focus on the framework rather than evaluations themselves is highly process oriented. There are only three recommendations, all of which are process related; this is a major flaw in the performance audit as it effectively leaves the DPMC off the hook notwithstanding the documented lack of substantive commitment to progressing an effective and substantive evaluation agenda. The discussion on independence of evaluations has all the robustness of a marshmallow. The discussion on publication and management responses is slightly better, more akin to fudge. Where are the robust recommendations on publishing evaluations undertaken with taxpayers’ funds, and ensuring that evaluations are not subject to being diluted through political interference, second-guessing, concern about vested interests and the like. We are told in para 3.10 of an instance where the Department agreed to an internal audit recommendation designed to strengthen evaluation activity, but then did not implement the recommendation. Given this behaviour, how are we believe that the Department will take any of the recommendations or less formal suggestions included in this report seriously?

Our scepticism is reinforced by the fact that notwithstanding the consistent rhetoric about the merits of evaluation of Indigenous programs for over a decade, the senior management of the nation’s premier department of state has been shown to be incapable of implementing an effective evaluation program. How are we to respond to the disjunction between the ANAO conclusion [in para.10]: ‘The department’s implementation and management of the IAS evaluation framework is partially effective’ (a generous finding given the actual content of the audit report and the limited results to date) and the Secretary’s letter in response in Appendix 1 which fails to acknowledge or address in any way the negative content of the audit. This deliberate blindness ought to be a matter of concern to the Minister, to the Government, and to First Nations peoples.

The Productivity Commission Indigenous Evaluation Project

In April 2019, the Treasurer requested the Productivity Commission (PC) to ‘develop a whole of Government Evaluation Strategy for policies and programs affecting Indigenous Australians’. The scope of the strategy includes a principles based framework for the evaluation of policies and programs affecting Indigenous Australians, identification of priorities for evaluation, and a description of the Productivity Commission’s approach for reviewing agencies conduct of evaluations against the strategy. The Treasurer’s Letter of Direction plus a recently released Issues Paper are available on the Productivity Commission’s website (link here).

The Issues Paper appears comprehensive and well structured. I don’t propose to summarise or discuss it at this point in time.

There are however a number of interesting issues embedded in the these developments. These include:

·         While the PC undoubtedly has substantial evaluation expertise, it is unclear how its proposed role in monitoring the effectiveness of programs will relate to the virtually identical role recently allocated to the new NIAA (see my previous post link here);

·         It is unclear whether the Treasurer was requesting advice (which is allowed under the PC’s legislation) or directing the PC to undertake an executive action (which does not appear to be within its statutory remit). To the extent that it is the latter, there are potential conflict of interest issues in the PC subsequently assessing and evaluation agency programs and policies utilising an evaluation framework it has developed and signed off on without any external approval;

·         It is also unclear what this new whole of government evaluation strategy/framework will mean for existing evaluation frameworks within agencies (including DPMC). There appears to be scope for substantial duplication, and there is also the risk or likelihood that the DPMC evaluation framework for the IAS, which took for excruciatingly slow years to be developed, and was audited by the ANAO at a cost of $335k, will be set aside in favour of the new whole of government evaluation framework to be promulgated in 2020;

·         Finally, there is a larger risk embedded in the establishment of an Indigenous Commissioner within the PC with a substantial role in oversighting the evaluation efforts of Commonwealth agencies related to Indigenous peoples. This larger risk is that the PC will feel disinclined in its mainstream inquiries and review to seriously consider policy issues related to the impact of mainstream programs on Indigenous peoples.


While the involvement of the Productivity Commission in Indigenous evaluation is in my view very positive, it is not without risk and challenges. To date, there has been virtually no public discussion of the benefits, risks, and challenges of the expanded role for the Productivity Commission in the Indigenous policy domain. For example, the proposed new arrangements do not fundamentally address the lack of independence inherent in program and policy agencies commissioning evaluations. The Commission’s current project is constrained by the remit contained in the Treasurer’s direction. There is a case for a broader assessment of the structure of evaluation in the Indigenous policy domain (and probably beyond). For example, there appears to be a strong case for the re-establishment of an independent statutory agency dedicated to undertaking key Indigenous evaluations such as previously existed with OEA.

In conclusion, the Government’s decision to bring the Productivity Commission into the policy and program evaluation space is welcome. It raises a number of issues, risks and challenges, including potential overlaps with the newly promulgated role of the National Indigenous Australians Agency. While a PC oversight role is welcome, it may not go far enough towards ensuring evaluations are consistently independent and free from political and other bias.

In relation to the DPMC (and from 1 July 2019 the NIAA), the ANAO ought to have recommended a major upgrade to the resourcing and management commitment to evaluation of policies and programs. There appears to have been a significant capability deficit in relation to building a robust and independent culture of evaluation. This will need to be a significant focus of the Minister and NIAA senior management going forward.

In relation to the ANAO, it seems to me it is time for an independent evaluation of the ANAO’s performance audit capabilities and performance. Perhaps that is something the Productivity Commission might take on as part of its priority setting for the new Indigenous Evaluation Strategy! After all, mainstream programs are increasingly impacting on First Nations, and the effectiveness of those programs and their oversight arrangements is a matter highly relevant to vulnerable people, including First Nations.

No comments:

Post a comment