Recent developments in mainstream evaluation: implications for Indigenous interests

 

Are these things then necessities?

Then let us meet them like necessities.

2 Henry IV, Act 3, scene 1

 

In this week’s budget, the Government has allocated new resources to improve the quality of public sector evaluations, and ultimately the quality of mainstream public sector investments across the board. It is a very minor outlay described in a short note on page 213 of Budget Paper 2 (link here):

$10.0 million over 4 years from 2023–24 (and $2.1 million per year ongoing) to establish a central evaluation function within Treasury to provide leadership and improve evaluation capability across Government, including support to agencies and leading a small number of flagship evaluations each year

 

It is too early to know what approach the new evaluation function (which hereafter I will refer to as an ‘Office’) will adopt, and thus how it is likely to impact Indigenous programs and thus ultimately Indigenous citizens and communities.

 

To provide some context, it is worth considering two separate developments from the last five years. The first development relates directly to evaluation.

 

In April 2019, then Treasurer Frydenberg commissioned the Productivity Commission to develop an Indigenous evaluation strategy. In his Direction Letter, the Treasurer described the scope of the proposed work in the following terms:

The Commission should develop an evaluation strategy for policies and programs affecting Indigenous Australians, to be utilised by all Australian Government agencies. As part of the strategy, the Commission should:

• establish a principles based framework for the evaluation of policies and programs affecting Indigenous Australians 

• identify priorities for evaluation 

• set out its approach for reviewing agencies' conduct of evaluations against the strategy.

 

The Productivity Commission published its report Indigenous Evaluation Strategy (link here) on 30 October 2020. The strategy was quite brief, but attached was a 500 page Background Paper.

 

On page 2, the Background Paper provides a one page summary of key points, which I have drastically cut back even further:

Key points

• Evaluation can answer questions about policy effectiveness, but both the quality and usefulness of evaluations of policies and programs affecting Aboriginal and Torres Strait Islander people are lacking…

• There is also no whole-of-government approach to evaluation priority setting. And while policy makers agree that evidence is critical for good policies, in practice there is little reliance on evaluation evidence when designing or modifying policies. 

• The Indigenous Evaluation Strategy (the Strategy) sets out a new approach. It provides a whole-of-government framework for Australian Government agencies for evaluating policies and programs affecting Aboriginal and Torres Strait Islander people. 

• The Strategy puts Aboriginal and Torres Strait Islander people at its centre….

• The Strategy’s proposed governance arrangements [which are essential architecture for an effective Strategy] include an Office of Indigenous Policy Evaluation (OIPE) and an Indigenous Evaluation Council (with all Aboriginal and Torres Strait Islander members). The OIPE and the Council would work in partnership to: monitor and report on agencies’ progress implementing the Strategy; identify evaluation priorities and potential cross-agency/topic evaluations; and provide evaluation leadership and guidance.

• A central clearinghouse for evidence on the effectiveness of policies and programs affecting Aboriginal and Torres Strait Islander people would strengthen accountability for undertaking good evaluations and improve diffusion of knowledge. 

 

While the PC’s Indigenous Evaluation Strategy is not without its flaws, its overarching message that evaluation has the potential to improve policy outcomes is certainly one worth taking seriously, particularly in circumstances where virtually everyone with any detailed knowledge of the Indigenous policy domain concludes that policy outcomes over the past few decades leave much to be desired.

 

Unfortunately, over the almost three years since the proposed Strategy was published, neither the former Government who commissioned the Productivity Commission’s work, nor the current Government have taken up the opportunity to expand the use of, and focus on, evaluation within the Indigenous policy domain, and as a consequence no government has sought to implement the recommendations included in the proposed Strategy either in part of in full.

 

This simple fact ‘bells the cat’ on the real levels of commitment by governments of all complexions to ensuring policy development and outcomes at least in the Indigenous policy domain are in fact fit for purpose. See my previous post (link here) for further detail. The deep scepticism of the Executive Branch to independent evaluation of policy and programs is driven by at least two sets of pressures:

·       for politicians, they are deeply averse to any process of independent evaluation that has the potential to provide ammunition to their political opponents (and thus to constraint the political opportunities that are available);

·       for bureaucrats, they abhor any process that expands the deeply constrained levels of transparency that ensure that the scope for flexibility in shifting administrative priorities and resource allocations incrementally is not constrained.

 

These two drivers are mutually reinforcing, and their combined impact is antithetical to effective democratic accountability and high quality governance.

 

The second contextual development was the successful negotiation of the National Agreement on Closing the Gap (link here), and in particular, the commitment of all parties, including all Australian jurisdictions to four priority reforms. Of particular importance in my mind is Priority Reform three:

58. The Parties commit to systemic and structural transformation of mainstream government organisations to improve accountability and respond to the needs of Aboriginal and Torres Strait Islander people. Priority Reform Three was strongly supported by the 2019 engagement process [emphasis added].

 

This institutional commitment is paralleled by the longstanding and inexorable increase in the comparative importance of mainstream programs to the quality of life of Indigenous citizens. One positive feature of this week’s budget was the focus on APS reform, and part of that is an explicit recognition that the National Agreement requires mainstream agencies to do better in relation to Indigenous issues and outcomes. See 2023 Budget Paper 4 at pages 8-9 (link here).

 

Given this background and context, the decision of the current Government to establish a new whole of government evaluation Office within the Treasury portfolio is to be welcomed. The Assistant Treasurer Andrew Leigh has been a longstanding advocate for greater use of evidence in policy development, particularly Randomised Control Trials.  It can be seen as a heavily cut down version of the proposal championed by Nicolas Gruen (link here) for an office of Evaluator General to be established to focus on public sector effectiveness and thus complement the focus of the ANAO in ensuring accountability and efficiency of public sector outlays (link here).

 

The challenge will be to identify a series of strategically important and high profile evaluation opportunities that lead directly to positive reform agendas. The new evaluation Office is too small to have an immediate impact, and will require time to build momentum and capability. The risk is that it will be vulnerable to abolition should it fail to demonstrate its utility in driving improved public sector outcomes. Of course, the new Office has a potential synergy with the Productivity Commission which may well be the subject of further reform over the next 12 months (link here). There may thus a strategic opportunity to build further evaluation capability across the public sector along with any future refocussing of the Productivity Commission.

 

While the ultimate shape of the new evaluation Office is still unclear, the obvious question for our purposes is what will this mean for evaluation in the Indigenous policy domain? It is too early to make definitive assessments on this issue, but it is clear that there will be a number of challenges over and above those facing mainstream agencies and policy sectors.

 

The first relates to the issue of involving Indigenous interests in evaluation activities that impact them. This becomes difficult when the subject of the evaluation is a mainstream program, especially if the data available is constrained. A second challenge is the extent to which Commonwealth policies and programs are implemented and delivered by the states, a trend reinforced by the policy architecture embedded within the National Agreement on Closing the Gap. 

No doubt there will be other issues that may arise: will the new evaluation Office differentiate between policy and programs; will the new Office have the capacity to evaluate the effectiveness of tax expenditures; what will be the relationship with the ANAO and other relevant agencies, and what will the coordination and working arrangements look like. All of these issues, and how they are decided, will impact Indigenous interests, for better or worse, but they will in very large measure be intangible impacts difficult to ascertain without the sort of close consideration that high quality evaluations might bring.

 

As someone who has taken an interest in the evaluation of Indigenous policy over thirty years, it seems useful if I lay out succinctly the key principles and policies that I would hope the new evaluation Office adopts in relation to evaluation in general and in particular in relation to the Indigenous policy domain. These are not intended to be comprehensive, but I do consider that they are among the necessary principles required to ensure mainstream evaluations make a positive contribution to the quality of policy outcomes, and importantly, to identifying potential reform agendas.

 

First, I would hope that the new evaluation Office adopts a rigorous and focussed approach to oversighting the quality of the entire approach to evaluation of each major agency. There is a potential overlap here with the role of the Department of Finance. The Office should develop a checklist of core competencies and create an overarching and publicly available data base of agency performance against a small number of core competencies. Of course, it will not be in a position to assess each agency every year, but as it does make assessments, they should be added to the data base and be published on the evaluation Office’s web site.

 

Second, I would hope that the new evaluation Office gives particular priority to assessing the independence of evaluations undertaken by agencies, rating them on a consistent scale.

 

Third, I would hope that the evaluation Office develops a mechanism which allows it to assess the extent of an agencies policy and program footprint that has been subject to evaluation over the past ten or fifteen years. See the discussion at pages 208-9 of the Indigenous Evaluation Strategy Background Paper.

 

Fourth, I would hope that the evaluation Office makes an assessment of each agency’s commitment to ensuring that ‘evaluation relevant’ data is being collected and ideally progressively made public. In the Indigenous space for example, it is now eight years since the last National Aboriginal and Torres Strait Islander Social Survey was undertaken by the ABS (link here). The funding of the NATSIS is contributed by relevant policy agencies, and a document released by the ABS in 2017 indicated that the aim was to undertake the survey every six years. Unfortunately, this has not occurred and there appears to be no current plans to undertake another version (link here).

 

Fifth, I would hope that the evaluation Office assesses and reports on the comparative performance of agency transparency in relation to publishing evaluation results in a timely fashion.

 

The Government has identified that the new evaluation Office will undertake a small number of flagship evaluations each year.  There will be a risk that the new Office gets lost in the detail of particular exercises, and ends up keeping busy on evaluations and issues that are tangential to the overall effectiveness of the Commonwealth’s evaluation efforts. In my view, it is this latter objective that should be the primary focus for the new Office. This is the real necessity. If the Office performs that task well, it will create substantial incentives for agencies, including the NIAA and its associated entities, to undertake policy and program evaluations in a much more effective manner than has been the case over the past few decades.

 

Addendum

For those interested, I have previously published posts on this blog related to evaluation issues (link here, link here, link here, and link here).

A more comprehensive conceptual treatment of the link between evaluation and policy can be found in my policy insights paper, Evaluation and review as drivers of reform in the Indigenous policy domain, available on the CAEPR, ANU website (link here).