Recent developments in mainstream evaluation: implications for Indigenous interests

 

Are these things then necessities?

Then let us meet them like necessities.

2 Henry IV, Act 3, scene 1

 

In this week’s budget, the Government has allocated new resources to improve the quality of public sector evaluations, and ultimately the quality of mainstream public sector investments across the board. It is a very minor outlay described in a short note on page 213 of Budget Paper 2 (link here):

$10.0 million over 4 years from 2023–24 (and $2.1 million per year ongoing) to establish a central evaluation function within Treasury to provide leadership and improve evaluation capability across Government, including support to agencies and leading a small number of flagship evaluations each year

 

It is too early to know what approach the new evaluation function (which hereafter I will refer to as an ‘Office’) will adopt, and thus how it is likely to impact Indigenous programs and thus ultimately Indigenous citizens and communities.

 

To provide some context, it is worth considering two separate developments from the last five years. The first development relates directly to evaluation.

 

In April 2019, then Treasurer Frydenberg commissioned the Productivity Commission to develop an Indigenous evaluation strategy. In his Direction Letter, the Treasurer described the scope of the proposed work in the following terms:

The Commission should develop an evaluation strategy for policies and programs affecting Indigenous Australians, to be utilised by all Australian Government agencies. As part of the strategy, the Commission should:

• establish a principles based framework for the evaluation of policies and programs affecting Indigenous Australians 

• identify priorities for evaluation 

• set out its approach for reviewing agencies' conduct of evaluations against the strategy.

 

The Productivity Commission published its report Indigenous Evaluation Strategy (link here) on 30 October 2020. The strategy was quite brief, but attached was a 500 page Background Paper.

 

On page 2, the Background Paper provides a one page summary of key points, which I have drastically cut back even further:

Key points

• Evaluation can answer questions about policy effectiveness, but both the quality and usefulness of evaluations of policies and programs affecting Aboriginal and Torres Strait Islander people are lacking…

• There is also no whole-of-government approach to evaluation priority setting. And while policy makers agree that evidence is critical for good policies, in practice there is little reliance on evaluation evidence when designing or modifying policies. 

• The Indigenous Evaluation Strategy (the Strategy) sets out a new approach. It provides a whole-of-government framework for Australian Government agencies for evaluating policies and programs affecting Aboriginal and Torres Strait Islander people. 

• The Strategy puts Aboriginal and Torres Strait Islander people at its centre….

• The Strategy’s proposed governance arrangements [which are essential architecture for an effective Strategy] include an Office of Indigenous Policy Evaluation (OIPE) and an Indigenous Evaluation Council (with all Aboriginal and Torres Strait Islander members). The OIPE and the Council would work in partnership to: monitor and report on agencies’ progress implementing the Strategy; identify evaluation priorities and potential cross-agency/topic evaluations; and provide evaluation leadership and guidance.

• A central clearinghouse for evidence on the effectiveness of policies and programs affecting Aboriginal and Torres Strait Islander people would strengthen accountability for undertaking good evaluations and improve diffusion of knowledge. 

 

While the PC’s Indigenous Evaluation Strategy is not without its flaws, its overarching message that evaluation has the potential to improve policy outcomes is certainly one worth taking seriously, particularly in circumstances where virtually everyone with any detailed knowledge of the Indigenous policy domain concludes that policy outcomes over the past few decades leave much to be desired.

 

Unfortunately, over the almost three years since the proposed Strategy was published, neither the former Government who commissioned the Productivity Commission’s work, nor the current Government have taken up the opportunity to expand the use of, and focus on, evaluation within the Indigenous policy domain, and as a consequence no government has sought to implement the recommendations included in the proposed Strategy either in part of in full.

 

This simple fact ‘bells the cat’ on the real levels of commitment by governments of all complexions to ensuring policy development and outcomes at least in the Indigenous policy domain are in fact fit for purpose. See my previous post (link here) for further detail. The deep scepticism of the Executive Branch to independent evaluation of policy and programs is driven by at least two sets of pressures:

·       for politicians, they are deeply averse to any process of independent evaluation that has the potential to provide ammunition to their political opponents (and thus to constraint the political opportunities that are available);

·       for bureaucrats, they abhor any process that expands the deeply constrained levels of transparency that ensure that the scope for flexibility in shifting administrative priorities and resource allocations incrementally is not constrained.

 

These two drivers are mutually reinforcing, and their combined impact is antithetical to effective democratic accountability and high quality governance.

 

The second contextual development was the successful negotiation of the National Agreement on Closing the Gap (link here), and in particular, the commitment of all parties, including all Australian jurisdictions to four priority reforms. Of particular importance in my mind is Priority Reform three:

58. The Parties commit to systemic and structural transformation of mainstream government organisations to improve accountability and respond to the needs of Aboriginal and Torres Strait Islander people. Priority Reform Three was strongly supported by the 2019 engagement process [emphasis added].

 

This institutional commitment is paralleled by the longstanding and inexorable increase in the comparative importance of mainstream programs to the quality of life of Indigenous citizens. One positive feature of this week’s budget was the focus on APS reform, and part of that is an explicit recognition that the National Agreement requires mainstream agencies to do better in relation to Indigenous issues and outcomes. See 2023 Budget Paper 4 at pages 8-9 (link here).

 

Given this background and context, the decision of the current Government to establish a new whole of government evaluation Office within the Treasury portfolio is to be welcomed. The Assistant Treasurer Andrew Leigh has been a longstanding advocate for greater use of evidence in policy development, particularly Randomised Control Trials.  It can be seen as a heavily cut down version of the proposal championed by Nicolas Gruen (link here) for an office of Evaluator General to be established to focus on public sector effectiveness and thus complement the focus of the ANAO in ensuring accountability and efficiency of public sector outlays (link here).

 

The challenge will be to identify a series of strategically important and high profile evaluation opportunities that lead directly to positive reform agendas. The new evaluation Office is too small to have an immediate impact, and will require time to build momentum and capability. The risk is that it will be vulnerable to abolition should it fail to demonstrate its utility in driving improved public sector outcomes. Of course, the new Office has a potential synergy with the Productivity Commission which may well be the subject of further reform over the next 12 months (link here). There may thus a strategic opportunity to build further evaluation capability across the public sector along with any future refocussing of the Productivity Commission.

 

While the ultimate shape of the new evaluation Office is still unclear, the obvious question for our purposes is what will this mean for evaluation in the Indigenous policy domain? It is too early to make definitive assessments on this issue, but it is clear that there will be a number of challenges over and above those facing mainstream agencies and policy sectors.

 

The first relates to the issue of involving Indigenous interests in evaluation activities that impact them. This becomes difficult when the subject of the evaluation is a mainstream program, especially if the data available is constrained. A second challenge is the extent to which Commonwealth policies and programs are implemented and delivered by the states, a trend reinforced by the policy architecture embedded within the National Agreement on Closing the Gap. 

No doubt there will be other issues that may arise: will the new evaluation Office differentiate between policy and programs; will the new Office have the capacity to evaluate the effectiveness of tax expenditures; what will be the relationship with the ANAO and other relevant agencies, and what will the coordination and working arrangements look like. All of these issues, and how they are decided, will impact Indigenous interests, for better or worse, but they will in very large measure be intangible impacts difficult to ascertain without the sort of close consideration that high quality evaluations might bring.

 

As someone who has taken an interest in the evaluation of Indigenous policy over thirty years, it seems useful if I lay out succinctly the key principles and policies that I would hope the new evaluation Office adopts in relation to evaluation in general and in particular in relation to the Indigenous policy domain. These are not intended to be comprehensive, but I do consider that they are among the necessary principles required to ensure mainstream evaluations make a positive contribution to the quality of policy outcomes, and importantly, to identifying potential reform agendas.

 

First, I would hope that the new evaluation Office adopts a rigorous and focussed approach to oversighting the quality of the entire approach to evaluation of each major agency. There is a potential overlap here with the role of the Department of Finance. The Office should develop a checklist of core competencies and create an overarching and publicly available data base of agency performance against a small number of core competencies. Of course, it will not be in a position to assess each agency every year, but as it does make assessments, they should be added to the data base and be published on the evaluation Office’s web site.

 

Second, I would hope that the new evaluation Office gives particular priority to assessing the independence of evaluations undertaken by agencies, rating them on a consistent scale.

 

Third, I would hope that the evaluation Office develops a mechanism which allows it to assess the extent of an agencies policy and program footprint that has been subject to evaluation over the past ten or fifteen years. See the discussion at pages 208-9 of the Indigenous Evaluation Strategy Background Paper.

 

Fourth, I would hope that the evaluation Office makes an assessment of each agency’s commitment to ensuring that ‘evaluation relevant’ data is being collected and ideally progressively made public. In the Indigenous space for example, it is now eight years since the last National Aboriginal and Torres Strait Islander Social Survey was undertaken by the ABS (link here). The funding of the NATSIS is contributed by relevant policy agencies, and a document released by the ABS in 2017 indicated that the aim was to undertake the survey every six years. Unfortunately, this has not occurred and there appears to be no current plans to undertake another version (link here).

 

Fifth, I would hope that the evaluation Office assesses and reports on the comparative performance of agency transparency in relation to publishing evaluation results in a timely fashion.

 

The Government has identified that the new evaluation Office will undertake a small number of flagship evaluations each year.  There will be a risk that the new Office gets lost in the detail of particular exercises, and ends up keeping busy on evaluations and issues that are tangential to the overall effectiveness of the Commonwealth’s evaluation efforts. In my view, it is this latter objective that should be the primary focus for the new Office. This is the real necessity. If the Office performs that task well, it will create substantial incentives for agencies, including the NIAA and its associated entities, to undertake policy and program evaluations in a much more effective manner than has been the case over the past few decades.

 

Addendum

For those interested, I have previously published posts on this blog related to evaluation issues (link here, link here, link here, and link here).

A more comprehensive conceptual treatment of the link between evaluation and policy can be found in my policy insights paper, Evaluation and review as drivers of reform in the Indigenous policy domain, available on the CAEPR, ANU website (link here).

 

NIAA management of evaluation

Words are no deeds

Hamlet Act 1 scene 3

 

The Department of Finance publishes the Commonwealth Evaluation Policy on its website (link here). It is succinct and not particularly prescriptive nor onerous. It lays out key principles, and seeks to promote an evaluative culture. As part of such a culture, it notes that:

 Leadership that is positive about learning from performance monitoring and evaluation activities is a necessary condition for delivering effective outcomes and providing quality advice to Government, the Parliament and the public.

 

It goes on to lay out key elements in such leadership, including a series of governance actions which support an evaluative culture within agencies.  The Finance website also includes a toolkit which provides a detailed set of tools to assist agencies manage the evaluation process.

 

I took the opportunity over the weekend to examine the NIAA website (link here) so as to seek to understand its current approach to evaluation.

 

The NIAA website includes a section on areas of focus for the agency, and one of them is listed as Evidence and Evaluation. I set out below an extract from the relevant section of the NIAA website (link here):

Evaluations and Evidence

Evidence

The work of the National Indigenous Australians Agency is underpinned by effective data and evidence. The NIAA provides advice and information to the Minister for Indigenous Australians, the Australian Government, State and Territory governments, organisations, providers and communities to inform policy development, programs and monitoring of the effectiveness of programs for Aboriginal and Torres Strait Islander peoples….

Evaluation

Evaluation helps us to gain an understanding of what works and what doesn't, for whom and why. This kind of knowledge can help us to learn and improve what we do, supporting decision-making with the best available evidence.

Evaluation is integral to continual improvement. It is not a one-off, or 'tick the box' exercise. Evaluation supports evidence-informed policy development, public accountability, learning and performance reporting. Evaluation needs to be planned across the lifecycle of a program, from the very start.

The NIAA promotes and supports a culture that focusses on evaluation and performance improvement. We do this by:

·         Providing information, support and advice that embeds the functions of evidence and evaluation into policy and program design cycles.

·         Ensuring that monitoring and evaluation priorities we support are well designed and delivered in collaboration with Indigenous Australians.

The Evaluation Framework guides the conduct and development of a stronger approach to evaluation of programs and activities under the Indigenous Advancement Strategy. The goals of the Framework are to:

·         Generate high quality evidence that is used to inform decision making.

·         Strengthen Indigenous leadership in evaluation.

·         Build capability by fostering a collaborative culture of evaluative thinking and continuous learning.

·         Emphasise collaboration and ethical ways of doing high quality evaluation at the forefront of evaluation practice in order to inform decision making.

·         Promote dialogue and deliberation to further develop the maturity of evaluation over time.

Indigenous Advancement Strategy Evaluation Work Plan 

Publication of the Evaluation Work Plan supports the commitment to transparency made in the Evaluation Framework. It provides details on evaluation activities and enabling activities that are planned, underway or completed by the NIAA.

Evaluation activities on the Work Plan are reviewed by members of the Indigenous Evaluation Committee and approved by the NIAA Executive Board.

 

The IAS evaluation work plan (link here) is dated 5 January and appears to have been substantially reconceptualised in recent times. The NIAA website describes what has occurred in the following terms:

The NIAA has moved to a dynamic version of the IAS Evaluation Work Plan to make it accessible to a broader audience, and to improve the timeliness of the information presented. Previous versions of the IAS Evaluation Work Plan (2017–18 through 2020–21) are still available.

In this dynamic version of the work plan, users can find information on past and current evaluations, and enabling activities, according to IAS Program areas in the tables below. New evaluations and enabling activities are added to the work plan as they come on line and updates to the status of evaluations are made as they progress through the different stages to completion.

 

In effect, the NIAA appear to have shifted away from an annual evaluation plan to one which is continuously updated. In effect, they have conflated two separate potential data sets. The first is the plan that has been adopted for each financial year. The second is the status report on progress against the plan, and recording of any additions and/or deletions. What is unclear or made more opaque with the new approach is the extent to which evaluations are progressing against the workplan (delays are not obvious unless you delve below the front page), how long completed and discontinued evaluations will stay listed (I would argue they should remain available for at least 5 years).

 

A closer examination of the listed evaluations in the ‘dynamic work plan’ reveals a number of issues that do not sit well with the Department of Finance advice regarding the development of an evaluation culture. For example, embedded in the descriptions of various evaluations are multiple inconsistencies and what might be described as deliberate circumlocutions. A number of evaluations have been completed, sometimes two years ago, and yet are listed as ‘publication pending’. Is this actually the case, or has a decision been taken to not publish them and quietly remove them from the work plan at some time in the future. Other evaluations have been discontinued, but without any real explanation as to why this has occurred, and whether funds were expended and evaluation activities undertaken before the discontinuation was decided upon.

 

My advice to the NIAA would be to develop a template for the evaluation workplan that lays out a consistent and comprehensive set of core information, ideally including dates evaluations were commissioned, were begun, were completed, and were published, the evaluator, a succinct description of the aim of the evaluation, and once completed, a succinct description of the results. Additionally, there seems no good reason why the cost of the evaluation is not provided, particularly as the value of any external consultancy payments would be available on the Commonwealth tender site.

 

A more important issue to my mind is that it is apparent that there are multiple evaluations of relatively insignificant programs, of virtually zero strategic significance. The standout example to my mind is the Dog Operations Unit evaluation listed as being underway. It was approved in August 2020, contract start in February 2022, and is (apparently) yet to be finalised. The evaluation description is:

NIAA funds the Northern Territory Police, Fire and Emergency Services (NTPFES) to provide a Dog Operations Unit within the community of Groote Eylandt with the aim to improve community safety and crime prevention through enhanced police capability. This evaluation will assess the design, implementation and outcomes of the Dog Operations Unit.

 

A September 2022 ABC media report suggests that the NT Police only have four dogs in their unit, and that they are based in Darwin (link here). A February 2023 media report relating to an arrest on Groote suggests that the Dog Operations Unit was deployed to Groote along with the Drone Unit and the Territory Response Unit (presumably from Darwin). Perhaps the NIAA funding has ceased. This evaluation strikes me as chopping wood for practice, and one must also ask, why is it that the NT Government is not prepared to fund the operations of a dog unit in a remote community if that is what is required. There are a host of further questions raised by this evaluation, but the bigger issue is why does the NIAA evaluation work plan include miniscule projects of zero strategic import and not larger programs such as remote housing.

 

When I checked the housing program, there is an item for remote housing:

Northern Territory Remote Housing National Partnership Agreement

Program: Program 1.5 - Remote Australia Strategies; Activity type: Evaluation strategy; Last updated: 5th October 2021; Not continuing.

The reporting framework under the National Partnership Agreement has only recently been endorsed, with further amendments expected. The four Northern Territory Land Councils have now prioritised a joint review of leasing and housing models in the Northern Territory over the evaluation strategy. The joint review will inform opportunities to improve remote housing delivery in the Northern Territory.

 

There are two evaluations listed for leasing and housing models in the Northern Territory:

Northern Australia White Paper: Land Tenure Reform Pilots

Program: Program 1.1 - Jobs, Land and Economy; Activity type: Evaluation; Contract start: May 2019; Provider: Yaran Business Services; Last updated: 4th February 2022. Publication pending:

The land tenure reform pilots measure supported a range of projects across Northern Australia to test innovation in land-based activities on the Indigenous estate that support economic development and boost investment in the north. The purpose of the evaluation is to understand the extent the measure has contributed to the goals of the Northern Australia White Paper which sets out the Government’s 20-year plan for investment and collaborative support to drive growth in Australia’s north.

Northern Australia White Paper: Township Leasing and Land Administration

Program: Program 1.1 - Jobs, Land and Economy; Activity type: Evaluation; Contract start: May 2019; Provider: Yaran Business Services; Last updated: 4th February 2022. Publication pending:

The Aboriginal Land Rights (Northern Territory) Act 1976 (Land Rights Act) was amended in 2006 to provide for section 19A ‘township leases’, which cover entire community areas on Aboriginal land to support economic development. Township leases can be held either by the Executive Director of Township Leasing (EDTL) on behalf of the Commonwealth or, more recently, by a local Aboriginal corporation representing traditional owners (community entity model).

The evaluation considers the communities that have taken up township leases in the Northern Territory (NT) from introduction to 2017 and how they have performed against initial expectations and policy objectives. As well as examining the direct benefits of township leasing to individual communities, the evaluation examines the extent to which township leasing provides social, cultural, economic benefits to Aboriginal peoples and communities in the NT.

 

These two evaluations appear to relate to the two evaluations apparently proposed by the NT Land Councils and focus on perfectly legitimate policy and program issues. It is unclear why they are yet to be made public. However, the fact that these are legitimate subjects for evaluation is hardly a justification for not evaluating what I understand is a four year $550m investment in remote housing in the NT by the Commonwealth (link here).

 

According to a February 2022 ANAO performance audit of remote housing in the NT (link here), by virtue of the existence of some 3500 forty year housing leases from traditional owners to the Commonwealth (which are subleased to the NTG), the Commonwealth retains an underlying interest in the majority of public housing in remote communities in the NT, which represent 59 percent of the remote housing portfolio in the NT (see para 1.23). That ANAO audit (which was not as broad as an evaluation) identified numerous shortcomings in the NIAA management of remote housing investment in the NT. The first 32 paragraphs provide a useful summary of the ANAO conclusions which are expressed in a dialect unique to the ANAO which I refer to as ‘muted bureaucratese’. I have cherry picked and reformatted a few sentences from that summary section to provide some concentrated flavour:

NIAA’s administration of funding for remote housing in the NT has been partly effective. The development of the National Partnership was partly effective. However, the National Partnership Implementation Plan has significant weaknesses, and advice to the minister did not include analysis of some of the National Partnership’s key parameters. NIAA has been partly effective in assessing the delivery of the program of works under the National Partnership. NIAA has not fully developed or implemented a risk-based approach to determining what assurance is necessary to verify the Northern Territory (NT) Government’s achievement against National Partnership targets. NIAA has been partly effective in ensuring that the National Partnership’s outcomes will be achieved. NIAA’s assessment of the delivery of capital works has been partly effective. NIAA’s assessment of the delivery of PTM services has been partly effective. NIAA does not gain assurance over the performance results reported to it by the NT Government, and has not always used the correct reporting period to assess the NT Government’s results and to recommend payments. NIAA has been partly effective in ensuring transparency about how money has been spent. However, NIAA does not have assurance that the NT Government will meet its $550 million co-contribution commitment over the life of the National Partnership. NIAA has not managed risks to the delivery of the National Partnership effectively…

 

Partly effective indeed. See my previous post on this performance audit for my own assessment (hint: it is tougher than the ANAO assessment: link here). Given these performance audit findings on what is a billion dollar joint Commonwealth / NTG investment program, and one which will necessarily continue into the future despite the time limits on the National Partnership, the case for discontinuing the proposed remote housing national partnership evaluation appears wafer thin. And this is just one of the issues embedded within the work plan.

 

Returning to the bigger picture, it is clear that the evaluations listed in the NIAA evaluation work plan raise myriad issues. One wonders whether there would be merit in undertaking a short sharp action oriented review of the management of the evaluation program by NIAA. I think it could be undertaken over ten days by a competent policy analyst.

 

The major problem with the NIAA evaluation workplan is that it provides no indication of the overall extent of the evaluations undertaken over the past three years and planned for the next three years. The result is that it is virtually impossible to assess the extent of NIAA’s evaluation efforts against the agency’s overall policy and program footprint.

 

A related issue is that there is no evidence of narrative provided to demonstrate how previous evaluations have contributed to revised or new program initiatives.

 

It is my assessment that the NIAA appears to be merely going through the motions: an evaluation framework document full of the expected rhetoric, a rather opaque Indigenous Evaluation Committee to provide independent strategic and technical advice, yet where it really counts, a ‘dynamic work plan’ replete with inconsistencies and an absence of strategic direction.

 

What is clear is that the reported budget announcement later this week of a centralised unit in Treasury to promote improved evaluation across the Commonwealth will have its work cut out in at least one area of public policy. In the meantime, it is worth remembering that the consequences of ineffective programs and policy is borne in large measure by disadvantaged citizens.

Diluting structural reform: the Productivity Commission’s Information Repository on Closing the Gap

 

Angels and ministers of grace defend us!

Be thou a spirit of health or goblin damn'd,

Bring with thee airs from heaven or blasts from hell,

Be thy intents wicked or charitable,

Thou com'st in such a questionable shape…

Hamlet, Act 1, scene 4.

 

 

The Productivity Commission launched a new Closing the Gap Information Repository on 23 June 2021 (link here).

 

On the same day, Minister Wyatt issued a media release (link here) extolling the Commission’s new web site. He claimed (inter alia) it was:

…another step towards delivering on the Government’s commitment to share data and support more informed decision making by all parties to the National Agreement…

…As well as supporting the comprehensive review of progress every three years, this measure includes developing and maintaining a data dashboard (presenting the most up-to-date information available) and an annual data compilation report (a point-in-time snapshot).

“The launch of the Productivity Commission’s information repository provides the evidence base we need to properly monitor progress that all parties are making towards Closing the Gap,” Minister Wyatt said.

“This initiative brings all the data together into one place so that people can readily see the current situation and trajectories of indicators for each target, providing a level of transparency and access that we haven’t had before…

…Minister Wyatt also said the Productivity Commission will play a key role in keeping all parties to the National Agreement accountable.

 

This initiative, which is required by the National Agreement on Closing the Gap (see clauses 116 & 117) (link here), is clearly a welcome and positive development.

 

Nevertheless, it deserves to be subject to critical assessment, as do the Minister’s claims.

 

The Productivity Commission makes plain that the information repository is still under development, and the web site is listed as being a ‘beta’ version, which normally refers to the release of software that has been through internal ‘alpha’ testing, and is now released for wider public testing. The Minister’s media release glosses over this qualification. The information repository’s beta status does mean that any critique made is potentially moot. Of course, the reality is that the dashboard will be constantly evolving as new data and information becomes available.

 

There are two contextual criticisms worth making upfront. First, contrary to the Minister’s comment that the initiative provides a ‘level of transparency and access that we haven’t had before’, we should remind ourselves that the present Government abolished in 2014 a number of oversight and implementation coordination entities including the COAG Reform Council and the Closing the Gap Clearinghouse. Both these mechanisms had important roles in publishing relevant data regarding the performance of the Closing the Gap framework.

 

Second, contrary to the Minister’s assertion that the repository ‘provides the evidence base we need to properly monitor progress’ and that it ‘brings all the data together into one place’, it seems clear that the intention is to limit the data included to social indicators and to provide no comprehensive account of the investments allocated by governments towards meeting the targets. It will be impossible to keep all parties accountable, and to assess responsibility for shortfalls in meeting targets if we are not in a position to see what levels of investment is being allocated to each target by each of the governments involved, and in particular, are able to monitor variations over time in those investment levels.

 

The third issue worth considering is the way the repository ‘dashboard’ treats the four strategic reforms included in the National Agreement on Closing the Gap and signed up to by all jurisdictions. These are not targets which can be measured quantitatively, but are policy reforms that will need to be sustained over time and inevitably involve policy relevant qualitative judgements to assess their effectiveness. Yet the Productivity Commission has taken the bizarre decision to treat them as if they are mere targets, and thus has devised somewhat artificially quantitative indicators that purport to assess performance.

 

The most egregious example relates to Strategic Reform Three which deals with transforming mainstream government organisations, spelt out in paragraphs 58 to 68 of the National Agreement (link here). The Agreement identifies six elements of a successful transformation in clause 59. The dashboard selects a number of indicators that relate to just one of those six elements (racism), invents a target, namely a ‘Decrease in the proportion of Aboriginal and Torres Strait Islander people who have experiences of racism’ with the implicit suggestion that this is the evidence we need to monitor progress in the implementation of the strategic reform, and will keep government parties accountable (to refer once again to the criteria propounded by the Minister). That implicit suggestion is absolutely wrong on both counts.

 

What is most surprising about the Productivity Commission’s approach here is that in its Indigenous Evaluation Strategy (link here) released in October 2020, and not yet responded to by the Government, the Commission was at pains to emphasise the importance of evaluating the impact on First Nations citizens of mainstream policies and programs. Unless it revises its approach, the very real risk will be that government agencies across eight jurisdictions will limit their purportedly transformational activities to those that relate to the narrow indicators identified in the dashboard. If this occurs, essential structural reform will have been transmuted into mere desirable change.

 

If, as the Minister suggests, the Productivity Commission is to play a ley role in  keeping all parties to the National Agreement accountable, and is to maintain the trust of the Indigenous and wider community in these matters, it will need to ensure that it is meticulous in aligning its dashboard, and the information provided by the repository with the actual terms of the Agreement negotiated by First Nations interests with all Australian governments.

 

Finally, on 25 June 2021, the Minister and Prime Minister issued a joint media statement  (link here) recording their second roundtable with representatives of the Coalition of Peaks. The statement is largely process oriented and provides little information of substance. It does confirm that the Government intends to publish its closing the gap implementation plan in August along with announcements on ‘associated investments’. At that point, the degree of seriousness being brought to the task of closing the gap by the Australian Government will become clearer.

 

 

Prospects for Reform: the forthcoming Indigenous Evaluation Strategy

 

What, must I hold a candle to my shames?

The Merchant of Venice Act 2, scene 6

 

 

The Productivity Commission Inquiry into an Evaluation Strategy is nearing completion. Its Draft Report was issued in early June 2020, and attracted 112 submission. The Final Strategy is scheduled to be submitted in October 2020 (link here).

 

In a previous post (link here), I bought together a series of links to various documents that I have authored or to which I contributed. Submissions on the Draft Strategy closed in early August and are available on the Productivity Commission website (link here). I haven’t reviewed all 66 submissions, but a quick scan suggests that the following submissions are worth a look for those interested: the Independent Members of the Indigenous Evaluation Council of the NIAA; the NIAA; Lateral Economics; APONT; Professor Don Weatherburn; ANTaR; and Ernst & Young.

 

The process from here is that following the publication of the Final Strategy, the Government will consider the recommendations and decide how to respond. There is no obligation on the Government to respond within any set timeframe. Even where it decides to implement recommendations, it may not announce them, nor give reasons. It may of course decide to leave the Strategy on the shelf, unimplemented.

 

Key recommendations in the Draft Report were for a new Office of Indigenous Policy Evaluation and for the creation of an Indigenous Evaluation Council. Many of the submissions referenced above commented on the desirable attributes of one or both of these bodies, as does the Draft Strategy itself.

 

What prompted me to write this post was the recent publication of an article on The Interpreter web site by Professor Stephen Howes from the ANU Crawford School of Public Policy on developments related to the Office of Development Effectiveness (ODE) within the Department of Foreign Affairs (link here).

 

I don’t propose to summarise his article (it is very short), apart from noting that the Office was established by the 2006 White Paper on Australian Development Assistance; was endorsed by the 2011 Aid Effectiveness Review and supplemented by an Independent Evaluation Committee (IEC); and has been widely seen as making a positive contribution to the quality of Australia’s development assistance.

 

The Howes article reports that the Government has this year abolished the ODE and the associated IEC, and replaced the function with a downgraded departmental evaluation section. Moreover, paralleling the Government’s abolition of the PM’s Indigenous Advisory Council (link here), the Government made no public announcement of its decisions. As a consequence, there has been no justification provided by the Government for the actions taken.

 

These developments raise the obvious question: given the lack of commitment to evaluation of our International Development assistance programs, what level of commitment will the Government muster for the forthcoming recommendations of the Productivity Commission in relation to Indigenous program effectiveness? Or put another way, is the Government signalling that it is not prepared to countenance a truly independent evaluation function for any of its programs, and if so, what are the implications for evaluation of the Indigenous policy domain?

Indigenous evaluation: selected links

   

His promises were as he then was, mighty,

But his performance, as he is now, nothing.

Henry VIII, Act 4, scene 2.

The Productivity Commission has recently published its draft report on Indigenous Evaluation, and is seeking further submissions by 3 August from the public.

I thought it timely to post links to the PC document, and to a few documents I have authored in relation to the topic of Indigenous evaluation. I wont add further commentary here; the documents speak for themselves.

Link to a previous post, ‘Marshmallow and Fudge: evaluation and the Indigenous policy domain’, here

Link to the Productivity Commission Draft Report on Indigenous evaluation here. The final report is due by the end of the year.

Link to my initial submission to the Productivity Commission here

Link to my recent evaluation policy insights paper on the CAEPR website here. It is titled Evaluation and review as drivers of reform in the Indigenous policy domain.

From the Abstract:

This Policy Insights Paper seeks to assess the influence of evaluation and review in influencing policy in the Indigenous affairs policy domain. The paper examines four high-level case studies of strategically significant policy issues within the Indigenous policy domain to assess the impact of evaluation in driving reform over time. 

Marshmallow and fudge: evaluation and the Indigenous policy domain

I want to cover three issues in this post. First, provide a very brief and selective account of the history of evaluation in relation to Commonwealth policies and programs. Second, to unpick some of the issues raised in the recent ANAO performance audit on ‘Evaluating Aboriginal and Torres Strait Islander Programs’ (link here). Third, to make some preliminary or provisional comments in relation to the Productivity Commission’s project to develop a whole of government Evaluation Strategy (link here). Paragraph numbers in square brackets refer to the ANAO report.

The Indigenous policy domain has a long history of concerns over accountability, waste and effectiveness that reaches back to the very beginnings of the Commonwealth’s decision to take on an active role in Indigenous affairs in the late 1960s. This in turn led to periodic bouts of intense media and political scrutiny into the propriety and effectiveness of public funding for Indigenous affairs. For example, a series of dramatic allegations (never confirmed) were aired in the Senate in 1988 alleging the existence of a ‘black mafia’ which in turn led to a Special Audit Reports into the Department of Aboriginal Affairs and the Aboriginal Development Commission by the Auditor General in March and October 1989. In turn, a desire to pre-empt and counter this sort of political pressure led to the establishment within the ATSIC legislation of the Office of Evaluation and Audit (OEA) headed by a ministerially appointed Director to oversight ATSIC’s programs and report to both the Commission and the Minister. After the abolition of ATSIC, OEA was shifted to, and eventually subsumed within, the ANAO.

The establishment of OEA broadly coincided with the adoption of program performance budgeting by the Commonwealth, which in turn led to an increased focus on performance monitoring and evaluation across the public sector. Since the abolition of ATSIC around 2005, the Indigenous Affairs portfolio has lodged in a number of key portfolios (DIMIA; FACS; FaHCSIA) each with their own program management and evaluation cultures, before finally coming to rest in the Prime Minster & Cabinet portfolio since 2013.

While it is arguable that more onerous standards of accountability have been applied to Indigenous programs over the past 30 years, it is certainly the case that concepts such as accountability, value for money, efficiency and effectiveness have been ongoing benchmarks (if only internally to government) in the administration of Indigenous affairs programs and policies. It is however almost 15 years since the last major accountability controversy in the Indigenous affairs portfolio which related to the activities of a number of senior ATSIC leaders, and which created the opportunity for ATSIC to be abolished by the Howard Government. For those interested in this history, I refer you to pages 27 – 30 in the recent CAEPR Policy Insights paper I co-authored with Neil Westbury (link here).Consequently, it is not surprising that the substantive (as opposed to rhetorical ) focus on accountability has diminished over time. In turn, the impetus for effective and substantive evaluation has diminished, even as the prevalence of rhetoric advocating greater use of evaluation has increased.

At the rhetorical level, there have been numerous reports and speeches by public sector leaders advocating greater use of evaluation as the solution to ongoing policy failure in Indigenous affairs. Thus in 2009, the Department of Finance and Deregulation’s Strategic Review of Indigenous Expenditure ‘stressed the need for a more rigorous approach to program evaluation at a whole of government level’ [para 1.4, p14].

In August 2016, in the Dungala Kaiela Oration, the Secretary of PMC, Martin Parkinson (link here and link here)  argued

A high proportion of what we fund has, at best, a weak evidence base of how it affects Indigenous peoples. We must gather evidence which shows we are improving the lives of Indigenous Australians. And if that evidence tells us otherwise, we must change our approach.

In December 2016 in a speech titled ‘Public policy and whats missing in action?’ (link here), the Deputy Chair of the Productivity Commission, Karen Chester, argued that:

While there is extensive reporting on the extent of Indigenous disadvantage, there is a lack of evidence about what works (and what does not) in bridging outcome gaps. And while evaluating the impacts of policies on Indigenous outcomes can be challenging, the challenges should not be seen as insurmountable….Stop looking for silver bullets and policy sound bites. And just get back to the dirt under the fingernail work of building evidence-based policy and building a much stronger evaluation culture (we need to know more about what works and why).

These views, which have an attractive internal logic, implicitly assume that Government programs have straightforward objectives and that if only we find appropriate or effective policy incentives, Indigenous behaviours will change in ways which lead to improved socio-economic outcomes. Both these assumptions are problematic in cross cultural contexts  and certainly not necessarily the case in the context of the Indigenous domain. Moreover, even if policies were technically able to improve outcomes, robust policy evaluations inevitably come into conflict with the deep-seated political imperatives to frame policies and programs to advance political objectives and also to avoid any admission of failure that might create political embarrassment for Ministers or the Government.

Thus notwithstanding the continuous stream of pro-evaluation rhetoric from public service leaders, the recent ANAO performance audit ‘Evaluating Aboriginal and Torres Strait Islander Programs’ outlines a sorry history of desultory performance by the Department of Prime Minister and Cabinet (DPMC) and its Indigenous Affairs Group (IAG). These results appear to derive from policymakers prioritisation of the perception of action over substantive action. Below, I selectively point to a number of issues raised by the ANAO audit which reinforce this conclusion. I recommend those interested read the whole report (link here).

The ANAO performance audit

The ANAO performance audit was aimed not at assessing the effectiveness of DPMC’s evaluations, but at assessing the effectiveness of the design and implementation of the department’s evaluation framework for the IAS [para. 6]. This more limited objective was arguably a mistake, since even a wholly effective design and implementation of the framework is merely a necessary but not sufficient prerequisite for successful evaluation of the IAS. Nevertheless, the audit does uncover and reveal a wide range of useful information about the DPMC approach and commitment to comprehensive evaluation of not only the IAS (Its most significant program) but to Indigenous related policies and programs more generally.

In chapter one, the ANAO outlined the background to the audit. In 2017, the ANAO performance audit of the Indigenous Advancement Strategy (IAS) (the largest program in the Indigenous affairs portfolio comprising five program streams and $4.8bn over four years) ‘found the Department did not effectively implement the strategy’ [Para 1.9]. See my previous blog post on the IAS audit (link here). In response, in February 2017 the then Minister announced funding of $40m over four years to strengthen IAS evaluation in the portfolio [para 1.10]. In February 2018, the Department released an IAS Evaluation Framework [para.1.12]. IAG program areas are responsible for conducting the majority of program evaluations under the framework.

In chapter two, the performance audit outlines significant delays in establishing the evaluation framework for the IAS, extending from March 2014 through to February 2018 [Figure 2.1, para 2.1]. The ANAO note that the IAS evaluation framework does not include references to the Government’s higher level objectives including the Closing the Gap framework [paras 2.26, 2.29]. The ANAO also note that the IAS evaluation framework ‘is intended to align with the wider role of the Productivity Commission in overseeing the development and implementation of a whole of government evaluation strategy of policies and programs that effect [sic] Indigenous Australians’. However because of delays in passing legislation and appointing a Commissioner, the Productivity Commission did not begin work until April 2019 and it won’t be complete until July 2020 [para. 2.31].

In chapter three, the ANAO assess implementation and management fo the IAS Framework. The ANAO note that the framework includes various commitments for activities designed to strengthen evaluation quality, but without any timeframes specified for implementation [para.3.3].The ANAO concluded that the performance targets for the IAS evaluation framework were focussed on the delivery of short term outputs (ie not longer term outcomes). The audit’s first recommendation [para 3.12] was that the Department ensure its performance information for evaluations are supported by a reliable methodology for measuring longer term outcomes. The audit’s second recommendation is largely process related [para 3.22]. The rest of the chapter is concerned with management oversight arrangements for evaluations. In a surprising and rather damning assessment, the ANAO found that there was limited evidence that the IAG Executive Board were involved in the oversight of the evaluation activities [para.3.31]. The Indigenous Evaluation Committee (which included some external members) raised concerns in late 2018 that a number of completed evaluations had not been published [para. 3.33]. The ANAO also noted delays in the Department developing effective mechanisms to track evaluation recommendations and management responses [paras.3.36-3.40].

In chapter four, which considers the application of the framework, the ANAO conclude that ‘As the department is still developing procedures to support the application of the IAS evaluation framework, it is too early to assess whether evaluations are being conducted in accordance with the framework’ [Box, page 37]. After assessing the Department’s processes for prioritising and selecting areas for evaluation, the ANAO recommended that the department ‘formalise its evaluation prioritisation process by developing structured criteria for assessing significance, contribution and risk and conducting a strategic analysis of gaps in evaluation coverage’ [para 4.10]. While the Department indicated it agreed with the recommendation, its response is so heavily caveated as to be meaningless. Moreover, it made no reference or commitment to undertaking a strategic gaps analysis as explicitly recommended [para 4.11].

Chapter four also included discussion of the desirability of independence within evaluation teams [paras 4.25 to 4.28]. The discussion is extraordinarily hedged and timid, and failed to make a substantive assessment of the actual independence of the evaluations undertaken. There is also a discussion of the issue of publication of completed evaluations and management responses [paras 4.37 to 4.41. See also appendix 4]. Of 35 evaluations on the Department’s 2018-19 workplan, 15 had not commenced. Of the remaining 20, 8 had been published and 12 withheld from publication. The ANAO discuss the reasons for non-publication. In at least four cases (involving very significant and sensitive program evaluations) the Department was waiting to brief the Minister or awaiting his noting of a brief [para 4.38; appendix 4]. In plain language, the Minister was preventing timely publication of the evaluations.

I found this performance audit disappointing for two reasons. First, it was clear from the inexorable delays and bureaucratic fudging – much focussed merely on the development of an evaluation framework against which actual evaluations might be assessed – that the Department (and the Minister – see para 2.7) were not prioritising evaluation activities. Primary responsibility rests with senior management, as there are indications throughout the audit that middle level officers were seeking to advance the agenda, but kept hitting blockages above. If one assumes that the senior management of the DPMC are competent (Which I do), then it is difficult to explain the disjunction between the rhetoric of the Secretary in 2016 and the appalling delays in making substantive progress between 2014 and 2019 as anything but a process of bureaucratic game playing. In particular, the bureaucratic imperative appears to be to contrive circumstances where the department is always appearing to be moving forward, but never reaches a destination.

The second reason the audit is disappointing is that the ANAO itself appears to have gone soft. It describes the audit as a performance audit, but it fails to robustly assess poor performance. Its focus on the framework rather than evaluations themselves is highly process oriented. There are only three recommendations, all of which are process related; this is a major flaw in the performance audit as it effectively leaves the DPMC off the hook notwithstanding the documented lack of substantive commitment to progressing an effective and substantive evaluation agenda. The discussion on independence of evaluations has all the robustness of a marshmallow. The discussion on publication and management responses is slightly better, more akin to fudge. Where are the robust recommendations on publishing evaluations undertaken with taxpayers’ funds, and ensuring that evaluations are not subject to being diluted through political interference, second-guessing, concern about vested interests and the like. We are told in para 3.10 of an instance where the Department agreed to an internal audit recommendation designed to strengthen evaluation activity, but then did not implement the recommendation. Given this behaviour, how are we believe that the Department will take any of the recommendations or less formal suggestions included in this report seriously?

Our scepticism is reinforced by the fact that notwithstanding the consistent rhetoric about the merits of evaluation of Indigenous programs for over a decade, the senior management of the nation’s premier department of state has been shown to be incapable of implementing an effective evaluation program. How are we to respond to the disjunction between the ANAO conclusion [in para.10]: ‘The department’s implementation and management of the IAS evaluation framework is partially effective’ (a generous finding given the actual content of the audit report and the limited results to date) and the Secretary’s letter in response in Appendix 1 which fails to acknowledge or address in any way the negative content of the audit. This deliberate blindness ought to be a matter of concern to the Minister, to the Government, and to First Nations peoples.

The Productivity Commission Indigenous Evaluation Project

In April 2019, the Treasurer requested the Productivity Commission (PC) to ‘develop a whole of Government Evaluation Strategy for policies and programs affecting Indigenous Australians’. The scope of the strategy includes a principles based framework for the evaluation of policies and programs affecting Indigenous Australians, identification of priorities for evaluation, and a description of the Productivity Commission’s approach for reviewing agencies conduct of evaluations against the strategy. The Treasurer’s Letter of Direction plus a recently released Issues Paper are available on the Productivity Commission’s website (link here).

The Issues Paper appears comprehensive and well structured. I don’t propose to summarise or discuss it at this point in time.

There are however a number of interesting issues embedded in the these developments. These include:

·         While the PC undoubtedly has substantial evaluation expertise, it is unclear how its proposed role in monitoring the effectiveness of programs will relate to the virtually identical role recently allocated to the new NIAA (see my previous post link here);

·         It is unclear whether the Treasurer was requesting advice (which is allowed under the PC’s legislation) or directing the PC to undertake an executive action (which does not appear to be within its statutory remit). To the extent that it is the latter, there are potential conflict of interest issues in the PC subsequently assessing and evaluation agency programs and policies utilising an evaluation framework it has developed and signed off on without any external approval;

·         It is also unclear what this new whole of government evaluation strategy/framework will mean for existing evaluation frameworks within agencies (including DPMC). There appears to be scope for substantial duplication, and there is also the risk or likelihood that the DPMC evaluation framework for the IAS, which took for excruciatingly slow years to be developed, and was audited by the ANAO at a cost of $335k, will be set aside in favour of the new whole of government evaluation framework to be promulgated in 2020;

·         Finally, there is a larger risk embedded in the establishment of an Indigenous Commissioner within the PC with a substantial role in oversighting the evaluation efforts of Commonwealth agencies related to Indigenous peoples. This larger risk is that the PC will feel disinclined in its mainstream inquiries and review to seriously consider policy issues related to the impact of mainstream programs on Indigenous peoples.

While the involvement of the Productivity Commission in Indigenous evaluation is in my view very positive, it is not without risk and challenges. To date, there has been virtually no public discussion of the benefits, risks, and challenges of the expanded role for the Productivity Commission in the Indigenous policy domain. For example, the proposed new arrangements do not fundamentally address the lack of independence inherent in program and policy agencies commissioning evaluations. The Commission’s current project is constrained by the remit contained in the Treasurer’s direction. There is a case for a broader assessment of the structure of evaluation in the Indigenous policy domain (and probably beyond). For example, there appears to be a strong case for the re-establishment of an independent statutory agency dedicated to undertaking key Indigenous evaluations such as previously existed with OEA.

In conclusion, the Government’s decision to bring the Productivity Commission into the policy and program evaluation space is welcome. It raises a number of issues, risks and challenges, including potential overlaps with the newly promulgated role of the National Indigenous Australians Agency. While a PC oversight role is welcome, it may not go far enough towards ensuring evaluations are consistently independent and free from political and other bias.

In relation to the DPMC (and from 1 July 2019 the NIAA), the ANAO ought to have recommended a major upgrade to the resourcing and management commitment to evaluation of policies and programs. There appears to have been a significant capability deficit in relation to building a robust and independent culture of evaluation. This will need to be a significant focus of the Minister and NIAA senior management going forward.

In relation to the ANAO, it seems to me it is time for an independent evaluation of the ANAO’s performance audit capabilities and performance. Perhaps that is something the Productivity Commission might take on as part of its priority setting for the new Indigenous Evaluation Strategy! After all, mainstream programs are increasingly impacting on First Nations, and the effectiveness of those programs and their oversight arrangements is a matter highly relevant to vulnerable people, including First Nations.