NIAA management of evaluation

Words are no deeds

Hamlet Act 1 scene 3

 

The Department of Finance publishes the Commonwealth Evaluation Policy on its website (link here). It is succinct and not particularly prescriptive nor onerous. It lays out key principles, and seeks to promote an evaluative culture. As part of such a culture, it notes that:

 Leadership that is positive about learning from performance monitoring and evaluation activities is a necessary condition for delivering effective outcomes and providing quality advice to Government, the Parliament and the public.

 

It goes on to lay out key elements in such leadership, including a series of governance actions which support an evaluative culture within agencies.  The Finance website also includes a toolkit which provides a detailed set of tools to assist agencies manage the evaluation process.

 

I took the opportunity over the weekend to examine the NIAA website (link here) so as to seek to understand its current approach to evaluation.

 

The NIAA website includes a section on areas of focus for the agency, and one of them is listed as Evidence and Evaluation. I set out below an extract from the relevant section of the NIAA website (link here):

Evaluations and Evidence

Evidence

The work of the National Indigenous Australians Agency is underpinned by effective data and evidence. The NIAA provides advice and information to the Minister for Indigenous Australians, the Australian Government, State and Territory governments, organisations, providers and communities to inform policy development, programs and monitoring of the effectiveness of programs for Aboriginal and Torres Strait Islander peoples….

Evaluation

Evaluation helps us to gain an understanding of what works and what doesn't, for whom and why. This kind of knowledge can help us to learn and improve what we do, supporting decision-making with the best available evidence.

Evaluation is integral to continual improvement. It is not a one-off, or 'tick the box' exercise. Evaluation supports evidence-informed policy development, public accountability, learning and performance reporting. Evaluation needs to be planned across the lifecycle of a program, from the very start.

The NIAA promotes and supports a culture that focusses on evaluation and performance improvement. We do this by:

·         Providing information, support and advice that embeds the functions of evidence and evaluation into policy and program design cycles.

·         Ensuring that monitoring and evaluation priorities we support are well designed and delivered in collaboration with Indigenous Australians.

The Evaluation Framework guides the conduct and development of a stronger approach to evaluation of programs and activities under the Indigenous Advancement Strategy. The goals of the Framework are to:

·         Generate high quality evidence that is used to inform decision making.

·         Strengthen Indigenous leadership in evaluation.

·         Build capability by fostering a collaborative culture of evaluative thinking and continuous learning.

·         Emphasise collaboration and ethical ways of doing high quality evaluation at the forefront of evaluation practice in order to inform decision making.

·         Promote dialogue and deliberation to further develop the maturity of evaluation over time.

Indigenous Advancement Strategy Evaluation Work Plan 

Publication of the Evaluation Work Plan supports the commitment to transparency made in the Evaluation Framework. It provides details on evaluation activities and enabling activities that are planned, underway or completed by the NIAA.

Evaluation activities on the Work Plan are reviewed by members of the Indigenous Evaluation Committee and approved by the NIAA Executive Board.

 

The IAS evaluation work plan (link here) is dated 5 January and appears to have been substantially reconceptualised in recent times. The NIAA website describes what has occurred in the following terms:

The NIAA has moved to a dynamic version of the IAS Evaluation Work Plan to make it accessible to a broader audience, and to improve the timeliness of the information presented. Previous versions of the IAS Evaluation Work Plan (2017–18 through 2020–21) are still available.

In this dynamic version of the work plan, users can find information on past and current evaluations, and enabling activities, according to IAS Program areas in the tables below. New evaluations and enabling activities are added to the work plan as they come on line and updates to the status of evaluations are made as they progress through the different stages to completion.

 

In effect, the NIAA appear to have shifted away from an annual evaluation plan to one which is continuously updated. In effect, they have conflated two separate potential data sets. The first is the plan that has been adopted for each financial year. The second is the status report on progress against the plan, and recording of any additions and/or deletions. What is unclear or made more opaque with the new approach is the extent to which evaluations are progressing against the workplan (delays are not obvious unless you delve below the front page), how long completed and discontinued evaluations will stay listed (I would argue they should remain available for at least 5 years).

 

A closer examination of the listed evaluations in the ‘dynamic work plan’ reveals a number of issues that do not sit well with the Department of Finance advice regarding the development of an evaluation culture. For example, embedded in the descriptions of various evaluations are multiple inconsistencies and what might be described as deliberate circumlocutions. A number of evaluations have been completed, sometimes two years ago, and yet are listed as ‘publication pending’. Is this actually the case, or has a decision been taken to not publish them and quietly remove them from the work plan at some time in the future. Other evaluations have been discontinued, but without any real explanation as to why this has occurred, and whether funds were expended and evaluation activities undertaken before the discontinuation was decided upon.

 

My advice to the NIAA would be to develop a template for the evaluation workplan that lays out a consistent and comprehensive set of core information, ideally including dates evaluations were commissioned, were begun, were completed, and were published, the evaluator, a succinct description of the aim of the evaluation, and once completed, a succinct description of the results. Additionally, there seems no good reason why the cost of the evaluation is not provided, particularly as the value of any external consultancy payments would be available on the Commonwealth tender site.

 

A more important issue to my mind is that it is apparent that there are multiple evaluations of relatively insignificant programs, of virtually zero strategic significance. The standout example to my mind is the Dog Operations Unit evaluation listed as being underway. It was approved in August 2020, contract start in February 2022, and is (apparently) yet to be finalised. The evaluation description is:

NIAA funds the Northern Territory Police, Fire and Emergency Services (NTPFES) to provide a Dog Operations Unit within the community of Groote Eylandt with the aim to improve community safety and crime prevention through enhanced police capability. This evaluation will assess the design, implementation and outcomes of the Dog Operations Unit.

 

A September 2022 ABC media report suggests that the NT Police only have four dogs in their unit, and that they are based in Darwin (link here). A February 2023 media report relating to an arrest on Groote suggests that the Dog Operations Unit was deployed to Groote along with the Drone Unit and the Territory Response Unit (presumably from Darwin). Perhaps the NIAA funding has ceased. This evaluation strikes me as chopping wood for practice, and one must also ask, why is it that the NT Government is not prepared to fund the operations of a dog unit in a remote community if that is what is required. There are a host of further questions raised by this evaluation, but the bigger issue is why does the NIAA evaluation work plan include miniscule projects of zero strategic import and not larger programs such as remote housing.

 

When I checked the housing program, there is an item for remote housing:

Northern Territory Remote Housing National Partnership Agreement

Program: Program 1.5 - Remote Australia Strategies; Activity type: Evaluation strategy; Last updated: 5th October 2021; Not continuing.

The reporting framework under the National Partnership Agreement has only recently been endorsed, with further amendments expected. The four Northern Territory Land Councils have now prioritised a joint review of leasing and housing models in the Northern Territory over the evaluation strategy. The joint review will inform opportunities to improve remote housing delivery in the Northern Territory.

 

There are two evaluations listed for leasing and housing models in the Northern Territory:

Northern Australia White Paper: Land Tenure Reform Pilots

Program: Program 1.1 - Jobs, Land and Economy; Activity type: Evaluation; Contract start: May 2019; Provider: Yaran Business Services; Last updated: 4th February 2022. Publication pending:

The land tenure reform pilots measure supported a range of projects across Northern Australia to test innovation in land-based activities on the Indigenous estate that support economic development and boost investment in the north. The purpose of the evaluation is to understand the extent the measure has contributed to the goals of the Northern Australia White Paper which sets out the Government’s 20-year plan for investment and collaborative support to drive growth in Australia’s north.

Northern Australia White Paper: Township Leasing and Land Administration

Program: Program 1.1 - Jobs, Land and Economy; Activity type: Evaluation; Contract start: May 2019; Provider: Yaran Business Services; Last updated: 4th February 2022. Publication pending:

The Aboriginal Land Rights (Northern Territory) Act 1976 (Land Rights Act) was amended in 2006 to provide for section 19A ‘township leases’, which cover entire community areas on Aboriginal land to support economic development. Township leases can be held either by the Executive Director of Township Leasing (EDTL) on behalf of the Commonwealth or, more recently, by a local Aboriginal corporation representing traditional owners (community entity model).

The evaluation considers the communities that have taken up township leases in the Northern Territory (NT) from introduction to 2017 and how they have performed against initial expectations and policy objectives. As well as examining the direct benefits of township leasing to individual communities, the evaluation examines the extent to which township leasing provides social, cultural, economic benefits to Aboriginal peoples and communities in the NT.

 

These two evaluations appear to relate to the two evaluations apparently proposed by the NT Land Councils and focus on perfectly legitimate policy and program issues. It is unclear why they are yet to be made public. However, the fact that these are legitimate subjects for evaluation is hardly a justification for not evaluating what I understand is a four year $550m investment in remote housing in the NT by the Commonwealth (link here).

 

According to a February 2022 ANAO performance audit of remote housing in the NT (link here), by virtue of the existence of some 3500 forty year housing leases from traditional owners to the Commonwealth (which are subleased to the NTG), the Commonwealth retains an underlying interest in the majority of public housing in remote communities in the NT, which represent 59 percent of the remote housing portfolio in the NT (see para 1.23). That ANAO audit (which was not as broad as an evaluation) identified numerous shortcomings in the NIAA management of remote housing investment in the NT. The first 32 paragraphs provide a useful summary of the ANAO conclusions which are expressed in a dialect unique to the ANAO which I refer to as ‘muted bureaucratese’. I have cherry picked and reformatted a few sentences from that summary section to provide some concentrated flavour:

NIAA’s administration of funding for remote housing in the NT has been partly effective. The development of the National Partnership was partly effective. However, the National Partnership Implementation Plan has significant weaknesses, and advice to the minister did not include analysis of some of the National Partnership’s key parameters. NIAA has been partly effective in assessing the delivery of the program of works under the National Partnership. NIAA has not fully developed or implemented a risk-based approach to determining what assurance is necessary to verify the Northern Territory (NT) Government’s achievement against National Partnership targets. NIAA has been partly effective in ensuring that the National Partnership’s outcomes will be achieved. NIAA’s assessment of the delivery of capital works has been partly effective. NIAA’s assessment of the delivery of PTM services has been partly effective. NIAA does not gain assurance over the performance results reported to it by the NT Government, and has not always used the correct reporting period to assess the NT Government’s results and to recommend payments. NIAA has been partly effective in ensuring transparency about how money has been spent. However, NIAA does not have assurance that the NT Government will meet its $550 million co-contribution commitment over the life of the National Partnership. NIAA has not managed risks to the delivery of the National Partnership effectively…

 

Partly effective indeed. See my previous post on this performance audit for my own assessment (hint: it is tougher than the ANAO assessment: link here). Given these performance audit findings on what is a billion dollar joint Commonwealth / NTG investment program, and one which will necessarily continue into the future despite the time limits on the National Partnership, the case for discontinuing the proposed remote housing national partnership evaluation appears wafer thin. And this is just one of the issues embedded within the work plan.

 

Returning to the bigger picture, it is clear that the evaluations listed in the NIAA evaluation work plan raise myriad issues. One wonders whether there would be merit in undertaking a short sharp action oriented review of the management of the evaluation program by NIAA. I think it could be undertaken over ten days by a competent policy analyst.

 

The major problem with the NIAA evaluation workplan is that it provides no indication of the overall extent of the evaluations undertaken over the past three years and planned for the next three years. The result is that it is virtually impossible to assess the extent of NIAA’s evaluation efforts against the agency’s overall policy and program footprint.

 

A related issue is that there is no evidence of narrative provided to demonstrate how previous evaluations have contributed to revised or new program initiatives.

 

It is my assessment that the NIAA appears to be merely going through the motions: an evaluation framework document full of the expected rhetoric, a rather opaque Indigenous Evaluation Committee to provide independent strategic and technical advice, yet where it really counts, a ‘dynamic work plan’ replete with inconsistencies and an absence of strategic direction.

 

What is clear is that the reported budget announcement later this week of a centralised unit in Treasury to promote improved evaluation across the Commonwealth will have its work cut out in at least one area of public policy. In the meantime, it is worth remembering that the consequences of ineffective programs and policy is borne in large measure by disadvantaged citizens.