There are multiple systemic structural barriers that result in health and wellbeing inequities experienced by Aboriginal and Torres Strait Islander peoples (hereafter referred to as Indigenous). The recent Overcoming Indigenous Disadvantage 2020 report highlights some of these barriers, emphasising the importance of addressing the social determinants and cultural determinants of health by delivering evidence-based and culturally responsive health programs and initiatives.1 While is it estimated that $5.9 billion (AUS) is spent annually on Indigenous health programs and services in Australia, there is scant evidence to illustrate what works (or does not work), for who and why.2, 3 A mapping exercise of Indigenous-specific programs in Australia reported that of 1082 programs delivered, over 90% were not evaluated to assess whether programs achieved their goals and objectives.3 Also, it is reported that only a few evaluations used culturally appropriate methods for assessing program effectiveness.3 Recently, there have been a growing number of robust evaluations that are culturally responsive, from which we can learn. These have important implications that can guide the commissioning of future evaluations. The deficiency of evaluations in number and quality is problematic and viewed as stemming from the lack of effective engagement of commissioners and evaluators with Indigenous people.4-7 Although it is recognised that commissioning is a process, often involving a contractual arrangement, there is no accepted definition of commissioning. Despite the lack of an agreed definition, they share some similarities, with most definitions featuring the terms of reference for the evaluation and criteria for selecting the evaluator, and associated processes. The commissioning process is guided by the commissioning organisation's policy frameworks and principles that set the terms of reference on opportunities for Indigenous leadership and engagement in decision-making.8 The commissioning of an evaluation also sets the stage for the type of evaluation that will be conducted, the questions that will be asked and whether and how Indigenous people are engaged in the evaluation. Over the past decade, there have been numerous government reports and Indigenous organisations calling for genuine partnership with commissioning agencies when assessing the impact of programs.4, 7-10 Indigenous communities have consistently stated, ‘nothing about us without us!’. A comprehensive review of health policies, including policies on commissioning found that the term 'Indigenous engagement' is often used and the importance of involving Indigenous community representatives in decision-making is frequently recognised. However, most policies do not go beyond 'consultation'. Mere consultation does not allow for Indigenous leadership in the commissioning process, therefore, tends to privilege non-Indigenous perspectives. This often leads to culturally unsafe evaluations and offers little in the way of benefit to the Indigenous communities who are the beneficiaries of the evaluated program. In instances where Indigenous leadership is recognised, it tends to be at the local level once the program and evaluation plans have already been identified by the commissioner.7 Such approaches to commissioning could be framed as 'participatory' or insofar as the commissioned evaluation approach makes provisions for the contracted evaluator for engaging local Indigenous people in the evaluation. Recent findings from a trans-Tasman study on culturally safe evaluation suggest that the way health programs and services are commissioned has a critical impact on whether communities are engaged in evaluation planning and implementation processes.11 Suppose funders were to listen to Indigenous people and ask the right evaluation questions at the initial program development stages? Then evaluations may be of more significant benefit to Indigenous communities. Unfortunately, current commissioning processes lack transparency and are often unclear how decisions made at the procurement or contracting stage influence what happens on the ground, in communities. Commissioned evaluations that do not support the active engagement of Indigenous people in leadership positions are prone to several problems. In the absence of authentic engagement, problems can arise from commissioners and evaluators who exert their decision-making authority in identifying the evaluation questions, preferences for the evaluation design and outcome measures.12-14 The lack of Indigenous peoples’ leadership and engagement in program design, delivery and evaluation is often driven by short timeframes in tendering processes.11, 15 It is not uncommon for providers to respond to a tender within 4-6 weeks and be required to design, deliver and evaluate programs to change well-entrenched attitudes and behaviours within 2 years. Many evaluations are commissioned after programs have been running for one or more years, which pose significant challenges in harmonising the evaluation's key performance indicators. Evaluations that do not build-in adequate Indigenous leadership and governance from the conception of the evaluation run significant risks associated with program duplication, poor coordination, harming the community, asking the wrong questions, accessing the wrong data sources and methodological unfeasibility.16 These issues also explain why, despite the level of funding and services over decades, there appears to be little improvement in health outcomes as evidenced by the Queensland Productivity Commission [Queensland Productivity Commission]. Consequently, many evaluations do not adequately capture Indigenous peoples’ lived experience with the program and subsequent measures of program success. International research reveals that commissioners of evaluation need more realistic timescales and openness to different methods and approaches, including more formative evaluation. They need to seek out help from experts and clients or professionals in the commissioning of evaluation.17 Indeed, globally it has been shown that commissioners of evaluations can often hold unrealistically high expectations for health improvements and cost savings as the outcome of integrated care initiatives.18 Evaluation questions and designs may be the problem as commissioners of evaluations jump to comparative or controlled research designs too quickly, ‘underestimating the value in process evaluations and embedded approaches that are able to describe the model, explore the variable penetration and fidelity, and undertake the necessary “sensemaking” before moving on to examining outcomes’.17, p3. Through the development of Indigenous-related strategies, policies and frameworks aimed at improving health outcomes, there has been some recognition that the commissioning of evaluations needs to improve. To overcome the scant evaluation evidence and poor financial return on the significant expenditure relating to Indigenous programs, the Productivity Commission, through a national consultation process, has developed a national Indigenous Evaluation Strategy (IES).19 The strategy highlights the need to strengthen both the evaluation capability and cultural capability of public servants engaged in commissioning evaluations of Indigenous programs and services. The IES identifies several 'leading practices' for agencies to centre Indigenous, perspectives and knowledge in the commissioning of evaluations. For planning, designing and conducting evaluations, agencies need to allocate sufficient time and resources for extensive engagement with Indigenous people during evaluations. Agencies also must strive to support Indigenous people's leadership in decisions related to data planning, collection and use. The Lowitja Institute has published an evaluation framework to improve Indigenous health.7 The publication recommends that tender processes, evaluation contracts and agreements align with the principles of working with Indigenous people and ethical frameworks. Although the National Health and Medical Research Council's Ethical conduct in research with Aboriginal and Torres Strait Islander peoples and communities: Guidelines for researchers and stakeholders has a focus on Indigenous engagement in the design and implementation of research, it can also be applied to the commissioning and conduct of evaluations.20 The Queensland Productivity Commission's Indigenous Inquiry Final Report also puts forward a range of models for engaging with communities and evaluation.21 If the principles and strategies outlined in the existing resources are implemented effectively, there is significant potential to strengthen the quality of evaluations in Indigenous health and wellbeing contexts. However, is this enough? Furthermore, are the implementation of these frameworks and strategies effectively monitored to ensure that Indigenous people are being engaged in leadership positions across all stages of a commissioned evaluation? We contend that a much greater focus needs to be paid to the commissioning of Indigenous health evaluations by commissioning bodies. We know very little about the commissioning models (eg, participatory, co-design), processes and approaches that support the evaluation of Indigenous programs. With no transparency, there can be little accountability. There needs to be a more nuanced understanding of the commissioning approaches employed to support Indigenous engagement and leadership, the most effective approaches and the contexts in which they would be most beneficial. Moreover, there is a need for deeper understandings of the complex interplay between commissioners, evaluators, service providers, other stakeholders and Indigenous communities. How such relationships are navigated, can greatly influence the conduct of the evaluation and ultimately the utility of evaluation findings, along with the actions taken or not taken over the longer term. It is now time to shift the Indigenous evaluation discourse to one which is focused on quality and comprehensiveness, not deficiency and mediocrity. Indeed, scholars have referred to this shortfall as the ‘gate keeping of research in Aboriginal communities’.22 Are you ready to join us in a new era of evaluating Indigenous health programs and services in a culturally-responsive way?
No takes yet. Share an insight, caveat, or question.
Finlay et al. (2021) studied this question.
Synapse has enriched 5 related papers. Consider them for comparative context: