- Systematic review
- Open Access
The effectiveness of implementation in Indigenous Australian healthcare: an overview of literature reviews
International Journal for Equity in Health volume 15, Article number: 47 (2016)
Effective implementation can maximise the beneficial impacts of health services. It is therefore important to review implementation in the context of Indigenous populations, who suffer some of the greatest disadvantage within developed countries. This paper analyses Aboriginal and Torres Strait Islander (hereafter Indigenous) Australian health implementation reviews to examine the research question: What is the effectiveness of implementation, as reported in the Indigenous Australian health implementation literature?
Eight databases were systematically searched to find reviews of Indigenous Australian health services and/or programs where implementation was the focus. Search terms included Aborigin* OR Indigen* OR Torres AND health AND service OR program* OR intervention AND implementation (or like terms) AND Australia AND review. Review findings were analysed through the lens of the PARiHS framework which theorises that successful implementation occurs through the interplay of evidence, context and facilitation. The review followed Cochrane methods but was not registered.
Six reviews were found; these encompassed 107 studies that considered health service/program implementation. Included studies described many health services implemented across Australia as not underpinned by rigorous impact evaluation; nevertheless implementers tended to prefer evidence-based interventions. Effective implementation was supported by clearly defined management systems, employment of Indigenous health workers as leaders, community control, partnerships, tailoring for diverse places and settings; and active facilitation methods. Short-term funding meant most studies focused on implementation in one site through pilot initiatives. Only two mentioned cost effectiveness. Indigenous Australian studies incorporated two elements not included in the PARiHS reference guide: the value of community control and equity of service provision across sites.
Comparison of the Indigenous Australian review findings against the PARiHS reference guide elements suggested a fledgling but growing state of Indigenous implementation research, and considerable scope to improve the effectiveness of implementation. Further research is required to explore Indigenous people’s understandings of what is important in healthcare implementation; particularly in relation to the value of community control and equity issues.
A previous study by the lead author of this paper claimed that there has been little attention in the implementation science literature paid to researching the implementation of health services and programs targeting Indigenous people globally . This claim was based on a prior systematic search of the Indigenous Australian transfer literature which produced only 14 studies  as well as a quick scoping search for global Indigenous health implementation literature in Implementation Science which resulted in only six papers. Two years on, the likelihood that effective implementation can maximise the beneficial impacts and cost effectiveness of health services or programs, means that it is important to review whether there has been an improvement in the scope of the health implementation literature targeting Indigenous population groups, who suffer some of the greatest disadvantage within developed countries [2–4]. The term implementation is used to refer to the processes by which a service or program is assimilated into use within an organisation .
The limited scope of Indigenous implementation literature in 2013 contrasts with the burgeoning nature of implementation science literature globally. For example, the systematic review of the global diffusion of health innovations literature by Greenhalgh et al.  appraised a hefty 1000 relevant full text papers and book chapters spanning the 40 years since Rogers  seminal book, Diffusion of innovations. Yet an update of the Greenhalgh et al.  review using similar search terms for just 8 years (2003–2011) identified more than double that rate - a further 404 publications that identified theoretical implementation models or empirical studies . The scope of conceptual understandings had also expanded – evidenced by the appearance of new concepts and terms since 2002—these included scaling, cross-organisational linkages and clusters, knowledge exchange, knowledge translation and social and professional knowledge networks.
Implementation research in health is growing because of its contribution to understanding how the evidence to practice gap in healthcare can be reduced and the benefits of health services and programs maximised through their effective implementation within diverse contexts . However, a consistent theme in the literature is that reducing the evidence into practice gap is difficult to achieve . The complexity of implementation accounts for part of this difficulty. Thus it is also important to understand the elements that comprise effective implementation, and whether and how these elements are described in the available Indigenous studies.
In Australia, the importance of understanding health implementation in the Indigenous context was recognised by the Lowitja Institute, Australia’s national institute for Aboriginal and Torres Strait Islander health research. In 2011, the Lowitja Institute held collaborative workshops for researchers and community members to inform the improvement of knowledge and understanding about the uptake and implementation of Indigenous health promotion approaches. A project was funded to systematically examine the characteristics, implementation and effects of Indigenous health promotion tools  and to develop an Indigenous health promotion tool implementation model . In a related Lowitja Institute project, Brands  reviewed the international implementation science literature to determine its relevance to guiding improvements in the implementation of Indigenous Australian health services and programs.
Systematic reviews (or overviews) of reviews provide a method for summarising or scoping the overall evidence from more than one systematic review, and encompassing the results from a combination of populations, interventions, conditions, contexts and outcomes . Reviews of reviews bring together the evidence in one place by synthesising or comparing the findings of related reviews to provide policy makers and practitioner with the evidence they need to improve implementation, taking account of variable quality and scope. This paper builds on the work of the aforementioned Lowitja projects by reviewing the reviews of the Indigenous Australian health implementation literature to determine the scope of the literature and “to make sense of [the] complexity [of implementation], and the elements that require attention if implementation is more likely to be successful” (, p. 3).
We applied Kitson et al.’s  theoretical model ‘Promoting Action on Research Implementation in Health Services’ (PARiHS) framework in mapping the elements which could provide explanations of the processes by which Indigenous health implementation occurs which could guide improvements in the acceptability, adoption, appropriateness, feasibility, replication, implementation cost, spread and/or sustainability of implemented health services or programs. Brands  identified the PARiHS framework as a particularly accessible and flexible international implementation framework that could be usefully applied to Indigenous Australian health services and programs.
The PARiHS framework is based on the theory that successful implementation of evidence into practice is determined by a planned facilitated process involving interplay between three elements: 1) the level and nature of the evidence for a health service or program proposed for adoption; 2) the context or environment into which the evidence is to be placed; and 3) the facilitation or method of implementation [13, 14]. Evidence is defined as the form of knowledge that is made implementable through services and programs, and includes knowledge from research, clinical expertise, and/or local knowledge from clients . Context refers to the environment or setting in which the proposed service or program is to be implemented (including the broad macro, organisational and individual factors) [14, 15]; and the readiness for implementation . Facilitation refers to the process by which change managers help individuals and teams to understand what they need to change and how they need to change it in order to apply evidence to practice [13–15]. Successful implementation encompasses three components: 1) an implementation plan and its realisation; 2) an evidence based practice innovation uptake (i.e., uptake of a clinical intervention and/or delivery system interventions); and 3) the achievement of patient and organisational outcomes . Whilst development of the PARiHS framework was based on observation of implementation practice and testing in four empirical case studies, it has since been further developed and applied in diverse situations .
International theoretical understandings of implementation may not adequately explain the implementation of health services and programs to Indigenous population groups within developed countries. However, the PARiHS framework was used in preference to Indigenous theoretical models because there are few theoretical conceptualisations of implementation processes within Indigenous Australian health settings, and those that exist are program-specific [1, 17]. Through applying PARiHS, we also attempted to determine the relevance and utility of this framework for understanding implementation in the Indigenous Australian health context. Hence, the three underexplored research questions were: 1) What is the scope of the Indigenous Australian health implementation literature; 2) What is the level and nature of the evidence that underpins implementation, the contexts into which the evidence is placed, and the methods for facilitating implementation; and 3) Is the PARiHS framework useful for understanding implementation in the Indigenous Australian health context? Thus, this systematic review of systematic reviews was conducted to appraise, summarise and bring together the findings of Indigenous Australian health implementation reviews and compare and contrast these with those elements outlined in the international PARiHS framework as being critical to successful implementation.
We did not develop a written review protocol, although the focus and methods of the systematic search were established prior to beginning work. The methods were based on Cochrane guidelines and prior reviews by the author team [2, 9, 18–21]. Reviews of the Indigenous Australian healthcare implementation literature were identified and classified using a process that was consistent with PRISMA-P (Preferred Reporting Item for Systematic Review and Meta-analysis) guidelines .
The search strategy is summarised in Fig. 1. First, eight electronic databases were searched: Informit, Infotrac, Blackwells Publishing, Proquest, Taylor and Francis, JStor, Medline and the Australian Indigenous HealthInfoNet. The following terms were searched in either the title or abstract, article or MESH heading of peer reviewed papers: Aborigin* OR Indigen* OR Torres AND health AND service OR program* OR intervention AND implementation (or like terms listed below) AND Australia AND review. In consideration of the broad and expanding international implementation literature, search terms related to implementation included: dissemination, extension, transfer, translation, adaptation, uptake, utilisation, spread and scaling.
Separate searches were performed for each database using database specific subject headings and keywords. The combined searches of the databases produced references that were imported into a bibliographic citation management software EndNote X7. The authors of this study also drew on their knowledge of Indigenous healthcare implementation literature to identify papers.
To capture evidence of Indigenous Australian health implementation, studies were included if they if they were reviews (systematic, rapid, narrative or other); focussed on health services and/or programs for Indigenous Australians; implementation was the focus of the review; and studies were published in the English language between 2005 and 2014 (inclusive) in the peer review literature. The search was limited to peer reviewed studies as a marker for quality reviews. Given that implementation is a relatively new field of study, a decade of literature was considered sufficient for analysing the majority of Indigenous health implementation studies and was feasible within the scope of this project.
Electronic database searching was completed on June 26th, 2015 and yielded 206 citations. An additional three reviews were added based on the authors’ knowledge. After duplicates were removed, 202 citations were screened by one author (JM) to remove articles that were clearly not relevant to the review based on the title, abstracts, journals and keywords of the articles. This screening resulted in 189 citations being excluded from the review.
For the studies considered to meet the review’s eligibility criteria, copies of full text articles were obtained (n = 13). Two reviewers (JM, RB) independently assessed the 13 articles with 77 % agreement. Discrepancies were resolved by discussion. Seven studies were excluded from the review for the following reasons: Not a review of the literature (n = 1), not Indigenous Australian (n = 3), not focussed on implementation (n = 3). Ultimately, six studies were included in the analysis, including two of the reviews that had been added based on the authors’ knowledge [2, 23].
Quality of studies
The quality appraisal methods used by the authors of the included reviews were taken at face value and are reported in Table 1.
Classification of studies
Aligned with the PARiHS framework, studies were categorised by the first author and year, service or program name, evidence that informed the service or program, organisational context and method for facilitating implementation.
Reviews and included studies
We identified six systematic searches of the Indigenous Australian health literature that focussed on implementation and fit the eligibility criteria (Table 1).
These six reviews incorporated 107 Indigenous Australian healthcare implementation studies (after removing duplicates) which included evaluations and descriptions of the implementation of services and programs to address diverse health conditions, services and systems. Only 11 of the studies were included in more than one review, with two studies appearing in three reviews. The six reviews focussed on chronic disease; smoking, nutrition, alcohol and physical activity (SNAP) interventions, suicide prevention, alcohol and other drug treatment and health promotion tools. The reviews utilised differing terms to describe similar concepts of implementation. In addition to the term “implementation” which is defined above, the related terms utilised were: 1) dissemination (the extent of uptake of evidence-based interventions by health-care providers) ; 2) transfer the process and practice by which an initiative is made available and accessible to a new setting through interactive engagement between organisational representatives and participants; 3) uptake (the decisions made, often by multiple agents, to make full use of an initiative as the best course of action available) ; and spread (the idea that a program expands to increase the number of people served) .
Only three of the reviews employed a measure of study quality, with the quality of studies generally found to be weak or moderate. For example, Gibson et al.  used the JBI SUMMARI tool and found the quality of qualitative and quantitative studies overall to be moderate, mainly due to insufficient description of the methodological approach. McCalman et al.  assessed the quality of quantitative evaluation studies using the Effective Public Health Practice Project (EPHPP) tool and qualitative studies using the Critical Appraisal Skills Program (CASP) tool, and found it to be strong for only five/74 (7 %) studies. McCalman et al.  rated quality using peer review and study design as quality measures and found that none of the evaluation studies were based on experimental research designs. These findings of generally poor methodological quality were not surprising since Indigenous health research globally and locally has been predominantly descriptive . Reflecting the logistical, ethical and methodological challenges associated with conducting rigorous intervention and implementation research in Indigenous health-care settings, few studies have met rigorous methodological criteria [27, 28].
Synthesis of findings
A synthesis of the findings from the Indigenous Australian health implementation reviews is provided here. We analysed the findings according to the research questions.
What is the level and nature of the evidence that underpins implementation?
Reviews stated that in many cases, the new services and programs that are constantly being introduced into Indigenous primary healthcare services had not been previously systematically evaluated to determine their effectiveness. For example, McCalman et al.  found that the impact of only 15 % of Indigenous health promotion tools implemented had been evaluated and McCalman et al.  found that only 31 % of 119 studies that reported transfer of an Indigenous health service or program had evaluated the impact of the service or program. Reviews recommended continuing and improving use of valid and reliable measures and rigorous research designs are required to accurately quantify the effect of implementation strategies [2, 9, 23, 29].
However, there was some evidence from reviews that implementers may have been aware of the research evidence and preferenced the implementation of evidence-based services or programs. For example, Clifford et al.  reported that all but one of the 11 studies included in their review explicitly reported using evidence-based resources and/or guidelines. McCalman et al.  found that of 37/119 transfer studies (31.1 %) evaluated the impact of the service or program compared to only 16.7 % of impact evaluation studies among the remaining 1192 publications reviewed. Thus, while the evidence for what works in improving Indigenous Australian health has been dubbed “the sorry state of the evidence base” (, p. 566), in some cases the available evidence was being accessed to inform the choice of which intervention to implement.
While the state of the research evidence can make it difficult for practitioners to confidently select innovations that will guide improvements in health practice, Gray et al.  argued that concerns about the scientific rigour of the research need to be balanced by the overriding importance of Indigenous control of the research process. Indigenous control encompasses the identification of the research topic and building the capacity of Indigenous researchers to disseminate, translate and implement results . When Indigenous people had control, the credibility of the evidence source was enhanced with Indigenous end users. Reviews also profit from Indigenous co-authorship, since the interpretation of study findings from an Indigenous worldview perspective demonstrates respect and increases the likelihood of a converging interpretation of the aims and targets of implementation, and hence of research benefit . Five of the six reviews [2, 9, 23, 26, 30] were identified as being co-authored by Indigenous researchers.
Paramount within Indigenous Australian health implementation reviews were issues related to what constitutes evidence. Reviews described implementation processes as best serving Indigenous populations when they drew from both the available scientific evidence as well as, importantly, the knowledge of local Indigenous people and healthcare workers to inform locally relevant program implementation. McCalman et al.  found that 70 % of publications that described or evaluated the development of Indigenous health promotion tools specified that community members were consulted or collaboratively involved in developing or adapting the tool. Gibson et al.  iterated the importance of involving communities in the design and implementation of services and programs to ensure that their particular health concerns were addressed. Similarly, Gray et al.  considered that optimising and maintaining investment required recognition of cultural difference in both the planning and delivery of alcohol services and programs. They stated (p. 487):
Aboriginal community controlled organisations (ACCOs), their practices and values reflect the groups that established them and which they serve. These cultural elements affect the relationships between Aboriginal and mainstream organisations, implementation of specific interventions within ACCOs, and patient-practitioner relationships… recognition of cultural differences is central to modifications to the AUDIT and ‘Drinkless’ materials by the SSWAHS team; and the clash of cultural values and failure to recognise differences, highlighted in the AADS study, demonstrates how provision of quality care can be undermined.
However, few reviews explicitly considered how Indigenous knowledge (such as the principles underpinning health programs and services) were reflected in program or service implementation. In one exception, in the context of suicide prevention, Ridani et al.  exemplified the importance of recognising the divergent interpretations of implementation from Western and Indigenous perspectives. Reflecting the importance of engagement with Indigenous knowledges in designing services and programs and in interpreting the outcomes of their implementation, the authors found that while non-Indigenous run programs were individualistic and treatment oriented, Indigenous run programs were more community-focussed and holistic. Additionally, Gray et al.  noted conflicts between varying knowledge sources, including: “divergent views regarding staff skills and competencies, including the relative importance of clinical and cultural competencies.” Thus, although reviews recognised the role of evidence from quality research, clinical experience and local knowledge to tailor services or programs appropriately for the context, this was reported inconsistently and evidence was likely to have been inconsistently embedded in service or program implementation.
What are the contexts into which the evidence is placed and how does implementation work within diverse health contexts?
Health services and programs are implemented across widely varying geographical, historical, social and cultural contexts. This diversity means that implementation efforts need to be tailored to local contexts and the effectiveness of implementation can vary considerably, even for the same service or program. At a local level, Indigenous Australian reviews documented the implementation of health services and programs by government departments, peak or “hub” provider organisations, primary healthcare organisations, partnerships and networks, training organisations, schools, regional health organisations, men’s groups, sports clubs, general practice and non-government organisations . Within organisations, implementation was facilitated by primary healthcare workers, health specialists, Indigenous community members, health promotion officers, policy makers, community/welfare workers, specialist alcohol and other drug, tobacco, mental health, and sexual health workers, and Indigenous and non-Indigenous academics . As employees, these individual change agents had acted for the organisations’ interests but maintained some discretion to interact, advocate and negotiate for how implementation was facilitated.
Some community and organisational contexts were more conducive to the successful implementation of health services and programs than others. Within organisational settings, studies identified the benefits of support from management. Clearly defined management structures and procedures were identified as important in the provision of Indigenous alcohol and other drug services; Gray et al.  highlighted the need to formalise processes and commitments within organisational policy and procedures. Community control was mentioned in several reviews as facilitating implementation of health services/programs to Indigenous community members. Mechanisms by which this occurred were by enabling community engagement with service or program design , promoting cultural activities and other community-focussed interventions that fostered a sense of community connectedness and pride , optimising spread by providing services targeted to Indigenous Australians ; and committing to program longevity [26, 29].
A key facilitator of implementation was the employment of local Indigenous health workers. Implementation was facilitated through their roles as cultural mentors to non-Indigenous staff and ability to provide a culturally safe service . Gibson et al. found that the participation of Indigenous health workers in all levels of decision making relating to chronic disease management was particularly important for implementing appropriate and effective services to Indigenous patients and their families.
Barriers to implementation included recruiting qualified staff, high staff turnover and the use of temporary staff. For example, Gray et al.  reported on a workplace survey to assess the needs for staff training within an alcohol and other drug service, and development of a training program. However, high staff turnover precluded completion of the outcome evaluation of the training program within the project time frame . Reviews emphasised the need for appropriate staff development, including time to attend training and a backfill of staff; a dedicated coordinator and/or facilitator role  and change management and communications plans.
A commitment to partnerships, collaboration and networking was also important [9, 23, 26, 30]. Gray et al.  reported on a partnership for the provision of counselling, withdrawal management and residential rehabilitation for Indigenous clients. The authors described the partnership as “fraught with tension” as a result of a range of structural, historical, cultural and personal factors—compounded by client complexity and the “paternalism of the funding agency” (p. 485). Despite these challenges, most staff interviewed considered the potential for partnerships and the need for such partnerships to be voluntary, equitable, accountable and based on trust. Partnerships between Indigenous practitioners and researchers were seen to be a positive outcome of the implementation process [9, 30]. A collegiate approach encompassing the sharing of ideas in the development of research proposals and projects, training, and the funding of projects was valued and led to significant improvements in research design, process and outcomes .
All reviews also cited macro political, social and economic factors, such as government policies, funding amounts and duration, and the economic and social determinants of Indigenous health, that influenced the implementation of health services and programs [2, 9, 26, 29, 30]. In particular, the provision of adequate and enduring funding for Indigenous health service and program implementation was required for health infrastructure and recruiting, training and support for additional members of the health workforce, as well as to continue the delivery of services or programs. For example, Gibson et al.  stated:
Indigenous-specific services often tended to rely on a multitude of short-term government funding arrangements which threaten their sustainability and result in overwhelming reporting requirements. Funding arrangements between Indigenous community-controlled health services and governments tend to be more complex than those between governments and general practice or tiers of government in Australia and elsewhere. One of the key issues to be considered during the design phase is therefore adequate funding for both the implementation and sustainability of an intervention.
Reviews critiqued the short-term nature of funding grants as inefficient; finding that the uncertainty of funding created a barrier to sustained implementation [2, 26, 29, 30]. For example, Gray et al.  found that the short-term nature of funding exacerbated problems in alcohol and other drug treatment such as the difficulty of recruiting qualified staff (particularly in rural and remote areas), establishing collaborative relationships and gaining community acceptance. In contrast, modest additional resources to alcohol and other drug projects produced change and enhanced outcomes through increased capacity to deliver services; improved case identification; increased client engagement; improved interagency and community collaboration; and development of more appropriate assessment tools and resources. As these became embedded in service provision, the initial investment continued to have a positive effect and this success led to further funding allocations by government agencies .
Studies reported wide variations in implementation between local areas. For example, McCalman et al.  highlighted the findings of an included study by Tursan d’Espaignet et al.  of significant improvements in birth weight following the introduction of the Strong Women, Strong Babies, Strong Culture Program in one group of Indigenous communities in the Northern Territory with no significant change in the second group. The authors concluded that there was a need to better understand how implementation of the same program differed across the community contexts. This finding implied a need to identify ways to support implementation of the strategies in areas of greatest need; thereby making the implementation of services and programs more effective and equitable. It also suggests that services/programs can not necessarily be directly transferred across contexts; for example, from mainstream general practice to Indigenous health services; between urban, rural and remote settings; or between communities.
What are the methods for facilitating implementation and what facilitation strategies work?
The methods for facilitating the delivery of Indigenous health services or programs affected the success of the service or program. Drawing from a typology of types of spread identified by Ovretveit , McCalman et al.  found that the most common type of transfer and implementation of health services and programs reported in studies was through the central development but decentralised implementation of the initiative; this involved community-based participation and adaptation of the intervention (12/21 or 57 % studies). Also found were studies of informal grass roots transfer (5/21 studies). An example was provided by Ridani et al.  who reported the lateral sharing of suicide prevention knowledge between communities as a process “vital in empowering communities to help each other” (p. 25). The third type of transfer found was hierarchical top-down implementation (3/21 studies).
Reviews considered that services and programs effective in non- Indigenous communities could not simply be implemented in Indigenous settings without consideration of cultural differences . Local Indigenous knowledge was considered important in tailoring mainstream health services and programs to fit the diverse contexts of Indigenous Australian health. For example, Gray et al.  noted that the provision of training and tailored outreach support resulted in modest evidence of improvements in alcohol screening in community controlled health services. However, McCalman et al.  reported that the majority of Indigenous health promotion tools were designed for national or state/territory use, with only 12 % developed or tailored for regional or local use.
Reviews cited the critical importance of Indigenous leadership, governance and involvement in all aspects of decision making and responses to issues affecting their lives. For example, Ridani et al.  found that suicide prevention programs that were wholly or partly Indigenous owned, employed creative and community-focussed strategies such as art classes, dancing events, theatrical showcases and cultural camps; whereas those not owned or run by Indigenous corporations used workshops as their main mode of delivery. Reviews cited the need for respect for Indigenous peoples’ information and knowledge systems, safe spaces for knowledge exchange, and strengthening of trust, engagement and participation within health and other systems.
Reviews also described how passive implementation strategies, such as the distribution of clinical guidelines or resources, were generally ineffective compared to the use of actively facilitated implementation, which increased readiness for implementation [9, 23, 26]. Active facilitation roles were provided by agents external to the implementation context, such as researchers, and by internal change agents, such as continuous quality improvement managers or health practitioners. Gibson et al.  cited the quality of patient/provider partnerships as one of five key factors that enabled or inhibited the implementation of services and programs aimed at improving chronic disease primary health care for Indigenous people. Developing respectful, trusting and safe relationships with providers was particularly important given historical policies and practices that excluded and discriminated against Indigenous people; and achieving this was often time consuming and required effort and understanding on the part of the provider.
In corollary, McCalman et al.  highlighted an included study by Hunter et al.  that found that distribution of clinical resources alone was not sufficient to ensure use. The evaluation of the National Recommendations for the Clinical Management of Alcohol-Related Problems in Indigenous Primary Care Settings found that although these clinical guidelines were produced through a series of workshops involving some 15 expert clinicians and workers in the area of Indigenous primary care and substance use; their dissemination to doctors and nurses working with Indigenous patients required 74 workshops conducted by an Indigenous and non-Indigenous medical practitioner and worker with expertise in Indigenous alcohol use. Particularly for medical practitioners, appropriate introduction by acknowledged experts not only increased use, but also positively influenced willingness to reflect on their interactions with Indigenous drinkers and to engage with alcohol-related problems using the guidelines as part of primary clinical care. Hence, the credibility of the new guidelines for end users was influenced not only by the use of an evidence-based intervention, but also by the perceived level of expertise and active facilitation of the implementation process.
From their review of dissemination strategies for Indigenous Australian SNAP interventions, Clifford et al.  found that half of the studies employed a combination of strategies that was less than optimal. Dissemination strategies included academic detailing (educational outreach or skills training); continuing medical education; reminder systems, and reinforcement contact (structured follow-up contact with practitioners to reinforce education or training). Most studies reported the tertiary prevention of diabetes care, rather than the uptake of secondary preventive interventions such as brief intervention in Indigenous primary health care settings.
What works in successfully implementing health services and programs for Indigenous Australians?
Several reviews reported a need to plan implementation. For example Gray et al.  urged: “to optimise and maintain investment, cultural difference needs to be recognised in both planning and delivery of alcohol interventions”. Yet generally, implementation planning was considered primarily when plans were abandoned due to unexpected contingencies. For example, Gray et al.  reported: “staff turnover led to abandonment of a plan to conduct a post-intervention evaluation survey” and “initially, it was planned to conduct the individual projects over a 12-month period. However, all projects exceeded this” (p. 487). The use of formative evaluation to refine implementation plans to account for such unanticipated factors was reported. For example, McCalman et al.  found that 28/119 (23.5 %) studies reported only process evaluation measures of reach, satisfaction, quality and how implementation occurred. However, these examples point to a need for improved reporting of implementation planning and refinement.
The reported outcomes of implementation strategies included the uptake of secondary and tertiary preventive interventions, tailored to clinic, patient and health-care provider factors in specific settings, and cost-effectively implemented across multiple Indigenous health-care services and programs . Also reported as positive outcomes were the close networks established between practitioners and researchers [9, 30]. Yet, consistent with perceptions of Indigenous people that research has produced little health benefits , the perceptions of Indigenous stakeholders in terms of the usefulness of such changes were not reported.
Gray et al.  noted that the transfer and uptake of services and programs encompasses a long-term process. Although all reviews cited the importance of sustaining service or program implementation, most of the included studies focused on singular incidents of implementation through pilot initiatives. Program longevity tended to be linked to community ownership . Ridani et al.  concluded:
Piecemeal and ad hoc approaches are unlikely to be effective and near impossible to evaluate. Perhaps the challenge in moving forward will be to determine how we can coordinate intervention efforts so that they not only have an effect, but they also demonstrate it.
The spread of services or programs to other sites was quantified by McCalman et al. . Of 1311 publications which they identified as dealing with Indigenous Australian health services, programs or innovations, 119 (9.1 %) referred to their transfer. Transfer or implementation was the primary focus of 21 (1.6 %) of these studies and was only considered by the remaining 98 (7.5 %) studies. The authors concluded that few studies focus on the process by which transfer or implementation of health services or programs occurred or their effectiveness in the new site.
The cost effectiveness of implementation was mentioned in only two of the six reviews [23, 30]. Based on an audit showing that few Indigenous people were accessing alcohol and other drug services in Sydney’s South West, Gray et al.  reported a project to assess the potential role of ‘community-based education and brief intervention’ in reducing harm. The intervention was labour intensive, comprising screening of community group members; interactive education sessions to increase awareness of alcohol-related harms, alcohol guidelines and availability of services; and feeding back screening scores and providing one-to-one brief interventions for those at risk. The authors suggested positive results and potential economies of funding, staffing and training if the approach became a routine element of service provision. Nevertheless, both Gray et al.  and Clifford et al.  suggested a need for further examination of cost-effectiveness.
The method used to search electronic databases may not have located all Indigenous Australian health implementation reviews published in the peer reviewed literature from 2005 to 2014. Only a small number of implementation-focussed reviews were found however, given the small number of primary studies, this was not surprising. Limitations in the findings of this analysis include potential bias given the small number of papers found and the first authorship of two of the reviews by the first author of this paper. Gaps in the information provided by the studies include foci on only a limited range of Indigenous health issues (chronic disease, SNAP interventions, suicide prevention, alcohol and other drug treatment and health promotion tools). Systematic reviews of reviews generally create a meta-analysis of the included reviews; however the descriptive foci of the included reviews meant that this was not feasible.
More than ten years ago, Ring and Brown  critiqued the slow progress in Indigenous Australian health improvement as due to “lack of commitment to and implementation of already existing policies”. The reviews which provided the basis for this paper suggest that there is a growing body of published research that describes or evaluates the implementation strategies that would reliably result in health improvement, assist in accessing hard-to-reach community members, or provide best value for money. However, the quality of studies is poor or moderate. This paper found value in the PARiHS theoretical model for identifying the broad elements critical to implementing Indigenous Australian health services and programs. Application of the PARiHS framework to the Indigenous Australian healthcare implementation literature identified a range of key policy and management issues where there is scope for improvement.
Table 2 provides a mapping of the findings of Indigenous Australian health implementation reviews to the PARiHS framework elements. Included in that table are the key elements highlighted in the PARiHS reference guide ; the comparable findings in the Indigenous Australian implementation reviews; elements that were highlighted in the PARiHS framework but missing from the Indigenous Australian reviews; and elements highlighted in the Indigenous Australian reviews but missing from PARiHS. It should be noted that the missing elements are not necessarily absent in practice; simply that they were not reported in reviews.
Comparison of the Indigenous Australian review findings against the elements outlined in the PARiHS reference guide found a high level of consistency. For example, although the implementation of many Indigenous health services and programs was not underpinned by high quality or tailored impact evaluation of their effects, where available, implementers preferred evidence-based programs. Implementation success for Indigenous health services and programs was enhanced by recognition of the value of local Indigenous knowledge, clearly defined management systems, commitments to employing Indigenous health workers as leaders, community control, effective partnerships, tailoring for diverse sites; and active facilitation methods. Reported barriers to implementation included poor staff recruitment and retention, inadequate staff development, the unavailability of designated implementation leaders and absence or inadequacy of implementation or communication plans. All reviews cited the importance of sustaining service or program implementation, yet short-term funding meant that most included studies focused on singular incidents of implementation through pilot initiatives, and only two mentioned cost effectiveness.
PARiHS was not useful for explaining the value of community control and improvement of equity of service provision across sites. However, comparison with the PARiHS framework did identify five elements that were missing from the Indigenous Australian health implementation reviews. These were the extent to which: 1) managers and other stakeholders supported implementation and saw it as appropriate to their roles; 2) internal facilitation resources, authority, skilled facilitators, communication channels and networks were available; 3) implementation fit with organisational priorities; 4) local or other evaluation data was used to inform practice and evaluate implementation; and 5) Indigenous stakeholders perceived the usefulness of changes through implementation.
Conclusion and implications
There are three key implications of this analysis of Indigenous health implementation reviews. First, to support the implementation of evidence-based interventions, methodologically rigorous evaluations of current services and programs are needed which use outcome measures with demonstrated reliability and validity to quantify the effect of changes in health service delivery. As well, further research is required to explore Indigenous people’s understandings, principles and knowledge of what is important in healthcare implementation; particularly in relation to the value of community control and equity issues. Second, contexts that support sustained health service or program implementation include the provision of adequate and enduring funding. The short-term nature of funding exacerbated problems in service delivery whilst modest additional resources produced change and enhanced outcomes, leading to further funding allocations. To improve the equity and effectiveness of service provision, support for the implementation of services or programs in areas of greatest need is warranted. Finally, implementation could be better supported through enabling Indigenous leadership, governance and involvement in implementation; tailoring services and programs; and active facilitation methods to fit the diverse contexts of Indigenous Australian health settings.
Promoting Action on Research Implementation in Health Services
McCalman J. The transfer of an Aboriginal Australian wellbeing program: a grounded theory study. Implement Sci. 2013a;8:129. doi:10.1186/1748-5908-8-129.
McCalman J, Tsey K, Clifford A, Earles W, Shakeshaft A, Bainbridge R. Applying what works: a systematic search of the transfer and implementation of promising Indigenous Australian health services and programs. BMC Public Health. 2012;12:600. doi:10.1186/1471-2458-12-600.
Ovretveit J. Widespread focused improvement: lessons from international health for spreading specific improvements to health services in high-income countries. Int J Qual Health Care. 2011;23(3):239–46. doi:10.1093/intqhc/mzr018.
Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide. Geneva: Alliance for Health Policy and Systems Research and World Health Organization; 2013. Retrieved from http://who.int/alliance-hpsr/alliancehpsr_irpguide.pdf.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi:10.1186/1748-5908-4-50.
Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou O. Diffusion of innovations in health service organisations. A systematic literature review. Malden: BMJ Books, Blackwell Publishing; 2005.
Rogers EM. Diffusion of innovations. New York: Free Press of Glencoe; 1962.
McCalman J. A grounded theory of program transfer: how an aboriginal empowerment initiative became “bigger than a program”. Cairns: (PhD), James Cook University; 2013b.
McCalman J, Tsey K, Bainbridge R, Rowley K, Percival N, O’Donoghue L, Judd J. The characteristics, implementation and effects of Aboriginal and Torres Strait Islander health promotion tools: a systematic literature search. BMC Public Health. 2014;14:712. doi:10.1186/1471-2458-14-712.
Percival N, McCalman J, Armit C, O’Donoghue L, Bainbridge R, Rowley K, Tsey K (in press). How are health promotion tools implemented? A theoretical model of implementation in Indigenous Australian primary health care. Health Promotion International.
Brands J. Implementation of innovations in Aboriginal and Torres Strait Islander health care. A review of the literature: interim report. Melbourne: Lowitja Institute; 2012.
Smith V, Devane D, Begley CM, Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011;11:15. doi:10.1186/1471-2288-11-15.
Stetler CB, Damschroder LJ, Helfrich CD, Hagedorn HJ. A guide for applying a revised version of the PARiHS framework for implementation. Implement Sci. 2011;6:99. doi:10.1186/1748-5908-6-99.
Kitson AL, Rycroft-Malone J, Harvey G, McCormack B, Seers K, Titchen A. Evaluating the successful implementation of evidence into practice using the PARIHS framework: theoretical and practical challenges. Implement Sci. 2008;3:1. doi:10.1186/1748-5908-3-1.
Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Qual Health Care. 1998;7:149–58.
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:53. doi:10.1186/s13012-015-0242-0.
Gardner K, Dowden M, Togni S, Bailie R. Understanding uptake of continuous quality improvement in Indigenous primary health care: lessons from a multi-site case study of the Audit and Best practice for Chronic Disease project. Implement Sci. 2010;5:21. doi:10.1186/1748-5908-5-21.
Bainbridge R, McCalman J, Clifford A, Tsey K. Cultural competency in the delivery of health services for Indigenous people. (Issues paper no. 13). Canberra: Closing the Gap Clearinghouse; 2015. Retrieved from http://www.aihw.gov.au/uploadedFiles/ClosingTheGap/Content/Our_publications/2015/ctgc-ip13.pdf.
Bainbridge R, Tsey K, McCalman J, Towle S. The quantity, nature, quality and characteristics of Aboriginal and Torres Strait Islander Australian mentoring literature: a systematic review. BMC Public Health. 2014;14:1263. doi:10.1186/1471-2458-14-1263.
Jackson N. Handbook: systematic reviews of health promotion and public health interventions. Melbourne: Cochrane Health Promotion and Public Health Field and the Victorian Health Promotion Foundation; 2007.
Jongen C, McCalman J, Bainbridge R, Tsey K. Aboriginal and Torres Strait Islander maternal and infant health and wellbeing: a systematic review of programs and services in Australian primary health care settings. BMC Pregnancy Childbirth. 2014;14:251. doi:10.1186/1471-2393-14-251.
Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(6):e1000097. doi:10.1371/journal.pmed.1000097.
Clifford A, Jackson Pulver L, Richmond R, Shakeshaft A, Ivers R. Disseminating best-evidence health care to Indigenous health-care settings and programs in Australia: identifying the gaps. Health Promot Int. 2009;24(4):404–14. doi:10.1093/heapro/dap039.
Sanson-Fisher R, Campbell E. Health research in Australia: its role in achieving the goals and targets. Health Promot J Austr. 1994;4(3):28–33.
Harris E, Little P. Applying a broader concept of scale to evaluate a funding strategy. Eval Exchange. 2010;XV(1):17.
Gibson O, Lisy K, Davy C, Aromataris E, Kite E, Lockwood C, Brown A. Enablers and barriers to the implementation of primary health care interventions for Indigenous people with chronic diseases: a systematic review. Implement Sci. 2015;10:71. doi:10.1186/s13012-015-0261-x.
Sanson-Fisher RW, Campbell EM, Perkins JJ, Blunden SV, Davis BB. Indigenous health research: a critical review of outputs over time. Australasian Medical Publishing Company. 2006;184(10):502–5.
Paul CL, Sanson-Fisher R, Stewart J, Anderson AE. Being sorry is not enough. The sorry state of the evidence base for improving the health of Indigenous populations. Am J Prev Med. 2010;38(5):566–8. doi:10.1016/j.amepre.2010.02.001.
Ridani R, Shand FL, Christensen H, McKay K, Tighe J, Burns J, Hunter E. Suicide prevention in Australian Aboriginal communities: a review of past and present programs. Suicide Life Threat Behav. 2015;45(1):111–40. doi:10.1111/sltb.12121.
Gray D, Wilson M, Allsop S, Saggers S, Wilkes E, Ober C. Barriers and enablers to the provision of alcohol treatment among Aboriginal Australians: a thematic review of five research projects. Drug Alcohol Rev. 2014;33(5):482–90. doi:10.1111/dar.12137.
Bainbridge R, Tsey K, McCalman J, Kinchin I, Saunders V, Watkin Lui F, Lawson K. No one’s discussing the elephant in the room: contemplating questions of research impact and benefit in aboriginal and Torres Strait Islander health research. BMC Public Health. 2015;15:696. doi:10.1186/s12889-015-2052-3.
Tursan d’Espaignet E, Measey ML, Carnegie MA, Mackerras D. Monitoring the ‘Strong Women, Strong Babies, Strong Culture Program’: the first eight years. J Paediatr Child Health. 2003;39(9):668–72. doi:10.1046/j.1440-1754.2003.00272.x.
Hunter E, Brown J, McCulloch B. Encouraging practitioners to use resources: evaluation of the national implementation of a resource to improve the clinical management of alcohol-related problems in Indigenous primary care settings. Drug Alcohol Rev. 2004;23:89–100. doi:10.1080/09595230410001645583.
Thomas DP, Bainbridge R, Tsey K. Changing discourses in Aboriginal and Torres Strait Islander health research 1914–2014. Med J Aust. 2014;201(1):S1–4. doi:10.5694/mja14.00114.
Ring IT, Brown N. Indigenous health: chronically inadequate responses to damning statistics. Med J Aust. 2002;177(11–12):629–31.
This paper has been produced as part of the activities of the Lowitja Institute, Australia’s national institute for Aboriginal and Torres Strait Islander health research, which incorporates the Cooperative Research Centre for Aboriginal and Torres Strait Islander Health. The Cooperative Research Centre program is an Australian Government initiative. Katrina Keith, Research Manager at The Cairns Institute, James Cook University completed the systematic database searches, formatted an early version of the manuscript and edited the final manuscript.
The authors declare that they have no competing interests.
JM led the review process and drafted the manuscript. JM and RB screened studies for eligibility. RB, NP and KT advised on the development of the review and checked the final manuscript. All authors read and approved the final manuscript.
RB is an Indigenous researcher; JM, NP and KT are non-Indigenous.
About this article
Cite this article
McCalman, J., Bainbridge, R., Percival, N. et al. The effectiveness of implementation in Indigenous Australian healthcare: an overview of literature reviews. Int J Equity Health 15, 47 (2016). https://0-doi-org.brum.beds.ac.uk/10.1186/s12939-016-0337-5
- Aboriginal and Torres Strait Islander
- Indigenous Australian
- Health services
- Health programs
- PARiHS framework
- Community control
- Health equity