Preparing for implementation research
Setting the scene for implementation research
IR is the systematic approach to identifying and addressing barriers to effective and quality delivery of health interventions, strategies and policies. IR seeks to identify the implementation factors or challenges that prevent proven interventions from improving health outcomes and achieving implementation goals in certain settings. IR supports the design and testing of solutions to overcome identified barriers. Multiple working definitions exist for IR, with some examples favoured by certain disciplines, funders or regions. As Peters and colleagues (2013) state:1
"Implementation research can consider any aspect of implementation, including the factors affecting implementation, the processes of implementation, and the results of implementation, including how to introduce potential solutions into a health system or how to promote their large-scale use and sustainability. The intent is to understand what, why and how interventions work in 'real world' settings and to test approaches to improving them."
For example, IR studies may seek to:
- identify implementation problems that hinder access to interventions, delivery of services, as well as factors that undermine the usability of effective, evidence-based interventions;
- understand barriers and challenges in a given setting and determine the best approach to implementing a new intervention;
- develop and test practical solutions to implement a new recommendation or intervention that can effectively address known problems or health system bottlenecks, and;
- identify how to adapt evidence-based interventions, tools, and services to achieve sustained health impacts in real-world settings.
What might this look like when using digital technologies for tuberculosis (TB) care?
- You are implementing a new digital technology that has demonstrated positive results in other settings but is not yielding the same benefits in your setting, and you want to better understand why the intervention is not performing optimally;
- You are interested in trialling a new digital technology but need to identify the most appropriate strategies to implement it, understand how an existing tool could be best adapted to suit your specific context, and/or to identify what the potential barriers and facilitators to effective implementation might be;
- You want to understand whether a new digital intervention is working optimally, and/or whether the intervention is suitable for scale up (and how).
At this point, it is assumed that the IR process is being led by a small group of colleagues or partners including those who initiated the IR process and those with direct experience or interest in the implementation challenges related to the digital technology.
Engaging relevant stakeholders
Implementation research is a needs-driven process and research questions must reflect real-life challenges and needs related to the implementation of a particular intervention. As such, it is important that relevant stakeholders who are or will be connected to the digital intervention are identified early in the IR process in order to benefit from their insights, experiences and knowledge. This will enhance your team's understanding of the implementation challenges and their potential causes and help you to develop research questions that can adequately address these issues.
What is a stakeholder? Anyone whose involvement and perspective is crucial to the success of an activity, and who is interested in, or affected by the implementation challenge in question. Involving stakeholders in the identification and framing of research objectives, for example, ensures acceptable research tailored to end-users and increases ownership and uptake of findings. Fig. 2 illustrates some potential stakeholders in relation to digital technology and TB.
A stakeholder analysis may be a useful exercise to help properly identify potential stakeholders who should be consulted and involved in the development of your IR study (Box 1).
Understanding the implementation challenge
This section is about identifying and understanding the key challenges that constrain the implementation of your digital technology intervention, which will be further explored in your IR study. The series of steps that follow should be led by the core IR team in collaboration with the identified stakeholders.
Reviewing the intervention purpose and implementation goal
The first step in identifying current or potential implementation challenges is to clarify the overall goal of the intervention and its implementation: What is the purpose of the digital intervention? What are the implementation goals you are trying to achieve? Consider whether the digital technology is intended to add impact to an existing, proven intervention (e.g. digital adherence technologies to support TB adherence), or is it designed to independently trigger a health outcome (e.g. do connected diagnostic platforms improve the quality and timeliness of services provided by TB care providers?). It is important that all stakeholders and relevant parties are in agreement about the overall goals and have the same expectations of the study. With the goals clearly defined, it will be easier to identify the current or potential challenges. Here are some examples:
Electronic medication monitors in China (patient care)
Intervention goal: To use electronic medication monitors (EMM) to support self-administered medication and improve treatment adherence among TB patients in China.
Implementation goal: To implement EMM as the standard of care for all drug-susceptible TB patients in 20 health facilities across six Chinese counties.
Mobile cash transfers as enablers for TB patient support in India (patient care)
Intervention goal: To provide financial support to TB patients via a monthly direct transfer to a patient's bank account, which can be used to promote nutritional status during TB treatment and improve patient outcomes.
Implementation goal: To provide direct benefit transfers to 100% of newly diagnosed TB patients in India from 2018.
National electronic TB recording and reporting system in South Africa (surveillance and monitoring)
Intervention goal: To enable permanent, real-time monitoring, recording and reporting of the national TB control programme to strengthen TB surveillance in South Africa.
Implementation goal: To ensure 100% of TB services across South Africa are connected to the national reporting system by reporting via the electronic TB register.
Connected TB diagnostics platform in South Africa (programme management)
Intervention goal: To enable the integration of multiple test results from other health areas (including HIV and malaria) along with the suite of possible TB test results (GeneXpert, smear, culture and LPA) to support enhanced TB diagnostic capacity and treatment initiation in South Africa.
Implementation goal: To install and pilot the connected diagnostics platform in two primary health care clinics in Johannesburg, South Africa.
eLearning platform for HIV-associated TB in Brazil (eLearning)
Intervention goal: To provide a flexible approach for health care workers in Brazil to gain skills for high quality, person-centred care for people with TB/HIV.
Implementation goal: To reach approximately 100 000 new health care workers within the national public health system with the online training over a three-year implementation period.
Appraising current implementation within the local context
Once the team has clarified the overall goals of the intervention and the implementation, this should be compared to the current implementation status or process in order to identify potential challenges or gaps and their causes. For teams who are already implementing a digital intervention, you may already have an idea about where and why you are facing challenges with implementation, however it is important that this review is done in a systematic manner with the involvement of relevant stakeholders. For teams who are considering implementing a new digital technology intervention, this step will help you consider potential challenges you may face when it comes to implementing the new technology, and to identify the best strategy for implementation.
The RE-AIM framework
Adopting a systematic approach to thinking about implementation can be aided by the use of a theoretical framework. While there are many different frameworks available, in this toolkit we have chosen to use the RE-AIM framework2 – one of the most widely adopted frameworks – to evaluate potential implementation challenges to digital interventions.
RE-AIM stands for Reach, Effectiveness, Adoption, Implementation and Maintenance and was originally developed to evaluate the potential or actual impact of public health interventions. It does so by assessing traditional health outcomes (i.e., effectiveness) alongside important implementation outcomes that provide context for an intervention. Table 2 presents a definition of each domain and how it applies to digital interventions; read through the table and reflect on how this relates to the digital intervention being implemented or considered.
Assessing your current or planned implementation in light of the five RE-AIM domains can help pinpoint specific challenges that need to be addressed, and these challenges may become the basis of your IR study. The following case studies illustrate how the applying the RE-AIM framework can assist in the identification of potential implementation challenges and areas for further investigation.
When considering potential implementation challenges, it is imperative to consider how local and contextual factors may influence your ability to meet your implementation goals, and the impact this has on the overall effectiveness of your digital technology intervention. Such factors play an important role in how a digital technology intervention is rolled out, used and accessed, maintained, scaled up and integrated into policies and practices.
IR takes place in 'real-world' settings and attempts to support better understanding of contextual factors neglected by or unexamined in more traditional forms of clinical research. These settings are influenced by factors such as the physical, socioeconomic, geographical and cultural environments, and access to health systems. For digital health, these factors can include the maturity of the existing in-country information and communication technology (ICT) infrastructure, technological literacy, attitudes and policies related to the use of digital health interventions, workforce capacity, and the alignment or appropriateness of digital technology within the existing digital ecosystem.
The local context also includes varying understanding of the need for behaviour change and transition to new practices, which often accompanies the introduction of innovative tools and new approaches, such as changing from an entrenched paper-based records system to an electronic recording system. A digital intervention will succeed only if it is taken up by users, adds value to existing TB services, and facilitates the desired outcome. As such, implementers must be aware of stakeholder motivations and interests, as well as potential barriers and resistance to the disruption of the status quo that may affect the fidelity of deployment, understanding that this may temper the possible benefits of new digital health interventions.
Digital technology solutions are often regarded as complex interventions due to the various components required to create change. It is not enough to simply introduce a new digital intervention without any regard for the broader context in which you are working. Understanding how or why an intervention may be successfully implemented requires an understanding of the contextual factors that may shape, influence, or create potential implementation challenges.
Some potential factors that may shape or influence the implementation of a digital intervention are shown in Fig. 3.
Reviewing the existing literature
Once you have a better understanding of some of the implementation challenges and potential contributing factors as they relate to your specific context and digital intervention, the next step is to review the literature to assist in the development and articulation of a specific research question that can be addressed by an IR study. For example, the literature may provide evidence in support of a particular type of digital intervention, may further describe or explain the magnitude of a specific challenge, contextualise known implementation challenges, and/or identify solutions previously or currently implemented by others. The literature review should look at digital health interventions with evidence of feasibility and usability in their intended setting, as well as established efficacy under study conditions. Special consideration should be given to interventions deployed in settings that closely resemble those that you are interested in. The following are some possible resources for literature:
Peer-reviewed, published literature
- PubMed is a free, full-text online search engine for biomedical and life science literature. Guidance on how to perform a comprehensive literature search using PubMed can be found here.
- Web of Science and Scopus are both subscription-based databases for peer-reviewed literature in the social and health sciences.
- Google scholar is a search engine for scholarly literature across all disciplines. Google scholar can provide access to peer-reviewed and grey and/or unpublished literature and provides greater access to full text articles compared to other subscription-based databases.
'Grey literature' refers to information that has not been peer-reviewed. This includes:
- Programmatic and technical documents that provide current information on the state of the TB epidemic and response in your country, such as national strategic plans for TB control, government or WHO-developed estimates for TB prevalence and incidence, national TB programme (NTP) data on case detection, TB treatment outcomes and/or patient sociodemographic characteristics, or epidemiological and programmatic review reports. This data may also highlight differences or variation in terms of TB epidemiology, coverage of services, patient characteristics, etc. among different sub-populations or regions. Experiences from pilot studies may only be reported in internal documents and could hold valuable lessons that point to relevant problems.
- Routine NTP monitoring data can provide a good indication of the overall performance of an intervention or programme and be a useful way to identify implementation constraints, or areas of poor performance.
- Conference archives may also be a good source of information to learn about innovative uses of digital technology for TB, such as The Union Conference.
Building an IR team
By this stage, you should have:
- identified the key implementation challenges;
- reflected on the local context and identified the various contextual factors that may be influencing or contributing to the implementation challenges;
- identified and consulted with relevant stakeholders to better understand the implementation challenges, develop some hypotheses about potential causes and contributing factors and gather additional information, and;
- reviewed the existing literature to learn more about implementation challenge including selecting some strategies to address it or understand it better.
Turning this information into a sound research objective that can be investigated by an IR study requires a suitably skilled research team. Building a research team is an iterative process. While you may already have a small core research team assembled, your team may need to expand as your IR study develops and additional needs and capacities become clear. For example, depending on your research objective and chosen study design, you may require qualitative researchers, health economics, statisticians etc., on your team.
In general, research teams are made up of the following people:
- Principal investigator/lead researcher.
- Research coordinator/project manager.
- Data manager.
- Data analyst.
- Research assistants/data collectors.
Illustrative job descriptions for these key roles are available in the Annex. These can be adapted to suit your specific needs and can be used to help with recruitment of additional research team members if needed. Any new recruits will need to be accounted for in the budget, as discussed further in Module 5.
In addition to the research team, you may wish to involve some key stakeholders to provide oversight or represent the communities you will be working with (refer back to your stakeholder analysis). While the research team will be in the driving seat for the research, the other stakeholders can act as a steering committee and help provide guidance, help with decision-making and act as a conduit to other stakeholders relevant to your study.
Fig 4. provides some examples of individuals who could provide oversight to your research project. Perhaps you have already identified some of these individuals through the stakeholder analysis discussed earlier in this module?
Developing an IR proposal
The remainder of this toolkit is focused on turning your implementation challenge into a feasible research objective that will be developed into an IR study. By completing the following modules, you will be guided through the development of an IR proposal that can be used to support fundraising efforts by your team. Fig. 5 shows the various proposal components that will be addressed in this toolkit. We have developed a Generic proposal template that can be used by your team to develop the study proposal.
As prospective donors or funders may have specific requirements or timelines that will need to be addressed in the IR proposal, it is preferable that IR team members identify potential funding opportunities at the beginning of the IR process. Donors may also have specific requirements about the layout or contents of an IR proposal. In such cases, it is recommended that you follow the donor requirements instead of using the generic template included in this toolkit.
The team should also explore opportunities for domestic funding provided by government agencies or national NGOs that can be used to support the IR project. There are also various external funders who may provide support for research projects such as multilateral organizations, bilateral donors and NGOs, private foundations and trusts.
such as the World health Organization (WHO), the United Nations Development Programme (UNDP) or the United Nations Children's Fund (UNICEF) among others, or special programmes such as the Special Programme for Research and Training in Tropical Diseases (TDR) and the Alliance for Health Policy and Systems Research (The Alliance). In particular, The Global Fund to Fight AIDS, TB, and Malaria (The Global Fund) provides crucial support to many NTPs and are supportive of both digital technology and implementation research as key strategies to support TB control efforts. Research teams are strongly encouraged to consider integrating IR proposals within upcoming Global Fund funding request development processes.
This refers to government donors and their development agencies, such as the Department for International Development (DFID, United Kingdom), Japan International Cooperation Agency (JICA, Japan), the Swedish International Development Cooperation Agency (SIDA, Sweden), and the United States Agency for International Development (USAID). Like their multilateral counterparts, almost all bilateral organisations have programmes or partners in low- and middle-resource settings that reserve part of programme and/or project budget for M&E, as well as IR. Additionally, government research funding agencies are also increasingly supporting IR and could also be a potential funding source. Examples include the National Institutes of Health (NIH, United States), the Canadian Institutes of Health Research (CIHR, Canada) and the Kenya Medical Research Institute (KEMRI).