This example is based on a study conducted to evaluate the implementation of a new electronic register for recording and reporting to support monitoring and evaluation efforts for the national antiretroviral therapy (ART) programme in South Africa. The electronic register was nationally adopted in 2010 and this evaluation focused on the implementation experiences in Western Cape Province, South Africa
To identify and contextualise enabling and constraining factors for implementation of the electronic register, to describe experiences and use of the register, and to make recommendations for its implementation in similar settings where standardisation of ART monitoring and evaluation has not been achieved.
Areas for further investigation: What proportion of TB patients are registered in the national electronic system? Are there differences between TB patients who are/not registered in the system (e.g. age, sex, regional distribution, public vs private sector TB patients, etc.)? What factors contribute to some patients being registered in the system but not others?
Areas for further investigation: What impact has the national electronic TB Register had on key programme performance indicators (e.g. case detection, treatment success, etc.) compared to the same indicators under a decentralized paper register system? Is the system cost-effective under programmatic conditions compared with previous tools?
Do NTP staff find the system meaningful for their work? Does it help inform them about what further interventions are needed in light of these data? Would the users recommend the system to others? What factors contribute to variation in effectiveness and cost-effectiveness across settings or sub-groups? What end-user and health system factors facilitated, or were barriers, to achieving the outcome of interest?
More than 3000 entry points into the national electronic system have been established across the public health and penitentiary systems, and nearly 11 000 individual users have been registered.
Areas for further investigation: What proportion of entry points have been established? How and why does usage change in public, private and other settings?
Areas for further investigation: What proportion of electronic records are completed as expected by the staff responsible for the data? Are there certain patient groups for which the data are more frequently missing (e.g. TB/HIV or multidrug-resistant TB (MDR-TB))? Is the electronic system being used as intended; if not, do users require additional training or other support strategies to ensure proper use? What are the costs/savings and infrastructural requirements associated with the national electronic system? What is the proportion of evaluated functionalities that are functioning correctly/needing adjustment/have bugs to be fixed at time of evaluation?
Areas for investigation: What proportion of settings and end-users are still using the electronic system as intended after the beginning of implementation? Are there any differences in key programme indicators across settings that do/do not continue to use the electronic system? What are the costs and infrastructure requirements associated with continued use of the register, including ongoing maintenance costs, such as licensing, upgrades and/or additional features/developer time? Are there any components of the electronic system that must be sustained, discontinued or modified after initial roll out, if so, why? Has acceptability of the TB register changed over time? What are different risk factors/externalities/threats for continued implementation?
This study explored various outcomes related to implementation. In-depth interviews aimed to explore perceptions and experiences related to implementation barriers and facilitators.
The population was the ART programme staff working at the facility, sub-district and district level in Cape Winelands district, Western Cape Province. Facilities in the sub-district were chosen based on their ART case load, decentralization status and historic quality of their ART paper-based register.
The Cape Winelands district is in Western Cape, South Africa and is a rural region with an estimated population of 750 000. The district includes five subdistricts and recorded around 12 000 patients receiving public ART services in the district as of 2012.
Intervention/ implementation strategy
The intervention was a new, pragmatic M&E system to monitor the national ART programme in South Africa developed by the Centre for Infectious Disease Epidemiology and Research (CIDER) at the University of Cape Town. The system adopted a three-tiered approach that included paper-based, stand-alone electronic, and networked electronic medical records systems. The second electronic tier acts as an electronic register and allowed for the simple digitization of the first-tier paper-based registers. The second tier was designed to bridge the paper-based registers, and the eventual networked electronic medical records system (tier 3). The three systems all capture the same essential ART M&E indicators required by the National Department of Health (DoH). The three-tiered system was official adopted by the National Department of Health in December 2010 to monitor the ART programme in all provinces of South Africa.
A 12-step implementation process was developed by CIDER and the Anova Health Institute that included a standardized master training course to ensure consistency of implementation across all districts. Steps included stakeholder meetings with relevant ART staff at the facility level to introduce the new system and requirements for roll out; site visits to identify available resources and planning requirements; facility evaluation to map out the recommended implementation; procurement of necessary equipment; training of relevant staff and establishment of the new M&E system; back-capturing of patient data captured on paper-based registers into the electronic system; data cleaning and validation. The remaining steps were considered part of the 'post implementation' phase and included activities such as follow-up training and support as facilities commence using the new M&E system within their routine practice.
Data collection, management and analysis
Sampling and recruitment
Purposive sampling was used to recruit participants from facilities across each of the five sub-districts. Participants from identified facilities were selected based on their involvement in the ART programme and experience with the implementation of the electronic register across the facility, sub-district and district level. In total, the study aimed to recruit 20 in-depth interview participants.
Data collection methods
A diverse sample of stakeholders was recruited at facility, sub-district, and district level via telephone or e-mail between January and June 2012. In-depth, semi-structured interviews were administered in the language of the participants' choosing by the research team using interview guides. Separate interview guides were used for the facility-based and sub-district staff respectively. The guides comprised open-ended questions with set probes, and Likert-scale questions to facilitate comparison between participants on specific questions relating to implementation. Questions focused on respondents' experiences of the implementation process and the perceived impact of the electronic register on the quality, access, and use of ART data as well as on service delivery.
Data management and analysis
Following consent, interviews were recorded and transcribed and translated into English if necessary. A team of three researchers coded the transcripts using a grounded theory approach and analysed thematically. The Likert-scaled answers provided an overview of responses per respondent type and facility, and were expected to highlight inconsistencies. Following coding, preliminary findings were shared with people who had been involved in the roll out and implementation process in order to validate, interpret and contextualize the findings.
Based on: Myburgh H, et al (2015). Implementation of an electronic monitoring and evaluation system for the antiretroviral treatment programme in the Cape Winelands District, South Africa: A qualitative evaluation (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0127223).