Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009-2012

Development of novel composite data quality scores to evaluate facility-level data quality in electronic data in Kenya: a nationwide retrospective cohort study

BACKGROUND

In this evaluation, we aim to strengthen Routine Health Information Systems (RHIS) through the digitization of data quality assessment (DQA) processes. We leverage electronic data from the Kenya Health Information System (KHIS) which is based on the District Health Information System version 2 (DHIS2) to perform DQAs at scale. We provide a systematic guide to developing composite data quality scores and use these scores to assess data quality in Kenya.

METHODS

We evaluated 187 HIV care facilities with electronic medical records across Kenya. Using quarterly, longitudinal KHIS data from January 2011 to June 2018 (total N = 30 quarters), we extracted indicators encompassing general HIV services including services to prevent mother-to-child transmission (PMTCT). We assessed the accuracy (the extent to which data were correct and free of error) of these data using three data-driven composite scores: 1) completeness score; 2) consistency score; and 3) discrepancy score. Completeness refers to the presence of the appropriate amount of data. Consistency refers to uniformity of data across multiple indicators. Discrepancy (measured on a Z-scale) refers to the degree of alignment (or lack thereof) of data with rules that defined the possible valid values for the data.

RESULTS

A total of 5,610 unique facility-quarters were extracted from KHIS. The mean completeness score was 61.1% [standard deviation (SD) = 27%]. The mean consistency score was 80% (SD = 16.4%). The mean discrepancy score was 0.07 (SD = 0.22). A strong and positive correlation was identified between the consistency score and discrepancy score (correlation coefficient = 0.77), whereas the correlation of either score with the completeness score was low with a correlation coefficient of -0.12 (with consistency score) and -0.36 (with discrepancy score). General HIV indicators were more complete, but less consistent, and less plausible than PMTCT indicators.

CONCLUSION

We observed a lack of correlation between the completeness score and the other two scores. As such, for a holistic DQA, completeness assessment should be paired with the measurement of either consistency or discrepancy to reflect distinct dimensions of data quality. Given the complexity of the discrepancy score, we recommend the simpler consistency score, since they were highly correlated. Routine use of composite scores on KHIS data could enhance efficiencies in DQA at scale as digitization of health information expands and could be applied to other health sectors beyondHIV clinics.

Evaluating the effectiveness of a mobile application to improve the quality, collection, and usability of forensic documentation of sexual violence

BACKGROUND

Survivors of sexual violence deserve timely and high-quality forensic examination, evidence collection, and documentation as part of comprehensive care. However, in many countries, the quality of medical-legal documentation is severely limited. MediCapt is an innovative digital application that enables clinicians to document forensic medical evidence as well as capture and securely store forensic photographs of injuries. This study evaluated the effectiveness and usability of MediCapt to document forensic medical evidence of sexual violence.

METHODS

This mixed-methods evaluation involved key-informant interviews, usability questionnaires, and forensic record reviews. Participants included clinicians, medical records personnel, information technology personnel, and health facility administrators, as well as law enforcement and legal professionals in Kenya.

RESULTS

The Physicians for Human Rights (PHR) data quality checklist found that using MediCapt led to significantly higher data-quality scores compared to paper-based forms. MediCapt forms scored higher on 23 of 26 checklist items. While a wide difference in quality was seen among paper-based forms, MediCapt appeared to both standardize and improve quality of documentation across sites. MediCapt strengths included data security and confidentiality, accuracy and efficiency, and supplemental documentation with photography. Weaknesses included infrastructure challenges, required technological proficiencies, and time to learn the new system. Although it is early to assess the impact of MediCapt on prosecutions, providers and law and justice sector professionals were optimistic about its usefulness. They identified MediCapt as appropriate for use with survivors of sexual violence and reported MediCapt's legibility and photography features had already been commended by the court.

CONCLUSION

MediCapt was well-received across all sectors, its use was perceived as feasible and sustainable, and it significantly improved the quality of collected forensic data. It is anticipated this improvement in forensic documentation will increase successful prosecutions, strengthen accountability for perpetrators, and improve justice for survivors.

Development and Testing Requirements for an Integrated Maternal and Child Health Information System in Iran: A Design Thinking Case Study

BACKGROUND

 Management of child health care can be negatively affected by incomplete recording, low data quality, and lack of data integration of health management information systems to support decision making and public health program needs. Given the importance of identifying key determinants of child health via capturing and integrating accurate and high-quality information, we aim to address this gap through the development and testing requirements for an integrated child health information system.

SUBJECTS AND METHODS

 A five-phase design thinking approach including empathizing, defining, ideation, prototyping, and testing was applied. We employed observations and interviews with the health workers at the primary health care network to identify end-users' challenges and needs using tools in human-centered design and focus group discussion. Then, a potential solution to the identified problems was developed as an integrated maternal and child health information system (IMCHIS) prototype and tested using Software Quality Requirements and Evaluation Model (SQuaRE) ISO/IEC 25000.

RESULTS

 IMCHIS was developed as a web-based system with 74 data elements and seven maternal and child health care requirements. The requirements of "child disease" with weight (0.26), "child nutrition" with weight (0.20), and "prenatal care" with weight (0.16) acquired the maximum weight coefficient. In the testing phase, the highest score with the weight coefficient of 0.48 and 0.73 was attributed to efficiency and functionality characteristics, focusing on software capability to fulfill the tasks that meet users' needs.

CONCLUSION

 Implementing a successful child health care system integrates both maternal and child health care information systems to track the effect of maternal conditions on child health and support managing performance and optimizing service delivery. The highest quality score of IMCHIS in efficiency and functionality characteristics confirms that it owns the capability to identify key determinants of child health.

Can routine health facility data be used to monitor subnational coverage of maternal, newborn and child health services in Uganda?

Background

Routine health facility data are a critical source of local monitoring of progress and performance at the subnational level. Uganda has been using district health statistics from facility data for many years. We aimed to systematically assess data quality and examine different methods to obtain plausible subnational estimates of coverage for maternal, newborn and child health interventions.

Methods

Annual data from the Uganda routine health facility information system 2015–2019 for all 135 districts were used, as well as national surveys for external comparison and the identification of near-universal coverage interventions. The quality of reported data on antenatal and delivery care and child immunization was assessed through completeness of facility reporting, presence of extreme outliers and internal data consistencies. Adjustments were made when necessary. The denominators for the coverage indicators were derived from population projections and health facility data on near-universal coverage interventions. The coverage results with different denominators were compared with the results from household surveys.

Results

Uganda’s completeness of reporting by facilities was near 100% and extreme outliers were rare. Inconsistencies in reported events, measured by annual fluctuations and between intervention consistency, were common and more among the 135 districts than the 15 subregions. The reported numbers of vaccinations were improbably high compared to the projected population of births or first antenatal visits – and especially so in 2015–2016. There were also inconsistencies between the population projections and the expected target population based on reported numbers of antenatal visits or immunizations. An alternative approach with denominators derived from facility data gave results that were more plausible and more consistent with survey results than based on population projections, although inconsistent results remained for substantive number of subregions and districts.

Conclusion

Our systematic assessment of the quality of routine reports of key events and denominators shows that computation of district health statistics is possible with transparent adjustments and methods, providing a general idea of levels and trends for most districts and subregions, but that improvements in data quality are essential to obtain more accurate monitoring.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06554-6.

A realist systematic review of evidence from low- and middle-income countries of interventions to improve immunization data use

BACKGROUND

The use of routine immunization data by health care professionals in low- and middle-income countries remains an underutilized resource in decision-making. Despite the significant resources invested in developing national health information systems, systematic reviews of the effectiveness of data use interventions are lacking. Applying a realist review methodology, this study synthesized evidence of effective interventions for improving data use in decision-making.

METHODS

We searched PubMed, POPLINE, Centre for Agriculture and Biosciences International Global Health, and African Journals Online for published literature. Grey literature was obtained from conference, implementer, and technical agency websites and requested from implementing organizations. Articles were included if they reported on an intervention designed to improve routine data use or reported outcomes related to data use, and targeted health care professionals as the principal data users. We developed a theory of change a priori for how we expect data use interventions to influence data use. Evidence was then synthesized according to data use intervention type and level of the health system targeted by the intervention.

RESULTS

The searches yielded 549 articles, of which 102 met our inclusion criteria, including 49 from peer-reviewed journals and 53 from grey literature. A total of 66 articles reported on immunization data use interventions and 36 articles reported on data use interventions for other health sectors. We categorized 68 articles as research evidence and 34 articles as promising strategies. We identified ten primary intervention categories, including electronic immunization registries, which were the most reported intervention type (n = 14). Among the research evidence from the immunization sector, 32 articles reported intermediate outcomes related to data quality and availability, data analysis, synthesis, interpretation, and review. Seventeen articles reported data-informed decision-making as an intervention outcome, which could be explained by the lack of consensus around how to define and measure data use.

CONCLUSIONS

Few immunization data use interventions have been rigorously studied or evaluated. The review highlights gaps in the evidence base, which future research and better measures for assessing data use should attempt to address.

Interventions to improve district-level routine health data in low-income and middle-income countries: a systematic review

BACKGROUND

Routine health information system(s) (RHIS) facilitate the collection of health data at all levels of the health system allowing estimates of disease prevalence, treatment and preventive intervention coverage, and risk factors to guide disease control strategies. This core health system pillar remains underdeveloped in many low-income and middle-income countries. Efforts to improve RHIS data coverage, quality and timeliness were launched over 10 years ago.

METHODS

A systematic review was performed across 12 databases and literature search engines for both peer-reviewed articles and grey literature reports on RHIS interventions. Studies were analysed in three stages: (1) categorisation of RHIS intervention components and processes; (2) comparison of intervention component effectiveness and (3) whether the post-intervention outcome improved above the WHO integrated disease surveillance response framework data quality standard of 80% or above.

RESULTS

5294 references were screened, resulting in 56 studies. Three key performance determinants-technical, organisational and behavioural-were proposed as critical to RHIS strengthening. Seventy-seven per cent [77%] of studies identified addressed all three determinants. The most frequently implemented intervention components were 'providing training' and 'using an electronic health management information systems'. Ninety-three per cent [93%] of pre-post or controlled trial studies showed improvements in one or more data quality outputs, but after applying a standard threshold of >80% post-intervention, this number reduced to 68%. There was an observed benefit of multi-component interventions that either conducted data quality training or that addressed improvement across multiple processes and determinants of RHIS.

CONCLUSION

Holistic data quality interventions that address multiple determinants should be continuously practised for strengthening RHIS. Studies with clearly defined and pragmatic outcomes are required for future RHIS improvement interventions. These should be accompanied by qualitative studies and cost analyses to understand which investments are needed to sustain high-quality RHIS in low-income and middle-income countries.

Data quality of the routine health management information system at the primary healthcare facility and district levels in Tanzania

BACKGROUND

Effective planning for disease prevention and control requires accurate, adequately-analysed, interpreted and communicated data. In recent years, efforts have been put in strengthening health management information systems (HMIS) in Sub-Saharan Africa to improve data accessibility to decision-makers. This study assessed the quality of routine HMIS data at primary healthcare facility (HF) and district levels in Tanzania.

METHODS

This cross-sectional study involved reviews of documents, information systems and databases, and collection of primary data from facility-level registers, tally sheets and monthly summary reports. Thirty-four indicators from Outpatient, Inpatient, Antenatal care, Family Planning, Post-natal care, Labour and Delivery, and Provider-Initiated Testing and Counselling service areas were assessed. Indicator records were tracked and compared across the process of data collection, compilation and submission to the district office. Copies of monthly report forms submitted by facilities to the district were also reviewed. The availability and utilization of HMIS tools were assessed, while completeness and data accuracy levels were quantified for each phase of the reporting system.

RESULTS

A total of 115 HFs (including hospitals, health centres, dispensaries) in 11 districts were involved. Registers (availability rate = 91.1%; interquartile range (IQR) 66.7-100%) and report forms (86.9%; IQR 62.2-100%) were the most utilized tools. There was a limited use of tally-sheets (77.8%; IQR 35.6-100%). Tools availability at the dispensary was 91.1%, health centre 82.2% and hospital 77.8%, and was low in urban districts. The availability rate at the district level was 65% (IQR 48-75%). Wrongly filled or empty cells in registers and poor adherence to the coding procedures were observed. Reports were highly over-represented in comparison to registers' records, with large differences observed at the HF phase of the reporting system. The OPD and IPD areas indicated the highest levels of mismatch between data source and district office. Indicators with large number of clients, multiple variables, disease categorization, or those linked with dispensing medicine performed poorly.

CONCLUSION

There are high variations in the tool utilisation and data accuracy at facility and district levels. The routine HMIS is weak and data at district level inaccurately reflects what is available at the source. These results highlight the need to design tailored and inter-service strategies for improving data quality.

Spatial-temporal patterns of malaria incidence in Uganda using HMIS data from 2015 to 2019

BACKGROUND

As global progress to reduce malaria transmission continues, it is increasingly important to track changes in malaria incidence rather than prevalence. Risk estimates for Africa have largely underutilized available health management information systems (HMIS) data to monitor trends. This study uses national HMIS data, together with environmental and geographical data, to assess spatial-temporal patterns of malaria incidence at facility catchment level in Uganda, over a recent 5-year period.

METHODS

Data reported by 3446 health facilities in Uganda, between July 2015 and September 2019, was analysed. To assess the geographic accessibility of the health facilities network, AccessMod was employed to determine a three-hour cost-distance catchment around each facility. Using confirmed malaria cases and total catchment population by facility, an ecological Bayesian conditional autoregressive spatial-temporal Poisson model was fitted to generate monthly posterior incidence rate estimates, adjusted for caregiver education, rainfall, land surface temperature, night-time light (an indicator of urbanicity), and vegetation index.

RESULTS

An estimated 38.8 million (95% Credible Interval [CI]: 37.9-40.9) confirmed cases of malaria occurred over the period, with a national mean monthly incidence rate of 20.4 (95% CI: 19.9-21.5) cases per 1000, ranging from 8.9 (95% CI: 8.7-9.4) to 36.6 (95% CI: 35.7-38.5) across the study period. Strong seasonality was observed, with June-July experiencing highest peaks and February-March the lowest peaks. There was also considerable geographic heterogeneity in incidence, with health facility catchment relative risk during peak transmission months ranging from 0 to 50.5 (95% CI: 49.0-50.8) times higher than national average. Both districts and health facility catchments showed significant positive spatial autocorrelation; health facility catchments had global Moran's I = 0.3 (p < 0.001) and districts Moran's I = 0.4 (p < 0.001). Notably, significant clusters of high-risk health facility catchments were concentrated in Acholi, West Nile, Karamoja, and East Central - Busoga regions.

CONCLUSION

Findings showed clear countrywide spatial-temporal patterns with clustering of malaria risk across districts and health facility catchments within high risk regions, which can facilitate targeting of interventions to those areas at highest risk. Moreover, despite high and perennial transmission, seasonality for malaria incidence highlights the potential for optimal and timely implementation of targeted interventions.

Improving the quality of routine maternal and newborn data captured in primary health facilities in Gombe State, Northeastern Nigeria: a before-and-after study

OBJECTIVES

Primary objective: to assess nine data quality metrics for 14 maternal and newborn health data elements, following implementation of an integrated, district-focused data quality intervention.

SECONDARY OBJECTIVE

to consider whether assessing the data quality metrics beyond completeness and accuracy of facility reporting offered new insight into reviewing routine data quality.

DESIGN

Before-and-after study design.

SETTING

Primary health facilities in Gombe State, Northeastern Nigeria.

PARTICIPANTS

Monitoring and evaluation officers and maternal, newborn and child health coordinators for state-level and all 11 local government areas (district-equivalent) overseeing 492 primary care facilities offering maternal and newborn care services.

INTERVENTION

Between April 2017 and December 2018, we implemented an integrated data quality intervention which included: introduction of job aids and regular self-assessment of data quality, peer-review and feedback, learning workshops, work planning for improvement, and ongoing support through social media.

OUTCOME MEASURES

9 metrics for the data quality dimensions of completeness and timeliness, internal consistency of reported data, and external consistency.

RESULTS

The data quality intervention was associated with improvements in seven of nine data quality metrics assessed including availability and timeliness of reporting, completeness of data elements, accuracy of facility reporting, consistency between related data elements, and frequency of outliers reported. Improvement differed by data element type, with content of care and commodity-related data improving more than contact-related data. Increases in the consistency between related data elements demonstrated improved internal consistency within and across facility documentation.

CONCLUSIONS

An integrated district-focused data quality intervention-including regular self-assessment of data quality, peer-review and feedback, learning workshops, work planning for improvement, and ongoing support through social media-can increase the completeness, accuracy and internal consistency of facility-based routine data.

Improving quality and use of routine health information system data in low- and middle-income countries: A scoping review

BACKGROUND

A routine health information system is one of the essential components of a health system. Interventions to improve routine health information system data quality and use for decision-making in low- and middle-income countries differ in design, methods, and scope. There have been limited efforts to synthesise the knowledge across the currently available intervention studies. Thus, this scoping review synthesised published results from interventions that aimed at improving data quality and use in routine health information systems in low- and middle-income countries.

METHOD

We included articles on intervention studies that aimed to improve data quality and use within routine health information systems in low- and middle-income countries, published in English from January 2008 to February 2020. We searched the literature in the databases Medline/PubMed, Web of Science, Embase, and Global Health. After a meticulous screening, we identified 20 articles on data quality and 16 on data use. We prepared and presented the results as a narrative.

RESULTS

Most of the studies were from Sub-Saharan Africa and designed as case studies. Interventions enhancing the quality of data targeted health facilities and staff within districts, and district health managers for improved data use. Combinations of technology enhancement along with capacity building activities, and data quality assessment and feedback system were found useful in improving data quality. Interventions facilitating data availability combined with technology enhancement increased the use of data for planning.

CONCLUSION

The studies in this scoping review showed that a combination of interventions, addressing both behavioural and technical factors, improved data quality and use. Interventions addressing organisational factors were non-existent, but these factors were reported to pose challenges to the implementation and performance of reported interventions.

Using routine health information data for research in low- and middle-income countries: a systematic review

BACKGROUND

Routine health information systems (RHISs) support resource allocation and management decisions at all levels of the health system, as well as strategy development and policy-making in many low- and middle-income countries (LMICs). Although RHIS data represent a rich source of information, such data are currently underused for research purposes, largely due to concerns over data quality. Given that substantial investments have been made in strengthening RHISs in LMICs in recent years, and that there is a growing demand for more real-time data from researchers, this systematic review builds upon the existing literature to summarize the extent to which RHIS data have been used in peer-reviewed research publications.

METHODS

Using terms 'routine health information system', 'health information system', or 'health management information system' and a list of LMICs, four electronic peer-review literature databases were searched from inception to February 202,019: PubMed, Scopus, EMBASE, and EconLit. Articles were assessed for inclusion based on pre-determined eligibility criteria and study characteristics were extracted from included articles using a piloted data extraction form.

RESULTS

We identified 132 studies that met our inclusion criteria, originating in 37 different countries. Overall, the majority of the studies identified were from Sub-Saharan Africa and were published within the last 5 years. Malaria and maternal health were the most commonly studied health conditions, although a number of other health conditions and health services were also explored.

CONCLUSIONS

Our study identified an increasing use of RHIS data for research purposes, with many studies applying rigorous study designs and analytic methods to advance program evaluation, monitoring and assessing services, and epidemiological studies in LMICs. RHIS data represent an underused source of data and should be made more available and further embraced by the research community in LMIC health systems.

A, Salim N, Zaman SB, Shamba D, Blencowe H, Ruysen H, El Arifeen S, Boggs D, Gordeev VS, Rahman QSU, Hossain T, Joshi E, Thapa S, Poudel RP, Poudel D, Chaudhary P, Karki R, Chitrakar B, Mkopi N, Wisiko A, Kitende AP, Shirati MR, Chingalo C, Semhando AO, Mtei C, Mwenisongole V, Bakuza JM, Kombo J, Mbaruku G, Lawn JE. Labour and delivery ward register data availability, quality, and utility - Every Newborn - birth indicators research tracking in hospitals (EN-BIRTH) study baseline analysis in three countries

BACKGROUND

Countries with the highest burden of maternal and newborn deaths and stillbirths often have little information on these deaths. Since over 81% of births worldwide now occur in facilities, using routine facility data could reduce this data gap. We assessed the availability, quality, and utility of routine labour and delivery ward register data in five hospitals in Bangladesh, Nepal, and Tanzania. This paper forms the baseline register assessment for the Every Newborn-Birth Indicators Research Tracking in Hospitals (EN-BIRTH) study.

METHODS

We extracted 21 data elements from routine hospital labour ward registers, useful to calculate selected maternal and newborn health (MNH) indicators. The study sites were five public hospitals during a one-year period (2016-17). We measured 1) availability: completeness of data elements by register design, 2) data quality: implausibility, internal consistency, and heaping of birthweight and explored 3) utility by calculating selected MNH indicators using the available data.

RESULTS

Data were extracted for 20,075 births. Register design was different between the five hospitals with 10-17 of the 21 selected MNH data elements available. More data were available for health outcomes than interventions. Nearly all available data elements were > 95% complete in four of the five hospitals and implausible values were rare. Data elements captured in specific columns were 85.2% highly complete compared to 25.0% captured in non-specific columns. Birthweight data were less complete for stillbirths than live births at two hospitals, and significant heaping was found in all sites, especially at 2500g and 3000g. All five hospitals recorded count data required to calculate impact indicators including; stillbirth rate, low birthweight rate, Caesarean section rate, and mortality rates.

CONCLUSIONS

Data needed to calculate MNH indicators are mostly available and highly complete in EN-BIRTH study hospital routine labour ward registers in Bangladesh, Nepal and Tanzania. Register designs need to include interventions for coverage measurement. There is potential to improve data quality if Health Management Information Systems utilization with feedback loops can be strengthened. Routine health facility data could contribute to reduce the coverage and impact data gap around the time of birth.

Institutionalized data quality assessments: a critical pathway to improving the accuracy of integrated disease surveillance data in Sierra Leone

Background

Public health agencies require valid, timely and complete health information for early detection of outbreaks. Towards the end of the Ebola Virus Disease (EVD) outbreak in 2015, the Ministry of Health and Sanitation (MoHS), Sierra Leone revitalized the Integrated Disease Surveillance and Response System (IDSR). Data quality assessments were conducted to monitor accuracy of IDSR data.

Methods

Starting 2016, data quality assessments (DQA) were conducted in randomly selected health facilities. Structured electronic checklist was used to interview district health management teams (DHMT) and health facility staff. We used malaria data, to assess data accuracy, as malaria was endemic in Sierra Leone. Verification factors (VF) calculated as the ratio of confirmed malaria cases recorded in health facility registers to the number of malaria cases in the national health information database, were used to assess data accuracy. Allowing a 5% margin of error, VF < 95% were considered over reporting while VF > 105 was underreporting. Differences in the proportion of accurate reports at baseline and subsequent assessments were compared using Z-test for two proportions.

Results

Between 2016 and 2018, four DQA were conducted in 444 health facilities where 1729 IDSR reports were reviewed. Registers and IDSR technical guidelines were available in health facilities and health care workers were conversant with reporting requirements. Overall data accuracy improved from over- reporting of 4.7% (VF 95.3%) in 2016 to under-reporting of 0.2% (VF 100.2%) in 2018. Compared to 2016, proportion of accurate IDSR reports increased by 14.8% (95% CI 7.2, 22.3%) in May 2017 and 19.5% (95% CI 12.5–26.5%) by 2018. Over reporting was more common in private clinics and not- for profit facilities while under-reporting was more common in lower level government health facilities. Leading reasons for data discrepancies included counting errors in 358 (80.6%) health facilities and missing source documents in 47 (10.6%) health facilities.

Conclusion

This is the first attempt to institutionalize routine monitoring of IDSR data quality in Sierra Leone. Regular data quality assessments may have contributed to improved data accuracy over time. Data compilation errors accounted for most discrepancies and should be minimized to improve accuracy of IDSR data.

Understanding the challenges associated with the use of data from routine health information systems in low- and middle-income countries: A systematic review

BACKGROUND

Routine health information systems (RHISs) are crucial to informing decision-making at all levels of the health system. However, the use of RHIS data in low- and middle-income countries (LMICs) is limited due to concerns regarding quality, accuracy, timeliness, completeness and representativeness.

OBJECTIVE

This study systematically reviewed technical, behavioural and organisational/environmental challenges that hinder the use of RHIS data in LMICs and strategies implemented to overcome these challenges.

METHOD

Four electronic databases were searched for studies describing challenges associated with the use of RHIS data and/or strategies implemented to circumvent these challenges in LMICs. Identified articles were screened against inclusion and exclusion criteria by two independent reviewers.

RESULTS

Sixty studies met the inclusion criteria and were included in this review, 55 of which described challenges in using RHIS data and 20 of which focused on strategies to address these challenges. Identified challenges and strategies were organised by their technical, behavioural and organisational/environmental determinants and by the core steps of the data process. Organisational/environmental challenges were the most commonly reported barriers to data use, while technical challenges were the most commonly addressed with strategies.

CONCLUSION

Despite the known benefits of RHIS data for health system strengthening, numerous challenges continue to impede their use in practice.

IMPLICATIONS

Additional research is needed to identify effective strategies for addressing the determinants of RHIS use, particularly given the disconnect identified between the type of challenge most commonly described in the literature and the type of challenge most commonly targeted for interventions.

Cascade Analysis: An Adaptable Implementation Strategy Across HIV and Non-HIV Delivery Platforms

BACKGROUND

Cascades have been used to characterize sequential steps within a complex health system and are used in diverse disease areas and across prevention, testing, and treatment. Routine data have great potential to inform prioritization within a system, but are often inaccessible to frontline health care workers (HCWs) who may have the greatest opportunity to innovate health system improvement.

METHODS

The cascade analysis tool (CAT) is an Excel-based, simple simulation model with an optimization function. It identifies the step within a cascade that could most improve the system. The original CAT was developed for HIV treatment and the prevention of mother-to-child transmission of HIV.

RESULTS

CAT has been adapted 7 times: to a mobile application for prevention of mother-to-child transmission; for hypertension screening and management and for mental health outpatient services in Mozambique; for pediatric and adolescent HIV testing and treatment, HIV testing in family planning, and cervical cancer screening and treatment in Kenya; and for naloxone distribution and opioid overdose reversal in the United States. The main domains of adaptation have been technical-estimating denominators and structuring steps to be binary sequential steps-as well as logistical-identifying acceptable approaches for data abstraction and aggregation, and not overburdening HCW.

DISCUSSION

CAT allows for prompt feedback to HCWs, increases HCW autonomy, and allows managers to allocate resources and time in an equitable manner. CAT is an effective, feasible, and acceptable implementation strategy to prioritize areas most requiring improvement within complex health systems, although adaptations are being currently evaluated.

Factors limiting data quality in the expanded programme on immunization in low and middle-income countries: A scoping review

Few public health interventions can match the immense achievements of immunization in terms of mortality and morbidity reduction. However, progress in reaching global coverage goals and achieving universal immunization coverage have stalled; with key stakeholders concerned about the accuracy of reported coverage figures. Incomplete and incorrect data has made it challenging to obtain an accurate overview of immunization coverage, particularly in low- and middle-income countries (LMIC). To date, only one literature review concerning immunization data quality exists. However, it only included articles from Gavi-eligible countries, did not go deep into the characteristics of the data quality problems, and used a narrow 'data quality' definition. This scoping review builds upon that work; exploring the "state of data quality" in LMIC, factors affecting data quality in these settings and potential means to improve it. Only a small volume of literature addressing immunization data quality in LMIC was found and definitions of 'data quality' varied widely. Data quality was, on the whole, considered poor in the articles included. Coverage numerators were seen to be inflated for official reports and denominators were inaccurate and infrequently adjusted. Numerous factors related to these deficiencies were reported, including health information system fragmentation, overreliance on targets and poor data management processes. Factors associated with health workers were noted most frequently. Authors suggested that data quality could be improved by ensuring proper data collection tools, increasing workers' capacities and motivation through training and supervision, whilst also ensuring adequate and timely feedback on the data collected. The findings of this scoping review can serve as the basis to identify and address barriers to good quality immunization data in LMICs. Overcoming said barriers is essential if immunization's historic successes are to continue.

Monitoring Pharmacy and Test Kit Stocks in Rural Mozambique: U.S. President's Emergency Plan for AIDS Relief Surveillance to Help Prevent Ministry of Health Shortages

Support of human immunodeficiency virus (HIV) and tuberculosis (TB) testing and treatment supported by President's Emergency Plan for AIDS Relief (PEPFAR) in Africa requires immense quantities of tests and medications. We sought to use central pharmacy supply data of Mozambique's rural Zambézia Province (2017 population ≈5.11 million persons; ≈12.6% adult HIV prevalence in 2016) to examine shortages, stockouts, and trends in availability. Using stock surveillance for 60 weeks in 2014-2015, we assessed availability of 36 medications [4 classes: adult antiretroviral (ARV) medications, pediatric ARVs, anti-TB medications, and antibiotics] and diagnostic test kits (2 rapid tests for HIV; 1 each for malaria and syphilis). We contrasted these to 2018-2019 data. We modeled pharmacy data using ordinal logistic regression, characterizing weekly product availability in four categories: good, adequate, shortage, or complete stockout. We found 166 (7_.7%) stockouts and 150 (6.9%) shortages among 2,160 weekly records. Earlier calendar time was associated with reduced medication supplies (p < .001). Certain medication/test kit classes were associated with reduced supply (p < .001). We found an interaction between time and medication class on the odds of reduced supply (p < .001). Pediatric ARVs had a 17.4 (95% confidence interval: 8.8-34.4) times higher odds of reduced medication supply compared with adult ARVs at study midpoint. Trends comparing the first and last weeks showed adult ARVs having 67% and pediatric having 71% lower odds of reduced supplies. Only adult ARV shortages improved amid growing demand. Data from 2018 to 2019 suggest continuing inventory management challenges. Monitoring of drug (especially pediatric) and test kit shortages is vital to ensure quality improvement to guarantee adequate supplies to enable patients and care providers to achieve sustained viral suppression. A central Mozambican drug repository in the nation's second largest Province continues to experience drug and rapid test kit stockouts.

Making Smarter Decisions Faster: Systems Engineering to Improve the Global Public Health Response to HIV

PURPOSE OF REVIEW

This review offers an operational definition of systems engineering (SE) as applied to public health, reviews applications of SE in the field of HIV, and identifies opportunities and challenges of broader application of SE in global health.

RECENT FINDINGS

SE involves the deliberate sequencing of three steps: diagnosing a problem, evaluating options using modeling or optimization, and providing actionable recommendations. SE includes diverse tools (from process improvement to mathematical modeling) applied to decisions at various levels (from local staffing decisions to planning national-level roll-out of new interventions). Contextual factors are crucial to effective decision-making, but there are gaps in understanding global decision-making processes. Integrating SE into pre-service training and translating SE tools to be more accessible could increase utilization of SE approaches in global health. SE is a promising, but under-recognized approach to improve public health response to HIV globally.

Practical Implications of a Relationship between Health Management Information System and Community Cohort-Based Malaria Incidence Rates

Global malaria burden is reducing with effective control interventions, and surveillance is vital to maintain progress. Health management information system (HMIS) data provide a powerful surveillance tool; however, its estimates of burden need to be better understood for effectiveness. We aimed to investigate the relationship between HMIS and cohort incidence rates and identify sources of bias in HMIS-based incidence. Malaria incidence was estimated using HMIS data from 15 health facilities in three subcounties in Uganda. This was compared with a gold standard of representative cohort studies conducted in children aged 0.5 to < 11 years, followed concurrently in these sites. Between October 2011 and September 2014, 153,079 children were captured through HMISs and 995 followed up through enhanced community cohorts in Walukuba, Kihihi, and Nagongera subcounties. Although HMISs substantially underestimated malaria incidence in all sites compared with data from the cohort studies, there was a strong linear relationship between these rates in the lower transmission settings (Walukuba and Kihihi), but not the lowest HMIS performance highest transmission site (Nagongera), with calendar year as a significant modifier. Although health facility accessibility, availability, and recording completeness were associated with HMIS incidence, they were not significantly associated with bias in estimates from any site. Health management information systems still require improvements; however, their strong predictive power of unbiased malaria burden when improved highlights the important role they could play as a cost-effective tool for monitoring trends and estimating impact of control interventions. This has important implications for malaria control in low-resource, high-burden countries.

Generating statistics from health facility data: the state of routine health information systems in Eastern and Southern Africa

Health facility data are a critical source of local and continuous health statistics. Countries have introduced web-based information systems that facilitate data management, analysis, use and visualisation of health facility data. Working with teams of Ministry of Health and country public health institutions analysts from 14 countries in Eastern and Southern Africa, we explored data quality using national-level and subnational-level (mostly district) data for the period 2013-2017. The focus was on endline analysis where reported health facility and other data are compiled, assessed and adjusted for data quality, primarily to inform planning and assessments of progress and performance. The analyses showed that although completeness of reporting was generally high, there were persistent data quality issues that were common across the 14 countries, especially at the subnational level. These included the presence of extreme outliers, lack of consistency of the reported data over time and between indicators (such as vaccination and antenatal care), and challenges related to projected target populations, which are used as denominators in the computation of coverage statistics. Continuous efforts to improve recording and reporting of events by health facilities, systematic examination and reporting of data quality issues, feedback and communication mechanisms between programme managers, care providers and data officers, and transparent corrections and adjustments will be critical to improve the quality of health statistics generated from health facility data.

Quality of routine facility data for monitoring priority maternal and newborn indicators in DHIS2: A case study from Gombe State, Nigeria

INTRODUCTION

Routine health information systems are critical for monitoring service delivery. District Heath Information System, version 2 (DHIS2) is an open source software platform used in more than 60 countries, on which global initiatives increasingly rely for such monitoring. We used facility-reported data in DHIS2 for Gombe State, north-eastern Nigeria, to present a case study of data quality to monitor priority maternal and neonatal health indicators.

METHODS

For all health facilities in DHIS2 offering antenatal and postnatal care services (n = 497) and labor and delivery services (n = 486), we assessed the quality of data for July 2016-June 2017 according to the World Health Organization data quality review guidance. Using data from DHIS2 as well as external facility-level and population-level household surveys, we reviewed three data quality dimensions-completeness and timeliness, internal consistency, and external consistency-and considered the opportunities for improvement.

RESULTS

Of 14 priority maternal and neonatal health indicators that could be tracked through facility-based data, 12 were included in Gombe's DHIS2. During July 2016-June 2017, facility-reported data in DHIS2 were incomplete at least 40% of the time, under-reported 10%-60% of the events documented in facility registers, and showed inconsistencies over time, between related indicators, and with an external data source. The best quality data elements were those that aligned with Gombe's health program priorities, particularly older health programs, and those that reflected contact indicators rather than indicators related to the provision of commodities or content of care.

CONCLUSION

This case study from Gombe State, Nigeria, demonstrates the high potential for effective monitoring of maternal and neonatal health using DHIS2. However, coordinated action at multiple levels of the health system is needed to maximize reporting of existing data; rationalize data flow; routinize data quality review, feedback, and supervision; and ensure ongoing maintenance of DHIS2.

Quality improvement and emerging global health priorities

Quality improvement approaches can strengthen action on a range of global health priorities. Quality improvement efforts are uniquely placed to reorient care delivery systems towards integrated people-centred health services and strengthen health systems to achieve Universal Health Coverage (UHC). This article makes the case for addressing shortfalls of previous agendas by articulating the critical role of quality improvement in the Sustainable Development Goal era. Quality improvement can stimulate convergence between health security and health systems; address global health security priorities through participatory quality improvement approaches; and improve health outcomes at all levels of the health system. Entry points for action include the linkage with antimicrobial resistance and the contentious issue of the health of migrants. The work required includes focussed attention on the continuum of national quality policy formulation, implementation and learning; alongside strengthening the measurement-improvement linkage. Quality improvement plays a key role in strengthening health systems to achieve UHC.

Evidence for national universal eye health plans

Many low- and middle-income countries use national eye-care plans to guide efforts to strengthen eye-care services. The World Health Organization recognizes that evidence is essential to inform these plans. We assessed how evidence was incorporated in a sample of 28 national eye-care plans generated since the Universal eye health: a global action plan 2014-2019 was endorsed by the World Health Assembly in 2013. Most countries (26, 93%) cited estimates of the prevalence of blindness and 18 countries (64%) had set targets for the cataract surgical rate in their plan. Other evidence was rarely cited or used to set measurable targets. No country cited evidence from systematic reviews or solution-based research. This limited use of evidence reflects its low availability, but also highlights incomplete use of existing evidence. For example, despite sex-disaggregated data and cataract surgical coverage being available from surveys in 20 countries (71%), these data were reported in the eye health plans of only nine countries (32%). Only three countries established sex-disaggregated indicators and only one country had set a target for cataract surgical coverage for future monitoring. Countries almost universally recognized the need to strengthen health information systems and almost one-third planned to undertake operational or intervention research. Realistic strategies need to be identified and supported to translate these intentions into action. To gain insights into how a country can strengthen its evidence-informed approach to eye-care planning, we reflect on the process underway to develop Kenya's seventh national plan (2019-2023).

In Madagascar, Use Of Health Care Services Increased When Fees Were Removed: Lessons For Universal Health Coverage

Despite overwhelming burdens of disease, health care access in most developing countries is extremely low. As governments work toward achieving universal health coverage, evidence on appropriate interventions to expand access in rural populations is critical for informing policies. Using a combination of population and health system data, we evaluated the impact of two pilot fee exemption interventions in a rural area of Madagascar. We found that fewer than one-third of people in need of health care accessed treatment when point-of-service fees were in place. However, when fee exemptions were introduced for targeted medicines and services, the use of health care increased by 65 percent for all patients, 52 percent for children under age five, and over 25 percent for maternity consultations. These effects were sustained at an average direct cost of US$0.60 per patient. The pilot interventions can become a key element of universal health care in Madagascar with the support of external donors.

The impact of routine data quality assessments on electronic medical record data quality in Kenya

Background

Routine Data Quality Assessments (RDQAs) were developed to measure and improve facility-level electronic medical record (EMR) data quality. We assessed if RDQAs were associated with improvements in data quality in KenyaEMR, an HIV care and treatment EMR used at 341 facilities in Kenya.

Methods

RDQAs assess data quality by comparing information recorded in paper records to KenyaEMR. RDQAs are conducted during a one-day site visit, where approximately 100 records are randomly selected and 24 data elements are reviewed to assess data completeness and concordance. Results are immediately provided to facility staff and action plans are developed for data quality improvement. For facilities that had received more than one RDQA (baseline and follow-up), we used generalized estimating equation models to determine if data completeness or concordance improved from the baseline to the follow-up RDQAs.

Results

27 facilities received two RDQAs and were included in the analysis, with 2369 and 2355 records reviewed from baseline and follow-up RDQAs, respectively. The frequency of missing data in KenyaEMR declined from the baseline (31% missing) to the follow-up (13% missing) RDQAs. After adjusting for facility characteristics, records from follow-up RDQAs had 0.43-times the risk (95% CI: 0.32–0.58) of having at least one missing value among nine required data elements compared to records from baseline RDQAs. Using a scale with one point awarded for each of 20 data elements with concordant values in paper records and KenyaEMR, we found that data concordance improved from baseline (11.9/20) to follow-up (13.6/20) RDQAs, with the mean concordance score increasing by 1.79 (95% CI: 0.25–3.33).

Conclusions

This manuscript demonstrates that RDQAs can be implemented on a large scale and used to identify EMR data quality problems. RDQAs were associated with meaningful improvements in data quality and could be adapted for implementation in other settings.

Availability and quality of routine morbidity data: review of studies in South Africa

Objectives

Routine health information systems (RHISs) provide data that are vital for planning and monitoring individual health. Data from RHISs could also be used for purposes for which they were not originally intended, provided that the data are of sufficient quality. For example, morbidity data could be used to inform burden of disease estimations, which serve as important evidence to prioritize interventions and promote health. The objective of this study was to identify and assess published quantitative assessments of data quality related to patient morbidity in RHISs in use in South Africa.

Materials and Methods

We conducted a review of literature published between 1994 and 2014 that assessed the quality of data in RHISs in South Africa. World Health Organization (WHO) data quality components were used as the assessment criteria.

Results

Of 420 references identified, 11 studies met the inclusion criteria. The studies were limited to tuberculosis and HIV. No study reported more than 3 WHO data quality components or provided a quantitative assessment of quality that could be used for burden of disease estimation.

Discussion

The included studies had limited geographical focus and evaluated different source data at different levels of the information system. All studies reported poor data quality.

Conclusion

This review confirmed concerns about the quality of data in RHISs, and highlighted the need for a comprehensive evaluation of the quality of patient-level morbidity data in RHISs in South Africa.

Data-driven quality improvement in low-and middle-income country health systems: lessons from seven years of implementation experience across Mozambique, Rwanda, and Zambia

BACKGROUND

Well-functioning health systems need to utilize data at all levels, from the provider, to local and national-level decision makers, in order to make evidence-based and needed adjustments to improve the quality of care provided. Over the last 7 years, the Doris Duke Charitable Foundation's African Health Initiative funded health systems strengthening projects at the facility, district, and/or provincial level to improve population health. Increasing data-driven decision making was a common strategy in Mozambique, Rwanda and Zambia. This paper describes the similar and divergent approaches to increase data-driven quality of care improvements (QI) and implementation challenge and opportunities encountered in these three countries.

METHODS

Eight semi-structured in-depth interviews (IDIs) were administered to program staff working in each country. IDIs for this paper included principal investigators of each project, key program implementers (medically-trained support staff, data managers and statisticians, and country directors), as well as Ministry of Health counterparts. IDI data were collected through field notes; interviews were not audio recorded. Data were analyzed using thematic analysis but no systematic coding was conducted. IDIs were supplemented through donor report abstractions, a structured questionnaire, one-on-one phone calls, and email exchanges with country program leaders to clarify and expand on key themes emerging from IDIs.

RESULTS

Project successes ranged from over 450 collaborative action-plans developed, implemented, and evaluated in Mozambique, to an increase from <10% to >80% of basic clinical protocols followed in intervention facilities in rural Zambia, and a shift from a lack of awareness of health data among health system staff to collaborative ownership of data and using data to drive change in Rwanda.

CONCLUSION

Based on common successes across the country experiences, we recommend future data-driven QI interventions begin with data quality assessments to promote that rapid health system improvement is possible, ensure confidence in available data, serve as the first step in data-driven targeted improvements, and improve staff data analysis and visualization skills. Explicit Ministry of Health collaborative engagement can ensure performance review is collaborative and internally-driven rather than viewed as an external "audit."

Tackling the hard problems: implementation experience and lessons learned in newborn health from the African Health Initiative

Background

The Doris Duke Charitable Foundation’s African Health Initiative supported the implementation of Population Health Implementation and Training (PHIT) Partnership health system strengthening interventions in designated areas of five countries: Ghana, Mozambique, Rwanda, Tanzania, and Zambia. All PHIT programs included health system strengthening interventions with child health outcomes from the outset, but all increasingly recognized the need to increase focus to improve health and outcomes in the first month of life. This paper uses a case study approach to describe interventions implemented in newborn health, compare approaches, and identify lessons learned across the programs’ collective implementation experience.

Methods

Case studies were built using quantitative and qualitative methods, applying the World Health Organization Health Systems Strengthening Framework, and maternal, newborn and child health continuum of care framework. We identified the following five primary themes in health systems strengthening intervention strategies used to target improvement in newborn health, which were incorporated by all PHIT projects with varying results: health service delivery at the community level (Tanzania), combining community and health facility level interventions (Zambia), participatory information feedback and clinical training (Ghana), performance review and enhancement (Mozambique), and integrated clinical and system-level improvement (Rwanda), and used individual case studies to illustrate each of these themes.

Results

Tanzania and Zambia included significant community-based components, including mobilization and sensitization for increased uptake of essential services, while Ghana, Mozambique, and Rwanda focused more efforts on improving the quality of services delivered once a patient enters a health facility. All countries included aspects that improved communication across levels of the health system, whether through district-wide data sharing and peer learning networks in Mozambique and Rwanda, or improved referral processes and systems in Tanzania, Zambia, and Ghana.

Conclusion

Key lessons learned include the importance of focusing intervention components on addressing drivers of neonatal mortality across the maternal and newborn care continuum at all levels of the health system, matching efforts to improve service utilization with provision of high quality facility-based services, and the critical role of leadership to catalyze improvements in newborn health.

Measuring health systems strength and its impact: experiences from the African Health Initiative

Background

Health systems are essential platforms for accessible, quality health services, and population health improvements. Global health initiatives have dramatically increased health resources; however, funding to strengthen health systems has not increased commensurately, partially due to concerns about health system complexity and evidence gaps demonstrating health outcome improvements. In 2009, the African Health Initiative of the Doris Duke Charitable Foundation began supporting Population Health Implementation and Training Partnership projects in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) to catalyze significant advances in strengthening health systems. This manuscript reflects on the experience of establishing an evaluation framework to measure health systems strength, and associate measures with health outcomes, as part of this Initiative.

Methods

Using the World Health Organization’s health systems building block framework, the Partnerships present novel approaches to measure health systems building blocks and summarize data across and within building blocks to facilitate analytic procedures. Three Partnerships developed summary measures spanning the building blocks using principal component analysis (Ghana and Tanzania) or the balanced scorecard (Zambia). Other Partnerships developed summary measures to simplify multiple indicators within individual building blocks, including health information systems (Mozambique), and service delivery (Rwanda). At the end of the project intervention period, one to two key informants from each Partnership’s leadership team were asked to list – in rank order – the importance of the six building blocks in relation to their intervention.

Results

Though there were differences across Partnerships, service delivery and information systems were reported to be the most common focus of interventions, followed by health workforce and leadership and governance. Medical products, vaccines and technologies, and health financing, were the building blocks reported to be of lower focus.

Conclusion

The African Health Initiative experience furthers the science of evaluation for health systems strengthening, highlighting areas for further methodological development – including the development of valid, feasible measures sensitive to interventions in multiple contexts (particularly in leadership and governance) and describing interactions across building blocks; in developing summary statistics to facilitate testing intervention effects on health systems and associations with health status; and designing appropriate analytic models for complex, multi-level open health systems.

Mentorship and coaching to support strengthening healthcare systems: lessons learned across the five Population Health Implementation and Training partnership projects in sub-Saharan Africa

Background

Despite global efforts to increase health workforce capacity through training and guidelines, challenges remain in bridging the gap between knowledge and quality clinical practice and addressing health system deficiencies preventing health workers from providing high quality care. In many developing countries, supervision activities focus on data collection, auditing and report completion rather than catalyzing learning and supporting system quality improvement. To address this gap, mentorship and coaching interventions were implemented in projects in five African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) as components of health systems strengthening (HSS) strategies funded through the Doris Duke Charitable Foundation’s African Health Initiative. We report on lessons learned from a cross-country evaluation.

Methods

The evaluation was designed based on a conceptual model derived from the project-specific interventions. Semi-structured interviews were administered to key informants to capture data in six categories: 1) mentorship and coaching goals, 2) selection and training of mentors and coaches, 3) integration with the existing systems, 4) monitoring and evaluation, 5) reported outcomes, and 6) challenges and successes. A review of project-published articles and technical reports from the individual projects supplemented interview information.

Results

Although there was heterogeneity in the approaches to mentorship and coaching and targeted areas of the country projects, all led to improvements in core health system areas, including quality of clinical care, data-driven decision making, leadership and accountability, and staff satisfaction. Adaptation of approaches to reflect local context encouraged their adoption and improved their effectiveness and sustainability.

Conclusion

We found that incorporating mentorship and coaching activities into HSS strategies was associated with improvements in quality of care and health systems, and mentorship and coaching represents an important component of HSS activities designed to improve not just coverage, but even further effective coverage, in achieving Universal Health Care.

Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): results from the African Health Initiative

Background

High-quality data are critical to inform, monitor and manage health programs. Over the seven-year African Health Initiative of the Doris Duke Charitable Foundation, three of the five Population Health Implementation and Training (PHIT) partnership projects in Mozambique, Rwanda, and Zambia introduced strategies to improve the quality and evaluation of routinely-collected data at the primary health care level, and stimulate its use in evidence-based decision-making. Using the Consolidated Framework for Implementation Research (CFIR) as a guide, this paper: 1) describes and categorizes data quality assessment and improvement activities of the projects, and 2) identifies core intervention components and implementation strategy adaptations introduced to improve data quality in each setting.

Methods

The CFIR was adapted through a qualitative theme reduction process involving discussions with key informants from each project, who identified two domains and ten constructs most relevant to the study aim of describing and comparing each country’s data quality assessment approach and implementation process. Data were collected on each project’s data quality improvement strategies, activities implemented, and results via a semi-structured questionnaire with closed and open-ended items administered to health management information systems leads in each country, with complementary data abstraction from project reports.

Results

Across the three projects, intervention components that aligned with user priorities and government systems were perceived to be relatively advantageous, and more readily adapted and adopted. Activities that both assessed and improved data quality (including data quality assessments, mentorship and supportive supervision, establishment and/or strengthening of electronic medical record systems), received higher ranking scores from respondents.

Conclusion

Our findings suggest that, at a minimum, successful data quality improvement efforts should include routine audits linked to ongoing, on-the-job mentoring at the point of service. This pairing of interventions engages health workers in data collection, cleaning, and analysis of real-world data, and thus provides important skills building with on-site mentoring. The effect of these core components is strengthened by performance review meetings that unify multiple health system levels (provincial, district, facility, and community) to assess data quality, highlight areas of weakness, and plan improvements.

Impact of a district-wide health center strengthening intervention on healthcare utilization in rural Rwanda: Use of interrupted time series analysis

Background

Evaluations of health systems strengthening (HSS) interventions using observational data are rarely used for causal inference due to limited data availability. Routinely collected national data allow use of quasi-experimental designs such as interrupted time series (ITS). Rwanda has invested in a robust electronic health management information system (HMIS) that captures monthly healthcare utilization data. We used ITS to evaluate impact of an HSS intervention to improve primary health care facility readiness on health service utilization in two rural districts of Rwanda.

Methods

We used controlled ITS analysis to compare changes in healthcare utilization at health centers (HC) that received the intervention (n = 13) to propensity score matched non-intervention health centers in Rwanda (n = 86) from January 2008 to December 2012. HC support included infrastructure renovation, salary support, medical equipment, referral network strengthening, and clinical training. Baseline quarterly mean outpatient visit rates and population density were used to model propensity scores. The intervention began in May 2010 and was implemented over a twelve-month period. We used monthly healthcare utilization data from the national Rwandan HMIS to study changes in the (1) number of facility deliveries per 10,000 women, (2) number of referrals for high risk pregnancy per 100,000 women, and (3) the number of outpatient visits performed per 1,000 catchment population.

Results

PHIT HC experienced significantly higher monthly delivery rates post-HSS during the April-June season than comparison (3.19/10,000, 95% CI: [0.27, 6.10]). In 2010, this represented a 13% relative increase, and in 2011, this represented a 23% relative increase. The post-HSS change in monthly rate of high-risk pregnancies referred increased slightly in intervention compared to control HC (0.03/10,000, 95% CI: [-0.007, 0.06]). There was a small immediate post-HSS increase in outpatient visit rates in intervention compared to control HC (6.64/1,000, 95% CI: [-13.52, 26.81]).

Conclusion

We failed to find strong evidence of post-HSS increases in outpatient visit rates or referral rates at health centers, which could be explained by small sample size and high baseline nation-wide health service coverage. However, our findings demonstrate that high quality routinely collected health facility data combined with ITS can be used for rigorous policy evaluation in resource-limited settings.

Moving the Agenda on Noncommunicable Diseases: Policy Implications of Mobile Phone Surveys in Low and Middle-Income Countries

The growing burden of noncommunicable diseases (NCDs), for example, cardiovascular diseases and chronic respiratory diseases, in low- and middle-income countries (LMICs) presents special challenges for policy makers, due to resource constraints and lack of timely data for decision-making. Concurrently, the increasing ubiquity of mobile phones in LMICs presents possibilities for rapid collection of population-based data to inform the policy process. The objective of this paper is to highlight potential benefits of mobile phone surveys (MPS) for developing, implementing, and evaluating NCD prevention and control policies. To achieve this aim, we first provide a brief overview of major global commitments to NCD prevention and control, and subsequently explore how countries can translate these commitments into policy action at the national level. Using the policy cycle as our frame of reference, we highlight potential benefits of MPS which include (1) potential cost-effectiveness of using MPS to inform NCD policy actions compared with using traditional household surveys; (2) timeliness of assessments to feed into policy and planning cycles; (3) tracking progress of interventions, hence assessment of reach, coverage, and distribution; (4) better targeting of interventions, for example, to high-risk groups; (5) timely course correction for suboptimal or non-effective interventions; (6) assessing fairness in financial contribution and financial risk protection for those affected by NCDs in the spirit of universal health coverage (UHC); and (7) monitoring progress in reducing catastrophic medical expenditure due to chronic health conditions in general, and NCDs in particular. We conclude that MPS have potential to become a powerful data collection tool to inform policies that address public health challenges such as NCDs. Additional forthcoming assessments of MPS in LMICs will inform opportunities to maximize this technology.

Root Causes of Poor Immunisation Data Quality and Proven Interventions: A Systematic Literature Review

Introduction

Effective allocation of resources and investments heavily rely on good quality data. As global investments in vaccines increases, particularly by organisations such as Gavi, The Vaccine Alliance, Switzerland, the demand for data which is accurate and representative is urgent. Understanding what causes poor immunisation data and how to address these problems are therefore key in maximizing investments, improving coverage and reducing risks of outbreaks.

Objective

Identify the root causes of poor immunisation data quality and proven solutions for guiding future data quality interventions.

Methods and Results

Qualitative systematic review of both scientific and grey literature using key words on immunisation and health information systems. Once screened, articles were classified either as identifying root causes of poor data quality or as an intervention to improve data quality. A total of 8,646 articles were initially identified which were screened and reduced to 26. Results were heterogeneous in methodology, settings and conclusions with a variation of outcomes. Key themes were underperformance in health facilities and limited Human Resource (HR) capacity at the peripheral level leading to data of poor quality. Repeated reference to a "culture" of poor data collection, reporting and use in low-income countries was found implying that it is the attitudes and subsequent behaviour of staff that prevents good quality data. Documented interventions mainly involved implementing Information Communication Technology (ICT) at the district level. However, without changes in HR capacity the skills and practices of staff remain a key impediment to reaching its full impact.

Discussion

There was a clear incompatibility between identified root causes, mainly being behavioural and organizational factors, and interventions introducing predominantly technical factors. More emphasis should be placed on interventions that build on current practices and skills in a gradual process in order to be more readily adopted by health workers. Major gaps in the literature exist mainly in the lack of assessment at central and intermediate levels and association between inaccurate target setting from outdated census data and poor data quality as well as limited documentation of interventions that target behaviour change and policy change. This prevents the ability to make informed decisions on best methodology for improving data quality.

Opportunities and challenges in conducting secondary analysis of HIV programmes using data from routine health information systems and personal health information

INTRODUCTION

HIV programme data from routine health information systems (RHIS) and personal health information (PHI) provide ample opportunities for secondary data analysis. However, these data pose unique opportunities and challenges for use in health system monitoring, along with process and impact evaluations.

METHODS

Analyses focused on retrospective case reviews of four of the HIV-related studies published in this JIAS supplement. We identify specific opportunities and challenges with respect to the secondary analysis of RHIS and PHI data.

RESULTS

Challenges working with both HIV-related RHIS and PHI included missing, inconsistent and implausible data; rapidly changing indicators; systematic differences in the utilization of services; and patient linkages over time and different data sources. Specific challenges among RHIS data included numerous registries and indicators, inconsistent data entry, gaps in data transmission, duplicate registry of information, numerator-denominator incompatibility and infrequent use of data for decision-making. Challenges specific to PHI included the time burden for busy providers, the culture of lax charting, overflowing archives for paper charts and infrequent chart review.

CONCLUSIONS

Many of the challenges that undermine effective use of RHIS and PHI data for analyses are related to the processes and context of collecting the data, excessive data requirements, lack of knowledge of the purpose of data and the limited use of data among those generating the data. Recommendations include simplifying data sources, analysis and reporting; conducting systematic data quality audits; enhancing the use of data for decision-making; promoting routine chart review linked with simple patient tracking systems; and encouraging open access to RHIS and PHI data for increased use.

An assessment of data quality in a multi-site electronic medical record system in Haiti

OBJECTIVES

Strong data quality (DQ) is a precursor to strong data use. In resource limited settings, routine DQ assessment (DQA) within electronic medical record (EMR) systems can be resource-intensive using manual methods such as audit and chart review; automated queries offer an efficient alternative. This DQA focused on Haiti's national EMR - iSanté - and included longitudinal data for over 100,000 persons living with HIV (PLHIV) enrolled in HIV care and treatment services at 95 health care facilities (HCF).

METHODS

This mixed-methods evaluation used a qualitative Delphi process to identify DQ priorities among local stakeholders, followed by a quantitative DQA on these priority areas. The quantitative DQA examined 13 indicators of completeness, accuracy, and timeliness of retrospective data collected from 2005 to 2013. We described levels of DQ for each indicator over time, and examined the consistency of within-HCF performance and associations between DQ and HCF and EMR system characteristics.

RESULTS

Over all iSanté data, age was incomplete in <1% of cases, while height, pregnancy status, TB status, and ART eligibility were more incomplete (approximately 20-40%). Suspicious data flags were present for <3% of cases of male sex, ART dispenses, CD4 values, and visit dates, but for 26% of cases of age. Discontinuation forms were available for about half of all patients without visits for 180 or more days, and >60% of encounter forms were entered late. For most indicators, DQ tended to improve over time. DQ was highly variable across HCF, and within HCFs DQ was variable across indicators. In adjusted analyses, HCF and system factors with generally favorable and statistically significant associations with DQ were University hospital category, private sector governance, presence of local iSante server, greater HCF experience with the EMR, greater maturity of the EMR itself, and having more system users but fewer new users. In qualitative feedback, local stakeholders emphasized lack of stable power supply as a key challenge to data quality and use of the iSanté EMR.

CONCLUSIONS

Variable performance on key DQ indicators across HCF suggests that excellent DQ is achievable in Haiti, but further effort is needed to systematize and routinize DQ approaches within HCFs. A dynamic, interactive "DQ dashboard" within iSanté could bring transparency and motivate improvement. While the results of the study are specific to Haiti's iSanté data system, the study's methods and thematic lessons learned holdgeneralized relevance for other large-scale EMR systems in resource-limited countries.

Health facility determinants and trends of ICD-10 outpatient psychiatric consultations across Sofala, Mozambique: time-series analyses from 2012 to 2014

BACKGROUND

Few peer-reviewed publications have taken a longitudinal or systems approach to mental healthcare (MH) utilization in low- and middle-income countries. We analyzed: (1) outpatient ICD-10 diagnoses over time and by gender; and (2) health facility determinants of MH service utilization.

METHODS

We reviewed a census of 15,856 outpatient psychiatric consultations conducted at Ministry clinics in Sofala province, Mozambique from January 2012-June 2014. Generalized estimating equations were used to model facility determinants of ICD-10 diagnoses.

RESULTS

Across the period, 48.9 % of consults were for epilepsy, 22.4 % for schizophrenia/delusional disorders, and 8.8 % for neurotic/stress-related disorders. The proportion of schizophrenia/delusional disorders has decreased over time (32 % in 2012; 13 % in 2014, p = 0.003), in favor of greater diversity of diagnoses. Epilepsy has increased significantly in absolute and proportional terms. Women are more likely to present for neurotic/stress-related conditions (12.8 % of consults for women, 5.7 % for men, p < 0.001), while men are more likely to present for substance use (1.9 % for women, 6.4 % for men, p < 0.001). Clinics with more psychiatric technicians have a 2.1-fold (CI: 1.2, 3.6) increased rate of schizophrenia/delusional disorder diagnoses. Rural clinics saw a higher proportion of epilepsy cases and a lower proportion of organic, substance use, schizophrenia, and mood disorder cases.

DISCUSSION AND CONCLUSIONS

Outpatient MH service provision is increasing in Mozambique, although currently focuses on epilepsy and schizophrenia/delusional disorders. Mid-level psychiatric providers appear to be associated with a higher proportion of schizophrenia/delusional disorder diagnoses. Due to diagnostic or utilization differences, rural clinics may be missing important cases of organic, substance use, schizophrenia, and mood disorders. Models and decision-support tools for mental healthcare integration with primary care practice are needed in Mozambique to allow further scale-up of mental health services.

Using routine health information systems for well-designed health evaluations in low- and middle-income countries

Routine health information systems (RHISs) are in place in nearly every country and provide routinely collected full-coverage records on all levels of health system service delivery. However, these rich sources of data are regularly overlooked for evaluating causal effects of health programmes due to concerns regarding completeness, timeliness, representativeness and accuracy. Using Mozambique's national RHIS (Módulo Básico) as an illustrative example, we urge renewed attention to the use of RHIS data for health evaluations. Interventions to improve data quality exist and have been tested in low-and middle-income countries (LMICs). Intrinsic features of RHIS data (numerous repeated observations over extended periods of time, full coverage of health facilities, and numerous real-time indicators of service coverage and utilization) provide for very robust quasi-experimental designs, such as controlled interrupted time-series (cITS), which are not possible with intermittent community sample surveys. In addition, cITS analyses are well suited for continuously evolving development contexts in LMICs by: (1) allowing for measurement and controlling for trends and other patterns before, during and after intervention implementation; (2) facilitating the use of numerous simultaneous control groups and non-equivalent dependent variables at multiple nested levels to increase validity and strength of causal inference; and (3) allowing the integration of continuous 'effective dose received' implementation measures. With expanded use of RHIS data for the evaluation of health programmes, investments in data systems, health worker interest in and utilization of RHIS data, as well as data quality will further increase over time. Because RHIS data are ministry-owned and operated, relying upon these data will contribute to sustainable national capacity over time.