Rethinking Antimicrobial Resistance Surveillance: A Role for Lot Quality Assurance Sampling

Global surveillance of antimicrobial resistance (AMR) is a key component of the 68th World Health Assembly Global Action Plan on AMR. Laboratory-based surveillance is inherently biased and lacks local relevance due to aggregation of data. We assessed the feasibility, sensitivity, and affordability of a population-based AMR survey using lot quality assurance sampling (LQAS), which classifies a population as having a high or low prevalence of AMR based on a priori defined criteria. Three studies were carried out in Medan and Bandung, Indonesia, between April 2014 and June 2017. LQAS classifications for 15 antibiotics were compared with AMR estimates from a conventional population-based survey, with an assessment of the cost of a single LQAS classification using microcosting methodology, among patients suspected of urinary tract infection at 11 sites in Indonesia. The sensitivity of LQAS was above 98%. The approach detected local variation in the prevalence of AMR across sites. Time to reach LQAS results ranged from 47 to 138 days. The average cost of an LQAS classification in a single facility was US$466. The findings indicate that LQAS-based AMR survey is a feasible, sensitive, and affordable strategy for population-based AMR surveys, providing essential data to inform local empirical treatment guidelines and antimicrobial stewardship efforts.

Institutionalizing and sustaining social change in health systems: the case of Uganda

The key to high impact health services is institutionalizing and sustaining programme evaluation. Uganda represents a success story in the use of a specific programme evaluation method: Lot Quality Assurance Sampling (LQAS). Institutionalization is defined by two C’s: competent programme evaluators and control mechanisms that effectively use evaluation data to improve health services. Sustainability means continued training and funding for the evaluation approach. Social science literature that researches institutionalization has emphasized ‘stability’, whereas in global health, the issue is determining how to improve the impact of services by ‘changing’ programmes. In Uganda, we measured the extent of the institutionalization and sustainability of evaluating programmes that produce change in nine districts sampled to represent three largely rural regions and varying levels of effective health programmes. We used the proportion of mothers with children aged 0–11 months who delivered in a health facility as the principal indicator to measure programme effectiveness. Interviews and focus groups were conducted among directors, evaluation supervisors, data collectors in the district health offices, and informant interviews conducted individually at the central government level. Seven of the nine districts demonstrated a high level of institutionalization of evaluation. The two others had only conducted one round of programme evaluation. When we control for the availability of health facilities, we find that the degree of institutionalization is moderately related to the prevalence of the delivery of a baby in a health facility. Evaluation was institutionalized at the central government level. Sustainability existed at both levels. Several measures indicate that lessons from the nine district case studies may be relevant to the 74 districts that had at least two rounds of programme evaluation. We note that there is an association between the evaluation data being used to change health services, and the four separate indicators being used to measure women's health and child survival services. We conclude that the two C’s (competent evaluators and control mechanisms) have been critical for sustaining programme evaluation in Uganda.

Comparing two survey methods of measuring health-related indicators: Lot Quality Assurance Sampling and Demographic Health Surveys

OBJECTIVES

Two common methods used to measure indicators for health programme monitoring and evaluation are the demographic and health surveys (DHS) and lot quality assurance sampling (LQAS); each one has different strengths. We report on both methods when utilised in comparable situations.

METHODS

We compared 24 indicators in south-west Uganda, where data for prevalence estimations were collected independently for the two methods in 2011 (LQAS: n = 8876; DHS: n = 1200). Data were stratified (e.g. gender and age) resulting in 37 comparisons. We used a two-sample two-sided Z-test of proportions to compare both methods.

RESULTS

The average difference between LQAS and DHS for 37 estimates was 0.062 (SD = 0.093; median = 0.039). The average difference among the 21 failures to reject equality of proportions was 0.010 (SD = 0.041; median = 0.009); among the 16 rejections, it was 0.130 (SD = 0.010, median = 0.118). Seven of the 16 rejections exhibited absolute differences of <0.10, which are clinically (or managerially) not significant; 5 had differences >0.10 and <0.20 (mean = 0.137, SD = 0.031) and four differences were >0.20 (mean = 0.261, SD = 0.083).

CONCLUSION

There is 75.7% agreement across the two surveys. Both methods yield regional results, but only LQAS provides information at less granular levels (e.g. the district level) where managerial action is taken. The cost advantage and localisation make LQAS feasible to conduct more frequently, and provides the possibility for real-time health outcomes monitoring.

Finding the gap: revealing local disparities in coverage of maternal, newborn and child health services in South Sudan using lot quality assurance sampling

OBJECTIVES

We adapted a rapid monitoring method to South Sudan, a new nation with one of the world's highest maternal and child mortality rates, aiming to assess coverage of maternal, neonatal and child health (MNCH) services at the time of independence, and introducing a monitoring and evaluation system (M&E) for equity-sensitive tracking of progress related to Millennium Development Goals (MDG) 4 and 5 at national, state and county levels to detect local variability.

METHODS

We conducted a national cross-sectional household survey among women from six client populations in all, but six of South Sudan's 79 counties. We used lot quality assurance sampling (LQAS) to measure coverage with diverse MNCH indicators to obtain information for national-, state- and county-level health system management decision-making.

RESULTS

National coverage of MNCH services was low for all maternal and neonatal care, child immunisation, and child care indicators. However, results varied across states and counties. Central Equatoria State (CES), where the capital is located, showed the highest coverage for most indicators (e.g. ≥4 antenatal care visits range: 4.5% in Jonglei to 40.1% in CES). Urban counties often outperformed rural ones.

CONCLUSIONS

This adaptation of LQAS to South Sudan demonstrates how it can be used in the future as an M&E system to track progress of MDGs at national, state and county levels to detect local disparities. Overall, our data reveal a desperate need for improving MNCH service coverage in all states.

Monitoring maternal, newborn, and child health interventions using lot quality assurance sampling in Sokoto State of northern Nigeria

BACKGROUND

Maternal mortality ratio and infant mortality rate are as high as 1,576 per 100,000 live births and 78 per 1,000 live births, respectively, in Nigeria's northwestern region, where Sokoto State is located. Using applicable monitoring indicators for tracking progress in the UN/WHO framework on continuum of maternal, newborn, and child health care, this study evaluated the progress of Sokoto toward achieving the Millennium Development Goals (MDGs) 4 and 5 by December 2015. The changes in outcomes in 2012-2013 associated with maternal and child health interventions were assessed.

DESIGN

We used baseline and follow-up lot quality assurance sampling (LQAS) data obtained in 2012 and 2013, respectively. In each of the surveys, data were obtained from 437 households sampled from 19 LQAS locations in each of the 23 local government areas (LGAs). The composite state-level coverage estimates of the respective indicators were aggregated from estimated LGA coverage estimates.

RESULTS

None of the nine indicators associated with the continuum of maternal, neonatal, and child care satisfied the recommended 90% coverage target for achieving MDGs 4 and 5. Similarly, the average state coverage estimates were lower than national coverage estimates. Marginal improvements in coverage were obtained in the demand for family planning satisfied, antenatal care visits, postnatal care for mothers, and exclusive breast-feeding. Antibiotic treatment for acute pneumonia increased significantly by 12.8 percentage points. The majority of the LGAs were classifiable as low-performing, high-priority areas for intensified program intervention.

CONCLUSIONS

Despite the limited time left in the countdown to December 2015, Sokoto State, Nigeria, is not on track to achieving the MDG 90% coverage of indicators tied to the continuum of maternal and child care, to reduce maternal and childhood mortality by a third by 2015. Targeted health system investments at the primary care level remain a priority, for intensive program scale-up to accelerate impact.

Lot quality assurance sampling to monitor supplemental immunization activity quality: an essential tool for improving performance in polio endemic countries

Monitoring the quality of supplementary immunization activities (SIAs) is a key tool for polio eradication. Regular monitoring data, however, are often unreliable, showing high coverage levels in virtually all areas, including those with ongoing virus circulation. To address this challenge, lot quality assurance sampling (LQAS) was introduced in 2009 as an additional tool to monitor SIA quality. Now used in 8 countries, LQAS provides a number of programmatic benefits: identifying areas of weak coverage quality with statistical reliability, differentiating areas of varying coverage with greater precision, and allowing for trend analysis of campaign quality. LQAS also accommodates changes to survey format, interpretation thresholds, evaluations of sample size, and data collection through mobile phones to improve timeliness of reporting and allow for visualization of campaign quality. LQAS becomes increasingly important to address remaining gaps in SIA quality and help focus resources on high-risk areas to prevent the continued transmission of wild poliovirus.

Improving the collection of knowledge, attitude and practice data with community surveys: a comparison of two second-stage sampling methods

BACKGROUND

Second-stage sampling techniques, including spatial segmentation, are widely used in community health surveys when reliable household sampling frames are not available. In India, an unresearched technique for household selection is used in eight states, which samples the house with the last marriage or birth as the starting point. Users question whether this last-birth or last-marriage (LBLM) approach introduces bias affecting survey results.

METHODS

We conducted two simultaneous population-based surveys. One used segmentation sampling; the other used LBLM. LBLM sampling required modification before assessment was possible and a more systematic approach was tested using last birth only. We compared coverage proportions produced by the two independent samples for six malaria indicators and demographic variables (education, wealth and caste). We then measured the level of agreement between the caste of the selected participant and the caste of the health worker making the selection.

RESULTS

No significant difference between methods was found for the point estimates of six malaria indicators, education, caste or wealth of the survey participants (range of P: 0.06 to >0.99). A poor level of agreement occurred between the caste of the health worker used in household selection and the caste of the final participant, (Κ = 0.185), revealing little association between the two, and thereby indicating that caste was not a source of bias.

CONCLUSIONS

Although LBLM was not testable, a systematic last-birth approach was tested. If documented concerns of last-birth sampling are addressed, this new method could offer an acceptable alternative to segmentation in India. However, inter-state caste variation could affect this result. Therefore, additional assessment of last birth is required before wider implementation is recommended.

Monitoring health interventions--who's afraid of LQAS?

Lot quality assurance sampling (LQAS) is used to evaluate health services. Subunits of a population (lots) are accepted or rejected according to the number of failures in a random sample (N) of a given lot. If failures are greater than decision value (d), we reject the lot and recommend corrective actions in the lot (i.e. intervention area); if they are equal to or less than d, we accept it. We used LQAS to monitor coverage during the last 3 days of a meningitis vaccination campaign in Niger. We selected one health area (lot) per day reporting the lowest administrative coverage in the previous 2 days. In the sampling plan we considered: N to be small enough to allow us to evaluate one lot per day, deciding to sample 16 individuals from the selected villages of each health area, using probability proportionate to population size; thresholds and d to vary according to administrative coverage reported; α ≤5% (meaning that, if we would have conducted the survey 100 times, we would have accepted the lot up to five times when real coverage was at an unacceptable level) and β ≤20% (meaning that we would have rejected the lot up to 20 times, when real coverage was equal or above the satisfactory level). We classified all three lots as with the acceptable coverage. LQAS appeared to be a rapid, simple, and statistically sound method for in-process coverage assessment. We encourage colleagues in the field to consider using LQAS in complement with other monitoring techniques such as house-to-house monitoring.

Assessing and improving data quality from community health workers: a successful intervention in Neno, Malawi

SETTING

A community health worker (CHW) program was established in Neno District, Malawi, in 2007 by Partners In Health in support of Ministry of Health activities. Routinely generated CHW data provide critical information for program monitoring and evaluation. Informal assessments of the CHW reports indicated poor quality, limiting the usefulness of the data.

OBJECTIVES

1) To establish the quality of aggregated measures contained in CHW reports; 2) to develop interventions to address poor data quality; and 3) to evaluate changes in data quality following the intervention.

DESIGN

We developed a lot quality assurance sampling-based data quality assessment tool to identify sites with high or low reporting quality. Following the first assessment, we identified challenges and best practices and followed the interventions with two subsequent assessments.

RESULTS

At baseline, four of five areas were classified as low data quality. After 8 months, all five areas had achieved high data quality, and the reports generated from our electronic database became consistent and plausible.

CONCLUSION

Program changes included improving the usability of the reporting forms, shifting aggregation responsibility to designated assistants and providing aggregation support tools. Local quality assessments and targeted interventions resulted in immediate improvements in data quality.

Cluster designs to assess the prevalence of acute malnutrition by lot quality assurance sampling: a validation study by computer simulation

Traditional lot quality assurance sampling (LQAS) methods require simple random sampling to guarantee valid results. However, cluster sampling has been proposed to reduce the number of random starting points. This study uses simulations to examine the classification error of two such designs, a 67x3 (67 clusters of three observations) and a 33x6 (33 clusters of six observations) sampling scheme to assess the prevalence of global acute malnutrition (GAM). Further, we explore the use of a 67x3 sequential sampling scheme for LQAS classification of GAM prevalence. Results indicate that, for independent clusters with moderate intracluster correlation for the GAM outcome, the three sampling designs maintain approximate validity for LQAS analysis. Sequential sampling can substantially reduce the average sample size that is required for data collection. The presence of intercluster correlation can impact dramatically the classification error that is associated with LQAS analysis.

Performance of the lot quality assurance sampling method compared to surveillance for identifying inadequately-performing areas in Matlab, Bangladesh

This paper compared the performance of the lot quality assurance sampling (LQAS) method in identifying inadequately-performing health work-areas with that of using health and demographic surveillance system (HDSS) data and examined the feasibility of applying the method by field-level programme supervisors. The study was carried out in Matlab, the field site of ICDDR,B, where a HDSS has been in place for over 30 years. The LQAS method was applied in 57 work-areas of community health workers in ICDDR,B-served areas in Matlab during July-September 2002. The performance of the LQAS method in identifying work-areas with adequate and inadequate coverage of various health services was compared with those of the HDSS. The health service-coverage indicators included coverage of DPT, measles, BCG vaccination, and contraceptive use. It was observed that the difference in the proportion of work-areas identified to be inadequately performing using the LQAS method with less than 30 respondents, and the HDSS was not statistically significant. The consistency between the LQAS method and the HDSS in identifying work-areas was greater for adequately-performing areas than inadequately-performing areas. It was also observed that the field managers could be trained to apply the LQAS method in monitoring their performance in reaching the target population.