Comparative diagnostic accuracy studies with an imperfect reference standard - a comparison of correction methods

The Arabic EAT-10 and FEES in dysphagia screening among cancer patients: a comparative prospective study

Head and neck cancer treatments, such as radiotherapy, chemotherapy, and surgery, have diverse effects on patients, leading to dysphagia as a significant post-treatment issue. This study aims to evaluate the effectiveness of the Arabic version of the EAT-10 screening instrument (A-EAT-10) using Fiber-Optic Endoscopic Evaluation of Swallowing (FEES) as an imperfect gold standard. Additionally, we seek to establish a correlation between A-EAT-10 and PEG tube insertion in head and neck cancer (HNC) patients. Our sample comprised 130 head and neck cancer patients with varying cancer types at King Hussein Cancer Center (KHCC). We followed these patients throughout their distinct treatment plans up to one month after their final treatment session. During follow-up visits, we administered the A-Eat-10 instrument to monitor dysphagia. FEES were conducted at the initial and concluding visits to compare results with A-EAT-10 scores. The results in our tests, assuming independence or dependence, demonstrated excellent agreement. A-EAT-10 exhibited outstanding predictive capabilities with an AUC ranging from 93 to 97%. A-EAT-10 tended to slightly overestimate dysphagia at later treatment stages by approximately 20% compared to FEES, with an RR of 1.2 (95% CI 0.91, 1.56, p-value = 0.21), indicating statistical insignificance. In conclusion, A-EAT-10 is an excellent option for dysphagia evaluation, offering non-invasive, straightforward, and cost-effective advantages compared to FEES. Its utility extends to predicting the need for PEG tube insertion at initial patient visits, making it a valuable tool for informed treatment decisions. Notably, A-EAT-10 demonstrates a diminishing correlation with FEES over time.

Diagnostic accuracy of chest ultrasound scan in the diagnosis of childhood tuberculosis

Chest Ultrasound Scan (CUS) has been utilized in place of CXR in the diagnosis of adult pneumonia with similar or higher sensitivity and specificity to CXR. However, there is a paucity of data on the use of CUS for the diagnosis of childhood TB. This study aimed to determine the diagnostic accuracy of CUS for childhood TB. This cross-sectional study was conducted at the Mulago National Referral Hospital in Uganda. Eighty children up to 14 years of age with presumptive TB were enrolled. They all had CUS and CXR performed and interpreted independently by radiologists. The radiologist who performed the CXR was blinded to the CUS findings, and vice versa. Radiologists noted whether TB was likely or unlikely. A two-by-two table was developed to compare the absolute number of children as either TB likely or TB unlikely on CXR or CUS. This was used to calculate the sensitivity and specificity of CUS when screening for TB in children, with a correction to accommodate the use of CXR as a reference test. The sensitivity of CUS was 64% (95% CI 48.5%-77.3%), while its specificity was 42.7% (95% CI 25.5%-60.8%). Both the CUS and CXR found 29 children with a likelihood of TB, and 27 children unlikely to have TB. CUS met the sensitivity target set by the WHO TPP for Triage, and it had a sensitivity and specificity comparable to that of CXR.

Comparison of the AmpFire® Multiplex HPV Assay to the Xpert® HPV Assay for detection of human papillomavirus and cervical disease in women with human immunodeficiency virus: a pragmatic performance evaluation

BACKGROUND

Low- and middle-income countries (LMICs) account for nearly 85% of the global cervical cancer burden, yet have the least access to high-performance screening. International guidelines recommend human papillomavirus testing (HPV) as primary screening, yet implementation is inhibited by the cost of HPV testing. Atila AmpFire® HPV Assay (AmpFire) is both affordable and easy to use, and offers individual genotyping. The objective of this study was to compare the performance of the AmpFire HPV assay to the Xpert® HPV assay in detection of both HPV and clinically significant cervical disease.

METHODS

We utilized stored cervical specimens from a prospective cohort study of women living with human immunodeficiency virus (HIV) in Botswana conducted from May to July 2018. Positive and negative percent agreement was calculated for the AmpFire and Xpert assays, as was detection of high-grade cervical dysplasia.

RESULTS

63 stored cervical specimens had detectable DNA after thawing and were included in the analysis. The positive percent agreement was 91.2% (95%CI 76.3-98.1) and negative percent agreement was 79.3% (95% CI 60.3-92.0). Six cases positive by AmpFire but negative by Xpert were HPV genotypes 35, 52 (n = 2), 58, 68, and co-infection with HPV 45 and 68. Both Xpert and AmpFire assays detected HPV in all 10 samples of women who had high-grade cervical dysplasia.

CONCLUSIONS

The AmpFire HPV assay demonstrated excellent analytic performance in both detection of HPV and clinically significant cervical disease. AmpFire HPV is a promising option to increase access to affordable, type-specific HPV screening for cervical cancer in LMICs.

Comparison of the AmpFire Multiplex HPV Assay to the Xpert HPV Assay for detection of human papillomavirus and cervical disease in women with human immunodeficiency virus: A pragmatic performance evaluation

Background: Low- and middle-income countries (LMICs) account for nearly 85% of the global cervical cancer burden, yet have the least access to high-performance screening. International guidelines recommend human papillomavirus testing (HPV) as primary screening, yet implementation is inhibited by the cost of HPV testing.Atila AmpFire HPV Assay (AmpFire) is both affordable and easy to use, and offers individual genotyping. The objective of this study was to compare the performance of the AmpFire HPV assay to the Xpert HPV assay in detection of both HPV and clinically significant cervical disease. Methods: We utilized stored cervical specimens from a prospective cohortstudy of women living with human immunodeficiency virus (HIV) in Botswana conducted from May to July 2018. Positive and negative percent agreement was calculated for the AmpFire and Xpert assays, as was detection of high-grade cervical dysplasia. Results : 63 stored cervical specimens haddetectable DNA after thawing and were included in the analysis. The positive percent agreement was 91.2% (95%CI: 76.3-98.1) and negative percent agreement was 79.3% (95% CI: 60.3-92.0). Six cases positive by AmpFire but negative by Xpert were HPV genotypes 35, 52 (n=2), 58, 68, and co-infection with HPV 45 and 68. Both Xpert and AmpFire assays detected HPV in all 10 samples of women who had high-grade cervical dysplasia. Conclusions : The AmpFire HPV assay demonstrated excellent analytic performance in both detection of HPV and clinically significant cervical disease. AmpFire HPV is a promising option to increase access to affordable, type-specific HPV screening for cervical cancer in LMICs.

Effects of Expert-Determined Reference Standards in Evaluating the Diagnostic Performance of a Deep Learning Model: A Malignant Lung Nodule Detection Task on Chest Radiographs

OBJECTIVE

Little is known about the effects of using different expert-determined reference standards when evaluating the performance of deep learning-based automatic detection (DLAD) models and their added value to radiologists. We assessed the concordance of expert-determined standards with a clinical gold standard (herein, pathological confirmation) and the effects of different expert-determined reference standards on the estimates of radiologists' diagnostic performance to detect malignant pulmonary nodules on chest radiographs with and without the assistance of a DLAD model.

MATERIALS AND METHODS

This study included chest radiographs from 50 patients with pathologically proven lung cancer and 50 controls. Five expert-determined standards were constructed using the interpretations of 10 experts: individual judgment by the most experienced expert, majority vote, consensus judgments of two and three experts, and a latent class analysis (LCA) model. In separate reader tests, additional 10 radiologists independently interpreted the radiographs and then assisted with the DLAD model. Their diagnostic performance was estimated using the clinical gold standard and various expert-determined standards as the reference standard, and the results were compared using the t test with Bonferroni correction.

RESULTS

The LCA model (sensitivity, 72.6%; specificity, 100%) was most similar to the clinical gold standard. When expert-determined standards were used, the sensitivities of radiologists and DLAD model alone were overestimated, and their specificities were underestimated (all p-values < 0.05). DLAD assistance diminished the overestimation of sensitivity but exaggerated the underestimation of specificity (all p-values < 0.001). The DLAD model improved sensitivity and specificity to a greater extent when using the clinical gold standard than when using the expert-determined standards (all p-values < 0.001), except for sensitivity with the LCA model (p = 0.094).

CONCLUSION

The LCA model was most similar to the clinical gold standard for malignant pulmonary nodule detection on chest radiographs. Expert-determined standards caused bias in measuring the diagnostic performance of the artificial intelligence model.

Correcting Diagnostic Test Sensitivity and Specificity for Patient Misclassifications Resulting from Use of an Imperfect Reference Standard

Investigational diagnostic tests are validated by using a reference standard (RS). If the RS is imperfect (i.e., it has sensitivity [Se] and/or specificity [Sp] < 1), incorrect values for the investigational test’s Se and Sp may result because of patient misclassification by the RS. Formulas were derived to correct a test’s Se and Sp that were determined by using an imperfect RS. The following derived formulas correct for misclassification and give the true numbers of disease-positive [nDP] and disease-negative patients [nDN] from the apparent number of disease-positive and disease-negative patients (anDP and anDN), and the Se and Sp of the RS (SeR, SpR): nDP = (anDP × SpR + anDN × SpR − anDN)/JR; nDN = (anDP × SeR + anDN × SeR − anDP)/JR, where JR is Youden’s Index for the RS (JR = SeR + SpR − 1). The following derived formulas give the correct Se and Sp of an investigational test (SeI and SpI): SeI = (anTPI × SpR − nDP × SeR × SpR + nDP × JR + nDN × SpR2 − nDN × SpR − SpR × anTNI + anTNI)/(nDP × JR); SpI = (anTPI − anTPI × SeR + nDP × SeR2 − nDP × SeR − SeR × nDN × SpR + nDN × JR + SeR × anTNI)/(nDN × JR), where anTPI is the apparent number of true-positive test results, and anTNI is the apparent number of true-negative test results. The derived formulas correct for patient misclassification by an imperfect RS and give the correct values of a diagnostic test’s Se and Sp.

Estimating Cancer Screening Sensitivity and Specificity Using Healthcare Utilization Data: Defining the Accuracy Assessment Interval

The effectiveness and efficiency of cancer screening in real-world settings depend on many factors, including test sensitivity and specificity. Outside of select experimental studies, not everyone receives a gold standard test that can serve as a comparator in estimating screening test accuracy. Thus, many studies of screening test accuracy use the passage of time to infer whether or not cancer was present at the time of the screening test, particularly for patients with a negative screening test. We define the accuracy assessment interval as the period of time after a screening test that is used to estimate the test's accuracy. We describe how the length of this interval may bias sensitivity and specificity estimates. We call for future research to quantify bias and uncertainty in accuracy estimates and to provide guidance on setting accuracy assessment interval lengths for different cancers and screening modalities.

Multicentre evaluation of two multiplex PCR platforms for the rapid microbiological investigation of nosocomial pneumonia in UK ICUs: the INHALE WP1 study

BACKGROUND

Culture-based microbiological investigation of hospital-acquired or ventilator-associated pneumonia (HAP or VAP) is insensitive, with aetiological agents often unidentified. This can lead to excess antimicrobial treatment of patients with susceptible pathogens, while those with resistant bacteria are treated inadequately for prolonged periods. Using PCR to seek pathogens and their resistance genes directly from clinical samples may improve therapy and stewardship.

METHODS

Surplus routine lower respiratory tract samples were collected from intensive care unit patients about to receive new or changed antibiotics for hospital-onset lower respiratory tract infections at 15 UK hospitals. Testing was performed using the BioFire FilmArray Pneumonia Panel (bioMérieux) and Unyvero Pneumonia Panel (Curetis). Concordance analysis compared machine and routine microbiology results, while Bayesian latent class (BLC) analysis estimated the sensitivity and specificity of each test, incorporating information from both PCR panels and routine microbiology.

FINDINGS

In 652 eligible samples; PCR identified pathogens in considerably more samples compared with routine microbiology: 60.4% and 74.2% for Unyvero and FilmArray respectively vs 44.2% by routine microbiology. PCR tests also detected more pathogens per sample than routine microbiology. For common HAP/VAP pathogens, FilmArray had sensitivity of 91.7%-100.0% and specificity of 87.5%-99.5%; Unyvero had sensitivity of 50.0%-100.0%%, and specificity of 89.4%-99.0%. BLC analysis indicated that, compared with PCR, routine microbiology had low sensitivity, ranging from 27.0% to 69.4%.

INTERPRETATION

Conventional and BLC analysis demonstrated that both platforms performed similarly and were considerably more sensitive than routine microbiology, detecting potential pathogens in patient samples reported as culture negative. The increased sensitivity of detection realised by PCR offers potential for improved antimicrobial prescribing.

Comparison of Immunochromatographic Test (ICT) and Filariasis Test Strip (FTS) for Detecting Lymphatic Filariasis Antigen in American Samoa, 2016

Circulating filarial antigen (Ag) prevalence, measured using rapid point-of-care tests, is the standard indicator used for monitoring and surveillance in the Global Program to Eliminate Lymphatic Filariasis. In 2015, the immunochromatographic test (ICT) was replaced with the filariasis test strip (FTS), which has higher reported sensitivity. Despite differences in sensitivity, no changes in recommended surveillance targets were made when the FTS was introduced. In 2016, we conducted lymphatic filariasis surveys in American Samoa using FTS, which found higher Ag prevalence than previous surveys that used ICT. To determine whether the increase was real, we assessed the concordance between FTS and ICT results by paired testing of heparinised blood from 179 individuals (63% FTS-positive). ICT had 93.8% sensitivity and 100% specificity for identifying FTS-positive persons, and sensitivity was not associated with age, gender, or presence of microfilariae. Based on these findings, if ICT had been used in the 2016 surveys, the results and interpretation would have been similar to those reported using FTS. American Samoa would have failed Transmission Assessment Survey (TAS) of Grade 1 and 2 children with either test, and community prevalence would not have been significantly different (4.1%, 95% CI, 3.3-4.9% with FTS vs. predicted 3.8%, 95%, CI: 3.1-4.6% with ICT).