Diagnostic accuracy of a commercially available, deep learning-based chest X-ray interpretation software for detecting culture-confirmed pulmonary tuberculosis

Breaking the threshold: Developing multivariable models using computer-aided chest X-ray analysis for tuberculosis triage

OBJECTIVES

Computer-aided detection (CAD) software packages quantify tuberculosis (TB)-compatible chest X-ray (CXR) abnormality as continuous scores. In practice, a threshold value is selected for binary CXR classification. We assessed the diagnostic accuracy of an alternative approach to applying CAD for TB triage: incorporating CAD scores in multivariable modeling.

METHODS

We pooled individual patient data from four studies. Separately, for two commercial CAD, we used logistic regression to model microbiologically confirmed TB. Models included CAD score, study site, age, sex, human immunodeficiency virus status, and prior TB. We compared specificity at target sensitivities ≥90% between the multivariable model and the current threshold-based approach for CAD use.

RESULTS

We included 4,733/5,640 (84%) participants with complete covariate data (median age 36 years; 45% female; 22% with prior TB; 22% people living with human immunodeficiency virus). A total of 805 (17%) had TB. Multivariable models demonstrated excellent performance (areas under the receiver operating characteristic curve [95% confidence interval]: software A, 0.91 [0.90-0.93]; software B, 0.92 [0.91-0.93]). Compared with threshold scores, multivariable models increased specificity (e.g., at 90% sensitivity, threshold vs model specificity [95% confidence interval]: software A, 71% [68-74%] vs 75% [74-77%]; software B, 69% [63-75%] vs 75% [74-77%]).

CONCLUSION

Using CAD scores in multivariable models outperformed the current practice of CAD-threshold-based CXR classification for TB diagnosis.

Development and validation of a deep learning model for detecting signs of tuberculosis on chest radiographs among US-bound immigrants and refugees

Immigrants and refugees seeking admission to the United States must first undergo an overseas medical exam, overseen by the US Centers for Disease Control and Prevention (CDC), during which all persons ≥15 years old receive a chest x-ray to look for signs of tuberculosis. Although individual screening sites often implement quality control (QC) programs to ensure radiographs are interpreted correctly, the CDC does not currently have a method for conducting similar QC reviews at scale. We obtained digitized chest radiographs collected as part of the overseas immigration medical exam. Using radiographs from applicants 15 years old and older, we trained deep learning models to perform three tasks: identifying abnormal radiographs; identifying abnormal radiographs suggestive of tuberculosis; and identifying the specific findings (e.g., cavities or infiltrates) in abnormal radiographs. We then evaluated the models on both internal and external testing datasets, focusing on two classes of performance metrics: individual-level metrics, like sensitivity and specificity, and sample-level metrics, like accuracy in predicting the prevalence of abnormal radiographs. A total of 152,012 images (one image per applicant; mean applicant age 39 years) were used for model training. On our internal test dataset, our models performed well both in identifying abnormalities suggestive of TB (area under the curve [AUC] of 0.97; 95% confidence interval [CI]: 0.95, 0.98) and in estimating sample-level counts of the same (-2% absolute percentage error; 95% CIC: -8%, 6%). On the external test datasets, our models performed similarly well in identifying both generic abnormalities (AUCs ranging from 0.89 to 0.92) and those suggestive of TB (AUCs from 0.94 to 0.99). This performance was consistent across metrics, including those based on thresholded class predictions, like sensitivity, specificity, and F1 score. Strong performance relative to high-quality radiological reference standards across a variety of datasets suggests our models may make reliable tools for supporting chest radiography QC activities at CDC.

Early detection of tuberculosis: a systematic review

Tuberculosis remains a significant global health challenge. Tuberculosis affects millions of individuals worldwide. Early detection of tuberculosis plays a relevant role in the management of treatment of tuberculosis. This systematic review will analyze the findings of several published studies on the topic of the early detection of tuberculosis. This systematic review highlights their methodologies and limitations as well as their contributions to our understanding of this pressing issue. Early detection of tuberculosis can be achieved through tuberculosis screening for contacts. Comprehensive health education for household contacts can be used as early detection. The in-house deep learning models can be used in the X-ray used for automatic detection of tuberculosis. Interferon gamma release assay, routine passive and active case detection, portable X-ray and nucleic acid amplification testing, and highly sensitive enzyme-linked immunosorbent assay tests play critical roles in improving tuberculosis detection.

Head-to-head comparison of diagnostic accuracy of TB screening tests: Chest-X-ray, Xpert TB host response, and C-reactive protein

Background

Accessible, accurate screening tests are necessary to advance tuberculosis (TB) case finding and early detection in high-burden countries. We compared the diagnostic accuracy of available TB triage tests.

Methods

We prospectively screened consecutive adults with ≥2 weeks of cough presenting to primary health centers in the Philippines, Vietnam, South Africa, Uganda, and India. All participants received the index tests: chest-X-ray (CXR), venous or capillary Cepheid Xpert TB Host Response (HR) testing, and point-of-care C-reactive protein (CRP) testing (Boditech iChroma II). CXR images were processed using computer-aided detection (CAD) algorithms. We assessed diagnostic accuracy against a microbiologic reference standard (sputum Xpert Ultra, culture). Optimal cut-points were chosen to achieve sensitivity ≥90% and maximize specificity. Two-test screening algorithms were considered, using two approaches: 1) sequential negative serial screening in which the second screening test is conducted only if the first is negative and positive is defined as positive on either test and 2) sequential positive serial screening, in which the second screening test is conducted only if the first is positive and positive is defined as positive on both tests.

Results

Between July 2021 and August 2022, 1,392 participants with presumptive TB had valid results on index tests and the reference standard, and 303 (22%) had confirmed TB. In head-to-head comparisons, CAD4TB v7 showed the highest specificity when using a cut-point that achieves 90% sensitivity (70.3% vs. 65.1% for Xpert HR, difference 95% CI 1.6 to 8.9; 49.7% for CRP, difference 95% CI 17.0 to 24.3). Among the possible two-test screening algorithms, three met WHO target product profile (TPP) minimum accuracy thresholds and had higher accuracy than any test alone. At 90% sensitivity, the specificity was 79.6% for Xpert HR-CAD4TB [sequential negative], 75.9% for CRP-CAD4TB [sequential negative], and 73.7% for Xpert HR-CAD4TB [sequential positive].

Conclusions

CAD4TB achieves TPP targets and outperforms Xpert HR and CRP. Combining screening tests further increased accuracy. Cost and feasibility of two-test screening algorithms should be explored.

Registration

NCT04923958.

AI for Detection of Tuberculosis: Implications for Global Health

Tuberculosis, which primarily affects developing countries, remains a significant global health concern. Since the 2010s, the role of chest radiography has expanded in tuberculosis triage and screening beyond its traditional complementary role in the diagnosis of tuberculosis. Computer-aided diagnosis (CAD) systems for tuberculosis detection on chest radiographs have recently made substantial progress in diagnostic performance, thanks to deep learning technologies. The current performance of CAD systems for tuberculosis has approximated that of human experts, presenting a potential solution to the shortage of human readers to interpret chest radiographs in low- or middle-income, high-tuberculosis-burden countries. This article provides a critical appraisal of developmental process reporting in extant CAD software for tuberculosis, based on the Checklist for Artificial Intelligence in Medical Imaging. It also explores several considerations to scale up CAD solutions, encompassing manufacturer-independent CAD validation, economic and political aspects, and ethical concerns, as well as the potential for broadening radiography-based diagnosis to other nontuberculosis diseases. Collectively, CAD for tuberculosis will emerge as a representative deep learning application, catalyzing advances in global health and health equity. Keywords: Computer-aided Diagnosis (CAD), Conventional Radiography, Thorax, Lung, Machine Learning Supplemental material is available for this article. © RSNA, 2024.

Computer-aided diagnostic accuracy of pulmonary tuberculosis on chest radiography among lower respiratory tract symptoms patients

Even though the Gaza Strip is a low pulmonary tuberculosis (TB) burden region, it is well-known that TB is primarily a socioeconomic problem associated with overcrowding, poor hygiene, a lack of fresh water, and limited access to healthcare, which is the typical case in the Gaza Strip. Therefore, this study aimed at assessing the accuracy of the automatic software computer-aided detection for tuberculosis (CAD4TB) in diagnosing pulmonary TB on chest radiography and compare the CAD4TB software reading with the results of geneXpert. Using a census sampling method, the study was conducted in radiology departments in the Gaza Strip hospitals between 1 December 2022 and 31 March 2023. A digital X-ray, printer, and online X-ray system backed by CAD4TBv6 software were used to screen patients with lower respiratory tract symptoms. GeneXpert analysis was performed for all patients having a score > 40. A total of 1,237 patients presenting with lower respiratory tract symptoms participated in this current study. Chest X-ray readings showed that 7.8% (n = 96) were presumptive for TB. The CAD4TBv6 scores showed that 11.8% (n = 146) of recruited patients were presumptive for TB. GeneXpert testing on sputum samples showed that 6.2% (n = 77) of those with a score > 40 on CAD4TB were positive for pulmonary TB. Significant differences were found in chest X-ray readings, CAD4TBv6 scores, and GeneXpert results among sociodemographic and health status variables (P-value < 0.05). The study showed that the incidence rate of TB in the Gaza Strip is 3.5 per 100,000 population in the Gaza strip. The sensitivity of the CAD4TBv6 score and the symptomatic review for tuberculosis with a threshold score of >40 is 80.2%, and the specificity is 94.0%. The positive Likelihood Ratio is 13.3%, Negative Likelihood Ratio is 0.2 with 7.8% prevalence. Positive Predictive Value is 52.7%, Negative Predictive Value is 98.3%, and accuracy is 92.9%. In a resource-limited country with a high burden of neglected disease, combining chest X-ray readings by CAD4TB and symptomatology is extremely valuable for screening a population at risk. CAD4TB is noticeably more efficient than other methods for TB screening and early diagnosis in people who would otherwise go undetected.

Benchmarking the diagnostic test accuracy of certified AI products for screening pulmonary tuberculosis in digital chest radiographs: Preliminary evidence from a rapid review and meta-analysis

BACKGROUND AND OBJECTIVE

The global market for AI systems used in lung tuberculosis (TB) detection has expanded significantly in recent years. Verifying their performance across diverse settings is crucial before medical organisations can invest in them and pursue safe, wide-scale deployment. The goal of this research was to synthesise the clinical evidence for the diagnostic accuracy of certified AI products designed for screening TB in chest X-rays (CXRs) compared to a microbiological reference standard.

METHODS

Four databases were searched between June to September 2022. Data concerning study methodology, system characteristics, and diagnostic accuracy metrics was extracted and summarised. Study bias was evaluated using QUADAS-2 and by examining sources of funding. Forest plots for diagnostic odds ratio (DOR) and summary receiver operating characteristic (SROC) curves were constructed for the AI products individually and collectively.

RESULTS

10 out of 3642 studies satisfied the review criteria however only 8 were subject to meta-analysis following bias assessment. Three AI products were evaluated with a 95 % confidence interval producing the following pooled estimates for accuracy rankings: qXR v2 (sensitivity of 0.944 [0.887-0.973], specificity of 0.692 [0.549-0.805], DOR of 3.63 [3.17-4.09], Lunit INSIGHT CXR v3.1 (sensitivity of 0.853 [0.787-0.901], specificity of 0.646 [0.627-0.665], DOR of 2.37 [1.96-2.78]), and CAD4TB v3.07 (sensitivity of 0.917 [0.848-0.956], specificity of 0.371 [0.336-0.408], DOR of 1.91 [1.4-2.47]). Overall, the products had a sensitivity of 0.903 (0.859-0.934), specificity of 0.526 (0.409-0.641), and DOR of 2.31 (1.78-2.84).

CONCLUSION

Current publicly available evidence indicates considerable variability in the diagnostic accuracy of available AI products although overall they have high sensitivity and modest specificity which is improving with time. These preliminary results are limited by the small number of studies and poor coverage for low TB burden settings. More research is needed to expand the clinical evidence base for the performance of AI products.

A prospective multicenter clinical research study validating the effectiveness and safety of a chest X-ray-based pulmonary tuberculosis screening software JF CXR-1 built on a convolutional neural network algorithm

Background

Chest radiography (chest X-ray or CXR) plays an important role in the early detection of active pulmonary tuberculosis (TB). In areas with a high TB burden that require urgent screening, there is often a shortage of radiologists available to interpret the X-ray results. Computer-aided detection (CAD) software employed with artificial intelligence (AI) systems may have the potential to solve this problem.

Objective

We validated the effectiveness and safety of pulmonary tuberculosis imaging screening software that is based on a convolutional neural network algorithm.

Methods

We conducted prospective multicenter clinical research to validate the performance of pulmonary tuberculosis imaging screening software (JF CXR-1). Volunteers under the age of 15 years, both with or without suspicion of pulmonary tuberculosis, were recruited for CXR photography. The software reported a probability score of TB for each participant. The results were compared with those reported by radiologists. We measured sensitivity, specificity, consistency rate, and the area under the receiver operating characteristic curves (AUC) for the diagnosis of tuberculosis. Besides, adverse events (AE) and severe adverse events (SAE) were also evaluated.

Results

The clinical research was conducted in six general infectious disease hospitals across China. A total of 1,165 participants were enrolled, and 1,161 were enrolled in the full analysis set (FAS). Men accounted for 60.0% (697/1,161). Compared to the results from radiologists on the board, the software showed a sensitivity of 94.2% (95% CI: 92.0-95.8%) and a specificity of 91.2% (95% CI: 88.5-93.2%). The consistency rate was 92.7% (91.1-94.1%), with a Kappa value of 0.854 (P = 0.000). The AUC was 0.98. In the safety set (SS), which consisted of 1,161 participants, 0.3% (3/1,161) had AEs that were not related to the software, and no severe AEs were observed.

Conclusion

The software for tuberculosis screening based on a convolutional neural network algorithm is effective and safe. It is a potential candidate for solving tuberculosis screening problems in areas lacking radiologists with a high TB burden.

Advances in development of new tuberculosis vaccines

PURPOSE OF REVIEW

Tuberculosis (TB) remains a global public health emergency and caused 1.6 million deaths in 2021. The aim of this review is to provide recent updates on advances in TB vaccine development for prevention and adjunct therapy.

RECENT FINDINGS

Targets use indications guiding late stage TB vaccine development have been established, namely: (i) Prevention of disease (PoD), (ii) Prevention of recurrent disease (PoR), (iii) Prevention of established infection in previously uninfected patients (PoI), and (iv) Adjunctive immunotherapy. Novel approaches include vaccines designed to induce immune responses beyond established CD4+, Th1-biased T cell immunity, novel animal models for use in challenge/protection studies, and controlled human infection models to generate vaccine efficacy data.

SUMMARY

Recent efforts at developing effective TB vaccines for prevention and adjunct treatment utilising new targets and technologies have yielded 16 candidate vaccines demonstrating proof of concept for inducing potentially protective immune responses to TB which is currently under evaluation in different stages of clinical trials.

Diagnostic Accuracy of the Artificial Intelligence Methods in Medical Imaging for Pulmonary Tuberculosis: A Systematic Review and Meta-Analysis

Tuberculosis (TB) remains one of the leading causes of death among infectious diseases worldwide. Early screening and diagnosis of pulmonary tuberculosis (PTB) is crucial in TB control, and tend to benefit from artificial intelligence. Here, we aimed to evaluate the diagnostic efficacy of a variety of artificial intelligence methods in medical imaging for PTB. We searched MEDLINE and Embase with the OVID platform to identify trials published update to November 2022 that evaluated the effectiveness of artificial-intelligence-based software in medical imaging of patients with PTB. After data extraction, the quality of studies was assessed using quality assessment of diagnostic accuracy studies 2 (QUADAS-2). Pooled sensitivity and specificity were estimated using a bivariate random-effects model. In total, 3987 references were initially identified and 61 studies were finally included, covering a wide range of 124,959 individuals. The pooled sensitivity and the specificity were 91% (95% confidence interval (CI), 89-93%) and 65% (54-75%), respectively, in clinical trials, and 94% (89-96%) and 95% (91-97%), respectively, in model-development studies. These findings have demonstrated that artificial-intelligence-based software could serve as an accurate tool to diagnose PTB in medical imaging. However, standardized reporting guidance regarding AI-specific trials and multicenter clinical trials is urgently needed to truly transform this cutting-edge technology into clinical practice.