Testing the performance, adequacy, and applicability of an artificial intelligence model for pediatric pneumonia diagnosis

BACKGROUND

Community-acquired Pneumonia (CAP) is a common childhood infectious disease. Deep learning models show promise in X-ray interpretation and diagnosis, but their validation should be extended due to limitations in the current validation workflow. To extend the standard validation workflow we propose doing a pilot test with the next characteristics. First, the assumption of perfect ground truth (100% sensitive and specific) is unrealistic, as high intra and inter-observer variability have been reported. To address this, we propose using Bayesian latent class models (BLCA) to estimate accuracy during the pilot. Additionally, assessing only the performance of a model without considering its applicability and acceptance by physicians is insufficient if we hope to integrate AI systems into day-to-day clinical practice. Therefore, we propose employing explainable artificial intelligence (XAI) methods during the pilot test to involve physicians and evaluate how well a Deep Learning model is accepted and how helpful it is for routine decisions as well as analyze its limitations by assessing the etiology. This study aims to apply the proposed pilot to test a deep Convolutional Neural Network (CNN)-based model for identifying consolidation in pediatric chest-X-ray (CXR) images already validated using the standard workflow.

METHODS

For the standard validation workflow, a total of 5856 public CXRs and 950 private CXRs were used to train and validate the performance of the CNN model. The performance of the model was estimated assuming a perfect ground truth. For the pilot test proposed in this article, a total of 190 pediatric chest-X-ray (CXRs) images were used to test the CNN model support decision tool (SDT). The performance of the model on the pilot test was estimated using extensions of the two-test Bayesian Latent-Class model (BLCA). The sensitivity, specificity, and accuracy of the model were also assessed. The clinical characteristics of the patients were compared according to the model performance. The adequacy and applicability of the SDT was tested using XAI techniques. The adequacy of the SDT was assessed by asking two senior physicians the agreement rate with the SDT. The applicability was tested by asking three medical residents before and after using the SDT and the agreement between experts was calculated using the kappa index.

RESULTS

The CRXs of the pilot test were labeled by the panel of experts into consolidation (124/176, 70.4%) and no-consolidation/other infiltrates (52/176, 29.5%). A total of 31/176 (17.6%) discrepancies were found between the model and the panel of experts with a kappa index of 0.6. The sensitivity and specificity reached a median of 90.9 (95% Credible Interval (CrI), 81.2-99.9) and 77.7 (95% CrI, 63.3-98.1), respectively. The senior physicians reported a high agreement rate (70%) with the system in identifying logical consolidation patterns. The three medical residents reached a higher agreement using SDT than alone with experts (0.66±0.1 vs. 0.75±0.2).

CONCLUSIONS

Through the pilot test, we have successfully verified that the deep learning model was underestimated when a perfect ground truth was considered. Furthermore, by conducting adequacy and applicability tests, we can ensure that the model is able to identify logical patterns within the CXRs and that augmenting clinicians with automated preliminary read assistants could accelerate their workflows and enhance accuracy in identifying consolidation in pediatric CXR images.

Oral saliva swab reverse transcription PCR for Covid-19 in the paediatric population

OBJECTIVES

To evaluate the performance of oral saliva swab (OSS) reverse transcription PCR (RT-PCR) compared with RT-PCR and antigen rapid diagnostic test (Ag-RDT) on nasopharyngeal swabs (NPS) for SARS-CoV-2 in children.

DESIGN

Cross-sectional multicentre diagnostic study.

SETTING

Study nested in a prospective, observational cohort (EPICO-AEP) performed between February and March 2021 including 10 hospitals in Spain.

PATIENTS

Children from 0 to 18 years with symptoms compatible with Covid-19 of ≤5 days of duration were included. Two NPS samples (Ag-RDT and RT-PCR) and one OSS sample for RT-PCR were collected.

MAIN OUTCOME

Performance of Ag-RDT and RT-PCR on NPS and RT-PCR on OSS sample for SARS-CoV-2.

RESULTS

1174 children were included, aged 3.8 years (IQR 1.7-9.0); 73/1174 (6.2%) patients tested positive by at least one of the techniques. Sensitivity and specificity of OSS RT-PCR were 72.1% (95% CI 59.7 to 81.9) and 99.6% (95% CI 99 to 99.9), respectively, versus 61.8% (95% CI 49.1 to 73) and 99.9% (95% CI 99.4 to 100) for the Ag-RDT. Kappa index was 0.79 (95% CI 0.72 to 0.88) for OSS RT-PCR and 0.74 (95% CI 0.65 to 0.84) for Ag-RDT versus NPS RT-PCR.

CONCLUSIONS

RT-PCR on the OSS sample is an accurate option for SARS-CoV-2 testing in children. A less intrusive technique for younger patients, who usually are tested frequently, might increase the number of patients tested.

Comparison of pneumonia features in children caused by SARS-CoV-2 and other viral respiratory pathogens

BACKGROUND

Pneumonia is a frequent manifestation of coronavirus disease 2019 (COVID-19) in hospitalized children.

METHODS

The study involved 80 hospitals in the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) Spanish Pediatric National Cohort. Participants were children <18 years, hospitalized with SARS-CoV-2 community-acquired pneumonia (CAP). We compared the clinical and radiological characteristics of SARS-CoV-2-associated CAP with CAP due to other viral etiologies from ValsDance (retrospective) cohort.

RESULTS

In total, 151 children with SARS-CoV-2-associated CAP and 138 with other viral CAP were included. Main clinical features of SARS-CoV-2-associated CAP were cough, fever, or dyspnea. Lymphopenia was found in 43% patients and 15% required admission to the pediatric intensive care unit (PICU). Chest X-ray revealed condensation (42%) and other infiltrates (58%). Compared with CAP from other viral pathogens, COVID-19 patients were older, with lower C-reactive protein (CRP) levels, less wheezing, and greater need of mechanical ventilation (MV). There were no differences in the use of continuous positive airway pressure (CPAP) or HVF, or PICU admission between groups.

CONCLUSION

SARS-CoV-2-associated CAP in children presents differently to other virus-associated CAP: children are older and rarely have wheezing or high CRP levels; they need less oxygen but more CPAP or MV. However, several features overlap and differentiating the etiology may be difficult. The overall prognosis is good.

Antibiotic Prescribing in Children Hospitalized With COVID-19 and Multisystem Inflammatory Syndrome in Spain: Prevalence, Trends, and Associated Factors

The SARS-CoV-2 pandemic has caused an increase in antibiotic use in different settings. We describe the antibiotic prescribing prevalence, associated factors and trends, as well as concomitant bacterial infections in children hospitalized with COVID-19 or multisystemic inflammatory syndrome related to SARS-CoV-2 in Spain.

Clinical spectrum of COVID-19 and risk factors associated with severity in Spanish children

We aimed to identify the spectrum of disease in children with COVID-19, and the risk factors for admission in paediatric intensive care units (PICUs). We conducted a multicentre, prospective study of children with SARS-CoV-2 infection in 76 Spanish hospitals. We included children with COVID-19 or multi-inflammatory syndrome (MIS-C) younger than 18 years old, attended during the first year of the pandemic. We enrolled 1200 children. A total of 666 (55.5%) were hospitalised, and 123 (18.4%) required admission to PICU. Most frequent major clinical syndromes in the cohort were mild syndrome (including upper respiratory tract infection and flu-like syndrome, skin or mucosae problems and asymptomatic), 44.8%; bronchopulmonary syndrome (including pneumonia, bronchitis and asthma flare), 18.5%; fever without a source, 16.2%; MIS-C, 10.6%; and gastrointestinal syndrome, 10%. In hospitalised children, the proportions were 28.5%, 25.7%, 16.5%, 19.1% and 10.2%, respectively. Risk factors associated with PICU admission were age in months (OR: 1.007; 95% CI 1.004 to 1.01), MIS-C (OR: 14.4, 95% CI 8.9 to 23.8), chronic cardiac disease (OR: 4.8, 95% CI 1.8 to 13), asthma or recurrent wheezing (OR: 2.5, 95% CI 1.2 to 5.2) and after excluding MIS-C patients, moderate/severe liver disease (OR: 8.6, 95% CI 1.6 to 47.6). However, asthmatic children were admitted into the PICU due to MIS-C or pneumonia, not due to asthma flare.Conclusion: Hospitalised children with COVID-19 usually present as one of five major clinical phenotypes of decreasing severity. Risk factors for PICU include MIS-C, elevation of inflammation biomarkers, asthma, moderate or severe liver disease and cardiac disease. What is Known: • All studies suggest that children are less susceptible to serious SARS-CoV-2 infection when compared to adults. Most studies describe symptoms at presentation. However, it remains unclear how these symptoms group together into clinically identifiable syndromes and the severity associated with them. What is New: • We have gathered the primary diagnoses into five major syndromes of decreasing severity: MIS-C, bronchopulmonary syndrome, gastrointestinal syndrome, fever without a source and mild syndrome. Classification of the children in one of the syndromes is unique and helps to assess the risk of critical illness and to define the spectrum of the disease instead of just describing symptoms and signs.