Integrating Domain Knowledge Into Deep Networks for Lung Ultrasound With Applications to COVID-19

Semi-supervised Ensemble Learning for Automatic Interpretation of Lung Ultrasound Videos

Point-of-care ultrasound (POCUS) stands as a safe, portable, and cost-effective imaging modality for swift bedside patient examinations. Specifically, lung ultrasonography (LUS) has proven useful in evaluating both acute and chronic pulmonary conditions. Despite its clinical value, automatic LUS interpretation remains relatively unexplored, particularly in multi-label contexts. This work proposes a novel deep learning (DL) framework tailored for interpreting lung POCUS videos, whose outputs are the finding(s) present in these videos (such as A-lines, B-lines, or consolidations). The pipeline, based on a residual (2+1)D architecture, initiates with a pre-processing routine for video masking and standardisation, and employs a semi-supervised approach to harness available unlabeled data. Additionally, we introduce an ensemble modeling strategy that aggregates outputs from models trained to predict distinct label sets, thereby leveraging the hierarchical nature of LUS findings. The proposed framework and its building blocks were evaluated through extensive experiments with both multi-class and multi-label models, highlighting its versatility. In a held-out test set, the categorical proposal, suited for expedite triage, achieved an average F1-score of 92.4%, while the multi-label proposal, helpful for patient management and referral, achieved an average F1-score of 70.5% across five relevant LUS findings. Overall, the semi-supervised methodology contributed significantly to improved performance, while the proposed hierarchy-aware ensemble provided moderate additional gains.

Leveraging anatomical constraints with uncertainty for pneumothorax segmentation

Background

Pneumothorax is a medical emergency caused by the abnormal accumulation of air in the pleural space-the potential space between the lungs and chest wall. On 2D chest radiographs, pneumothorax occurs within the thoracic cavity and outside of the mediastinum, and we refer to this area as "lung + space." While deep learning (DL) has increasingly been utilized to segment pneumothorax lesions in chest radiographs, many existing DL models employ an end-to-end approach. These models directly map chest radiographs to clinician-annotated lesion areas, often neglecting the vital domain knowledge that pneumothorax is inherently location-sensitive.

Methods

We propose a novel approach that incorporates the lung + space as a constraint during DL model training for pneumothorax segmentation on 2D chest radiographs. To circumvent the need for additional annotations and to prevent potential label leakage on the target task, our method utilizes external datasets and an auxiliary task of lung segmentation. This approach generates a specific constraint of lung + space for each chest radiograph. Furthermore, we have incorporated a discriminator to eliminate unreliable constraints caused by the domain shift between the auxiliary and target datasets.

Results

Our results demonstrated considerable improvements, with average performance gains of 4.6%, 3.6%, and 3.3% regarding intersection over union, dice similarity coefficient, and Hausdorff distance. These results were consistent across six baseline models built on three architectures (U-Net, LinkNet, or PSPNet) and two backbones (VGG-11 or MobileOne-S0). We further conducted an ablation study to evaluate the contribution of each component in the proposed method and undertook several robustness studies on hyper-parameter selection to validate the stability of our method.

Conclusions

The integration of domain knowledge in DL models for medical applications has often been underemphasized. Our research underscores the significance of incorporating medical domain knowledge about the location-specific nature of pneumothorax to enhance DL-based lesion segmentation and further bolster clinicians' trust in DL tools. Beyond pneumothorax, our approach is promising for other thoracic conditions that possess location-relevant characteristics.

Deep learning approaches for automated classification of neonatal lung ultrasound with assessment of human-to-AI interrater agreement

Neonatal respiratory disorders pose significant challenges in clinical settings, often requiring rapid and accurate diagnostic solutions for effective management. Lung ultrasound (LUS) has emerged as a promising tool to evaluate respiratory conditions in neonates. This evaluation is mainly based on the interpretation of visual patterns (horizontal artifacts, vertical artifacts, and consolidations). Automated interpretation of these patterns can assist clinicians in their evaluations. However, developing AI-based solutions for this purpose is challenging, primarily due to the lack of annotated data and inherent subjectivity in expert interpretations. This study aims to propose an automated solution for the reliable interpretation of patterns in LUS videos of newborns. We employed two distinct strategies. The first strategy is a frame-to-video-level approach that computes frame-level predictions from deep learning (DL) models trained from scratch (F2V-TS) along with fine-tuning pre-trained models (F2V-FT) followed by aggregation of those predictions for video-level evaluation. The second strategy is a direct video classification approach (DV) for evaluating LUS data. To evaluate our methods, we used LUS data from 34 neonatal patients comprising of 70 exams with annotations provided by three expert human operators (3HOs). Results show that within the frame-to-video-level approach, F2V-FT achieved the best performance with an accuracy of 77% showing moderate agreement with the 3HOs. while the direct video classification approach resulted in an accuracy of 72%, showing substantial agreement with the 3HOs, our proposed study lays down the foundation for reliable AI-based solutions for newborn LUS data evaluation.

Active learning based on multi-enhanced views for classification of multiple patterns in lung ultrasound images

There are several main patterns in lung ultrasound (LUS) images, including A-lines, B-lines, consolidation and pleural effusion. LUS images of healthy lungs typically only exhibit A-lines, while other patterns may emerge or coexist in LUS images associated with different lung diseases. The accurate categorization of these primary patterns is pivotal for effective lung disease screening. However, two challenges complicate the classification task: the first is the inherent blurring of feature differences between main patterns due to ultrasound imaging properties; and the second is the potential coexistence of multiple patterns in a single case, with only the most dominant pattern being clinically annotated. To address these challenges, we propose the active learning based on multi-enhanced views (MEVAL) method to achieve more precise pattern classification in LUS. To accentuate feature differences between multiple patterns, we introduce a feature enhancement module by applying vertical linear fitting and k-means clustering. The multi-enhanced views are then employed in parallel with the original images, thus enhancing MEVAL's awareness of feature differences between multiple patterns. To tackle the patterns coexistence issue, we propose an active learning strategy based on confidence sets and misclassified sets. This strategy enables the network to simultaneously recognize multiple patterns by selectively labeling of a small number of images. Our dataset comprises 5075 LUS images, with approximately 4% exhibiting multiple patterns. Experimental results showcase the effectiveness of the proposed method in the classification task, with accuracy of 98.72%, AUC of 0.9989, sensitivity of 98.76%, and specificity of 98.16%, which outperforms than the state-of-the-art deep learning-based methods. A series of comprehensive ablation studies suggest the effectiveness of each proposed component and show great potential in clinical application.

Lung Ultrasound Spectroscopy Applied to the Differential Diagnosis of Pulmonary Diseases: An In Vivo Multicenter Clinical Study

Lung ultrasound (LUS) is an important imaging modality to assess the state of the lung surface. However, current LUS approaches are based on subjective interpretation of imaging artifacts, which results in poor specificity as quantitative evaluation lacks. The latter could be improved by adopting LUS spectroscopy of vertical artifacts. Indeed, parameterizing these artifacts with native frequency, bandwidth, and total intensity ( [Formula: see text]) already showed potentials in differentiating pulmonary fibrosis (PF). In this study, we acquired radio frequency (RF) data from 114 patients. These data (representing the largest LUS RF dataset worldwide) were acquired by utilizing a multifrequency approach, implemented with an ULtrasound Advanced Open Platform (ULA-OP). Convex (CA631) and linear (LA533) probes (Esaote, Florence, Italy) were utilized to acquire RF data at three (2, 3, and 4 MHz), and four (3, 4, 5, and 6 MHz) imaging frequencies. A multifrequency analysis was conducted on vertical artifacts detected in patients having cardiogenic pulmonary edema (CPE), pneumonia, or PF. These artifacts were characterized by the three abovementioned parameters, and their mean values were used to project each patient into a feature space having up to three dimensions. Binary classifiers were used to evaluate the performance of these three mean features in differentiating patients affected by CPE, pneumonia, and PF. Acquisitions of multifrequency data performed with linear probe lead to accuracies up to 85.43% in the differential diagnosis of these diseases (convex probes' maximum accuracy was 74.51%). Moreover, the results showed high potentials of mean [Formula: see text] (by itself or combined with other features) in improving LUS specificity.

FLUEnT: Transformer for detecting lung consolidations in videos using fused lung ultrasound encodings

Pneumonia is the leading cause of death among children around the world. According to WHO, a total of 740,180 lives under the age of five were lost due to pneumonia in 2019. Lung ultrasound (LUS) has been shown to be particularly useful for supporting the diagnosis of pneumonia in children and reducing mortality in resource-limited settings. The wide application of point-of-care ultrasound at the bedside is limited mainly due to a lack of training for data acquisition and interpretation. Artificial Intelligence can serve as a potential tool to automate and improve the LUS data interpretation process, which mainly involves analysis of hyper-echoic horizontal and vertical artifacts, and hypo-echoic small to large consolidations. This paper presents, Fused Lung Ultrasound Encoding-based Transformer (FLUEnT), a novel pediatric LUS video scoring framework for detecting lung consolidations using fused LUS encodings. Frame-level embeddings from a variational autoencoder, features from a spatially attentive ResNet-18, and encoded patient information as metadata combiningly form the fused encodings. These encodings are then passed on to the transformer for binary classification of the presence or absence of consolidations in the video. The video-level analysis using fused encodings resulted in a mean balanced accuracy of 89.3 %, giving an average improvement of 4.7 % points in comparison to when using these encodings individually. In conclusion, outperforming the state-of-the-art models by an average margin of 8 % points, our proposed FLUEnT framework serves as a benchmark for detecting lung consolidations in LUS videos from pediatric pneumonia patients.

Knowledge fused latent representation from lung ultrasound examination for COVID-19 pneumonia severity assessment

COVID-19 pneumonia severity assessment is of great clinical importance, and lung ultrasound (LUS) plays a crucial role in aiding the severity assessment of COVID-19 pneumonia due to its safety and portability. However, its reliance on qualitative and subjective observations by clinicians is a limitation. Moreover, LUS images often exhibit significant heterogeneity, emphasizing the need for more quantitative assessment methods. In this paper, we propose a knowledge fused latent representation framework tailored for COVID-19 pneumonia severity assessment using LUS examinations. The framework transforms the LUS examination into latent representation and extracts knowledge from regions labeled by clinicians to improve accuracy. To fuse the knowledge into the latent representation, we employ a knowledge fusion with latent representation (KFLR) model. This model significantly reduces errors compared to approaches that lack prior knowledge integration. Experimental results demonstrate the effectiveness of our method, achieving high accuracy of 96.4 % and 87.4 % for binary-level and four-level COVID-19 pneumonia severity assessments, respectively. It is worth noting that only a limited number of studies have reported accuracy for clinically valuable exam level assessments, and our method surpass existing methods in this context. These findings highlight the potential of the proposed framework for monitoring disease progression and patient stratification in COVID-19 pneumonia cases.

Subtracting-adding strategy for necrotic lesion segmentation in osteonecrosis of the femoral head

PURPOSE

Osteonecrosis of the femoral head (ONFH) is a severe bone disease that can progressively lead to hip dysfunction. Accurately segmenting the necrotic lesion helps in diagnosing and treating ONFH. This paper aims at enhancing deep learning models for necrosis segmentation.

METHODS

Necrotic lesions of ONFH are confined to the femoral head. Considering this domain knowledge, we introduce a preprocessing procedure, termed the "subtracting-adding" strategy, which explicitly incorporates this domain knowledge into the downstream deep neural network input. This strategy first removes the voxels outside the predefined volume of interest to "subtract" irrelevant information, and then it concatenates the bone mask with raw data to "add" anatomical structure information.

RESULTS

Each of the tested off-the-shelf networks performed better with the help of the "subtracting-adding" strategy. The dice similarity coefficients increased by 10.93%, 9.23%, 9.38% and 1.60% for FCN, HRNet, SegNet and UNet, respectively. The improvements in FCN and HRNet were statistically significant.

CONCLUSIONS

The "subtracting-adding" strategy enhances the performance of general-purpose networks in necrotic lesion segmentation. This strategy is compatible with various semantic segmentation networks, alleviating the need to design task-specific models.

Combining quantitative and qualitative analysis for scoring pleural line in lung ultrasound

Objective.Accurate assessment of pleural line is crucial for the application of lung ultrasound (LUS) in monitoring lung diseases, thereby aim of this study is to develop a quantitative and qualitative analysis method for pleural line.Approach.The novel cascaded deep learning model based on convolution and multilayer perceptron was proposed to locate and segment the pleural line in LUS images, whose results were applied for quantitative analysis of textural and morphological features, respectively. By using gray-level co-occurrence matrix and self-designed statistical methods, eight textural and three morphological features were generated to characterize the pleural lines. Furthermore, the machine learning-based classifiers were employed to qualitatively evaluate the lesion degree of pleural line in LUS images.Main results.We prospectively evaluated 3770 LUS images acquired from 31 pneumonia patients. Experimental results demonstrated that the proposed pleural line extraction and evaluation methods all have good performance, with dice and accuracy of 0.87 and 94.47%, respectively, and the comparison with previous methods found statistical significance (P< 0.001 for all). Meanwhile, the generalization verification proved the feasibility of the proposed method in multiple data scenarios.Significance.The proposed method has great application potential for assessment of pleural line in LUS images and aiding lung disease diagnosis and treatment.

Low-complexity lung ultrasound video scoring by means of intensity projection-based video compression

Since the outbreak of COVID-19, efforts have been made towards semi-quantitative analysis of lung ultrasound (LUS) data to assess the patient's condition. Several methods have been proposed in this regard, with a focus on frame-level analysis, which was then used to assess the condition at the video and prognostic levels. However, no extensive work has been done to analyze lung conditions directly at the video level. This study proposes a novel method for video-level scoring based on compression of LUS video data into a single image and automatic classification to assess patient's condition. The method utilizes maximum, mean, and minimum intensity projection-based compression of LUS video data over time. This enables to preserve hyper- and hypo-echoic data regions, while compressing the video down to a maximum of three images. The resulting images are then classified using a convolutional neural network (CNN). Finally, the worst predicted score given among the images is assigned to the corresponding video. The results show that this compression technique can achieve a promising agreement at the prognostic level (81.62%), while the video-level agreement remains comparable with the state-of-the-art (46.19%). Conclusively, the suggested method lays down the foundation for LUS video compression, shifting from frame-level to direct video-level analysis of LUS data.

Can Artificial Intelligence Aid Diagnosis by Teleguided Point-of-Care Ultrasound? A Pilot Study for Evaluating a Novel Computer Algorithm for COVID-19 Diagnosis Using Lung Ultrasound

With the 2019 coronavirus disease (COVID-19) pandemic, there is an increasing demand for remote monitoring technologies to reduce patient and provider exposure. One field that has an increasing potential is teleguided ultrasound, where telemedicine and point-of-care ultrasound (POCUS) merge to create this new scope. Teleguided POCUS can minimize staff exposure while preserving patient safety and oversight during bedside procedures. In this paper, we propose the use of teleguided POCUS supported by AI technologies for the remote monitoring of COVID-19 patients by non-experienced personnel including self-monitoring by the patients themselves. Our hypothesis is that AI technologies can facilitate the remote monitoring of COVID-19 patients through the utilization of POCUS devices, even when operated by individuals without formal medical training. In pursuit of this goal, we performed a pilot analysis to evaluate the performance of users with different clinical backgrounds using a computer-based system for COVID-19 detection using lung ultrasound. The purpose of the analysis was to emphasize the potential of the proposed AI technology for improving diagnostic performance, especially for users with less experience.

Deep Learning for Detection and Localization of B-Lines in Lung Ultrasound

Lung ultrasound (LUS) is an important imaging modality used by emergency physicians to assess pulmonary congestion at the patient bedside. B-line artifacts in LUS videos are key findings associated with pulmonary congestion. Not only can the interpretation of LUS be challenging for novice operators, but visual quantification of B-lines remains subject to observer variability. In this work, we investigate the strengths and weaknesses of multiple deep learning approaches for automated B-line detection and localization in LUS videos. We curate and publish, BEDLUS, a new ultrasound dataset comprising 1,419 videos from 113 patients with a total of 15,755 expert-annotated B-lines. Based on this dataset, we present a benchmark of established deep learning methods applied to the task of B-line detection. To pave the way for interpretable quantification of B-lines, we propose a novel "single-point" approach to B-line localization using only the point of origin. Our results show that (a) the area under the receiver operating characteristic curve ranges from 0.864 to 0.955 for the benchmarked detection methods, (b) within this range, the best performance is achieved by models that leverage multiple successive frames as input, and (c) the proposed single-point approach for B-line localization reaches an F 1-score of 0.65, performing on par with the inter-observer agreement. The dataset and developed methods can facilitate further biomedical research on automated interpretation of lung ultrasound with the potential to expand the clinical utility.

Benchmark methodological approach for the application of artificial intelligence to lung ultrasound data from COVID-19 patients: From frame to prognostic-level

Automated ultrasound imaging assessment of the effect of CoronaVirus disease 2019 (COVID-19) on lungs has been investigated in various studies using artificial intelligence-based (AI) methods. However, an extensive analysis of state-of-the-art Convolutional Neural Network-based (CNN) models for frame-level scoring, a comparative analysis of aggregation techniques for video-level scoring, together with a thorough evaluation of the capability of these methodologies to provide a clinically valuable prognostic-level score is yet missing within the literature. In addition to that, the impact on the analysis of the posterior probability assigned by the network to the predicted frames as well as the impact of temporal downsampling of LUS data are topics not yet extensively investigated. This paper takes on these challenges by providing a benchmark analysis of methods from frame to prognostic level. For frame-level scoring, state-of-the-art deep learning models are evaluated with additional analysis of best performing model in transfer-learning settings. A novel cross-correlation based aggregation technique is proposed for video and exam-level scoring. Results showed that ResNet-18, when trained from scratch, outperformed the existing methods with an F1-Score of 0.659. The proposed aggregation method resulted in 59.51%, 63.29%, and 84.90% agreement with clinicians at the video, exam, and prognostic levels, respectively; thus, demonstrating improved performances over the state of the art. It was also found that filtering frames based on the posterior probability shows higher impact on the LUS analysis in comparison to temporal downsampling. All of these analysis were conducted over the largest standardized and clinically validated LUS dataset from COVID-19 patients.

New International Guidelines and Consensus on the Use of Lung Ultrasound

Following the innovations and new discoveries of the last 10 years in the field of lung ultrasound (LUS), a multidisciplinary panel of international LUS experts from six countries and from different fields (clinical and technical) reviewed and updated the original international consensus for point-of-care LUS, dated 2012. As a result, a total of 20 statements have been produced. Each statement is complemented by guidelines and future developments proposals. The statements are furthermore classified based on their nature as technical (5), clinical (11), educational (3), and safety (1) statements.

Weakly supervised segmentation of COVID-19 infection with local lesion coherence on CT images

At the end of 2019, a novel coronavirus, COVID-19, was ravaging the world, wreaking havoc on public health and the global economy. Today, although Reverse Transcription-Polymerase Chain Reaction (RT-PCR) is the gold standard for COVID-19 clinical diagnosis, it is a time-consuming and labor-intensive procedure. Simultaneously, an increasing number of individuals are seeking for better alternatives to RT-PCR. As a result, automated identification of COVID-19 lung infection in computed tomography (CT) images may help traditional diagnostic approaches in determining the severity of the disease. Unfortunately, a shortage of labeled training sets makes using AI deep learning algorithms to accurately segregate diseased regions in CT scan challenging. We design a simple and effective weakly supervised learning strategy for COVID-19 CT image segmentation to overcome the segmentation issue in the absence of adequate labeled data, namely LLC-Net. Unlike others weakly supervised work that uses a complex training procedure, our LLC-Net is relatively easy and repeatable. We propose a Local Self-Coherence Mechanism to accomplish label propagation based on lesion area labeling characteristics for weak labels that cannot offer comprehensive lesion areas, hence forecasting a more complete lesion area. Secondly, when the COVID-19 training samples are insufficient, the Scale Transform for Self-Correlation is designed to optimize the robustness of the model to ensure that the CT images are consistent in the prediction results from different angles. Finally, in order to constrain the segmentation accuracy of the lesion area, the Lesion Infection Edge Attention Module is used to improve the information expression ability of edge modeling. Experiments on public datasets demonstrate that our method is more effective than other weakly supervised methods and achieves a new state-of-the-art performance.

Multi-objective automatic analysis of lung ultrasound data from COVID-19 patients by means of deep learning and decision trees

COVID-19 raised the need for automatic medical diagnosis, to increase the physicians' efficiency in managing the pandemic. Among all the techniques for evaluating the status of the lungs of a patient with COVID-19, lung ultrasound (LUS) offers several advantages: portability, cost-effectiveness, safety. Several works approached the automatic detection of LUS imaging patterns related COVID-19 by using deep neural networks (DNNs). However, the decision processes based on DNNs are not fully explainable, which generally results in a lack of trust from physicians. This, in turn, slows down the adoption of such systems. In this work, we use two previously built DNNs as feature extractors at the frame level, and automatically synthesize, by means of an evolutionary algorithm, a decision tree (DT) that aggregates in an interpretable way the predictions made by the DNNs, returning the severity of the patients' conditions according to a LUS score of prognostic value. Our results show that our approach performs comparably or better than previously reported aggregation techniques based on an empiric combination of frame-level predictions made by DNNs. Furthermore, when we analyze the evolved DTs, we discover properties about the DNNs used as feature extractors. We make our data publicly available for further development and reproducibility.

State of the Art in Lung Ultrasound, Shifting from Qualitative to Quantitative Analyses

Lung ultrasound (LUS) has been increasingly expanding since the 1990s, when the clinical relevance of vertical artifacts was first reported. However, the massive spread of LUS is only recent and is associated with the coronavirus disease 2019 (COVID-19) pandemic, during which semi-quantitative computer-aided techniques were proposed to automatically classify LUS data. In this review, we discuss the state of the art in LUS, from semi-quantitative image analysis approaches to quantitative techniques involving the analysis of radiofrequency data. We also discuss recent in vitro and in silico studies, as well as research on LUS safety. Finally, conclusions are drawn highlighting the potential future of LUS.

Automatic deep learning-based consolidation/collapse classification in lung ultrasound images for COVID-19 induced pneumonia

Our automated deep learning-based approach identifies consolidation/collapse in LUS images to aid in the identification of late stages of COVID-19 induced pneumonia, where consolidation/collapse is one of the possible associated pathologies. A common challenge in training such models is that annotating each frame of an ultrasound video requires high labelling effort. This effort in practice becomes prohibitive for large ultrasound datasets. To understand the impact of various degrees of labelling precision, we compare labelling strategies to train fully supervised models (frame-based method, higher labelling effort) and inaccurately supervised models (video-based methods, lower labelling effort), both of which yield binary predictions for LUS videos on a frame-by-frame level. We moreover introduce a novel sampled quaternary method which randomly samples only 10% of the LUS video frames and subsequently assigns (ordinal) categorical labels to all frames in the video based on the fraction of positively annotated samples. This method outperformed the inaccurately supervised video-based method and more surprisingly, the supervised frame-based approach with respect to metrics such as precision-recall area under curve (PR-AUC) and F1 score, despite being a form of inaccurate learning. We argue that our video-based method is more robust with respect to label noise and mitigates overfitting in a manner similar to label smoothing. The algorithm was trained using a ten-fold cross validation, which resulted in a PR-AUC score of 73% and an accuracy of 89%. While the efficacy of our classifier using the sampled quaternary method significantly lowers the labelling effort, it must be verified on a larger consolidation/collapse dataset, our proposed classifier using the sampled quaternary video-based method is clinically comparable with trained experts' performance.

Deep Learning-Based Classification of Reduced Lung Ultrasound Data From COVID-19 Patients

The application of lung ultrasound (LUS) imaging for the diagnosis of lung diseases has recently captured significant interest within the research community. With the ongoing COVID-19 pandemic, many efforts have been made to evaluate LUS data. A four-level scoring system has been introduced to semiquantitatively assess the state of the lung, classifying the patients. Various deep learning (DL) algorithms supported with clinical validations have been proposed to automate the stratification process. However, no work has been done to evaluate the impact on the automated decision by varying pixel resolution and bit depth, leading to the reduction in size of overall data. This article evaluates the performance of DL algorithm over LUS data with varying pixel and gray-level resolution. The algorithm is evaluated over a dataset of 448 LUS videos captured from 34 examinations of 20 patients. All videos are resampled by a factor of 2, 3, and 4 of original resolution, and quantized to 128, 64, and 32 levels, followed by score prediction. The results indicate that the automated scoring shows negligible variation in accuracy when it comes to the quantization of intensity levels only. Combined effect of intensity quantization with spatial down-sampling resulted in a prognostic agreement ranging from 73.5% to 82.3%.These results also suggest that such level of prognostic agreement can be achieved over evaluation of data reduced to 32 times of its original size. Thus, laying foundation to efficient processing of data in resource constrained environments.