Interrater and intrarater agreement on heart murmurs

OBJECTIVE

To investigate interrater and intrarater agreement between physicians and medical students on heart sound classification from audio recordings, and factors predicting agreement with a reference classification.

DESIGN

Intra- and interrater agreement study.

SUBJECTS

Seventeen GPs and eight cardiologists from Norway and the Netherlands, eight medical students from Norway.

MAIN OUTCOME MEASURES

Proportion of agreement and kappa coefficients for intrarater agreement and agreement with a reference classification.

RESULTS

The proportion of intrarater agreement on the presence of any murmur was 83% on average, with a median kappa of 0.64 (range k = 0.09-0.86) for all raters, and 0.65, 0.69, and 0.61 for GPs, cardiologist, and medical students, respectively.The proportion of agreement with the reference on any murmur was 81% on average, with a median kappa of 0.67 (range 0.29-0.90) for all raters, and 0.65, 0.69, and 0.51 for GPs, cardiologists, and medical students, respectively.Distinct murmur, more than five years of clinical practice, and cardiology specialty were most strongly associated with the agreement, with ORs of 2.41 (95% CI 1.63-3.58), 2.19 (1.58-3.04), and 2.53 (1.46-4.41), respectively.

CONCLUSION

We observed fair but variable agreement with a reference on heart murmurs, and physician experience and specialty, as well as murmur intensity, were the factors most strongly associated with agreement.Key points:Heart auscultation is the main physical examination of the heart, but we lack knowledge of inter- and intrarater agreement on heart sounds.• Physicians identified heart murmurs from heart sound recordings fairly reliably compared with a reference classification, and with fair intrarater agreement.• Both intrarater agreement and agreement with the reference showed considerable variation between doctors• Murmur intensity, more than five years in clinical practice, and cardiology specialty were most strongly linked to agreement with the reference.

Prevalence and clinical associations of wheezes and crackles in the general population: the Tromsø study

BACKGROUND

Wheezes and crackles are well-known signs of lung diseases, but can also be heard in apparently healthy adults. However, their prevalence in a general population has been sparsely described. The objective of this study was to determine the prevalence of wheezes and crackles in a large general adult population and explore associations with self-reported disease, smoking status and lung function.

METHODS

We recorded lung sounds in 4033 individuals 40 years or older and collected information on self-reported disease. Pulse oximetry and spirometry were carried out. We estimated age-standardized prevalence of wheezes and crackles and associations between wheezes and crackles and variables of interest were analyzed with univariable and multivariable logistic regressions.

RESULTS

Twenty-eight percent of individuals had wheezes or crackles. The age-standardized prevalence of wheezes was 18.6% in women and 15.3% in men, and of crackles, 10.8 and 9.4%, respectively. Wheezes were mostly found during expiration and crackles during inspiration. Significant predictors of expiratory wheezes in multivariable analyses were age (10 years increase - OR 1.18, 95%CI 1.09-1.30), female gender (1.45, 1.2-1.8), self-reported asthma (1.36, 1.00-1.83), and current smoking (1.70, 1.28-2.23). The most important predictors of inspiratory crackles were age (1.76, 1.57-1.99), current smoking, (1.94, 1.40-2.69), mMRC ≥2 (1.79, 1.18-2.65), SpO₂ (0.88, 0.81-0.96), and FEV₁ Z-score (0.86, 0.77-0.95).

CONCLUSIONS

Nearly over a quarter of adults present adventitious lung sounds on auscultation. Age was the most important predictor of adventitious sounds, particularly crackles. The adventitious sounds were also associated with self-reported disease, current smoking and measures of lung function. The presence of findings in two or more auscultation sites was associated with a higher risk of decreased lung function than solitary findings.

Convolutional Neural Network for Breathing Phase Detection in Lung Sounds

We applied deep learning to create an algorithm for breathing phase detection in lung sound recordings, and we compared the breathing phases detected by the algorithm and manually annotated by two experienced lung sound researchers. Our algorithm uses a convolutional neural network with spectrograms as the features, removing the need to specify features explicitly. We trained and evaluated the algorithm using three subsets that are larger than previously seen in the literature. We evaluated the performance of the method using two methods. First, discrete count of agreed breathing phases (using 50% overlap between a pair of boxes), shows a mean agreement with lung sound experts of 97% for inspiration and 87% for expiration. Second, the fraction of time of agreement (in seconds) gives higher pseudo-kappa values for inspiration (0.73–0.88) than expiration (0.63–0.84), showing an average sensitivity of 97% and an average specificity of 84%. With both evaluation methods, the agreement between the annotators and the algorithm shows human level performance for the algorithm. The developed algorithm is valid for detecting breathing phases in lung sound recordings.

Adventitious and Normal Lung Sounds in the General Population: Comparison of Standardized and Spontaneous Breathing

BACKGROUND

For clinical practice and research, it would be easier to auscultate lung sounds without simultaneously measuring air flow. This study evaluated whether the presence of adventitious lung sounds and the characteristics of normal lung sounds differ between spontaneous and standardized breathing in a general population.

METHODS

A cross-sectional study was conducted with 116 subjects (53.4% female, mean age 59.2 ± 11.6 y). The subjects reported heart/lung diseases and the degree of dyspnea, and spirometry was carried out. Lung sounds were recorded at 6 chest locations, first during spontaneous breathing and then during breathing with a standardized air flow of 1.5 L/s. Crackles and wheezes were identified by 4 observers. Intensity and frequency of normal lung sounds in the 100-2,000 Hz band were determined.

RESULTS

Inspiratory crackles were heard in 19 subjects (16.4%) during spontaneous breathing and in 18 subjects during standardized breathing (15.5%). Only 5 subjects were identified with both methods (kappa = 0.13). Expiratory wheezes were heard in 18 subjects (15.5%) during spontaneous breathing and in 23 subjects during standardized breathing (19.8%). Nine subjects were identified with both methods (kappa = 0.32). The mean intensity and median frequency of normal lung sounds were significantly higher during standardized breathing than during spontaneous breathing, both at inspiration (23.1 dB vs 20.1 dB and 391.6 Hz vs 367.3 Hz) and expiration (20 dB vs17.6 dB and 376.3 Hz vs 355 Hz). Dyspnea was more frequently reported when expiratory wheezes were present, but this association was only statistically significant during standardized breathing (P = .03). During spontaneous breathing, increased mean intensity and median frequency during expiration were associated with an increased reporting of heart/lung diseases (P = .02 and P = .01, respectively).

CONCLUSIONS

The mode of breathing had an impact on both adventitious and normal lung sounds. Although adventitious sounds were found with similar frequency between the modes of breathing, less than half of these subjects were identified with both methods. Spontaneous breathing was not inferior to standardized breathing in reflecting lung disease.

International perception of lung sounds: a comparison of classification across some European borders

Introduction

Lung auscultation is helpful in the diagnosis of lung and heart diseases; however, the diagnostic value of lung sounds may be questioned due to interobserver variation. This situation may also impair clinical research in this area to generate evidence-based knowledge about the role that chest auscultation has in a modern clinical setting. The recording and visual display of lung sounds is a method that is both repeatable and feasible to use in large samples, and the aim of this study was to evaluate interobserver agreement using this method.

Methods

With a microphone in a stethoscope tube, we collected digital recordings of lung sounds from six sites on the chest surface in 20 subjects aged 40 years or older with and without lung and heart diseases. A total of 120 recordings and their spectrograms were independently classified by 28 observers from seven different countries. We employed absolute agreement and kappa coefficients to explore interobserver agreement in classifying crackles and wheezes within and between subgroups of four observers.

Results

When evaluating agreement on crackles (inspiratory or expiratory) in each subgroup, observers agreed on between 65% and 87% of the cases. Conger's kappa ranged from 0.20 to 0.58 and four out of seven groups reached a kappa of ≥0.49. In the classification of wheezes, we observed a probability of agreement between 69% and 99.6% and kappa values from 0.09 to 0.97. Four out of seven groups reached a kappa ≥0.62.

Conclusions

The kappa values we observed in our study ranged widely but, when addressing its limitations, we find the method of recording and presenting lung sounds with spectrograms sufficient for both clinic and research. Standardisation of terminology across countries would improve international communication on lung auscultation findings.

Combining multivariate statistics and the think-aloud protocol to assess Human-Computer Interaction barriers in symptom checkers

Symptom checkers are software tools that allow users to submit a set of symptoms and receive advice related to them in the form of a diagnosis list, health information or triage. The heterogeneity of their potential users and the number of different components in their user interfaces can make testing with end-users unaffordable. We designed and executed a two-phase method to test the respiratory diseases module of the symptom checker Erdusyk. Phase I consisted of an online test with a large sample of users (n=53). In Phase I, users evaluated the system remotely and completed a questionnaire based on the Technology Acceptance Model. Principal Component Analysis was used to correlate each section of the interface with the questionnaire responses, thus identifying which areas of the user interface presented significant contributions to the technology acceptance. In the second phase, the think-aloud procedure was executed with a small number of samples (n=15), focusing on the areas with significant contributions to analyze the reasons for such contributions. Our method was used effectively to optimize the testing of symptom checker user interfaces. The method allowed kept the cost of testing at reasonable levels by restricting the use of the think-aloud procedure while still assuring a high amount of coverage. The main barriers detected in Erdusyk were related to problems understanding time repetition patterns, the selection of levels in scales to record intensities, navigation, the quantification of some symptom attributes, and the characteristics of the symptoms.