A framework for rigorous evaluation of human performance in human and machine learning comparison studies

Comparative analysis of personal protective equipment nonadherence detection: computer vision versus human observers

OBJECTIVES

Human monitoring of personal protective equipment (PPE) adherence among healthcare providers has several limitations, including the need for additional personnel during staff shortages and decreased vigilance during prolonged tasks. To address these challenges, we developed an automated computer vision system for monitoring PPE adherence in healthcare settings. We assessed the system performance against human observers detecting nonadherence in a video surveillance experiment.

MATERIALS AND METHODS

The automated system was trained to detect 15 classes of eyewear, masks, gloves, and gowns using an object detector and tracker. To assess how the system performs compared to human observers in detecting nonadherence, we designed a video surveillance experiment under 2 conditions: variations in video durations (20, 40, and 60 seconds) and the number of individuals in the videos (3 versus 6). Twelve nurses participated as human observers. Performance was assessed based on the number of detections of nonadherence.

RESULTS

Human observers detected fewer instances of nonadherence than the system (parameter estimate -0.3, 95% CI -0.4 to -0.2, P < .001). Human observers detected more nonadherence during longer video durations (parameter estimate 0.7, 95% CI 0.4-1.0, P < .001). The system achieved a sensitivity of 0.86, specificity of 1, and Matthew's correlation coefficient of 0.82 for detecting PPE nonadherence.

DISCUSSION

An automated system simultaneously tracks multiple objects and individuals. The system performance is also independent of observation duration, an improvement over human monitoring.

CONCLUSION

The automated system presents a potential solution for scalable monitoring of hospital-wide infection control practices and improving PPE usage in healthcare settings.

Comparing Human-Level and Machine Learning Model Performance in White Blood Cell Morphology Assessment

INTRODUCTION

There is an increasing research focus on the role of machine learning in the haematology laboratory, particularly in blood cell morphologic assessment. Human-level performance is an important baseline and goal for machine learning. This study aims to assess the interobserver variability and human-level performance in blood cell morphologic assessment.

METHODS

A dataset of 1000 single white blood cell images were independently labelled by 10 doctors and morphology scientists. Interobserver variability was calculated using Fleiss' kappa. Observers' labels were then separated into consensus labels used to determine ground truth, and performance labels used to assess observer performance. A machine learning model was trained and assessed using the same cell images. Explainability images (XRAI and IG) were generated for each of the test images.

RESULTS

The Fleiss kappa for all 10 observers was 0.608, indicating substantial agreement between observers. The accuracy of human observers was 95%, with sensitivity 72% and specificity 97%. The accuracy of the machine learning model was 95%, with sensitivity 71% and specificity 97%. The model shared similar performance across labels when compared to humans. Explainability metrics demonstrated that the machine learning model was able to differentiate between the cytoplasm and nucleus of the cells, and used these features to perform predictions.

CONCLUSION

The substantial, though not perfect, agreement between human observers highlights the inherent subjectivity in white blood cell morphologic assessment. A machine learning model performed similarly to human observers in single white blood cell identification. Further research is needed to compare human-level and machine learning performance in ways that more closely reflect the typical process of morphologic assessment.

Development of a Machine Learning Model for the Classification of Enterobius vermicularis Egg

Enterobius vermicularis (pinworm) infections are a significant global health issue, affecting children predominantly in environments like schools and daycares. Traditional diagnosis using the scotch tape technique involves examining E. vermicularis eggs under a microscope. This method is time-consuming and depends heavily on the examiner's expertise. To improve this, convolutional neural networks (CNNs) have been used to automate the detection of pinworm eggs from microscopic images. In our study, we enhanced E. vermicularis egg detection using a CNN benchmarked against leading models. We digitized and augmented 40,000 images of E. vermicularis eggs (class 1) and artifacts (class 0) for comprehensive training, using an 80:20 training-validation and a five-fold cross-validation. The proposed CNN model showed limited initial performance but achieved 90.0% accuracy, precision, recall, and F1-score after data augmentation. It also demonstrated improved stability with an ROC-AUC metric increase from 0.77 to 0.97. Despite its smaller file size, our CNN model performed comparably to larger models. Notably, the Xception model achieved 99.0% accuracy, precision, recall, and F1-score. These findings highlight the effectiveness of data augmentation and advanced CNN architectures in improving diagnostic accuracy and efficiency for E. vermicularis infections.

A single fast Hebbian-like process enabling one-shot class addition in deep neural networks without backbone modification

One-shot learning, the ability to learn a new concept from a single instance, is a distinctive brain function that has garnered substantial interest in machine learning. While modeling physiological mechanisms poses challenges, advancements in artificial neural networks have led to performances in specific tasks that rival human capabilities. Proposing one-shot learning methods with these advancements, especially those involving simple mechanisms, not only enhance technological development but also contribute to neuroscience by proposing functionally valid hypotheses. Among the simplest methods for one-shot class addition with deep learning image classifiers is "weight imprinting," which uses neural activity from a new class image data as the corresponding new synaptic weights. Despite its simplicity, its relevance to neuroscience is ambiguous, and it often interferes with original image classification, which is a significant drawback in practical applications. This study introduces a novel interpretation where a part of the weight imprinting process aligns with the Hebbian rule. We show that a single Hebbian-like process enables pre-trained deep learning image classifiers to perform one-shot class addition without any modification to the original classifier's backbone. Using non-parametric normalization to mimic brain's fast Hebbian plasticity significantly reduces the interference observed in previous methods. Our method is one of the simplest and most practical for one-shot class addition tasks, and its reliance on a single fast Hebbian-like process contributes valuable insights to neuroscience hypotheses.

Perception without preconception: comparison between the human and machine learner in recognition of tissues from histological sections

Deep neural networks (DNNs) have shown success in image classification, with high accuracy in recognition of everyday objects. Performance of DNNs has traditionally been measured assuming human accuracy is perfect. In specific problem domains, however, human accuracy is less than perfect and a comparison between humans and machine learning (ML) models can be performed. In recognising everyday objects, humans have the advantage of a lifetime of experience, whereas DNN models are trained only with a limited image dataset. We have tried to compare performance of human learners and two DNN models on an image dataset which is novel to both, i.e. histological images. We thus aim to eliminate the advantage of prior experience that humans have over DNN models in image classification. Ten classes of tissues were randomly selected from the undergraduate first year histology curriculum of a Medical School in North India. Two machine learning (ML) models were developed based on the VGG16 (VML) and Inception V2 (IML) DNNs, using transfer learning, to produce a 10-class classifier. One thousand (1000) images belonging to the ten classes (i.e. 100 images from each class) were split into training (700) and validation (300) sets. After training, the VML and IML model achieved 85.67 and 89% accuracy on the validation set, respectively. The training set was also circulated to medical students (MS) of the college for a week. An online quiz, consisting of a random selection of 100 images from the validation set, was conducted on students (after obtaining informed consent) who volunteered for the study. 66 students participated in the quiz, providing 6557 responses. In addition, we prepared a set of 10 images which belonged to different classes of tissue, not present in training set (i.e. out of training scope or OTS images). A second quiz was conducted on medical students with OTS images, and the ML models were also run on these OTS images. The overall accuracy of MS in the first quiz was 55.14%. The two ML models were also run on the first quiz questionnaire, producing accuracy between 91 and 93%. The ML models scored more than 80% of medical students. Analysis of confusion matrices of both ML models and all medical students showed dissimilar error profiles. However, when comparing the subset of students who achieved similar accuracy as the ML models, the error profile was also similar. Recognition of 'stomach' proved difficult for both humans and ML models. In 04 images in the first quiz set, both VML model and medical students produced highly equivocal responses. Within these images, a pattern of bias was uncovered-the tendency of medical students to misclassify 'liver' tissue. The 'stomach' class proved most difficult for both MS and VML, producing 34.84% of all errors of MS, and 41.17% of all errors of VML model; however, the IML model committed most errors in recognising the 'skin' class (27.5% of all errors). Analysis of the convolution layers of the DNN outlined features in the original image which might have led to misclassification by the VML model. In OTS images, however, the medical students produced better overall score than both ML models, i.e. they successfully recognised patterns of similarity between tissues and could generalise their training to a novel dataset. Our findings suggest that within the scope of training, ML models perform better than 80% medical students with a distinct error profile. However, students who have reached accuracy close to the ML models, tend to replicate the error profile as that of the ML models. This suggests a degree of similarity between how machines and humans extract features from an image. If asked to recognise images outside the scope of training, humans perform better at recognising patterns and likeness between tissues. This suggests that 'training' is not the same as 'learning', and humans can extend their pattern-based learning to different domains outside of the training set.