Avoid Oversimplifications in Machine Learning: Going beyond the Class-Prediction Accuracy

Development and validation of radiomics models for the prediction of diagnosis of classic trigeminal neuralgia

The study aims to develop a magnetic resonance imaging (MRI)-based radiomics model for the diagnosis of classic trigeminal neuralgia (cTN). This study involved 350 patients with cTN and 100 control participants. MRI data were collected retrospectively for all the enrolled subjects. The symptomatic side trigeminal nerve regions of patients and both sides of the trigeminal nerve regions of control participants were manually labeled on MRI images. Radiomics features of the areas labeled were extracted. Principle component analysis (PCA) and least absolute shrinkage and selection operator (LASSO) regression were utilized as the preliminary feature reduction methods to decrease the high dimensionality of radiomics features. Machine learning methods were established, including LASSO logistic regression, support vector machine (SVM), and Adaboost methods, evaluating each model's diagnostic abilities using 10-fold cross-validation. All the models showed excellent diagnostic ability in predicting trigeminal neuralgia. A prospective study was conducted, 20 cTN patients and 20 control subjects were enrolled to validate the clinical utility of all models. Results showed that the radiomics models based on MRI can predict trigeminal neuralgia with high accuracy, which could be used as a diagnostic tool for this disorder.

Systems Biology and Omics Approaches for Complex Human Diseases

For many years, there has been general interest in developing virtual cells or digital twin models [...].

Application of artificial intelligence for overall survival risk stratification in oropharyngeal carcinoma: A validation of ProgTOOL

BACKGROUND

In recent years, there has been a surge in machine learning-based models for diagnosis and prognostication of outcomes in oncology. However, there are concerns relating to the model's reproducibility and generalizability to a separate patient cohort (i.e., external validation).

OBJECTIVES

This study primarily provides a validation study for a recently introduced and publicly available machine learning (ML) web-based prognostic tool (ProgTOOL) for overall survival risk stratification of oropharyngeal squamous cell carcinoma (OPSCC). Additionally, we reviewed the published studies that have utilized ML for outcome prognostication in OPSCC to examine how many of these models were externally validated, type of external validation, characteristics of the external dataset, and diagnostic performance characteristics on the internal validation (IV) and external validation (EV) datasets were extracted and compared.

METHODS

We used a total of 163 OPSCC patients obtained from the Helsinki University Hospital to externally validate the ProgTOOL for generalizability. In addition, PubMed, OvidMedline, Scopus, and Web of Science databases were systematically searched according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

RESULTS

The ProgTOOL produced a predictive performance of 86.5% balanced accuracy, Mathew's correlation coefficient of 0.78, Net Benefit (0.7) and Brier score (0.06) for overall survival stratification of OPSCC patients as either low-chance or high-chance. In addition, out of a total of 31 studies found to have used ML for the prognostication of outcomes in OPSCC, only seven (22.6%) reported a form of EV. Three studies (42.9%) each used either temporal EV or geographical EV while only one study (14.2%) used expert as a form of EV. Most of the studies reported a reduction in performance when externally validated.

CONCLUSION

The performance of the model in this validation study indicates that it may be generalized, therefore, bringing recommendations of the model for clinical evaluation closer to reality. However, the number of externally validated ML-based models for OPSCC is still relatively small. This significantly limits the transfer of these models for clinical evaluation and subsequently reduces the likelihood of the use of these models in daily clinical practice. As a gold standard, we recommend the use of geographical EV and validation studies to reveal biases and overfitting of these models. These recommendations are poised to facilitate the implementation of these models in clinical practice.

Deep learning neural network derivation and testing to distinguish acute poisonings

INTRODUCTION

Acute poisoning is a significant global health burden, and the causative agent is often unclear. The primary aim of this pilot study was to develop a deep learning algorithm that predicts the most probable agent a poisoned patient was exposed to from a pre-specified list of drugs.

RESEARCH DESIGN & METHODS

Data were queried from the National Poison Data System (NPDS) from 2014 through 2018 for eight single-agent poisonings (acetaminophen, diphenhydramine, aspirin, calcium channel blockers, sulfonylureas, benzodiazepines, bupropion, and lithium). Two Deep Neural Networks (PyTorch and Keras) designed for multi-class classification tasks were applied.

RESULTS

There were 201,031 single-agent poisonings included in the analysis. For distinguishing among selected poisonings, PyTorch model had specificity of 97%, accuracy of 83%, precision of 83%, recall of 83%, and a F1-score of 82%. Keras had specificity of 98%, accuracy of 83%, precision of 84%, recall of 83%, and a F1-score of 83%. The best performance was achieved in the diagnosis of single-agent poisoning in diagnosing poisoning by lithium, sulfonylureas, diphenhydramine, calcium channel blockers, then acetaminophen, in PyTorch (F1-score = 99%, 94%, 85%, 83%, and 82%, respectively) and Keras (F1-score = 99%, 94%, 86%, 82%, and 82%, respectively).

CONCLUSION

Deep neural networks can potentially help in distinguishing the causative agent of acute poisoning. This study used a small list of drugs, with polysubstance ingestions excluded.Reproducible source code and results can be obtained at https://github.com/ashiskb/npds-workspace.git.

Role of brain 2-[18F]fluoro-2-deoxy-D-glucose-positron-emission tomography as survival predictor in amyotrophic lateral sclerosis

PURPOSE

The identification of prognostic tools in amyotrophic lateral sclerosis (ALS) would improve the design of clinical trials, the management of patients, and life planning. We aimed to evaluate the accuracy of brain 2-[¹⁸F]fluoro-2-deoxy-D-glucose-positron-emission tomography (2-[¹⁸F]FDG-PET) as an independent predictor of survival in ALS.

METHODS

A prospective cohort study enrolled 418 ALS patients, who underwent brain 2-[¹⁸F]FDG-PET at diagnosis and whose survival time was available. We discretized the survival time in a finite number of classes in a data-driven fashion by employing a k-means-like strategy. We identified "hot brain regions" with maximal power in discriminating survival classes, by evaluating the Laplacian scores in a class-aware fashion. We retained the top-m features for each class to train the classification systems (i.e., a support vector machine, SVM), using 10% of the ALS cohort as test set.

RESULTS

Data were discretized in three survival profiles: 0-2 years, 2-5 years, and > 5 years. SVM resulted in an error rate < 20% for two out of three classes separately. As for class one, the discriminant clusters included left caudate body and anterior cingulate cortex. The most discriminant regions were bilateral cerebellar pyramid in class two, and right cerebellar dentate nucleus, and left cerebellar nodule in class three.

CONCLUSION

Brain 2-[¹⁸F]FDG-PET along with artificial intelligence was able to predict with high accuracy the survival time range in our ALS cohort. Healthcare professionals can benefit from this prognostic tool for planning patients' management and follow-up. 2-[¹⁸F]FDG-PET represents a promising biomarker for individual patients' stratification in clinical trials. The lack of a multicentre external validation of the model warrants further studies to evaluate its generalization capability.

Awareness: An empirical model

In this work, we face the time-honored problem of the contraposition/integration of analytical and intuitive knowledge, and the impact of such interconnection on the onset of awareness resulting from human decision-making processes. Borrowing the definitions of concepts like intuition, tacit knowledge, uncertainty, metacognition, and emotions from the philosophical, psychological, decision theory, and economic points of view, we propose a skeletonized mathematical model grounded on Markov Decision Processes of these multifaceted concepts. Behavioral patterns that emerged from the solutions of the model enabled us to understand some relevant properties of the interaction between explicit (mainly analytical) and implicit (mainly holistic) knowledge. The impact of the roles played by the same factors for both styles of reasoning and different stages of the decision process has been evaluated. We have found that awareness emerges as a dynamic process allowing the decision-maker to switch from habitual to optimal behavior, resulting from a feedback mechanism of self-observation. Furthermore, emotions are embedded in the model as inner factors, possibly fostering the onset of awareness.

Artificial Intelligence in Orthopedic Radiography Analysis: A Narrative Review

Artificial intelligence (AI) in medicine is a rapidly growing field. In orthopedics, the clinical implementations of AI have not yet reached their full potential. Deep learning algorithms have shown promising results in computed radiographs for fracture detection, classification of OA, bone age, as well as automated measurements of the lower extremities. Studies investigating the performance of AI compared to trained human readers often show equal or better results, although human validation is indispensable at the current standards. The objective of this narrative review is to give an overview of AI in medicine and summarize the current applications of AI in orthopedic radiography imaging. Due to the different AI software and study design, it is difficult to find a clear structure in this field. To produce more homogeneous studies, open-source access to AI software codes and a consensus on study design should be aimed for.

Doppelgänger Spotting in Biomedical Gene Expression Data

Doppelgänger effects (DEs) occur when samples exhibit chance similarities such that, when split across training and validation sets, inflates the trained machine learning (ML) model performance. This inflationary effect causes misleading confidence on the deployability of the model. Thus, so far, there are no tools for doppelgänger identification or standard practices to manage their confounding implications. We present doppelgangerIdentifier, a software suite for doppelgänger identification and verification. Applying doppelgangerIdentifier across a multitude of diseases and data types, we show the pervasive nature of DEs in biomedical gene expression data. We also provide guidelines toward proper doppelgänger identification by exploring the ramifications of lingering batch effects from batch imbalances on the sensitivity of our doppelgänger identification algorithm. We suggest doppelgänger verification as a useful procedure to establish baselines for model evaluation that may inform on whether feature selection and ML on the data set may yield meaningful insights.

•

Doppelgänger effects inflate the machine learning performance

•

Doppelgänger effects exist in RNA-Seq and microarray gene expression data

•

Developed doppelgangerIdentifier, a software to identify and verify doppelgängers

•

Provide guidelines for proper doppelgänger identification

Detecting Unusual Intravenous Infusion Alerting Patterns with Machine Learning Algorithms

OBJECTIVE

To detect unusual infusion alerting patterns using machine learning (ML) algorithms as a first step to advance safer inpatient intravenous administration of high-alert medications.

MATERIALS AND METHODS

We used one year of detailed propofol infusion data from a hospital. Interpretable and clinically relevant variables were feature engineered, and data points were aggregated per calendar day. A univariate (maximum times-limit) moving range (mr) control chart was used to simulate clinicians' common approach to identifying unusual infusion alerting patterns. Three different unsupervised multivariate ML-based anomaly detection algorithms (Local Outlier Factor, Isolation Forest, and k-Nearest Neighbors) were used for the same purpose. Results from the control chart and ML algorithms were compared.

RESULTS

The propofol data had 3,300 infusion alerts, 92% of which were generated during the day shift and seven of which had a times-limit greater than 10. The mr-chart identified 15 alert pattern anomalies. Different thresholds were set to include the top 15 anomalies from each ML algorithm. A total of 31 unique ML anomalies were grouped and ranked by agreeability. All algorithms agreed on 10% of the anomalies, and at least two algorithms agreed on 36%. Each algorithm detected one specific anomaly that the mr-chart did not detect. The anomaly represented a day with 71 propofol alerts (half of which were overridden) generated at an average rate of 1.06 per infusion, whereas the moving alert rate for the week was 0.35 per infusion.

DISCUSSION

These findings show that ML-based algorithms are more robust than control charts in detecting unusual alerting patterns. However, we recommend using a combination of algorithms, as multiple algorithms serve a benchmarking function and allow researchers to focus on data points with the highest algorithm agreeability.

CONCLUSION

Unsupervised ML algorithms can assist clinicians in identifying unusual alert patterns as a first step toward achieving safer infusion practices.

Detecting Unusual Intravenous Infusion Alerting Patterns with Machine Learning Algorithms

OBJECTIVE

To detect unusual infusion alerting patterns using machine learning (ML) algorithms as a first step to advance safer inpatient intravenous administration of high-alert medications.

MATERIALS AND METHODS

We used one year of detailed propofol infusion data from a hospital. Interpretable and clinically relevant variables were feature engineered, and data points were aggregated per calendar day. A univariate (maximum times-limit) moving range (mr) control chart was used to simulate clinicians' common approach to identifying unusual infusion alerting patterns. Three different unsupervised multivariate ML-based anomaly detection algorithms (Local Outlier Factor, Isolation Forest, and k-Nearest Neighbors) were used for the same purpose. Results from the control chart and ML algorithms were compared.

RESULTS

The propofol data had 3,300 infusion alerts, 92% of which were generated during the day shift and seven of which had a times-limit greater than 10. The mr-chart identified 15 alert pattern anomalies. Different thresholds were set to include the top 15 anomalies from each ML algorithm. A total of 31 unique ML anomalies were grouped and ranked by agreeability. All algorithms agreed on 10% of the anomalies, and at least two algorithms agreed on 36%. Each algorithm detected one specific anomaly that the mr-chart did not detect. The anomaly represented a day with 71 propofol alerts (half of which were overridden) generated at an average rate of 1.06 per infusion, whereas the moving alert rate for the week was 0.35 per infusion.

DISCUSSION

These findings show that ML-based algorithms are more robust than control charts in detecting unusual alerting patterns. However, we recommend using a combination of algorithms, as multiple algorithms serve a benchmarking function and allow researchers to focus on data points with the highest algorithm agreeability.

CONCLUSION

Unsupervised ML algorithms can assist clinicians in identifying unusual alert patterns as a first step toward achieving safer infusion practices.

Systems serology detects functionally distinct coronavirus antibody features in children and elderly

The hallmarks of COVID-19 are higher pathogenicity and mortality in the elderly compared to children. Examining baseline SARS-CoV-2 cross-reactive immunological responses, induced by circulating human coronaviruses (hCoVs), is needed to understand such divergent clinical outcomes. Here we show analysis of coronavirus antibody responses of pre-pandemic healthy children (n = 89), adults (n = 98), elderly (n = 57), and COVID-19 patients (n = 50) by systems serology. Moderate levels of cross-reactive, but non-neutralizing, SARS-CoV-2 antibodies are detected in pre-pandemic healthy individuals. SARS-CoV-2 antigen-specific Fcγ receptor binding accurately distinguishes COVID-19 patients from healthy individuals, suggesting that SARS-CoV-2 infection induces qualitative changes to antibody Fc, enhancing Fcγ receptor engagement. Higher cross-reactive SARS-CoV-2 IgA and IgG are observed in healthy elderly, while healthy children display elevated SARS-CoV-2 IgM, suggesting that children have fewer hCoV exposures, resulting in less-experienced but more polyreactive humoral immunity. Age-dependent analysis of COVID-19 patients, confirms elevated class-switched antibodies in elderly, while children have stronger Fc responses which we demonstrate are functionally different. These insights will inform COVID-19 vaccination strategies, improved serological diagnostics and therapeutics.

Extensions of the External Validation for Checking Learned Model Interpretability and Generalizability

We discuss the validation of machine learning models, which is standard practice in determining model efficacy and generalizability. We argue that internal validation approaches, such as cross-validation and bootstrap, cannot guarantee the quality of a machine learning model due to potentially biased training data and the complexity of the validation procedure itself. For better evaluating the generalization ability of a learned model, we suggest leveraging on external data sources from elsewhere as validation datasets, namely external validation. Due to the lack of research attractions on external validation, especially a well-structured and comprehensive study, we discuss the necessity for external validation and propose two extensions of the external validation approach that may help reveal the true domain-relevant model from a candidate set. Moreover, we also suggest a procedure to check whether a set of validation datasets is valid and introduce statistical reference points for detecting external data problems.

AI Naturalists Might Hold the Key to Unlocking Biodiversity Data in Social Media Imagery

The increasing availability of digital images, coupled with sophisticated artificial intelligence (AI) techniques for image classification, presents an exciting opportunity for biodiversity researchers to create new datasets of species observations. We investigated whether an AI plant species classifier could extract previously unexploited biodiversity data from social media photos (Flickr). We found over 60,000 geolocated images tagged with the keyword "flower" across an urban and rural location in the UK and classified these using AI, reviewing these identifications and assessing the representativeness of images. Images were predominantly biodiversity focused, showing single species. Non-native garden plants dominated, particularly in the urban setting. The AI classifier performed best when photos were focused on single native species in wild situations but also performed well at higher taxonomic levels (genus and family), even when images substantially deviated from this. We present a checklist of questions that should be considered when undertaking a similar analysis.