Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

Identifying and Minimizing Incentives for Competing Interests in Sports Medicine Publications

Academics in sports medicine as well as other medical fields are generally expected to publish research and opinions in peer-reviewed journals. The peer-review process is intended to protect against the publication of flawed research and unsubstantiated claims. However, both financial and non-financial competing interests may result in sub-optimal results by affecting investigators, editors, peer reviewers, academic institutions, and publishers. In this article, we focus on the non-financial competing interests created in our current academic system. Because these competing interests are embedded in our current scholastic framework, the potential biases are difficult to quantify. To minimize the effect of these competing interests, we review and highlight some underlying incentives for each stakeholder and some potential solutions to mitigate their effects.

Identifying patients with undiagnosed small intestinal neuroendocrine tumours in primary care using statistical and machine learning: model development and validation study

BACKGROUND

Neuroendocrine tumours (NETs) are increasing in incidence, often diagnosed at advanced stages, and individuals may experience years of diagnostic delay, particularly when arising from the small intestine (SI). Clinical prediction models could present novel opportunities for case finding in primary care.

METHODS

An open cohort of adults (18+ years) contributing data to the Optimum Patient Care Research Database between 1st Jan 2000 and 30th March 2023 was identified. This database collects de-identified data from general practices in the UK. Model development approaches comprised logistic regression, penalised regression, and XGBoost. Performance (discrimination and calibration) was assessed using internal-external cross-validation. Decision analysis curves compared clinical utility.

RESULTS

Of 11.7 million individuals, 382 had recorded SI NET diagnoses (0.003%). The XGBoost model had the highest AUC (0.869, 95% confidence interval [CI]: 0.841-0.898) but was mildly miscalibrated (slope 1.165, 95% CI: 1.088-1.243; calibration-in-the-large 0.010, 95% CI: -0.164 to 0.185). Clinical utility was similar across all models.

DISCUSSION

Multivariable prediction models may have clinical utility in identifying individuals with undiagnosed SI NETs using information in their primary care records. Further evaluation including external validation and health economics modelling may identify cost-effective strategies for case finding for this uncommon tumour.

PET/CT deep learning prognosis for treatment decision support in esophageal squamous cell carcinoma

OBJECTIVES

The clinical decision-making regarding choosing surgery alone (SA) or surgery followed by postoperative adjuvant chemotherapy (SPOCT) in esophageal squamous cell carcinoma (ESCC) remains controversial. We aim to propose a pre-therapy PET/CT image-based deep learning approach to improve the survival benefit and clinical management of ESCC patients.

METHODS

This retrospective multicenter study included 837 ESCC patients from three institutions. Prognostic biomarkers integrating six networks were developed to build an ESCC prognosis (ESCCPro) model and predict the survival probability of ESCC patients treated with SA and SPOCT. Patients who did not undergo surgical resection were in a control group. Overall survival (OS) was the primary end-point event. The expected improvement in survival prognosis with the application of ESCCPro to assign treatment protocols was estimated by comparing the survival of patients in each subgroup. Seven clinicians with varying experience evaluated how ESCCPro performed in assisting clinical decision-making.

RESULTS

In this retrospective multicenter study, patients receiving SA had a median OS 9.2 months longer than controls. No significant differences in survival were found between SA patients with predicted poor outcomes and the controls (p > 0.05). It was estimated that if ESCCPro was used to determine SA and SPOCT eligibility, the median OS in the ESCCPro-recommended SA group and SPOCT group would have been 15.3 months and 24.9 months longer, respectively. In addition, ESCCPro also significantly improved prognosis accuracy, certainty, and the efficiency of clinical experts.

CONCLUSION

ESCCPro assistance improved the survival benefit of ESCC patients and the clinical decision-making among the two treatment approaches.

CRITICAL RELEVANCE STATEMENT

The ESCCPro model for treatment decision-making is promising to improve overall survival in ESCC patients undergoing surgical resection and patients undergoing surgery followed by postoperative adjuvant chemotherapy.

KEY POINTS

ESCC is associated with a poor prognosis and unclear ideal treatments. ESCCPro predicts the survival of patients with ESCC and the expected benefit from SA. ESCCPro improves clinicians' stratification of patients' prognoses.

Artificial Intelligence in Orthodontics: Critical Review

With increasing digitalization in orthodontics, certain orthodontic manufacturing processes such as the fabrication of indirect bonding trays, aligner production, or wire bending can be automated. However, orthodontic treatment planning and evaluation remains a specialist's task and responsibility. As the prediction of growth in orthodontic patients and response to orthodontic treatment is inherently complex and individual, orthodontists make use of features gathered from longitudinal, multimodal, and standardized orthodontic data sets. Currently, these data sets are used by the orthodontist to make informed, rule-based treatment decisions. In research, artificial intelligence (AI) has been successfully applied to assist orthodontists with the extraction of relevant data from such data sets. Here, AI has been applied for the analysis of clinical imagery, such as automated landmark detection in lateral cephalograms but also for evaluation of intraoral scans or photographic data. Furthermore, AI is applied to help orthodontists with decision support for treatment decisions such as the need for orthognathic surgery or for orthodontic tooth extractions. One major challenge in current AI research in orthodontics is the limited generalizability, as most studies use unicentric data with high risks of bias. Moreover, comparing AI across different studies and tasks is virtually impossible as both outcomes and outcome metrics vary widely, and underlying data sets are not standardized. Notably, only few AI applications in orthodontics have reached full clinical maturity and regulatory approval, and researchers in the field are tasked with tackling real-world evaluation and implementation of AI into the orthodontic workflow.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique.

STUDY DESIGN AND SETTING

We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

A systematic review of cardiac surgery clinical prediction models that include intra-operative variables

BACKGROUND

Most cardiac surgery clinical prediction models (CPMs) are developed using pre-operative variables to predict post-operative outcomes. Some CPMs are developed with intra-operative variables, but none are widely used. The objective of this systematic review was to identify CPMs with intra-operative variables that predict short-term outcomes following adult cardiac surgery.

METHODS

Ovid MEDLINE and EMBASE databases were searched from inception to December 2022, for studies developing a CPM with at least one intra-operative variable. Data were extracted using a critical appraisal framework and bias assessment tool. Model performance was analysed using discrimination and calibration measures.

RESULTS

A total of 24 models were identified. Frequent predicted outcomes were acute kidney injury (9/24 studies) and peri-operative mortality (6/24 studies). Frequent pre-operative variables were age (18/24 studies) and creatinine/eGFR (18/24 studies). Common intra-operative variables were cardiopulmonary bypass time (16/24 studies) and transfusion (13/24 studies). Model discrimination was acceptable for all internally validated models (AUC 0.69-0.91). Calibration was poor (15/24 studies) or unreported (8/24 studies). Most CPMs were at a high or indeterminate risk of bias (23/24 models). The added value of intra-operative variables was assessed in six studies with statistically significantly improved discrimination demonstrated in two.

CONCLUSION

Weak reporting and methodological limitations may restrict wider applicability and adoption of existing CPMs that include intra-operative variables. There is some evidence that CPM discrimination is improved with the addition of intra-operative variables. Further work is required to understand the role of intra-operative CPMs in the management of cardiac surgery patients.

Critical appraisal of machine learning prognostic models for acute pancreatitis: protocol for a systematic review

Acute pancreatitis (AP) is an acute inflammatory disorder that is common, costly, and is increasing in incidence worldwide with over 300,000 hospitalizations occurring yearly in the United States alone. As its course and outcomes vary widely, a critical knowledge gap in the field has been a lack of accurate prognostic tools to forecast AP patients' outcomes. Despite several published studies in the last three decades, the predictive performance of published prognostic models has been found to be suboptimal. Recently, non-regression machine learning models (ML) have garnered intense interest in medicine for their potential for better predictive performance. Each year, an increasing number of AP models are being published. However, their methodologic quality relating to transparent reporting and risk of bias in study design has never been systematically appraised. Therefore, through collaboration between a group of clinicians and data scientists with appropriate content expertise, we will perform a systematic review of papers published between January 2021 and December 2023 containing artificial intelligence prognostic models in AP. To systematically assess these studies, the authors will leverage the CHARMS checklist, PROBAST tool for risk of bias assessment, and the most current version of the TRIPOD-AI. (Research Registry ( http://www.reviewregistry1727 .).

The application and use of artificial intelligence in cancer nursing: A systematic review

PURPOSE

Artificial Intelligence is being applied in oncology to improve patient and service outcomes. Yet, there is a limited understanding of how these advanced computational techniques are employed in cancer nursing to inform clinical practice. This review aimed to identify and synthesise evidence on artificial intelligence in cancer nursing.

METHODS

CINAHL, MEDLINE, PsycINFO, and PubMed were searched using key terms between January 2010 and December 2022. Titles, abstracts, and then full texts were screened against eligibility criteria, resulting in twenty studies being included. Critical appraisal was undertaken, and relevant data extracted and analysed. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed.

RESULTS

Artificial intelligence was used in numerous areas including breast, colorectal, liver, and ovarian cancer care among others. Algorithms were trained and tested on primary and secondary datasets to build predictive models of health problems related to cancer. Studies reported this led to improvements in the accuracy of predicting health outcomes or identifying variables that improved outcome prediction. While nurses led most studies, few deployed an artificial intelligence based digital tool with cancer nurses in a real-world setting as studies largely focused on developing and validating predictive models.

CONCLUSION

Electronic cancer nursing datasets should be established to enable artificial intelligence techniques to be tested and if effective implemented in digital prediction and other AI-based tools. Cancer nurses need more education on machine learning and natural language processing, so they can lead and contribute to artificial intelligence developments in oncology.

Leveraging machine learning to distinguish between bacterial and viral induced pharyngitis using hematological markers: a retrospective cohort study

Accurate differentiation between bacterial and viral-induced pharyngitis is recognized as essential for personalized treatment and judicious antibiotic use. From a cohort of 693 patients with pharyngitis, data from 197 individuals clearly diagnosed with bacterial or viral infections were meticulously analyzed in this study. By integrating detailed hematological insights with several machine learning algorithms, including Random Forest, Neural Networks, Decision Trees, Support Vector Machine, Naive Bayes, and Lasso Regression, for potential biomarkers were identified, with an emphasis being placed on the diagnostic significance of the Monocyte-to-Lymphocyte Ratio. Distinct inflammatory signatures associated with bacterial infections were spotlighted in this study. An innovation introduced in this research was the adaptation of the high-accuracy Lasso Regression model for the TI-84 calculator, with an AUC (95% CI) of 0.94 (0.925-0.955) being achieved. Using this adaptation, pivotal laboratory parameters can be input on-the-spot and infection probabilities can be computed subsequently. This methodology embodies an improvement in diagnostics, facilitating more effective distinction between bacterial and viral infections while fostering judicious antibiotic use.