The uncertainty with using risk prediction models for individual decision making: an exemplar cohort study examining the prediction of cardiovascular disease in English primary care

Current status of artificial intelligence methods for skin cancer survival analysis: a scoping review

Skin cancer mortality rates continue to rise, and survival analysis is increasingly needed to understand who is at risk and what interventions improve outcomes. However, current statistical methods are limited by inability to synthesize multiple data types, such as patient genetics, clinical history, demographics, and pathology and reveal significant multimodal relationships through predictive algorithms. Advances in computing power and data science enabled the rise of artificial intelligence (AI), which synthesizes vast amounts of data and applies algorithms that enable personalized diagnostic approaches. Here, we analyze AI methods used in skin cancer survival analysis, focusing on supervised learning, unsupervised learning, deep learning, and natural language processing. We illustrate strengths and weaknesses of these approaches with examples. Our PubMed search yielded 14 publications meeting inclusion criteria for this scoping review. Most publications focused on melanoma, particularly histopathologic interpretation with deep learning. Such concentration on a single type of skin cancer amid increasing focus on deep learning highlight growing areas for innovation; however, it also demonstrates opportunity for additional analysis that addresses other types of cutaneous malignancies and expands the scope of prognostication to combine both genetic, histopathologic, and clinical data. Moreover, researchers may leverage multiple AI methods for enhanced benefit in analyses. Expanding AI to this arena may enable improved survival analysis, targeted treatments, and outcomes.

Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

BACKGROUND

Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.

METHODS

This retrospective cohort study used 5909 patients recruited from 1999 to 2012 for model development, and 3199 patients recruited from 2012 to 2015 for model validation. Patients were recruited at oncology referral or general centers and underwent an ultrasound examination and surgery ≤ 120 days later. We developed models using standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM). We used nine clinical and ultrasound predictors but developed models with or without CA125.

RESULTS

Most tumors were benign (3980 in development and 1688 in validation data), secondary metastatic tumors were least common (246 and 172). The c-statistic (AUROC) to discriminate benign from any type of malignant tumor ranged from 0.89 to 0.92 for models with CA125, from 0.89 to 0.91 for models without. The multiclass c-statistic ranged from 0.41 (SVM) to 0.55 (XGBoost) for models with CA125, and from 0.42 (SVM) to 0.51 (standard MLR) for models without. Multiclass calibration was best for RF and XGBoost. Estimated probabilities for a benign tumor in the same patient often differed by more than 0.2 (20% points) depending on the model. Net Benefit for diagnosing malignancy was similar for algorithms at the commonly used 10% risk threshold, but was slightly higher for RF at higher thresholds. Comparing models, between 3% (XGBoost vs. NN, with CA125) and 30% (NN vs. SVM, without CA125) of patients fell on opposite sides of the 10% threshold.

CONCLUSION

Although several models had similarly good performance, individual probability estimates varied substantially.

Understanding the risk of developing weight-related complications associated with different body mass index categories: a systematic review

BACKGROUND

Obesity and overweight are major risk factors for several chronic diseases. There is limited systematic evaluation of risk equations that predict the likelihood of developing an obesity or overweight associated complication. Predicting future risk is essential for health economic modelling. Availability of future treatments rests upon a model's ability to inform clinical and decision-making bodies.  This systematic literature review aimed to identify studies reporting (1) equations that calculate the risk for individuals with obesity, or overweight with a weight-related complication (OWRC), of developing additional complications, namely T2D, cardiovascular (CV) disease (CVD), acute coronary syndrome, stroke, musculoskeletal disorders, knee replacement/arthroplasty, or obstructive sleep apnea; (2) absolute or proportional risk for individuals with severe obesity, obesity or OWRC developing T2D, a CV event or mortality from knee surgery, stroke, or an acute CV event.

METHODS

Databases (MEDLINE and Embase) were searched for English language reports of population-based cohort analyses or large-scale studies in Australia, Canada, Europe, the UK, and the USA between January 1, 2011, and March 29, 2021. Included reports were quality assessed using an adapted version of the Newcastle Ottawa Scale.

RESULTS

Of the 60 included studies, the majority used European cohorts. Twenty-nine reported a risk prediction equation for developing an additional complication. The most common risk prediction equations were logistic regression models that did not differentiate between body mass index (BMI) groups (particularly above 40 kg/m²) and lacked external validation. The remaining included studies (31 studies) reported the absolute or proportional risk of mortality (29 studies), or the risk of developing T2D in a population with obesity and with prediabetes or normal glucose tolerance (NGT) (three studies), or a CV event in populations with severe obesity with NGT or T2D (three studies). Most reported proportional risk, predominantly a hazard ratio.

CONCLUSION

More work is needed to develop and validate these risk equations, specifically in non-European cohorts and that distinguish between BMI class II and III obesity. New data or adjustment of the current risk equations by calibration would allow for more accurate decision making at an individual and population level.

Breaking bad news in the era of artificial intelligence and algorithmic medicine: an exploration of disclosure and its ethical justification using the hedonic calculus

An appropriate ethical framework around the use of Artificial Intelligence (AI) in healthcare has become a key desirable with the increasingly widespread deployment of this technology. Advances in AI hold the promise of improving the precision of outcome prediction at the level of the individual. However, the addition of these technologies to patient-clinician interactions, as with any complex human interaction, has potential pitfalls. While physicians have always had to carefully consider the ethical background and implications of their actions, detailed deliberations around fast-moving technological progress may not have kept up. We use a common but key challenge in healthcare interactions, the disclosure of bad news (likely imminent death), to illustrate how the philosophical framework of the 'Felicific Calculus' developed in the eighteenth century by Jeremy Bentham, may have a timely quasi-quantitative application in the age of AI. We show how this ethical algorithm can be used to assess, across seven mutually exclusive and exhaustive domains, whether an AI-supported action can be morally justified.

Machine learning for the life-time risk prediction of Alzheimer's disease: a systematic review

Alzheimer’s disease is a neurodegenerative disorder and the most common form of dementia. Early diagnosis may assist interventions to delay onset and reduce the progression rate of the disease. We systematically reviewed the use of machine learning algorithms for predicting Alzheimer’s disease using single nucleotide polymorphisms and instances where these were combined with other types of data. We evaluated the ability of machine learning models to distinguish between controls and cases, while also assessing their implementation and potential biases. Articles published between December 2009 and June 2020 were collected using Scopus, PubMed and Google Scholar. These were systematically screened for inclusion leading to a final set of 12 publications. Eighty-five per cent of the included studies used the Alzheimer's Disease Neuroimaging Initiative dataset. In studies which reported area under the curve, discrimination varied (0.49–0.97). However, more than half of the included manuscripts used other forms of measurement, such as accuracy, sensitivity and specificity. Model calibration statistics were also found to be reported inconsistently across all studies. The most frequent limitation in the assessed studies was sample size, with the total number of participants often numbering less than a thousand, whilst the number of predictors usually ran into the many thousands. In addition, key steps in model implementation and validation were often not performed or unreported, making it difficult to assess the capability of machine learning models.

Prognostic prediction models for chronic postsurgical pain in adults: a systematic review

ABSTRACT

Chronic postsurgical pain (CPSP) affects an estimated 10% to 50% of adults depending on the type of surgical procedure. Clinical prediction models can help clinicians target preventive strategies towards patients at high risk for CPSP. Therefore, the objective of this systematic review was to identify and describe existing prediction models for CPSP in adults. A systematic search was performed in MEDLINE, Embase, PsychINFO, and the Cochrane Database of Systematic Reviews in March 2020 for English peer-reviewed studies that used data collected between 2000 and 2020. Studies that developed, validated, or updated a prediction model in adult patients who underwent any surgical procedure were included. Two reviewers independently screened titles, abstracts, and full texts for eligibility; extracted data; and assessed risk of bias using the Prediction model Risk of Bias Assessment Tool. The search identified 2037 records; 28 articles were reviewed in full text. Fifteen studies reporting on 19 prediction models were included; all were at high risk of bias. Model discrimination, measured by the area under receiver operating curves or c-statistic, ranged from 0.690 to 0.816. The most common predictors identified in final prediction models included preoperative pain in the surgical area, preoperative pain in other areas, age, sex or gender, and acute postsurgical pain. Clinical prediction models may support prevention and management of CPSP, but existing models are at high risk of bias that affects their reliability to inform practice and generalizability to wider populations. Adherence to standardized guidelines for clinical prediction model development is necessary to derive a prediction model of value to clinicians.

Consistency of ranking was evaluated as new measure for prediction model stability: longitudinal cohort study

OBJECTIVE

Clinical risk prediction models are generally assessed on population level with a lack of measures that evaluate their stability at predicting risks of individual patients. This study evaluated the use of ranking as a measure to assess individual level stability between risk prediction models.

STUDY DESIGN AND SETTING

A large patient cohort (3.66 million patients with 0.11 million cardiovascular events) extracted from the Clinical Practice Research Datalink was used in the exemplar of cardiovascular disease risk prediction.

RESULTS

It was found that 15 models (including machine learning and statistical models) had similar population-level model performance (C statistics about 0.88). For patients with high absolute risks, the models were more consistent in ranking of risk predictions (interquartile range (IQR) of differences in rank percentiles -0.6 to 1.0), but inconsistent in absolute risk (IQR of differences in absolute risk -18.8 to 9.0). At low risk, the reverse was true with inconsistent ranking but more consistent absolute risk.

CONCLUSION

Consistency of ranking of individual risk predictions is a useful measure to assess risk prediction models providing complementary information to absolute risk stability. Model developing guidelines including "TRIPOD" and "PROBAST" should incorporate ranking to assess individual level stability between risk prediction models.

Predicting asthma-related crisis events using routine electronic healthcare data

BACKGROUND

There is no published algorithm predicting asthma crisis events (Accident and Emergency (A&E) attendance, hospitalisation or death) using routinely available electronic health record (EHR) data.

AIM

To develop an algorithm to identify individuals at high risk of an asthma crisis event.

DESIGN AND SETTING

Database analysis from primary care EHRs.

METHOD

Multivariable logistic regression was applied to a dataset of 61,861 people with asthma from England and Scotland using the Clinical Practice Research Datalink. External validation was performed using the Secure Anonymised Information Linkage databank of 174,240 patients from Wales. Outcomes were one or more hospitalisation (development dataset) and asthma-related hospitalisation, A&E attendance or death (validation dataset) within a 12-month period.

RESULTS

Risk factors for asthma-related crisis events included previous hospitalisation, older age, underweight, smoking and blood eosinophilia. The prediction algorithm had acceptable predictive ability with a Receiver Operating Characteristic (ROC) of 0.71 (0.70, 0.72) in the validation dataset. Using a cut-point based on the 7% of the population at greatest risk results in a positive predictive value of 5.7% (95% CI 5.3 - 6.1) and a negative predictive value of 98.9% (98.9 - 99.0), with sensitivity of 28.5% (26.7 - 30.3) and specificity of 93.3% (93.2 - 93.4); they had an event risk of 6.0% compared 1.1% for the remaining population. Eighteen people would be "needed to follow" to identify one admission.

CONCLUSIONS

This externally validated algorithm has acceptable predictive ability for identifying patients at high risk of asthma-related crisis events and excluding individuals not at high risk.

Artificial intelligence in oncology: From bench to clinic

In the past few years, Artificial Intelligence (AI) techniques have been applied to almost every facet of oncology, from basic research to drug development and clinical care. In the clinical arena where AI has perhaps received the most attention, AI is showing promise in enhancing and automating image-based diagnostic approaches in fields such as radiology and pathology. Robust AI applications, which retain high performance and reproducibility over multiple datasets, extend from predicting indications for drug development to improving clinical decision support using electronic health record data. In this article, we review some of these advances. We also introduce common concepts and fundamentals of AI and its various uses, along with its caveats, to provide an overview of the opportunities and challenges in the field of oncology. Leveraging AI techniques productively to provide better care throughout a patient's medical journey can fuel the predictive promise of precision medicine.

An assessment of the potential miscalibration of cardiovascular disease risk predictions caused by a secular trend in cardiovascular disease in England

Background

A downwards secular trend in the incidence of cardiovascular disease (CVD) in England was identified through previous work and the literature. Risk prediction models for primary prevention of CVD do not model this secular trend, this could result in over prediction of risk for individuals in the present day. We evaluate the effects of modelling this secular trend, and also assess whether it is driven by an increase in statin use during follow up.

Methods

We derived a cohort of patients (1998–2015) eligible for cardiovascular risk prediction from the Clinical Practice Research Datalink with linked hospitalisation and mortality records (N = 3,855,660). Patients were split into development and validation cohort based on their cohort entry date (before/after 2010). The calibration of a CVD risk prediction model developed in the development cohort was tested in the validation cohort. The calibration was also assessed after modelling the secular trend. Finally, the presence of the secular trend was evaluated under a marginal structural model framework, where the effect of statin treatment during follow up is adjusted for.

Results

Substantial over prediction of risks in the validation cohort was found when not modelling the secular trend. This miscalibration could be minimised if one was to explicitly model the secular trend. The reduction in risk in the validation cohort when introducing the secular trend was 35.68 and 33.24% in the female and male cohorts respectively. Under the marginal structural model framework, the reductions were 33.31 and 32.67% respectively, indicating increasing statin use during follow up is not the only the cause of the secular trend.

Conclusions

Inclusion of the secular trend into the model substantially changed the CVD risk predictions. Models that are being used in clinical practice in the UK do not model secular trend and may thus overestimate the risks, possibly leading to patients being treated unnecessarily. Wider discussion around the modelling of secular trends in a risk prediction framework is needed.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-020-01173-x.

Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar

OBJECTIVE

To assess the consistency of machine learning and statistical techniques in predicting individual level and population level risks of cardiovascular disease and the effects of censoring on risk predictions.

DESIGN

Longitudinal cohort study from 1 January 1998 to 31 December 2018.

SETTING AND PARTICIPANTS

3.6 million patients from the Clinical Practice Research Datalink registered at 391 general practices in England with linked hospital admission and mortality records.

MAIN OUTCOME MEASURES

Model performance including discrimination, calibration, and consistency of individual risk prediction for the same patients among models with comparable model performance. 19 different prediction techniques were applied, including 12 families of machine learning models (grid searched for best models), three Cox proportional hazards models (local fitted, QRISK3, and Framingham), three parametric survival models, and one logistic model.

RESULTS

The various models had similar population level performance (C statistics of about 0.87 and similar calibration). However, the predictions for individual risks of cardiovascular disease varied widely between and within different types of machine learning and statistical models, especially in patients with higher risks. A patient with a risk of 9.5-10.5% predicted by QRISK3 had a risk of 2.9-9.2% in a random forest and 2.4-7.2% in a neural network. The differences in predicted risks between QRISK3 and a neural network ranged between -23.2% and 0.1% (95% range). Models that ignored censoring (that is, assumed censored patients to be event free) substantially underestimated risk of cardiovascular disease. Of the 223 815 patients with a cardiovascular disease risk above 7.5% with QRISK3, 57.8% would be reclassified below 7.5% when using another model.

CONCLUSIONS

A variety of models predicted risks for the same patients very differently despite similar model performances. The logistic models and commonly used machine learning models should not be directly applied to the prediction of long term risks without considering censoring. Survival models that consider censoring and that are explainable, such as QRISK3, are preferable. The level of consistency within and between models should be routinely assessed before they are used for clinical decision making.

On predictions in critical care: The individual prognostication fallacy in elderly patients

Predicting the future course of critical conditions involves personal experience, heuristics and statistical models. Although these methods may perform well for some cases and population averages, they suffer from substantial shortcomings when applied to individual patients. The reasons include methodological problems of statistical modeling as well as limitations of cross-sectional data sampling. Accurate predictions for individual patients become crucial when they have to guide irreversible decision-making. This notably applies to triage situations in response to a lack of healthcare resources. We will discuss these issues and argue that analysing longitudinal data obtained from time-limited trials in intensive care can provide a more robust approach to individual prognostication.

Receptivity of Hospitalized Older Adults and Family Caregivers to Prognostic Information about Aging, Injury, and Frailty: A Qualitative Study

BACKGROUND

Frailty is the leading prognosticator for poor outcomes and palliative care among older adults. Delivery of negative prognostic information entails potentially difficult conversations about decline and death.

OBJECTIVE

The study aims were to: 1) examine hospitalized older adults' and family caregivers' receptivity to general (vs. individualized) prognostic information about frailty, injury, and one-year outcomes; and 2) determine information needs based on prognostic information.

DESIGN

Provision of general prognostic information followed by semi-structured interview questions. We deductively analyzed qualitative data within the context of problematic integration theory.

SETTING

An academic medical center in the Southeast region of the U.S.

PARTICIPANTS

Purposive sampling was utilized to obtain a distribution of patients across the frailty continuum (non-frail [N=10], pre-frail [N=9], frail [9=6]). Twenty-five older adults (≥ age 65) hospitalized for a primary injury (e.g. fall) and 15 family caregivers of hospitalized patients were enrolled.

METHODS

Hospitalized older patients and family caregivers were shown prognostic information about one-year outcomes of injured older adults in the form of simple pictographs. Semi-structured interview questions were administered immediately afterwards. The interviews were audio-recorded, transcribed, and analyzed using qualitative content analysis. Demographic and medical information data were used to contextualize the responses during analysis.

RESULTS

Overall, participants (patients [56%], caregivers [73%]) were open to receiving prognostic information. A small number of family caregivers (N=3) expressed reservations about the frankness of the information and suggested delivery through a softer approach or not at all. Qualitative data was coded using categories and constructs of problematic integration theory. Four codes (personalizing the evidence, vivid understanding, downhill spiral, realities of aging) reflected probabilistic and evaluative orientation categories of problematic integration theory. One code (fatalism vs. hope) represented manifestations of ambivalence and ambiguity in the theory; and another code (exceptionalism) represented divergence and impossibility. Two codes (role of thought processes, importance of faith) reflected forms of resolutions as described in problematic integration theory. Information needs based on prognostic information revealed four additional codes: give it to me straight, what can I do? what can I expect? and how can I prevent decline? A consistently reported desire of both patients and caregivers was for honesty and hope from providers.

CONCLUSION

This study supports the use of general prognostic information in conversations about aging, injury, frailty and patient outcomes. Incorporating prognostic information into communication aids can facilitate shared decision making before end-of-life is imminent.

The present and future scope of real-world evidence research in diabetes: What questions can and cannot be answered and what might be possible in the future?

The last decade has witnessed an exponential growth in the opportunities to collect and link health-related data from multiple resources, including primary care, administrative, and device data. The availability of these "real-world," "big data" has fuelled also an intense methodological research into methods to handle them and extract actionable information. In medicine, the evidence generated from "real-world data" (RWD), which are not purposely collected to answer biomedical questions, is commonly termed "real-world evidence" (RWE). In this review, we focus on RWD and RWE in the area of diabetes research, highlighting their contributions in the last decade; and give some suggestions for future RWE diabetes research, by applying well-established and less-known tools to direct RWE diabetes research towards better personalized approaches to diabetes care. We underline the essential aspects to consider when using RWD and the key features limiting the translational potential of RWD in generating high-quality and applicable RWE. Only if viewed in the context of other study designs and statistical methods, with its pros and cons carefully considered, RWE will exploit its full potential as a complementary or even, in some cases, substitutive source of evidence compared to the expensive evidence obtained from randomized controlled trials.

Correction to: The uncertainty with using risk prediction models for individual decision making: an exemplar cohort study examining the prediction of cardiovascular disease in English primary care

The original article [1] contained an error in the abstract. The mentioned cohort size now correctly states 'N = 3,855,660'.

Beyond the hype of big data and artificial intelligence: building foundations for knowledge and wisdom

Big data, coupled with the use of advanced analytical approaches, such as artificial intelligence (AI), have the potential to improve medical outcomes and population health. Data that are routinely generated from, for example, electronic medical records and smart devices have become progressively easier and cheaper to collect, process, and analyze. In recent decades, this has prompted a substantial increase in biomedical research efforts outside traditional clinical trial settings. Despite the apparent enthusiasm of researchers, funders, and the media, evidence is scarce for successful implementation of products, algorithms, and services arising that make a real difference to clinical care. This article collection provides concrete examples of how "big data" can be used to advance healthcare and discusses some of the limitations and challenges encountered with this type of research. It primarily focuses on real-world data, such as electronic medical records and genomic medicine, considers new developments in AI and digital health, and discusses ethical considerations and issues related to data sharing. Overall, we remain positive that big data studies and associated new technologies will continue to guide novel, exciting research that will ultimately improve healthcare and medicine-but we are also realistic that concerns remain about privacy, equity, security, and benefit to all.