The Proliferation of Reports on Clinical Scoring Systems: Issues About Uptake and Clinical Utility

Medical calculators derived synthetic cohorts: a novel method for generating synthetic patient data

This study shows that we can use synthetic cohorts created from medical risk calculators to gain insights into how risk estimations, clinical reasoning, data-driven subgrouping, and the confidence in risk calculator scores are connected. When prediction variables aren't evenly distributed in these synthetic cohorts, they can be used to group similar cases together, revealing new insights about how cohorts behave. We also found that the confidence in predictions made by these calculators can vary depending on patient characteristics. This suggests that it might be beneficial to include a "normalized confidence" score in future versions of these calculators for healthcare professionals. We plan to explore this idea further in our upcoming research.

[Comparison of different prediction models for the occurrence of nausea and vomiting in the postoperative phase : A systematic qualitative comparison based on prospectively defined quality indicators]

BACKGROUND

Various prognostic prediction models exist for evaluating the risk of nausea and vomiting in the postoperative period (PONV). So far, no systematic comparison of these prognostic scores is available.

METHOD

A systematic literature search was carried out in seven medical databases to find publications on prognostic PONV models. Identified scores were assessed against prospectively defined quality criteria, including generalizability, validation and clinical relevance of the models.

RESULTS

The literature search revealed 62 relevant publications with a total of 81,834 patients which could be assigned to 8 prognostic models. The simplified Apfel score performed best, primarily because it was extensively validated. The Van den Bosch score and Sinclair score tied for second place. The simplified Koivuranta score was in third place.

CONCLUSION

The qualitative analysis highlights the strengths and weaknesses of each prediction system based on predetermined standardized quality criteria.

Quantifying health facility service readiness for small and sick newborn care: comparing standards-based and WHO level-2 + scoring for 64 hospitals implementing with NEST360 in Kenya, Malawi, Nigeria, and Tanzania

BACKGROUND

Service readiness tools are important for assessing hospital capacity to provide quality small and sick newborn care (SSNC). Lack of summary scoring approaches for SSNC service readiness means we are unable to track national targets such as the Every Newborn Action Plan targets.

METHODS

A health facility assessment (HFA) tool was co-designed by Newborn Essential Solutions and Technologies (NEST360) and UNICEF with four African governments. Data were collected in 68 NEST360-implementing neonatal units in Kenya, Malawi, Nigeria, and Tanzania (September 2019-March 2021). Two summary scoring approaches were developed: a) standards-based, including items for SSNC service readiness by health system building block (HSBB), and scored on availability and functionality, and b) level-2 + , scoring items on readiness to provide WHO level-2 + clinical interventions. For each scoring approach, scores were aggregated and summarised as a percentage and equally weighted to obtain an overall score by hospital, HSBB, and clinical intervention.

RESULTS

Of 1508 HFA items, 1043 (69%) were included in standards-based and 309 (20%) in level-2 + scoring. Sixty-eight neonatal units across four countries had median standards-based scores of 51% [IQR 48-57%] at baseline, with variation by country: 62% [IQR 59-66%] in Kenya, 49% [IQR 46-51%] in Malawi, 50% [IQR 42-58%] in Nigeria, and 55% [IQR 53-62%] in Tanzania. The lowest scoring was family-centred care [27%, IQR 18-40%] with governance highest scoring [76%, IQR 71-82%]. For level-2 + scores, the overall median score was 41% [IQR 35-51%] with variation by country: 50% [IQR 44-53%] in Kenya, 41% [IQR 35-50%] in Malawi, 33% [IQR 27-37%] in Nigeria, and 41% [IQR 32-52%] in Tanzania. Readiness to provide antibiotics by culture report was the highest-scoring intervention [58%, IQR 50-75%] and neonatal encephalopathy management was the lowest-scoring [21%, IQR 8-42%]. In both methods, overall scores were low (< 50%) for 27 neonatal units in standards-based scoring and 48 neonatal units in level-2 + scoring. No neonatal unit achieved high scores of > 75%.

DISCUSSION

Two scoring approaches reveal gaps in SSNC readiness with no neonatal units achieving high scores (> 75%). Government-led quality improvement teams can use these summary scores to identify areas for health systems change. Future analyses could determine which items are most directly linked with quality SSNC and newborn outcomes.

Diagnostic algorithm allows for a scientifically robust and reliable retrospective diagnosis using textual evidence from mid-19th century Basel, Switzerland

OBJECTIVE

Diagnosing disease from the past using historic textual sources can be controversial as to its accuracy. To overcome these objections, an empirical approach to the historical clinical data was developed. The approach follows a standardised, objective, and systematic evaluation, satisfying the requirements of the philosophy of science.

MATERIAL

Physician-managed medical records of mid-19th century patients reported to have suffered from tuberculosis.

METHOD

A diagnostic algorithm, quantifying clinical data into a scoring system, was developed based on criteria recorded in the medical sources. The findings were compared to the autopsy results using the Receiver Operating Characteristics method.

RESULTS

The generated scoring system correctly predicted the diagnosis of tuberculosis in 86% of patients in the study. 6% false negatives and 8% false positives were predicted.

CONCLUSIONS

It is possible to retrospectively diagnose in a reliable and scientifically robust manner under certain conditions. It is important to embed the clinical data into the historical context. A general rejection of retrospective diagnosis is unsubstantiated. Well-designed, disease-specific, and source adapted medical scoring systems are new approaches and overcome criticism raised against retrospective diagnosis.

SIGNIFICANCE

This new approach utilises diverse historic sources and potentially leads to reliable retrospective diagnosis of most common diseases of the past.

LIMITATIONS

Selection bias of the records allocated. Quality of the historic sources utilized. Restricted statistical assessment potential of historic sources.

SUGGESTIONS FOR FURTHER RESEARCH

Development of disease- and epoch-specific medical score systems.

Posterior Instrumented Spinal Surgery Outcomes in the Elderly: A Comparison of the 5-Item and 11-Item Modified Frailty Indices

STUDY DESIGN

Retrospective Cohort.

OBJECTIVES

To validate the most concise risk stratification system to date, the 5-item modified frailty index (mFI-5), and compare its effectiveness with the established 11-item modified frailty index (mFI-11) in the elderly population undergoing posterior instrumented spine surgery.

METHODS

A single centre retrospective review of posterior instrumented spine surgeries in patients aged 65 years and older was conducted. The primary outcome was rate of post-operative major complications (Clavien-Dindo Classification ≥ 4). Secondary outcome measures included rate of all complications, 6-month mortality and surgical site infection. Multi-variate analysis was performed and adjusted receiver operating characteristic curves were generated and compared by DeLong's test. The indices were correlated with Spearman's rho.

RESULTS

272 cases were identified. The risk of major complications was independently associated with both the mFI-5 (OR 1.89, 95% CI 1.01-3.55, P = .047) and mFI-11 (OR 3.73, 95% CI 1.90-7.30, P = .000). Both the mFI-5 and mFI-11 were statistically significant predictors of risk of all complications (P = .007 and P = .003), surgical site infection (P = .011 and P = .003) and 6-month mortality (P = .031 and P = .000). Adjusted ROC curves determined statistically similar c-statistics for major complications (.68 vs .68, P = .64), all complications (.66 vs .64, P = .10), surgical site infection (.75 vs .75, P = .76) and 6-month mortality (.83 vs .81, P = .21). The 2 indices correlated very well with a Spearman's rho of .944.

CONCLUSIONS

The mFI-5 and mFI-11 are equally effective predictors of postoperative morbidity and mortality in this population. The brevity of the mFI-5 is advantageous in facilitating its daily clinical use.

DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record

OBJECTIVE

Heatlhcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable, and reliable machine learning models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient, safe and high-quality manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher-created models into a widely used electronic medical record system.

MATERIALS AND METHODS

We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within electronic medical record software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact.

RESULTS

We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating 12 machine learning models trained using electronic medical record data that predict laboratory diagnostic results, triggered by clinician button-clicks in Stanford Health Care's electronic medical record.

DISCUSSION

Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. When possible, we recommend using prospectively estimated performance measures during silent trials to make final go decisions for model deployment.

CONCLUSION

Machine learning applications in healthcare are extensively researched, but successful translations to the bedside are rare. By describing DEPLOYR, we aim to inform machine learning deployment best practices and help bridge the model implementation gap.

Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes

Medical experts may use Artificial Intelligence (AI) systems with greater trust if these are supported by 'contextual explanations' that let the practitioner connect system inferences to their context of use. However, their importance in improving model usage and understanding has not been extensively studied. Hence, we consider a comorbidity risk prediction scenario and focus on contexts regarding the patients' clinical state, AI predictions about their risk of complications, and algorithmic explanations supporting the predictions. We explore how relevant information for such dimensions can be extracted from Medical guidelines to answer typical questions from clinical practitioners. We identify this as a question answering (QA) task and employ several state-of-the-art Large Language Models (LLM) to present contexts around risk prediction model inferences and evaluate their acceptability. Finally, we study the benefits of contextual explanations by building an end-to-end AI pipeline including data cohorting, AI risk modeling, post-hoc model explanations, and prototyped a visual dashboard to present the combined insights from different context dimensions and data sources, while predicting and identifying the drivers of risk of Chronic Kidney Disease (CKD) - a common type-2 diabetes (T2DM) comorbidity. All of these steps were performed in deep engagement with medical experts, including a final evaluation of the dashboard results by an expert medical panel. We show that LLMs, in particular BERT and SciBERT, can be readily deployed to extract some relevant explanations to support clinical usage. To understand the value-add of the contextual explanations, the expert panel evaluated these regarding actionable insights in the relevant clinical setting. Overall, our paper is one of the first end-to-end analyses identifying the feasibility and benefits of contextual explanations in a real-world clinical use case. Our findings can help improve clinicians' usage of AI models.

APLUS: A Python library for usefulness simulations of machine learning models in healthcare

Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.

Clinical risk calculators informing the decision to admit: A methodologic evaluation and assessment of applicability

INTRODUCTION

Clinical prediction and decision tools that generate outcome-based risk stratification and/or intervention recommendations are prevalent. Appropriate use and validity of these tools, especially those that inform complex clinical decisions, remains unclear. The objective of this study was to assess the methodologic quality and applicability of clinical risk scoring tools used to guide hospitalization decision-making.

METHODS

In February 2021, a comprehensive search was performed of a clinical calculator online database (mdcalc.com) that is publicly available and well-known to clinicians. The primary reference for any calculator tool informing outpatient versus inpatient disposition was considered for inclusion. Studies were restricted to the adult, acute care population. Those focused on obstetrics/gynecology or critical care admission were excluded. The Wasson-Laupacis framework of methodologic standards for clinical prediction rules was applied to each study.

RESULTS

A total of 22 calculators provided hospital admission recommendations for 9 discrete medical conditions using adverse events (14/22), mortality (6/22), or confirmatory diagnosis (2/22) as outcomes of interest. The most commonly met methodologic standards included mathematical technique description (22/22) and clinical sensibility (22/22) and least commonly met included reproducibility of the rule (1/22) and measurement of effect on clinical use (1/22). Description of the studied population was often lacking, especially patient race/ethnicity (2/22) and mental or behavioral health (0/22). Only one study reported any item related to social determinants of health.

CONCLUSION

Studies commonly do not meet rigorous methodologic standards and often fail to report pertinent details that would guide applicability. These clinical tools focus primarily on specific disease entities and clinical variables, missing the breadth of information necessary to make a disposition determination and raise significant validation and generalizability concerns.

Artificial intelligence-enabled decision support in nephrology

Kidney pathophysiology is often complex, nonlinear and heterogeneous, which limits the utility of hypothetical-deductive reasoning and linear, statistical approaches to diagnosis and treatment. Emerging evidence suggests that artificial intelligence (AI)-enabled decision support systems - which use algorithms based on learned examples - may have an important role in nephrology. Contemporary AI applications can accurately predict the onset of acute kidney injury before notable biochemical changes occur; can identify modifiable risk factors for chronic kidney disease onset and progression; can match or exceed human accuracy in recognizing renal tumours on imaging studies; and may augment prognostication and decision-making following renal transplantation. Future AI applications have the potential to make real-time, continuous recommendations for discrete actions and yield the greatest probability of achieving optimal kidney health outcomes. Realizing the clinical integration of AI applications will require cooperative, multidisciplinary commitment to ensure algorithm fairness, overcome barriers to clinical implementation, and build an AI-competent workforce. AI-enabled decision support should preserve the pre-eminence of wisdom and augment rather than replace human decision-making. By anchoring intuition with objective predictions and classifications, this approach should favour clinician intuition when it is honed by experience.

Developing and Implementing Predictive Models in a Learning Healthcare System: Traditional and Artificial Intelligence Approaches in the Veterans Health Administration

Predicting clinical risk is an important part of healthcare and can inform decisions about treatments, preventive interventions, and provision of extra services. The field of predictive models has been revolutionized over the past two decades by electronic health record data; the ability to link such data with other demographic, socioeconomic, and geographic information; the availability of high-capacity computing; and new machine learning and artificial intelligence methods for extracting insights from complex datasets. These advances have produced a new generation of computerized predictive models, but debate continues about their development, reporting, validation, evaluation, and implementation. In this review we reflect on more than 10 years of experience at the Veterans Health Administration, the largest integrated healthcare system in the United States, in developing, testing, and implementing such models at scale. We report lessons from the implementation of national risk prediction models and suggest an agenda for research. Expected final online publication date for the Annual Review of Biomedical Data Science, Volume 5 is August 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Diagnosing capillary leak in critically ill patients: development of an innovative scoring instrument for non-invasive detection

BACKGROUND

The concomitant occurrence of the symptoms intravascular hypovolemia, peripheral edema and hemodynamic instability is typically named Capillary Leak Syndrome (CLS) and often occurs in surgical critical ill patients. However, neither a unitary definition nor standardized diagnostic criteria exist so far. We aimed to investigate common characteristics of this phenomenon with a subsequent scoring system, determining whether CLS contributes to mortality.

METHODS

We conducted this single-center, observational, multidisciplinary, prospective trial in two separately run surgical ICUs of a tertiary academic medical center. 200 surgical patients admitted to the ICU and 30 healthy volunteers were included. Patients were clinically diagnosed as CLS or No-CLS group (each N = 100) according to the grade of edema, intravascular hypovolemia, hemodynamic instability, and positive fluid balance by two independent attending physicians with > 10 years of experience in ICU. We performed daily measurements with non-invasive body impedance electrical analysis, ultrasound and analysis of serum biomarkers to generate objective diagnostic criteria. Receiver operating characteristics were used, while we developed machine learning models to increase diagnostic specifications for our scoring model.

RESULTS

The 30-day mortility was increased among CLS patients (12 vs. 1%, P = 0.002), while showing higher SOFA-scores. Extracellular water was increased in patients with CLS with higher echogenicity of subcutaneous tissue [29(24-31) vs. 19(16-21), P < 0.001]. Biomarkers showed characteristic alterations, especially with an increased angiopoietin-2 concentration in CLS [9.9(6.2-17.3) vs. 3.7(2.6-5.6)ng/mL, P < 0.001]. We developed a score using seven parameters (echogenicity, SOFA-score, angiopoietin-2, syndecan-1, ICAM-1, lactate and interleukin-6). A Random Forest prediction model boosted its diagnostic characteristics (AUC 0.963, P < 0.001), while a two-parameter decision tree model showed good specifications (AUC 0.865).

CONCLUSIONS

Diagnosis of CLS in critically ill patients is feasible by objective, non-invasive parameters using the CLS-Score. A simplified two-parameter diagnostic approach can enhance clinical utility. CLS contributes to mortality and should, therefore, classified as an independent entity.

TRIAL REGISTRATION

German Clinical Trials Registry (DRKS No. 00012713), Date of registration 10/05/2017, www.drks.de.

Machine learning: principles and applications for thoracic surgery

OBJECTIVES

Machine learning (ML) has experienced a revolutionary decade with advances across many disciplines. We seek to understand how recent advances in ML are going to specifically influence the practice of surgery in the future with a particular focus on thoracic surgery.

METHODS

Review of relevant literature in both technical and clinical domains.

RESULTS

ML is a revolutionary technology that promises to change the way that surgery is practiced in the near future. Spurred by an advance in computing power and the volume of data produced in healthcare, ML has shown remarkable ability to master tasks that had once been reserved for physicians. Supervised learning, unsupervised learning and reinforcement learning are all important techniques that can be leveraged to improve care. Five key applications of ML to cardiac surgery include diagnostics, surgical skill assessment, postoperative prognostication, augmenting intraoperative performance and accelerating translational research. Some key limitations of ML include lack of interpretability, low quality and volumes of relevant clinical data, ethical limitations and difficulties with clinical implementation.

CONCLUSIONS

In the future, the practice of cardiac surgery will be greatly augmented by ML technologies, ultimately leading to improved surgical performance and better patient outcomes.

Derivation and Validation of a 10-Year Risk Score for Symptomatic Abdominal Aortic Aneurysm: Cohort Study of Nearly 500 000 Individuals

BACKGROUND

Abdominal aortic aneurysm (AAA) can occur in patients who are ineligible for routine ultrasound screening. A simple AAA risk score was derived and compared with current guidelines used for ultrasound screening of AAA.

METHODS

United Kingdom Biobank participants without previous AAA were split into a derivation cohort (n=401 820, 54.6% women, mean age 56.4 years, 95.5% White race) and validation cohort (n=83 816). Incident AAA was defined as first hospital inpatient diagnosis of AAA, death from AAA, or an AAA-related surgical procedure. A multivariable Cox model was developed in the derivation cohort into an AAA risk score that did not require blood biomarkers. To illustrate the sensitivity and specificity of the risk score for AAA, a theoretical threshold to refer patients for ultrasound at 0.25% 10-year risk was modeled. Discrimination of the risk score was compared with a model of US Preventive Services Task Force (USPSTF) AAA screening guidelines.

RESULTS

In the derivation cohort, there were 1570 (0.40%) cases of AAA over a median 11.3 years of follow-up. Components of the AAA risk score were age (stratified by smoking status), weight (stratified by smoking status), antihypertensive and cholesterol-lowering medication use, height, diastolic blood pressure, baseline cardiovascular disease, and diabetes. In the validation cohort, over 10 years of follow-up, the C-index for the model of the USPSTF guidelines was 0.705 (95% CI, 0.678-0.733). The C-index of the risk score as a continuous variable was 0.856 (95% CI, 0.837-0.878). In the validation cohort, the USPSTF model yielded sensitivity 63.9% and specificity 71.3%. At the 0.25% 10-year risk threshold, the risk score yielded sensitivity 82.1% and specificity 70.7% while also improving the net reclassification index compared with the USPSTF model +0.176 (95% CI, 0.120-0.232). A combined model, whereby risk scoring was combined with the USPSTF model, also improved prediction compared with USPSTF alone (net reclassification index +0.101 [95% CI, 0.055-0.147]).

CONCLUSIONS

In an asymptomatic general population, a risk score based on patient age, height, weight, and medical history may improve identification of asymptomatic patients at risk for clinical events from AAA. Further development and validation of risk scores to detect asymptomatic AAA are needed.

Learning decision thresholds for risk stratification models from aggregate clinician behavior

Using a risk stratification model to guide clinical practice often requires the choice of a cutoff—called the decision threshold—on the model’s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 pooled cohort equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision making following model deployment.

Systematic Review of Approaches to Preserve Machine Learning Performance in the Presence of Temporal Dataset Shift in Clinical Medicine

OBJECTIVE

The change in performance of machine learning models over time as a result of temporal dataset shift is a barrier to machine learning-derived models facilitating decision-making in clinical practice. Our aim was to describe technical procedures used to preserve the performance of machine learning models in the presence of temporal dataset shifts.

METHODS

Studies were included if they were fully published articles that used machine learning and implemented a procedure to mitigate the effects of temporal dataset shift in a clinical setting. We described how dataset shift was measured, the procedures used to preserve model performance, and their effects.

RESULTS

Of 4,457 potentially relevant publications identified, 15 were included. The impact of temporal dataset shift was primarily quantified using changes, usually deterioration, in calibration or discrimination. Calibration deterioration was more common (n = 11) than discrimination deterioration (n = 3). Mitigation strategies were categorized as model level or feature level. Model-level approaches (n = 15) were more common than feature-level approaches (n = 2), with the most common approaches being model refitting (n = 12), probability calibration (n = 7), model updating (n = 6), and model selection (n = 6). In general, all mitigation strategies were successful at preserving calibration but not uniformly successful in preserving discrimination.

CONCLUSION

There was limited research in preserving the performance of machine learning models in the presence of temporal dataset shift in clinical medicine. Future research could focus on the impact of dataset shift on clinical decision making, benchmark the mitigation strategies on a wider range of datasets and tasks, and identify optimal strategies for specific settings.

A framework for making predictive models useful in practice

OBJECTIVE

To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.

MATERIALS AND METHODS

We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models' predictions. Factors included nonclinical reasons that make ACP inappropriate: limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases. We also quantified the relative benefits of increasing capacity for inpatient ACP versus outpatient ACP.

RESULTS

Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow based on a model's predictions. However, the reduction can be mitigated by creating an outpatient ACP workflow. Given limited resources to either add capacity for inpatient ACP versus developing outpatient ACP capability, the latter is likely to provide more benefit to patient care.

DISCUSSION

The benefit of using a predictive model for identifying patients for interventions is highly dependent on the capacity to execute the workflow triggered by the model. We provide a framework for quantifying the impact of healthcare delivery factors and work capacity constraints on achieved benefit.

CONCLUSION

An analysis of the sensitivity of the net benefit realized by a predictive model triggered clinical workflow to various healthcare delivery factors is necessary for making predictive models useful in practice.

Clinical integration of machine learning for curative-intent radiation treatment of patients with prostate cancer

Machine learning (ML) holds great promise for impacting healthcare delivery; however, to date most methods are tested in 'simulated' environments that cannot recapitulate factors influencing real-world clinical practice. We prospectively deployed and evaluated a random forest algorithm for therapeutic curative-intent radiation therapy (RT) treatment planning for prostate cancer in a blinded, head-to-head study with full integration into the clinical workflow. ML- and human-generated RT treatment plans were directly compared in a retrospective simulation with retesting (n = 50) and a prospective clinical deployment (n = 50) phase. Consistently throughout the study phases, treating physicians assessed ML- and human-generated RT treatment plans in a blinded manner following a priori defined standardized criteria and peer review processes, with the selected RT plan in the prospective phase delivered for patient treatment. Overall, 89% of ML-generated RT plans were considered clinically acceptable and 72% were selected over human-generated RT plans in head-to-head comparisons. RT planning using ML reduced the median time required for the entire RT planning process by 60.1% (118 to 47 h). While ML RT plan acceptability remained stable between the simulation and deployment phases (92 versus 86%), the number of ML RT plans selected for treatment was significantly reduced (83 versus 61%, respectively). These findings highlight that retrospective or simulated evaluation of ML methods, even under expert blinded review, may not be representative of algorithm acceptance in a real-world clinical setting when patient care is at stake.

Improving hospital readmission prediction using individualized utility analysis

OBJECTIVE

Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Our objective was to determine whether ML models for allocating readmission-mitigating interventions have different usefulness based on their overall utility and discriminative ability.

MATERIALS AND METHODS

We conducted a retrospective utility analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69 [19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for lowering readmission risk based on the reduction in the 90-day cost.

RESULTS

Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating interventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient, and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model performing the best under different intervention settings.

CONCLUSION

We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.

MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care

The rise of digital data and computing power have contributed to significant advancements in artificial intelligence (AI), leading to the use of classification and prediction models in health care to enhance clinical decision-making for diagnosis, treatment and prognosis. However, such advances are limited by the lack of reporting standards for the data used to develop those models, the model architecture, and the model evaluation and validation processes. Here, we present MINIMAR (MINimum Information for Medical AI Reporting), a proposal describing the minimum information necessary to understand intended predictions, target populations, and hidden biases, and the ability to generalize these emerging technologies. We call for a standard to accurately and responsibly report on AI in health care. This will facilitate the design and implementation of these models and promote the development and use of associated clinical decision support tools, as well as manage concerns regarding accuracy and bias.

Current opinion and emerging trends on the treatment, diagnosis, and prevention of pregnancy-associated venous thromboembolic disease: a review

Pregnancy associated venous thromboembolism (PA-VTE) is a leading cause of maternal morbidity and mortality worldwide. Despite the availability of international guidance on the prevention, diagnosis and treatment, practice differs between countries and clinical institutions. The evidence base in this area is limited due to the vulnerable population who are affected, with the majority of guidelines deriving their recommendations from experience in surgical and medical venous thromboembolic disease. This review includes best evidence in PA-VTE management, highlighting specific literature which supports current diagnosis, prevention, and treatment strategies. Additionally, we hope to demonstrate emerging trends in the field through discussion of ongoing trials designed to progress towards evidence-based practice in the context of PA-VTE.

Developing a delivery science for artificial intelligence in healthcare

Artificial Intelligence (AI) has generated a large amount of excitement in healthcare, mostly driven by the emergence of increasingly accurate machine learning models. However, the promise of AI delivering scalable and sustained value for patient care in the real world setting has yet to be realized. In order to safely and effectively bring AI into use in healthcare, there needs to be a concerted effort around not just the creation, but also the delivery of AI. This AI "delivery science" will require a broader set of tools, such as design thinking, process improvement, and implementation science, as well as a broader definition of what AI will look like in practice, which includes not just machine learning models and their predictions, but also the new systems for care delivery that they enable. The careful design, implementation, and evaluation of these AI enabled systems will be important in the effort to understand how AI can improve healthcare.

Ordinal labels in machine learning: a user-centered approach to improve data validity in medical settings

BACKGROUND

Despite the vagueness and uncertainty that is intrinsic in any medical act, interpretation and decision (including acts of data reporting and representation of relevant medical conditions), still little research has focused on how to explicitly take this uncertainty into account. In this paper, we focus on the representation of a general and wide-spread medical terminology, which is grounded on a traditional and well-established convention, to represent severity of health conditions (for instance, pain, visible signs), ranging from Absent to Extreme. Specifically, we will study how both potential patients and doctors perceive the different levels of the terminology in both quantitative and qualitative terms, and if the embedded user knowledge could improve the representation of ordinal values in the construction of machine learning models.

METHODS

To this aim, we conducted a questionnaire-based research study involving a relatively large sample of 1,152 potential patients and 31 clinicians to represent numerically the perceived meaning of standard and widely-applied labels to describe health conditions. Using these collected values, we then present and discuss different possible fuzzy-set based representations that address the vagueness of medical interpretation by taking into account the perceptions of domain experts. We also apply the findings of this user study to evaluate the impact of different encodings on the predictive performance of common machine learning models in regard to a real-world medical prognostic task.

RESULTS

We found significant differences in the perception of pain levels between the two user groups. We also show that the proposed encodings can improve the performances of specific classes of models, and discuss when this is the case.

CONCLUSIONS

In perspective, our hope is that the proposed techniques for ordinal scale representation and ordinal encoding may be useful to the research community, and also that our methodology will be applied to other widely used ordinal scales for improving validity of datasets and bettering the results of machine learning tasks.

Ethical dilemmas due to the Covid-19 pandemic

The devastating pandemic that has stricken the worldwide population induced an unprecedented influx of patients in ICUs, raising ethical concerns not only surrounding triage and withdrawal of life support decisions, but also regarding family visits and quality of end-of-life support. These ingredients are liable to shake up our ethical principles, sharpen our ethical dilemmas, and lead to situations of major caregiver sufferings. Proposals have been made to rationalize triage policies in conjunction with ethical justifications. However, whatever the angle of approach, imbalance between utilitarian and individual ethics leads to unsolvable discomforts that caregivers will need to overcome. With this in mind, we aimed to point out some critical ethical choices with which ICU caregivers have been confronted during the Covid-19 pandemic and to underline their limits. The formalized strategies integrating the relevant tools of ethical reflection were disseminated without deviating from usual practices, leaving to intensivists the ultimate choice of decision.

Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study

OBJECTIVE

In this study, we used human factors (HF) methods and principles to design a clinical decision support (CDS) that provides cognitive support to the pulmonary embolism (PE) diagnostic decision-making process in the emergency department. We hypothesised that the application of HF methods and principles will produce a more usable CDS that improves PE diagnostic decision-making, in particular decision about appropriate clinical pathway.

MATERIALS AND METHODS

We conducted a scenario-based simulation study to compare a HF-based CDS (the so-called CDS for PE diagnosis (PE-Dx CDS)) with a web-based CDS (MDCalc); 32 emergency physicians performed various tasks using both CDS. PE-Dx integrated HF design principles such as automating information acquisition and analysis, and minimising workload. We assessed all three dimensions of usability using both objective and subjective measures: effectiveness (eg, appropriate decision regarding the PE diagnostic pathway), efficiency (eg, time spent, perceived workload) and satisfaction (perceived usability of CDS).

RESULTS

Emergency physicians made more appropriate diagnostic decisions (94% with PE-Dx; 84% with web-based CDS; p<0.01) and performed experimental tasks faster with the PE-Dx CDS (on average 96 s per scenario with PE-Dx; 117 s with web-based CDS; p<0.001). They also reported lower workload (p<0.001) and higher satisfaction (p<0.001) with PE-Dx.

CONCLUSIONS

This simulation study shows that HF methods and principles can improve usability of CDS and diagnostic decision-making. Aspects of the HF-based CDS that provided cognitive support to emergency physicians and improved diagnostic performance included automation of information acquisition (eg, auto-populating risk scoring algorithms), minimisation of workload and support of decision selection (eg, recommending a clinical pathway). These HF design principles can be applied to the design of other CDS technologies to improve diagnostic safety.