Consensus Statement on Electronic Health Predictive Analytics: A Guiding Framework to Address Challenges

Developer Perspectives on Potential Harms of Machine Learning Predictive Analytics in Health Care: Qualitative Analysis

BACKGROUND

Machine learning predictive analytics (MLPA) is increasingly used in health care to reduce costs and improve efficacy; it also has the potential to harm patients and trust in health care. Academic and regulatory leaders have proposed a variety of principles and guidelines to address the challenges of evaluating the safety of machine learning-based software in the health care context, but accepted practices do not yet exist. However, there appears to be a shift toward process-based regulatory paradigms that rely heavily on self-regulation. At the same time, little research has examined the perspectives about the harms of MLPA developers themselves, whose role will be essential in overcoming the "principles-to-practice" gap.

OBJECTIVE

The objective of this study was to understand how MLPA developers of health care products perceived the potential harms of those products and their responses to recognized harms.

METHODS

We interviewed 40 individuals who were developing MLPA tools for health care at 15 US-based organizations, including data scientists, software engineers, and those with mid- and high-level management roles. These 15 organizations were selected to represent a range of organizational types and sizes from the 106 that we previously identified. We asked developers about their perspectives on the potential harms of their work, factors that influence these harms, and their role in mitigation. We used standard qualitative analysis of transcribed interviews to identify themes in the data.

RESULTS

We found that MLPA developers recognized a range of potential harms of MLPA to individuals, social groups, and the health care system, such as issues of privacy, bias, and system disruption. They also identified drivers of these harms related to the characteristics of machine learning and specific to the health care and commercial contexts in which the products are developed. MLPA developers also described strategies to respond to these drivers and potentially mitigate the harms. Opportunities included balancing algorithm performance goals with potential harms, emphasizing iterative integration of health care expertise, and fostering shared company values. However, their recognition of their own responsibility to address potential harms varied widely.

CONCLUSIONS

Even though MLPA developers recognized that their products can harm patients, public, and even health systems, robust procedures to assess the potential for harms and the need for mitigation do not exist. Our findings suggest that, to the extent that new oversight paradigms rely on self-regulation, they will face serious challenges if harms are driven by features that developers consider inescapable in health care and business environments. Furthermore, effective self-regulation will require MLPA developers to accept responsibility for safety and efficacy and know how to act accordingly. Our results suggest that, at the very least, substantial education will be necessary to fill the "principles-to-practice" gap.

Exploring patient perspectives on how they can and should be engaged in the development of artificial intelligence (AI) applications in health care

BACKGROUND

Artificial intelligence (AI) is a rapidly evolving field which will have implications on both individual patient care and the health care system. There are many benefits to the integration of AI into health care, such as predicting acute conditions and enhancing diagnostic capabilities. Despite these benefits potential harms include algorithmic bias, inadequate consent processes, and implications on the patient-provider relationship. One tool to address patients' needs and prevent the negative implications of AI is through patient engagement. As it currently stands, patients have infrequently been involved in AI application development for patient care delivery. Furthermore, we are unaware of any frameworks or recommendations specifically addressing patient engagement within the field of AI in health care.

METHODS

We conducted four virtual focus groups with thirty patient participants to understand of how patients can and should be meaningfully engaged within the field of AI development in health care. Participants completed an educational module on the fundamentals of AI prior to participating in this study. Focus groups were analyzed using qualitative content analysis.

RESULTS

We found that participants in our study wanted to be engaged at the problem-identification stages using multiple methods such as surveys and interviews. Participants preferred that recruitment methodologies for patient engagement included both in-person and social media-based approaches with an emphasis on varying language modalities of recruitment to reflect diverse demographics. Patients prioritized the inclusion of underrepresented participant populations, longitudinal relationship building, accessibility, and interdisciplinary involvement of other stakeholders in AI development. We found that AI education is a critical step to enable meaningful patient engagement within this field. We have curated recommendations into a framework for the field to learn from and implement in future development.

CONCLUSION

Given the novelty and speed at which AI innovation is progressing in health care, patient engagement should be the gold standard for application development. Our proposed recommendations seek to enable patient-centered AI application development in health care. Future research must be conducted to evaluate the effectiveness of patient engagement in AI application development to ensure that both AI application development and patient engagement are done rigorously, efficiently, and meaningfully.

Epistemic Rights and Responsibilities of Digital Simulacra for Biomedicine

Big data and AI have enabled digital simulation for prediction of future health states or behaviors of specific individuals, populations or humans in general. "Digital simulacra" use multimodal datasets to develop computational models that are virtual representations of people or groups, generating predictions of how systems evolve and react to interventions over time. These include digital twins and virtual patients for in silico clinical trials, both of which seek to transform research and health care by speeding innovation and bridging the epistemic gap between population-based research findings and their application to the individual. Nevertheless, digital simulacra mark a major milestone on a trajectory to embrace the epistemic culture of data science and a potential abandonment of medical epistemological concepts of causality and representation. In doing so, "data first" approaches potentially shift moral attention from actual patients and principles, such as equity, to simulated patients and patient data.

Prediction tool Development and Implementation in pharmacy praCTice (PreDICT) proposed guidance

PURPOSE

Proposed guidance is presented for Prediction tool Development and Implementation in pharmacy praCTice (PreDICT). This guidance aims to assist pharmacists and their collaborators with planning, developing, and implementing custom risk prediction tools for use by pharmacists in their own health systems or practice settings. We aimed to describe general considerations that would be relevant to most prediction tools designed for use in health systems or other pharmacy practice settings.

SUMMARY

The PreDICT proposed guidance is organized into 3 sequential phases: (1) planning, (2) development and validation, and (3) testing and refining prediction tools for real-world use. Each phase is accompanied by a checklist of considerations designed to be used by pharmacists or their trainees (eg, residents) during the planning or conduct of a prediction tool project. Commentary and a worked example are also provided to highlight some of the most relevant and impactful considerations for each phase.

CONCLUSION

The proposed guidance for PreDICT is a pharmacist-focused set of checklists for planning, developing, and implementing prediction tools in pharmacy practice. The list of considerations and accompanying commentary can be used as a reference by pharmacists or their trainees before or during the completion of a prediction tool project.

How can Big Data Analytics Support People-Centred and Integrated Health Services: A Scoping Review

Introduction:

Health systems in high-income countries face a variety of challenges calling for a systemic approach to improve quality and efficiency. Putting people in the centre is the main idea of the WHO model of people-centred and integrated health services. Integrating health services is fuelled by an integration of health data with great potentials for decision support based on big data analytics. The research question of this paper is “How can big data analytics support people-centred and integrated health services?”

Methods:

A scoping review following the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses – Scoping Review (PRISMA-ScR) statement was conducted to gather information on how big data analytics can support people-centred and integrated health services. The results were summarized in a role model of a people-centred and integrated health services platform illustrating which data sources might be integrated and which types of analytics might be applied to support the strategies of the people-centred and integrated health services framework to become more integrated across the continuum of care. Additional rapid literature reviews were conducted to generate frequency distributions of the most often used data types and analytical methods in the medical literature. Finally, the main challenges connected with big data analytics were worked out based on a content analysis of the results from the scoping literature review.

Results:

Based on the results from the rapid literature reviews the most often used data sources for big data analytics (BDA) in healthcare were biomarkers (39.3%) and medical images (30.9%). The most often used analytical models were support vector machines (27.3%) and neural networks (20.4%). The people-centred and integrated health services framework defines different strategic interventions for health services to become more integrated. To support all aspects of these interventions a comparably integrated platform of health-related data would be needed, so that a role model labelled as people-centred health platform was developed. Based on integrated data the results of the scoping review (n = 72) indicate, that big data analytics could for example support the strategic intervention of tailoring personalized health plans (43.1%), e.g. by predicting individual risk factors for different therapy options. Also BDA might enhance clinical decision support tools (31.9%), e.g. by calculating risk factors for disease uptake or progression. BDA might also assist in designing population-based services (26.4% by clustering comparable individuals in manageable risk groups e.g. mentored by specifically trained, non-medical professionals. The main challenges of big data analytics in healthcare were categorized in regulatory, (information-) technological, methodological, and cultural issues, whereas methodological challenges were mentioned most often (55.0%), followed by regulatory challenges (43.7%).

Discussion:

The BDA applications presented in this literature review are based on findings which have already been published. For some important components of the framework on people-centred care like enhancing the role of community care or establishing intersectoral partnerships between health and social care institutions only few examples of enabling big data analytical tools were found in the literature. Quite the opposite does this mean that these strategies have less potential value, but rather that the source systems in these fields need to be further developed to be suitable for big data analytics.

Conclusions:

Big data analytics can support people-centred and integrated health services e.g. by patient similarity stratifications or predictions of individual risk factors. But BDA fails to unfold its full potential until data source systems are still disconnected and actions towards a comprehensive and people-centred health-related data platform are politically insufficiently incentivized. This work highlighted the potential of big data analysis in the context of the model of people-centred and integrated health services, whereby the role model of the person-centered health platform can be used as a blueprint to support strategies to improve person-centered health care. Likely because health data is extremely sensitive and complex, there are only few practical examples of platforms to some extent already capable of merging and processing people-centred big data, but the integration of health data can be expected to further proceed so that analytical opportunities might also become reality in the near future.

Guidelines for artificial intelligence in medicine: A literature review and content analysis of frameworks (Preprint)

Background

Artificial intelligence (AI) is rapidly expanding in medicine despite a lack of consensus on its application and evaluation.

Objective

We sought to identify current frameworks guiding the application and evaluation of AI for predictive analytics in medicine and to describe the content of these frameworks. We also assessed what stages along the AI translational spectrum (ie, AI development, reporting, evaluation, implementation, and surveillance) the content of each framework has been discussed.

Methods

We performed a literature review of frameworks regarding the oversight of AI in medicine. The search included key topics such as “artificial intelligence,” “machine learning,” “guidance as topic,” and “translational science,” and spanned the time period 2014-2022. Documents were included if they provided generalizable guidance regarding the use or evaluation of AI in medicine. Included frameworks are summarized descriptively and were subjected to content analysis. A novel evaluation matrix was developed and applied to appraise the frameworks’ coverage of content areas across translational stages.

Results

Fourteen frameworks are featured in the review, including six frameworks that provide descriptive guidance and eight that provide reporting checklists for medical applications of AI. Content analysis revealed five considerations related to the oversight of AI in medicine across frameworks: transparency, reproducibility, ethics, effectiveness, and engagement. All frameworks include discussions regarding transparency, reproducibility, ethics, and effectiveness, while only half of the frameworks discuss engagement. The evaluation matrix revealed that frameworks were most likely to report AI considerations for the translational stage of development and were least likely to report considerations for the translational stage of surveillance.

Conclusions

Existing frameworks for the application and evaluation of AI in medicine notably offer less input on the role of engagement in oversight and regarding the translational stage of surveillance. Identifying and optimizing strategies for engagement are essential to ensure that AI can meaningfully benefit patients and other end users.

Electronic patient-reported outcomes and machine learning in predicting immune-related adverse events of immune checkpoint inhibitor therapies

BACKGROUND

Immune-checkpoint inhibitors (ICIs) have introduced novel immune-related adverse events (irAEs), arising from various organ systems without strong timely dependency on therapy dosing. Early detection of irAEs could result in improved toxicity profile and quality of life. Symptom data collected by electronic (e) patient-reported outcomes (PRO) could be used as an input for machine learning (ML) based prediction models for the early detection of irAEs.

METHODS

The utilized dataset consisted of two data sources. The first dataset consisted of 820 completed symptom questionnaires from 34 ICI treated advanced cancer patients, including 18 monitored symptoms collected using the Kaiku Health digital platform. The second dataset included prospectively collected irAE data, Common Terminology Criteria for Adverse Events (CTCAE) class, and the severity of 26 irAEs. The ML models were built using extreme gradient boosting algorithms. The first model was trained to detect the presence and the second the onset of irAEs.

RESULTS

The model trained to predict the presence of irAEs had an excellent performance based on four metrics: accuracy score 0.97, Area Under the Curve (AUC) value 0.99, F1-score 0.94 and Matthew's correlation coefficient (MCC) 0.92. The prediction of the irAE onset was more difficult with accuracy score 0.96, AUC value 0.93, F1-score 0.66 and MCC 0.64 but the model performance was still at a good level.

CONCLUSION

The current study suggests that ML based prediction models, using ePRO data as an input, can predict the presence and onset of irAEs with a high accuracy, indicating that ePRO follow-up with ML algorithms could facilitate the detection of irAEs in ICI-treated cancer patients. The results should be validated with a larger dataset. Trial registration Clinical Trials Register (NCT3928938), registration date the 26th of April, 2019.

Early renal function trajectories, cytomegalovirus serostatus and long-term graft outcomes in kidney transplant recipients

BACKGROUND

Improved recognition of factors influencing graft survival has led to better short-term kidney transplant outcomes. However, efforts to prevent long-term graft decline and improve graft survival have seen more modest improvements. The adoption of electronic health records has enabled better recording and identification of donor-recipient factors through the use of modern statistical techniques. We have previously shown in a prevalent renal transplant population that episodes of rapid deterioration are associated with graft loss.

METHODS

Estimated glomerular filtration rates (eGFR) between 3 and 27 months after transplantation were collected from 310 kidney transplant recipients. We utilised a Bayesian approach to estimate the most likely eGFR trajectory as a smooth curve from an average of 10,000 Monte Carlo samples. The probability of having an episode of rapid deterioration (decline greater than 5 ml/min/1.73 m2 per year in any 1-month period) was calculated. Graft loss and mortality data was collected over a median follow-up period of 8 years. Factors associated with having an episode of rapid deterioration and associations with long-term graft loss were explored.

RESULTS

In multivariable Cox Proportional Hazard analysis, a probability greater than 0.8 of rapid deterioration was associated with long-term death-censored graft loss (Hazard ratio 2.17; 95% Confidence intervals [CI] 1.04-4.55). In separate multivariable logistic regression models, cytomegalovirus (CMV) serostatus donor positive to recipient positive (Odds ratio [OR] 3.82; 95%CI 1.63-8.97), CMV donor positive (OR 2.06; 95%CI 1.15-3.68), and CMV recipient positive (OR 2.03; 95%CI 1.14-3.60) were associated with having a greater than 0.8 probability of an episode of rapid deterioration.

CONCLUSIONS

Early episodes of rapid deterioration are associated with long-term death-censored graft loss and are associated with cytomegalovirus seropositivity. Further study is required to better manage these potentially modifiable risks factors and improve long-term graft survival.

Development of Electronic Health Record-Based Prediction Models for 30-Day Readmission Risk Among Patients Hospitalized for Acute Myocardial Infarction

IMPORTANCE

In the US, more than 600 000 adults will experience an acute myocardial infarction (AMI) each year, and up to 20% of the patients will be rehospitalized within 30 days. This study highlights the need for consideration of calibration in these risk models.

OBJECTIVE

To compare multiple machine learning risk prediction models using an electronic health record (EHR)-derived data set standardized to a common data model.

DESIGN, SETTING, AND PARTICIPANTS

This was a retrospective cohort study that developed risk prediction models for 30-day readmission among all inpatients discharged from Vanderbilt University Medical Center between January 1, 2007, and December 31, 2016, with a primary diagnosis of AMI who were not transferred from another facility. The model was externally validated at Dartmouth-Hitchcock Medical Center from April 2, 2011, to December 31, 2016. Data analysis occurred between January 4, 2019, and November 15, 2020.

EXPOSURES

Acute myocardial infarction that required hospital admission.

MAIN OUTCOMES AND MEASURES

The main outcome was thirty-day hospital readmission. A total of 141 candidate variables were considered from administrative codes, medication orders, and laboratory tests. Multiple risk prediction models were developed using parametric models (elastic net, least absolute shrinkage and selection operator, and ridge regression) and nonparametric models (random forest and gradient boosting). The models were assessed using holdout data with area under the receiver operating characteristic curve (AUROC), percentage of calibration, and calibration curve belts.

RESULTS

The final Vanderbilt University Medical Center cohort included 6163 unique patients, among whom the mean (SD) age was 67 (13) years, 4137 were male (67.1%), 1019 (16.5%) were Black or other race, and 933 (15.1%) were rehospitalized within 30 days. The final Dartmouth-Hitchcock Medical Center cohort included 4024 unique patients, with mean (SD) age of 68 (12) years; 2584 (64.2%) were male, 412 (10.2%) were rehospitalized within 30 days, and most of the cohort were non-Hispanic and White. The final test set AUROC performance was between 0.686 to 0.695 for the parametric models and 0.686 to 0.704 for the nonparametric models. In the validation cohort, AUROC performance was between 0.558 to 0.655 for parametric models and 0.606 to 0.608 for nonparametric models.

CONCLUSIONS AND RELEVANCE

In this study, 5 machine learning models were developed and externally validated to predict 30-day readmission AMI hospitalization. These models can be deployed within an EHR using routinely collected data.

Identifying Ethical Considerations for Machine Learning Healthcare Applications

Along with potential benefits to healthcare delivery, machine learning healthcare applications (ML-HCAs) raise a number of ethical concerns. Ethical evaluations of ML-HCAs will need to structure the overall problem of evaluating these technologies, especially for a diverse group of stakeholders. This paper outlines a systematic approach to identifying ML-HCA ethical concerns, starting with a conceptual model of the pipeline of the conception, development, implementation of ML-HCAs, and the parallel pipeline of evaluation and oversight tasks at each stage. Over this model, we layer key questions that raise value-based issues, along with ethical considerations identified in large part by a literature review, but also identifying some ethical considerations that have yet to receive attention. This pipeline model framework will be useful for systematic ethical appraisals of ML-HCA from development through implementation, and for interdisciplinary collaboration of diverse stakeholders that will be required to understand and subsequently manage the ethical implications of ML-HCAs.

Including Social and Behavioral Determinants in Predictive Models: Trends, Challenges, and Opportunities

In an era of accelerated health information technology capability, health care organizations increasingly use digital data to predict outcomes such as emergency department use, hospitalizations, and health care costs. This trend occurs alongside a growing recognition that social and behavioral determinants of health (SBDH) influence health and medical care use. Consequently, health providers and insurers are starting to incorporate new SBDH data sources into a wide range of health care prediction models, although existing models that use SBDH variables have not been shown to improve health care predictions more than models that use exclusively clinical variables. In this viewpoint, we review the rationale behind the push to integrate SBDH data into health care predictive models and explore the technical, strategic, and ethical challenges faced as this process unfolds across the United States. We also offer several recommendations to overcome these challenges to reach the promise of SBDH predictive analytics to improve health and reduce health care disparities.

Examination and diagnosis of electronic patient records and their associated ethics: a scoping literature review

Background

Electronic patient record (EPR) technology is a key enabler for improvements to healthcare service and management. To ensure these improvements and the means to achieve them are socially and ethically desirable, careful consideration of the ethical implications of EPRs is indicated. The purpose of this scoping review was to map the literature related to the ethics of EPR technology. The literature review was conducted to catalogue the prevalent ethical terms, to describe the associated ethical challenges and opportunities, and to identify the actors involved. By doing so, it aimed to support the future development of ethics guidance in the EPR domain.

Methods

To identify journal articles debating the ethics of EPRs, Scopus, Web of Science, and PubMed academic databases were queried and yielded 123 eligible articles. The following inclusion criteria were applied: articles need to be in the English language; present normative arguments and not solely empirical research; include an abstract for software analysis; and discuss EPR technology.

Results

The medical specialty, type of information captured and stored in EPRs, their use and functionality varied widely across the included articles. Ethical terms extracted were categorised into clusters ‘privacy’, ‘autonomy’, ‘risk/benefit’, ‘human relationships’, and ‘responsibility’. The literature shows that EPR-related ethical concerns can have both positive and negative implications, and that a wide variety of actors with rights and/or responsibilities regarding the safe and ethical adoption of the technology are involved.

Conclusions

While there is considerable consensus in the literature regarding EPR-related ethical principles, some of the associated challenges and opportunities remain underdiscussed. For example, much of the debate is presented in a manner more in keeping with a traditional model of healthcare and fails to take account of the multidimensional ensemble of factors at play in the EPR era and the consequent need to redefine/modify ethical norms to align with a digitally-enabled health service. Similarly, the academic discussion focuses predominantly on bioethical values. However, approaches from digital ethics may also be helpful to identify and deliberate about current and emerging EPR-related ethical concerns.

Integrating machine-generated mortality estimates and behavioral nudges to promote serious illness conversations for cancer patients: Design and methods for a stepped-wedge cluster randomized controlled trial

INTRODUCTION

Patients with cancer often receive care that is not aligned with their personal values and goals. Serious illness conversations (SICs) between clinicians and patients can help increase a patient's understanding of their prognosis, goals and values.

METHODS AND ANALYSIS

In this study, we describe the design of a stepped-wedge cluster randomized trial to evaluate the impact of an intervention that employs machine learning-based prognostic algorithms and behavioral nudges to prompt oncologists to have SICs with patients at high risk of short-term mortality. Data are collected on documented SICs, documented advance care planning discussions, and end-of-life care utilization (emergency room and inpatient admissions, chemotherapy and hospice utilization) for patients of all enrolled clinicians.

CONCLUSION

This trial represents a novel application of machine-generated mortality predictions combined with behavioral nudges in the routine care of outpatients with cancer. Findings from the trial may inform strategies to encourage early serious illness conversations and the application of mortality risk predictions in clinical settings.

TRIAL REGISTRATION

Clinicaltrials.gov Identifier: NCT03984773.

Transformation of the Doctor-Patient Relationship: Big Data, Accountable Care, and Predictive Health Analytics

The medical profession is steeped in traditions that guide its practice. These traditions were developed to preserve the well-being of patients. Transformations in science, technology, and society, while maintaining a self-governance structure that drives the goal of care provision, have remained hallmarks of the profession. The purpose of this paper is to examine ethical challenges in health care as it relates to Big Data, Accountable Care Organizations, and Health Care Predictive Analytics using the principles of biomedical ethics laid out by Beauchamp and Childress (autonomy, beneficence, non-maleficence, and justice). Among these are the use of Electronic Health Records within stipulations of the Health Insurance Portability and Accountability Act. Clinicians are well-positioned to impact health policy development to address ethical issues associated with the use of Big Data, Accountable Care, and Health Care Predictive Analytics as we work to transform the doctor-patient relationship towards improving population health outcomes and creating a healthier society.

Unstructured Text in EMR Improves Prediction of Death after Surgery in Children

Text fields in electronic medical records (EMR) contain information on important factors that influence health outcomes, however, they are underutilized in clinical decision making due to their unstructured nature. We analyzed 6497 inpatient surgical cases with 719,308 free text notes from Le Bonheur Children’s Hospital EMR. We used a text mining approach on preoperative notes to obtain a text-based risk score to predict death within 30 days of surgery. In addition, we evaluated the performance of a hybrid model that included the text-based risk score along with structured data pertaining to clinical risk factors. The C-statistic of a logistic regression model with five-fold cross-validation significantly improved from 0.76 to 0.92 when text-based risk scores were included in addition to structured data. We conclude that preoperative free text notes in EMR include significant information that can predict adverse surgery outcomes.

Secondary use of routine data in hospitals: description of a scalable analytical platform based on a business intelligence system

We describe a scalable platform for research-oriented analyses of routine data in hospitals, which evolved from a state-of-the-art business intelligence architecture for enterprise resource planning. This platform involves an in-memory database management system for data modeling and analytics and a high-performance cluster for more computing-intensive analytical tasks. Setting up platforms for research-oriented analyses is a highly dynamic, time-consuming, and costly process. In some health care institutions, effective research platforms may be derived from existing business intelligence systems.

American Joint Replacement Registry Risk Calculator Does Not Predict 90-day Mortality in Veterans Undergoing Total Joint Replacement

BACKGROUND

The American Joint Replacement Registry (AJRR) Total Joint Risk Calculator uses demographic and clinical parameters to provide risk estimates for 90-day mortality and 2-year periprosthetic joint infection (PJI). The tool is intended to help surgeons counsel their Medicare-eligible patients about their risk of death and PJI after total joint arthroplasty (TJA). However, for a predictive risk model to be useful, it must be accurate when applied to new patients; this has yet to be established for this calculator.

QUESTIONS/PURPOSES

To produce accuracy metrics (ie, discrimination, calibration) for the AJRR mortality calculator using data from Medicare-eligible patients undergoing TJA in the Veterans Health Administration (VHA), the largest integrated healthcare system in the United States, where more than 10,000 TJAs are performed annually.

METHODS

We used the AJRR calculator to predict risk of death within 90 days of surgery among 31,214 VHA patients older than 64 years of age who underwent primary TJA; data was drawn from the Veterans Affairs Surgical Quality Improvement Project (VASQIP) and VA Corporate Data Warehouse (CDW). We then used VHA mortality data to evaluate the extent to which the AJRR calculator estimates distinguished individuals who died compared with those who did not (C-statistic), and graphically depicted the relationship between estimated risk and observed mortality (calibration). As a secondary evaluation of the calculator, a sample of 39,300 patients younger than 65 years old was assigned to the youngest age group available to the user (65-69 years) as might be done in real-world practice.

RESULTS

C-statistics for 90-day mortality for the older samples were 0.62 (95% CI, 0.60-0.64) and for the younger samples they were 0.46 (95% CI, 0.43-0.49), suggesting poor discrimination. Calibration analysis revealed poor correspondence between deciles of predicted risk and observed mortality rates. Poor discrimination and calibration mean that patients who died will frequently have a lower estimated risk of death than surviving patients.

CONCLUSIONS

For Medicare-eligible patients receiving TJA in the VA, the AJRR risk calculator had a poor performance in the prediction of 90-day mortality. There are several possible reasons for the model's poor performance. Veterans Health Administration patients, 97% of whom were men, represent only a subset of the broader Medicare population. However, applying the calculator to a subset of the target population should not affect its accuracy. Other reasons for poor performance include a lack of an underlying statistical model in the calculator's implementation and simply the challenge of predicting rare events. External validation in a more representative sample of Medicare patients should be conducted to before assuming this tool is accurate for its intended use.

LEVEL OF EVIDENCE

Level I, diagnostic study.

Numeracy and Understanding of Quantitative Aspects of Predictive Models: A Pilot Study

BACKGROUND

The assessment of user preferences for performance characteristics of patient-oriented clinical prediction models is lacking. It is unknown if complex statistical aspects of prediction models are readily understandable by a general audience.

OBJECTIVE

A pilot study was conducted among nonclinical audiences to determine the feasibility of interpreting statistical concepts that describe the performance of prediction models.

METHODS

We conducted a cross-sectional electronic survey using the Amazon Mechanical Turk platform. The survey instrument included educational modules about predictive models, sensitivity, specificity, and confidence intervals (CIs). Follow-up questions tested participants' abilities to interpret these characteristics with both verbatim and gist knowledge. Objective and subjective numeracy were assessed using previously validated instruments. We also tested understanding of these concepts when embedded in a sample discrete choice experiment task to establish feasibility for future elicitation of preferences using a discrete choice experiment design. Multivariable linear regression was used to identify factors associated with correct interpretation of statistical concepts.

RESULTS

Among 534 respondents who answered all nine questions, the mean correct responses was 95.9% (95% CI, 93.8-97.4) for sensitivity, 93.1% (95% CI, 90.5-95.0) for specificity, and 86.6% (95% CI, 83.3-89.3) for CIs. Verbatim interpretation was high for all concepts, but significantly higher than gist only for CIs (p < 0.001). Scores on each discrete choice experiment tasks were slightly lower in each category. Both objective and subjective numeracy were positively associated with an increased proportion of correct responses (p < 0.001).

CONCLUSION

These results suggest that a nonclinical audience can interpret quantitative performance measures of predictive models with very high accuracy. Future development of patient-facing clinical prediction models can feasibly incorporate patient preferences for model features into their development.

Inclusion of Unstructured Clinical Text Improves Early Prediction of Death or Prolonged ICU Stay

OBJECTIVES

Early prediction of undesired outcomes among newly hospitalized patients could improve patient triage and prompt conversations about patients' goals of care. We evaluated the performance of logistic regression, gradient boosting machine, random forest, and elastic net regression models, with and without unstructured clinical text data, to predict a binary composite outcome of in-hospital death or ICU length of stay greater than or equal to 7 days using data from the first 48 hours of hospitalization.

DESIGN

Retrospective cohort study with split sampling for model training and testing.

SETTING

A single urban academic hospital.

PATIENTS

All hospitalized patients who required ICU care at the Beth Israel Deaconess Medical Center in Boston, MA, from 2001 to 2012.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Among eligible 25,947 hospital admissions, we observed 5,504 (21.2%) in which patients died or had ICU length of stay greater than or equal to 7 days. The gradient boosting machine model had the highest discrimination without (area under the receiver operating characteristic curve, 0.83; 95% CI, 0.81-0.84) and with (area under the receiver operating characteristic curve, 0.89; 95% CI, 0.88-0.90) text-derived variables. Both gradient boosting machines and random forests outperformed logistic regression without text data (p < 0.001), whereas all models outperformed logistic regression with text data (p < 0.02). The inclusion of text data increased the discrimination of all four model types (p < 0.001). Among those models using text data, the increasing presence of terms "intubated" and "poor prognosis" were positively associated with mortality and ICU length of stay, whereas the term "extubated" was inversely associated with them.

CONCLUSIONS

Variables extracted from unstructured clinical text from the first 48 hours of hospital admission using natural language processing techniques significantly improved the abilities of logistic regression and other machine learning models to predict which patients died or had long ICU stays. Learning health systems may adapt such models using open-source approaches to capture local variation in care patterns.

Prediction Models for 30-Day Mortality and Complications After Total Knee and Hip Arthroplasties for Veteran Health Administration Patients With Osteoarthritis

BACKGROUND

Statistical models to preoperatively predict patients' risk of death and major complications after total joint arthroplasty (TJA) could improve the quality of preoperative management and informed consent. Although risk models for TJA exist, they have limitations including poor transparency and/or unknown or poor performance. Thus, it is currently impossible to know how well currently available models predict short-term complications after TJA, or if newly developed models are more accurate. We sought to develop and conduct cross-validation of predictive risk models, and report details and performance metrics as benchmarks.

METHODS

Over 90 preoperative variables were used as candidate predictors of death and major complications within 30 days for Veterans Health Administration patients with osteoarthritis who underwent TJA. Data were split into 3 samples-for selection of model tuning parameters, model development, and cross-validation. C-indexes (discrimination) and calibration plots were produced.

RESULTS

A total of 70,569 patients diagnosed with osteoarthritis who received primary TJA were included. C-statistics and bootstrapped confidence intervals for the cross-validation of the boosted regression models were highest for cardiac complications (0.75; 0.71-0.79) and 30-day mortality (0.73; 0.66-0.79) and lowest for deep vein thrombosis (0.59; 0.55-0.64) and return to the operating room (0.60; 0.57-0.63).

CONCLUSIONS

Moderately accurate predictive models of 30-day mortality and cardiac complications after TJA in Veterans Health Administration patients were developed and internally cross-validated. By reporting model coefficients and performance metrics, other model developers can test these models on new samples and have a procedure and indication-specific benchmark to surpass.

Calibration drift in regression and machine learning models for acute kidney injury

Objective

Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.

Materials and Methods

Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.

Results

Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.

Conclusions

Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Uses and Misuses of Patient- and Neighborhood-level Social Determinants of Health Data

Health care leaders in the US are actively exploring strategies to identify and address patients' social and economic hardships as part of high-quality clinical care. The result has been a proliferation of screening tools and interventions related to patients' social determinants of health, but little guidance on effective strategies to implement them. Some of these tools rely on patient- or household-level screening data collected from patients during medical encounters. Other tools rely on data available at the neighborhood-level that can be used to characterize the environment in which patients live or to approximate patients' social or economic risks. Four case examples were selected from different health care organizations to illustrate strengths and limitations of using patient- or neighborhood-level social and economic needs data to inform a range of interventions. This work can guide health care investments in this rapidly evolving arena.

Development and Validation of a Multi-Algorithm Analytic Platform to Detect Off-Target Mechanical Ventilation

Healthcare-specific analytic software is needed to process the large volumes of streaming physiologic waveform data increasingly available from life support devices such as mechanical ventilators. Detection of clinically relevant events from these data streams will advance understanding of critical illness, enable real-time clinical decision support, and improve both clinical outcomes and patient experience. We used mechanical ventilation waveform data (VWD) as a use case to address broader issues of data access and analysis including discrimination between true events and waveform artifacts. We developed an open source data acquisition platform to acquire VWD, and a modular, multi-algorithm analytic platform (ventMAP) to enable automated detection of off-target ventilation (OTV) delivery in critically-ill patients. We tested the hypothesis that use of artifact correction logic would improve the specificity of clinical event detection without compromising sensitivity. We showed that ventMAP could accurately detect harmful forms of OTV including excessive tidal volumes and common forms of patient-ventilator asynchrony, and that artifact correction significantly improved the specificity of event detection without decreasing sensitivity. Our multi-disciplinary approach has enabled automated analysis of high-volume streaming patient waveform data for clinical and translational research, and will advance the study and management of critically ill patients requiring mechanical ventilation.

Machine learning approaches to personalize early prediction of asthma exacerbations

Patient telemonitoring results in an aggregation of significant amounts of information about patient disease trajectory. However, the potential use of this information for early prediction of exacerbations in adult asthma patients has not been systematically evaluated. The aim of this study was to explore the utility of telemonitoring data for building machine learning algorithms that predict asthma exacerbations before they occur. The study dataset comprised daily self-monitoring reports consisting of 7001 records submitted by adult asthma patients during home telemonitoring. Predictive modeling included preparation of stratified training datasets, predictive feature selection, and evaluation of resulting classifiers. Using a 7-day window, a naive Bayesian classifier, adaptive Bayesian network, and support vector machines were able to predict asthma exacerbation occurring on day 8, with sensitivity of 0.80, 1.00, and 0.84; specificity of 0.77, 1.00, and 0.80; and accuracy of 0.77, 1.00, and 0.80, respectively. Our study demonstrated that machine learning techniques have significant potential in developing personalized decision support for chronic disease telemonitoring systems. Future studies may benefit from a comprehensive predictive framework that combines telemonitoring data with other factors affecting the likelihood of developing acute exacerbation. Approaches implemented for advanced asthma exacerbation prediction may be extended to prediction of exacerbations in patients with other chronic health conditions.