A literature review of quality assessment and applicability to HTA of risk prediction models of coronary heart disease in patients with diabetes

This literature review had two objectives: to identify models for predicting the risk of coronary heart diseases in patients with diabetes (DM); and to assess model quality in terms of risk of bias (RoB) and applicability for the purpose of health technology assessment (HTA). We undertook a targeted review of journal articles published in English, Dutch, Chinese, or Spanish in 5 databases from 1st January 2016 to 18th December 2022, and searched three systematic reviews for the models published after 2012. We used PROBAST (Prediction model Risk Of Bias Assessment Tool) to assess RoB, and used findings from Betts et al. 2019, which summarized recommendations and criticisms of HTA agencies on cardiovascular risk prediction models, to assess model applicability for the purpose of HTA. As a result, 71 % and 67 % models reporting C-index showed good discrimination abilities (C-index >= 0.7). Of the 26 model studies and 30 models identified, only one model study showed low RoB in all domains, and no model was fully applicable for HTA. Since the major cause of high RoB is inappropriate use of analysis method, we advise clinicians to carefully examine the model performance declared by model developers, and to trust a model if all PROBAST domains except analysis show low RoB and at least one validation study conducted in the same setting (e.g. country) is available. Moreover, since general model applicability is not informative for HTA, novel adapted tools may need to be developed.

Consensus Statement on Electronic Health Predictive Analytics: A Guiding Framework to Address Challenges

Context:

The recent explosion in available electronic health record (EHR) data is motivating a rapid expansion of electronic health care predictive analytic (e-HPA) applications, defined as the use of electronic algorithms that forecast clinical events in real time with the intent to improve patient outcomes and reduce costs. There is an urgent need for a systematic framework to guide the development and application of e-HPA to ensure that the field develops in a scientifically sound, ethical, and efficient manner.

Objectives:

Building upon earlier frameworks of model development and utilization, we identify the emerging opportunities and challenges of e-HPA, propose a framework that enables us to realize these opportunities, address these challenges, and motivate e-HPA stakeholders to both adopt and continuously refine the framework as the applications of e-HPA emerge.

Methods:

To achieve these objectives, 17 experts with diverse expertise including methodology, ethics, legal, regulation, and health care delivery systems were assembled to identify emerging opportunities and challenges of e-HPA and to propose a framework to guide the development and application of e-HPA.

Findings:

The framework proposed by the panel includes three key domains where e-HPA differs qualitatively from earlier generations of models and algorithms (Data Barriers, Transparency, and Ethics) and areas where current frameworks are insufficient to address the emerging opportunities and challenges of e-HPA (Regulation and Certification; and Education and Training). The following list of recommendations summarizes the key points of the framework:

Data Barriers: Establish mechanisms within the scientific community to support data sharing for predictive model development and testing.

Transparency: Set standards around e-HPA validation based on principles of scientific transparency and reproducibility.

Ethics: Develop both individual-centered and society-centered risk-benefit approaches to evaluate e-HPA.

Regulation and Certification: Construct a self-regulation and certification framework within e-HPA.

Education and Training: Make significant changes to medical, nursing, and paraprofessional curricula by including training for understanding, evaluating, and utilizing predictive models.