Measuring performance in health care: case-mix adjustment by boosted decision trees

Opportunities and Challenges of Prognostic Models for Extremely Preterm Infants

Predicting the short- and long-term outcomes of extremely preterm infants remains a challenge. Multivariable prognostic models might be valuable tools for clinicians, parents, and policymakers for providing accurate outcome estimates. In this perspective, we discuss the opportunities and challenges of using prognostic models in extremely preterm infants at population and individual levels. At a population level, these models could support the development of guidelines for decisions about treatment limits and may support policy processes such as benchmarking and resource allocation. At an individual level, these models may enhance prenatal counselling conversations by considering multiple variables and improving transparency about expected outcomes. Furthermore, they may improve consistency in projections shared with parents. For the development of prognostic models, we discuss important considerations such as predictor and outcome measure selection, clinical impact assessment, and generalizability. Lastly, future recommendations for developing and using prognostic models are suggested. Importantly, the purpose of a prognostic model should be clearly defined, and integrating these models into prenatal counselling requires thoughtful consideration.

Predicting Suicidal Behavior Without Asking About Suicidal Ideation: Machine Learning and the Role of Borderline Personality Disorder Criteria

OBJECTIVE

Identifying predictors contributing to suicide risk could help prevent suicides via targeted interventions. However, using only known risk factors may not yield accurate enough results. Furthermore, risk models typically rely on suicidal ideation, even though people often withhold this information.

METHOD

This study examined the contribution of various predictors to the accuracy of six machine learning models for identifying suicidal behavior in a prison population (n = 353), including borderline personality disorder (BPD) and antisocial personality disorder (APD) criteria, and compared how excluding data about suicidal ideation affects accuracy.

RESULTS

Results revealed that gradient tree boosting accurately identified individuals with suicidal behavior, even without relying on questions about suicidal ideation (AUC = 0.875, F1 = 0.846). Furthermore, the model maintained this accuracy with only 29 predictors. Meeting five or more diagnostic criteria of BPD was an important risk factor for suicidal behavior. APD criteria, in the presence of other predictors, did not substantially improve accuracy. Additionally, it may be possible to implement a decision tree model to assess individuals at risk of suicide, without focusing upon suicidal ideation.

CONCLUSIONS

These findings highlight that modern classification algorithms do not necessarily require information about suicidal ideation for modeling suicide and self-harm behavior.

Data mining techniques utilizing latent class models to evaluate emergency department revisits

BACKGROUND

The use of machine learning techniques is especially pertinent to the composite and challenging conditions of emergency departments (EDs). Repeat ED visits (i.e. revisits) are an example of potentially inappropriate utilization of resources that can be forecasted by these techniques.

OBJECTIVE

To track the ED revisit risk over time using the hidden Markov model (HMM) as a major latent class model. Given the HMM states, we carried out forecasting of future ED revisits with various data mining models.

METHODS

Information integrated from four distributed sources (e.g. electronic health records and health information exchange) was integrated into four HMMs which capture the relationships between an observed and a hidden progression that shift over time through a series of hidden states in an adult patient population.

RESULTS

Assimilating a pre-analysis of the various patients by applying latent class models and directing them to well-known classifiers functioned well. The performance was significantly better than without utilizing pre-analysis of HMM for all prediction models (classifiers(.

CONCLUSIONS

These findings suggest that one prospective approach to advanced risk prediction is to leverage the longitudinal nature of health care data by exploiting patients' between state variation.

Prioritizing Highway Safety Manual's crash prediction variables using boosted regression trees

The Highway Safety Manual (HSM) recommends using the empirical Bayes (EB) method with locally derived calibration factors to predict an agency's safety performance. However, the data needs for deriving these local calibration factors are significant, requiring very detailed roadway characteristics information. Many of the data variables identified in the HSM are currently unavailable in the states' databases. Moreover, the process of collecting and maintaining all the HSM data variables is cost-prohibitive. Prioritization of the variables based on their impact on crash predictions would, therefore, help to identify influential variables for which data could be collected and maintained for continued updates. This study aims to determine the impact of each independent variable identified in the HSM on crash predictions. A relatively recent data mining approach called boosted regression trees (BRT) is used to investigate the association between the variables and crash predictions. The BRT method can effectively handle different types of predictor variables, identify very complex and non-linear association among variables, and compute variable importance. Five years of crash data from 2008 to 2012 on two urban and suburban facility types, two-lane undivided arterials and four-lane divided arterials, were analyzed for estimating the influence of variables on crash predictions. Variables were found to exhibit non-linear and sometimes complex relationship to predicted crash counts. In addition, only a few variables were found to explain most of the variation in the crash data.

enDNA-Prot: identification of DNA-binding proteins by applying ensemble learning

DNA-binding proteins are crucial for various cellular processes, such as recognition of specific nucleotide, regulation of transcription, and regulation of gene expression. Developing an effective model for identifying DNA-binding proteins is an urgent research problem. Up to now, many methods have been proposed, but most of them focus on only one classifier and cannot make full use of the large number of negative samples to improve predicting performance. This study proposed a predictor called enDNA-Prot for DNA-binding protein identification by employing the ensemble learning technique. Experiential results showed that enDNA-Prot was comparable with DNA-Prot and outperformed DNAbinder and iDNA-Prot with performance improvement in the range of 3.97–9.52% in ACC and 0.08–0.19 in MCC. Furthermore, when the benchmark dataset was expanded with negative samples, the performance of enDNA-Prot outperformed the three existing methods by 2.83–16.63% in terms of ACC and 0.02–0.16 in terms of MCC. It indicated that enDNA-Prot is an effective method for DNA-binding protein identification and expanding training dataset with negative samples can improve its performance. For the convenience of the vast majority of experimental scientists, we developed a user-friendly web-server for enDNA-Prot which is freely accessible to the public.

Performance evaluation of public non-profit hospitals using a BP artificial neural network: the case of Hubei Province in China

To provide a reference for evaluating public non-profit hospitals in the new environment of medical reform, we established a performance evaluation system for public non-profit hospitals. The new “input-output” performance model for public non-profit hospitals is based on four primary indexes (input, process, output and effect) that include 11 sub-indexes and 41 items. The indicator weights were determined using the analytic hierarchy process (AHP) and entropy weight method. The BP neural network was applied to evaluate the performance of 14 level-3 public non-profit hospitals located in Hubei Province. The most stable BP neural network was produced by comparing different numbers of neurons in the hidden layer and using the “Leave-one-out” Cross Validation method. The performance evaluation system we established for public non-profit hospitals could reflect the basic goal of the new medical health system reform in China. Compared with PLSR, the result indicated that the BP neural network could be used effectively for evaluating the performance public non-profit hospitals.

Applications of New Technologies and New Methods in ZHENG Differentiation

With the hope to provide an effective approach for personalized diagnosis and treatment clinically, Traditional Chinese Medicine (TCM) is being paid increasing attention as a complementary and alternative medicine. It performs treatment based on ZHENG (TCM syndrome) differentiation, which could be identified as clinical special phenotypes by symptoms and signs of patients. However, it caused skepticism and criticism because ZHENG classification only depends on observation, knowledge, and clinical experience of TCM practitioners, which is lack of objectivity and repeatability. Scientists have done fruitful researches for its objectivity and standardization. Compared with traditional four diagnostic methods (looking, listening and smelling, asking, and touching), in this paper, the applications of new technologies and new methods on the ZHENG differentiation were systemically reviewed, including acquisition, analysis, and integration of clinical data or information. Furthermore, the characteristics and application range of these technologies and methods were summarized. It will provide reference for further researches.

iDNA-Prot: identification of DNA binding proteins using random forest with grey model

DNA-binding proteins play crucial roles in various cellular processes. Developing high throughput tools for rapidly and effectively identifying DNA-binding proteins is one of the major challenges in the field of genome annotation. Although many efforts have been made in this regard, further effort is needed to enhance the prediction power.

By incorporating the features into the general form of pseudo amino acid composition that were extracted from protein sequences via the “grey model” and by adopting the random forest operation engine, we proposed a new predictor, called iDNA-Prot, for identifying uncharacterized proteins as DNA-binding proteins or non-DNA binding proteins based on their amino acid sequences information alone. The overall success rate by iDNA-Prot was 83.96% that was obtained via jackknife tests on a newly constructed stringent benchmark dataset in which none of the proteins included has pairwise sequence identity to any other in a same subset. In addition to achieving high success rate, the computational time for iDNA-Prot is remarkably shorter in comparison with the relevant existing predictors. Hence it is anticipated that iDNA-Prot may become a useful high throughput tool for large-scale analysis of DNA-binding proteins.

As a user-friendly web-server, iDNA-Prot is freely accessible to the public at the web-site on http://icpr.jci.edu.cn/bioinfo/iDNA-Prot or http://www.jci-bioinfo.cn/iDNA-Prot. Moreover, for the convenience of the vast majority of experimental scientists, a step-by-step guide is provided on how to use the web-server to get the desired results.

Multilevel latent class casemix modelling: a novel approach to accommodate patient casemix

Background

Using routinely collected patient data we explore the utility of multilevel latent class (MLLC) models to adjust for patient casemix and rank Trust performance. We contrast this with ranks derived from Trust standardised mortality ratios (SMRs).

Methods

Patients with colorectal cancer diagnosed between 1998 and 2004 and resident in Northern and Yorkshire regions were identified from the cancer registry database (n = 24,640). Patient age, sex, stage-at-diagnosis (Dukes), and Trust of diagnosis/treatment were extracted. Socioeconomic background was derived using the Townsend Index. Outcome was survival at 3 years after diagnosis. MLLC-modelled and SMR-generated Trust ranks were compared.

Results

Patients were assigned to two classes of similar size: one with reasonable prognosis (63.0% died within 3 years), and one with better prognosis (39.3% died within 3 years). In patient class one, all patients diagnosed at stage B or C died within 3 years; in patient class two, all patients diagnosed at stage A, B or C survived. Trusts were assigned two classes with 51.3% and 53.2% of patients respectively dying within 3 years. Differences in the ranked Trust performance between the MLLC model and SMRs were all within estimated 95% CIs.

Conclusions

A novel approach to casemix adjustment is illustrated, ranking Trust performance whilst facilitating the evaluation of factors associated with the patient journey (e.g. treatments) and factors associated with the processes of healthcare delivery (e.g. delays). Further research can demonstrate the value of modelling patient pathways and evaluating healthcare processes across provider institutions.

Multilevel latent class casemix modelling: a novel approach to accommodate patient casemix

BACKGROUND

Using routinely collected patient data we explore the utility of multilevel latent class (MLLC) models to adjust for patient casemix and rank Trust performance. We contrast this with ranks derived from Trust standardised mortality ratios (SMRs).

METHODS

Patients with colorectal cancer diagnosed between 1998 and 2004 and resident in Northern and Yorkshire regions were identified from the cancer registry database (n = 24,640). Patient age, sex, stage-at-diagnosis (Dukes), and Trust of diagnosis/treatment were extracted. Socioeconomic background was derived using the Townsend Index. Outcome was survival at 3 years after diagnosis. MLLC-modelled and SMR-generated Trust ranks were compared.

RESULTS

Patients were assigned to two classes of similar size: one with reasonable prognosis (63.0% died within 3 years), and one with better prognosis (39.3% died within 3 years). In patient class one, all patients diagnosed at stage B or C died within 3 years; in patient class two, all patients diagnosed at stage A, B or C survived. Trusts were assigned two classes with 51.3% and 53.2% of patients respectively dying within 3 years. Differences in the ranked Trust performance between the MLLC model and SMRs were all within estimated 95% CIs.

CONCLUSIONS

A novel approach to casemix adjustment is illustrated, ranking Trust performance whilst facilitating the evaluation of factors associated with the patient journey (e.g. treatments) and factors associated with the processes of healthcare delivery (e.g. delays). Further research can demonstrate the value of modelling patient pathways and evaluating healthcare processes across provider institutions.

Performance evaluation of inpatient service in Beijing: a horizontal comparison with risk adjustment based on Diagnosis Related Groups

BACKGROUND

The medical performance evaluation, which provides a basis for rational decision-making, is an important part of medical service research. Current progress with health services reform in China is far from satisfactory, without sufficient regulation. To achieve better progress, an effective tool for evaluating medical performance needs to be established. In view of this, this study attempted to develop such a tool appropriate for the Chinese context.

METHODS

Data was collected from the front pages of medical records (FPMR) of all large general public hospitals (21 hospitals) in the third and fourth quarter of 2007. Locally developed Diagnosis Related Groups (DRGs) were introduced as a tool for risk adjustment and performance evaluation indicators were established: Charge Efficiency Index (CEI), Time Efficiency Index (TEI) and inpatient mortality of low-risk group cases (IMLRG), to reflect respectively work efficiency and medical service quality. Using these indicators, the inpatient services' performance was horizontally compared among hospitals. Case-mix Index (CMI) was used to adjust efficiency indices and then produce adjusted CEI (aCEI) and adjusted TEI (aTEI). Poisson distribution analysis was used to test the statistical significance of the IMLRG differences between different hospitals.

RESULTS

Using the aCEI, aTEI and IMLRG scores for the 21 hospitals, Hospital A and C had relatively good overall performance because their medical charges were lower, LOS shorter and IMLRG smaller. The performance of Hospital P and Q was the worst due to their relatively high charge level, long LOS and high IMLRG. Various performance problems also existed in the other hospitals.

CONCLUSION

It is possible to develop an accurate and easy to run performance evaluation system using Case-Mix as the tool for risk adjustment, choosing indicators close to consumers and managers, and utilizing routine report forms as the basic information source. To keep such a system running effectively, it is necessary to improve the reliability of clinical information and the risk-adjustment ability of Case-Mix.

Regression tree boosting to adjust health care cost predictions for diagnostic mix

OBJECTIVE

To assess the ability of regression tree boosting to risk-adjust health care cost predictions, using diagnostic groups and demographic variables as inputs. Systems for risk-adjusting health care cost, described in the literature, have consistently employed deterministic models to account for interactions among diagnostic groups, simplifying their statistical representation, but sacrificing potentially useful information. An alternative is to use a statistical learning algorithm such as regression tree boosting that systematically searches the data for consequential interactions, which it automatically incorporates into a risk-adjustment model that is customized to the population under study.

DATA SOURCE

Administrative data for over 2 million enrollees in indemnity, preferred provider organization (PPO), and point-of-service (POS) plans from Thomson Medstat's Commercial Claims and Encounters database.

STUDY DESIGN

The Agency for Healthcare Research and Quality's Clinical Classification Software (CCS) was used to sort 2001 diagnoses into 260 diagnosis categories (DCs). For each plan type (indemnity, PPO, and POS), boosted regression trees and main effects linear models were fitted to predict concurrent (2001) and prospective (2002) total health care cost per patient, given DCs and demographic variables.

PRINCIPAL FINDINGS

Regression tree boosting explained 49.7-52.1 percent of concurrent cost variance and 15.2-17.7 percent of prospective cost variance in independent test samples. Corresponding results for main effects linear models were 42.5-47.6 percent and 14.2-16.6 percent.

CONCLUSIONS

The combination of regression tree boosting and a diagnostic grouping scheme, such as CCS, represents a competitive alternative to risk-adjustment systems that use complex deterministic models to account for interactions among diagnostic groups.