Identification of Genes Related to Endoplasmic Reticulum Stress (ERS) in Chronic Obstructive Pulmonary Disease (COPD) and Clinical Validation

Objective

Endoplasmic reticulum stress (ERS) is key in chronic obstructive pulmonary disease (COPD) incidence and progression. This study aims to identify potential ERS-related genes in COPD through bioinformatics analysis and clinical experiments.

Methods

We first obtained a COPD-related mRNA expression dataset (GSE38974) from the Gene Expression Omnibus (GEO) database. The R software was then used to identify potential differentially expressed genes (DEGs) of COPD-related ERS (COPDERS). Subsequently, the identified DEGs were subjected to protein-protein interaction (PPI), correlation, Gene Ontology (GO) enrichment, and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway enrichment analyses. Following that, qRT-PCR was used to examine the RNA expression of six ERS-related DEGs in blood samples obtained from the COPD and control groups. The genes were also subjected to microRNA analysis. Finally, a correlation analysis was performed between the DEGs and key clinical indicators.

Results

Six ERS-related DEGs (five upregulated and one downregulated) were identified based on samples drawn from 23 COPD patients and nine healthy individuals enrolled in the study. Enrichment analysis revealed multiple ERS-related pathways. The qRT-PCR and mRNA microarray bioinformatics analysis results showed consistent STC2, APAF1, BAX, and PTPN1 expressions in the COPD and control groups. Additionally, hsa-miR-485-5p was identified through microRNA prediction and DEG analysis. A correlation analysis between key genes and clinical indicators in COPD patients demonstrated that STC2 was positively and negatively correlated with eosinophil count (EOS) and lymphocyte count (LYM), respectively. On the other hand, PTPN1 showed a strong correlation with pulmonary function indicators.

Conclusion

Four COPDERS-related key genes (STC2, APAF1, BAX, and PTPN1) were identified through bioinformatics analysis and clinical validation, and the expressions of some genes exhibited a significant correlation with the selected clinical indicators. Furthermore, hsa-miR-485-5p was identified as a potential key target in COPDERS, but its precise mechanism remains unclear.

A framework for understanding label leakage in machine learning for health care

INTRODUCTION

The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule."

FRAMEWORK

We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice.

RECOMMENDATIONS

Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.

The value of parental medical records for the prediction of diabetes and cardiovascular disease: a novel method for generating and incorporating family histories

OBJECTIVE

To determine whether data-driven family histories (DDFH) derived from linked EHRs of patients and their parents can improve prediction of patients' 10-year risk of diabetes and atherosclerotic cardiovascular disease (ASCVD).

MATERIALS AND METHODS

A retrospective cohort study using data from Israel's largest healthcare organization. A random sample of 200 000 subjects aged 40-60 years on the index date (January 1, 2010) was included. Subjects with insufficient history (<1 year) or insufficient follow-up (<10 years) were excluded. Two separate XGBoost models were developed-1 for diabetes and 1 for ASCVD-to predict the 10-year risk for each outcome based on data available prior to the index date of January 1, 2010.

RESULTS

Overall, the study included 110 734 subject-father-mother triplets. There were 22 153 cases of diabetes (20%) and 11 715 cases of ASCVD (10.6%). The addition of parental information significantly improved prediction of diabetes risk (P < .001), but not ASCVD risk. For both outcomes, maternal medical history was more predictive than paternal medical history. A binary variable summarizing parental disease state delivered similar predictive results to the full parental EHR.

DISCUSSION

The increasing availability of EHRs for multiple family generations makes DDFH possible and can assist in delivering more personalized and precise medicine to patients. Consent frameworks must be established to enable sharing of information across generations, and the results suggest that sharing the full records may not be necessary.

CONCLUSION

DDFH can address limitations of patient self-reported family history, and it improves clinical predictions for some conditions, but not for all, and particularly among younger adults.

Development of a prediction model for the acquisition of extended spectrum beta-lactam-resistant organisms in U.S. international travellers

BACKGROUND

Extended spectrum beta-lactamase producing Enterobacterales (ESBL-PE) present a risk to public health by limiting the efficacy of multiple classes of beta-lactam antibiotics against infection. International travellers may acquire these organisms and identifying individuals at high risk of acquisition could help inform clinical treatment or prevention strategies.

METHODS

We used data collected from a cohort of 528 international travellers enrolled in a multicentre US-based study to derive a clinical prediction rule (CPR) to identify travellers who developed ESBL-PE colonization, defined as those with new ESBL positivity in stool upon return to the United States. To select candidate features, we used data collected from pre-travel and post-travel questionnaires, alongside destination-specific data from external sources. We utilized LASSO regression for feature selection, followed by random forest or logistic regression modelling, to derive a CPR for ESBL acquisition.

RESULTS

A CPR using machine learning and logistic regression on 10 features has an internally cross-validated area under the receiver operating characteristic curve (cvAUC) of 0.70 (95% confidence interval 0.69-0.71). We also demonstrate that a four-feature model performs similarly to the 10-feature model, with a cvAUC of 0.68 (95% confidence interval 0.67-0.69). This model uses traveller's diarrhoea, and antibiotics as treatment, destination country waste management rankings and destination regional probabilities as predictors.

CONCLUSIONS

We demonstrate that by integrating traveller characteristics with destination-specific data, we could derive a CPR to identify those at highest risk of acquiring ESBL-PE during international travel.

Validation of Machine Learning Model Performance in Predicting Blood Transfusion After Primary and Revision Total Hip Arthroplasty

BACKGROUND

The rates of blood transfusion following primary and revision total hip arthroplasty (THA) remain as high as 9% and 18%, respectively, contributing to patient morbidity and healthcare costs. Existing predictive tools are limited to specific populations, thereby diminishing their clinical applicability. This study aimed to externally validate our previous institutionally developed machine learning (ML) algorithms to predict the risk of postoperative blood transfusion following primary and revision THA using national inpatient data.

METHODS

Five ML algorithms were trained and validated using data from 101,266 primary THA and 8,594 revision THA patients from a large national database to predict postoperative transfusion risk after primary and revision THA. Models were assessed and compared based on discrimination, calibration, and decision curve analysis.

RESULTS

The most important predictors of transfusion following primary and revision THA were preoperative hematocrit (<39.4%) and operation time (>157 minutes), respectively. All ML models demonstrated excellent discrimination (area under the curve (AUC) >0.8) in primary and revision THA patients, with artificial neural network (AUC = 0.84, slope = 1.11, intercept = -0.04, Brier score = 0.04), and elastic-net-penalized logistic regression (AUC = 0.85, slope = 1.08, intercept = -0.01, and Brier score = 0.12) performing best, respectively. On decision curve analysis, all 5 models demonstrated a higher net benefit than the conventional strategy of intervening for all or no patients in both patient cohorts.

CONCLUSIONS

This study successfully validated our previous institutionally developed ML algorithms for the prediction of blood transfusion following primary and revision THA. Our findings highlight the potential generalizability of predictive ML tools developed using nationally representative data in THA patients.

Developmental trajectories of premorbid functioning predict cognitive remediation treatment response in first-episode schizophrenia

BACKGROUND

Cognitive development after schizophrenia onset can be shaped by interventions such as cognitive remediation, yet no study to date has investigated whether patterns of early behavioral development may predict later cognitive changes following intervention. We therefore investigated the extent to which premorbid adjustment trajectories predict cognitive remediation gains in schizophrenia.

METHODS

In a total sample of 215 participants (170 first-episode schizophrenia participants and 45 controls), we classified premorbid functioning trajectories from childhood through late adolescence using the Cannon-Spoor Premorbid Adjustment Scale. For the 62 schizophrenia participants who underwent 6 months of computer-assisted, bottom-up cognitive remediation interventions, we identified MATRICS Consensus Cognitive Battery scores for which participants demonstrated mean changes after intervention, then evaluated whether developmental trajectories predicted these changes.

RESULTS

Growth mixture models supported three premorbid functioning trajectories: stable-good, deteriorating, and stable-poor adjustment. Schizophrenia participants demonstrated significant cognitive remediation gains in processing speed, verbal learning, and overall cognition. Notably, participants with stable-poor trajectories demonstrated significantly greater improvements in processing speed compared to participants with deteriorating trajectories.

CONCLUSIONS

This is the first study to our knowledge to characterize the associations between premorbid functioning trajectories and cognitive remediation gains after schizophrenia onset, indicating that 6 months of bottom-up cognitive remediation appears to be sufficient to yield a full standard deviation gain in processing speed for individuals with early, enduring functioning difficulties. Our findings highlight the connection between trajectories of premorbid and postmorbid functioning in schizophrenia and emphasize the utility of considering the lifespan developmental course in personalizing therapeutic interventions.

Using measures of race to make clinical predictions: Decision making, patient health, and fairness

The use of race measures in clinical prediction models is contentious. We seek to inform the discourse by evaluating the inclusion of race in probabilistic predictions of illness that support clinical decision making. Adopting a static utilitarian framework to formalize social welfare, we show that patients of all races benefit when clinical decisions are jointly guided by patient race and other observable covariates. Similar conclusions emerge when the model is extended to a two-period setting where prevention activities target systemic drivers of disease. We also discuss non-utilitarian concepts that have been proposed to guide allocation of health care resources.

Improving an Electronic Health Record-Based Clinical Prediction Model Under Label Deficiency: Network-Based Generative Adversarial Semisupervised Approach

BACKGROUND

Observational biomedical studies facilitate a new strategy for large-scale electronic health record (EHR) utilization to support precision medicine. However, data label inaccessibility is an increasingly important issue in clinical prediction, despite the use of synthetic and semisupervised learning from data. Little research has aimed to uncover the underlying graphical structure of EHRs.

OBJECTIVE

A network-based generative adversarial semisupervised method is proposed. The objective is to train clinical prediction models on label-deficient EHRs to achieve comparable learning performance to supervised methods.

METHODS

Three public data sets and one colorectal cancer data set gathered from the Second Affiliated Hospital of Zhejiang University were selected as benchmarks. The proposed models were trained on 5% to 25% labeled data and evaluated on classification metrics against conventional semisupervised and supervised methods. The data quality, model security, and memory scalability were also evaluated.

RESULTS

The proposed method for semisupervised classification outperforms related semisupervised methods under the same setup, with the average area under the receiver operating characteristics curve (AUC) reaching 0.945, 0.673, 0.611, and 0.588 for the four data sets, respectively, followed by graph-based semisupervised learning (0.450, 0.454, 0.425, and 0.5676, respectively) and label propagation (0.475,0.344, 0.440, and 0.477, respectively). The average classification AUCs with 10% labeled data were 0.929, 0.719, 0.652, and 0.650, respectively, comparable to that of the supervised learning methods logistic regression (0.601, 0.670, 0.731, and 0.710, respectively), support vector machines (0.733, 0.720, 0.720, and 0.721, respectively), and random forests (0.982, 0.750, 0.758, and 0.740, respectively). The concerns regarding the secondary use of data and data security are alleviated by realistic data synthesis and robust privacy preservation.

CONCLUSIONS

Training clinical prediction models on label-deficient EHRs is indispensable in data-driven research. The proposed method has great potential to exploit the intrinsic structure of EHRs and achieve comparable learning performance to supervised methods.

Machine learning predictive models in neurosurgery: an appraisal based on the TRIPOD guidelines. Systematic review

OBJECTIVE

In recent years, machine learning models for clinical prediction have become increasingly prevalent in the neurosurgical literature. However, little is known about the quality of these models, and their translation to clinical care has been limited. The aim of this systematic review was to empirically determine the adherence of machine learning models in neurosurgery with standard reporting guidelines specific to clinical prediction models.

METHODS

Studies describing the development or validation of machine learning predictive models published between January 1, 2020, and January 10, 2023, across five neurosurgery journals (Journal of Neurosurgery, Journal of Neurosurgery: Spine, Journal of Neurosurgery: Pediatrics, Neurosurgery, and World Neurosurgery) were included. Studies where the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines were not applicable, radiomic studies, and natural language processing studies were excluded.

RESULTS

Forty-seven studies featuring a machine learning-based predictive model in neurosurgery were included. The majority (53%) of studies were single-center studies, and only 15% of studies externally validated the model in an independent cohort of patients. The median compliance across all 47 studies was 82.1% (IQR 75.9%-85.7%). Giving details of treatment (n = 17 [36%]), including the number of patients with missing data (n = 11 [23%]), and explaining the use of the prediction model (n = 23 [49%]) were identified as the TRIPOD criteria with the lowest rates of compliance.

CONCLUSIONS

Improved adherence to TRIPOD guidelines will increase transparency in neurosurgical machine learning predictive models and streamline their translation into clinical care.

Accounting for uncertainty in training data to improve machine learning performance in predicting new disease activity in early multiple sclerosis

Introduction

Machine learning (ML) has great potential for using health data to predict clinical outcomes in individual patients. Missing data are a common challenge in training ML algorithms, such as when subjects withdraw from a clinical study, leaving some samples with missing outcome labels. In this study, we have compared three ML models to determine whether accounting for label uncertainty can improve a model's predictions.

Methods

We used a dataset from a completed phase-III clinical trial that evaluated the efficacy of minocycline for delaying the conversion from clinically isolated syndrome to multiple sclerosis (MS), using the McDonald 2005 diagnostic criteria. There were a total of 142 participants, and at the 2-year follow-up 81 had converted to MS, 29 remained stable, and 32 had uncertain outcomes. In a stratified 7-fold cross-validation, we trained three random forest (RF) ML models using MRI volumetric features and clinical variables to predict the conversion outcome, which represented new disease activity within 2 years of a first clinical demyelinating event. One RF was trained using subjects with the uncertain labels excluded (RF_exclude), another RF was trained using the entire dataset but with assumed labels for the uncertain group (RF_naive), and a third, a probabilistic RF (PRF, a type of RF that can model label uncertainty) was trained on the entire dataset, with probabilistic labels assigned to the uncertain group.

Results

Probabilistic random forest outperformed both the RF models with the highest AUC (0.76, compared to 0.69 for RF_exclude and 0.71 for RF_naive) and F1-score (86.6% compared to 82.6% for RF_exclude and 76.8% for RF_naive).

Conclusion

Machine learning algorithms capable of modeling label uncertainty can improve predictive performance in datasets in which a substantial number of subjects have unknown outcomes.

A Novel Risk Assessment Model Predicts Major Bleeding Risk at Admission in Medical Inpatients

Background

Venous thromboembolism (VTE) is the leading cause of preventable hospital death in the US. Guidelines from the American College of Chest Physicians and American Society for Hematology recommend providing pharmacological VTE prophylaxis to acutely or critically ill medical patients at acceptable bleeding risk, but there is currently only one validated risk assessment model (RAM) for estimating bleeding risk. We developed a RAM using risk factors at admission and compared it with the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) model.

Methods

A total of 46,314 medical patients admitted to a Cleveland Clinic Health System hospital from 2017-2020 were included. Data were split into training (70%) and validation (30%) sets with equivalent bleeding event rates in each set. Potential risk factors for major bleeding were identified from the IMPROVE model and literature review. Penalized logistic regression using LASSO was performed on the training set to select and regularize important risk factors for the final model. The validation set was used to assess model calibration and discrimination and compare performance with IMPROVE. Bleeding events and risk factors were confirmed through chart review.

Results

The incidence of major in-hospital bleeding was 0.58%. Active peptic ulcer (OR = 5.90), prior bleeding (OR = 4.24), and history of sepsis (OR = 3.29) were the strongest independent risk factors. Other risk factors included age, male sex, decreased platelet count, increased INR, increased PTT, decreased GFR, ICU admission, CVC or PICC placement, active cancer, coagulopathy, and in-hospital antiplatelet drug, steroid, or SSRI use. In the validation set, the Cleveland Clinic Bleeding Model (CCBM) had better discrimination than IMPROVE (0.86 vs. 0.72, p < .001) and, at equivalent sensitivity (54%), categorized fewer patients as high-risk (6.8% vs. 12.1%, p < .001).

Conclusions

From a large population of medical inpatients, we developed and validated a RAM to accurately predict bleeding risk at admission. The CCBM may be used in conjunction with VTE risk calculators to decide between mechanical and pharmacological prophylaxis for at-risk patients.

Machine Learning for Postoperative Continuous Recovery Scores of Oncology Patients in Perioperative Care with Data from Wearables

Assessing post-operative recovery is a significant component of perioperative care, since this assessment might facilitate detecting complications and determining an appropriate discharge date. However, recovery is difficult to assess and challenging to predict, as no universally accepted definition exists. Current solutions often contain a high level of subjectivity, measure recovery only at one moment in time, and only investigate recovery until the discharge moment. For these reasons, this research aims to create a model that predicts continuous recovery scores in perioperative care in the hospital and at home for objective decision making. This regression model utilized vital signs and activity metrics measured using wearable sensors and the XGBoost algorithm for training. The proposed model described continuous recovery profiles, obtained a high predictive performance, and provided outcomes that are interpretable due to the low number of features in the final model. Moreover, activity features, the circadian rhythm of the heart, and heart rate recovery showed the highest feature importance in the recovery model. Patients could be identified with fast and slow recovery trajectories by comparing patient-specific predicted profiles to the average fast- and slow-recovering populations. This identification may facilitate determining appropriate discharge dates, detecting complications, preventing readmission, and planning physical therapy. Hence, the model can provide an automatic and objective decision support tool.

Developing and Validating a Prediction Model For Death or Critical Illness in Hospitalized Adults, an Opportunity for Human-Computer Collaboration

Hospital early warning systems that use machine learning (ML) to predict clinical deterioration are increasingly being used to aid clinical decision-making. However, it is not known how ML predictions complement physician and nurse judgment. Our objective was to train and validate a ML model to predict patient deterioration and compare model predictions with real-world physician and nurse predictions.

DESIGN

Retrospective and prospective cohort study.

SETTING

Academic tertiary care hospital.

PATIENTS

Adult general internal medicine hospitalizations.

MEASUREMENTS AND MAIN RESULTS

We developed and validated a neural network model to predict in-hospital death and ICU admission in 23,528 hospitalizations between April 2011 and April 2019. We then compared model predictions with 3,374 prospectively collected predictions from nurses, residents, and attending physicians about their own patients in 960 hospitalizations between April 30, and August 28, 2019. ML model predictions achieved clinician-level accuracy for predicting ICU admission or death (ML median F1 score 0.32 [interquartile range (IQR) 0.30-0.34], AUC 0.77 [IQ 0.76-0.78]; clinicians median F1-score 0.33 [IQR 0.30-0.35], AUC 0.64 [IQR 0.63-0.66]). ML predictions were more accurate than clinicians for ICU admission. Of all ICU admissions and deaths, 36% occurred in hospitalizations where the model and clinicians disagreed. Combining human and model predictions detected 49% of clinical deterioration events, improving sensitivity by 16% compared with clinicians alone and 24% compared with the model alone while maintaining a positive predictive value of 33%, thus keeping false alarms at a clinically acceptable level.

CONCLUSIONS

ML models can complement clinician judgment to predict clinical deterioration in hospital. These findings demonstrate important opportunities for human-computer collaboration to improve prognostication and personalized medicine in hospital.

Identifying Depression Early in Adolescence: assessing the performance of a risk score for future onset of depression in an independent Brazilian sample

OBJECTIVE

The Identifying Depression Early in Adolescence Risk Score (IDEA-RS) was recently developed in Brazil using data from the Pelotas 1993 Birth Cohort to estimate the individualized probability of developing depression in adolescence. This model includes 11 sociodemographic variables and has been assessed in longitudinal studies from four other countries. We aimed to test the performance of IDEA-RS in an independent, community-based, school-attending sample within the same country: the Brazilian High-Risk Cohort.

METHODS

Standard external validation, refitted, and case mix-corrected models were used to predict depression among 1442 youth followed from a mean age of 13.5 years at baseline to 17.7 years at follow-up, using probabilities calculated with IDEA-RS coefficients.

RESULTS

The area under the curve was 0.65 for standard external validation, 0.70 for the case mix-corrected model, and 0.69 for the refitted model, with discrimination consistently above chance for predicting depression in the new dataset. There was some degree of miscalibration, corrected by model refitting (calibration-in-the-large reduced from 0.77 to 0).

CONCLUSION

IDEA-RS was able to parse individuals with higher or lower probability of developing depression beyond chance in an independent Brazilian sample. Further steps should include model improvements and additional studies in populations with high levels of subclinical symptoms to improve clinical decision making.

Clinical Predictors for Upper Limb Recovery after Stroke Rehabilitation: Retrospective Cohort Study

After stroke, recovery of upper limb (UL) motor function is enhanced by a high dose of rehabilitation and is supposed to be supported by attentive functions. However, their mutual influence during rehabilitation is not well known yet. The aim of this retrospective observational cohort study was to investigate the association between rehabilitation dose and motor and cognitive functions, during UL motor recovery. Inpatients with first unilateral stroke, without time restrictions from onset, and undergoing at least 15 h of rehabilitation were enrolled. Data on dose and modalities of rehabilitation received, together with motor and cognitive outcomes before and after therapy, were collected. Fugl-Meyer values for the Upper Extremity were the primary outcome measure. Logistic regression models were used to detect any associations between UL motor improvement and motor and cognitive-linguistic features at acceptance, regarding dose of rehabilitation received. Thirty-five patients were enrolled and received 80.57 ± 30.1 h of rehabilitation on average. Manual dexterity, level of independence and UL motor function improved after rehabilitation, with no influence of attentive functions on motor recovery. The total amount of rehabilitation delivered was the strongest factor (p = 0.031) influencing the recovery of UL motor function after stroke, whereas cognitive-linguistic characteristics were not found to influence UL motor gains.

Derivation and external validation of clinical prediction rules identifying children at risk of linear growth faltering

Background

Nearly 150 million children under-5 years of age were stunted in 2020. We aimed to develop a clinical prediction rule (CPR) to identify children likely to experience additional stunting following acute diarrhea, to enable targeted approaches to prevent this irreversible outcome.

Methods

We used clinical and demographic data from the Global Enteric Multicenter Study (GEMS) to build predictive models of linear growth faltering (decrease of ≥0.5 or ≥1.0 in height-for-age z-score [HAZ] at 60-day follow-up) in children ≤59 months presenting with moderate-to-severe diarrhea, and community controls, in Africa and Asia. We screened variables using random forests, and assessed predictive performance with random forest regression and logistic regression using fivefold cross-validation. We used the Etiology, Risk Factors, and Interactions of Enteric Infections and Malnutrition and the Consequences for Child Health and Development (MAL-ED) study to (1) re-derive, and (2) externally validate our GEMS-derived CPR.

Results

Of 7639 children in GEMS, 1744 (22.8%) experienced severe growth faltering (≥0.5 decrease in HAZ). In MAL-ED, we analyzed 5683 diarrhea episodes from 1322 children, of which 961 (16.9%) episodes experienced severe growth faltering. Top predictors of growth faltering in GEMS were: age, HAZ at enrollment, respiratory rate, temperature, and number of people living in the household. The maximum area under the curve (AUC) was 0.75 (95% confidence interval [CI]: 0.75, 0.75) with 20 predictors, while 2 predictors yielded an AUC of 0.71 (95% CI: 0.71, 0.72). Results were similar in the MAL-ED re-derivation. A 2-variable CPR derived from children 0-23 months in GEMS had an AUC = 0.63 (95% CI: 0.62, 0.65), and AUC = 0.68 (95% CI: 0.63, 0.74) when externally validated in MAL-ED.

Conclusions

Our findings indicate that use of prediction rules could help identify children at risk of poor outcomes after an episode of diarrheal illness. They may also be generalizable to all children, regardless of diarrhea status.

Funding

This work was supported by the National Institutes of Health under Ruth L. Kirschstein National Research Service Award NIH T32AI055434 and by the National Institute of Allergy and Infectious Diseases (R01AI135114).

Fitting joint models of longitudinal observations and time to event by sequential Bayesian updating

Joint modelling of longitudinal measurements and time to event, with longitudinal and event submodels coupled by latent state variables, has wide application in biostatistics. Standard methods for fitting these models require numerical integration to marginalize over the trajectories of the latent states, which is computationally prohibitive for high-dimensional data and for the large data sets that are generated from electronic health records. This paper describes an alternative model-fitting approach based on sequential Bayesian updating, which allows the likelihood to be factorized as the product of the likelihoods of a state-space model and a Poisson regression model. Updates for linear Gaussian state-space models can be efficiently generated with a Kalman filter and the approach can be implemented with existing software. An application to a publicly available data set is demonstrated.

Reporting of Model Performance and Statistical Methods in Studies That Use Machine Learning to Develop Clinical Prediction Models: Protocol for a Systematic Review

BACKGROUND

With the growing excitement of the potential benefits of using machine learning and artificial intelligence in medicine, the number of published clinical prediction models that use these approaches has increased. However, there is evidence (albeit limited) that suggests that the reporting of machine learning-specific aspects in these studies is poor. Further, there are no reviews assessing the reporting quality or broadly accepted reporting guidelines for these aspects.

OBJECTIVE

This paper presents the protocol for a systematic review that will assess the reporting quality of machine learning-specific aspects in studies that use machine learning to develop clinical prediction models.

METHODS

We will include studies that use a supervised machine learning algorithm to develop a prediction model for use in clinical practice (ie, for diagnosis or prognosis of a condition or identification of candidates for health care interventions). We will search MEDLINE for studies published in 2019, pseudorandomly sort the records, and screen until we obtain 100 studies that meet our inclusion criteria. We will assess reporting quality with a novel checklist developed in parallel with this review, which includes content derived from existing reporting guidelines, textbooks, and consultations with experts. The checklist will cover 4 key areas where the reporting of machine learning studies is unique: modelling steps (order and data used for each step), model performance (eg, reporting the performance of each model compared), statistical methods (eg, describing the tuning approach), and presentation of models (eg, specifying the predictors that contributed to the final model).

RESULTS

We completed data analysis in August 2021 and are writing the manuscript. We expect to submit the results to a peer-reviewed journal in early 2022.

CONCLUSIONS

This review will contribute to more standardized and complete reporting in the field by identifying areas where reporting is poor and can be improved.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews CRD42020206167; https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=206167.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR1-10.2196/30956.

Logistic Regression and Machine Learning Models Cannot Discriminate Between Satisfied and Dissatisfied Total Knee Arthroplasty Patients

BACKGROUND

Approximately 20% of total knee arthroplasty (TKA) patients are found to be dissatisfied or unsure of their satisfaction at 1-year post-surgery. This study attempted to predict 1-year post-surgery dissatisfied/unsure TKA patients with pre-surgery and surgical variables using logistic regression and machine learning methods.

METHODS

A retrospective analysis of patients who underwent primary TKA for osteoarthritis between 2012 and 2016 at a single institution was completed. Patients were split into satisfied and dissatisfied/unsure groups. Potential predictor variables included the following: demographic information, patella re-surfaced, posterior collateral ligament sacrificed, and subscales from the Knee Society Knee Scoring System, the Knee Society Clinical Rating System, the Western Ontario and McMaster Universities Osteoarthritis Index, and the 12-Item Short Form Health Survey version 2. Logistic regression and 6 different machine learning methods were used to create prediction models. Model performance was evaluated using discrimination (AUC [area under the receiver operating characteristic curve]) and calibration (Brier score, Cox intercept, and Cox slope) metrics.

RESULTS

There were 1432 eligible patients included in the analysis, 313 were considered to be dissatisfied/unsure. When evaluating discrimination, the logistic regression (AUC = 0.736) and extreme gradient boosted tree (AUC = 0.713) models performed best. When evaluating calibration, the logistic regression (Brier score = 0.141, Cox intercept = 0.241, and Cox slope = 1.31) and gradient boosted tree (Brier score = 0.149, Cox intercept = 0.054, and Cox slope = 1.158) models performed best.

CONCLUSION

The models developed in this study do not perform well enough as discriminatory tools to be used in a clinical setting. Further work needs to be done to improve the performance of pre-surgery TKA dissatisfaction prediction models.

A dynamic model to predict long-term outcomes in patients with prolonged disorders of consciousness

PURPOSE

It is important to predict the prognosis of patients with prolonged disorders of consciousness (DOC). This study established and validated a nomogram and corresponding web-based calculator to predict outcomes for patients with prolonged DOC.

METHODS

All data were obtained from the First Affiliated Hospital of Nanchang University and the Shangrao Hospital of Traditional Chinese Medicine. Predictive variables were identified by univariate and multiple logistic regression analyses. Receiver operating characteristic curves, calibration curves, and a decision curve analysis (DCA) were utilized to assess the predictive accuracy, discriminative ability, and clinical utility of the model, respectively.

RESULTS

Independent prognostic factors, such as age, Glasgow coma scale score, state of consciousness, and brainstem auditory-evoked potential grade were integrated into a nomogram. The model demonstrated good discrimination in the training and validation cohorts, with area-under-the-curve values of 0.815 (95% confidence interval [CI]: 0.748-0.882) and 0.805 (95% CI: 0.727-0.883), respectively. The calibration plots and DCA demonstrated good model performance and clear clinical benefits in both cohorts.

CONCLUSIONS

Based on our nomogram, we developed an effective, simple, and accurate model of a web-based calculator that may help individualize healthcare decision-making. Further research is warranted to optimize the system and update the predictors.

Capturing adolescents in need of psychiatric care with psychopathological symptoms: A population-based cohort study

BACKGROUND

The current study aims to overcome past methodological limitations and capture adolescents in need of psychiatric care with psychopathological symptoms in a cohort with unrestricted access to mental health professionals.

METHODS

The study source population consisted of a random sample of adolescents aged 16-17 years (N=1,369) assessed by the Israeli Draft Board. An adapted version of the Brief Symptom Inventory was used to identify clinically relevant psychopathological symptoms with scores categorized as severe if they were in the top 10th percentile of symptoms, otherwise not severe. An independent interview with a subsequent referral to a mental health professional was used to categorize adolescents in need of psychiatric care. To examine the association between severe psychopathological symptoms and the need for psychiatric care, logistic regression models were fitted unadjusted and adjusted for age, sex, and intellectual assessment scores. Adjusted classification measures were estimated to examine the utility of severe psychopathological symptoms for clinical prediction of need for psychiatric care.

RESULTS

Information on 1,283 adolescents was available in the final analytic sample. Logistic regression modeling showed a statistically significant (p<0.001) association between self-reported severe psychopathological symptoms and the need for psychiatric care (OR adjusted: 4.38; 95% CI: 3.55-5.40). Severe psychopathological symptoms had a classification accuracy of 83% (CI: 81%-85%).

CONCLUSIONS

Severe psychopathological symptoms, although accounting for a fair proportion of treatment seeking, would perhaps be better useful for classification purposes alongside other variables rather than in isolation.

Flexible co-data learning for high-dimensional prediction

Clinical research often focuses on complex traits in which many variables play a role in mechanisms driving, or curing, diseases. Clinical prediction is hard when data is high-dimensional, but additional information, like domain knowledge and previously published studies, may be helpful to improve predictions. Such complementary data, or co-data, provide information on the covariates, such as genomic location or P-values from external studies. We use multiple and various co-data to define possibly overlapping or hierarchically structured groups of covariates. These are then used to estimate adaptive multi-group ridge penalties for generalized linear and Cox models. Available group adaptive methods primarily target for settings with few groups, and therefore likely overfit for non-informative, correlated or many groups, and do not account for known structure on group level. To handle these issues, our method combines empirical Bayes estimation of the hyperparameters with an extra level of flexible shrinkage. This renders a uniquely flexible framework as any type of shrinkage can be used on the group level. We describe various types of co-data and propose suitable forms of hypershrinkage. The method is very versatile, as it allows for integration and weighting of multiple co-data sets, inclusion of unpenalized covariates and posterior variable selection. For three cancer genomics applications we demonstrate improvements compared to other models in terms of performance, variable selection stability and validation.

Identifying digenic disease genes via machine learning in the Undiagnosed Diseases Network

Rare diseases affect millions of people worldwide, and discovering their genetic causes is challenging. More than half of the individuals analyzed by the Undiagnosed Diseases Network (UDN) remain undiagnosed. The central hypothesis of this work is that many of these rare genetic disorders are caused by multiple variants in more than one gene. However, given the large number of variants in each individual genome, experimentally evaluating combinations of variants for potential to cause disease is currently infeasible. To address this challenge, we developed the digenic predictor (DiGePred), a random forest classifier for identifying candidate digenic disease gene pairs by features derived from biological networks, genomics, evolutionary history, and functional annotations. We trained the DiGePred classifier by using DIDA, the largest available database of known digenic-disease-causing gene pairs, and several sets of non-digenic gene pairs, including variant pairs derived from unaffected relatives of UDN individuals. DiGePred achieved high precision and recall in cross-validation and on a held-out test set (PR area under the curve > 77%), and we further demonstrate its utility by using digenic pairs from the recent literature. In contrast to other approaches, DiGePred also appropriately controls the number of false positives when applied in realistic clinical settings. Finally, to enable the rapid screening of variant gene pairs for digenic disease potential, we freely provide the predictions of DiGePred on all human gene pairs. Our work enables the discovery of genetic causes for rare non-monogenic diseases by providing a means to rapidly evaluate variant gene pairs for the potential to cause digenic disease.

A Machine Learning-Based Algorithm for the Prediction of Intensive Care Unit Delirium (PRIDE): Retrospective Study

Background

Delirium frequently occurs among patients admitted to the intensive care unit (ICU). There is limited evidence to support interventions to treat or resolve delirium in patients who have already developed delirium. Therefore, the early recognition and prevention of delirium are important in the management of critically ill patients.

Objective

This study aims to develop and validate a delirium prediction model within 24 hours of admission to the ICU using electronic health record data. The algorithm was named the Prediction of ICU Delirium (PRIDE).

Methods

This is a retrospective cohort study performed at a tertiary referral hospital with 120 ICU beds. We only included patients who were 18 years or older at the time of admission and who stayed in the medical or surgical ICU. Patients were excluded if they lacked a Confusion Assessment Method for the ICU record from the day of ICU admission or if they had a positive Confusion Assessment Method for the ICU record at the time of ICU admission. The algorithm to predict delirium was developed using patient data from the first 2 years of the study period and validated using patient data from the last 6 months. Random forest (RF), Extreme Gradient Boosting (XGBoost), deep neural network (DNN), and logistic regression (LR) were used. The algorithms were externally validated using MIMIC-III data, and the algorithm with the largest area under the receiver operating characteristics (AUROC) curve in the external data set was named the PRIDE algorithm.

Results

A total of 37,543 cases were collected. After patient exclusion, 12,409 remained as our study population, of which 3816 (30.8%) patients experienced delirium incidents during the study period. Based on the exclusion criteria, out of the 96,016 ICU admission cases in the MIMIC-III data set, 2061 cases were included, and 272 (13.2%) delirium incidents occurred. The average AUROCs and 95% CIs for internal validation were 0.916 (95% CI 0.916-0.916) for RF, 0.919 (95% CI 0.919-0.919) for XGBoost, 0.881 (95% CI 0.878-0.884) for DNN, and 0.875 (95% CI 0.875-0.875) for LR. Regarding the external validation, the best AUROC were 0.721 (95% CI 0.72-0.721) for RF, 0.697 (95% CI 0.695-0.699) for XGBoost, 0.655 (95% CI 0.654-0.657) for DNN, and 0.631 (95% CI 0.631-0.631) for LR. The Brier score of the RF model is 0.168, indicating that it is well-calibrated.

Conclusions

A machine learning approach based on electronic health record data can be used to predict delirium within 24 hours of ICU admission. RF, XGBoost, DNN, and LR models were used, and they effectively predicted delirium. However, with the potential to advise ICU physicians and prevent ICU delirium, prospective studies are required to verify the algorithm’s performance.

ASPECT scores of patients with focal intracerebral hemorrhage were correlated with their short- and medium-term functional outcomes

PURPOSE

The Alberta Stroke Program Early CT Score (ASPECTS) is widely used to guide thrombolytic therapy and predict the functional outcome of patients with acute ischemic stroke (AIS). Whether ASPECTS can predict the functional outcome of patients with intracerebral hemorrhage (ASPECTS-H) remains unclear.

METHODS

Patients with primary intracerebral hemorrhage (ICH) were collected and retrospectively analyzed. ASPECTS-H was assessed at admission. Patients were followed up at 30 days and 90 days after the onset of ICH. Occurrence of death within 90 days after ICH was the primary endpoint. Modified Rankin Scale (mRS) ≥ 3 was considered a poor functional outcome.

RESULTS

A total of 149 patients met eligibility criteria; 61 (40.9%) had poor functional outcome at 30 days, and 37 (24.8%) had poor functional outcome at 90 days. Using binary logistic regression modeling, we found that a low ASPECTS-H was associated with a poor functional outcome. The risk ratio of a low ASPECTS-H was 2.31 at 30 days (P = 0.000; 95% CI, 1.560-3.421) and 2.711 at 90 days (P = 0.000; 95% CI, 1.677-4.381). The optimal cutoff value of ASPECTS-H to discriminate good and poor 30-day and 90-day outcomes was 7.5 (Sensitivity_30-day = 0.636, 1-Specificity_{30 - day} = 0.311; Sensitivity_90-day = 0.580, 1-Specificity_90-day = 0.270).

CONCLUSIONS

A low ASPECTS-H was an indicator of poor short-term and long-term functional outcomes of ICH.

Validation of E-PRE-DELIRIC in cardiac surgical ICU delirium: A retrospective cohort study

BACKGROUND

The early prediction model for delirium in intensive care units (ICUs)-E-PRE-DELIRIC-has been created to predict delirium development during the length of stay in ICUs. However, there have been few early predictive models for delirium in the cardiac surgical ICU (CSICU), and the predictive ability of the E-PRE-DELIRIC among patients following cardiac surgeries is still unknown.

AIMS AND OBJECTIVES

To validate the performance of E-PRE-DELIRIC in CSICU.

DESIGN

A retrospective cohort study.

METHODS

Data were retrospectively extracted from the electronic records for patients admitted in CSICU from January 2018 to December 2018 in a tertiary teaching hospital in China. Adult patients were included following the criteria of the E-PRE-DELIRIC model. Predictors, including age, history of cognitive impairment, history of alcohol abuse, urgent admission, use of corticosteroids, respiratory failure, blood urea nitrogen, and mean arterial pressure, at the time of ICU admission were retrieved, and delirium was assessed twice a day using the Confusion Assessment Method for the ICU. The performance of the E-PRE-DELIRIC model was evaluated by area under receiver operator characteristic curve, precision-recall curve (AUPRC), Hosmer-Lemeshow (HL) test, and calibration belt.

RESULTS

Of the 725 patients included, 120 (16.6%) developed delirium. The AUROC was 0.54 (95% confidence interval [CI], 0.48-0.59), and the AUPRC was 0.18 (95% CI, 0.12-0.20). The HL test showed a significant difference between predicted probability and delirium occurrence (χ² = 17.326, P = .027), and the overestimation chance of the E-PRE-DELIRIC score was 0.24 to 0.43.

CONCLUSION

The E-PRE-DELIRIC model has poor-to-fair predictive value in this study; thus, its application among the CSICU patients is limited. Development of reliable and validated tools for early prediction of delirium in CSICU is required.

RELEVANCE TO CLINICAL PRACTICE

Early prediction of delirium risk at CSICU admission is of vital importance and could provide timely information to caregivers. However, the E-PRE-DELIRIC model should be applied cautiously in the CSICU because of the significant probability of over-estimating the risk of developing delirium.

Accuracy of the Palliative Prognostic Score With or Without Clinicians' Prediction of Survival in Patients With Far Advanced Cancer

CONTEXT

Previous studies suggest that clinicians' prediction of survival (CPS) may have reduced the accuracy of objective indicators for prognostication in palliative care.

OBJECTIVES

We aimed to examine the accuracy of CPS alone, compared to the original Palliative Prognostic Score (PaP), and five clinical/laboratory variables of the PaP in patients with far advanced cancer.

METHODS

We compared the discriminative accuracy of three prediction models (the PaP-CPS [the score of the categorical CPS of PaP], PaP without CPS [sum of the scores of only the objective variables of PaP], and PaP total score) across 3 settings: inpatient palliative care consultation team, palliative care unit, and home palliative care. We computed the area under receiver operating characteristic curve (AUROC) for 30-day survival and concordance index (C-index) to compare the discriminative accuracy of these three models.

RESULTS

We included a total of 1534 subjects with median survival of 34.0 days. The AUROC and C-index in the three settings were 0.816-0.896 and 0.732-0.799 for the PaP total score, 0.808-0.884 and 0.713-0.782 for the PaP-CPS, and 0.726-0.815 and 0.672-0.728 for the PaP without CPS, respectively. The PaP total score and PaP-CPS showed similar AUROCs and C-indices across the three settings. The PaP total score had significantly higher AUROCs and C-indices than the PaP without CPS across the three settings.

CONCLUSION

Overall, the PaP total score, PaP-CPS, and PaP without CPS showed good discriminative performances. However, the PaP total score and PaP-CPS were significantly more accurate than the PaP without CPS.

Risk Prediction of Cognitive Decline after Stroke

BACKGROUND AND PURPOSE

Cognitive decline is one of the major outcomes after stroke. We have developed and evaluated a risk predictive tool of post-stroke cognitive decline and assessed its clinical utility.

METHODS

In this population-based cohort, 4,783 patients with first-ever stroke from the South London Stroke Register (1995-2010) were included in developing the model. Cognitive impairment was measured using the Mini Mental State Examination (cut off 24/30) and the Abbreviated Mental Test (cut off 8/10) at 3-months and yearly thereafter. A penalised mixed-effects linear model was developed and temporal-validated in a new cohort consisted of 1,718 stroke register participants recruited from (2011-2018). Prediction errors on discrimination and calibration were assessed. The clinical utility of the model was evaluated using prognostic accuracy measurements and decision curve analysis.

RESULTS

The overall predictive model showed good accuracy, with root mean squared error of 0.12 and R2 of 73%. Good prognostic accuracy for predicting severe cognitive decline was observed AUC: (88%, 95% CI [85-90]), (89.6%, 95% CI [86-92]), (87%, 95% CI [85-91]) at 3 months, one and 5 years respectively. Average predicted recovery patterns were analysed by age, stroke subtype, Glasgow-coma scale, and left-stroke and showed variability. DECISION: curve analysis showed an increased clinical benefit, particularly at threshold probabilities of above 15% for predictive risk of cognitive impairment.

CONCLUSIONS

The derived prognostic model seems to accurately screen the risk of post-stroke cognitive decline. Such prediction could support the development of more tailored management evaluations and identify groups for further study and future trials.

Erratum: Improving the Prediction of Benign or Malignant Breast Masses Using a Combination of Image Biomarkers and Clinical Parameters

[This corrects the article DOI: 10.3389/fonc.2021.629321.].

Dr. Agent: Clinical predictive model via mimicked second opinions

OBJECTIVE

Prediction of disease phenotypes and their outcomes is a difficult task. In practice, patients routinely seek second opinions from multiple clinical experts for complex disease diagnosis. Our objective is to mimic such a practice of seeking second opinions by training 2 agents with different focuses: the primary agent studies the most recent visit of the patient to learn the current health status, and then the second-opinion agent considers the entire patient history to obtain a more global view.

MATERIALS AND METHODS

Our approach Dr. Agent augments recurrent neural networks with 2 policy gradient agents. Moreover, Dr. Agent is customized with various patient demographics information and learns a dynamic skip connection to focus on the relevant information over time. We trained Dr. Agent to perform 4 clinical prediction tasks on the publicly available MIMIC-III (Medical Information Mart for Intensive Care) database: (1) in-hospital mortality prediction, (2) acute care phenotype classification, (3) physiologic decompensation prediction, and (4) forecasting length of stay. We compared the performance of Dr. Agent against 4 baseline clinical predictive models.

RESULTS

Dr. Agent outperforms baseline clinical prediction models across all 4 tasks in terms of all metrics. Compared with the best baseline model, Dr. Agent achieves up to 15% higher area under the precision-recall curve on different tasks.

CONCLUSIONS

Dr. Agent can comprehensively model the long-term dependencies of patients' health status while considering patients' demographics using 2 agents, and therefore achieves better prediction performance on different clinical prediction tasks.

How good are clinicians in predicting the presence of Pseudomonas spp. in diabetic foot infections? A prospective clinical evaluation

Introduction

The most frequently prescribed empirical antibiotic agents for mild and moderate diabetic foot infections (DFIs) are amino-penicillins and second-generation cephalosporins that do not cover Pseudomonas spp. Many clinicians believe they can predict the involvement of Pseudomonas in a DFI by visual and/or olfactory clues, but no data support this assertion.

Methods

In this prospective observational study, we separately asked 13 experienced (median 11 years) healthcare workers whether they thought the Pseudomonas spp. would be implicated in the DFI. Their predictions were compared with the results of cultures of deep/intraoperative specimens and/or the clinical remission of DFI achieved with antibiotic agents that did not cover Pseudomonas.

Results

Among 221 DFI episodes in 88 individual patients, intraoperative tissue cultures grew Pseudomonas in 22 cases (10%, including six bone samples). The presence of Pseudomonas was correctly predicted with a sensitivity of 0.32, specificity of 0.84, positive predictive value of 0.18 and negative predictive value 0.92. Despite two feedbacks of the interim results and a 2-year period, the clinicians' predictive performance did not improve.

Conclusion

The combined visual and olfactory performance of experienced clinicians in predicting the presence of Pseudomonas in a DFI was moderate, with better specificity than sensitivity, and did not improve over time. Further investigations are needed to determine whether clinicians should use a negative prediction of the presence of Pseudomonas in a DFI, especially in settings with a high prevalence of pseudomonal DFIs.

Improving the Prediction of Benign or Malignant Breast Masses Using a Combination of Image Biomarkers and Clinical Parameters

Background: Breast cancer is one of the leading causes of death in female cancer patients. The disease can be detected early using Mammography, an effective X-ray imaging technology. The most important step in mammography is the classification of mammogram patches as benign or malignant. Classically, benign or malignant breast tumors are diagnosed by radiologists' interpretation of mammograms based on clinical parameters. However, because masses are heterogeneous, clinical parameters supply limited information on mammography mass. Therefore, this study aimed to predict benign or malignant breast masses using a combination of image biomarkers and clinical parameters. Methods: We trained a deep learning (DL) fusion network of VGG16 and Inception-V3 network in 5,996 mammography images from the training cohort; DL features were extracted from the second fully connected layer of the DL fusion network. We then developed a combined model incorporating DL features, hand-crafted features, and clinical parameters to predict benign or malignant breast masses. The prediction performance was compared between clinical parameters and the combination of the above features. The strengths of the clinical model and the combined model were subsequently validated in a test cohort (n = 244) and an external validation cohort (n = 100), respectively. Results: Extracted features comprised 30 hand-crafted features, 27 DL features, and 5 clinical features (shape, margin type, breast composition, age, mass size). The model combining the three feature types yielded the best performance in predicting benign or malignant masses (AUC = 0.961) in the test cohort. A significant difference in the predictive performance between the combined model and the clinical model was observed in an independent external validation cohort (AUC: 0.973 vs. 0.911, p = 0.019). Conclusion: The prediction of benign or malignant breast masses improves when image biomarkers and clinical parameters are combined; the combined model was more robust than clinical parameters alone.

Enhancing the prediction of childhood asthma remission: Integrating clinical factors with microRNAs

The novel integration of baseline clinical and microRNA variables significantly improves the long-term individualized prediction of childhood asthma remission by early adulthood compared to using clinical variables alone.

In vitro to Clinical Translation of Combinatorial Effects of Doxorubicin and Abemaciclib in Rb-Positive Triple Negative Breast Cancer: A Systems-Based Pharmacokinetic/Pharmacodynamic Modeling Approach

Background

Doxorubicin (DOX) and its pegylated liposomal formulation (L_DOX) are the standard of care for triple-negative breast cancer (TNBC). However, resistance to DOX often occurs, motivating the search for alternative treatment approaches. The retinoblastoma protein (Rb) is a potential pharmacological target for TNBC treatment since its expression has been associated with resistance to DOX-based therapy.

Methods

DOX (0.01–20 μM) combination with abemaciclib (ABE, 1–6 μM) was evaluated over 72 hours on Rb-positive (MDA-MB-231) and Rb-negative (MDA-MB-468) TNBC cells. Combination indices (CI) for DOX+ABE were calculated using Compusyn software. The TNBC cell viability time-course and fold-change from the control of phosphorylated-Rb (pRb) protein expression were measured with CCK8-kit and enzyme-linked immunosorbent assay. A cell-based pharmacodynamic (PD) model was developed, where pRb protein dynamics drove cell viability response. Clinical pharmacokinetic (PK) models for DOX, L_DOX, and ABE were developed using data extracted from the literature. After scaling cancer cell growth to clinical TNBC tumor growth, the time-to-tumor progression (TTP) was predicted for human dosing regimens of DOX, ABE, and DOX+ABE.

Results

DOX and ABE combinations were synergistic (CI<1) in MDA-MB-231 and antagonistic (CI>1) in MDA-MB-468. The maximum inhibitory effects (Imax) for both drugs were set to one. The drug concentrations producing 50% of Imax for DOX and ABE were 0.565 and 2.31 μM (MDA-MB-231) and 0.121 and 1.61 μM (MDA-MB-468). The first-orders rate constants of abemaciclib absorption (k_a) and doxorubicin release from L_DOX (k_Rel) were estimated at 0.31 and 0.013 h⁻¹. Their linear clearances were 21.7 (ABE) and 32.1 L/h (DOX). The estimated TTP for intravenous DOX (75 mg/m2 every 21 days), intravenous L_DOX (50 mg/m² every 28 days), and oral ABE (200 mg twice a day) were 125, 31.2, and 8.6 days shorter than drug-free control. The TTP for DOX+ABE and L_DOX+ABE were 312 days and 47.5 days shorter than control, both larger than single-agent DOX, suggesting improved activity with the DOX+ABE combination.

Conclusion

The developed translational systems-based PK/PD model provides an in vitro-to-clinic modeling platform for DOX+ABE in TNBC. Although model-based simulations suggest improved outcomes with combination over monotherapy, tumor relapse was not prevented with the combination. Hence, DOX+ABE may not be an effective treatment combination for TNBC.

Construction and validation of a nomogram for patients with skin cancer

Skin cancer is a common malignant tumor in human beings. At present, the construction of clinical prediction models mainly focuses on malignant melanoma and no researchers have constructed clinical prediction models for all kind of skin cancer to predict the prognosis of skin cancer. We used patient data collected from the surveillance, epidemiology, and end results program database to construct and validate our model for clinical prediction of skin cancer, hoping to provide a reference for clinical treatment of skin cancer.R software was used for univariate and multivariate Cox regression analysis of variables to screen out factors that have an impact on the survival of skin cancer patients. Then the prognostic model of skin cancer patients was constructed and the nomogram was drawn. Concordance Index (C-index), receiver operating characteristic (ROC) curve and calibration curve were used to evaluate the clinical prediction model.A total of 3180 skin cancer patients were included in this study. We constructed nomogram, a 3-year and 5-year clinical prediction model for skin cancer patients. We used C-index to evaluate the accuracy of nomogram model, and the result of C-index was 0.728, 95%CI (0.703-0.753). The nomogram model was evaluated by ROC curve. The area under the curve values of the ROC curve for 3-year survival rate and 5-year survival rate were 0.732 and 0.768 respectively. The model calibration diagram of the modeling group also shows that the model exhibits high accuracy.The nomogram model of postoperative survival of patients with skin cancer, based on the surveillance, epidemiology, and end results program database of patients with skin cancer, has shown good stability and accuracy in multi-method validation.

Assessing the Usefulness of the Prevexair Smartphone Application in the Follow-Up High-Risk Patients with COPD

Introduction

This manuscript analyzes the exacerbations recorded by the Prevexair application through the daily analysis of symptoms in high-risk patients with COPD and explores its usefulness in assessing clinical stability with respect to that reported in visits.

Patients and Methods

This study is a multi-centre cohort of COPD patients with the exacerbator phenotype who were monitored over 6 months. The Prevexair application was installed on the patients' smartphones. Patients used the app to record symptom changes, use of medication and use of healthcare resources. It is not established a recommended action plan when worsening of symptoms. At their clinical visit during the follow-up period, patients were asked about exacerbations suffered during these 6 months of monitoring. The investigators who conducted the visit were blinded about the Prevexair app records.

Results

The patients experienced a total of 185 exacerbations according to daily records in the app whereas only 64 exacerbations were recalled during medical visits. Perception became more accurate for severe exacerbations (kappa 0.6577), although we found no factors that predicted poor recall. The proportion of 72.5% patients were classified as unstable if the exacerbations captured by Prevexair were used to define stability, versus 47.8% if the exacerbations recall in visit was used. Two-thirds of the exacerbations recorded in the Prevexair application were not reported to doctors during their clinical visits. Almost half were treated with oral corticosteroids and/or antibiotics and more than one-quarter of the exacerbations treated did not seek medical attention.

Conclusion

The findings of this cohort study confirm that patients do not always remember the exacerbations suffered during their medical visit. The prevexair application is useful in monitoring COPD patients at high risk, in order to a better assessment of exacerbations of COPD during medical visits. Further research must be carried out to evaluate this strategy in clinical practice.

Deep Learning Improves Osteonecrosis Prediction of Femoral Head After Internal Fixation Using Hybrid Patient and Radiograph Variables

Femoral neck fractures (FNFs) are a great public health problem that leads to a high incidence of death and dysfunction. Osteonecrosis of the femoral head (ONFH) after internal fixation of FNF is a frequently reported complication and a major cause for reoperation. Early intervention can prevent osteonecrosis aggravation at the preliminary stage. However, at present, failure to diagnose asymptomatic ONFH after FNF fixation hinders effective intervention at early stages. The primary objective of this study was to develop a predictive model for postoperative ONFH using deep learning (DL) methods developed using plain X-ray radiographs and hybrid patient variables. A two-center retrospective study of patients who underwent closed reduction and cannulated screw fixation was performed. We trained a convolutional neural network (CNN) model using postoperative pelvic radiographs and the output regressive radiograph variables. A less experienced orthopedic doctor, and an experienced orthopedic doctor also evaluated and diagnosed the patients using postoperative pelvic radiographs. Hybrid nomograms were developed based on patient and radiograph variables to determine predictive performance. A total of 238 patients, including 95 ONFH patients and 143 non-ONFH patients, were included. A CNN model was trained using postoperative radiographs and output radiograph variables. The accuracy of the validation set was 0.873 for the CNN model, and the algorithm achieved an area under the curve (AUC) value of 0.912 for the prediction. The diagnostic and predictive ability of the algorithm was superior to that of the two doctors, based on the postoperative X-rays. The addition of DL-based radiograph variables to the clinical nomogram improved predictive performance, resulting in an AUC of 0.948 (95% CI, 0.920-0.976) and better calibration. The decision curve analysis showed that adding the DL increased the clinical usefulness of the nomogram compared with a clinical approach alone. In conclusion, we constructed a DL facilitated nomogram that incorporated a hybrid of radiograph and patient variables, which can be used to improve the prediction of preoperative osteonecrosis of the femoral head after internal fixation.

Clinical Predictive Models for COVID-19: Systematic Study

BACKGROUND

COVID-19 is a rapidly emerging respiratory disease caused by SARS-CoV-2. Due to the rapid human-to-human transmission of SARS-CoV-2, many health care systems are at risk of exceeding their health care capacities, in particular in terms of SARS-CoV-2 tests, hospital and intensive care unit (ICU) beds, and mechanical ventilators. Predictive algorithms could potentially ease the strain on health care systems by identifying those who are most likely to receive a positive SARS-CoV-2 test, be hospitalized, or admitted to the ICU.

OBJECTIVE

The aim of this study is to develop, study, and evaluate clinical predictive models that estimate, using machine learning and based on routinely collected clinical data, which patients are likely to receive a positive SARS-CoV-2 test or require hospitalization or intensive care.

METHODS

Using a systematic approach to model development and optimization, we trained and compared various types of machine learning models, including logistic regression, neural networks, support vector machines, random forests, and gradient boosting. To evaluate the developed models, we performed a retrospective evaluation on demographic, clinical, and blood analysis data from a cohort of 5644 patients. In addition, we determined which clinical features were predictive to what degree for each of the aforementioned clinical tasks using causal explanations.

RESULTS

Our experimental results indicate that our predictive models identified patients that test positive for SARS-CoV-2 a priori at a sensitivity of 75% (95% CI 67%-81%) and a specificity of 49% (95% CI 46%-51%), patients who are SARS-CoV-2 positive that require hospitalization with 0.92 area under the receiver operator characteristic curve (AUC; 95% CI 0.81-0.98), and patients who are SARS-CoV-2 positive that require critical care with 0.98 AUC (95% CI 0.95-1.00).

CONCLUSIONS

Our results indicate that predictive models trained on routinely collected clinical data could be used to predict clinical pathways for COVID-19 and, therefore, help inform care and prioritize resources.

Clinical Features to Predict 22q11.2 Deletion Syndrome Proven by Molecular Genetic Testing

The 22q11.2 deletion syndrome (22q11.2 DS) is the most common microdeletion syndrome with a wide variety of clinical features. However, as there are no clinical criteria for diagnosis, confirmation is solely done by genetic tests if clinicians recognize the syndrome. Therefore, we aimed to identify clinical features that may help clinicians recognize 22q11.2 DS. Participants with at least two anomalies were enrolled, complete patient history and physical examinations were performed, then multiplex ligation-dependent probe amplification (MLPA) analysis for 22q11.2 DS was utilized. We identified 11/48 (23%) cases with 22q11.2 DS. Palatal anomalies, hypocalcemia, and ≥3 affected body systems were highly significant presentations in the 22q11.2 DS group versus the group without deletion ( p  < 0.05). Therefore, a comprehensive physical examination is crucial at identifying any subtle features which may lead to testing and a definite diagnosis.

Reporting quality of studies using machine learning models for medical diagnosis: a systematic review

AIMS

We conducted a systematic review assessing the reporting quality of studies validating models based on machine learning (ML) for clinical diagnosis, with a specific focus on the reporting of information concerning the participants on which the diagnostic task was evaluated on.

METHOD

Medline Core Clinical Journals were searched for studies published between July 2015 and July 2018. Two reviewers independently screened the retrieved articles, a third reviewer resolved any discrepancies. An extraction list was developed from the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guideline. Two reviewers independently extracted the data from the eligible articles. Third and fourth reviewers checked, verified the extracted data as well as resolved any discrepancies between the reviewers.

RESULTS

The search results yielded 161 papers, of which 28 conformed to the eligibility criteria. Detail of data source was reported in 24 of the 28 papers. For all of the papers, the set of patients on which the ML-based diagnostic system was evaluated was partitioned from a larger dataset, and the method for deriving such set was always reported. Information on the diagnostic/non-diagnostic classification was reported well (23/28). The least reported items were the use of reporting guideline (0/28), distribution of disease severity (8/28 patient flow diagram (10/28) and distribution of alternative diagnosis (10/28). A large proportion of studies (23/28) had a delay between the conduct of the reference standard and ML tests, while one study did not and four studies were unclear. For 15 studies, it was unclear whether the evaluation group corresponded to the setting in which the ML test will be applied to.

CONCLUSION

All studies in this review failed to use reporting guidelines, and a large proportion of them lacked adequate detail on participants, making it difficult to replicate, assess and interpret study findings.

PROSPERO REGISTRATION NUMBER

CRD42018099167.

Prediction of lithium response using clinical data

OBJECTIVE

Promptly establishing maintenance therapy could reduce morbidity and mortality in patients with bipolar disorder. Using a machine learning approach, we sought to evaluate whether lithium responsiveness (LR) is predictable using clinical markers.

METHOD

Our data are the largest existing sample of direct interview-based clinical data from lithium-treated patients (n = 1266, 34.7% responders), collected across seven sites, internationally. We trained a random forest model to classify LR-as defined by the previously validated Alda scale-against 180 clinical predictors.

RESULTS

Under appropriate cross-validation procedures, LR was predictable in the pooled sample with an area under the receiver operating characteristic curve of 0.80 (95% CI 0.78-0.82) and a Cohen kappa of 0.46 (0.4-0.51). The model demonstrated a particularly low false-positive rate (specificity 0.91 [0.88-0.92]). Features related to clinical course and the absence of rapid cycling appeared consistently informative.

CONCLUSION

Clinical data can inform out-of-sample LR prediction to a potentially clinically relevant degree. Despite the relevance of clinical course and the absence of rapid cycling, there was substantial between-site heterogeneity with respect to feature importance. Future work must focus on improving classification of true positives, better characterizing between- and within-site heterogeneity, and further testing such models on new external datasets.

A Tutorial on Mechanical Decision-Making for Personnel and Educational Selection

In decision-making, it is important not only to use the correct information but also to combine information in an optimal way. There are robust research findings that a mechanical combination of information for personnel and educational selection matches or outperforms a holistic combination of information. However, practitioners and policy makers seldom use mechanical combination for decision-making. One of the important conditions for scientific results to be used in practice and to be part of policy-making is that results are easily accessible. To increase the accessibility of mechanical judgment prediction procedures, we (1) explain in detail how mechanical combination procedures work, (2) provide examples to illustrate these procedures, and (3) discuss some limitations of mechanical decision-making.

Novel pediatric-automated respiratory score using physiologic data and machine learning in asthma

OBJECTIVES

Manual clinical scoring systems are the current standard used for acute asthma clinical care pathways. No automated system exists that assesses disease severity, time course, and treatment impact in pediatric acute severe asthma exacerbations.

WORKING HYPOTHESIS

machine learning applied to continuous vital sign data could provide a novel pediatric-automated asthma respiratory score (pARS) by using the manual pediatric asthma score (PAS) as the clinical care standard.

METHODS

Continuous vital sign monitoring data (heart rate, respiratory rate, and pulse oximetry) were merged with the health record data including a provider-determined PAS in children between 2 and 18 years of age admitted to the pediatric intensive care unit (PICU) for status asthmaticus. A cascaded artificial neural network (ANN) was applied to create an automated respiratory score and validated by two approaches. The ANN was compared with the Normal and Poisson regression models.

RESULTS

Out of an initial group of 186 patients, 128 patients met inclusion criteria. Merging physiologic data with clinical data yielded >37 000 data points for model training. The pARS score had good predictive accuracy, with 80% of the pARS values within ±2 points of the provider-determined PAS, especially over the mid-range of PASs (6-9). The Poisson and Normal distribution regressions yielded a smaller overall median absolute error.

CONCLUSIONS

The pARS reproduced the manually recorded PAS. Once validated and studied prospectively as a tool for research and for physician decision support, this methodology can be implemented in the PICU to objectively guide treatment decisions.

Submissions from the SPRINT Data Analysis Challenge on clinical risk prediction: a cross-sectional evaluation

OBJECTIVES

To collate and systematically characterise the methods, results and clinical performance of the clinical risk prediction submissions to the Systolic Blood Pressure Intervention Trial (SPRINT) Data Analysis Challenge.

DESIGN

Cross-sectional evaluation.

DATA SOURCES

SPRINT Challenge online submission website.

STUDY SELECTION

Submissions to the SPRINT Challenge for clinical prediction tools or clinical risk scores.

DATA EXTRACTION

In duplicate by three independent reviewers.

RESULTS

Of 143 submissions, 29 met our inclusion criteria. Of these, 23/29 (79%) reported prediction models for an efficacy outcome (20/23 [87%] of these used the SPRINT study primary composite outcome, 14/29 [48%] used a safety outcome, and 4/29 [14%] examined a combined safety/efficacy outcome). Age and cardiovascular disease history were the most common variables retained in 80% (12/15) of the efficacy and 60% (6/10) of the safety models. However, no two submissions included an identical list of variables intending to predict the same outcomes. Model performance measures, most commonly, the C-statistic, were reported in 57% (13/23) of efficacy and 64% (9/14) of safety model submissions. Only 2/29 (7%) models reported external validation. Nine of 29 (31%) submissions developed and provided evaluable risk prediction tools. Using two hypothetical vignettes, 67% (6/9) of the tools provided expected recommendations for a low-risk patient, while 44% (4/9) did for a high-risk patient. Only 2/29 (7%) of the clinical risk prediction submissions have been published to date.

CONCLUSIONS

Despite use of the same data source, a diversity of approaches, methods and results was produced by the 29 SPRINT Challenge competition submissions for clinical risk prediction. Of the nine evaluable risk prediction tools, clinical performance was suboptimal. By collating an overview of the range of approaches taken, researchers may further optimise the development of risk prediction tools in SPRINT-eligible populations, and our findings may inform the conduct of future similar open science projects.

Prediction of skin anti-aging clinical benefits of an association of ingredients from marine and maritime origins: Ex vivo evaluation using a label-free quantitative proteomic and customized data processing approach

BACKGROUND

The application of ingredients from marine and maritime origins is increasingly common in skin care products, driven by consumer expectations for natural ingredients. However, these ingredients are typically studied for a few isolated in vitro activities.

OBJECTIVES

The purpose of this study was to carry out a comprehensive evaluation of the activity on the skin of an association of ingredients from marine and maritime origins using label-free quantitative proteomic analysis, in order to predict the clinical benefits if used in a skin care product.

METHODS

An aqueous gel containing 6.1% of ingredients from marine and maritime origins (amino acid-enriched giant kelp extract, trace element-enriched seawater, dedifferentiated sea fennel cells) was topically applied on human skin explants. The skin explants' proteome was analyzed in a label-free manner by high-performance liquid nano-chromatography coupled with tandem mass spectrometry. A specific data processing pipeline (CORAVALID) providing an objective and comprehensive interpretation of the statistically relevant biological activities processed the results.

RESULTS

Compared to untreated skin explants, 64 proteins were significantly regulated by the gel treatment (q-value ≤ 0.05). Computer data processing revealed an activity of the ingredients on the epidermis and the dermis. These significantly regulated proteins are involved in gene expression, cell survival and metabolism, inflammatory processes, dermal extracellular matrix synthesis, melanogenesis and keratinocyte proliferation, migration, and differentiation.

CONCLUSIONS

These results suggest that the tested ingredients could help to preserve a healthy epidermis and dermis, and possibly to prevent the visible signs of skin aging.

Calibration drift in regression and machine learning models for acute kidney injury

Objective

Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.

Materials and Methods

Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.

Results

Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.

Conclusions

Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Prospective Validation of a Screening Tool to Identify Older Adults in Need of a Driving Evaluation

OBJECTIVES

To prospectively validate and refine the 5-item "CRASH" screening tool for identifying older drivers needing a behind-the-wheel (BTW) test.

DESIGN

Prospective observational study.

SETTING

Geriatric and internal medicine primary care clinics affiliated with a tertiary care hospital and a local BTW program.

PARTICIPANTS

Cognitively intact drivers aged 65 and older (N = 315).

MEASUREMENTS

Participants completed baseline questionnaire (including CRASH tool) and assessments and BTW test (evaluator blinded to questionnaire results) and participated in 1-month telephone follow-up. Analysis included descriptive statistics and examination of predictive ability of the CRASH tool to discriminate normal (pass) from abnormal (conditional pass or fail) on the BTW test, with logistic regression and CART techniques for tool refinement.

RESULTS

Two hundred sixty-six participants (84%) had a BTW test; of these, 17% had a normal rating and 83% an abnormal rating. Forty-five percent of those with an abnormal score were advised to limit driving under particular conditions. Neither the CRASH tool nor its individual component variables were significantly associated with the summary BTW score; in refined models with other variables, the best-performing tool had approximately 67% sensitivity and specificity for an abnormal BTW score. Most participants found the BTW test useful and were willing to pay a median of $50. At 1-month follow-up, no participants had stopped driving.

CONCLUSION

The CRASH screening tool cannot be recommended for use in clinical practice. Findings on older adults' perceived utility of the BTW test and the stability of driving patterns at 1-month follow-up could be useful for future research studies and for design of older driver programs.

Reliability and Utility of the Surprise Question in CKD Stages 4 to 5

BACKGROUND

Prognostic uncertainty is one barrier to engaging in goals-of-care discussions in chronic kidney disease (CKD). The surprise question ("Would you be surprised if this patient died in the next 12 months?") is a tool to assist in prognostication. However, it has not been studied in non-dialysis-dependent CKD and its reliability is unknown.

STUDY DESIGN

Observational study.

SETTING & PARTICIPANTS

388 patients at least 60 years of age with non-dialysis-dependent CKD stages 4 to 5 who were seen at an outpatient nephrology clinic.

PREDICTOR

Trinary (ie, Yes, Neutral, or No) and binary (Yes or No) surprise question response.

OUTCOMES

Mortality, test-retest reliability, and blinded inter-rater reliability.

MEASUREMENTS

Baseline comorbid conditions, Charlson Comorbidity Index, cause of CKD, and baseline laboratory values (ie, serum creatinine/estimated glomerular filtration rate, serum albumin, and hemoglobin).

RESULTS

Median patient age was 71 years with median follow-up of 1.4 years, during which time 52 (13%) patients died. Using the trinary surprise question, providers responded Yes, Neutral, and No for 202 (52%), 80 (21%), and 106 (27%) patients, respectively. About 5%, 15%, and 27% of Yes, Neutral, and No patients died, respectively (P<0.001). Trinary surprise question inter-rater reliability was 0.58 (95% CI, 0.42-0.72), and test-retest reliability was 0.63 (95% CI, 0.54-0.72). The trinary surprise question No response had sensitivity and specificity of 55% and 76%, respectively (95% CIs, 38%-71% and 71%-80%, respectively). The binary surprise question had sensitivity of 66% (95% CI, 49%-80%; P=0.3 vs trinary), but lower specificity of 68% (95% CI, 63%-73%; P=0.02 vs trinary).

LIMITATIONS

Single center, small number of deaths.

CONCLUSIONS

The surprise question associates with mortality in CKD stages 4 to 5 and demonstrates moderate to good reliability. Future studies should examine how best to deploy the surprise question to facilitate advance care planning in advanced non-dialysis-dependent CKD.

The accuracy of clinicians' predictions of survival in advanced cancer: a review

The process of formulating an accurate survival prediction is often difficult but important, as it influences the decisions of clinicians, patients, and their families. The current article aims to review the accuracy of clinicians' predictions of survival (CPS) in advanced cancer patients. A literature search of Cochrane CENTRAL, EMBASE, and MEDLINE was conducted to identify studies that reported clinicians' prediction of survival in advanced cancer patients. Studies were included if the subjects consisted of advanced cancer patients and the data reported on the ability of clinicians to predict survival, with both estimated and observed survival data present. Studies reporting on the ability of biological and molecular markers to predict survival were excluded. Fifteen studies that met the inclusion and exclusion criteria were identified. Clinicians in five studies underestimated patients' survival (estimated to observed survival ratio between 0.5 and 0.92). In contrast, 12 studies reported clinicians' overestimation of survival (ratio between 1.06 and 6). CPS in advanced cancer patients is often inaccurate and overestimated. Given these findings, clinicians should be aware of their tendency to be overoptimistic. Further investigation of predictive patient and clinician characteristics is warranted to improve clinicians' ability to predict survival.

Objective Bayesian model selection for Cox regression

There is now a large literature on objective Bayesian model selection in the linear model based on the g-prior. The methodology has been recently extended to generalized linear models using test-based Bayes factors. In this paper, we show that test-based Bayes factors can also be applied to the Cox proportional hazards model. If the goal is to select a single model, then both the maximum a posteriori and the median probability model can be calculated. For clinical prediction of survival, we shrink the model-specific log hazard ratio estimates with subsequent calculation of the Breslow estimate of the cumulative baseline hazard function. A Bayesian model average can also be employed. We illustrate the proposed methodology with the analysis of survival data on primary biliary cirrhosis patients and the development of a clinical prediction model for future cardiovascular events based on data from the Second Manifestations of ARTerial disease (SMART) cohort study. Cross-validation is applied to compare the predictive performance with alternative model selection approaches based on Harrell's c-Index, the calibration slope and the integrated Brier score. Finally, a novel application of Bayesian variable selection to optimal conditional prediction via landmarking is described. Copyright © 2016 John Wiley & Sons, Ltd.