Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets

Developing and evaluating prediction models in rehabilitation populations

This article presents a 3-part framework for developing and evaluating prediction models in rehabilitation populations. First, a process for developing and refining prognostic research questions and the scientific approach to prediction models is presented. Primary components of the scientific approach include the study design and sampling of patients, outcome measurement, selecting predictor variable(s), minimizing methodologic sources of bias, assuring a sufficient sample size for statistical power, and selecting an appropriate statistical model. Examples focus on prediction modeling using samples of rehabilitation patients. Second, a brief overview for statistically building and validating multivariable prediction models is provided, which includes the following 7 steps: data inspection, coding of predictors, model specification, model estimation, model performance, model validation, and model presentation. Third, we propose a set of primary considerations for evaluating prediction model studies using specific quality indicators as criteria to help stakeholders evaluate the quality of a prediction model study. Lastly, we offer perspectives on the future development and use of rehabilitation prediction models.

Early life stress and inflammatory mechanisms of fatigue in the Coronary Artery Risk Development in Young Adults (CARDIA) study

Fatigue is highly prevalent and causes serious disruption in quality of life. Although cross-sectional studies suggest childhood adversity is associated with adulthood fatigue, longitudinal evidence of this relationship and its specific biological mechanisms have not been established. This longitudinal study examined the association between early life stress and adulthood fatigue and tested whether this association was mediated by low-grade systemic inflammation as indexed by circulating C-reactive protein (CRP) and interleukin-6 (IL-6). In the Coronary Artery Risk Development in Young Adults (CARDIA) study, a population-based longitudinal study conducted in 4 US cities, early life stress was retrospectively assessed in 2716 African-American and white adults using the Risky Families Questionnaire at Year 15 examination (2000-2001, ages 33-45 years). Fatigue as indexed by a loss of subjective vitality using the Vitality Subscale of the 12-item Short Form Health Survey was assessed at both Years 15 and 20. While CRP was measured at both Years 15 and 20, IL-6 was measured only at Year 20. Early life stress assessed at Year 15 was associated with adulthood fatigue at Year 20 after adjustment for sociodemographic characteristics, body-mass index, medication use, medical comorbidity, smoking, alcohol consumption, physical activity, current stress, pain, sleep disturbance as well as Year 15 fatigue (adjusted beta 0.047, P=0.007). However, neither CRP nor IL-6 was a significant mediator of this association. In summary, early life stress assessed in adulthood was associated with fatigue 5 years later, but this association was not mediated by low-grade systemic inflammation.

Prognostic factors for a favourable long-term outcome from an integrative psychotherapeutic nursing home programme

BACKGROUND/AIMS

The prevalence of multiple psychiatric symptoms (MPS) in psychogeriatric patients is about 80%. MPS have negative effects on caregivers; 70-80% of caregivers are moderately to heavily burdened. We tested an integrative psychotherapeutic programme (IRR) focused on MPS as well as caregiver burden. To develop decision rules in indicating IRR, prognostic potentialities of diagnostic and functional baseline variables for a favourable outcome of IRR were identified.

METHODS

Patients with a DSM-IV classification of dementia, amnestic disorders or other cognitive disorders were followed in a randomised controlled trial, comparing IRR (n=81) with usual nursing home care (n=87). Assessments at T1 (intake) and T2 (6 months' follow-up).

RESULTS

In the combined prognostic models Alzheimer dementia showed significant prognostic qualities for improvement on NPI sum severity (OR 3.01), IRR on general burden and competence of caregiver (OR 2.29 and 3.34). Cognitive functions had low prognostic value.

CONCLUSION

Prognostic modelling of positive change on severity of MPS and caregiver burden was feasible. Applying three decision rules, all resulted in IRR as indicated intervention. It seems justified to refer psychogeriatric patients suffering from a broad range of cognitive function disorders, specifically patients with dementia of the Alzheimer type, to the IRR programme.

Perinatal risk-indicators for long-term respiratory morbidity among preterm or very low birth weight neonates

OBJECTIVES

To develop prediction models for long-term respiratory morbidity. To explore if respiratory distress syndrome (RDS) is a risk-indicator for long-term respiratory morbidity and to identify other perinatal risk-indicators for long-term respiratory morbidity.

STUDY DESIGN

In the Dutch POPS cohort 1338 live born infants delivered in The Netherlands in 1983, either before 32 completed weeks gestation and/or with a birth weight below 1500 g, were followed prospectively. We used multivariable logistic regression analyses to construct three prediction models for respiratory morbidity at 2, 5 and 19 years of age.

RESULTS

At 2 years of age, maternal smoking (adjusted OR 1.5, 95% CI 1.0-2.4), prolonged rupture of membranes (adjusted OR 2.3, 95% CI 1.3-4.1), pre-eclampsia (adjusted OR 1.9, 95% CI 1.1-4.1), male gender (adjusted OR 1.5, 95% CI 1.1-2.0) and BPD (adjusted OR 1.9, 95% CI 1.1-3.2) were significantly associated with respiratory morbidity. Prolonged rupture of membranes (adjusted OR 3.7, 95% CI 1.6-8.5), family history of asthma (adjusted OR 5.9, 95% CI 2.7-13.0) and BPD (adjusted OR 1.8, 95% CI 1.1-3.0) were significantly associated with respiratory morbidity at 5 years of age. At 19 years of age only higher social class was associated with decreased respiratory morbidity (adjusted OR 0.64, 95% CI 0.41-0.99). The areas under the curves (AUC) were 0.65, 0.71 and 0.61 respectively. The prediction models for respiratory morbidity at 2 and 5 years of age showed a good calibration, while the calibration plot for respiratory morbidity at 19 year was less optimal.

CONCLUSIONS

RDS is not a risk-indicator for long-term respiratory morbidity at 2, 5 and 19 years in this cohort (OR 1.2, 95% 0.88-1.7; 1.3, 95% 0.88-2.0; OR 0.91, 95% 0.56-1.5 respectively). Future obstetric studies interested in the effect of a specific perinatal intervention on long-term respiratory morbidity, should consider taking bronchopulmonary dysplasia (BPD) as primary outcome instead of RDS.

The prediction of functional decline in older hospitalised patients

BACKGROUND

thirty to sixty per cent of older patients experience functional decline after hospitalisation, associated with an increase in dependence, readmission, nursing home placement and mortality. First step in prevention is the identification of patients at risk.

OBJECTIVE

to develop and validate a prediction model to assess the risk of functional decline in older hospitalised patients.

DESIGN

development study: cohort study (n = 492). Validation study: secondary data analysis of a cohort study (n = 484) in an independent population. Both with follow-up after 3 months. Functional decline was defined as a decline of at least one point on the Katz ADL index at follow-up compared with pre-admission status.

SETTING

development study: general internal medicine wards of two university hospitals and one regional hospital. Validation study: general internal wards of an university hospital.

SUBJECTS

patients ≥65 years acutely admitted and hospitalised for at least 48 h.

RESULTS

thirty-five per cent of all patients in the development cohort and 32% in the validation cohort developed functional decline. A four-item model could accurately predict functional decline with an AUC of 0.71. At threshold 2 sensitivity, specificity, positive and negative predictive values were 87, 39, 43 and 85%, respectively. In the validation study, this was, respectively, 0.68, 89, 41, 41 and 89%.

CONCLUSION

pre-admission need for assistance in instrumental activities of daily living, use of a walking device, need for assistance in travelling and no education after age 14, are the items of a prediction model to identify older patients at risk for functional decline following hospital admission. The strength of the model is that it relies on four simple questions and this makes it easy to use in clinical practice and easy to administer.

A multi-institutional study of pancreatic cancer in Harris County, Texas: race predicts treatment and survival

BACKGROUND

Racial disparities exist for patients with pancreatic cancer. This observation has primarily been noted in the Surveillance, Epidemiology, and End Results database and has focused primarily on whites and African Americans. We sought to determine if these disparities exist in a local, racially diverse patient population.

METHODS

Retrospective review of a pancreatic cancer tumor registry from two hospital systems from 1998 to 2010. Clinicopathologic parameters were recorded. Statistical analysis was performed by analysis of variance, Chi square test, Kaplan-Meier survival analysis, log rank test, and regression models.

RESULTS

A total of 1039 patients were identified for this study. Hispanic and African American patients presented at an earlier age when compared to whites. There was no difference in gender or stage at presentation between racial groups. Adjusted for stage, race was predictive of chemotherapy administration. Independent predictors of increased mortality included male gender, African American race, stage at diagnosis, and older age.

CONCLUSIONS

Despite adjusting for covariates, survival remains lowest for African American patients. Further investigation is needed to understand the effect of race and how it mediates treatment and survival in those with pancreatic cancer.

Predicting prostate biopsy outcome: prostate health index (phi) and prostate cancer antigen 3 (PCA3) are useful biomarkers

Indication for prostate biopsy is presently mainly based on prostate-specific antigen (PSA) serum levels and digital-rectal examination (DRE). In view of the unsatisfactory accuracy of these two diagnostic exams, research has focused on novel markers to improve pre-biopsy prostate cancer detection, such as phi and PCA3. The purpose of this prospective study was to assess the diagnostic accuracy of phi and PCA3 for prostate cancer using biopsy as gold standard. Phi index (Beckman coulter immunoassay), PCA3 score (Progensa PCA3 assay) and other established biomarkers (tPSA, fPSA and %fPSA) were assessed before a 18-core prostate biopsy in a group of 251 subjects at their first biopsy. Values of %p2PSA and phi were significantly higher in patients with PCa compared with PCa-negative group (p<0.001) and also compared with high grade prostatic intraepithelial neoplasia (HGPIN) (p<0.001). PCA3 score values were significantly higher in PCa compared with PCa-negative subjects (p<0.001) and in HGPIN vs PCa-negative patients (p<0.001). ROC curve analysis showed that %p2PSA, phi and PCA3 are predictive of malignancy. In conclusion, %p2PSA, phi and PCA3 may predict a diagnosis of PCa in men undergoing their first prostate biopsy. PCA3 score is more useful in discriminating between HGPIN and non-cancer.

Survival in MS: a randomized cohort study 21 years after the start of the pivotal IFNβ-1b trial

OBJECTIVE

To examine the effects of interferon beta (IFNβ)-1b on all-cause mortality over 21 years in the cohort of 372 patients who participated in the pivotal randomized clinical trial (RCT), retaining (in the analysis) the original randomized treatment-assignments.

METHODS

For this randomized long-term cohort study, the primary outcome, defined before data collection, was the comparison of all-cause mortality between the IFNβ-1b 250 μg and placebo groups from the time of randomization through the entire 21-year follow-up interval (intention-to-treat, log-rank test for Kaplan-Meier survival curves). All other survival outcomes were secondary.

RESULTS

After a median of 21.1 years from RCT enrollment, 98.4%(366 of 372) of patients were identified, and, of these, 81 deaths were recorded (22.1% [81 of 366]). Patients originally randomly assigned to IFNβ-1b 250 μg showed a significant reduction in all-cause mortality over the 21-year period compared with placebo (p = 0.0173), with a hazard ratio of 0.532 (95% confidence interval 0.314-0.902). The hazard rate of death at long-term follow-up by Kaplan-Meier estimates was reduced by 46.8% among IFNβ-1b 250 μg-treated patients (46.0% among IFNβ-1b 50 μg-treated patients) compared with placebo. Baseline variables did not influence the observed treatment effect.

CONCLUSIONS

There was a significant survival advantage in this cohort of patients receiving early IFNβ-1b treatment at either dose compared with placebo. Near-complete ascertainment, together with confirmatory findings from both active treatment groups, strengthens the evidence for an IFNβ-1b benefit on all-cause mortality.

CLASSIFICATION OF EVIDENCE

This study provides Class III evidence that early treatment with IFNβ-1b is associated with prolonged survival in initially treatment-naive patients with relapsing-remitting multiple sclerosis.

Longitudinal evaluation of CA-125 velocity and prediction of ovarian cancer

OBJECTIVE

To determine whether CA-125 velocity is a statistically significant predictor of ovarian cancer and develop a classification rule to screen for ovarian cancer.

METHODS

In the ovarian component of the PLCO cancer screening trial, 28,038 women aged 55-74 had at least two CA-125 screening tests. Ovarian cancer was diagnosed in 72 (0.26%) women. A multiple logistic regression model was developed to evaluate CA-125 velocity and other related covariates as predictors of ovarian cancer. Predictive accuracy was assessed by the concordance index and measures of discrimination and calibration while the fit of the model was assessed by the Hosmer and Lemeshow's goodness-of-fit χ(2)test.

RESULTS

CA-125 velocity decreased as the number of CA-125 measurements increased but was unaffected by age at baseline screen and family history of ovarian cancer. The average velocity (19.749U/ml per month) of the cancer group was more than 500 times the average velocity (0.035U/ml per month) of the non-cancer group.

CONCLUSION

Among six covariates used in the model, CA-125 velocity and time intervals between baseline and second to last screening test and between last two screening tests were statistically significant predictors of ovarian cancer. The chance of having ovarian cancer increased as velocity increased, and the chance decreased when the time intervals between baseline and the second to last screening test and between last two screening tests of an individual increased.

Modeling chemical interaction profiles: II. Molecular docking, spectral data-activity relationship, and structure-activity relationship models for potent and weak inhibitors of cytochrome P450 CYP3A4 isozyme

Polypharmacy increasingly has become a topic of public health concern, particularly as the U.S. population ages. Drug labels often contain insufficient information to enable the clinician to safely use multiple drugs. Because many of the drugs are bio-transformed by cytochrome P450 (CYP) enzymes, inhibition of CYP activity has long been associated with potentially adverse health effects. In an attempt to reduce the uncertainty pertaining to CYP-mediated drug-drug/chemical interactions, an interagency collaborative group developed a consensus approach to prioritizing information concerning CYP inhibition. The consensus involved computational molecular docking, spectral data-activity relationship (SDAR), and structure-activity relationship (SAR) models that addressed the clinical potency of CYP inhibition. The models were built upon chemicals that were categorized as either potent or weak inhibitors of the CYP3A4 isozyme. The categorization was carried out using information from clinical trials because currently available in vitro high-throughput screening data were not fully representative of the in vivo potency of inhibition. During categorization it was found that compounds, which break the Lipinski rule of five by molecular weight, were about twice more likely to be inhibitors of CYP3A4 compared to those, which obey the rule. Similarly, among inhibitors that break the rule, potent inhibitors were 2–3 times more frequent. The molecular docking classification relied on logistic regression, by which the docking scores from different docking algorithms, CYP3A4 three-dimensional structures, and binding sites on them were combined in a unified probabilistic model. The SDAR models employed a multiple linear regression approach applied to binned 1D ¹³C-NMR and 1D ¹⁵N-NMR spectral descriptors. Structure-based and physical-chemical descriptors were used as the basis for developing SAR models by the decision forest method. Thirty-three potent inhibitors and 88 weak inhibitors of CYP3A4 were used to train the models. Using these models, a synthetic majority rules consensus classifier was implemented, while the confidence of estimation was assigned following the percent agreement strategy. The classifier was applied to a testing set of 120 inhibitors not included in the development of the models. Five compounds of the test set, including known strong inhibitors dalfopristin and tioconazole, were classified as probable potent inhibitors of CYP3A4. Other known strong inhibitors, such as lopinavir, oltipraz, quercetin, raloxifene, and troglitazone, were among 18 compounds classified as plausible potent inhibitors of CYP3A4. The consensus estimation of inhibition potency is expected to aid in the nomination of pharmaceuticals, dietary supplements, environmental pollutants, and occupational and other chemicals for in-depth evaluation of the CYP3A4 inhibitory activity. It may serve also as an estimate of chemical interactions via CYP3A4 metabolic pharmacokinetic pathways occurring through polypharmacy and nutritional and environmental exposures to chemical mixtures.

Predicting infectious complications in neutropenic children and young people with cancer (IPD protocol)

BACKGROUND

A common and potentially life-threatening complication of the treatment of childhood cancer is infection, which frequently presents as fever with neutropenia. The standard management of such episodes is the extensive use of intravenous antibiotics, and though it produces excellent survival rates of over 95%, it greatly inconveniences the three-fourths of patients who do not require such aggressive treatment. There have been a number of studies which have aimed to develop risk prediction models to stratify treatment. Individual participant data (IPD) meta-analysis in therapeutic studies has been developed to improve the precision and reliability of answers to questions of treatment effect and recently have been suggested to be used to answer questions regarding prognosis and diagnosis to gain greater power from the frequently small individual studies.

DESIGN

In the IPD protocol, we will collect and synthesise IPD from multiple studies and examine the outcomes of episodes of febrile neutropenia as a consequence of their treatment for malignant disease. We will develop and evaluate a risk stratification model using hierarchical regression models to stratify patients by their risk of experiencing adverse outcomes during an episode. We will also explore specific practical and methodological issues regarding adaptation of established techniques of IPD meta-analysis of interventions for use in synthesising evidence derived from IPD from multiple studies for use in predictive modelling contexts.

DISCUSSION

Our aim in using this model is to define a group of individuals at low risk for febrile neutropenia who might be treated with reduced intensity or duration of antibiotic therapy and so reduce the inconvenience and cost of these episodes, as well as to define a group of patients at very high risk of complications who could be subject to more intensive therapies. The project will also help develop methods of IPD predictive modelling for use in future studies of risk prediction.

Multivariate modeling of complications with data driven variable selection: guarding against overfitting and effects of data set size

PURPOSE

Multivariate modeling of complications after radiotherapy is frequently used in conjunction with data driven variable selection. This study quantifies the risk of overfitting in a data driven modeling method using bootstrapping for data with typical clinical characteristics, and estimates the minimum amount of data needed to obtain models with relatively high predictive power.

MATERIALS AND METHODS

To facilitate repeated modeling and cross-validation with independent datasets for the assessment of true predictive power, a method was developed to generate simulated data with statistical properties similar to real clinical data sets. Characteristics of three clinical data sets from radiotherapy treatment of head and neck cancer patients were used to simulate data with set sizes between 50 and 1000 patients. A logistic regression method using bootstrapping and forward variable selection was used for complication modeling, resulting for each simulated data set in a selected number of variables and an estimated predictive power. The true optimal number of variables and true predictive power were calculated using cross-validation with very large independent data sets.

RESULTS

For all simulated data set sizes the number of variables selected by the bootstrapping method was on average close to the true optimal number of variables, but showed considerable spread. Bootstrapping is more accurate in selecting the optimal number of variables than the AIC and BIC alternatives, but this did not translate into a significant difference of the true predictive power. The true predictive power asymptotically converged toward a maximum predictive power for large data sets, and the estimated predictive power converged toward the true predictive power. More than half of the potential predictive power is gained after approximately 200 samples. Our simulations demonstrated severe overfitting (a predicative power lower than that of predicting 50% probability) in a number of small data sets, in particular in data sets with a low number of events (median: 7, 95th percentile: 32). Recognizing overfitting from an inverted sign of the estimated model coefficients has a limited discriminative value.

CONCLUSIONS

Despite considerable spread around the optimal number of selected variables, the bootstrapping method is efficient and accurate for sufficiently large data sets, and guards against overfitting for all simulated cases with the exception of some data sets with a particularly low number of events. An appropriate minimum data set size to obtain a model with high predictive power is approximately 200 patients and more than 32 events. With fewer data samples the true predictive power decreases rapidly, and for larger data set sizes the benefit levels off toward an asymptotic maximum predictive power.

Development and validation of clinical prediction models: marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming

OBJECTIVE

Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods.

STUDY DESIGN AND SETTING

We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration.

RESULTS

There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations.

CONCLUSION

The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.

Age at diagnosis of inflammatory bowel disease influences early development of colorectal cancer in inflammatory bowel disease patients: a nationwide, long-term survey

BACKGROUND

Data on clinical characteristics of patients with inflammatory bowel disease (IBD)-related colorectal cancer (CRC) are scarce and mainly originate from tertiary referral centres. We studied patient and disease characteristics of IBD-related CRC in a nationwide IBD cohort in general hospitals. Main outcome parameters were time to develop CRC, and factors associated with early CRC development.

METHODS

All IBD patients diagnosed with CRC between 1 January 1990 and 1 July 2006 were identified using a nationwide automated pathology database (PALGA). Patient charts were assessed to confirm diagnosis and collect clinical data. Early CRC was defined as CRC diagnosed less than 8 years after IBD diagnosis. Statistical analysis was performed using descriptive statistics, independent t tests, binary logistic regression and Cox-regression analysis.

RESULTS

Diagnosis of IBD-related CRC was confirmed in 251 patients (171 ulcerative colitis, 77 Crohn's disease, 3 unclassified colitis), 161 males (64 %). Median time from IBD diagnosis to CRC diagnosis was 12 years (IQR 4-20); 89 patients (35 %) developed early CRC. Type of IBD, gender, concomitant PSC, pseudopolyps, extent of inflammation, and medication use were not related to early CRC (p > 0.05). IBD diagnosis at older age (HR for 10 years older age 2.25; 95 % CI 1.92-2.63) was related to early CRC. Twenty-three patients (12 %) had been included in a surveillance programme prior to CRC diagnosis. Patients in the surveillance group had a significantly better tumor stage (p = 0.004).

CONCLUSIONS

We emphasize the problem of a high proportion of IBD-associated CRCs developing before the recommended start of surveillance. Therefore, we suggest that older age at IBD onset could be an additional factor to start surveillance in IBD patients.

Maternal pregnancy-specific anxiety is associated with child executive function at 6-9 years age

Because fetal brain development proceeds at an extremely rapid pace, early life experiences have the potential to alter the trajectory of neurodevelopment, which may increase susceptibility for developmental and neuropsychiatric disorders. There is evidence that prenatal maternal stress and anxiety, especially worries specifically related to being pregnant, influence neurodevelopmental outcomes. In the current prospective longitudinal study, we included 89 women for whom serial data were available for pregnancy-specific anxiety, state anxiety, and depression at 15, 19, 25, 31, and 37 weeks gestation. When the offspring from the target pregnancy were between 6 and 9 years of age, their executive function was assessed. High levels of mean maternal pregnancy-specific anxiety over the course of gestation were associated with lower inhibitory control in girls only and lower visuospatial working memory performance in boys and girls. Higher-state anxiety and depression also were associated with lower visuospatial working memory performance. However, neither state anxiety nor depression explained any additional variance after accounting for pregnancy-specific anxiety. The findings contribute to the literature supporting an association between pregnancy-specific anxiety and cognitive development and extend our knowledge about the persistence of this effect until middle childhood.

Decision support tool for early differential diagnosis of acute lung injury and cardiogenic pulmonary edema in medical critically ill patients

BACKGROUND

At the onset of acute hypoxic respiratory failure, critically ill patients with acute lung injury (ALI) may be difficult to distinguish from those with cardiogenic pulmonary edema (CPE). No single clinical parameter provides satisfying prediction. We hypothesized that a combination of those will facilitate early differential diagnosis.

METHODS

In a population-based retrospective development cohort, validated electronic surveillance identified critically ill adult patients with acute pulmonary edema. Recursive partitioning and logistic regression were used to develop a decision support tool based on routine clinical information to differentiate ALI from CPE. Performance of the score was validated in an independent cohort of referral patients. Blinded post hoc expert review served as gold standard.

RESULTS

Of 332 patients in a development cohort, expert reviewers (κ, 0.86) classified 156 as having ALI and 176 as having CPE. The validation cohort had 161 patients (ALI = 113, CPE = 48). The score was based on risk factors for ALI and CPE, age, alcohol abuse, chemotherapy, and peripheral oxygen saturation/Fio(2) ratio. It demonstrated good discrimination (area under curve [AUC] = 0.81; 95% CI, 0.77-0.86) and calibration (Hosmer-Lemeshow [HL] P = .16). Similar performance was obtained in the validation cohort (AUC = 0.80; 95% CI, 0.72-0.88; HL P = .13).

CONCLUSIONS

A simple decision support tool accurately classifies acute pulmonary edema, reserving advanced testing for a subset of patients in whom satisfying prediction cannot be made. This novel tool may facilitate early inclusion of patients with ALI and CPE into research studies as well as improve and rationalize clinical management and resource use.

Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting

BACKGROUND

The World Health Organisation estimates that by 2030 there will be approximately 350 million people with type 2 diabetes. Associated with renal complications, heart disease, stroke and peripheral vascular disease, early identification of patients with undiagnosed type 2 diabetes or those at an increased risk of developing type 2 diabetes is an important challenge. We sought to systematically review and critically assess the conduct and reporting of methods used to develop risk prediction models for predicting the risk of having undiagnosed (prevalent) or future risk of developing (incident) type 2 diabetes in adults.

METHODS

We conducted a systematic search of PubMed and EMBASE databases to identify studies published before May 2011 that describe the development of models combining two or more variables to predict the risk of prevalent or incident type 2 diabetes. We extracted key information that describes aspects of developing a prediction model including study design, sample size and number of events, outcome definition, risk predictor selection and coding, missing data, model-building strategies and aspects of performance.

RESULTS

Thirty-nine studies comprising 43 risk prediction models were included. Seventeen studies (44%) reported the development of models to predict incident type 2 diabetes, whilst 15 studies (38%) described the derivation of models to predict prevalent type 2 diabetes. In nine studies (23%), the number of events per variable was less than ten, whilst in fourteen studies there was insufficient information reported for this measure to be calculated. The number of candidate risk predictors ranged from four to sixty-four, and in seven studies it was unclear how many risk predictors were considered. A method, not recommended to select risk predictors for inclusion in the multivariate model, using statistical significance from univariate screening was carried out in eight studies (21%), whilst the selection procedure was unclear in ten studies (26%). Twenty-one risk prediction models (49%) were developed by categorising all continuous risk predictors. The treatment and handling of missing data were not reported in 16 studies (41%).

CONCLUSIONS

We found widespread use of poor methods that could jeopardise model development, including univariate pre-screening of variables, categorisation of continuous risk predictors and poor handling of missing data. The use of poor methods affects the reliability of the prediction model and ultimately compromises the accuracy of the probability estimates of having undiagnosed type 2 diabetes or the predicted risk of developing type 2 diabetes. In addition, many studies were characterised by a generally poor level of reporting, with many key details to objectively judge the usefulness of the models often omitted.

Atrial fibrillation and flutter outcomes and risk determination (AFFORD): design and rationale

BACKGROUND

Atrial fibrillation (AF) is the most common sustained arrhythmia affecting over 700,000 individuals in Japan and 2.2 million in the USA. The proper management of patients with AF is critical due to the well-documented association with heart failure and stroke. A strategy to better define the emergency department (ED) management, admission decisions, and spectrum of risk from low to high is needed.

METHODS AND SUBJECTS

The atrial fibrillation and flutter outcomes and risk determination investigation is a prospective, observational cohort study to develop a multivariable clinical prediction rule that accurately estimates risk for adverse outcomes in patients presenting to the ED with symptomatic AF. We will enroll 430 patients at 2 sites who present to the ED with symptomatic AF defined as a new or established diagnosis of AF or atrial flutter that require ED evaluation for a complaint thought related to their rhythm disturbance. The study's endpoint is to develop an accurate, objective, internally validated, reliable clinical prediction rule to risk-stratify ED patients presenting with AF exacerbations. The rule will incorporate patient history and examination findings and laboratory studies obtained upon ED presentation, as well as trends over the first 2 h of care. This investigation's primary outcome is the incidence of any AF-related adverse event at 5 days and 30 days. We expect to complete the study by the end of 2014. The study was registered at Clinicaltrials.gov NCT01138644.

Magnetic resonance imaging for evaluation of Crohn's disease: validation of parameters of severity and quantitative index of activity

BACKGROUND

The use of magnetic resonance imaging (MRI) for assessment of Crohn's disease (CD) is expanding. The aim of this study is to define and provide an external validation of the MRI predictors of active CD, severe CD, and a quantitative Magnetic Resonance Index of Activity (MaRIA).

METHODS

In all, 48 patients with clinically active (n = 29) or inactive (n = 19) CD underwent ileocolonoscopy (reference standard) and MRI. T2-weighted and pre- and postcontrast-enhanced T1-weighted sequences were acquired. Endoscopic activity was evaluated by the Crohn's Disease Endoscopic Index of Severity (CDEIS), and also classified as absent, mild (inflammation without ulcers), or severe (presence of ulceration).

RESULTS

In complete agreement with a previous derivation study, independent predictors of disease severity using CDEIS as a reference were wall thickness, relative contrast enhancement (RCE), presence of edema, and ulcers on MRI. Estimation of activity in each segment using this regression model, or another with simplified coefficients (MaRIA(S) = 1.5*wall thickness + 0.02*RCE + 5*edema + 10*ulceration) correlated with CDEIS (r = 0.798, P< 0.001; r = 0.80 P < 0.001, respectively). In the validation cohort both indexes had a high and equal accuracy for diagnosis of active disease: receiver operator characteristic (ROC) area 0.93, sensitivity 0.87, specificity 0.87 using a cutoff point ≥ 7, and for diagnosis of severe disease: ROC area 0.96, sensitivity 0.92, specificity 0.92 using a cutoff point ≥ 11. The total of segment values (MaRIA(T)) correlated with global CDEIS (r = 0.83, P< 0.001).

CONCLUSIONS

The MRI variables that should be evaluated in clinical practice to diagnose active CD and severe CD are validated, as well as the quantitative index of activity for use in research studies.

Changing family routines at kindergarten entry predict biomarkers of parental stress

This study tested associations among parenting stress prior to a child’s kindergarten entry, the sustainability of family routines, and biomarkers of stress among parents following the kindergarten transition. Parents (N = 51) with higher prekindergarten scores on the Parenting Stress Index Short Form reported lower Family Routines Inventory scores following school entry relative to their baseline. Declining family routines, in turn, were associated following kindergarten entry with greater 5-day mean and variance in evening cortisol, and higher C-reactive protein, an inflammatory mediator. However, only the cortisol findings remained significant controlling for baseline physiology. These findings support a family systems, social-ecological approach to life course development, wherein even mild challenges posed by children’s normative transitions may reveal differences in parents’ biobehavioral functioning.

Effect of marriage on duration of chest pain associated with acute myocardial infarction before seeking care

BACKGROUND

Coronary artery disease is the most common cause of death in the Western world, and being married decreases the risk of death from cardiovascular causes. We aimed to determine whether marital status was a predictor of the duration of chest pain endured by patients with acute myocardial infarction before they sought care and whether the patient's sex modified the effect.

METHODS

We conducted a retrospective, population-based cohort analysis of patients with acute myocardial infarction admitted to 96 acute care hospitals in Ontario, Canada, from April 2004 to March 2005. We excluded patients who did not experience chest pain. Using multivariable regression analyses, we assessed marital status in relation to delayed presentation to hospital (more than six hours from onset of pain), both overall and stratified by sex. In patients who reported the exact duration of chest pain, we assessed the effect of marital status on the delay in seeking care.

RESULTS

Among 4403 eligible patients with acute myocardial infarction, the mean age was 67.3 (standard deviation 13.6) years, and 1486 (33.7%) were women. Almost half (2037 or 46.3%) presented to a hospital within two hours, and 3240 (73.6%) presented within six hours. Overall, 75.3% (2317/3079) of married patients, 67.9% (188/277) of single patients, 68.5% (189/276) of divorced patients and 70.8% (546/771) of widowed patients presented within six hours of the onset of chest pain. Being married was associated with lower odds of delayed presentation (odds ratio [OR] 0.46, 95% confidence interval [CI] 0.30-0.71, p < 0.001) relative to being single. Among men, the OR was 0.35 (95% CI 0.21-0.59, p < 0.001), whereas among women the effect of marital status was not significant (OR 1.36, 95% CI 0.49-3.73, p = 0.55).

INTERPRETATION

Among men experiencing acute myocardial infarction with chest pain, being married was associated with significantly earlier presentation for care, a benefit that was not observed for married women. Earlier presentation for medical care appears to be one reason for the observed lower risk of cardiovascular death among married men, relative to their single counterparts.

Multivariate predictive model for dyslexia diagnosis

Dyslexia is a specific disorder of language development that mainly affects reading. Etiological researches have led to multiple hypotheses which induced various diagnosis methods and rehabilitation treatments so that many different tests are used by practitioners to identify dyslexia symptoms. Our purpose is to determine a subset of the most efficient ones by integrating them into a multivariate predictive model. A set of screening tasks that are the most commonly used and representative of the different cognitive aspects of dyslexia was proposed to 78 children from elementary school (mean age = 9 years ± 7 months) exempt from identified reading difficulties and to 35 dyslexic children attending a specialized consultation for dyslexia. We proposed a multi-step procedure: within each category, we first selected the most representative tasks using principal component analysis and then we implemented logistic regression models on the preselected variables. Spelling and reading tasks were considered separately. The model with the best predictive performance includes eight variables from four categories of tasks and classifies correctly 94% of the children. The sensitivity (91%) and the specificity (95%) are both high. Forty minutes are necessary to complete the test.

Predictors of outcome of anorectal biofeedback therapy in patients with constipation

BACKGROUND

Anorectal biofeedback therapy (BFT) is a safe and effective treatment in patients with constipation. Given the high prevalence of constipation and therefore high demand for BFT, there is a need to prioritise patients.

AIMS

To explore clinical features and anorectal physiology which predict success or failure of BFT and to derive a statistical model which helps to predict the success of BFT.

METHODS

A total of 102 patients with constipation referred for BFT were evaluated. All patients underwent comprehensive clinical and anorectal function assessment, including balloon expulsion testing. The BFT protocol consisted of a comprehensive 6-weekly visit programme comprising instruction on toilet behaviour and abdominal breathing, achieving adequate rectal pressure and anal relaxation, and balloon expulsion and rectal sensory retraining. Success of BFT was based on an improvement in global bowel satisfaction.

RESULTS

Harder stool consistency (P=0.009), greater willingness to participate (P<0.001), higher resting anal sphincter pressure (P=0.04) and prolonged balloon expulsion time (P=0.02) correlated with an improvement in bowel satisfaction score. A longer duration of laxative use (P=0.049) correlated with no improvement in bowel satisfaction score. Harder stools, shorter duration of laxative use, higher straining rectal pressure and prolonged balloon expulsion independently predicted successful BFT. A model (S(i) = (p)∑ β(j)X(ij), where β represents a regression coefficient, X is a given predictive variable and S(i) is the weighted index score for each individual) incorporating these four variables enabled prediction of successful BFT, with sensitivity and specificity of 0.79 and 0.81, respectively.

CONCLUSIONS

Important clinical and anorectal physiological features were found to be associated with outcome of anorectal biofeedback therapy in patients with constipation. This information and the predictive model will assist clinicians to prioritise patients for anorectal biofeedback therapy.

The Acute Asthma Severity Assessment Protocol (AASAP) study: objectives and methods of a study to develop an acute asthma clinical prediction rule

Acute asthma exacerbations are one of the most common reasons for paediatric emergency department visits and hospitalisations, and a relapse frequently necessitates repeat urgent care. While care plans exist, there are no acute asthma prediction rules (APRs) to assess severity and predict outcome. The primary objective of the Acute Asthma Severity Assessment Protocol study is to develop a multivariable APR for acute asthma exacerbations in paediatric patients. A prospective, convenience sample of paediatric patients aged 5-17 years with acute asthma exacerbations who present to an urban, academic, tertiary paediatric emergency department was enrolled. The study protocol and data analysis plan conform to accepted biostatistical and clinical standards for clinical prediction rule development. Modelling of the APR will be performed once the entire sample size of 1500 has accrued. It is anticipated that the APR will improve resource utilisation in the emergency department, aid in standardisation of disease assessment and allow physician and non-physician providers to participate in earlier objective decision making. The objective of this report is to describe the study objectives and detailed methodology of the Acute Asthma Severity Assessment Protocol study.

Perinatal risk indicators for long-term neurological morbidity among preterm neonates

OBJECTIVE

Many obstetric interventions are performed to improve long-term neonatal outcome. However, long-term neonatal outcome is usually not a primary outcome because it is time-consuming and expensive. The aim of this project was to identify different perinatal risk indicators and to develop prediction models for neurologic morbidity at 2 and 5 years of age.

STUDY DESIGN

Data from a Dutch cohort study of preterm and small-for-gestational-age infants was used. Neonates who were born in The Netherlands in 1983 with a gestational age of <34 weeks and without congenital abnormalities were included (n = 753). Infants were divided in 3 groups: no handicap, minor handicap, and major handicap.

RESULTS

Common risk indicators for major handicaps at 2 and 5 years of age were male sex (odds ratio, 2.7 and 3.0, respectively), seizures after ≥2 days of life (odds ratio, 5.8 and 5.8, respectively), and intracranial hemorrhage (odds ratio, 3.8 and 2.6, respectively).

CONCLUSION

In this cohort, male sex, intracranial hemorrhage, and seizures seem to be important risk indicators for long-term neurologic morbidity.

[Regression modeling strategies]

Multivariable regression models are widely used in health science research, mainly for two purposes: prediction and effect estimation. Various strategies have been recommended when building a regression model: a) use the right statistical method that matches the structure of the data; b) ensure an appropriate sample size by limiting the number of variables according to the number of events; c) prevent or correct for model overfitting; d) be aware of the problems associated with automatic variable selection procedures (such as stepwise), and e) always assess the performance of the final model in regard to calibration and discrimination measures. If resources allow, validate the prediction model on external data.

A prediction model for left ventricular mass in patients at high cardiovascular risk

BACKGROUND

Left ventricular (LV) mass has a continuous relation with cardiovascular risk, and regression of LV mass induced by pharmacological treatment is associated with improved prognosis. Therefore, early identification of patients with a large LV mass is desired. We developed a model to predict LV mass in individual hypertensives at high cardiovascular risk.

DESIGN AND METHODS

We analyzed data of 536 hypertensives with symptomatic extracardiac atherosclerotic disease or marked risk factors for atherosclerosis from a cross-sectional study in a tertiary referral center. LV mass was measured by cardiac MRI. We developed the prediction rule with multivariable linear regression analysis and stepwise backward elimination. Internal validation was assessed with bootstrap sampling to obtain an estimate of model performance (R²) that may be expected for new patients.

RESULTS

Important predictors for LV mass included sex, height, body mass index, systolic blood pressure, and previous aneurysm of the abdominal aorta. R² of the prediction model was 45% after internal validation, which was considerably higher than the R² of previously reported models (range 1-38%). Addition of electrocardiography data showed limited improvement of the model performance (R²=47%).

CONCLUSION

We present a prediction model for LV mass in hypertensives at high cardiovascular risk. After external validation, this model may be used in clinical practice to estimate LV mass for early identification of large LV mass. The predictions of the model may support appropriate medical care in the prevention of cardiovascular disease.

Validation and refinement of the Millennium Criteria for atopic dermatitis

There is no gold standard for a definite diagnosis of atopic dermatitis. For the time being, several lists of diagnostic criteria have been proposed, some of them in actual use. The Millennium Criteria have been proposed to diagnose atopic dermatitis and to differentiate it from atopiform dermatitis. Our aim was to further refine the Millennium Criteria into a manageable set that can differentiate between atopic and atopiform dermatitis and other entities. The hereby refined Millennium Criteria will be compared with the UK Working Party Criteria and the Hanifin & Rajka Criteria. Data of 210 included patients were used. After multiple logistic regression, a minimum set of five criteria was identified as best discriminators: (i) typical morphology; (ii) early age of onset; (iii) Dennie-Morgan fold; (iv) historical and (v) actual flexural involvement. The refined Millennium Criteria were constituted from these criteria. When comparing the different list for validity in diagnosing atopic dermatitis, the refined Millennium Criteria showed a sensitivity of 81.8% and a specificity of 98.8% compared to a sensitivity of 97.7% and specificity of 72.9% of the UK Criteria and a sensitivity of 100% and specificity of 48.8% of the Hanifin & Rajka Criteria. This refinement and validity study shows that the refined Millennium Criteria are a valid tool to diagnose atopic and atopiform dermatitis in a hospital-based setting and therefore could be incorporated in clinical practice and trials.

Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects

OBJECTIVE

To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.

STUDY DESIGN AND SETTING

Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P<0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.

RESULTS

Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005-2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66-1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54-0.96). Mean credibility of the 461 meta-analyses in 2005 was 63-84% depending on the assumptions made. Credibility estimates changed >20% in 19-31 (24-39%) of the 80 updated meta-analyses.

CONCLUSIONS

Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.

Systematic review and meta-analysis of the discriminatory performance of risk prediction rules in febrile neutropaenic episodes in children and young people

Introduction

Febrile neutropaenia is a frequently occurring and occasionally life-threatening complication of treatment for childhood cancer, yet many children are aggressively over-treated. We aimed to undertake a systematic review and meta-analysis to summarise evidence on the discriminatory ability and predictive accuracy of clinical decision rules (CDR) of risk stratification in febrile neutropaenic episodes.

Methods

The review was conducted in accordance with Centre for Reviews and Dissemination methods, using random effects models to undertake meta-analysis. It was registered with the HTA Registry of systematic reviews, CRD32009100453.

Results

We found 20 studies describing 16 different CDR assessed in 8388 episodes of FNP. No study compared different approaches and only one CDR had been subject to testing across multiple datasets. This review cannot conclude that any system is more effective or reliable than any other.

Conclusion

To maximise the value of the information already collected by these and other cohorts of children with febrile neutropaenia, an individual-patient-data (IPD) meta-analysis is required to develop and test new and existing CDR to improve stratification and optimise therapy.

An empirical approach to model selection through validation for censored survival data

Medical prognostic models can be designed to predict the future course or outcome of disease progression after diagnosis or treatment. The existing variable selection methods may be precluded by full model advocates when we build a prediction model owing to their estimation bias and selection bias in right-censored time-to-event data. If our objective is to optimize predictive performance by some criterion, we can often achieve a reduced model that has a little bias with low variance, but whose overall performance is enhanced. To accomplish this goal, we propose a new variable selection approach that combines Stepwise Tuning in the Maximum Concordance Index (STMC) with Forward Nested Subset Selection (FNSS) in two stages. In the first stage, the proposed variable selection is employed to identify the best subset of risk factors optimized with the concordance index using inner cross-validation for optimism correction in the outer loop of cross-validation, yielding potentially different final models for each of the folds. We then feed the intermediate results of the prior stage into another selection method in the second stage to resolve the overfitting problem and to select a final model from the variation of predictors in the selected models. Two case studies on relatively different sized survival data sets as well as a simulation study demonstrate that the proposed approach is able to select an improved and reduced average model under a sufficient sample and event size compared with other selection methods such as stepwise selection using the likelihood ratio test, Akaike Information Criterion (AIC), and lasso. Finally, we achieve better final models in each dataset than their full models by most measures. These results of the model selection models and the final models are assessed in a systematic scheme through validation for the independent performance.

The risk of inflammatory bowel disease-related colorectal carcinoma is limited: results from a nationwide nested case-control study

OBJECTIVES

The risk for inflammatory bowel disease (IBD)-related colorectal cancer (CRC) remains a matter of debate. Initial reports mainly originate from tertiary referral centers, and conflict with more recent studies. Overall, epidemiology of IBD-related CRC is relevant to strengthen the basis of surveillance guidelines. We performed a nationwide nested case-control study to assess the risk for IBD-related CRC and associated prognostic factors in general hospitals.

METHODS

IBD patients diagnosed with CRC between January 1990 and July 2006 in 78 Dutch general hospitals were identified as cases, using a nationwide automated pathology database. Control IBD patients without CRC were randomly selected. Clinical data were collected from detailed chart review. Poisson regression analysis was used for univariable and multivariable analyses.

RESULTS

A total of 173 cases were identified through pathology and chart review and compared with 393 controls. The incidence rate of IBD-related CRC was 0.04%. Risk factors for IBD-related CRC were older age, concomitant primary sclerosing cholangitis (PSC, relative ratio (RR) per year duration 1.05; 95% confidence interval (CI) 1.01-1.10), pseudopolyps (RR 1.92; 95% CI 1.28-2.88), and duration of IBD (RR per year 1.04; 95% CI 1.02-1.05). Using immunosuppressive therapy (odds ratio (OR) 0.3; 95% CI 0.16-0.56, P<0.001) or anti-tumor necrosis factor (TNF) (OR 0.09; 95% CI 0.01-0.68, P<0.02) was protective.

CONCLUSIONS

We found a limited risk for developing IBD-related CRC in The Netherlands. Age, duration of PSC and IBD, concomitant pseudopolyps, and use immunosuppressives or anti-TNF were strong prognostic factors in general hospitals.

Performance of using multiple stepwise algorithms for variable selection

Some research studies in the medical literature use multiple stepwise variable selection (SVS) algorithms to build multivariable models. The purpose of this study is to determine whether the use of multiple SVS algorithms in tandem (stepwise agreement) is a valid variable selection procedure. Computer simulations were developed to address stepwise agreement. Three popular SVS algorithms were tested (backward elimination, forward selection, and stepwise) on three statistical methods (linear, logistic, and Cox proportional hazards regression). Other simulation parameters explored were the sample size, number of predictors considered, degree of correlation between pairs of predictors, p-value-based entrance and exit criteria, predictor type (normally distributed or binary), and differences between stepwise agreement between any two or all three algorithms. Among stepwise methods, the rate of agreement, agreement on a model including only those predictors truly associated with the outcome, and agreement on a model containing the predictors truly associated with the outcome were measured. These rates were dependent on all simulation parameters. Mostly, the SVS algorithms agreed on a final model, but rarely on a model with only the true predictors. Sample size and candidate predictor pool size are the most influential simulation conditions. To conclude, stepwise agreement is often a poor strategy that gives misleading results and researchers should avoid using multiple SVS algorithms to build multivariable models. More research on the relationship between sample size and variable selection is needed.