The effect of disease-prevalence adjustments on the accuracy of a logistic prediction model

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Extracranial-Intracranial Bypass for Moyamoya Disease: The Influence of Racial and Socioeconomic Disparities on Outcomes - A National Inpatient Sample Analysis

BACKGROUND

Extracranial-intracranial (EC-IC) bypass is an established therapeutic option for Moyamoya disease (MMD). However, little is known about the effects of racial and ethnic disparities on outcomes. This study assessed trends in EC-IC bypass outcomes among MMD patients stratified by race and ethnicity.

METHODS

Utilizing the US National Inpatient Sample, we identified MMD patients undergoing EC-IC bypass between 2002 and 2020. Demographic and hospital-level data were collected. Multivariable analysis was conducted to identify independent factors associated with outcomes. Trend analysis was performed using piecewise joinpoint regression.

RESULTS

Out of 14,062 patients with MMD, 1771 underwent EC-IC bypass. Of these, 60.59% were White, 17.56% were Black, 12.36% were Asians, 8.47% were Hispanic, and 1.02% were Native Americans. Nonhome discharge was noted in 21.7% of cases, with a 6.7% death and 3.8% postoperative neurologic complications rates. EC-IC bypass was more commonly performed in Native Americans (23.38%) and Asians (17.76%). Hispanics had the longest mean length of stay (8.4 days) and lower odds of nonhome discharge compared to Whites (odds ratio: 0.64; 95% confidence interval: 0.40-1.03; P = 0.04). Patients with Medicaid, private insurance, self-payers, and insurance paid by other governments had lower odds of nonhome discharge than those with Medicare.

CONCLUSION

This study highlights racial and socioeconomic disparities in EC-IC bypass for patients with MMD. Despite these disparities, we did not find any significant difference in the quality of care. Addressing these disparities is essential for optimizing MMD outcomes.

Estimation of fault probability in medium voltage feeders through calibration techniques in classification models

Machine Learning is currently a well-suited approach widely adopted for solving data-driven problems in predictive maintenance. Data-driven approaches can be used as the main building block in risk-based assessment and analysis tools for Transmission and Distribution System Operators in modern Smart Grids. For this purpose, a suitable Decision Support System should be able of providing not only early warnings, such as the detection of faults in real time, but even an accurate probability estimate of outages and failures. In other words, the performance of classification systems, at least in these cases, needs to be assessed even in terms of reliable outputting posterior probabilities, a really important feature that, in general, classifiers very often do not offer. In this paper are compared several state-of-the-art calibration techniques along with a set of simple new proposed techniques, with the aim of calibrating fuzzy scoring values of a custom-made evolutionary-cluster-based hybrid classifier trained on a set of a real-world dataset of faults collected within the power grid that feeds the city of Rome, Italy. Comparison results show that in real-world cases calibration techniques need to be assessed carefully depending on the scores distribution and the proposed techniques are a valid alternative to the ones existing in the technical literature in terms of calibration performance, computational efficiency and flexibility.

External Validation of the Modified 4C Deterioration Model and 4C Mortality Score for COVID-19 Patients in a Swiss Tertiary Hospital

Prognostic models to predict the deterioration and mortality risk in COVID-19 patients are utterly needed to assist in informed decision making. Most of these models, however, are at high risk of bias, model overfitting, and unclear reporting. Here, we aimed to externally validate the modified (urea was omitted) 4C Deterioration Model and 4C Mortality Score in a cohort of Swiss COVID-19 patients and, second, to evaluate whether the inclusion of the neutrophil-to-lymphocyte ratio (NLR) improves the predictive performance of the models. We conducted a retrospective single-centre study with adult patients hospitalized with COVID-19. Both prediction models were updated by including the NLR. Model performance was assessed via the models' discriminatory performance (area under the curve, AUC), calibration (intercept and slope), and their performance overall (Brier score). For the validation of the 4C Deterioration Model and Mortality Score, 546 and 527 patients were included, respectively. In total, 133 (24.4%) patients met the definition of in-hospital deterioration. Discrimination of the 4C Deterioration Model was AUC = 0.78 (95% CI 0.73-0.82). A total of 55 (10.44%) patients died in hospital. Discrimination of the 4C Mortality Score was AUC = 0.85 (95% CI 0.79-0.89). There was no evidence for an incremental value of the NLR. Our data confirm the role of the modified 4C Deterioration Model and Mortality Score as reliable prediction tools for the risk of deterioration and mortality. There was no evidence that the inclusion of NLR improved model performance.

Marginal versus conditional odds ratios when updating risk prediction models

Risk prediction models often need to be updated when applied to new settings. A simple updating method involves fixed odds ratio transformation of predicted risks to adjust the model for outcome prevalence in the new setting. When a sample from the target population is available, the gold standard is to use a logistic regression model to estimate this odds ratio. A simpler method has been proposed that calculates this odds ratio from the prevalence estimates in the original and new samples. We show that the marginal odds ratio estimated in this way is generally closer to one than the correct (conditional) odds ratio; thus, the simpler method should be avoided when individual-level data are available. When prevalence estimates are the only information at hand, we suggest an approximate method for recovering the conditional odds ratio from the variance of predicted risks in the development sample. Brief simulations and examples show that this approach reduces undercorrection, often substantially.

Assessing a Patient's Individual Risk of Biopsy-detectable Prostate Cancer: Be Aware of Case Mix Heterogeneity and A Priori Likelihood

The relation between prostate-specific antigen (PSA) and other relevant prebiopsy information is often combined in a risk calculator (RC). If the setting for RC use differs from that in which it was developed, there is a risk of making clinical decisions based on incorrect estimates of the absolute risk. The ERSPC-MRI RC predicts clinically significant prostate cancer (csPC; Gleason ≥3 + 4) on targeted and systematic biopsy using information on PSA, digital rectal examination, prostate volume, age, previous negative biopsy, and Prostate Imaging-Recording and Data System score. This calculator was developed on a clinical cohort of 961 men (2012-2017) with a csPC prevalence of 36%. Discrimination was good (area under the receiver operating characteristic curve 0.84). With the increasing use of multiparametric magnetic resonance imaging, we foresee that this RC will also be used for men with a lower a priori likelihood of PC. We investigated the effect of such a scenario on individual risk predictions. A small update of the intercept for the calculator can restore the accuracy to support decision-making with locally valid risk estimates. PATIENT SUMMARY: Decisions on who to refer for a prostate biopsy with its risk of sepsis and overdiagnosis require more than a prostate-specific antigen test. A prediction tool may take other relevant prebiopsy information into account, but may need to be updated to contemporary center-specific settings to provide accurate estimates of the risk of having prostate cancer.

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

BACKGROUND

Prior to implementing predictive models in novel settings, analyses of calibration and clinical usefulness remain as important as discrimination, but they are not frequently discussed. Calibration is a model's reflection of actual outcome prevalence in its predictions. Clinical usefulness refers to the utilities, costs, and harms of using a predictive model in practice. A decision analytic approach to calibrating and selecting an optimal intervention threshold may help maximize the impact of readmission risk and other preventive interventions.

OBJECTIVES

To select a pragmatic means of calibrating predictive models that requires a minimum amount of validation data and that performs well in practice. To evaluate the impact of miscalibration on utility and cost via clinical usefulness analyses.

MATERIALS AND METHODS

Observational, retrospective cohort study with electronic health record data from 120,000 inpatient admissions at an urban, academic center in Manhattan. The primary outcome was thirty-day readmission for three causes: all-cause, congestive heart failure, and chronic coronary atherosclerotic disease. Predictive modeling was performed via L1-regularized logistic regression. Calibration methods were compared including Platt Scaling, Logistic Calibration, and Prevalence Adjustment. Performance of predictive modeling and calibration was assessed via discrimination (c-statistic), calibration (Spiegelhalter Z-statistic, Root Mean Square Error [RMSE] of binned predictions, Sanders and Murphy Resolutions of the Brier Score, Calibration Slope and Intercept), and clinical usefulness (utility terms represented as costs). The amount of validation data necessary to apply each calibration algorithm was also assessed.

RESULTS

C-statistics by diagnosis ranged from 0.7 for all-cause readmission to 0.86 (0.78-0.93) for congestive heart failure. Logistic Calibration and Platt Scaling performed best and this difference required analyzing multiple metrics of calibration simultaneously, in particular Calibration Slopes and Intercepts. Clinical usefulness analyses provided optimal risk thresholds, which varied by reason for readmission, outcome prevalence, and calibration algorithm. Utility analyses also suggested maximum tolerable intervention costs, e.g., $1720 for all-cause readmissions based on a published cost of readmission of $11,862.

CONCLUSIONS

Choice of calibration method depends on availability of validation data and on performance. Improperly calibrated models may contribute to higher costs of intervention as measured via clinical usefulness. Decision-makers must understand underlying utilities or costs inherent in the use-case at hand to assess usefulness and will obtain the optimal risk threshold to trigger intervention with intervention cost limits as a result.

A risk score for the prediction of advanced age-related macular degeneration: development and validation in 2 prospective cohorts

PURPOSE

To develop a clinical eye-specific prediction model for advanced age-related macular degeneration (AMD).

DESIGN

The Age-Related Eye Disease Study (AREDS) cohort followed up for 8 years served as the training dataset, and the Blue Mountains Eye Study (BMES) cohort followed up for 10 years served as the validation dataset.

PARTICIPANTS

A total of 4507 AREDS participants (contributing 1185 affected vs. 6992 unaffected eyes) and 2169 BMES participants (contributing 69 affected vs. 3694 unaffected eyes).

METHODS

Using Bayes' theorem in a logistic model, we used 8 baseline predictors-age, sex, education level, race, smoking status, and presence of pigment abnormality, soft drusen, and maximum drusen size-to devise and validate a macular risk scoring system (MRSS). We assessed the performance of the MRSS by calculating sensitivity, specificity, and the area under the receiver operating characteristic curve (i.e., c-index).

MAIN OUTCOME MEASURES

Advanced AMD.

RESULTS

The internally validated c-indexAREDS (0.88; 95% confidence interval, 0.87-0.89) and the externally validated c-indexBMES (0.91; 95% confidence interval, 0.88-0.95) suggested excellent performance of the MRSS. The sensitivity and specificity at the optimal macular risk score cutoff point of 0 were 87.6% and 73.6%, respectively. An application for the iPhone and iPad also was developed as a practical tool for the MRSS.

CONCLUSIONS

The MRSS was developed and validated to provide satisfactory accuracy and generalizability. It may be used to screen patients at risk of developing advanced AMD.

Predicting the Coexistence of an Endometrial Adenocarcinoma in the Presence of Atypical Complex Hyperplasia: Immunohistochemical Analysis of Endometrial Samples

Objective

This study aimed to determine whether immunohistochemical markers in complex atypical endometrial hyperplasia could predict the presence of a concurrent endometrial carcinoma.

Methods

Endometrial biopsies of 39 patients with complex atypical hyperplasia were selected retrospectively between 1999 and 2006. Only patients who underwent a hysterectomy were included. A coexisting endometrial carcinoma was present in 25 patients (64%). Immunohistochemical analysis was performed on formalin-fixed paraffin-embedded sections of the endometrial biopsies, using antibodies for MIB-1, β-catenin, E-cadherin, p53, PTEN, CD44, HER2-neu, survivin, COX-2, tenascin, and bcl-2. To evaluate the potential utility of these markers, a prediction model was constructed.

Results

In the univariate analysis, expressions of both PTEN and HER2-neu were significantly different between the groups with and without a coexisting endometrial carcinoma (P < 0.05). Loss of PTEN staining was found in 13 (54%) and 1 (7%) of the patients with and without a coexistent carcinoma, respectively (odds ratio, 16.55; 95% confidence interval [CI], 1.87–146.65). HER2-neu expression was found in only 2 (8.6%) and 6 (43%) patients with and without a coexistent carcinoma, respectively, and was excluded from further analysis because of its low expression. A prediction model containing PTEN expression only showed an area under the curve of 73.4% (95% CI, 57.3%–89.6%). After adding MIB-1 and p53, discriminative power improved to 87.2% (95% CI, 75.1%–99.3%).

Conclusions

This study showed that PTEN expression in complex endometrial hyperplasia is a promising factor for the prediction of the presence of a coexisting endometrial carcinoma, and prediction may even better when MIB-1 and p53 expressions are considered simultaneously.

Diagnostic accuracy of free and total metanephrines in plasma and fractionated metanephrines in urine of patients with pheochromocytoma

BACKGROUND

Plasma free and urinary metanephrines are recognized biomarkers for the assessment of pheochromocytoma. Plasma total metanephrines with a long half-life may represent another useful biomarker.

OBJECTIVE

The aim of this study is to evaluate the diagnostic performances of plasma total metanephrines alone or combined with free metanephrines and fractionated 24-h urinary metanephrines.

METHODS

A retrospective, case-control diagnostic test study was conducted between 1999 and 2007 in two university hospitals in Switzerland and two institutions in France. The patients included 46 cases with histologically proven pheochromocytoma, and 181 controls suspected of tumor with negative investigations and 3-year follow-up. None had renal dysfunction. Sensitivity and specificity were compared after expressing each measurement result as a ratio over its upper reference limit, adding the ratios of normetanephrine and metanephrine, and defining cut-off values of 1 or 2 for this sum.

RESULTS

Applying a cut-off value of 1, plasma free and total metanephrines and urinary fractionated metanephrines had similar sensitivities of 96% (95% confidence interval, 86-99%), 95% (85-99%), and 95% (84-99%) along with similar specificities of 89% (83-94%), 91% (84-95%), and 86% (80-91%). A cut-off of 2 for the sum of ratios over reference limit improves the specificity, and it can be used for a confirmation test based on another biomarker taken among the three biomarkers.

CONCLUSION

All three metanephrine-based tests perform equivalently for diagnosing pheochromocytoma in the absence of renal insufficiency, and can be conveniently associated two by two for confirming/excluding tumor.

Predicting postresection hydrocephalus in pediatric patients with posterior fossa tumors

OBJECT

Approximately 30% of children with posterior fossa tumors exhibit hydrocephalus after tumor resection. Recent literature has suggested that prophylactic endoscopic third ventriculostomy diminishes the risk of this event. Because the majority of patients will not have postoperative hydrocephalus, a preoperative clinical prediction rule that identifies patients at high or low risk for postresection hydrocephalus would be helpful to optimize the care of these children.

METHODS

The authors evaluated a derivation cohort of 343 consecutive children with posterior fossa tumors who underwent treatment between 1989 and 2003. Multivariate methods were used on these data to generate the Canadian Preoperative Prediction Rule for Hydrocephalus. The rule's estimated risk of postresection hydrocephalus was compared with risk observed in 111 independent patients in the validation cohort.

RESULTS

Variables identified as significant in predicting postresection hydrocephalus were age < 2 years (score of 3), papilledema (score of 1), moderate to severe hydrocephalus (score of 2), cerebral metastases (score of 3), and specific estimated tumor pathologies (score of 1). Patients with scores > or = 5 were deemed as high risk. Predicted probabilities for the high- and low-risk groups were 0.73 and 0.25, respectively, from the derivation cohort, and 0.59 and 0.14 after prevalence adjustment compared with the observed values of 0.42 and 0.17 in the validation cohort.

CONCLUSIONS

A patient's score on the Preoperative Prediction Rule for Hydrocephalus will allow improved patient counseling and surgical planning by identifying patients at high risk of developing postresection hydrocephalus. These patients might selectively be exposed to the risks of preresection CSF diversion to improve outcome.

Relationship of predictive modeling to receiver operating characteristics

BACKGROUND

Discriminatory capabilities of a measurement technique can be assessed by a receiver operating characteristic (ROC) curve analysis (specifically, area under the curve [AUC]) and predictive modeling (predictive accuracy and positive predictive value). Theoretically, predictive accuracy is dependent on disease prevalence while AUC assessments are not.

OBJECTIVE

To compare the effect of changes in disease prevalence on ROC AUC analysis and predictive modeling.

METHODS

For this comparison, a data set with 72 individuals with coronary artery disease (CAD) and 1,857 individuals without CAD was used. A validated CAD score with a demonstrated AUC of 0.80 was applied. Disease prevalence within the study sample was altered by randomly removing non-CAD patients from the original sample. Predictive accuracy and positive predictive value of the CAD score were calculated using 2 x 2 contingency tables. Three threshold values of the CAD score were applied centering on a value for which sensitivity and specificity were equal.

RESULTS

For a chosen CAD score threshold value (eg, 60), sensitivity (0.74), specificity (0.75), and AUC (0.81) did not change significantly while positive predictive value increased (10%-70%) as disease prevalence increased from 4% to 44%. Changes in predictive accuracy were dependent on the selected test threshold value. Predictive accuracy increased (54%-68%), did not change (74%-75%), or decreased (88%-70%) with the same increase in disease prevalence for threshold values of 50, 60, and 70, respectively.

CONCLUSIONS

The ROC AUC and predictive accuracy are stable diagnostic characteristics, whereas positive predictive value is greatly influenced by disease prevalence.

Simple optimal weighting of cases and controls in case-control studies

Researchers of uncommon diseases are often interested in assessing potential risk factors. Given the low incidence of disease, these studies are frequently case-control in design. Such a design allows a sufficient number of cases to be obtained without extensive sampling and can increase efficiency; however, these case-control samples are then biased since the proportion of cases in the sample is not the same as the population of interest. Methods for analyzing case-control studies have focused on utilizing logistic regression models that provide conditional and not causal estimates of the odds ratio. This article will demonstrate the use of the prevalence probability and case-control weighted targeted maximum likelihood estimation (MLE), as described by van der Laan (2008), in order to obtain causal estimates of the parameters of interest (risk difference, relative risk, and odds ratio). It is meant to be used as a guide for researchers, with step-by-step directions to implement this methodology. We will also present simulation studies that show the improved efficiency of the case-control weighted targeted MLE compared to other techniques.

What percent luminal stenosis should be used to define angiographic coronary artery disease for noninvasive test evaluation?

BACKGROUND

There has been controversy over what is the best angiographic luminal dimension criterion associated with ischemia for evaluating diagnostic tests. If one assumes that ST-segment depression or scores are indicators of ischemia, then whatever angiographic criteria best discriminates those with ischemic and nonischemic responses would be the best angiographic marker for ischemia. To study this, we calculated the area under the ROC curves for ST depression and scores at different angiographic cut-points in order to determine the best angiographic cut-point for defining ischemia-producing coronary disease.

METHODS

Twelve hundred and seventy-six consecutive males without prior MI with a mean age of 59 +/- 11 years who had undergone exercise testing and coronary angiography were analyzed in this study. We calculated the number of patients of this population that would be considered to have coronary artery disease at different cut-points for angiographic luminal stenosis. For example, 59% of the patients had significant CAD when disease was defined as 50% or greater coronary lumen stenosis of any coronary vessel while 49% of the patients had significant CAD when disease was defined as 70% or greater coronary lumen stenosis. Cut-points were considered between 40 to 100% coronary lumen stenosis. ROC analysis was then performed comparing ST depression and treadmill scores at each of these cut-points.

RESULTS

The cut-point for coronary lumen stenosis that returned the highest AUC for ST depression and scores was between 70 and 80% coronary luminal stenosis. However, the difference between the 50% and 75% luminal stenosis criteria was minimal.

CONCLUSION

It appears that the best cut-point for defining significant angiographic disease when evaluating diagnostic tests of ischemia is 75% or greater coronary luminal stenosis.

Nomographic representation of logistic regression models: a case study using patient self-assessment data

Logistic regression models are widely used in medicine, but difficult to apply without the aid of electronic devices. In this paper, we present a novel approach to represent logistic regression models as nomograms that can be evaluated by simple line drawings. As a case study, we show how data obtained from a questionnaire-based patient self-assessment study on the risks of developing melanoma can be used to first identify a subset of significant covariates, build a logistic regression model, and finally transform the model to a graphical format. The advantage of the nomogram is that it can easily be mass-produced, distributed and evaluated, while providing the same information as the logistic regression model it represents.

The incidence and prediction of automatically detected intraoperative cardiovascular events in noncardiac surgery

The objective of this study was to evaluate prognostic models for quality assurance purposes in predicting automatically detected intraoperative cardiovascular events (CVE) in 58458 patients undergoing noncardiac surgery. To this end, we assessed the performance of two established models for risk assessment in anesthesia, the Revised Cardiac Risk Index (RCRI) and the ASA physical status classification. We then developed two new models. CVEs were detected from the database of an electronic anesthesia record-keeping system. Logistic regression was used to build a complex and a simple predictive model. Performance of the prognostic models was assessed using analysis of discrimination and calibration. In 5249 patients (17.8%) of the evaluation (n = 29437) and 5031 patients (17.3%) of the validation cohorts (n = 29021), a minimum of one CVE was detected. CVEs were associated with significantly more frequent hospital mortality (2.1% versus 1.0%; P < 0.01). The new models demonstrated good discriminative power, with an area under the receiver operating characteristic curve (AUC) of 0.709 and 0.707 respectively. Discrimination of the ASA classification (AUC 0.647) and the RCRI (AUC 0.620) were less. Neither the two new models nor ASA classification nor the RCRI showed acceptable calibration. ASA classification and the RCRI alone both proved unsuitable for the prediction of intraoperative CVEs.

IMPLICATIONS

The objective of this study was to evaluate prognostic models for quality assurance purposes to predict the occurrence of automatically detected intraoperative cardiovascular events in 58,458 patients undergoing noncardiac surgery. Two newly developed models showed good discrimination but, because of reduced calibration, their clinical use is limited. The ASA physical status classification and the Revised Cardiac Risk Index are unsuitable for the prediction of intraoperative cardiovascular events.

Continuous probabilistic prediction of angiographically significant coronary artery disease using electron beam tomography

BACKGROUND

We sought to incorporate electron beam tomography-derived calcium scores in a model for prediction of angiographically significant coronary artery disease (CAD). Such a model could greatly facilitate clinical triage in symptomatic patients with no known CAD.

METHODS AND RESULTS

We examined 1851 patients with suspected CAD who underwent coronary angiography for clinical indications. An electron beam tomographic scan was performed in all patients. Total per-patient calcium scores and separate scores for the major coronary arteries were added to logistic regression models to calculate a posterior probability of the severity and extent of angiographic disease. These models were designed to be continuous, adjusted for age and sex, corrected for verification bias, and independently validated in terms of their incremental diagnostic accuracy. The overall sensitivity was 95%, and specificity was 66% for coronary calcium to predict obstructive disease on angiography. With calcium scores >20, >80, and >100, the sensitivity to predict stenosis decreased to 90%, 79%, and 76%, whereas the specificity increased to 58%, 72%, and 75%, respectively. The logistic regression model exhibited excellent discrimination (receiver operating characteristic curve area, 0.842+/-0.023) and calibration (chi2 goodness of fit, 8.95; P=0.442).

CONCLUSIONS

Electron beam tomographic calcium scanning provides incremental and independent power in predicting the severity and extent of angiographically significant CAD in symptomatic patients, in conjunction with pretest probability of disease. This algorithm is most useful when applied to an intermediate-risk population.

Free/total PSA ratio improves differentiation of benign and malignant disease of the prostate: critical analysis of two different test populations

OBJECTIVES

To evaluate the ability of free PSA (fPSA), total PSA (tPSA), and the free/total PSA (f/t PSA) ratio to differentiate between benign prostate disease (benign prostatic hyperplasia [BPH] and no evidence of malignancy [NEM]) and prostate cancer (CaP) using two different testing populations, and to compare predictive probabilities for the two test populations.

METHODS

One test population consisted of sera from 531 men with clinically well-defined and biopsy-confirmed BPH (n = 255) or CaP (n = 276), with tPSA values ranging from 2 to 20 ng/mL. All of these serum samples were retrospective and obtained from patients evaluated in academic settings before any treatment. A second test population consisted of a prospective analysis of sera obtained from 4870 men, collected by urologists throughout the United States and processed at a single pathology laboratory. All these patients had a systematic biopsy evaluated and diagnosed at the same pathology laboratory, with the diagnosis categorized as either NEM (n = 2961) or CaP (n = 1909). No additional information on concurrent disease or pre- or current treatment status was known for this test population. For both populations, two tPSA reflex range groups, 2 to 10 and 2 to 20 ng/mL, were evaluated.

RESULTS

Both test populations benefited from the application of either fPSA alone or the f/t PSA ratio to differentiate benign from malignant disease (t test P value less than 0.001). The receiver operating characteristic (ROC) curve for the f/t PSA ratio had an area under the curve (AUC) of 72% for n = 531 versus 63% for n = 4870, irrespective of the tPSA reflex range. Average fPSA values demonstrated a linear correlation to a range of tPSA concentrations for both test populations. Predictive probabilities (adjusted for established cancer prevalence rates in the academic population [n = 531]) calculated using f/t PSA ratios also demonstrated their value in contrasting the performance characteristics in the two test populations.

CONCLUSIONS

The fPSA and f/t PSA ratio improved the differentiation of benign disease and CaP in two different patient samples. The f/t PSA ratio demonstrated an increased sensitivity and specificity when applied to differentiate clinically well-defined BPH and CaP (n = 531). The differences in the results between the two test samples are probably attributable to the variability of the patient's disease and treatment status in the larger, less refined, community-based population. The use of predictive probabilities provides the opportunity to provide patient-specific cancer probabilities instead of using population-based specific single cutoffs.

Validation of a predictive model for postoperative pulmonary complications

OBJECTIVE

To test or validate a previously reported model for predicting postoperative pulmonary complications (PPCs) after elective abdominal surgical procedures.

DESIGN

Prospective, descriptive.

SETTING

Four midwestern hospitals serving a diverse population of patients.

PATIENTS

Two hundred seventy-six adult patients who had undergone abdominal surgery (51% men, 49% women; mean age 54.1 +/- 5.3 years).

OUTCOME MEASURES

PPC developed in 26.4%.

DATA COLLECTION

Data were collected preoperatively during a brief interview and a pulmonary physical examination and on the first 6 postoperative days.

RESULTS

A six risk-factor model was tested in this sample of subjects. The model validated relatively well in the sample of 276 subjects with use of the basic criteria of correct classification, sensitivity, and specificity. However, when a new model was developed from this sample, differing risk factors emerged as significant independent predictors.

CONCLUSIONS

Further research is needed to assess the stability of the risk factors and test the models in differing settings and populations of patients.

Effect of an explicit decision-support tool on decisions to prescribe antibiotics for sore throat

Studies of scoring rules for sore throat have failed to show that they lower antibiotic prescription rates. The authors studied the effect of an explicit decision-support tool, incorporating a modified score, on antibiotic-prescription decisions. Four hundred and fifty family physicians received an information package, a score card, and a recording form to use during one sore-throat encounter. The physicians randomly received either a control form or an intervention form that required them to interact with the score during the clinical recording process. There was a trend towards a reduction in antibiotic prescriptions (21%, p=0.09) in the physicians using the intervention form. A greater reduction (45%, p=0.06) was observed for patients whose probabilities of infection with group A streptococcus were low. Sore-throat-scoring rules may reduce unnecessary antibiotic prescriptions if physicians are specifically cued to use them during clinical encounters and appropriate management responses are linked to score estimates for the likelihood of group A streptococcus infection.

An agreement approach to predict severe angiographic coronary artery disease with clinical and exercise test data

OBJECTIVE

To demonstrate that an agreement approach to applying equations on the basis of clinical and exercise test variables is an accurate, self-calibrating, and cost-efficient method for predicting severe coronary artery disease in clinical populations.

DESIGN

Retrospective analysis of consecutive patients with complete data from exercise testing and coronary angiography referred for evaluation of possible coronary artery disease. After developing an equation in a training set, this equation and two other equations developed by other investigators were validated in a test set. The study was performed at two university-affiliated Veteran's Affairs medical centers.

PATIENTS

1080 consecutive men studied between 1985 and 1995 who had coronary angiography within 3 months of the treadmill test. The population was randomly divided into a training set of 701 patients and a test set of 379 patients. Patients with previous coronary artery bypass surgery, valvular heart disease, marked degrees of resting ST depression, and left bundle branch block were excluded.

MEASUREMENTS

Recording of clinical and exercise test data along with visual interpretation of the electrocardiogram recordings on standardized forms and abstraction of visually interpreted angiographic data from clinical catheterization reports.

RESULTS

Simple clinical and exercise test variables improved the standard application of exercise-induced ST criteria for predicting severe coronary artery disease. By setting probability thresholds for severe disease of <20% and >40% for the three prediction equations, the agreement approach divided the test set into three groups: low risk (patients with all three equations predicting <21% probability of severe coronary disease), no agreement, and high risk (all three equations with >39% probability) for severe coronary artery disease. Because the patients in the no agreement group would be sent for further testing and would eventually be correctly classified, the sensitivity of the agreement approach was 89% and the specificity was 96%. The agreement approach appeared to be unaffected by disease prevalence, missing data, variable definitions, or even angiographic criteria.

CONCLUSIONS

Requiring diagnosis of severe coronary disease to be dependent on agreement between these three equations has made them likely to function in all clinical populations. The agreement approach should be an efficient method for the evaluation of populations with varying prevalence of coronary artery disease, limiting the use of more expensive noninvasive and invasive testing to patients with a higher probability of left main or triple-vessel coronary artery disease. This approach provides a strategy that can be applied by inputting the results of basic clinical assessment into a programmable calculator or a computer to assist the practitioner in deciding when further evaluation is appropriate, thus assuring patients access to subspecialty care.

Differential performance of TRISS-like in early and late blunt trauma deaths

OBJECTIVES

(1) To independently validate the Trauma and Injury Severity Score-Like (TRISS-Like) model derived by Offner et al. (Revision of TRISS for intubated patients. J Trauma. 1992;32:32-35) in a population of Canadian blunt trauma victims, and (2) to compare the ability of this model to predict mortality in early and late trauma deaths.

STUDY POPULATION

Prospective cohort of blunt trauma cases with Injury Severity Score > 12 identified from the Ontario Trauma Registry over a 5-year period.

STUDY DESIGN

The TRISS-Like model consisting of age, Injury Severity Score, systolic blood pressure, and best motor response of the Glasgow Coma Scale was evaluated as to its ability to predict mortality by determining the sensitivity, specificity, and the area under the receiver operating characteristic curve. The sample was then divided into early (< or = 7 days) and late mortality subgroups in which model performance was evaluated with respect to time of death.

RESULTS

A total of 7,703 patients were included in this analysis. The overall mortality was 12.3%. The TRISS-Like model allowed for assessment of an additional 23% of patients than would standard TRISS and performed with a sensitivity of 97.1%, specificity of 39.8% and an area under the receiver operating characteristic curve of 0.873. Analysis of mortality with respect to time demonstrated that 75% of deaths occurred by day 7. The specificity and receiver operating characteristic area increased in the early (< or = 7 days) subgroup, 46.5% and 0.935, respectively, compared with 20.8% and 0.778 in the late mortality group.

CONCLUSIONS

TRISS-Like demonstrated similar performance to that reported with the standard TRISS model but with the additional advantage that it is more generalizable because it can be applied to intubated patients. TRISS-Like demonstrated substantially superior performance in early trauma deaths compared with those that occurred late. This differential performance may be because the model does not include risk factors for late mortality.

Development and validation of a clinical score to estimate the probability of coronary artery disease in men and women presenting with suspected coronary disease

PURPOSE

Guidelines for the management of patients with suspected coronary disease have emphasized stratification into groups with low, intermediate, and high probability of significant coronary disease. Previously derived clinical prediction rules have been difficult to apply in clinical settings. The purpose of this study was to develop and validate a clinical score that facilitates this stratification process.

PATIENTS AND METHODS

We performed a retrospective analysis of prospectively acquired data from 915 patients with suspected coronary disease and normal resting electrocardiograms who presented for exercise testing at a university hospital. All patients subsequently underwent coronary angiography. Analysis included logistic regression with significant coronary disease (> or = 1 vessel with a > or = 50% lesion) presence as the dependent variable and clinical variables as independent variables. From this analysis, a coronary disease score was developed to estimate prevalence of coronary disease from clinical variables. Validation of this score was performed in a separate prospectively acquired cohort of 348 patients.

RESULTS

For the entire validation group, the prevalence of significant coronary disease was 16% (10/63) in the low probability group, 44% (86/195) in the intermediate probability group, and 69% (62/90) in the high probability group. Both men and women were stratified equally well into the 3 probability groups.

CONCLUSION

The clinical score is an easily memorized and accurate method for categorizing patients with suspected but not proven coronary disease and normal resting electrocardiograms into clinically meaningful probability groups upon which decisions concerning appropriate diagnostic test selection could potentially be based.

Free and complexed prostate-specific antigen serum ratios to predict probability of primary prostate cancer and benign prostatic hyperplasia

OBJECTIVES

Ratios of free to total prostate-specific antigen (f/t PSA ratio) improved differentiation of benign prostatic hyperplasia (BPH) from prostate cancer (CaP). Using sera obtained at least 1 month prior to biopsy-confirmed diagnosis and logistic regression adjusted for disease prevalence, probability curves are constructed to predict the presence of CaP.

METHODS

The patient population included 122 (44%) BPH sera and 155 (56%) prostate carcinoma sera collected prior to any therapy. The total PSA range = 2.0-20.0 ng/mL; median age = 69 years. External reference standards for both free and total PSA assays were used to standardize the assays and correct the ratio. Probability curves and tables for cancer incidence were formulated for a subset of the total test population (total PSA range = 2.0-10.0 ng/mL; 98 BPH, 118 CaP patients) by using logistic regression and prior cancer prevalence statistics derived from a published patient screening study.

RESULTS

Median f/t PSA ratios were 0.18 and 0.12 in the overall sample and 0.19 and 0.12 in the subset for BPH and CaP, respectively (P = 0.0001). The median total PSA concentrations for BPH and CaP were 5.8 and 6.7 ng/mL when total PSA range = 2.0-20.0 ng/mL and were 4.9 and 5.9 ng/mL when total PSA range = 2.0-10.0, respectively.

CONCLUSIONS

Cancer probability curves were constructed to help guide decisions concerning biopsy and other aspects of prostate cancer disease management. Further validation of this approach in another series of patients is necessary and is planned.