The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis

Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets

Logistic regression analysis may well be used to develop a prognostic model for a dichotomous outcome. Especially when limited data are available, it is difficult to determine an appropriate selection of covariables for inclusion in such models. Also, predictions may be improved by applying some sort of shrinkage in the estimation of regression coefficients. In this study we compare the performance of several selection and shrinkage methods in small data sets of patients with acute myocardial infarction, where we aim to predict 30-day mortality. Selection methods included backward stepwise selection with significance levels alpha of 0.01, 0.05, 0. 157 (the AIC criterion) or 0.50, and the use of qualitative external information on the sign of regression coefficients in the model. Estimation methods included standard maximum likelihood, the use of a linear shrinkage factor, penalized maximum likelihood, the Lasso, or quantitative external information on univariable regression coefficients. We found that stepwise selection with a low alpha (for example, 0.05) led to a relatively poor model performance, when evaluated on independent data. Substantially better performance was obtained with full models with a limited number of important predictors, where regression coefficients were reduced with any of the shrinkage methods. Incorporation of external information for selection and estimation improved the stability and quality of the prognostic models. We therefore recommend shrinkage methods in full models including prespecified predictors and incorporation of external information, when prognostic models are constructed in small data sets.

Validation of the pneumonia severity index. Importance of study-specific recalibration

OBJECTIVE

To evaluate the predictive validity and calibration of the pneumonia severity-of-illness index (PSI) in patients with community-acquired pneumonia (CAP).

PATIENTS

Randomly selected patients (n = 1,024) admitted with CAP to 22 community hospitals.

MEASUREMENTS AND MAIN RESULTS

Medical records were abstracted to obtain prognostic information used in the PSI. The discriminatory ability of the PSI to identify patients who died and the calibration of the PSI across deciles of risk were determined. The PSI discriminates well between patients with high risk of death and those with a lower risk. In contrast, calibration of the PSI was poor, and the PSI predicted about 2.4 times more deaths than actually occurred in our population of patients with CAP.

CONCLUSIONS

We found that the PSI had good discriminatory ability. The original PSI overestimated absolute risk of death in our population. We describe a simple approach to recalibration, which corrected the overestimation in our population. Recalibration may be needed when transporting this prediction rule across populations.

Effect of an explicit decision-support tool on decisions to prescribe antibiotics for sore throat

Studies of scoring rules for sore throat have failed to show that they lower antibiotic prescription rates. The authors studied the effect of an explicit decision-support tool, incorporating a modified score, on antibiotic-prescription decisions. Four hundred and fifty family physicians received an information package, a score card, and a recording form to use during one sore-throat encounter. The physicians randomly received either a control form or an intervention form that required them to interact with the score during the clinical recording process. There was a trend towards a reduction in antibiotic prescriptions (21%, p=0.09) in the physicians using the intervention form. A greater reduction (45%, p=0.06) was observed for patients whose probabilities of infection with group A streptococcus were low. Sore-throat-scoring rules may reduce unnecessary antibiotic prescriptions if physicians are specifically cued to use them during clinical encounters and appropriate management responses are linked to score estimates for the likelihood of group A streptococcus infection.

Application of the Center for Epidemiologic Studies Depression Scale among first-visit psychiatric patients: a new approach to improve its performance

BACKGROUND

Although the Center for Epidemiologic Studies Depression Scale (CES-D) is an internationally popular self-rating scale for depression both in community and clinical settings, extant literature concerning its validity has several shortcomings. The present paper aimed to overcome these problems.

METHODS

We applied newer assessment technology of receiver operating characteristics (ROC) analyses and stratum-specific likelihood ratios (SSLRs) and cross-validated the results in the 'training' and 'testing' data sets of 591 patients representing various clinical settings all over Japan.

RESULTS

The ROC analyses demonstrated that the CES-D had moderate convergent and discriminant validity to detect major depressive episodes among first-visit psychiatric patients. Selecting single optimal cutoffs, however, failed to arrive at consistent results across various settings. The efficacy of the instrument was most conveniently transportable into clinical practices when converted into SSLRs, which were 0.35 (95% CI: 0.25-0.49) for the score range 0-29, 2.3 (1.8-3.1) for the score range 30-49, and 11.7 (3.1-44.0) for the scores above 50. In addition, the SSLRs proved to be generalizable not only across various clinical settings in our sample but also across psychiatric, primary care and community samples in the published reports.

CONCLUSION

Clinicians and clinical epidemiologists can apply the SSLRs of the CES-D to various settings to estimate the probability of suffering from a major depressive episode in a convenient and intuitive manner.

Effectiveness versus efficacy: the limited value in clinical practice of high dose dexamethasone suppression testing in the differential diagnosis of adrenocorticotropin-dependent Cushing's syndrome

High dose dexamethasone suppression testing has been widely employed in the differentiation between pituitary ACTH-dependent hypercortisolism [Cushing's disease (CD)] and the ectopic ACTH syndrome. We hypothesized that the high dose dexamethasone suppression test as it is performed in practice does not improve the ability to differentiate between these two types of ACTH-dependent Cushing's syndrome. Cases were drawn from 112 consecutive patients with ACTH-dependent Cushing's syndrome, who were then classified based upon results of inferior petrosal sinus sampling for ACTH levels. Analysis of test characteristics of high dose dexamethasone suppression testing was performed in the 73 patients for whom results are available. Statistical modeling was performed using the 68 cases with complete data on all assessed variables. Logistic regression models were used to predict the probability of pituitary-dependent Cushing's syndrome (CD) given the results of high dose dexamethasone suppression testing before and after adjustment for the contribution of a series of potential covariates. Of the 112 patients with ACTH-dependent Cushing's syndrome, 15.2% had the ectopic ACTH syndrome, and the remainder had pituitary-dependent Cushing's syndrome (CD). Patients with the ectopic ACTH syndrome were significantly older (mean, 51.9 vs. 40.2), were more likely to be male (58.8% vs. 27.4%), had shorter duration of clinical findings (mean, 11.6 vs. 39.9 months), were more likely to have hypokalemia (50% vs. 8.6%), had higher baseline 24-h urinary free cortisol [mean, 8317 vs. 1164 nmol/day (3015 vs. 422 microg)] and plasma ACTH levels [mean, 47 vs. 17 pmol/L (210 vs. 78 pg/mL)] and were less likely to suppress urinary free cortisol or plasma cortisol with high dose dexamethasone using the standard criterion of 50% or more suppression compared with patients with pituitary-dependent Cushing's syndrome. Based upon the standard criterion, the sensitivity and specificity of the high dose dexamethasone suppression test for the diagnosis of pituitary-dependent Cushing's syndrome were 81.0% and 66.7%, respectively. Although the mean percent suppression was significantly greater for patients with CD than for those with the ectopic ACTH syndrome (72.2% vs. 41.3%), the range of suppression was 0-99% for each diagnosis. The area under the receiver operating characteristic curve was 0.710 (95% confidence interval, 0.541-0.879). Logistic regression models were used to evaluate the probability of CD given the responsiveness to high dose dexamethasone suppression testing before and after adjustment for the potential contributions of other factors. A model including all of the variables (age, sex, duration, presence of hypokalemia, urinary free cortisol, and plasma ACTH) had a diagnostic accuracy of 92.7%. A model including all of these variables plus a binary variable indicating whether the patient met the criterion of suppression by 50% or more resulted in 95.6% accuracy, whereas substitution of this binary variable by percent suppression resulted in a model with 94.1% accuracy. There were no statistically significant differences among these models; their values for the c statistic, which is equivalent to the area under the curve in a receiver operating characteristic analysis, were all greater than 0.9. Logistic regression models indicate that the results of the dexamethasone suppression test add little to the differential diagnosis of ACTH-dependent Cushing's syndrome, especially after taking other clinical information into account. In our patient population, the sensitivity and specificity of the dexamethasone suppression test were less than those reported by others. However, because 20-33% of cases of ectopic ACTH syndrome are misdiagnosed with these logistic regression models, other techniques are necessary to achieve greater diagnostic accuracy.

A consensus approach to diagnosing coronary artery disease based on clinical and exercise test data

OBJECTIVE

To demonstrate that a consensus approach for combining prediction equations based on clinical and exercise test variables derived from different populations can stratify patients referred for possible coronary artery disease (CAD) into low-, intermediate-, and high-risk groups.

DESIGN

Retrospective analysis of consecutive patients with complete data from exercise testing and coronary angiography referred for evaluation of possible CAD. After derivation of a logistic equation in our own training set of patients, this equation, along with two other equations developed independently by other investigators, was validated in a test set. The validation strategy for the consensus approach included the following: (1) calculation of probability scores for each patient using each logistic equation independently; (2) determination of probability thresholds in the training set to divide the patients into three groups-low risk (prevalence CAD <5%), intermediate risk (5 to 70%), and high risk (>70% prevalence of CAD); (3) using agreement among at least two of three of the prediction equations to generate "consensus" for each patient; and (4) application of the consensus approach thresholds to the test set of patients.

SETTINGS

Two university-affiliated Veteran's Affairs medical centers.

PATIENTS

We studied 718 consecutive men between 1985 and 1995 who had coronary angiography within 3 months of an exercise treadmill test for suspected CAD. The population was randomly divided into a training set of 429 patients and a test set of 289 patients. Patients with previous myocardial infarction or coronary artery bypass surgery, valvular heart disease, left bundle branch block, or any Q waves present on their resting ECG were excluded from the study.

MEASUREMENTS

Recording of clinical and exercise test data along with visual interpretation of the ECG recordings on standardized forms and abstraction of visually interpreted angiographic data from clinical catheterization reports.

RESULTS

We demonstrated that by using simple clinical and exercise test variables, we could improve on the standard use of ECG criteria during exercise testing for diagnosing CAD. Using the consensus approach divided the test set into populations with low, intermediate, and high risk for CAD. Since the patients in the intermediate group would be sent for further testing and would eventually be correctly classified, the sensitivity of the consensus approach is 94% and the specificity is 92%. The consensus approach controls for varying disease prevalence, missing data, inconsistency in variable definition, and varying angiographic criterion for stenosis severity. The percent of correct diagnoses increased from the 67% for standard exercise ECG analysis and from the 80% for multivariable predictive equations alone to >90% correct diagnoses for the consensus approach.

CONCLUSIONS

The consensus approach has made population-specific logistic regression equations portable to other populations. Excellent diagnostic characteristics can be obtained using simple data and measurements. The consensus approach is best applied utilizing a programmable calculator or a computer program to simplify the process of calculating the probability of CAD using the three equations.

Test characteristics and decision rules

We have demonstrated using several examples how different test characteristics can be used to assist clinicians in making better decisions for their patients. These probabilistic models may seem confusing and difficult to implement. Some general rules may help, such as SnNout and SpPin. Clinicians should know the test characteristics and decision rules for the acute problems they may face. For chronic conditions, advanced planning may be helpful. Electronic medical record systems may be able to incorporate these at the user interface. The improvements in hand-held computers may bring clinical decision-support systems directly to the point of service. We may also begin to see laboratories report test characteristics for important conditions as likelihood ratios (we already see estimates of the risk of heart disease corresponding to different lipid ratios). We also suspect that the medical literature will report likelihood ratios more frequently. As practice networks develop more sophisticated disease-tracking mechanisms, clinicians will be able to obtain estimates of disease prevalence more appropriate to their practice. Ultimately, for physicians to make better decisions, appropriate data are needed, including accurate estimates of test characteristics and of disease probability.

The effect of disease-prevalence adjustments on the accuracy of a logistic prediction model

The accuracy of a logistic prediction model is degraded when it is transported to populations with outcome prevalences different from that of the population used to derive the model. The resultant errors can have major clinical implications. Accordingly, the authors developed a logistic prediction model with respect to the noninvasive diagnosis of coronary disease based on 1,824 patients who underwent exercise testing and coronary angiography, varied the prevalence of disease in various "test" populations by random sampling of the original "derivation" population, and determined the accuracy of the logistic prediction model before and after the application of a mathematical algorithm designed to adjust only for these differences in prevalence. The accuracy of each prediction model was quantified in terms of receiver operating characteristic (ROC) curve area (discrimination) and chi-square goodness-of-fit (calibration). As the prevalence of the test population diverged from the prevalence of the derivation population, discrimination improved (ROC-curve areas increased from 0.82 +/- 0.02 to 0.87 +/- 0.03; p < 0.05), and calibration deteriorated (chi-square goodness-of-fit statistics increased from 9 to 154; p < 0.05). Following adjustment of the logistic intercept for differences in prevalence, discrimination was unchanged and calibration improved (maximum chi-square goodness-of-fit fell from 154 to 16). When the adjusted algorithm was applied to three geographically remote populations with prevalences that differed from that of the derivation population, calibration improved 87%, while discrimination fell by 1%. Thus, prevalence differences produce statistically significant and potentially clinically important errors in the accuracy of logistic prediction models. These errors can potentially be mitigated by use of a relatively simple mathematical correction algorithm.

Prospective validation of artificial neural network trained to identify acute myocardial infarction

BACKGROUND

Artificial neural networks apply non-linear statistics to pattern recognition problems. One such problem is acute myocardial infarction (AMI), a diagnosis which, in a patient presenting as an emergency, can be difficult to confirm. We report here a prospective comparison of the diagnostic accuracy of a network and that of physicians, on the same patients with suspected AMI.

METHODS

Emergency department physicians who evaluated 1070 patients 18 years or older presenting to the emergency department of a teaching hospital in California, USA with anterior chest pain indicated whether they thought these patients had sustained a myocardial infarction. The network analysed the patient data collected by the physicians during their evaluations and also generated a diagnosis.

FINDINGS

The physicians had a diagnostic sensitivity and specificity for myocardial infarction of 73.3% (95% confidence interval 63.3-83.3%) and 81.1% (78.7-83.5%), respectively, while the network had a diagnostic sensitivity and specificity of 96.0% (91.2-100%) and 96.0% (94.8-97.2%), respectively. Only 7% of patients had had an AMI, a low frequency but typical for anterior chest pain.

INTERPRETATION

The application of non-linear neural computational analysis via an artificial neural network to the clinical diagnosis of myocardial infarction appears to have significant potential.

Derivation of a decision rule for the use of radiography in acute knee injuries

STUDY OBJECTIVE

To derive a highly sensitive decision rule for the selective use of radiography in acute knee injuries.

DESIGN

Prospectively administered survey.

SETTING

Emergency departments of two university hospitals.

PARTICIPANTS

Convenience sample of 1,047 adults with acute knee injuries.

RESULTS

Attending emergency physicians assessed each patient for 23 standardized clinical findings, which were recorded on data collection forms. A total of 127 patients was examined independently by two physicians to determine interobserver agreement. The outcome measure was fracture of the knee. Any patients who did not have ED radiography underwent a structured telephone interview to determine the possibility of a missed fracture. Those variables found to be both reliable (highest kappa values) and strongly associated with a fracture (highest chi 2 values) were further analyzed by a recursive-partitioning multivariate technique. The derived decision rule included the following variables: (1) age 55 years or older, (2) tenderness at the head of the fibula, (3) isolated tenderness of the patella, (4) inability to flex to 90 degrees, and (5) inability to bear weight both immediately and in the ED (four steps). The presence of one or more of these findings would have identified the 68 fractures in the study population with a sensitivity of 1.0 (95% confidence interval [Cl], .95 to 1.0) and a specificity of .54 (95% Cl, .51 to .57). Application of the rule would have led to a 28.0% relative reduction in the use of radiography from 68.6% to 49.4% in the study population.

CONCLUSION

A practical, highly sensitive, and reliable decision rule for the use of radiography in acute knee injuries has been derived. Clinical application should await prospective validation of the rule.

You can lead a horse to water--improving physicians' knowledge of probabilities may not affect their decisions

OBJECTIVES

To determine whether improving physicians' judgments of the probability of streptococcal pharyngitis for patients with sore throats would affect their use of antibiotics and affect the variation in such use.

DESIGN

Post-hoc retrospective analysis of data previously collected as part of a controlled trial.

SETTINGS

University student health services in Pennsylvania and Nebraska.

PATIENTS

Sequential patients with pharyngitis seen before and after the time clinicians received either an experimental educational intervention designed to improve probabilistic diagnostic judgments (at the Pennsylvania site) or a control intervention, a standard lecture (at the Nebraska site). The clinician-subjects were the primary case physicians practicing at either site.

MEASUREMENTS

Clinical variables prospectively recorded by the clinicians, probability assessments, and treatment decisions.

RESULTS

At the experimental site, despite marked decreases in clinicians' overestimations of disease probability after the intervention, the proportion of patients prescribed antibiotics showed a trend toward increasing: 100/290 (34.5%) pre-intervention, 90/225 (40%) post-intervention. The intervention did not decrease practice variation between individual doctors. Univariable and multivariable analyses showed no major change in the relationships between clinical variables and treatment decisions after the intervention. At the control site there was no major change in probability judgments or treatment decisions after the intervention.

CONCLUSIONS

Teaching physicians to make better judgments of disease probability may not alter their treatment decisions.

Prediction of failure to survive following in-hospital cardiopulmonary resuscitation: comparison of two predictive instruments

The purpose of this study is to compare two clinical predictive rules, the pre-arrest-morbidity (PAM) index and the prognosis-after-resuscitation (PAR) score, which predict failure to survive following in-hospital cardiopulmonary resuscitation (CPR). The study population consisted of 274 consecutive adult patients who underwent CPR at University College Hospital in Galway, Ireland over a 2-year period. The PAM and PAR scores were calculated from the most recent data available for each variable prior to cardiac arrest. Performance of the predictive scores was compared using Student's t-test, Pearson chi-square, Fisher's exact test, and receiver-operating characteristic (ROC) curves where appropriate. The PAM index identified 23 patients with a score > 4, while the PAR score identified 59 patients with a score > 5, none of whom survived. The sensitivity of the PAR score for the prediction of failure to survive was 23.7%, while that of the PAM index was 9.2%; neither index incorrectly identified a patient as a non-survivor who eventually survived. The PAR score also had a greater area under the ROC curve, although this difference was not statistically significant (P = 0.07). In summary, the PAR score performed better than the PAM index in the identification of patients who are unlikely to survive following CPR. Although further confirmation is necessary, it may provide useful prognostic information to physicians and patients involved with decisions about do-not-resuscitate orders.

Failure to validate a predictive model for refusal of care to emergency-department patients

OBJECTIVE

To determine whether previously developed triage criteria for refusal of care to patients presenting to an emergency department (ED) with nonurgent problems could be validated for an independent patient population.

METHODS

A convenience sample of 534 adults presenting to a municipal hospital ED between July 1, 1992, and October 15, 1992, who met preestablished criteria for refusal of care were entered into a prospective, observational, cohort study. The single target outcome variable was hospitalization. In order to optimize the criteria's performance, both the triage nurse and the physician caring for the patient had to agree that all criteria for "refusal of care" were specifically met. No patient was refused care, nor was a patient's management or disposition interfered with in any way by the investigators. All patients were followed until hospital admission or release from the ED.

RESULTS

Six (1.1%) of 534 patients (95% CI 0.4-2.4) who met the criteria for refusal of care were hospitalized. This represents a greater than 50-fold difference in incidence of hospitalization when compared with that found by other investigators, who reported that only 0.02% (95% CI 0.0004-0.04) of those patients who were refused care subsequently required hospitalization (p < 10 (-7)).

CONCLUSION

The authors were unable to validate a previously developed predictive model for refusal of care to patients presenting to an ED. Refusal of care to selected ED patients based on current guidelines is not a viable solution to overcrowding. Alternative strategies must be sought.

Validation of a clinical prediction rule for the differential diagnosis of acute meningitis

OBJECTIVE

To attempt to validate a previously reported clinical prediction rule derived to assist in distinguishing between acute bacterial meningitis and acute viral meningitis.

DESIGN

Retrospective chart review of patients treated at five hospitals between 1981 and 1990. The criterion standard for bacterial meningitis was a positive cerebrospinal fluid (CSF) or blood culture or a positive test for bacterial antigen in the CSF. For viral meningitis, the criterion standard was a positive viral culture from CSF, stool, or blood or a discharge diagnosis of viral meningitis with no other etiology evident.

SETTING

Two Department of Veterans Affairs (VA) hospitals, two county hospitals, and one private hospital, each affiliated with one of two medical schools.

PATIENTS

All persons aged more than 17 years who were hospitalized over a ten-year period at one of five academically affiliated hospitals for the management of acute meningitis.

MEASUREMENTS AND MAIN RESULTS

Sixty-two cases of bacterial meningitis and 98 cases of viral meningitis were confirmed. With all patients included, the discriminatory power of the model as measured by the area under the receiver operating characteristic curve (AUC) was 0.977 (95% CI, 0.957-0.997), compared with the AUC of 0.97 in the derivation set of the original publication. The AUCs (95% CIs) for data subsets were: Dallas cases 0.994 (0.986-1.0). Milwaukee cases 0.912 (0.834-0.990); ages 18-39 years 0.952 (0.892-1.0), ages 40-59 years 0.99 (0.951-1.0), and age > or = 60 years 0.955 (0.898-1.0).

CONCLUSIONS

The authors conclude that the clinical prediction rule proved robust when applied to a geographically distinct population comprised exclusively of adults. There was sustained performance of the model when applied to cases from each city and from three age strata. Prospective validation of this prediction rule will be necessary to confirm its utility in clinical practice.

Practice variation in the management of pharyngitis: the importance of variability in patients' clinical characteristics and in physicians' responses to them

The objective of this study was to assess whether geographic differences in antibiotic-prescribing rates for patients with pharyngitis could be explained by intersite differences in patients' clinical characteristics and in how physicians responded to these clinical cues when making decisions. As part of the initial phase of a prospective controlled trial to improve physicians' diagnostic ability, the authors enrolled cohorts of consecutive patients seen at staff-model--HMO student health services in Pennsylvania and Nebraska. Physicians' decisions whether to prescribe antibiotics for 310 consecutive patients presenting with pharyngitis to the former and 214 such patients presenting to the latter at the time of the initial visit were examined. There was a large discrepancy between the antibiotic-prescribing rates at the student health services in Pennsylvania, 106/310, 32.4%, and Nebraska, 156/214, 72.9%. The clinical variables significantly independently associated with treatment at both sites in a logistic regression model were fever, adjusted odds ratio = 2.1 (95% CI = 1.1, 3.8); exudates, 5.4 (2.8, 10); palatine petechiae, 6.5 (1.5, 28); rhinorrhea, 0.46, (0.25, 0.85); and high risk of complications, 3.8 (1.04, 14). There was a significant interaction between site and anterior cervical adenopathy, 5.5 (1.6, 19); and a borderline interaction between site and rhinorrhea, 2.4 (0.89, 6.7). Site was not a significant independent predictor of treatment, 1.8 (0.45, 6.6.). Practice variation was related to geographic differences in patients' clinical characteristics and in how physicians responded to these factors when prescribing antibiotics. How physicians weight patients' clinical characteristics when making decisions may be an important element of their "practice styles."

Integrating stratum-specific likelihood ratios with the analysis of ROC curves

Data used to construct receiver operating characteristic (ROC) curves and to calculate the area under the curve (ROC AUC) can be used to derive stratum-specific likelihood ratios (SSLRs) with their 95% confidence intervals (95% CIs). The purpose of this study was to determine whether useful information can be obtained by adding SSLRs to the analysis of ROC curves. The authors analyzed four previously reported sets of data: 1) serum creatine kinase (SCK) for diagnosing acute myocardial infarction (AMI) in the coronary care unit (CCU); 2) SCK in the evaluation of chest pain in the emergency center (EC); 3) four predictor variables in the diagnosis of strep throat; and 4) the ordinal assessment of computed tomographic (CT) images. Use of SCK in the CCU produced four strata that had posttest probabilities that were highly discriminating, whereas SCK in the EC resulted in only two strata with limited discriminating ability. In either study the cutpoint at which the SSLR changed from less than to greater than 1.0 was higher than the reported upper normal for the test, thereby quantitating spectrum bias. The maximum number of strata of predictor signs and symptoms for strep throat was three rather than the five used in previous studies. With a larger sample size or pooling, four strata could probably be developed. With CT images, "definitely normal," "probably normal," and "questionable" were collapsed to one negative stratum. "Probably abnormal" became the true "questionable" stratum and "definitely abnormal" was the only positive stratum. The authors conclude that additional useful information is obtained by deriving stratum-specific likelihood ratios as part of the analysis of an ROC curve.

Evaluation of high-yield criteria for chest radiography in acute exacerbation of chronic obstructive pulmonary disease

STUDY OBJECTIVES

The objectives of this study were to assess the incidence of abnormal chest radiographs and to test the validity of previously developed high-yield criteria. There is disagreement about the need for chest radiography in acute exacerbation of chronic obstructive pulmonary disease, although high-yield criteria have been developed.

DESIGN

Retrospective chart review study.

SETTING

County-owned, university-affiliated, urban emergency department.

PARTICIPANTS

ED patients seen between January 1988 and July 1991 with chronic obstructive pulmonary disease.

RESULTS

Eight hundred forty-seven ED visits were identified; medical records were available for 742. Radiographs were not taken in 8%, leaving 685 ED visits in the study. One hundred nine patients (16%) had significant abnormalities, including 88 new infiltrates, two new lung masses, one pneumothorax, and 20 episodes of pulmonary edema. A history of congestive heart failure and fever was associated with abnormalities, as were findings of rales, pedal edema, and jugular venous distension. There was no association with WBC count, temperature, coronary artery disease, chest pain, or sputum production. Previously published high-yield criteria had a sensitivity of .76; specificity, .41; positive predictive value, 20; negative predictive value, .90; and accuracy, .47.

CONCLUSION

Radiographic abnormalities are common findings in acute exacerbation of chronic obstructive pulmonary disease. We found that almost one fourth of radiographic abnormalities are not predictable on the basis of previously developed high-yield criteria. Routine chest radiography should be considered in patients with acute exacerbation of chronic obstructive pulmonary disease to diagnose treatable, radiographically apparent abnormalities.

Validation of probabilistic predictions

Current advances in high-speed computing and increased availability of statistical software have led to widespread use of statistical methods for the development of computerized protocols predictive of binary health outcomes. If these predictive algorithms are to be used in settings other than those for which they were developed, e.g., applied in a different geographic setting or extrapolated for use in a slightly different population, then they should be carefully validated to ensure appropriate application. Miller et al. (Stat Med. 1991) provided a comprehensive methodology for external validation of logistic prediction models, and applied these methods in a temporal validation setting. In this article, the authors emphasize how these methods can be applied to general forms of probabilistic predictions and provide several SAS macros for computation of the desired statistics.

An algorithm for a selective use of throat swabs in the diagnosis of group A streptococcal pharyngo-tonsillitis in general practice

A prospective evaluation was made of an algorithm for a selective use of throat swabs in patients with sore throat in general practice. The algorithm states that a throat swab should be obtained (a) in all children younger than 15 years; (b) in patients aged 15 years or more who have pain on swallowing and at least three of four signs (enlarged or hyperaemic tonsils; exudate; enlarged or tender angular lymph nodes; and a temperature > or = 38 degrees C); and (c) in adults aged 15-44 years with pain on swallowing and one or two of the four signs, but not both cough and coryza. Group A streptococci were found by laboratory culture in 30% of throat swabs from 1783 patients. Using these results as the reference, the algorithm was 95% sensitive and 26% specific, and assigned 80% of the patients to be swabbed. Its positive and negative predictive values in this setting were 36% and 92%, respectively. It is concluded that this algorithm may be useful in general practice.

Computer probability estimates of angiographic coronary artery disease: transportability and comparison with cardiologists' estimates

A computer algorithm for estimating probabilities of any significant coronary obstruction and triple vessel/left main obstructions was derived, validated, and compared with the assessments of cardiac clinician angiographers. The algorithm performed at least as well as the clinicians when the latter knew the identity of the patients whose angiograms they had decided to perform. The clinicians were more accurate when they did not know the identity of the subjects but worked from tabulated objective data. Referral and value induced bias may affect physician judgment in assessing disease probability. Application of computer aids or consultation with cardiologists not directly involved with patient management may assist in more rational assessments and decision making.

Reproducibility of predictor variables from a validated clinical rule

It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regression. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p < .0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250.(ABSTRACT TRUNCATED AT 250 WORDS)

The impact of a rapid screen for streptococcal pharyngitis on clinical decision making in the emergency department

The impact of a rapid streptococcal screening test (RSS) on clinical decision making in the management of patients with acute pharyngitis was evaluated. Physicians managing 95 ambulatory patients with the complaint of sore throat were asked to estimate the probability of group A beta-hemolytic streptococcal pharyngitis (GRABS) in each patient and whether they intended to treat with antibiotics both before and after the results of the RSS were known. Simultaneous throat cultures were obtained for 80 patients at the discretion of the treating physicians. Regression analysis revealed that the RSS result was an important independent predictor for the posttest decision to treat with antibiotics and for the estimated probability of disease. The mean absolute log-likelihood ratio was used to quantify the effect of the RSS on diagnostic certainty; a greater contribution to diagnostic certainty occurred when the RSS was positive. Physicians would have prescribed an antibiotic 49 times without the benefit of the RSS and prescribed an antibiotic 48 times given the RSS result. Without the RSS, the treatment decision contradicted the throat culture result in 25 cases. With the RSS, this occurred in 26 cases. The RSS, while influencing decision making, especially when positive, did not decrease the use of antibiotics.

Confidence intervals for post-test probability

Confidence intervals are a natural way to describe the uncertainty of post-test probability in diagnostic tests. We consider confidence intervals for two different scenarios. At a site, for example, hospital emergency room or student health centre, with measured values of disease prevalence, sensitivity and specificity available, the confidence interval is similar to results in the literature, but at a site where measured values of these indices are unavailable, we develop a method, using the values of disease prevalence, sensitivity and specificity from other sites, to obtain a confidence interval for post-test probability. We use the diagnosis of strep throat to illustrate the results. We also obtain confidence intervals from simulations to compare with the results of both scenarios.

The trauma triage rule: a new, resource-based approach to the prehospital identification of major trauma victims

STUDY OBJECTIVE

To develop a new trauma decision rule.

DESIGN

Retrospective clinical review.

SETTING

Level I trauma center.

TYPE OF PARTICIPANTS

1,004 injured adults.

MEASUREMENTS AND MAIN RESULTS

A new trauma decision rule was derived from 1,004 injured adult patients using a new operational definition of major trauma. The rule, termed the Trauma Triage Rule, defines a major trauma victim as any injured adult patient whose systolic blood pressure is less than 85 mm Hg; whose motor component of the Glasgow Coma Score is less than 5; or who has sustained penetrating trauma of the head, neck, or trunk. Using the operational definition of major trauma, the rule had a sensitivity of 92% and a specificity of 92% when tested on the 1,004-patient cohort.

CONCLUSION

The Trauma Triage Rule may significantly reduce overtriage while only minimally increasing undertriage. This approach must be validated prospectively before it can be used in the prehospital setting.

Hemodynamic status in critically ill patients with and without acute heart disease

Physicians have been urged to reduce the use of the pulmonary artery catheter. However, there are no guidelines to help the clinician make the decision to use or withhold invasive monitoring in the individual patient. This study was designed to examine the accuracy of physician estimates of cardiac function in a spectrum of patients with hemodynamic instability to determine whether differences in accuracy among subgroups would suggest subgroups of patients who could be managed without invasive measurements. Physician estimates of cardiac index were found to be sufficiently accurate in patients without acute heart disease that initial management without invasive monitoring may be appropriate in selected cases. However, due to the general inaccuracy of physician estimates, efforts to improve the accuracy of clinical judgments of cardiac function and hemodynamic status should be pursued with vigor in patients both with and without acute cardiac dysfunction.

Using the patient's history to estimate the probability of coronary artery disease: a comparison of primary care and referral practices

PURPOSE

According to probability theory, the interpretation of new information should depend on the prior probability of disease. We asked if this principle applies to interpreting the history in patients with chest pain. We compared the prevalence of coronary artery disease (CAD) in patients who had similar histories but who came from populations with different disease prevalence.

PATIENTS AND METHODS

We studied two high-disease-prevalence populations (patients referred for coronary arteriography) and two low-disease-prevalence populations (patients from primary care practices). We used clinical characteristics of one arteriography population to develop a logistic rule for estimating the probability of coronary artery narrowing. The number of clinical findings determined the logistic score, which was proportional to the prevalence of CAD.

RESULTS

The prevalence of CAD was much lower in the primary care population than in the arteriography population, even when patients with similar logistic scores, and thus similar clinical histories, were compared.

CONCLUSION

A clinician must take account of the overall prevalence of disease in the clinical setting when using the patient's history to estimate the probability of disease. Failure to observe this caution may lead to errors in test selection and interpretation.

Clinical findings associated with acute proximal deep vein thrombosis: a basis for quantifying clinical judgment

PURPOSE

To identify clinical findings useful in estimating the probability of acute proximal deep vein thrombosis (DVT).

PATIENTS AND METHODS

The records of 355 symptomatic patients who underwent ascending venography were reviewed. Data on 76 clinical items were collected using standardized forms. Venograms were interpreted according to standard criteria and interobserver agreement was evaluated in a sample of 119 venograms. Independent clinical correlates of proximal DVT were identified using multivariate discriminant analysis in 236 randomly chosen patients; they were tested in the remaining 119 patients.

RESULTS

Acute proximal DVT was shown by venogram in 96 patients (27%). Five independent clinical correlates of proximal DVT--swelling above the knee of the affected leg, swelling below the knee, recent immobility, cancer, fever--predicted proximal DVT in the testing group; in patients with none, one, or two or more of these clinical findings, proximal DVT was present in 5%, 15%, and 42%, respectively. If venography had been performed only in patients with one or more of the five factors, 97% of cases of proximal DVT would have been diagnosed and venography would have been avoided in 26% of patients with normal test results.

CONCLUSION

These data provide a quantitative basis for estimating the probability of proximal DVT on the basis of clinical findings in symptomatic patients. How these findings can best be integrated with noninvasive testing and venography into diagnostic strategies for DVT remains to be determined.

The failure of prehospital trauma prediction rules to classify trauma patients accurately

Clinical prediction rules are used extensively by most regionalized trauma systems to identify which patients have sustained major injuries. Because of reported high misclassification rates of some of these rules and the known global difficulty of transporting prediction rules, four such rules (the Trauma Score, the CRAMS Scale, the Revised Trauma Score, and the Prehospital Index) and two newly derived rules were statistically analyzed using a cohort of 2,434 injured patients. All rules accurately predicted mortality with a minimum sensitivity and specificity of 85%. However, not one of the rules was able accurately to identify surviving patients who had sustained major injuries. In this instance, no rule was able to achieve a sensitivity of at least 70% while achieving a specificity of 70%. These results suggest that the problem with trauma prediction rules lies in the inherent limitations of the clinical data on which they are based. In view of this, the usefulness of existing prehospital trauma predictive rules must be questioned.

Rational decision making based on history: adult sore throats

Primary care physicians are often required to make preliminary evaluations based only on the patient's history, especially during telephone encounters about sore throats. The authors studied adults with sore throats to determine whether patients can be stratified into higher and lower risks of strep throat by history alone. They first obtained data from 517 patients seen in an emergency room. Providers graded symptoms on a four-point scale (absent, mild, moderate, or severe). Initial analyses showed that prediction based on history should include three variables: fever, difficulty in swallowing, and cough. For ease of computation, these were consolidated into one score, "history" (= fever history + difficulty in swallowing - cough). This score was used to develop a model that predicts the probability of infection with group A beta-hemolytic streptococcus, and the model's performance was tested in two additional patient groups. The predictive accuracy of the "history" score was confirmed in all patient groups, despite differences in providers and disease prevalences. Primary care physicians may use this model to help them make decisions in situations such as telephone encounters without using additional data.

A computer protocol to predict myocardial infarction in emergency department patients with chest pain

To achieve more appropriate triage to the coronary care unit of patients presenting with acute chest pain, we used clinical data on 1379 patients at two hospitals to construct a simple computer protocol to predict the presence of myocardial infarction. When we tested this protocol prospectively in 4770 patients at two university hospitals and four community hospitals, the computer-derived protocol had a significantly higher specificity (74 vs. 71 percent) in predicting the absence of infarction than physicians deciding whether to admit patients to the coronary care unit, and it had a similar sensitivity in detecting the presence of infarction (88.0 vs. 87.8 percent). Decisions based solely on the computer protocol would have reduced the admission of patients without infarction to the coronary care unit by 11.5 percent without adversely affecting the admission of patients in whom emergent complications developed that required intensive care. Although this protocol should not be used to override careful clinical judgment in individual cases, the computer protocol for the most part yields accurate estimates of the probability of myocardial infarction. Decisions about admission to the coronary care unit based on the protocol would have been as effective as those actually made by the unaided physicians who cared for the patients, and less costly. Whether physicians who are aided by the protocol perform better than unaided physicians cannot be determined without further study.