Using computer modeling to help identify patient subgroups in clinical data repositories

OBJECTIVE

The ability to accurately and efficiently identify patient cases of interest in a hospital information system has many important clinical, research, educational and administrative uses. The identification of cases of interest sometimes can be difficult. This paper describes a two-stage method for searching for cases of interest.

DESIGN

First, a Boolean search is performed using coded database variables. The user classifies the retrieved cases as being of interest or not. Second, based on the user-classified cases, a computer model of the patient cases of interest is constructed. The model is then used to help locate additional cases. These cases provide an augmented training set for constructing a new computer model of the cases of interest. This cycle of modeling and user classification continues until halted by the user.

MEASUREMENTS

This paper describes a pilot study in which this method is used to identify the records of patients who have venous thrombosis.

RESULTS

The results indicate that computer modeling enhances the identification of patient cases of interest.

Representing and developing temporally abstracted knowledge as a means towards facilitating time modeling in medical decision-support systems

The utilization of the appropriate level of temporal abstraction is an important aspect of time modeling. We discuss some aspects of the relation of temporal abstraction to important knowledge engineering parameters such as model correctness, ease of model specification, knowledge availability, query completeness, inference tractability, and semantic clarity. We propose that versatile and efficient time-modeling formalisms should encompass ways to represent and reason at more than one level of abstraction, and we discuss such a hybrid formalism. Although many research efforts have concentrated on the automation of specific temporal abstractions, much research needs to be done in understanding and developing provably optimal abstractions. We provide an initial framework for studying this problem in a manner that is independent of the particular problem domain and knowledge representation, and suggest several research challenges that appear worth pursuing.

An evaluation of machine-learning methods for predicting pneumonia mortality

This paper describes the application of eight statistical and machine-learning methods to derive computer models for predicting mortality of hospital patients with pneumonia from their findings at initial presentation. The eight models were each constructed based on 9847 patient cases and they were each evaluated on 4352 additional cases. The primary evaluation metric was the error in predicted survival as a function of the fraction of patients predicted to survive. This metric is useful in assessing a model's potential to assist a clinician in deciding whether to treat a given patient in the hospital or at home. We examined the error rates of the models when predicting that a given fraction of patients will survive. We examined survival fractions between 0.1 and 0.6. Over this range, each model's predictive error rate was within 1% of the error rate of every other model. When predicting that approximately 30% of the patients will survive, all the models have an error rate of less than 1.5%. The models are distinguished more by the number of variables and parameters that they contain than by their error rates; these differences suggest which models may be the most amenable to future implementation as paper-based guidelines.

Human causal discovery from observational data

Utilizing Bayesian belief networks as a model of causality, we examined medical students' ability to discover causal relationships from observational data. Nine sets of patient cases were generated from relatively simple causal belief networks by stochastic simulation. Twenty participants examined the data sets and attempted to discover the underlying causal relationships. Performance was poor in general, except at discovering the absence of a causal relationship. This work supports the potential for combining human and computer methods for causal discovery.

A temporal analysis of QMR

OBJECTIVE

To understand better the trade-offs of not incorporating explicit time in Quick Medical Reference (QMR), a diagnostic system in the domain of general internal medicine, along the dimensions of expressive power and diagnostic accuracy.

DESIGN

The study was conducted in two phases. Phase I was a descriptive analysis of the temporal abstractions incorporated in QMR's terms. Phase II was a pseudo-prospective controlled experiment, measuring the effect of history and physical examination temporal content on the diagnostic accuracy of QMR.

MEASUREMENTS

For each QMR finding that would fit our operational definition of temporal finding, several parameters describing the temporal nature of the finding were assessed, the most important ones being: temporal primitives, time units, temporal uncertainty, processes, and patterns. The history, physical examination, and initial laboratory results of 105 consecutive patients admitted to the Pittsburgh University Presbyterian Hospital were analyzed for temporal content and factors that could potentially influence diagnostic accuracy (these included: rareness of primary diagnosis, case length, uncertainty, spatial/causal information, and multiple diseases).

RESULTS

776 findings were identified as temporal. The authors developed an ontology describing the terms utilized by QMR developers to express temporal knowledge. The authors classified the temporal abstractions found in QMR in 116 temporal types, 11 temporal templates, and a temporal hierarchy. The odds of QMR's making a correct diagnosis in high temporal complexity cases is 0.7 the odds when the temporal complexity is lower, but this result is not statistically significant (95% confidence interval = 0.27-1.83).

CONCLUSIONS

QMR contains extensive implicit time modeling. These results support the conclusion that the abstracted encoding of time in the medical knowledge of QMR does not induce a diagnostic performance penalty.

Representing CARE rules in a decision-theoretic formalism

Improvement in the performance of reminder systems may be facilitated by the use of new representations. A decision-theoretic representation, for example, may enable a reminder system to represent and reason about the probabilities that a reminder will be a true or a false alarm and the relative utilities of these events. We extended a previously described decision-theoretic model to include such events. The model now represents explicitly the uncertainty, costs, and benefits of sending a reminder. We also extended the model to remove an assumption of reminder independence. As a step towards testing a hypothesis that this approach will support better performance than a rule-based approach, we analyzed a set of CARE rules and showed that our representation can represent these rules.

Progress in the development of trauma systems in the United States. Results of a national survey

OBJECTIVE

To examine the status of trauma system development and key structural and operational characteristics of these systems.

DESIGN AND SETTING

National survey of trauma systems with enabling state statute, regulation, or executive orders and for which designated trauma centers were present.

PARTICIPANTS

Trauma system administrators and directors of 37 state and regional organizations that had legal authority to administer trauma systems, which represented a response rate of 90.2%.

MAIN OUTCOME MEASURES

Trauma system components that had been implemented or were under development.

RESULTS

From 1988 to 1993, the number of states meeting one set of criteria for a complete trauma system criteria increased from two to five. The most common deficiency in establishing trauma systems was failure to limit the number of designated trauma centers based on community need. Although most existing trauma systems have developed formal processes for designating trauma centers, prehospital triage protocols to allow hospital bypass, and centralized trauma registries, several systems lack standardized policies for interhospital transfer and systemwide evaluation.

CONCLUSION

State and regional organizations have accomplished a great deal but still have substantial work ahead in developing comprehensive trauma systems. Research is needed to better understand the relationship between trauma volume and outcomes of care as well as the impact of trauma system structure and operational characteristics on care delivery. Improved measures of patient outcome are also needed so that effective system evaluation can take place.

The use of misclassification costs to learn rule-based decision support models for cost-effective hospital admission strategies

Cost-effective health care is at the forefront of today's important health-related issues. A research team at the University of Pittsburgh has been interested in lowering the cost of medical care by attempting to define a subset of patients with community-acquire pneumonia for whom outpatient therapy is appropriate and safe. Sensitivity and specificity requirements for this domain make it difficult to use rule-based learning algorithms with standard measures of performance based on accuracy. This paper describes the use of misclassification costs to assist a rule-based machine-learning program in deriving a decision-support aid for choosing outpatient therapy for patients with community-acquired pneumonia.

A new formalism for temporal modeling in medical decision-support systems

We present a new mathematical formalism, which we call modifiable temporal belief networks (MTBNs) that extends the concept of an ordinary belief network (BN) to incorporate a dynamic causal structure and explicit temporal semantics. An important feature of MTBNs is that they allow portions of the model to be abstract and portions of it to be temporally explicit. We show how this property can lead to substantial knowledge acquisition and computational complexity savings. In addition to temporal modeling, the language of MTBNs can be an important analytical tool, as well as temporal language for causal discovery.

Evaluation of a belief-network-based reminder system that learns from utility feedback

PRETRIEVE is a belief-network-based, unsolicited information-retrieval system that performs machine learning based on user feedback. We report here on the document-ordering and document-retrieval performance of PRETRIEVE. We developed a test collection of 410 judgments of document utility in a simulated medical order-entry context. We characterized the validity of these judgments, which were elicited from domain experts, by measuring interrater and intrarater reproducibility. We developed a measure of the quality of document orderings similar to the ROC-curve analysis used to evaluate document-retrieval systems. We found that the ordering performance of the PRETRIEVE system was (1) substantially better than random, (2) somewhat less than ideal, and (3) superior to that of versions of the PRETRIEVE system that used relevance feedback instead of utility feedback. Under a set of assumptions, which we make explicit, we found that the documents retrieved by a version of PRETRIEVE that modeled time cost were of higher utility than those retrieved by a similar rule-based system.

Building a medical multimedia database system to integrate clinical information: an application of high-performance computing and communications technology

The rapid growth of diagnostic-imaging technologies over the past two decades has dramatically increased the amount of nontextual data generated in clinical medicine. The architecture of traditional, text-oriented, clinical information systems has made the integration of digitized clinical images with the patient record problematic. Systems for the classification, retrieval, and integration of clinical images are in their infancy. Recent advances in high-performance computing, imaging, and networking technology now make it technologically and economically feasible to develop an integrated, multimedia, electronic patient record. As part of The National Library of Medicine's Biomedical Applications of High-Performance Computing and Communications program, we plan to develop Image Engine, a prototype microcomputer-based system for the storage, retrieval, integration, and sharing of a wide range of clinically important digital images. Images stored in the Image Engine database will be indexed and organized using the Unified Medical Language System Metathesaurus and will be dynamically linked to data in a text-based, clinical information system. We will evaluate Image Engine by initially implementing it in three clinical domains (oncology, gastroenterology, and clinical pathology) at the University of Pittsburgh Medical Center.

A temporal analysis of QMR: abstracted temporal representation and reasoning and initial assessment of diagnostic performance trade-offs

Explicit temporal representation and reasoning (TRR) in medical decision-support systems (MDSS) is generally considered to be a useful but often neglected aspect of system design and implementation. Given the great burden of explicit TRR both in knowledge acquisition and computational efficiency, developers of general-purpose large-scale systems typically utilize implicit (i.e., abstracted) forms of TRR. We are interested in understanding better the trade-offs of not incorporating explicit TRR in large general-purpose MDSS along the dimensions of system expressive power and diagnostic accuracy. In particular, we examine the types of abstracted TRR employed in QMR, a diagnostic system in the domain of general internal medicine, and the high-level effects of such an implicit treatment of time in the system's diagnostic performance. We present our findings and discuss implications for MDSS design and implementation practices.

An evaluation of explanations of probabilistic inference

Providing explanations of the conclusions of decision-support systems can be viewed as presenting inference results in a manner that enhances the user's insight into how these results were obtained. The ability to explain inferences has been demonstrated to be an important factor in making medical decision-support systems acceptable for clinical use. Although many researchers in artificial intelligence have explored the automatic generation of explanations for decision-support systems based on symbolic reasoning, research in automated explanation of probabilistic results has been limited. We present the results of an evaluation study of INSITE, a program that explains the reasoning of decision-support systems based on Bayesian belief networks. In the domain of anesthesia, we compared subjects who had access to a belief network with explanations of the inference results to control subjects who used the same belief network without explanations. We show that, compared to control subjects, the explanation subjects demonstrated greater diagnostic accuracy, were more confident about their conclusions, were more critical of the belief network, and found the presentation of the inference results more clear.

Data explorer: a prototype expert system for statistical analysis

The inadequate analysis of medical research data, due mainly to the unavailability of local statistical expertise, seriously jeopardizes the quality of new medical knowledge. Data Explorer is a prototype Expert System that builds on the versatility and power of existing statistical software, to provide automatic analyses and interpretation of medical data. The system draws much of its power by using belief network methods in place of more traditional, but difficult to automate, classical multivariate statistical techniques. Data Explorer identifies statistically significant relationships among variables, and using power-size analysis, belief network inference/learning and various explanatory techniques helps the user understand the importance of the findings. Finally the system can be used as a tool for the automatic development of predictive/diagnostic models from patient databases.

Decision-theoretic information pretrieval: a generalization of reminding

Reminder systems and clinical medical librarian services often provide information to clinicians without requiring that a clinician actively seek information. This characteristic may explain in part the effectiveness and high clinician acceptance of these systems. We term systems with this characteristic "information pretrieval systems" to distinguish them from information retrieval systems, which require a clinician to articulate an information need in the form of a query. Because of the increasing importance of information pretrieval systems in medical care, we have developed a decision-theoretic model of an ideal information pretrieval system. In this paper, we present this model and suggest its use as an analytic framework for understanding existing approaches, and as a formal basis for a functioning pretrieval system.

Evaluation of a Meta-1-based automatic indexing method for medical documents

This paper describes MetaIndex, an automatic indexing program that creates symbolic representations of documents for the purpose of document retrieval. MetaIndex uses a simple transition network parser to recognize a language that is derived from the set of main concepts in the Unified Medical Language System Metathesaurus (Meta-1). MetaIndex uses a hierarchy of medical concepts, also derived from Meta-1, to represent the content of documents. The goal of this approach is to improve document retrieval performance by better representation of documents. An evaluation method is described, and the performance of MetaIndex on the task of indexing the Slice of Life medical image collection is reported.

Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. II. Evaluation of diagnostic performance

We have developed a probabilistic reformulation of the Quick Medical Reference (QMR) system. In Part I of this two-part series, we described a two-level, multiply connected belief-network representation of the QMR knowledge base and a simulation algorithm to perform probabilistic inference on the reformulated knowledge base. In Part II of this series, we report on an evaluation of the probabilistic QMR, in which we compare the performance of QMR to that of our probabilistic system on cases abstracted from continuing medical education materials from Scientific American Medicine. In addition, we analyze empirically several components of the probabilistic model and simulation algorithm.

Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. I. The probabilistic model and inference algorithms

In Part I of this two-part series, we report the design of a probabilistic reformulation of the Quick Medical Reference (QMR) diagnostic decision-support tool. We describe a two-level multiply connected belief-network representation of the QMR knowledge base of internal medicine. In the belief-network representation of the QMR knowledge base, we use probabilities derived from the QMR disease profiles, from QMR imports of findings, and from National Center for Health Statistics hospital-discharge statistics. We use a stochastic simulation algorithm for inference on the belief network. This algorithm computes estimates of the posterior marginal probabilities of diseases given a set of findings. In Part II of the series, we compare the performance of QMR to that of our probabilistic system on cases abstracted from continuing medical education materials from Scientific American Medicine. In addition, we analyze empirically several components of the probabilistic model and simulation algorithm.

Algorithms for Bayesian belief-network precomputation

Bayesian belief networks provide an intuitive and concise means of representing probabilistic relationships among the variables in expert systems. A major drawback to this methodology is its computational complexity. We present an introduction to belief networks, and describe methods for precomputing, or caching, part of a belief network based on metrics of probability and expected utility. These algorithms are examples of a general method for decreasing expected running time for probabilistic inference. We first present the necessary background, and then present algorithms for producing caches based on metrics of expected probability and expected utility. We show how these algorithms can be applied to a moderately complex belief network, and present directions for future research.

Synthesis of 5 beta,17 alpha-19-norpregn-20-yne-3 beta,17-diol and of 5 beta,17 alpha-19-norpregn-20-yne-3 alpha,17-diol, human metabolites of norethindrone

A convenient synthesis of both 5 beta,17 alpha-19-norpregn-20-yne-3 beta,17-diol (1) and 5 beta,17 alpha-19-norpregn-20-yne-3 alpha,17-diol (2) in multigram quantities from estr-4-ene-3,17-dione is reported. Full characterization of these often-cited human metabolites of norethindrone is presented for the first time.