Validation of the medical expert system PNEUMON-IA

Developing a pneumonia diagnosis ontology from multiple knowledge sources

Background: Pneumonia is difficult to differentiate from other pulmonary diseases because it shares many symptoms with these diseases. Diagnosing pneumonia in clinical practice would benefit from having access to a codified representation of clinical knowledge. An ontology represents a well-established paradigm for such codification. Objectives: The goal of this research is to create Pneumonia Diagnosis Ontology (PNADO) that brings together the medical knowledge dispersed among multiple medical knowledge sources. Material and Methods: We used several clinical practice guidelines (CPGs) describing the pneumonia diagnostic process as a starting point in developing PNADO. Preliminary version of PNADO was subsequently expanded to cover a broader range of the concepts by reusing ontologies from Open Biological and Biomedical Ontology (OBO) Foundry and BioPortal. PNADO was evaluated by examining relevant concepts from the pneumonia-specific systematic reviews, using patient data from the MIMIC-III clinical dataset, and by clinical domain experts. Results: PNADO is a comprehensive ontology and has a rich set of classes and properties that cover different types of pneumonia, pathogens, symptoms, clinical signs, laboratory tests and imaging, clinical findings, complications, and diagnoses. Conclusion: PNADO unifies pneumonia diagnostic concepts from multiple knowledge sources. It is available in the BioPortal repository.

A practical review and taxonomy of fuzzy expert systems: methods and applications

Purpose

Expert systems are computer-based systems that mimic the logical processes of human experts or organizations to give advice in a specific domain of knowledge. Fuzzy expert systems use fuzzy logic to handle uncertainties generated by imprecise, incomplete and/or vague information. The purpose of this paper is to present a comprehensive review of the methods and applications in fuzzy expert systems.

Design/methodology/approach

The authors have carefully reviewed 281 journal publications and 149 conference proceedings published over the past 37 years since 1982. The authors grouped the journal publications and conference proceedings separately accordingly to the methods, application domains, tools and inference systems.

Findings

The authors have synthesized the findings and proposed useful suggestions for future research directions. The authors show that the most common use of fuzzy expert systems is in the medical field.

Originality/value

Fuzzy logic can be used to manage uncertainty in expert systems and solve problems that cannot be solved effectively with conventional methods. In this study, the authors present a comprehensive review of the methods and applications in fuzzy expert systems which could be useful for practicing managers developing expert systems under uncertainty.

Expert systems in clinical microbiology

This review aims to discuss expert systems in general and how they may be used in medicine as a whole and clinical microbiology in particular (with the aid of interpretive reading). It considers rule-based systems, pattern-based systems, and data mining and introduces neural nets. A variety of noncommercial systems is described, and the central role played by the EUCAST is stressed. The need for expert rules in the environment of reset EUCAST breakpoints is also questioned. Commercial automated systems with on-board expert systems are considered, with emphasis being placed on the "big three": Vitek 2, BD Phoenix, and MicroScan. By necessity and in places, the review becomes a general review of automated system performances for the detection of specific resistance mechanisms rather than focusing solely on expert systems. Published performance evaluations of each system are drawn together and commented on critically.

Evaluating detection and diagnostic decision support systems for bioterrorism response

We evaluated the usefulness of detection systems and diagnostic decision support systems for bioterrorism response. We performed a systematic review by searching relevant databases (e.g., MEDLINE) and Web sites for reports of detection systems and diagnostic decision support systems that could be used during bioterrorism responses. We reviewed over 24,000 citations and identified 55 detection systems and 23 diagnostic decision support systems. Only 35 systems have been evaluated: 4 reported both sensitivity and specificity, 13 were compared to a reference standard, and 31 were evaluated for their timeliness. Most evaluations of detection systems and some evaluations of diagnostic systems for bioterrorism responses are critically deficient. Because false-positive and false-negative rates are unknown for most systems, decision making on the basis of these systems is seriously compromised. We describe a framework for the design of future evaluations of such systems.

Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance

Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.

Evaluation of clinical information systems. What can be evaluated and what cannot?

The evaluation of clinical information systems is essential as they are increasingly used in clinical routine and may even influence patient outcome on the basis of reminder functions and decision support. Therefore we try to answer three questions in this paper: what to evaluate; how to evaluate; how to interpret the results. Those key questions lead to the discussion of goals, methods and results of evaluation studies in a common context. We will compare the objectivist and the subjectivist evaluation approach and illustrate the evaluation process itself in some detail, discussing different phases of software development and potential evaluation techniques in each phase. We use four different practical examples of evaluation studies that were conducted in various settings to demonstrate how defined evaluation goals may be achieved with a limited amount of resources. This also illustrates advantages, limitations and costs of the different evaluation methods and techniques that may be used when evaluating clinical information systems.

Renoir, Pneumon-IA and Terap-IA: three medical applications based on fuzzy logic

The research at the IIIA has produced over more than a decade two versions of a tool for developing knowledge-based systems: Milord and Milord II. This tool has been mainly used for the development of medical applications. In this paper we summarize the Milord II approximate reasoning approach based on fuzzy sets, and three medical applications: rheumatology diagnosis (Renoir), pneumonia diagnosis (Pneumon-IA) and pneumonia treatment (Terap-IA).

Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited

In the validation of medical expert systems, agreement among different human specialists on a random sample of cases may be taken as a substitute to a missing gold standard. Distance measures between pairs of experts, extensively described in previous studies, do not take into account the influence of chance-expected agreement. A weighted kappa index, with three different weighting schemes, is proposed as an alternative to be applied in the general situation of N cases assessed by E experts about K possible diagnoses, each of them qualified with one of G ordinal categories. A hierarchical cluster analysis, applied to the kappa matrices generated, allows for the classification of the expert system among clinical specialists, providing a relative assessment of its diagnostic ability. The above methodology is applied to the validation of two medical expert systems, PNEUMON-IA and RENOIR.

A reliability study for evaluating information extraction from radiology reports

GOAL

To assess the reliability of a reference standard for an information extraction task.

SETTING

Twenty-four physician raters from two sites and two specialties judged whether clinical conditions were present based on reading chest radiograph reports.

METHODS

Variance components, generalizability (reliability) coefficients, and the number of expert raters needed to generate a reliable reference standard were estimated.

RESULTS

Per-rater reliability averaged across conditions was 0.80 (95% CI, 0.79-0.81). Reliability for the nine individual conditions varied from 0.67 to 0.97, with central line presence and pneumothorax the most reliable, and pleural effusion (excluding CHF) and pneumonia the least reliable. One to two raters were needed to achieve a reliability of 0.70, and six raters, on average, were required to achieve a reliability of 0.95. This was far more reliable than a previously published per-rater reliability of 0.19 for a more complex task. Differences between sites were attributable to changes to the condition definitions.

CONCLUSION

In these evaluations, physician raters were able to judge very reliably the presence of clinical conditions based on text reports. Once the reliability of a specific rater is confirmed, it would be possible for that rater to create a reference standard reliable enough to assess aggregate measures on a system. Six raters would be needed to create a reference standard sufficient to assess a system on a case-by-case basis. These results should help evaluators design future information extraction studies for natural language processors and other knowledge-based systems.

Comparison between diagnoses of human experts and a neurotologic expert system

The decision-making ability of a recently developed neurotologic expert system was compared with the diagnoses of six physicians. Five of the physicians were residents and one was a specialist in the field of otolaryngology. The test patients were randomly selected from vertiginous patients referred to an otolaryngology clinic. The expert system and the physicians first had identical information on patient history, symptoms, and tests. During the second phase of the study the physicians were allowed to use the full medical records. The correct diagnoses were certified by an experienced specialist in neurotology. The expert system did better in decision-making when both the expert system and the physicians had identical information on patients. However, when the physicians were allowed to use patient's complete medical records, they surpassed the expert system. The expert system diagnosed 65% of the cases, while the physicians first diagnosed 54% of the cases, and then with complete information, 69% of the cases. From the patients' medical records, the physicians obtained information on the time perspective of the symptoms and the progression of the disease. These aspects will be used to further improve the expert system.

Decision support system to assist mechanical ventilation in the adult respiratory distress syndrome

This paper presents a knowledge-based decision support system to assist mechanical ventilation in patients with the Adult Respiratory Distress Syndrome (DSSARDS). The knowledge base uses clinical algorithms developed from interviews and seminars with experts. The system contains 140 rules, applies backward chaining and was built on an IBM-PC compatible microcomputer. Clinical and physiological data and ventilator settings were used for suggestions of ventilatory support mode (VSMODE) and settings (MVSET) and for hemodynamic evaluation and therapy (HEMO). Success rates (s) and kappa coefficient (k) were used to measure agreement between DSSARDS and physicians at 4 decision steps related to: beginning of mechanical ventilation (FIRSTSET), VSMODE, MVSET and HEMO, DSSARDS prototype was evaluated in a development phase with 6 patients aged 48.6 +/- 15.9 years. Agreement results for 142 decision steps were: FIRSTSET k = 0.90, s = 0.93; VSMODE k = 0.76, s = 0.92; HEMO k = 0.58, s = 0.70, MVSET k = 0.86, s = 0.92 (p < 0.05 for all k). Improvements in the knowledge base were performed mainly in HEMO and VSMODE modules. The subsequent test phase studied 5 patients aged 54.8 +/- 11.0 years in a total of 900 decision steps. Results were: FIRSTSET k = 0.93, s = 0.95; VSMODE k = 0.93, s = 0.96; HEMO k = 0.97, s = 0.99, MVSET k = 0.96, s = 0.97 (p < 0.05 for all k). The results indicate significant agreement between DSSARDS and physicians for all decision steps. This suggests that DSSARDS may be used as a support for decision making and a training tool for mechanical ventilation in patients with the adult respiratory distress syndrome.

Real time validation of paediatric biochemical reports using the Valab-Biochem system

Validation of biochemical reports must be fast and clinically accurate to be of assistance to clinicians. Considerable skill is required to analyse the consistency of different data in the report and to consider influences on the data. When performed throughout the day, such analysis is time-consuming and uncertain. We therefore decided to use a computer-assisted validation system, Valab-Biochem. Its decisions result from a decision tree based primarily on the intrinsic consistency of the data, validation ranges and patients' sex, age and hospital ward. Three hundred randomly chosen reports were simultaneously submitted to Valab-Biochem and to five biologists in order to analyse the computer's findings. The sensitivity of Valab-Biochem was 80% compared to biologists' consensus decision, which was taken as the gold standard. The specificity was 78%. This system provided autonomous assessment of the reports and could be used as an initial screen to assist biologists and focus attention on potentially inconsistent reports.

Case-based learning of plans and goal states in medical diagnosis

We introduce a case-based system, BOLERO, that learns both plans and goal states. The major aim is that of improving the performance of a rule-based diagnosis system by adapting its behavior using the most recent information available about a patient. On the one hand BOLERO gets knowledge from cases in the form of diagnostic plans that are represented as sequences of decision steps. The advantages of this representation include: (1) retrieval and adaptation of parts of plans (steps) appropriate to the current problem state; (2) generation of new plans not previously available in memory; and (3) learning from experience, both from successful or failed plans. On the other hand, since goal states are sets of final diagnosis likelihoods they are not known beforehand, i.e. goal states are not defined and the system has to learn to recognize them. For this reason BOLERO has a case-based method that uses solutions of past cases to recognize a diagnostic state as a goal state of a new planning problem. BOLERO and a rule-based system are integrated into a meta-level architecture in which we emphasize the collaboration of both systems in solving problems. The rule-based system executes the plans generated by BOLERO. As a consequence of the execution of plans, the rule-based system furnishes BOLERO with new information with which BOLERO can generate a new plan to adapt the reasoning process of the rule-based system into correspondence with the recent available data. All the methods have been designed to be useful for medical diagnosis and have been tested in the domain of diagnosing pneumonia.

Comparison of inference results of two otoneurological expert systems

In this paper, two different otoneurological expert systems, Vertigo and One, the latter developed by us, are considered. The expert systems are evaluated as regards their correctness in reasoning diagnoses. In the light of our data collected from randomly selected test patients, One, being a newer technique, is more effective, since it could infer more cases than vertigo did. All the data was also evaluated and diagnosed by otoneurological specialists, independently of the expert systems, to guarantee objectivity in evaluation of the results of the expert systems.

An essay on power of expert systems versus human expertise

In connection with several recent studies of medical informatics, the usefulness and use of expert systems have been both criticized and defended. We have examined the issue of the inference power of expert systems compared to that of human experts. At an abstract level we have shown that there is no doubt that expert systems could successfully complement human experts within strictly limited and well-defined specialties, and actually be of reasonable aid in diagnosis, provided that the expert systems have been correctly and effectively elaborated. Also practical experiments were conducted with our recently implemented expert system.