Analysis of underlying causes of inter-expert disagreement in retinopathy of prematurity diagnosis. Application of machine learning principles

OBJECTIVE

Inter-expert variability in image-based clinical diagnosis has been demonstrated in many diseases including retinopathy of prematurity (ROP), which is a disease affecting low birth weight infants and is a major cause of childhood blindness. In order to better understand the underlying causes of variability among experts, we propose a method to quantify the variability of expert decisions and analyze the relationship between expert diagnoses and features computed from the images. Identification of these features is relevant for development of computer-based decision support systems and educational systems in ROP, and these methods may be applicable to other diseases where inter-expert variability is observed.

METHODS

The experiments were carried out on a dataset of 34 retinal images, each with diagnoses provided independently by 22 experts. Analysis was performed using concepts of Mutual Information (MI) and Kernel Density Estimation. A large set of structural features (a total of 66) were extracted from retinal images. Feature selection was utilized to identify the most important features that correlated to actual clinical decisions by the 22 study experts. The best three features for each observer were selected by an exhaustive search on all possible feature subsets and considering joint MI as a relevance criterion. We also compared our results with the results of Cohen's Kappa [36] as an inter-rater reliability measure.

RESULTS

The results demonstrate that a group of observers (17 among 22) decide consistently with each other. Mean and second central moment of arteriolar tortuosity is among the reasons of disagreement between this group and the rest of the observers, meaning that the group of experts consider amount of tortuosity as well as the variation of tortuosity in the image.

CONCLUSION

Given a set of image-based features, the proposed analysis method can identify critical image-based features that lead to expert agreement and disagreement in diagnosis of ROP. Although tree-based features and various statistics such as central moment are not popular in the literature, our results suggest that they are important for diagnosis.

OBSERVER AND FEATURE ANALYSIS ON DIAGNOSIS OF RETINOPATHY OF PREMATURITY

Retinopathy of prematurity (ROP) is a disease affecting low-birth weight infants and is a major cause of childhood blindness. However, human diagnoses is often subjective and qualitative. We propose a method to analyze the variability of expert decisions and the relationship between the expert diagnoses and features. The analysis is based on Mutual Information and Kernel Density Estimation on features. The experiments are carried out on a dataset of 34 retinal images diagnosed by 22 experts. The results show that a group of observers decide consistently with each other and there are popular features that have a high correlation with labels.

Physician group practice turnaround. Focus on physician behavior

Physician behavior is an important driver in a group practice turnaround. A strong leadership team, giving physicians ownership, governance and proper incentives, creates the necessary behavior changes. It's also critical to have budget discipline, identify and make the hard decisions, provide effective care management, employ a sound business infrastructure and good contracting, and attend to physician morale.

An evaluation of UMLS as a controlled terminology for the Problem List Toolkit

We are developing a set of software components--the Problem List Toolkit (PL-Tk)--to support operations on clinical problem labels. An adaptation of the National Library of Medicine's Unified Medical Language System (UMLS) provides general vocabulary services to domain-specific software components. Our initial investigation centers on the inclusion in UMLS of problem labels used in the Beth Israel Deaconess Medical Center's Online Medical Record (OMR). We also explore the semantic typing of problem labels matched in UMLS. We have operationally defined a clinical problem to derive its semantic type from classes of terms representing findings or processes typically requiring diagnostic evaluation or therapeutic management in clinical practice. Of 1262 unique OMR problem labels, 999 terms (79%) have matches in UMLS. 986 of 999 terms (99%) map to the UMLS concept of the corresponding lexical match. 952 of 999 terms (95%) have semantic types that comply with our operational definition of clinical problems. These 952 terms (75%) constitute Version 1.0 of the problem list vocabulary B196. Matching terms with inappropriate semantic types raise issues regarding requirements for PL-Tk, typing of existing UMLS terms, and the adequacy of our operational definition for clinical problems. UMLS provides a large repertoire of pre-coordinated terms that are used as problem labels in a heavily used computer-based patient record system. The semantic type hierarchy provides a framework for the consistent use of clinical concepts in problem lists such that clinical problem labels represent "good" clinical problems.

The role of compositionality in standardized problem list generation

Compositionality is the ability of a Vocabulary System to record non-atomic strings. In this manuscript we define the types of composition, which can occur. We will then propose methods for both server based and client-based composition. We will differentiate the terms Pre-Coordination, Post-Coordination, and User-Directed Coordination. A simple grammar for the recording of terms with concept level identification will be presented, with examples from the Unified Medical Language System's (UMLS) Metathesaurus. We present an implementation of a Window's NT based client application and a remote Internet Based Vocabulary Server, which makes use of this method of compositionality. Finally we will suggest a research agenda which we believe is necessary to move forward toward a more complete understanding of compositionality. This work has the promise of paving the way toward a robust and complete Problem List Entry Tool.

Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System

VOCABULARY: The Mayo problem list vocabulary is a clinically derived lexicon created from the entries made to the Mayo Clinic's Master Sheet Index and the problem list entries made to the Impression/ Report/Plan section of the Clinical Notes System over the last three years. The vocabulary was reduced by eliminating repetition including lexical variants, spelling errors, and qualifiers (Administrative or Operational terms). Qualifiers are re-coordinated with other terms, at run-time, which greatly increased the number of input strings which our system is capable of recognizing.

IMPLEMENTATION

The Problem Manager is implemented using standard windows tools in a Windows NT environment. The interface is designed using Object Pascal. HTTP calls are passed over the World Wide Web to a UNIX based vocabulary server. The server returns a document, which is read into Object Pascal structures, parsed, filtered and displayed.

STUDY

This paper reports the results of a recent Usability Trial focused on assessing the viability of this mechanism for standardized problem entry. Eight clinicians engaged in eleven scenarios and responded as to their satisfaction with the systems performance. These responses were observed, videotaped and tabulated. Clinicians in this study were able to find acceptable diagnoses in 91.1% of the scenarios. The response time was acceptable in 92.5% of the scenarios. The presentation of related terms was stated to be useful in at least one scenario by seven of the eight participants. All clinicians wanted to make use of shortcuts which would minimize the amount of typing necessary to encode the concept they were searching for (e.g. Abbreviations, Word Completion).

CONCLUSIONS

Clinicians are willing to choose a canonical term from a suggested list (as opposed to their own wording). Clinicians want an "intelligent" system, which would suggest terms within a category (e.g. Types of "Migraine"). They are able to make functional use of our system, in its current state of development. Finally, all clinicians appreciate the value of encoding their problems in a standardized vocabulary, toward improved research, education and practice.

Treatment of proximal deep-vein thrombosis using subcutaneously administered calcium heparin: comparison with intravenous sodium heparin

In a prospective, randomized clinical trial we compared the efficacy of subcutaneously (SC) administered (every 8 h) calcium heparin to intravenous (IV) sodium heparin in the treatment of proximal deep-vein thrombosis (DVT). A secondary objective was to give enough heparin to achieve a therapeutic anticoagulant effect by the end of the first 24 h. Five of 36 patients (14%) in the SC heparin group failed to achieve a therapeutic anticoagulant effect by the end of the first 24 h compared to 2 of 23 patients (9%) in the IV group (p = NS; 95% CI for true difference = -11.7% to 22.1%). Two of 31 patients (6.5%) in the SC group had venographic evidence of clot propagation compared to 1 of 19 patients (5.3%) in the IV group (p = NS; 95% CI for true difference = -12.4% to 14.8%). The rate of major hemorrhagic complications was similar in each group (approximately 15%). We conclude: (1) using a large initial dose of SC heparin, a therapeutic anticoagulant effect can be readily achieved within 24 h, and (2) combining the results of this trial with previous studies, the efficacy of SC administered calcium appears to be comparable to IV sodium heparin.

Cell-free assembly of a polyoma-like particle from empty capsids and DNA

A polyoma-like particle (PLP) is formed when polyoma DNA and purified empty capsids are incubated in a cell-free system. The DNA of this new particle is protected against the action of pancreatic DNase. The density of the purified PLP in CsCl is 1.32 g/cm3, which is intermediate between that of polyoma virions (1.34 g/cm3) and empty capsids (1.29 g/cm3). Purified PLP sediments at 190 S in sucrose and is stable in solutions of high ionic strength. When the DNA is extracted from PLP by the use of detergent and phenol, it is found to be doublestranded with a molecular weight of approximately 1.1 x 10(6). The particles are stable in CsCl at 4 degrees for at least 5 months. Electron micrographs indicate that highly purified PLPs stained with 2% PTA have the same appearance as polyoma capsids. Neither aggregates nor complexes bound by loose ionic bonds appear reasonable to explain these results. The evidence indicates that the DNA of this new polyoma-like particle, made under cell-free conditions, is protected by the capsid.

Measurement of antibody and antigen concentrations with a sodium sulphate glutaraldehyde technique

Gamma-globulins can be crosslinked at low concentrations by simultaneous treatment with sodium sulphate and glutaraldehyde (SSG-method). This precipitate can be used for the assay of antibody or antigens. Examples are given for the measurement of (1) anti-dinitrophenyl antibody by reaction of the precipitate with 125I-dinitrophenylovalbumin, (2) IgA by inhibition of an anti-IgA precipitate with an 125I-labelled IgA myeloma protein, (3) idiotypic antibody by inhibition of the binding of a 125I-labelled myeloma protein by an anti-idiotype precipitate.

Ir-gene control of antibody class production

The antibody response in mice to DNP-insulin is under Ir-gene control. The Ir gene defects in two strains have been analyzed. In both cases the IgG immune response was impaired whereas the IgM response was not affected. One H-2 gene haplotype was characterized by lack of IgG response, independent of the immunization protocol. A second H-2 haplotype manifested a low response of IgG after immunization with Bordetella pertussis as an adjuvant but a high response after complete Freund's adjuvant. It is proposed that a low level of T cell help induces the production of IgM antibodies, intermediate levels allow few IgG clones to develop, and high levels induce a heterogeneous IgG response.

Ir gene control of carrier recognition II. Unexpected low immunogenicity of guinea pig insulin

The immune response of mice to guinea pig insulin (as well as to insulins from other species) is controlled by immune response genes. The immunogenicity of guinea pig insulin is generally much lower than that of bovine insulin, through there are 14 sequence differences between mouse insulin and guinea pig insulin as compared to 5 sequence differences between mouse insulin and bovine insulin, respectively. The low immunogenicity of guinea pig insulin is indicated by a low frequency of responder strains and the requirement for high antigen doses for immunisation. Guinea pig and bovine insulin do not cross-react at the antibody level, but there seems to be some cross-recognition at the carrier level, as indicated by the capability of guinea pig insulin primed spleen cells to provide help for the pig insulin carrier in cooperative cell transfer.