On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

RATIONALE AND OBJECTIVES

A new methodology that analyzes receiver operating characteristic (ROC) data sets based on jackknifing and that considers both case and reader variability has been proposed. The purpose of this investigation was to compare results using this method to those using commonly reported methodology.

METHODS

ROC data sets using discrete and continuous rating scales were analyzed using the proposed jackknifing method, and results were compared to analysis of the same data sets using the paired t test.

RESULTS

The two methodologies did not result in the same significance levels, and in some cases, the difference was sufficient to affect conclusions regarding comparisons of diagnostic modalities. The probability value for the jackknifing procedure is based on large sample distribution theory, and its appropriateness is unknown for sample sizes used in practice. Also, the jackknifing technique was found to be sensitive to outliers resulting when data from the computer programs used to estimate area under the ROC curve failed to converge.

CONCLUSION

Although the proposed methodology yields reasonable results, several fundamental and practical issues must be addressed before it can be used widely as the analytic method of choice in ROC studies comparing different imaging techniques or reading environments.

Diagnostic reading session: temporal patterns and case-order effects

The authors analyzed receiver-operating-characteristic studies to determine temporal patterns and performance as a function of the elapsed time in a reading session. Nineteen radiologists each read as many as 300 chest images with use of seven different display modalities, including conventional and laser-printed film and high-resolution soft display. With a computerized reporting system, the ratio of observers' interpretation rates (time to diagnosis) were recorded for the last five and 10 compared with the first five and 10 of 30-40 cases seen in sessions lasting 45-110 minutes. Observers tended to accelerate their interpretation as the sessions progressed by an average of 15% (P < .001). The acceleration was consistent for all readers (both fast and slow) with a variety of display modes under the nonrestricted time environment.

Clinical outcome of cardiac patients with negative thallium-201 SPECT and positive rubidium-82 PET myocardial perfusion imaging

In a previous comparison of 202 consecutive patients who underwent myocardial perfusion imaging with both 201Tl SPECT and 82Rb PET, 27 patients were identified as having true-positive 82Rb images, but false-negative 201Tl images. The purpose of this report is to determine the effect of correct image interpretation of coronary artery disease on the final management of those patients and compare it to the previous management scheme wherein a negative image was usually accepted as the end point unless clinical symptoms dictated otherwise. A follow-up study of the clinical course and outcome of these studies showed that 63% (17/27) of the patients with a true-positive 82Rb PET image were recommended for revascularization procedures. It is doubtful that this majority of patients would have received either surgical or interventional management based on the false-negative 201Tl SPECT procedure alone.

Practical issues of experimental ROC analysis. Selection of controls

Receiver operating characteristics (ROC) analysis has been used in many medical imaging applications during the past decade. With the recent expansion of the ROC methodology to multi-disease studies, several reports have begun to address both the theoretical and experimental design issues associated with such studies. While the appropriate selection, classification, and verification of actually positive cases is carefully addressed in the literature, similar considerations are rarely given to the selection of actually negative controls for these studies. In this paper, theoretical considerations and experimental data are provided to demonstrate the significance of this very issue.

A 3' splice site-binding sequence in the catalytic core of a group I intron

Ribozymes use specific RNA-RNA interactions for substrate binding and active-site formation. Self-splicing group I introns have approximately 70 nucleotides constituting the core, a region containing sequences and structures indispensable for catalytic function. The catalytic core must interact with the substrates used for the two steps of the self-splicing reaction, that is, guanosine, the 5'-splice-site helix (P1) and the 3' splice site. Mutational evidence suggests that core sequences near segment J6/7 that joins the base-paired stems P6 and P7, and the bulged base of P7(5'), participate in binding guanosine substrate, but nothing is known about the interactions between the core, the 5'-splice-site helix and the 3' splice site. On the basis of comparative sequence data, it has been suggested that two specific bases in the catalytic core of group I introns might form a binding sequence for the 3' splice site. Here we present genetic evidence that such a binding site exists in the core of the Tetrahymena large subunit ribosomal RNA intron. We demonstrate that this pairing, termed P9.0, is functionally important in the exon ligation step of self-splicing, but is not itself responsible for 3'-splice-site selection.

Effect of two rating formats in multi-disease ROC study of chest images

ROC methodology has been expanded in recent years to include multi-disease experiments. To accommodate these changes, different rating formats, general or disease specific, can be used. No experimental data are available concerning the possible effects of the rating format on the results of these studies. We performed a multi-observer, multi-disease study in which 196 chest images were rated using a format where each disease was evaluated individually and one in which the cases were evaluated without scoring a specific disease. The results indicate that for our data set, the overall assessment of accuracy was not significantly affected by the study format used. Thus, in spite of the difficulties in selecting appropriate controls and the necessity of reassessing sample size considerations, the disease-specific format appears to produce an assessment of accuracy equivalent to that produced by the general format. This equivalence permits the use of the disease-specific approach since it more closely simulates the readers' true environment and is more appropriate for comparing imaging systems that may have a relative accuracy that is disease specific.

Distribution of two basement membrane proteoglycans through hair follicle development and the hair growth cycle in the rat

The distribution of two distinct populations of basement membrane proteoglycans has been monitored through hair growth development in the rat embryo and subsequent hair growth cycle. An antiserum against a small heparan sulfate proteoglycan uniformly stained the dermal-epidermal junction of embryonic rats throughout the period of hair follicle formation. On the other hand, monoclonal antibodies recognizing a basement membrane-specific chondroitin sulfate proteoglycan only weakly stained 16-d embryo dermal-epidermal junction, but strong staining was associated with hair follicle buds as they developed. Through the hair growth cycle, it was found that the heparan sulfate proteoglycan persisted around the follicles, while the chondroitin sulfate proteoglycan decreased in amount through catagen until it was undetectable at the base and dermal papilla of the telogen follicle. As anagen commenced, expression of the chondroitin sulfate proteoglycan was again demonstrated. It therefore appears that a basement membrane-specific proteoglycan shows variation in its distribution in rat skin, expression correlating with morphogenetic activity in hair follicles. It is possible that this newly described basement membrane component is involved in the complex processes of dermal-epidermal interaction that lead to skin appendage formation and growth.

Use of the 82Sr/82Rb generator in clinical PET studies

The use of the 82Sr/82Rb generator in clinical positron emission tomography (PET) studies of myocardial perfusion has been described. An infusion pump is used to deliver the short-lived 82Rb from the generator to the patient. Various characteristics of the generator and the infusion system are described. The 82Rb yield was 69.8 +/- 13.3% and the 82Sr breakthrough was always less than the limit of 0.02 microCi/mCi 82Rb. The yield of 82Rb increased with the flow rate and the potency of the generator. Patients with coronary artery disease were studied for myocardial perfusion abnormalities by the 82Rb PET technique and images of excellent diagnostic quality were obtained.

Evolutionary nucleotide replacements in DNA

With the increasing availability of analytical information on mRNA molecules, it is now possible to compare homologous nucleotide sequences from different organisms and to draw conclusions about their evolution. Such comparisons have shown that silent changes in codons occur more frequently than nucleotide replacements that produce changes in amino acid sequences (code-altering changes). Furthermore, there is an important difference between amino acid sequence comparisons and nucleotide sequence comparisons. The former show only differences in amino acid residues, but the latter show several types of differences when corresponding codons are compared. Single-base replacements may be degenerate (silent) or expressed as amino acid replacements. Two-base codon changes may be degenerate, single-base changes, or be visible as such. Three-base codon changes may be degenerate (involving serine), simulate either single-base or two-base changes or be visible as such. All nine types of change are found in comparisons of genes from the viruses phi X174 and G4. The relative numbers of these nine types as based on all possible interchanges between all 61 amino acid codons were listed by Holmquist et al. and are shown in Table 1. We discuss these results in the light of the significance of nucleotide changes in molecular evolution.

Fixation of a deleterious allele at one of two "duplicate" loci by mutation pressure and random drift

We consider a diploid population and assume two gene loci with two alleles each, A and a at one locus and B and b at the second locus. Mutation from wild-type alleles A and B to deleterious alleles a and b occurs with mutation rates va and vb, respectively. We assume that alleles are completely recessive and that only the double recessive genotype aabb shows a deleterious effect with relative fitness 1-epsilon. Then, it can be shown that if va greater than vb mutant a becomes fixed in the population by mutation pressure and a mutation-selection balance is ultimately attained with respect to the B/b locus alone. The main aim of this paper is to investigate the situation in which va = vb exactly. In this case a neutral equilibrium is attained and either locus can drift to fixation for the mutant allele. Diffusion models are developed to treat the stochastic process involved whereby the deleterious mutant eventually becomes fixed in one of the two duplicated loci by random sampling drift in finite populations. In particular, the equation for the average time until fixation of mutant a or b is derived, and this is solved numerically for some combinations of parameters 4Nev and 4Ne epsilon, where v is the mutation rate (va = vb = v) and Ne is the effective size of the population. Monte Carlo experiments have been performed (using a device termed "pseudo sampling variable") to supplement the numerical analysis.

Unimodality, symmetry and the step-state hypothesis of electrophoretic variation in natural populations

The population frequency distributions of electromorphs of polymorphic loci, when ordered by electrophoretic mobility, tend strongly and significantly to be both unimodal and symmetrical. Such distributions are predicted by all step-change models and their generality in published data can be construed as supportive of the step-change hypothesis. On the other hand, unimodality and symmetry might also be due to artifactual "unit perception" biases that affect the interpretation and reporting of electrophoretic data. In any case, it appears that perceived electromorphs are highly heterogeneous.

Polyallelic mutational equilibria

A new deterministic formulation is derived of the equilibrium between mutation and natural selection, which takes into account(a) the possibility of many allelic mutation states, (b) selection coefficients of the order of magnitude of the mutation rate and (c) the possibility of further mutation of already mutant alleles. The frequencies of classes of alleles 0, 1, 2, n mutant steps removed from the type allele are shown to form a Poisson distribution, with a mean and variance of the mutation rate divided by the coefficient of selection against each incremental mutational step.--This formulation is interpreted in terms of the expected frequencies of electromorphs, defined as classes of alleles characterized by common electrophoretic mobilities of their protein products. Electromorph frequencies are predicted to form stable unimodal distributions of relatively few phenotypic classes. Common electromorph frequencies found throughout the ranges of species with large population sizes are interpreted as being a uniquely electrophoretic phenomenon; band patterns on starch and acrylamide gels are phenotypes, not genotypes. It is predicted that individual electromorphs are highly heterogenous with regard to amino acid sequence.

Isoallele frequencies in very large populations

The frequencies of electrophoretically distinguishable allelic forms of enzymes may be very different from the corresponding frequencies of structurally distinct forms, because many sequence variants may have identical electrophoretic charge. In large populations such frequencies will be determined largely by the number of amino acid sites that are free to vary. The number of distinguishable electrophoretic variants will remain fairly small. Beyond some limiting size, no further effect of population size on allele frequencies is expected, so isolated large populations will have closely similar allele frequencies if polymorphism is due largely to mutation and drift. The most common electrophoretic alleles are expected to be flanked by the next most common, with the rarer alleles increasingly distal. Neither strong selection nor mutation/drift interpretations of enzyme polymorphism are yet disproven, nor is any point between these extremes.