An accurate and precise methodology for routine determination of the false-negative rate of Papanicolaou smear screening

Frequency and outcome of cervical cancer prevention failures in the United States

We measured the frequency and outcome of cervical cancer prevention failures that occurred in the Papanicolaou (Pap) and colposcopy testing phases involving 1,646,580 Pap tests in 4 American hospital systems between January 1, 1998, and December 31, 2004. We defined a screening failure as a 2-step or greater discordant Pap test result and follow-up biopsy diagnosis. A total of 5,278 failures were detected (0.321% of all Pap tests); 48% and 52% of failures occurred in the Pap test and colposcopy phases, respectively. Missed squamous cancers (1 in 187,786 Pap tests), glandular cancers (1 in 19,426 Pap tests), and high-grade lesions (1 in 6,870 Pap tests) constituted 4.1% of all failures. Unnecessary repeated tests or diagnostic delays occurred in 70.8% and 63.9% of failures involving high- and low-grade lesions, respectively. We conclude that cervical cancer prevention practices are remarkably successful in preventing squamous cancers, although a high frequency of failures results in low-impact negative outcomes.

Comparing methods to measure error in gynecologic cytology and surgical pathology

CONTEXT

Both gynecologic cytology and surgical pathology use similar methods to measure diagnostic error, but differences exist between how these methods have been applied in the 2 fields.

OBJECTIVE

To compare the application of methods of error detection in gynecologic cytology and surgical pathology.

DATA SOURCES

Review of the literature.

CONCLUSIONS

There are several different approaches to measuring error, all of which have limitations. Measuring error using reproducibility as the gold standard is a common method to determine error. While error rates in gynecologic cytology are well characterized and methods for objectively assessing error in the legal setting have been developed, meaningful methods to measure error rates in clinical practice are not commonly used and little is known about the error rates in this setting. In contrast, in surgical pathology the error rates are not as well characterized and methods for assessing error in the legal setting are not as well defined, but methods to measure error in actual clinical practice have been characterized and preliminary data from these methods are now available concerning the error rates in this setting.

Rescreening in cervical cytology for quality control. When bad data is worse than no data or what works, what doesn't, and why

The currently mandated methods to measure the sensitivity of Papanicolaou smear interpretation (including the 5-year look back and review of 10% of negative smears) are misleading. They do not allow one to measure the true sensitivity of the test and allow only a small fraction of errors to be detected and corrected. Rapid prescreening and automated screening are the only methods that seem practically feasible, and will allow the sensitivity of the method to be measured on a routine basis, and, thereby allow a reduction in overall errors. Although emerging technologies, such as HPV testing, may allow alternative methods of measuring sensitivity, the most appropriate way to use and interpret these data in this context are not yet fully developed. Unfortunately, at present there seems to be little desire to change the way things are done, and, therefore, to accurately measure sensitivity in cervical cytology. The first task that needs to be undertaken in fixing a problem is to admit that one exists. At present, most laboratory directors believe that their own laboratory is performing satisfactorily. They may well be, although the laboratories lack an analytic method to demonstrate this, and, therefore, the need for better QA methods does not seem to be acute. There is some educational value to the currently practiced and mandated performance measures, the 5-year look back and review of 10% of negative smears. Most laboratory directors seem to be happy with their QA methods and are not concerned that the data that are derived from the 10% review of negative slides does not reflect their actual sensitivity of screening in any meaningful way. Unfortunately, the forces that are currently in place in the United States ensure that accurate measures of the sensitivity of cervical cytology interpretation are unlikely to be implemented beyond the level of individual experimentation. As long as the expectation of cytologists is that the error rates are significantly less than they actually are; as long as there is significant legal and financial risk to actually measuring the true sensitivity; and as long as fictitious measures of performance are not only advocated but mandated; the confluence of incentives will ensure that the true sensitivity of the test will never be measured on a routine basis. Despite all of this, it is possible that cervical cytology screening may, in fact, already be performing at an optimal level. Being able to measure this operating performance may not effect any improvement to the overall process. The ultimate arbiter in this debate will always be the demonstration of a reduction of cervical cancer morbidity and mortality with any new measure implemented. At the present time, there is only one solution to the quality control issue; a force from outside the system must change the balance of the aforementioned incentives. The promise of data from European experiences with rapid rescreening may show that this method is effective and accurate. Such data might make the current methods that are in use in the United States more open to change. So yes, the answer is that "bad" data may be worse than no data at all. The bad data that we have been collecting for more than a decade is as effective a trap as anyone could have devised to ensure that actually measuring the performance of cervical smear interpretation does not happen. The only question that remains is, "How we will be able to escape?"

Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology

CONTEXT

Numerous studies address the accuracy or positive predictive value of cytologic interpretations for defined histopathologic entities. The reproducibility (precision) of cytologic interpretation is less well defined.

OBJECTIVE

To establish and compare the reproducibility (precision) of cytologic interpretation in gynecologic cytopathology, as reflected in the educational program of the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology (PAP).

METHODS

The pathologists' interpretations for both validated (25 745 responses) and educational conventional (14 353 responses) slides in the PAP program for 2001 were analyzed. The frequency of exact matches between the reference and pathologists' interpretation for each of the cytologic interpretative categories was identified, and the cumulative distributions of exact match rates were derived. chi2 Tests by reference interpretations were used for cytodiagnostic categories, least and most reproducible groupings, and high-grade (HSIL) versus low-grade (LSIL) squamous intraepithelial lesions.

RESULTS

Pathologists' interpretations of negative, Candida, Trichomonas, herpes, and LSIL were characterized by a high degree of exact matching, while interpretations of repair, HSIL, adenocarcinoma, and squamous cell carcinomas were characterized by a lesser degree of exact matching (reproducibility). Pathologists' cytologic interpretations of HSIL were significantly less reproducible than those of LSIL.

CONCLUSION

The cytologic interpretations of the most significant categories (HSIL, squamous cell carcinoma, and adenocarcinoma) are less precise than those of specific infection (Candida, Trichomonas, and yeast), negative, and LSIL categories. Cytologic interpretations of LSIL are made with greater precision than those of HSIL and may represent a more appropriate endpoint to measure the precision performance of gynecologic cytology laboratories.

Measuring sensitivity in gynecologic cytology: a review

BACKGROUND

The sensitivity of gynecologic cytology has been measured in several different ways. The current review summarizes the major sources of bias and the results of these efforts.

METHODS

In the current study, a review of the literature was conducted.

RESULTS

The major sources of bias in measuring the sensitivity of gynecologic cytology are a lack of reproducibility, bias in the review process, bias in case selection, selection and correction of the gold standard, and the value of surrogate markers. Despite these sources of variation, the sensitivity of the Papanicolaou (Pap) smear is relatively stable, ranging from 50-75% if a single consistent threshold is used, to up to 94% if either Autopap-directed rescreening or thin-layer methods are used to diagnose high-grade squamous intraepithelial lesions using a threshold of atypical squamous cells of undetermined significance. Methods for the routine evaluation of sensitivity currently are not available and may be difficult to devise.

CONCLUSIONS

The sensitivity of the Pap smear in study situations is well known. Whether these reflect performance in real life is not known, and methods to compare performance in real life are not available.

Quantifying the value of in-house consultation in surgical pathology

In-house consultation is a well-known method to improve diagnostic accuracy and agreement, but the technique has not been well studied. We reviewed the results of in-house consultation in a large private hospital practice setting for a 1-month period and determined its effect on diagnostic accuracy using the final sign-out as the "gold standard." During this 1-month period, 352 cases were reviewed as in-house consultations. Initial complete agreement was found in 315 (89.5%) cases. Using the initial diagnosis as the test case and the final sign-out as the gold standard, of the 37 discrepant cases, 4 (1.1%) were thought to represent false-negative results, (0.3%) a false-positive result, 3 (0.9%) differences in type, and 29 (8.2%) differences in diagnostic threshold. Disagreements in 10 cases were thought to be potentially clinically significant. Internal consultation was obtained on approximately 20% of all cases seen in the laboratory and disagreements were found in 2% of all cases. Internal consultation has a significant and measurable impact on the practice of surgical pathology.