The human false-negative rate of rescreening Pap tests. Measured in a two-arm prospective clinical trial

Histological coherent Raman imaging: a prognostic review

Histopathology plays a central role in diagnosis of many diseases including solid cancers. Efforts are underway to transform this subjective art to an objective and quantitative science. Coherent Raman imaging (CRI), a label-free imaging modality with sub-cellular spatial resolution and molecule-specific contrast possesses characteristics which could support the qualitative-to-quantitative transition of histopathology. In this work we briefly survey major themes related to modernization of histopathology, review applications of CRI to histopathology and, finally, discuss potential roles for CRI in the transformation of histopathology that is already underway.

Performance of rapid prescreening and 100% rapid review as internal quality control methods for cervical cytopathology

BACKGROUND

An objective of quality control for cervical cytopathology is reducing high rates of false-negative results of laboratory tests. Therefore, methods to review smears such as rapid prescreening and 100% rapid review, which have shown better performance detecting false-negative results, have been widely used. The performance of rapid prescreening and the performance of 100% rapid review as internal quality control methods for cervical cytology examinations were evaluated.

METHODS

For 24 months, 9318 conventional cervical cytology smears underwent rapid prescreening and routine screening. The 100% rapid review method was performed for 8244 smears classified as negative during routine screening. Any discordant results underwent detailed review to define the final diagnosis. This was considered the gold standard for evaluating the performance of rapid prescreening and 100% rapid review.

RESULTS

Routine screening showed increases of 13.3% and 11.5% in the detection of abnormal smears with rapid prescreening and 100% rapid review, respectively. The relative percentage variation showed a 38.1% increase in the diagnosis of atypical squamous cells of undetermined significance with routine screening and rapid prescreening and a 12.5% increase in the diagnosis of atypical squamous cells, cannot exclude high-grade squamous intraepithelial lesion with both rapid prescreening and 100% rapid review. Sensitivity rates of rapid prescreening and routine screening were 48.2% and 83.2%, respectively. Sensitivity rates of rapid prescreening and 100% rapid review were 65.7% and 57.8%, respectively, for detecting false-negative results.

CONCLUSIONS

Inclusion of rapid prescreening and/or 100% rapid review improved the diagnostic sensitivity of the cervical cytology examination and reduced false-negative results of routine screening and can provide good quality control.

A Multi-Institutional Feasibility Study on the Use of Automated Screening Systems for Quality Control Rescreening of Cervical Cytology

OBJECTIVE

To evaluate the efficacy of the automated screening system FocalPoint for cervical cytology quality control (QC) rescreening. False-negative rates (FNRs) were evaluated by a multi-institutional retrospective study.

STUDY DESIGN

Cervical cytology slides that had already been reported as negative for intraepithelial lesion or malignancy (NILM) were chosen arbitrarily for FocalPoint rescreening. Slides stratified into the highest 15% probability of being abnormal were rescreened by a cytotechnologist. The slides that were abnormal were reevaluated by a cytopathologist to be false negatives.

RESULTS

Rescreening of 12,000 slides, i.e. 9,000 conventional slides and 3,000 liquid-based cytology (LBC) slides, was performed; 9,826 (7,393 conventional and 2,433 LBC) were satisfactory for FocalPoint (2,174 were determined unsatisfactory) and those within the highest 15% of probability (1,496, i.e. 1,123 conventional and 373 LBC) were rescreened. As a result, 117 (96 conventional and 21 LBC) were determined as abnormal (other than NILM) and the FNR was 1.19%. Among these 117 slides, 40 (35 conventional and 5 LBC) were determined as high-grade squamous intraepithelial lesion and greater (HSIL+).

CONCLUSION

Of 117 (1.19%) abnormal slides detected, 40 (0.41%) were determined to be HSIL+. This result suggests that FocalPoint is effective for QC rescreening of cervical cytology.

Fourier-ring descriptor to characterize rare circulating cells from images generated using immunofluorescence microscopy

We address the problem of subclassification of rare circulating cells using data driven feature selection from images of candidate circulating tumor cells from patients diagnosed with breast, prostate, or lung cancer. We determine a set of low level features which can differentiate among candidate cell types. We have implemented an image representation based on concentric Fourier rings (FRDs) which allow us to exploit size variations and morphological differences among cells while being rotationally invariant. We discuss potential clinical use in the context of treatment monitoring for cancer patients with metastatic disease.

Changing the cytology laboratory information system to improve cytology performance

A central tenet in the Patient Protection and Affordable Care Act is the increased use of information technology to improve patient care. However, areas for improvement in cytology are not well defined. Improvements in information technology could improve quality assessment in gynecologic cytology, but the cytology community must identify and ask for changes in information technology that can improve the care of patients. Cancer (Cancer Cytopathol) 2014;122:87-91. © 2013 American Cancer Society.

Prospective and retrospective review of gynecologic cytopathology: findings from the College of American Pathologists Gynecologic Cytopathology Quality Consensus Conference working group 2

CONTEXT

Two quality metrics for gynecologic cytology are the subject of this review: "prospective rescreening" and "retrospective rescreening."

OBJECTIVE

To offer consensus best practice approaches based on the College of American Pathologists' laboratory-based survey funded by the Centers for Disease Control and Prevention.

DESIGN

The College of American Pathologists submitted a paper-based survey to 1245 laboratories. After review of initial results, follow-up Web-based survey results, and a literature review, consensus best practice statements were presented at a national consensus conference. These statements were discussed and voted upon by conference participants. Results.-A total of 541 laboratories responded to survey questions about prospective and retrospective rescreening. Most laboratories (>85%) prospectively rescreen more than 10% of Pap tests interpreted as negative for intraepithelial lesion or malignancy. Most (72%) report inclusion of less than 20% high-risk cases. Most laboratories use multiple measures to define "high risk." Most laboratories (96.2%) retrospectively rescreen Pap tests from the preceding 5 years only. In most laboratories (71.4%) only Pap test results with high-grade squamous intraepithelial lesion or worse prompt retrospective review.

CONCLUSIONS

The number of Pap tests from high-risk patients should be maximized in prospective and retrospective rescreening. Unsatisfactory Pap tests should also be included. All readily identifiable high-risk human papillomavirus-positive cases with an interpretation of negative for intraepithelial lesion or malignancy should be prospectively rescreened. Cervical biopsy results with high-grade cervical intraepithelial neoplasia or worse (CIN 2+) should trigger retrospective rescreening. Regular feedback should be provided to cytotechnologists and cytopathologists. Upgraded diagnoses from negative for intraepithelial lesion or malignancy to atypical squamous cells, cannot exclude high-grade squamous intraepithelial lesion, should be monitored.

Effectiveness of rapid prescreening and 10% rescreening in liquid-based Papanicolaou testing

Although rapid prescreening (RPS) has been shown to be an effective quality control procedure for detecting false-negative conventional Papanicolaou (Pap) tests, RPS has not been widely implemented in the United States. In our laboratory, cytotechnologists performed RPS in 3,567 liquid-based Pap tests: 1,911 SurePath (BD Diagnostics-TriPath, Burlington, NC) preparations that were manually screened and 1,656 ThinPrep Pap tests (Hologic, Bedford, MA) that were imaged using the ThinPrep Imaging System (Hologic). We compared the sensitivity of RPS, 10% rescreening (R-10%), and routine screening (RS). In contrast with previously published findings, we found that RS + RPS did not improve screening sensitivity compared with RS + R-10%. These results support the following hypotheses: (1) Higher baseline RS sensitivity as a result of Pap test diagnoses standardization implemented for quality improvement purposes decreases the performance impact of RPS. (2) R-10% and RPS quality assurance methods detect diagnostic failures caused by different types of cognitive errors.

Quality Improvement in Cytology: Where Do We Go From Here?

Context.—Cytology is a success because of the many quality controls used to ensure the accuracy of its results. Nevertheless, additional information is becoming available to the cytologist, often from untraditional sources, and the best way to use that information to improve the quality of cytology is not yet known.

Objective.—To review ways to use new information to improve the quality of cytology.

Data Sources.—Review of relevant literature.

Results.—Information contained in many sources can be used in new ways to improve the quality of cytology. These include the timing of cytologic and histologic correlation, electronic medical records, workload information, prior aspirations, and molecular tests.

Conclusions.—To maintain their high standard of excellence, cytologists should seek to define the most appropriate way to incorporate this new information into their interpretation of individual cases.

Does the time of day or weekday affect screening accuracy? A pilot correlation study with cytotechnologist workload and abnormal rate detection using the ThinPrep Imaging System

BACKGROUND

Workload is extensively regulated in gynecologic cytology. However, sensitive monitors of excessive workload are not available.

METHODS

We measured the variation in abnormal (atypical squamous cells [ASC], low-grade squamous intraepithelial lesion [LSIL], and high-grade squamous intraepithelial lesion [HSIL]) rates for 4 cytotechnologists (CTs) among different days of the week and at different times during the day while they were performing primary screening with the ThinPrep Imaging System.

RESULTS

Three of 4 CTs detected significantly less abnormal cases on 1 day of the week than another (1 Monday, 2 Friday). Two of those CTs detected significantly fewer total abnormal cases in the afternoon than in the morning; this was strongly correlated with increased speed in the afternoon and decreased detection of ASC cases. HPV + rates for ASC cases dropped as the abnormal rate dropped. The third CT detected significantly fewer ASC cases in the morning; this was counterbalanced by an increase detection of LSIL cases, suggesting a shift in diagnostic threshold between the AM and PM. The difference in abnormal detection rates between morning and afternoon correlated with a false-negative fraction of 0.96.

CONCLUSIONS

There are significant differences in detection rates of abnormal cases between days of the week and the morning and afternoon. Correlating abnormal rates and workload between the morning and afternoon may represent a sensitive way to detect excessive workload. Because individual CTs may have different responses to workload and no overall pattern emerged, data on their workload and performance need to be tracked individually.

The Becton Dickinson FocalPoint GS Imaging System: clinical trials demonstrate significantly improved sensitivity for the detection of important cervical lesions

Location-guided screening in cervical cytology offers a potentially significant advance over routine manual screening. A prospective, 2-armed, masked clinical trial of the BD FocalPoint GS Imaging System using SurePath slides (BD Diagnostics-TriPath, Burlington, NC) compared routine manual screening and quality control rescreening with computer-assisted, field-of-view screening and device-directed quality control rescreening. The results obtained in the 2 arms were compared with adjudicated reference diagnoses for each slide. Sensitivity, specificity, and negative predictive value were calculated for the detection of atypical squamous cells of undetermined significance and greater (ASC-US+), low-grade squamous intraepithelial lesion and greater (LSIL+), and high-grade squamous intraepithelial lesion and greater (HSIL+) groups. We evaluated 12,313 slides. The detection sensitivities for HSIL+ were increased by 19.6% (P < .0001) and for LSIL+ were increased by 9.8% (P < .0001) in the computer-assisted arm, with small statistically significant decreases in specificity. For ASC-US+ sensitivity and specificity, the study arms were not statistically different. Use of this system might be expected to improve accuracy for clinically important entities without increasing equivocal case detection.

Surrogate indicators of sensitivity in gynecologic cytology: can they be used to improve the measurement of sensitivity in the laboratory?

Background:

Measuring the sensitivity of screening in gynecologic cytology in real life is problematic. However, other quality measures may correlate with sensitivity, including the atypical squamous cells (ASC)/squamous intraepithelial lesion (SIL) ratio. Whether these other measures can function as “surrogate indicators” for sensitivity and improve the assessment of sensitivity in the laboratory is not known.

Materials and Methods:

We compared multiple quality measures with true screening sensitivity in a variety of situations.

Results:

The abnormal rate, ASC rate, and ASC/SIL ratio were all highly correlated (r =.83 or greater) with sensitivity when the overall laboratory sensitivity was low (85%) but became less correlated (.64 or less) or uncorrelated when the screening sensitivity was higher (88% or 95%, respectively). Sensitivity was more highly correlated with the abnormal rate than the ASC/SIL ratio at low screening sensitivity. While thresholds could be set that were highly sensitive and specific for suboptimal screening, these thresholds were often less than one standard deviation away from the mean.

Conclusion:

The correlation of the abnormal rate and the ASC/SIL ratio with sensitivity depends on overall sensitivity. Standards to define minimum screening sensitivity can be defined, but these standards are relatively narrow. These features may limit the utility of these quality measures as surrogates for sensitivity.

Measuring the significance of workload on performance of cytotechnologists in gynecologic cytology: a study using rapid prescreening

BACKGROUND

Workload is extensively regulated and often used as a measure of quality in gynecologic cytology. Whether workload correlates with the sensitivity of screening in gynecologic cytology is not known.

METHODS

The sensitivity of gynecologic cytology screening was measured over an 8-month period using the result of full screening coupled with the results of rapid prescreening. Sensitivity results were then correlated with daily workload volumes and the experience level of individual cytotechnologists.

RESULTS

Rapid prescreening had an average sensitivity of 41.9% for atypical squamous cells of undetermined significance (ASCUS) and above. Full screening had a corrected sensitivity of 82.2% for ASCUS and above. Rapid prescreening increased the sensitivity of the laboratory to 89.9%. The sensitivity of full screening was significantly different between cytotechnologists (79.2% vs 99%, P < .001), but was not correlated with years of experience, sensitivity of rapid prescreening, or workload (all P > .05). When sensitivity and workload were examined on a monthly basis, there was no significant difference between sensitivity either as a group or individually at the highest and lowest workloads (P > .40 for all).

CONCLUSIONS

Screeners sensitivity in gynecologic cytology appears to be unrelated to the experience level of individual cytotechnologists or to their workload at the levels examined.

Longitudinal clinical evaluation of a novel antibody cocktail for detection of high-grade squamous intraepithelial lesions on cervical cytology specimens

CONTEXT

Although cervical cancer screening by cytology is successful, the test continues to show relatively poor operating characteristics. Cell cycle markers may enhance detection of high-grade squamous intraepithelial lesions.

OBJECTIVE

To determine the clinical usefulness of ProExC, an immunocytochemical assay for cell cycle components, performed on routine cervical cytology samples.

DESIGN

Cervical cytology samples were collected using the SurePath method. Residual cells remaining after preparation of the Papanicolaou-stained slide were used to make a second slide for ProExC staining using an indirect polymer-based immunoperoxidase method. Only adequately cellular slides were evaluated for the presence of nuclear staining within cytologically abnormal epithelial cells. Results were correlated with clinical follow-up.

RESULTS

Six hundred twenty-four samples were satisfactorily cellular and stained. Correlation with clinical follow-up for subsequent cervical intraepithelial neoplasia 2+ on biopsy/high-grade squamous intraepithelial lesion on cytology (CIN 2+/HSIL) showed that 434 results were true negative, 78 true positive, 18 false-negative, and 94 false-positive, resulting in a sensitivity/specificity of 81%/82%. When ProExC results were combined with any level of cytologic atypia, sensitivity for CIN 2+/HSIL was 92% and specificity was 84%.

CONCLUSIONS

ProExC shows promise as an aid in enhancing the sensitivity and specificity of cervical cytology for subsequent CIN 2+/HSIL and may be useful in identifying those cervical lesions most apt to progress.

The dysfunctional federally mandated proficiency test in cytopathology: a statistical analysis

Proficiency testing in cytopathology and in other disciplines should be based on firm statistical and scientific foundations, because test theory in general is a heavily statistical subject. Statistical considerations have demonstrated that the design of "short" proficiency tests in cytopathology, including the current federally mandated test, fundamentally is unsound because of the lack of sufficient validity and reliability. Examinees too frequently are misclassified by such short-format tests: Competent examinees fail the test in surprisingly high numbers, whereas most of the examinees who have insufficient cytologic skills eventually pass the test after the allowed retakes. Only dichotomous tests are suitable for accurate computation of the effects of test design on reliability, but the statistical conclusions also are generalizable to nondichotomous tests. In conclusion, the current federally mandated proficiency test cannot reliably measure the level of expertise of cytologists and, thus, cannot assure that only adequately skilled individuals evaluate Papanicolaou test samples. To render the test suitable for its intended purpose, the authors believe that complete redesign of the test, with the participation of experts in modern test theory, would be advisable.

Agreement among surgical pathologists evaluating routine histologic sections of digits amputated from cats and dogs

Agreement among pathologists interpreting histologic specimens is an area of interest within human pathology, but little work in this area has been reported in the veterinary literature. Agreement among pathologists evaluating routine histologic sections of amputated digits from cats and dogs submitted to multiple diagnostic centers was examined. Histologic sections from surgical specimens were reviewed in a blinded fashion by two pathologists, and a comparison to the original diagnosis, as stated in the diagnostic report, was recorded. A total of 513 cases were reviewed, and complete agreement was reached in 409 (79.7%). Of the 104 instances of disagreement, 77 (74.0%) were considered to be of clinical significance. The diagnosis of keratoacanthoma was disagreed with in 19 of 21 diagnoses (90.4%). No other individual diagnosis was similarly disputed. The overall level of disagreement is large and is similar to that reported in human pathology and suggests that further study of this issue would be useful in veterinary pathology.

Comparing methods to measure error in gynecologic cytology and surgical pathology

CONTEXT

Both gynecologic cytology and surgical pathology use similar methods to measure diagnostic error, but differences exist between how these methods have been applied in the 2 fields.

OBJECTIVE

To compare the application of methods of error detection in gynecologic cytology and surgical pathology.

DATA SOURCES

Review of the literature.

CONCLUSIONS

There are several different approaches to measuring error, all of which have limitations. Measuring error using reproducibility as the gold standard is a common method to determine error. While error rates in gynecologic cytology are well characterized and methods for objectively assessing error in the legal setting have been developed, meaningful methods to measure error rates in clinical practice are not commonly used and little is known about the error rates in this setting. In contrast, in surgical pathology the error rates are not as well characterized and methods for assessing error in the legal setting are not as well defined, but methods to measure error in actual clinical practice have been characterized and preliminary data from these methods are now available concerning the error rates in this setting.

Use of whole slide imaging in surgical pathology quality assurance: design and pilot validation studies

By imaging large numbers of slides automatically at high resolution, modem automated whole slide imaging (WSI) systems have the potential to become useful tools in pathology practice. This article describes a pilot validation study for use of automated high-speed WSI systems for surgical pathology quality assurance (QA). This was a retrospective comparative study in which 24 full genitourinary cases (including 47 surgical parts and 391 slides) were independently reviewed with traditional microscopy and whole slide digital images. Approximately half the cases had neoplasia in the diagnostic line. At the end of the study, diagnostic discrepancies were evaluated by a pathology consensus committee. The study pathologists felt that the traditional and WSI methods were comparable for case review. They reported no difference in perceived case complexity or diagnostic confidence between the methods. There were 4 clinically insignificant discrepancies with the signed-out cases: 2 from glass slide and 2 with WSI review. Of the 2 discrepancies reported by the WSI method, the committee agreed with the reviewer once and the original report once. At the end of the study, the participants agreed that automated WSI is a viable potential modality for surgical pathology QA, especially in multifacility health systems that would like to establish interfacility QA. The participants felt that major issues limiting the implementation of WSI-based QA did not involve image acquisition or quality but rather image management issues such as the pathologist's interface, the hospital's network, and integration with the laboratory information system.

Results of multiple-slide, blinded review of Papanicolaou slides in the context of litigation. Determining what can be detected regularly and reliably

BACKGROUND

Multiple-slide, blinded review has been endorsed by several cytology and pathology organizations as the most appropriate method for the review of cervicovaginal specimens in the context of litigation. This process involves review of litigation slides in a blinded manner by multiple independent cytotechnologists and the comparison of those results with those of validation cases that are comprised of known abnormalities with biopsy follow-up. To the authors' knowledge, the results of this method have not been previously published.

METHODS

The results of the blinded review program at the Center for Cytopathology and Molecular Research at the Medical University of South Carolina for the years 1998-2004 were reviewed.

RESULTS

A total of 135 litigation slides and 122 validation slides were reviewed. The interpretations of these cases were found to be significantly different (P < 0.001). Litigation cases were significantly more likely to be interpreted as either negative for intraepithelial lesion (NIL) or atypical squamous cells/atypical glandular cells (ASC/AGC) (P < 0.001). The results appeared to be independent of the individual cytotechnologists involved. Approximately 10% of litigation cases were called at least ASC/AGC by all observers and 4% were interpreted as NIL by all observers. For litigation cases, ASC/AGC was found to be just as reproducible as high-grade squamous intraepithelial lesion. The results demonstrated that only 10% of litigation cases are regularly and reliably identified as abnormal, whereas a single review as performed by an expert cytologist can be expected to classify 56% of cases as abnormal.

CONCLUSIONS

This program suggests that a majority of litigation cases are not regularly and reliably identified as abnormal, and a single review will routinely overestimate the percentage of cases that are identified regularly and reliably.

Rapid prescreening of papanicolaou smears

BACKGROUND

Efficient quality control (QC) is essential to ensure high sensitivity of Papanicolaou (Pap) smears. For this purpose, rescreening of 10% random negative smears is ineffective. Rapid rescreening (RR) of all negative Pap smears is more practical and has received widespread acceptance, especially in Europe, although its sensitivity is difficult to monitor and its retrospective nature may influence the vigilance of the screeners. The method of rapid prescreening (RPS) overcomes these drawbacks because rapid review of Pap smears occurs before routine full screening.

METHODS

All routine conventional Pap smears over 2 months underwent RPS by 12 cytotechnologists. Approximately 30 seconds were allowed to prescreen each slide. The presence of abnormal cells (atypical squamous cells of undetermined significance [ASCUS] or above), infection or endometrial cells detected on RPS was documented. All slides subsequently underwent routine full screening. Results of both screening methods were compared.

RESULTS

Of a total of 8364 Pap smears, 310 (3.7%) cases were categorized as abnormal after final diagnosis. Of those, 135 were also detected on RPS (sensitivity of 43.5%). Seventeen abnormal cases were detected only on RPS: these consisted of 13 ASCUS cases, 3 low-grade squamous intraepithelial lesions, and 1 high-grade squamous intraepithelial lesion. The sensitivity of RPS for infections and endometrial cells was 51.6% and 28.3%, respectively. Implementation of RPS did not significantly impact the work flow in our laboratory.

CONCLUSIONS

RPS is an efficient and practical QC tool. It is a reliable method with which to monitor sensitivity and reduce the false-negative rate, and because it is done before finalizing the case, it allows for timely corrections to the diagnosis and avoids the need to amend reports.

Measuring the Significance of Field Validation in the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology: How Good Are the Experts?

Context.—Expert opinion is often used as a gold standard for gynecologic cytology in the evaluation of new technologies, in the legal setting, and in the validation of cases for use in educational programs and proficiency testing. However, the reliability of expert opinion alone in selecting slides of a specific cytodiagnosis that can be reproducibly and reliably identified by subsequent reviewers has not been determined.

Objective.—To assess the ability of expert opinion to select slides that are validated in subsequent reviews.

Design.—In the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology, each case in every cytodiagnostic category is accepted for circulation only after review by 3 expert cytopathologists. The percentage of these cases that could not be reliably and reproducibly identified by program participants for each cytodiagnostic category (“failed field validation”) was determined during the duration of the program from 1989 to 2004.

Results.—More than 10 000 conventional smears and ThinPrep cases were selected by the expert panel for circulation. Of these selected slides, 19% of conventional smears and 15% of ThinPrep specimens failed field validation. Compared with the overall slide performance, significantly higher percentages (P &lt; .001) of conventional smears with reference diagnoses of unsatisfactory (51.7%), repair (58%), or low-grade intraepithelial lesion (31.8%) and of ThinPrep specimens with reference diagnoses of unsatisfactory (54.5%) and repair (76.9%) failed field validation. In contrast, significantly lower percentages of conventional smears with reference diagnoses of squamous cell carcinoma (4.5%), high-grade squamous intraepithelial lesion (9%), Trichomonas vaginalis infection (11.7%), or herpes (9.9%) and of ThinPrep specimens with reference diagnoses of adenocarcinoma (5.1%), herpes (2.1%), and fungal organism consistent with Candida (8.4%) failed field validation (P &lt; .001 for all).

Conclusions.—Between 15% and 19% of gynecologic cytologic cases that have been selected by expert cytopathologists as good examples of cytodiagnostic abnormalities fail field validation. The proportion of cases failing field validation varies with cytodiagnostic category, but it occurs in all cytodiagnostic entities.

Blinded review of Papanicolaou smears in the context of litigation

BACKGROUND

Blinded review has been endorsed by several cytology and pathology organizations as the most appropriate method for the review of cervicovaginal specimens in the context of litigation. Methods for determining the statistical validity of this method were evaluated.

METHODS

First, the authors calculated the sample size needed to detect various differences in case difficulty or ease of interpretation, in which ease of interpretation is defined as the percentage of the time a case could be identified as abnormal by routine screening. Very easy cases could be identified most reliably, whereas more difficult cases were detected less regularly and less reliably. Using construct sample sizes, the authors calculated the number of abnormal reviews that may be helpful to conclude that the case's difficulty or ease of interpretation was statistically significantly different from another case of a specified difficulty. Finally, they examined the effect of using two separate cutoff parameters to make these distinctions.

RESULTS

Depending on the threshold chosen, it was determined that improvements in the statistically meaningful distinctions may be made using 15 or 20 reviews. To distinguish between routine false-negative cases (ease of detection, 40%) and routine cases (ease of detection, 80%), the thresholds may be set at 5 of 10 reviews (a case that would not be detected regularly and reliably in any given laboratory) and 7 of 10 reviews (defining a case that would be identified regularly and reliably), respectively.

CONCLUSIONS

The authors provide data that can be used to interpret the results of a blinded review in a statistically appropriate manner. To improve the utility of blinded reviews, the standards are defined explicitly.

Rescreening in cervical cytology for quality control. When bad data is worse than no data or what works, what doesn't, and why

The currently mandated methods to measure the sensitivity of Papanicolaou smear interpretation (including the 5-year look back and review of 10% of negative smears) are misleading. They do not allow one to measure the true sensitivity of the test and allow only a small fraction of errors to be detected and corrected. Rapid prescreening and automated screening are the only methods that seem practically feasible, and will allow the sensitivity of the method to be measured on a routine basis, and, thereby allow a reduction in overall errors. Although emerging technologies, such as HPV testing, may allow alternative methods of measuring sensitivity, the most appropriate way to use and interpret these data in this context are not yet fully developed. Unfortunately, at present there seems to be little desire to change the way things are done, and, therefore, to accurately measure sensitivity in cervical cytology. The first task that needs to be undertaken in fixing a problem is to admit that one exists. At present, most laboratory directors believe that their own laboratory is performing satisfactorily. They may well be, although the laboratories lack an analytic method to demonstrate this, and, therefore, the need for better QA methods does not seem to be acute. There is some educational value to the currently practiced and mandated performance measures, the 5-year look back and review of 10% of negative smears. Most laboratory directors seem to be happy with their QA methods and are not concerned that the data that are derived from the 10% review of negative slides does not reflect their actual sensitivity of screening in any meaningful way. Unfortunately, the forces that are currently in place in the United States ensure that accurate measures of the sensitivity of cervical cytology interpretation are unlikely to be implemented beyond the level of individual experimentation. As long as the expectation of cytologists is that the error rates are significantly less than they actually are; as long as there is significant legal and financial risk to actually measuring the true sensitivity; and as long as fictitious measures of performance are not only advocated but mandated; the confluence of incentives will ensure that the true sensitivity of the test will never be measured on a routine basis. Despite all of this, it is possible that cervical cytology screening may, in fact, already be performing at an optimal level. Being able to measure this operating performance may not effect any improvement to the overall process. The ultimate arbiter in this debate will always be the demonstration of a reduction of cervical cancer morbidity and mortality with any new measure implemented. At the present time, there is only one solution to the quality control issue; a force from outside the system must change the balance of the aforementioned incentives. The promise of data from European experiences with rapid rescreening may show that this method is effective and accurate. Such data might make the current methods that are in use in the United States more open to change. So yes, the answer is that "bad" data may be worse than no data at all. The bad data that we have been collecting for more than a decade is as effective a trap as anyone could have devised to ensure that actually measuring the performance of cervical smear interpretation does not happen. The only question that remains is, "How we will be able to escape?"

Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology

CONTEXT

Numerous studies address the accuracy or positive predictive value of cytologic interpretations for defined histopathologic entities. The reproducibility (precision) of cytologic interpretation is less well defined.

OBJECTIVE

To establish and compare the reproducibility (precision) of cytologic interpretation in gynecologic cytopathology, as reflected in the educational program of the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology (PAP).

METHODS

The pathologists' interpretations for both validated (25 745 responses) and educational conventional (14 353 responses) slides in the PAP program for 2001 were analyzed. The frequency of exact matches between the reference and pathologists' interpretation for each of the cytologic interpretative categories was identified, and the cumulative distributions of exact match rates were derived. chi2 Tests by reference interpretations were used for cytodiagnostic categories, least and most reproducible groupings, and high-grade (HSIL) versus low-grade (LSIL) squamous intraepithelial lesions.

RESULTS

Pathologists' interpretations of negative, Candida, Trichomonas, herpes, and LSIL were characterized by a high degree of exact matching, while interpretations of repair, HSIL, adenocarcinoma, and squamous cell carcinomas were characterized by a lesser degree of exact matching (reproducibility). Pathologists' cytologic interpretations of HSIL were significantly less reproducible than those of LSIL.

CONCLUSION

The cytologic interpretations of the most significant categories (HSIL, squamous cell carcinoma, and adenocarcinoma) are less precise than those of specific infection (Candida, Trichomonas, and yeast), negative, and LSIL categories. Cytologic interpretations of LSIL are made with greater precision than those of HSIL and may represent a more appropriate endpoint to measure the precision performance of gynecologic cytology laboratories.

Metaanalysis of the accuracy of rapid prescreening relative to full screening of pap smears

BACKGROUND

Efficient quality assurance and improvement measures are essential ingredients in a well organized cytology-based program for cervical carcinoma screening. Various pap smear review procedures, aiming for optimization of accuracy, are described throughout the literature. Evaluation and synthesis of those methods are needed. In a previous study, we pooled data on the diagnostic quality of rapid reviewing (RR) of cervical smears initially reported as normal or unsatisfactory. We now focus on rapid prescreening (RPS) of unreported smears.

METHODS

Six published studies on the accuracy of RPS relative to subsequent full screening were pooled using metaanalytic methods. Individual and pooled sensitivity, specificity, and predictive values were assessed using forest plots. Random effect pooling methods were used for interstudy heterogeneity. Variation in sensitivity according to influencing factors was explored by metaregression.

RESULTS

The pooled average sensitivity of RPS was 64.9% (95% confidence interval [CI] 50.7-79.1%) for all abnormalities, 72.6% (95% CI 60.6-85.2%) for low-grade lesions or more severe, and 85.7% (95% CI 77.8-93.6%) for high-grade lesions or more severe. The pooled specificity was estimated at 96.8% (CI 95.8-97.8%). The sensitivity increased significantly with duration of screening and decreased with workload. Almost 3% of all abnormal slides were detected only by RPS (2.8%; CI 0.0-5.8%). This is comparable to the proportion of false-negative smears detectable by RR.

CONCLUSIONS

Rapid prescreening has a high yield for severe dysplasia and shows diagnostic properties that support its use as a quality control procedure in cytologic laboratories. We showed previously that RR is superior to full reviewing of a 10% random sample of negative slides (10% FR). Because the yield of additional abnormalities found by RR and RPS is comparable, we expect RPS to be more efficient than 10% FR as well.

Measuring sensitivity in gynecologic cytology: a review

BACKGROUND

The sensitivity of gynecologic cytology has been measured in several different ways. The current review summarizes the major sources of bias and the results of these efforts.

METHODS

In the current study, a review of the literature was conducted.

RESULTS

The major sources of bias in measuring the sensitivity of gynecologic cytology are a lack of reproducibility, bias in the review process, bias in case selection, selection and correction of the gold standard, and the value of surrogate markers. Despite these sources of variation, the sensitivity of the Papanicolaou (Pap) smear is relatively stable, ranging from 50-75% if a single consistent threshold is used, to up to 94% if either Autopap-directed rescreening or thin-layer methods are used to diagnose high-grade squamous intraepithelial lesions using a threshold of atypical squamous cells of undetermined significance. Methods for the routine evaluation of sensitivity currently are not available and may be difficult to devise.

CONCLUSIONS

The sensitivity of the Pap smear in study situations is well known. Whether these reflect performance in real life is not known, and methods to compare performance in real life are not available.

Blinded review as a method for quality improvement in surgical pathology

CONTEXT

Several studies have shown that blinded review, because it is less biased and may improve vigilance, is an excellent method for detecting errors and improving performance in gynecologic cytology. The value of blinded review in surgical pathology is not known.

OBJECTIVE

To determine the value of blinded review in surgical pathology.

METHODS

Five hundred ninety-two biopsy cases were reviewed without knowledge of the original diagnosis or history, and the results were compared with those of the original diagnosis.

RESULTS

Complete agreement was obtained in 567 (96%) of 592 cases. The technique of blinded review of biopsy material had a sensitivity of 98%, failing to identify a lesion in 7 cases; no cases of malignancy were missed. The specificity was 100%. Differences in diagnostic threshold were the most common source of disagreement. False-negative cases were identified by the technique and were clinically significant. Power studies show that the number of cases requiring review to identify significant errors are large, but potentially achievable by blinded review.

CONCLUSION

Blinded review is a sensitive and effective method for identifying areas of disagreement, including false-negative cases, and for decreasing errors in surgical pathology biopsy material.

Aberrant expression of E-cadherin in cervical intraepithelial neoplasia correlates with a false-negative Papanicolaou smear

OBJECTIVES

E-cadherin is responsible for cell adhesion in normal cervical epithelium. It is normally absent in the superficial epithelial layers, allowing for exfoliation. We investigated the correlation between E-cadherin distribution and Papanicolaou smear in subjects with cervical dysplasia.

STUDY DESIGN

Tissue samples from 25 women with cervical dysplasia were tested for E-cadherin, beta-catenin, and alpha-catenin expression by immunohistochemistry. The expression pattern of these proteins, whether full thickness or restricted to the basal layers, was correlated with the Papanicolaou smear result.

RESULTS

Of 12 women with normal Papanicolaou smears, 10 of 11 informative cases demonstrated E-cadherin expression throughout all epithelial layers. Eight of 10 informative cases with an abnormal Papanicolaou smear showed E-cadherin only at the basal layers. Alpha-catenin was distributed throughout the entire epithelium in samples of all 25 women.

CONCLUSIONS

Expression of E-cadherin throughout all epithelial layers was correlated with a false-negative Papanicolaou smear. It is likely that aberrant persistence of E-cadherin in these lesions interferes with the exfoliation of abnormal cells.

Quantifying the value of in-house consultation in surgical pathology

In-house consultation is a well-known method to improve diagnostic accuracy and agreement, but the technique has not been well studied. We reviewed the results of in-house consultation in a large private hospital practice setting for a 1-month period and determined its effect on diagnostic accuracy using the final sign-out as the "gold standard." During this 1-month period, 352 cases were reviewed as in-house consultations. Initial complete agreement was found in 315 (89.5%) cases. Using the initial diagnosis as the test case and the final sign-out as the gold standard, of the 37 discrepant cases, 4 (1.1%) were thought to represent false-negative results, (0.3%) a false-positive result, 3 (0.9%) differences in type, and 29 (8.2%) differences in diagnostic threshold. Disagreements in 10 cases were thought to be potentially clinically significant. Internal consultation was obtained on approximately 20% of all cases seen in the laboratory and disagreements were found in 2% of all cases. Internal consultation has a significant and measurable impact on the practice of surgical pathology.

Estimating the percentage of Papanicolaou smears that can be reproducibly identified: modeling Papanicolaou smear interpretation based on multiple blinded rescreenings

BACKGROUND

Multiple blinded rescreenings of Papanicolaou (Pap) smears for litigation purposes is based on the assumption that a subset of Pap smears can be reproducibly identified. The size of this subset is not known.

METHODS

To estimate the size of the subset of Pap smears that can be reproducibly identified, a model was constructed based on the results of repeated blinded screenings in the AutoPap Primary Screening System Trial. Additional analysis came from data in the literature.

RESULTS

Routine and AutoPap-assisted screening both have a detection rate for all detected abnormal cases of < 50%. Models with only two subsets or types of slides each with a different detection rate correlated well with the available data. Data from multiple rapid reviews strongly supported the existence of additional definable subsets. Although the percentage of cases with an expected detection rate of 100% in a three-subset model might have been as high as 30% of the abnormal cases detected in a single review, all estimates that included a second subset of slides with at least a 50% detection rate limited the percentage of slides in the 100% sensitive subset of slides to < 2% of all abnormal slides and < 6% of all abnormal slides detected by a single screening.

CONCLUSIONS

Repeated screenings of Pap smears allowed more accurate models of the sensitivity of Pap-smear screening and the overall incidence of abnormal cases. The data strongly supported the existence of multiple subsets of Pap smears, which can be defined by repeated blinded rescreenings. The percentage of slides that can be reproducibly identified was small.