Effects of observer variation in population screening for cervical carcinoma

Analysis of error in calculating the false-negative rate in the interpretation of cervicovaginal smears: the need to review abnormal cases

BACKGROUND

Determining the false-negative rate (FNR) of cervicovaginal smear interpretation is a necessary step for any quality assessment and improvement program. All tests estimate the FNR, but the accuracy of these estimates varies from test to test. Two methods for determining the FNR have been proposed, specifically "seeding" of the initial screening population with smears from patients with a known diagnosis and rescreening a random sample of negative smears. However, the accuracy of neither method is known.

METHODS

A review of the literature, an analysis of the sources of error, and an estimate of their magnitude was performed for each method.

RESULTS

Seeding has a large sampling error, and more important, the FNR that this test measures does not reflect the FNR of the laboratory as a whole. Random rescreening underestimates the FNR of primary screening by the FNR of rescreening. Currently, the FNR of rescreening is not known, not measured, and may be high. Nevertheless, the FNR of rescreening and the false-positive rate (FPR) of initial screening both can be measured by rescreening abnormal cases. Knowledge of both the FNR and the FPR of initial screening allows the efficiency of cervicovaginal smear interpretation to be measured, which may be a better measure of overall accuracy than the FNR alone.

CONCLUSIONS

Random, blinded rescreening of normal and abnormal smears can more accurately measure the FNR of screening than rescreening of normal smears alone.

Consequences of neural network technology for cervical screening: increase in diagnostic consistency and positive scores

BACKGROUND

Screening programs for the early detection of cervical carcinoma are criticized because of the problem of false-negative diagnoses. A successful approach for solving this problem is applying neural network technology (PAP-NET) to assist the cytotechnologist (CT) in finding the (few) abnormal cells in the smear.

METHODS

In 3 consecutive years (1992, 1993, and 1994), 25,767 smears were screened conventionally and 65,527 with the aid of PAPNET by 7 CTs. For each CT, the scores for atypias of undetermined significance, squamous or glandular (ASCUC/AGUS according to the Bethesda classification system), indicated by Positive 1, for low grade precursor lesions, by Positive II, for high grade lesions and invasive carcinoma, by Positive III, were calculated for both screening methods. The histologic scores were also calculated.

RESULTS

The mean positive scores of the seven CTs were higher for PAPNET than for conventional screening, and the coefficients of variability were lower. For Positive III smears, the consistency in screening was significantly higher for PAPNET than for conventional screening. The higher histologically positive scores for carcinoma in situ and invasive carcinoma indicated an increased screening sensitivity.

CONCLUSIONS

As demonstrated by the improvement in the performances of all CTs involved, screening efficacy was enhanced by the use of neural network technology.

Bethesda classification of cervicovaginal smears: reproducibility and viral correlates

Fifty-five cervicovaginal smears from women with squamous intraepithelial lesions (SILs) were independently evaluated on two separate occasions by four cytopathologists using a binary classification system (the Bethesda system). Smears were categorized as low-grade (LSIL) or high-grade (HSIL) using previously published criteria. All women had subsequent cervical biopsies containing human papillomavirus (HPV) DNA amplified with the polymerase chain reaction and typed by restriction fragment polymorphism analysis. Three or more observers agreed on classification in 49 of 55 cases (87%); unanimous diagnoses were rendered in 31 cases (56%). Interobserver and intraobserver reproducibility ranged from fair to near-excellent (kappa values 0.40 to 0.63; 0.63 to 0.74, respectively). HPV types included HPV 16 (27%), 18 (7%), 31 (9%), 35 (4%), 39 (4%), 6 (10%), 11 (2%), novel types (30%), and multiple types (4%). High-risk HPV types (16, 18, 31, 35, and 39) were significantly associated (P = .03) with consensus HSIL diagnoses (agreement of three or more observers). This was primarily because of the strong association of HPV 16 with HSIL (P = .001). After excluding HPV 16, the other high-risk HPV types (18, 31, 35, and 39) were no longer significantly associated with consensus HSIL diagnoses (P > .5). Conversely, LSIL diagnoses were significantly associated with non-high-risk HPV types (all HPV types except 16, 18, 31, 35, and 39; P = .006). Binary cytological classification of cervicovaginal SILs is reproducible among cytopathologists. Such classification correlates well with most low-risk HPV types and with the prototypic high-risk HPV 16 but not with other high-risk HPV types.

Observer variation in the histological diagnosis of plasmacytoid leukemia (marine anemia)

The reliability of past histological criteria for the diagnosis of plasmacytoid leukemia was investigated by studying observer variation in the diagnosis of the disease. Participants blindly evaluated a series of histological samples and classified them as positive, negative or questionable cases of the disease. Intra- and interpathologist agreement were used to assess reliability and were determined by calculating the observed agreement in diagnosis and the kappa statistic. There was poor overall agreement for the classification of sample cases. Questionable cases were most frequently misclassified but marked disagreement also existed for positive and negative samples. Access to historical information prior to histological examination influenced agreement. The results show that previously described histological criteria alone were insufficient for establishing the diagnosis of plasmacytoid leukemia.

Interobserver variability of cervical smears with squamous-cell abnormalities: a Philadelphia study

The reproducibility of reporting squamous lesions by the Bethesda System (TBS) was evaluated by distributing 20 slides to be classified among 5 panelists considered experts in the field of cytopathology. Four cases were chosen for their classic morphology and the remainder were foreseen to produce possible discrepancies within one diagnostic category. For 7/20 (35%) cases there was unanimous agreement. Participants disagreed within one category of magnitude for seven (35%) cases. In six (30%) cases there was a range of more than one category disagreement. However, additional written comments modifying TBS diagnoses often diminished the clinical significance of these discrepancies. We conclude that despite the important role of TBS in standardization of Pap smear reports, a great degree of subjectivity exists in classifying squamous abnormalities without "classic" morphology. The lack of reproducibility should be taken into account in cytology proficiency testing.

Retrospective analysis of non-correlating cervical smears and colposcopically directed biopsies

The purpose of this study is to evaluate the cause of discrepancies between non-correlating cytologic and histologic cervical samples. The biopsy results of 433 women examined colposcopically were compared to their referral cervical smears (RS). There was a discrepancy between the RS and the subsequent biopsy in 120 women (28%). One hundred of these 120 RS were available for review; and in each case, a reason for the discrepancy was established and classified as RS overcall, RS undercall, RS sampling error, or biopsy sampling error. Fifty-one discrepant RS were overcalled. They were reported initially as condyloma (19), mild dysplasia (22), and moderate dysplasia (10). One RS was undercalled. Nine RS were not diagnostic of the biopsy-proven lesion due to smear sampling error. The discrepancies in the remaining 39 cases were due to biopsy sampling error. Twenty-one of these 39 cases had additional biopsies or smears that confirmed the presence of condyloma/dysplasia, and 18 had negative follow-up. In summary, discrepancies were a result of pathologists' interpretative error, predominantly overcalls, in 52% of non-correlating cases, and smear or biopsy sampling error in the remaining 48%.

Screening for cervical cancer--should the routine be challenged?

Despite the extensiveness of cervical cancer screening programs, certain major issues are still at stake: (a) substantial false negative rates are common, mostly due to impaired test quality, (b) the optimal length of interval between screenings is uncertain, being a function of available resources and the physicians' attitude, (c) the postmenopausal population, in which incidence is highest, is inadequately tapped. Efforts must be made to optimise the process through more stringent control measures, and a more comprehensive cover of the target population.

[Not Available]

This study assesses the follow-up provided to patients of a family medicine clinic in Quebec who had abnormal cervical smears that were possibly precancerous or cancerous. Less follow-up was provided in the case of benign atypical smears, probably because the role of benign atypia in the pathogenesis of cervical cancer is as yet unclear.

Variations in interpretation of gynecologic cytology photomicrographs

This was a study of variations in interpretations of photomicrographs of gynecologic cytology specimens. Two photomicrographs from each of 20 gynecologic cytology specimens were examined by 10 cytotechnologists and 10 pathologists. In only two cases was there agreement among all participants regarding the primary entity illustrated. The greatest agreement in interpretation was for benign entities such as repair reaction, koilocytotic changes, benign endocervical cells, and herpes simplex. There was little uniformity in the recognition of intraepithelial neoplasia and invasive carcinoma, or these lesions are difficult to illustrate using photomicrographs. Most abnormal cases were identified at least as atypical, but in many cases the range of responses was great -- for example, atypical squamous metaplasia to squamous carcinoma. There was more uniformity among cytotechnologists than pathologists in the recognition of the cytologic entities presented. The results of this study illustrate the difficulty of selecting photomicrographs for testing purposes and the lack of uniformity in the interpretation of such photomicrographs.

Interobserver variability in the interpretation of epithelial ovarian cancer

The interobserver diagnostic reproducibility for epithelial ovarian neoplasia was studied. The histologic sections of 68 ovarian tumors from 34 patients were independently reviewed by two pathologists from different institutions, without knowledge of the clinical course. Each observer rendered 68 diagnoses. The interobserver agreement rate for histologic type was 60% and for histologic grade 66%. In the 23 instances in which a discrepancy in grade occurred, 5 (23%) were disagreements in the diagnosis of borderline versus malignant tumors. On analysis of the diagnostic variability with regard to histologic type, one observer (A) classified 60% of the tumors as undifferentiated, while the other observer (B) classified 59% of the tumors as serous. In an attempt to understand the reasons for the diagnostic disagreements, the observers were asked to simultaneously reexamine the material. This occurred 6 months after the initial review and they were unaware of their original diagnoses. The diagnostic differences with regard to histologic type were mainly due to (1) tumor cell heterogeneity, and (2) difficulty in discriminating between serous and undifferentiated tumors. The differences in grade were largely related to the use of different criteria. Observer A used mitotic counts while observer B used glandular pattern and its replacement by solid sheets of epithelium. Observer B consistently assigned a higher grade to the tumors. In making therapeutic decisions, clinicians should be aware of the interobserver diagnostic variability and the reasons for this variability.

Measuring repeatability and validity of histological diagnosis--a brief review with some practical examples

Evaluation of histological diagnosis requires an index of agreement (to measure repeatability and validity) together with a method of assessing bias. Cohen's kappa statistic appears to be the most suitable tool for measuring levels of agreement, which if unsatisfactory may be caused by bias. Further study of bias is possible by examining levels of agreement for each diagnostic category or by searching for categories of disagreement in which more observations occur than would be expected by chance alone. This article gives reasons for choosing the kappa statistic, with examples illustrating its calculation and the investigation of bias.

Invasive squamous cell carcinoma of the uterine cervix. II. Reproducibility of a histopathologic malignancy grading system

A histopathologic malignancy grading system for invasive squamous carcinoma of the uterine cervix has previously been presented. The grading results in a total malignancy point value for every patient (range 8-24). This score has been proven to have a significant prognostic value for the individual patient. The inter-observer reproducibility of this system has been examined by two pathologists, on the pretreatment biopsies of 100 patients and the intra-observer reproducibility on 190 patients (observer one) and on 100 patients (observer two). These biopsies were estimated twice and three times, respectively, without knowledge of former results and at an interval of 2 to 6 months. The inter-observer correlation coefficient was 0.54, the intra-observer correlation coefficients 0.94 and 0.74, respectively.

Reliability of histo-pathological diagnosis of squamous epithelial changes of the uterine cervix

The reliability of histological diagnosis of squamous epithelial changes was tested by letting 13 pathologists read 1,001 consecutive cervical biopsies twice. Intra-observer and inter-observer agreement, variance, and deviation of diagnosis were determined. The diagnostic ability showed great individual variation and no significant correlation to experience in pathology. The diagnosis of invasive cancer had a high diagnostic specificity, and the diagnostic sensitivity of the diagnosis of no significant epithelial changes was high too. The reliability of the diagnosis of dysplasia and carcinoma in situ proved unsatisfactory.

Malignant round-cell tumours of bone: an analytical histological study from the Cancer Research Campaign's bone tumour panel

A study of 40 cases of malignant round-cell tumour of one was made from the files of the Cancer Research Campaign's Bone Tumour Panel. Five pathologists made a careful study of observer error, involving repeated examination of routine paraffin sections, to determine whether the cases were a homogeneous group or a collection of differing sub-groups. Cell outline, nuclear staining, nuclear pleomorphism, conspicuous nucleoli, reticulin pattern and intracellular glycogen were the histological features selected for study. For each feature, the results were analysed to assess the importance of differences between tumours, between samples of tissue from the same tumour, and between observers. It is concluded that round-cell tumours of bone are a heterogeneous group, although completely distinct sub-groups could not be identified. Certain histological features tend to be associated, and it is reasonable to distinguish on histological grounds between Ewing's sarcoma and reticulum-cell sarcoma, although some tumours are not typical of either group.

Chronic mastopathy and breast cancer. A follow-up study

Over an average period of seven years 2,9000 cases of benign breast lesions diagnosed by biopsy between 1948 and 1973 in the Department of Pathology, Kaiser Foundation Hospital, Oakland, were followed for breast cancer development. When classified according to traditional diagnostic categories, the cancer incidence per 1,000 person-years varies between 2.7 and 7.9 and appears to be elevated in comparison to expectations obtained from the Third National Cancer Survey, San Francisco Bay Area. Two thousand four hundred biopsies were also scored by the Black-Chabou method. There is an upward trend in the breast cancer incidence as the atypia score rises, a finding which confirms conclusions from a retrospective case-control study by Black et al.

Agreement and disagreement between pathologists in histological diagnosis

Studies on differences of observation and interpretation between pathologists in connection with histological diagnosis are rare. The present paper gives a general account of such a study, by members of the Cancer Research Campaign's Bone Tumour Panel, of certain histological features of ‘malignant round-cell tumours’ of bone. Significant differences were found, not only between the different cases studied, but between the observations of the different pathologists, and between the repeated observations of individual pathologists.