Interobserver variability: comparison between liquid-based and conventional preparations in gynecologic cytology

Interobserver agreement in the interpretation of anal cytology

BACKGROUND

Anal cytology represents a tool for anal cancer screening in high-risk populations. In addition to accuracy, the reproducibility of the interpretation is of key importance. The authors evaluated the agreement of anal cytologic interpretation between two cytopathologists.

METHODS

Liquid-based cytologic slides from human immunodeficiency virus (HIV)-negative men who have sex with men (MSM) were evaluated by two readers with at least 10 years of expertise in cervical cytology. Cases with a discordant interpretation were reviewed, and a consensus was reached. Human papillomavirus (HPV) genotyping was performed using a proprietary HPV genotyping test. Unweighted and weighted Cohen kappa and 95% confidence interval (CI) values were calculated.

RESULTS

Overall, 713 slides that were adequate for interpretation were evaluated (MSM: median age, 33 years). An HPV test was performed on 620 samples (87.0%). Considering a dichotomous interpretation (negative for intraepithelial lesion or malignancy vs. atypical squamous cells of undetermined significance or worse), the crude agreement between the two readers was 93.3% (kappa = 0.82; 95% CI, 0.77-0.87). Once a consensus for discordant cases was reached, the best agreement was found for the negative for intraepithelial lesion or malignancy category (511 of 528 samples; 96.8%), whereas the atypical squamous cells of undetermined significance category showed the lowest agreement (90 of 117 samples, 76.9%). Considering the individual cytologic categories, overall agreement was 92.1% (kappa = 0.85; 95% CI, 0.81-0.89). The discordant interpretations were not associated with high-risk HPV infection, HPV16 infection, or MSM age.

CONCLUSIONS

The results indicating excellent interobserver agreement in this study substantiate the use of anal cytology in the setting of human immunodeficiency virus-negative MSM.

Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability

Kappa statistics have been widely used in the pathology literature to compare interobserver diagnostic variability (IOV) among different pathologists but there has been limited discussion about the clinical significance of kappa scores. Five representative and recent pathology papers were queried using clinically relevant specific questions to learn how IOV was evaluated and how the clinical applicability of results was interpreted. The papers supported our anecdotal impression that pathologists usually assess IOV using Cohen's or Fleiss' kappa statistics and interpret the results using some variation of the scale proposed by Landis and Koch. The papers did not cite or propose specific guidelines to comment on the clinical applicability of results. The solutions proposed to decrease IOV included the development of better diagnostic criteria and additional educational efforts, but the possibility that the entities themselves represented a continuum of morphologic findings rather than distinct diagnostic categories was not considered in any of the studies. A dataset from a previous study of IOV reported by Thunnissen et al. was recalculated to estimate percent agreement among 19 international lung pathologists for the diagnosis of 74 challenging lung neuroendocrine neoplasms. Kappa scores and diagnostic sensitivity, specificity, positive and negative predictive values were calculated using the majority consensus diagnosis for each case as the gold reference diagnosis for that case. Diagnostic specificity estimates among multiple pathologists were > 90%, although kappa scores were considerably more variable. We explain why kappa scores are of limited clinical applicability in pathology and propose the use of positive and negative percent agreement and diagnostic specificity against a gold reference diagnosis to evaluate IOV among two and multiple raters, respectively.

[The Spanish Society of Cytology: Quality control program of gynecological cytology]

INTRODUCTION

In Spain, the guidelines for cervical cancer screening include a recommendation to enroll in external quality control programs. The Spanish Society of Cytology (SEC) has initiated its own quality control program of gynecological cytology (QCPGC).

AIM

To describe and discuss the results of the second round of SEĆs QCPGC.

MATERIAL AND METHOD

The cases are selected by a group of expert cytologists. The cases with an agreement of 75% of four cytopathologists were used. The cases were scanned with Aperio. The scanned cases not available were excluded. We included a total of 23 cases, 1 negative, 15 low grade lesions (4 ASCUS and 11 LSIL) and 7 high grade lesions (1 ASCH and 6 HSIL). Sixteen cases were studied with ThinPrep™ platform and in 7 cases the SurePath™ platform was used.

RESULTS

Sixteen hospitals participated. The global mean concordance was 70.6%. The mean concordance in the type of lesion was 63.1%. The concordance was 71.9% in negative diagnoses, 56.2% in ASCUS, 69.5% in LSIL and 82.8% in HSIL The discordant cases were diagnosed more frequently as negative and ASCUS. 4.4% of cases had major discordances (HSIL or ASCH versus negatives).

CONCLUSIONS

Our results are similar to those reported in the literature, with very few severe discordances.

Interobserver variability of cervical cytology in HIV-infected women

OBJECTIVES

Our objectives were to determine the reproducibility of cytological specimen interpretation between two pathologists in human immunodeficiency virus (HIV)-infected women (from the VIHGY, ANRS CO17 study of human papillomavirus genital pathology among HIV-positive women) and to analyse the improvement, if any, between conventional and liquid-based cytology (LBC) interpretations.

MATERIALS AND METHODS

A sample of all abnormal and 40% of randomly selected normal Papanicolaou (Pap) tests was randomly ordered and read blindly by a second pathologist using the revised Bethesda terminology 2001. For both conventional and liquid-based preparations, unweighted and Cicchetti-Allison-weighted kappa and their 95% confidence intervals (CIs) were calculated. Kappa values were then compared using the Altman rule to classify the reproducibility of cytological specimen interpretation.

RESULTS

Two hundred and seventy-seven conventional Pap tests were reviewed, including 79 abnormal and 10 unsatisfactory results. Overall agreement between the two observers was 78%, with an estimated Cicchetti-Allison-weighted kappa of 0.69 (95%CI, 0.61-0.77). The corresponding values for the 268 LBCs, including 123 abnormal and two unsatisfactory results, were 84% and 0.82 (95%CI, 0.76-0.87), respectively. The reproducibility of LBC interpretations was significantly higher than that of conventional preparations (P = 0.009) and, for both laboratories, the percentages of unsatisfactory results were significantly lower for LBC.

CONCLUSION

In HIV-infected women in the combination antiretroviral therapy era, the strength of agreement was better for LBCs than for conventional preparations, with a lower percentage of unsatisfactory results. When available, LBC should be preferred because of its higher reproducibility.

Inter- and intra-operator variability in the reading of indirect immunofluorescence assays for the serological diagnosis of scrub typhus and murine typhus

Inter- and intra-observer variation was examined among six microscopists who read 50 scrub typhus (ST) and murine typhus (MT) indirect immunofluorescence assay (IFA) immunoglobulin M (IgM) slides. Inter-observer agreement was moderate (κ = 0.45) for MT and fair (κ = 0.32) for ST, and was significantly correlated with experience (P = 0.03 and P = 0.004, respectively); κ-scores for intra-observer agreement between morning and afternoon readings (range = 0.35–0.86) were not correlated between years of experience for ST and MT IFAs (Spearman's ρ = 0.31, P = 0.54 and P = 0.14, respectively; P = 0.78). Storage at 4°C for 2 days showed a change from positive to negative in 20–32% of slides. Although the titers did not dramatically change after 14 days of storage, the final interpretation (positive to negative) did change in 36–50% of samples, and it, therefore, recommended that slides should be read as soon as possible after processing.

Evaluation of agreement between conventional and liquid-based cytology in cervical cancer early detection based on analysis of 2,091 smears: experience at the Brazilian National Cancer Institute

The aim of this work was to evaluate the agreement between conventional cytology (CC) and liquid-based cytology (LBC) in cervical cancer early detection. The results of CC and LBC (DNACitoliq in 2,059 women aged 25-59 years were compared. The percent agreement, kappa coefficient, prevalence-adjusted bias-adjusted kappa coefficient (PABAK), and Chamberlain's percent positive agreement (PPA) were calculated. The percent agreement between the two methods was very good (80% and 79%, respectively, for normal versus ASCUS+; and normal versus ASCUS, AGUS and LSIL+ vs. HSIL+). The kappa coefficient indicates slight agreement (0.26 and 0.23, respectively), but when PABAK was used the agreement was good (0.61 and 0.68, respectively). PPA was high for normal results (79.2%) and low for the remaining categories. To conclude, in this study, agreement between LBC and CC was only good for normal results, which involves the majority of cases and positively influences the overall agreement rate.

Barriers to adoption of recent technology in cervical screening

The Pap smear is one of the modern success stories in the field of preventive medicine. Since its introduction as a screening test, there has been a dramatic reduction in the incidence of cervical cancer. However, the search for a better screening test continues. The new technologies, including liquid-based cytology (LBC), Human Papilloma Virus (HPV) testing and automated or machine-assisted screening have been introduced. However, there is continuous debate about whether society's limited resources are better spent on reaching the underserved rather than on these technologies. Another question is whether these technologies create yet another kind of disparity in delivering preventive care. For example, despite the wide use of LBC (99% of tests submitted to our laboratory are LBC), conventional Pap smears are still used to screen/follow up some women. It is not clear why some providers continue to prefer conventional smear over LBC and what are the barriers for adopting LBC in cervical cancer screening. We hypothesize the lower cost of conventional compared to LBC Pap testing, patient's lower socio-economic indices, a patient's medical history and provider's subspecialty/training all appear to play a role in the choice of using conventional Pap testing rather than LBC. Unintentionally, this choice results in repeat testing, delayed treatment and potentially higher costs than intended. The ultimate goal of this review article is to understand and explore possible barriers and disparities to adopting new technology in cancer screening.

The significance of certification in liquid-based cytology and performance in the college of american pathologists interlaboratory comparison program in cervicovaginal cytopathology

CONTEXT

Morphology training and certification provided by the manufacturer is required before laboratories are allowed to prepare and interpret liquid-based cytology (LBC) gynecologic slides. Whether this certification is associated with improved LBC interpretation performance is not known.

OBJECTIVE

To determine whether LBC certification correlates with performance.

DESIGN

The results of the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytopathology for the year 2004 were reviewed. During this year a question on the answer form asked participants whether they were certified in LBC preparations. Participant responses were analyzed at 2 levels of agreement with the reference diagnosis of unknown glass slides mailed quarterly.

RESULTS

For cytotechnologists and pathologists, participants who were LBC certified had a significantly lower total discordant rate to series than participants who were not certified for validated LBC ThinPrep (Cytyc Corp, Marlborough, Mass) preparations (P < .001 for both groups). The results were similar for validated conventional preparations for pathologists but not for cytotechnologists. The differences were present across most individual diagnostic series. Differences were most striking for low-grade squamous intraepithelial lesions. In addition, cytotechnologists and pathologists who were certified had a significantly lower total discordant rate to the exact diagnosis for validated LBC ThinPrep slides than those who were not certified (discordant rates for cytotechnologists, 14.14% and 23.36%, respectively, P < .001; for pathologists, 15.99% and 24.07%, respectively, P < .001).

CONCLUSION

Although not all confounding factors were excluded in this study, LBC certification is associated with a significantly lower discordant rate to both series and exact diagnosis for validated LBC ThinPrep preparations.

Histopathologic assessment of hot-spot microvessel density and vascular patterns in glioblastoma: Poor observer agreement limits clinical utility as prognostic factors: a translational research project of the European Organization for Research and Treatment of Cancer Brain Tumor Group

BACKGROUND

Hot-spot microvessel density (MVD) and vascular patterns have been reported as histopathologic factors that influence prognosis in retrospective series of malignant gliomas. To investigate clinical utility, the authors systematically studied observer agreement on MVD and vascular patterns and the influence of repeatedly assessed data on patient outcomes in 2 independent glioblastoma series.

METHODS

MVD and vascular patterns were assessed retrospectively by 5 observers in 1) a retrospectively compiled glioblastoma series that included 110 patients and 2) a glioblastoma series that included 233 patients who were treated within a randomized trial. MVD was determined in the field of greatest density ("hot-spot"). Predominantly classic or bizarre vascular patterns were determined by using a previously defined algorithm.

RESULTS

Observer agreement on MVD was highly variable (range of kappa values, 0.464-0.901). The worst observer agreement was achieved when both the selection of hot-spots and MVD counts were performed independently. Survival analysis did not show a consistent association between repeatedly assessed MVD and patient outcome. Observer agreement on vascular patterns was poor (kappa = 0.297). Survival analysis did not show a consistent association between repeatedly assessed vascular patterns and patient outcome.

CONCLUSIONS

Observer agreement on hot-spot MVD and vascular patterns in patients with glioblastoma was poor in independent assessments. MVD and vascular patterns were not associated consistently with patient outcome. Based on these findings, the authors concluded that poor observer agreement limits the clinical utility of histopathologically assessed hot-spot MVD and vascular patterns as prognostic factors in patients with glioblastoma. Improved methodologies for morphologic assessment of glioblastoma vascularization need to be identified.

A comparative study of two devices used for cervical cell sampling raises some doubts about liquid-based cytology

The objective of this study was to compare the quality of smears obtained by a newly designed cervical cell sampler (the implement) to those obtained with the cervex brush, using liquid-based cytology (LBC). A prospective randomized controlled trial with the approval of the local ethics committees after informing the Medical Devices Agency in the UK was used in this study. The study was carried out in colposcopy clinics in two district general NHS Trust Hospitals in London. A total of 200 women were recruited (100 from each hospital). The women were randomized into two groups, each of them having two smears at the same time. One hundred (50%) of the women had their first smear with the new implement, and the other 100 (50%) had their first smear with the cervex brush. The main outcome of this study is good-quality smears, assessed by evidence of effective sampling of the transformation zone, including immature metaplastic cells and also endocervical cells. Eighty-five percent of the smears taken by the new implement showed good-quality smears compared to 91% of those taken by the cervex brush, a statistically marginally nonsignificant result, with P value = 0.052 (McNemar exact test). A larger number of good-quality smears, judged by evidence of sampling of transformation zone, were obtained with the cervex brush, though the difference was statistically marginally nonsignificant, using the British society for clinical cytology criteria. Surprisingly, despite the use of LBC, there was a higher rate of inadequate smears obtained with both the devices (8% with each) compared to our previous study using the conventional slide test (1.3% with the new implement and 1.8% with the Jordan's spatula).

Interobserver agreement in the interpretation of anal intraepithelial neoplasia

BACKGROUND

Anal carcinoma incidence is increasing, and is highest among men with human immunodeficiency virus (HIV) infection who have sex with men. Anal carcinoma and anal intraepithelial neoplasia (AIN) are ascertained on tissue histology, but requires invasive procedures. Screening for AIN using anal cytology was suggested. The authors evaluated agreement on cytologic and biopsy specimens from HIV-positive men undergoing anal carcinoma screening.

METHODS

One hundred twenty-nine HIV-positive men with a history of anal-receptive intercourse underwent anal cytology, anoscopy, and biopsy. Four pathologists independently assessed cytology and biopsy specimens and reached consensus for discordant cases.

RESULTS

Each pathologist evaluated 120 cytology and 155 biopsy specimens. The weighted kappa value for overall agreement was 0.54 (95% confidence interval [CI], 0.49-0.59) for cytology specimens and 0.59 (95%CI, 0.55-0.63) for biopsy specimens. The median kappa values for pairwise agreement among pathologists and for agreement with consensus were, respectively, 0.69 and 0.77 for cytology and 0.66 and 0.75 for biopsy. At least 3 pathologists were in agreement for 92 (76.7%) cytology and 134 (86.5%) biopsy specimens. Reliability for the Bethesda classification system was at least moderate, except for the cytologic category of atypical squamous cells of undetermined significance (kappa = 0.12). Fourteen of 29 (48.3%) cytology specimens and 36 of 47 (76.6%) biopsy specimens with consensus interpretation of high-grade squamous intraepithelial lesions (HSIL) were interpreted originally as HSIL by > or = 3 pathologists. The kappa value for agreement with consensus distinguishing HSIL from non-HSIL ranged from 0.55 to 0.88 for cytology specimens and from 0.76 to 0.94 for biopsy specimens.

CONCLUSIONS

Agreement for cytologic and biopsy interpretations was generally at least moderate. Nevertheless, these results supported the need for disease indicators with greater reliability.

Results of the implementation of liquid-based cytology-SurePath in the Ontario screening program

BACKGROUND

The objective of the current study was to evaluate the adequacy and detection rates of SurePath after its implementation in Ontario.

METHODS

The detection and adequacy rates of the SurePath liquid-based cytology system (SP-LBC) were calculated for manually reviewed slides of the year 2002. The adequacy and detection rates from this study group were compared with a historical conventional smear (CS) group from the same laboratories during the same period of the previous year.

RESULTS

The SP-LBC study group consisted of 352,680 specimens with cytodiagnoses and the CS group included 378,990 specimens. The unsatisfactory rate for SP-LBC (0.24%) was less than that of the CS group (0.58%). The detection rate of atypical squamous cells (ASC+) by the SP-LBC group (4.69%) was greater than that of the CS group (3.81%), as was the detection rate of low-grade squamous intraepithelial lesions (LSIL+; 2.13% vs. 1.50% in the CS group). There was only a trend toward increased detection of high-grade squamous intraepithelial lesions (HSIL+) in the SP-LBC group (0.34%) relative to the CS group (0.31%), because the detection rate for carcinoma by SP-LBC declined.

CONCLUSIONS

The implementation of SP-LBC has been followed by better specimen adequacy and detection rates for ASC+, LSIL+, and a trend of increased detection of HSIL+ relative to CS practice. To determine sensitivity rates, a histopathologic database for cervical carcinoma and precancer needs to be established.

Comparison of Performance of Conventional and ThinPrep Gynecologic Preparations in the College of American Pathologists Gynecologic Cytology Program

Context.—Results of clinical trials suggest that interpretation of liquid-based cytology preparations is more accurate and is associated with less screening error than interpretation of conventional preparations.

Objective.—In this study, the performance of participants in interpreting ThinPrep (TP) preparations was compared with participants' performance on conventional Papanicolaou tests in the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology (PAP).

Design.—The results of the PAP from the year 2002 were reviewed, and the discordancies to series and exact-match error rates for the 2 cytologic methods were compared.

Results.—For this study, a total of 89 815 interpretations from conventional smears and 20 886 interpretations from TP samples were analyzed. Overall, interpretations of TP preparations had both significantly fewer false-positive (1.6%) and false-negative (1.3%) rates than those of conventional smears (P = .001 and P = .02, respectively) for validated or validated-equivalent slides, as assessed by concordance with the correct diagnostic series. In this assessment of concordance to series, interpretations of educational TP and conventional preparations were similar, except for high-grade squamous intraepithelial lesion, in which the performance was significantly worse for educational TP preparations (false-negative rate of 8.1% vs 4.1% for conventional smears, P &lt; .001). When interpretations were matched to the exact diagnosis, validated-equivalent TP preparations were generally more accurate for diagnoses in the 100 series and 200 series than were conventional smears. Notably, for the reference diagnosis of squamous cell carcinoma, the exact-match error rate on validated equivalent TP slides was significantly greater than that of conventional slides (44.5% vs 23.1%, P &lt; .001). Interpretations of educational TP preparations also had a significantly higher error rate in matching to the exact reference diagnosis for squamous cell carcinoma (33.7% vs 22.8%, P = .007).

Conclusions.—Overall, TP preparations in this program were associated with significantly lower error rates than conventional smears for both validated and educational cases. However, unlike the negative for intraepithelial lesion and malignancy, not otherwise specified, low-grade squamous intraepithelial lesion, and adenocarcinoma cytodiagnostic challenges, participants' responses indicated some difficulty in recognizing high-grade squamous intraepithelial lesion and squamous cell carcinoma.

Reproducibility of Subclassification of Squamous Intraepithelial Lesions: Conventional versus ThinPrep Paps

OBJECTIVE

Liquid-based cytologic methods are increasingly used, and classification of squamous intraepithelial lesions (SIL) affects patient management. This study compared interobserver reproducibility in SIL subclassification on conventional (CV) and ThinPrep (TP) cytologic specimens.

MATERIALS AND METHODS

Four reviewers independently subclassified SIL on 69 CV and 60 TP Paps. Specimens were retrieved by computer search of biopsy-confirmed SIL cases. A consensus interpretation of low-grade SIL (LSIL) or high-grade SIL (HSIL) was assigned when three or four observers agreed.

RESULTS

All four observers agreed in 40 of 69 CV with consensus reached in 56 of 69 CV Paps (81%; 20 LSIL, 36 HSIL). For TP Paps, 38 of 60 had 100% agreement, with consensus reached in 56 of 60 TP Paps (93%; 28 LSIL, 26 HSIL, 2 SIL, difficult to grade). kappa values for all four observers were 0.48 for CV (fair agreement) and 0.63 for TP (substantial). Pairwise kappa values ranged from 0.44 to 0.60 for CV and 0.54 to 0.76 for TP. Most of the nonconsensus cases included SIL, difficult to grade interpretations; in several, the original cytologic or biopsy SIL classification, or both, was also indeterminate, or cytologic and biopsy results did not correlate exactly. High-grade biopsies followed 15% of LSIL CV and 36% of LSIL TP.

CONCLUSIONS

Interobserver reproducibility in SIL subclassification may be better on TP Paps; however, both CV and TP have indeterminate lesions with low interobserver agreement. The TP specimens did not show improved correlation with histologic analysis, and specimens with consensus do not always have correlating biopsy findings.