Analysis of error in calculating the false-negative rate in the interpretation of cervicovaginal smears: the need to review abnormal cases

Corn leaf disease: insightful diagnosis using VGG16 empowered by explainable AI

The agricultural sector is pivotal to food security and economic stability worldwide. Corn holds particular significance in the global food industry, especially in developing countries where agriculture is a cornerstone of the economy. However, corn crops are vulnerable to various diseases that can significantly reduce yields. Early detection and precise classification of these diseases are crucial to prevent damage and ensure high crop productivity. This study leverages the VGG16 deep learning (DL) model to classify corn leaves into four categories: healthy, blight, gray spot, and common rust. Despite the efficacy of DL models, they often face challenges related to the explainability of their decision-making processes. To address this, Layer-wise Relevance Propagation (LRP) is employed to enhance the model's transparency by generating intuitive and human-readable heat maps of input images. The proposed VGG16 model, augmented with LRP, outperformed previous state-of-the-art models in classifying corn leaf diseases. Simulation results demonstrated that the model not only achieved high accuracy but also provided interpretable results, highlighting critical regions in the images used for classification. By generating human-readable explanations, this approach ensures greater transparency and reliability in model performance, aiding farmers in improving their crop yields.

Surrogate indicators of sensitivity in gynecologic cytology: can they be used to improve the measurement of sensitivity in the laboratory?

Background:

Measuring the sensitivity of screening in gynecologic cytology in real life is problematic. However, other quality measures may correlate with sensitivity, including the atypical squamous cells (ASC)/squamous intraepithelial lesion (SIL) ratio. Whether these other measures can function as “surrogate indicators” for sensitivity and improve the assessment of sensitivity in the laboratory is not known.

Materials and Methods:

We compared multiple quality measures with true screening sensitivity in a variety of situations.

Results:

The abnormal rate, ASC rate, and ASC/SIL ratio were all highly correlated (r =.83 or greater) with sensitivity when the overall laboratory sensitivity was low (85%) but became less correlated (.64 or less) or uncorrelated when the screening sensitivity was higher (88% or 95%, respectively). Sensitivity was more highly correlated with the abnormal rate than the ASC/SIL ratio at low screening sensitivity. While thresholds could be set that were highly sensitive and specific for suboptimal screening, these thresholds were often less than one standard deviation away from the mean.

Conclusion:

The correlation of the abnormal rate and the ASC/SIL ratio with sensitivity depends on overall sensitivity. Standards to define minimum screening sensitivity can be defined, but these standards are relatively narrow. These features may limit the utility of these quality measures as surrogates for sensitivity.

Assuring the quality of quality assurance: seeding abnormal slides into the negative Papanicolaou smears that will be rapid rescreened

BACKGROUND

Rapid rescreening (RR) of negative Papanicolaou smears (PS) is used in many countries as a quality-assurance measure. Seeding of abnormal slides has been suggested as a way to increase the sensitivity of this procedure. Since 2004, the authors have carried out RR with seeding before issuing reports. In this article, they describe their experience.

METHODS

Abnormal seeds were sourced from the previous day's high-grade cases, both squamous and glandular. Slides were evaluated for the 'degree of difficulty' (which was defined as the number of fields required to find (fields-to-find [FTF]) the abnormality), relabeled, and redotted to make them indistinguishable from the routine RR work. The number of seeds found/missed, the identity of the screener, the type of seeded abnormality, the degree of difficulty of the seed, and the mapping technique used all were recorded. The cytologists also were surveyed about their views on seeding.

RESULTS

Overall, 14.8% of 3082 high-grade seeds were missed during RR. There was no relation between seeds missed and the mapping technique used. However, the difficulty of the seed was relevant to the number missed and ranged from 8.3% when the FTF was <5 to 36% when the FTF was >10 (P = .000). The difference between intraepithelial seeds and invasive seeds was significant for squamous seeds (P = .031) but not for glandular seeds. Glandular seeds also were more likely to be missed than squamous seeds (23.1% vs 14.3%; P = .002). Most cytologists believed that seeding was a good idea and that seeds increased their level of vigilance.

CONCLUSIONS

The authors' experience demonstrated that routine seeding is practicable for both conventional and liquid-based slides. With the advent of the human papillomavirus vaccine, abnormalities will become rarer, and seeding will be necessary to maintain the alertness of cytologists.

Measuring the significance of workload on performance of cytotechnologists in gynecologic cytology: a study using rapid prescreening

BACKGROUND

Workload is extensively regulated and often used as a measure of quality in gynecologic cytology. Whether workload correlates with the sensitivity of screening in gynecologic cytology is not known.

METHODS

The sensitivity of gynecologic cytology screening was measured over an 8-month period using the result of full screening coupled with the results of rapid prescreening. Sensitivity results were then correlated with daily workload volumes and the experience level of individual cytotechnologists.

RESULTS

Rapid prescreening had an average sensitivity of 41.9% for atypical squamous cells of undetermined significance (ASCUS) and above. Full screening had a corrected sensitivity of 82.2% for ASCUS and above. Rapid prescreening increased the sensitivity of the laboratory to 89.9%. The sensitivity of full screening was significantly different between cytotechnologists (79.2% vs 99%, P < .001), but was not correlated with years of experience, sensitivity of rapid prescreening, or workload (all P > .05). When sensitivity and workload were examined on a monthly basis, there was no significant difference between sensitivity either as a group or individually at the highest and lowest workloads (P > .40 for all).

CONCLUSIONS

Screeners sensitivity in gynecologic cytology appears to be unrelated to the experience level of individual cytotechnologists or to their workload at the levels examined.

Rapid prescreening of papanicolaou smears

BACKGROUND

Efficient quality control (QC) is essential to ensure high sensitivity of Papanicolaou (Pap) smears. For this purpose, rescreening of 10% random negative smears is ineffective. Rapid rescreening (RR) of all negative Pap smears is more practical and has received widespread acceptance, especially in Europe, although its sensitivity is difficult to monitor and its retrospective nature may influence the vigilance of the screeners. The method of rapid prescreening (RPS) overcomes these drawbacks because rapid review of Pap smears occurs before routine full screening.

METHODS

All routine conventional Pap smears over 2 months underwent RPS by 12 cytotechnologists. Approximately 30 seconds were allowed to prescreen each slide. The presence of abnormal cells (atypical squamous cells of undetermined significance [ASCUS] or above), infection or endometrial cells detected on RPS was documented. All slides subsequently underwent routine full screening. Results of both screening methods were compared.

RESULTS

Of a total of 8364 Pap smears, 310 (3.7%) cases were categorized as abnormal after final diagnosis. Of those, 135 were also detected on RPS (sensitivity of 43.5%). Seventeen abnormal cases were detected only on RPS: these consisted of 13 ASCUS cases, 3 low-grade squamous intraepithelial lesions, and 1 high-grade squamous intraepithelial lesion. The sensitivity of RPS for infections and endometrial cells was 51.6% and 28.3%, respectively. Implementation of RPS did not significantly impact the work flow in our laboratory.

CONCLUSIONS

RPS is an efficient and practical QC tool. It is a reliable method with which to monitor sensitivity and reduce the false-negative rate, and because it is done before finalizing the case, it allows for timely corrections to the diagnosis and avoids the need to amend reports.

Rescreening in cervical cytology for quality control. When bad data is worse than no data or what works, what doesn't, and why

The currently mandated methods to measure the sensitivity of Papanicolaou smear interpretation (including the 5-year look back and review of 10% of negative smears) are misleading. They do not allow one to measure the true sensitivity of the test and allow only a small fraction of errors to be detected and corrected. Rapid prescreening and automated screening are the only methods that seem practically feasible, and will allow the sensitivity of the method to be measured on a routine basis, and, thereby allow a reduction in overall errors. Although emerging technologies, such as HPV testing, may allow alternative methods of measuring sensitivity, the most appropriate way to use and interpret these data in this context are not yet fully developed. Unfortunately, at present there seems to be little desire to change the way things are done, and, therefore, to accurately measure sensitivity in cervical cytology. The first task that needs to be undertaken in fixing a problem is to admit that one exists. At present, most laboratory directors believe that their own laboratory is performing satisfactorily. They may well be, although the laboratories lack an analytic method to demonstrate this, and, therefore, the need for better QA methods does not seem to be acute. There is some educational value to the currently practiced and mandated performance measures, the 5-year look back and review of 10% of negative smears. Most laboratory directors seem to be happy with their QA methods and are not concerned that the data that are derived from the 10% review of negative slides does not reflect their actual sensitivity of screening in any meaningful way. Unfortunately, the forces that are currently in place in the United States ensure that accurate measures of the sensitivity of cervical cytology interpretation are unlikely to be implemented beyond the level of individual experimentation. As long as the expectation of cytologists is that the error rates are significantly less than they actually are; as long as there is significant legal and financial risk to actually measuring the true sensitivity; and as long as fictitious measures of performance are not only advocated but mandated; the confluence of incentives will ensure that the true sensitivity of the test will never be measured on a routine basis. Despite all of this, it is possible that cervical cytology screening may, in fact, already be performing at an optimal level. Being able to measure this operating performance may not effect any improvement to the overall process. The ultimate arbiter in this debate will always be the demonstration of a reduction of cervical cancer morbidity and mortality with any new measure implemented. At the present time, there is only one solution to the quality control issue; a force from outside the system must change the balance of the aforementioned incentives. The promise of data from European experiences with rapid rescreening may show that this method is effective and accurate. Such data might make the current methods that are in use in the United States more open to change. So yes, the answer is that "bad" data may be worse than no data at all. The bad data that we have been collecting for more than a decade is as effective a trap as anyone could have devised to ensure that actually measuring the performance of cervical smear interpretation does not happen. The only question that remains is, "How we will be able to escape?"

Precision in gynecologic cytologic interpretation: a study from the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology

CONTEXT

Numerous studies address the accuracy or positive predictive value of cytologic interpretations for defined histopathologic entities. The reproducibility (precision) of cytologic interpretation is less well defined.

OBJECTIVE

To establish and compare the reproducibility (precision) of cytologic interpretation in gynecologic cytopathology, as reflected in the educational program of the College of American Pathologists Interlaboratory Comparison Program in Cervicovaginal Cytology (PAP).

METHODS

The pathologists' interpretations for both validated (25 745 responses) and educational conventional (14 353 responses) slides in the PAP program for 2001 were analyzed. The frequency of exact matches between the reference and pathologists' interpretation for each of the cytologic interpretative categories was identified, and the cumulative distributions of exact match rates were derived. chi2 Tests by reference interpretations were used for cytodiagnostic categories, least and most reproducible groupings, and high-grade (HSIL) versus low-grade (LSIL) squamous intraepithelial lesions.

RESULTS

Pathologists' interpretations of negative, Candida, Trichomonas, herpes, and LSIL were characterized by a high degree of exact matching, while interpretations of repair, HSIL, adenocarcinoma, and squamous cell carcinomas were characterized by a lesser degree of exact matching (reproducibility). Pathologists' cytologic interpretations of HSIL were significantly less reproducible than those of LSIL.

CONCLUSION

The cytologic interpretations of the most significant categories (HSIL, squamous cell carcinoma, and adenocarcinoma) are less precise than those of specific infection (Candida, Trichomonas, and yeast), negative, and LSIL categories. Cytologic interpretations of LSIL are made with greater precision than those of HSIL and may represent a more appropriate endpoint to measure the precision performance of gynecologic cytology laboratories.

One hundred percent thorough quality control rescreening of liquid-based monolayers in cervicovaginal cytopathology

BACKGROUND

Quality control (QC) of Papanicolaou (Pap) smear testing focuses on the identification of screening errors by 10% random, rapid, or thorough manual rescreening of some portion of negative smears. One hundred percent thorough manual rescreening has been reported to be the most effective method of identifying screening errors in conventional Pap smears (CP), but to the authors' knowledge no experience with this QC method has been reported for the ThinPrep Pap test (TP). The current study reports the estimated screening error rate of TP as determined by a QC program using 100% thorough rescreening.

METHODS

All TP samples received at the study institution between January 1, 1999 and December 31, 2000 and initially screened as negative underwent thorough manual QC rescreening.

RESULTS

A total of 53,419 TP samples were received during the study period. Of these, 5,368 cases (10%) initially were interpreted as abnormal. A total of 47,247 cases (88.4%) were rescreened. Abnormalities were identified in 804 additional cases, for a screening error rate of 13.0%. Of the 804 cases, 678 (84.3%) were atypical squamous cells of undetermined significance, 116 (14.4%) were low-grade squamous intraepithelial lesions, and 10 (1.2%) were high-grade squamous intraepithelial lesions. No tumors were identified on rescreening.

CONCLUSIONS

In the current study, 100% thorough rescreening of TP samples was found to result in the detection of a significant number of abnormalities that would have been missed by routine random 10% QC rescreening. The screening error rate determined by 100% thorough QC rescreening of TP is comparable to that reported for CP QC rescreening.

Blinded review as a method for quality improvement in surgical pathology

CONTEXT

Several studies have shown that blinded review, because it is less biased and may improve vigilance, is an excellent method for detecting errors and improving performance in gynecologic cytology. The value of blinded review in surgical pathology is not known.

OBJECTIVE

To determine the value of blinded review in surgical pathology.

METHODS

Five hundred ninety-two biopsy cases were reviewed without knowledge of the original diagnosis or history, and the results were compared with those of the original diagnosis.

RESULTS

Complete agreement was obtained in 567 (96%) of 592 cases. The technique of blinded review of biopsy material had a sensitivity of 98%, failing to identify a lesion in 7 cases; no cases of malignancy were missed. The specificity was 100%. Differences in diagnostic threshold were the most common source of disagreement. False-negative cases were identified by the technique and were clinically significant. Power studies show that the number of cases requiring review to identify significant errors are large, but potentially achievable by blinded review.

CONCLUSION

Blinded review is a sensitive and effective method for identifying areas of disagreement, including false-negative cases, and for decreasing errors in surgical pathology biopsy material.

Quantifying the value of in-house consultation in surgical pathology

In-house consultation is a well-known method to improve diagnostic accuracy and agreement, but the technique has not been well studied. We reviewed the results of in-house consultation in a large private hospital practice setting for a 1-month period and determined its effect on diagnostic accuracy using the final sign-out as the "gold standard." During this 1-month period, 352 cases were reviewed as in-house consultations. Initial complete agreement was found in 315 (89.5%) cases. Using the initial diagnosis as the test case and the final sign-out as the gold standard, of the 37 discrepant cases, 4 (1.1%) were thought to represent false-negative results, (0.3%) a false-positive result, 3 (0.9%) differences in type, and 29 (8.2%) differences in diagnostic threshold. Disagreements in 10 cases were thought to be potentially clinically significant. Internal consultation was obtained on approximately 20% of all cases seen in the laboratory and disagreements were found in 2% of all cases. Internal consultation has a significant and measurable impact on the practice of surgical pathology.

An accurate and precise methodology for routine determination of the false-negative rate of Papanicolaou smear screening

BACKGROUND

Although the false-negative rate (FNR) is the most important quality control measure for Papanicolaou smear screening, accurate, precise, and feasible methods for determining this value are lacking.

METHODS

The author undertook an analysis and review of the literature.

RESULTS

The best estimates of the FNR using atypical squamous cells of undetermined significance (ASCUS) as a threshold range from 17% to 61%. Sources of error in the accuracy of this measure that must be accounted for include the FNR of the review method, differences in diagnostic thresholds between the original diagnostic method and the review method, and differences in diagnostic accuracy between the original diagnostic method and the review method. Statistically precise (valid to within 10%) measurement of this value in laboratories with an ASCUS+ rate of 7% can be made from interlaboratory rescreening of approximately 1200-1500 randomly selected normal and abnormal slides along with both laboratories rediagnosing without rescreening 300-400 benign cellular change and ASCUS slides to determine the difference in diagnostic threshold. Consensus for each slide is not required with this method. Changing the threshold to low grade squamous intraepithelial lesion (best estimate FNR, 14-58%) requires review of significantly more slides to achieve the same statistical level of precision.

CONCLUSIONS

Detailed analysis of the sources of error in determining the FNR allow creation of methods that are relatively unbiased, feasible, and testable and whose accuracy and precision can be determined.

The human false-negative rate of rescreening Pap tests. Measured in a two-arm prospective clinical trial

BACKGROUND

Routine quality control rescreening often is used to calculate the false-negative rate (FNR) of gynecologic cytology. Theoretic analysis suggests that this is not appropriate, due to the high FNR of rescreening and the inability to actually measure it. The authors sought to determine the FNR of manual rescreening in a large, prospective, two-arm clinical trial using an analytic instrument in the evaluation.

METHODS

The results of the Autopap System Clinical Trial, encompassing 25,124 analyzed slides, were reviewed. The false-negative and false-positive rates at various thresholds were determined for routine primary screening, routine rescreening, Autopap primary screening, and Autopap rescreening by using a simple, standard methodology.

RESULTS

The FNR of routine manual rescreening at the level of atypical squamous cells of undetermined significance (ASCUS) was 73%, more than 3 times the FNR of primary screening; 11 cases were detected. The FNR of Autopap rescreening was 34%; 80 cases were detected. Routine manual rescreening decreased the laboratory FNR by less than 1%; Autopap rescreening reduced the overall laboratory FNR by 5.7%. At the same time, the false-positive rate for Autopap screening was significantly less than that of routine manual screening at the ASCUS level (4.7% vs. 5.6%; P < 0.0001). Rescreening with the Autopap system remained more sensitive than manual rescreening at the low grade squamous intraepithelial lesions threshold (FNR of 58.8% vs. 100%, respectively), although the number of cases rescreened was low.

CONCLUSIONS

Routine manual rescreening cannot be used to calculate the FNR of primary screening. Routine rescreening is an extremely ineffective method to detect error and thereby decrease a laboratory's FNR. The Autopap system is a much more effective way of detecting errors within a laboratory and reduces the laboratory's FNR by greater than 25%.

Measuring and reporting errors in surgical pathology. Lessons from gynecologic cytology

Substantial improvements in measuring and reporting errors in gynecologic cytology have been made during the last decade. Measuring and reporting errors in surgical pathology recently has gained renewed interest. However, review of current literature demonstrates mistakes in how these data are measured and reported. Error rates have been reported from review of consecutive material, biopsy material, and consultation material and range from 0.25% to 43%. Errors have been divided into anatomic regions and specimen types and separated according to their clinical significance. However, to be comparable, errors must be reported in reference to the incidence of disease and not to overall caseload. Blinding and reviewer error have been addressed only rarely, and the true incidence of errors is almost certainly higher than reported. "Gold standards" are not well defined. In addition, available data strongly suggest that the greatest source of error is with false-negative diagnoses, which are detected only rarely by review of consultation material. Most of these issues have been addressed in the gynecologic cytology literature. Errors in surgical pathology are more common than generally believed, and efforts should be made to define methods that allow appropriate interlaboratory comparisons.

A practical problem with calculating the false-negative rate of Papanicolaou smear interpretation by rescreening negative cases alone

BACKGROUND

Rescreening negative Papanicolaou (Pap) smears alone is the most commonly employed method of determining the false-negative rate (FNR) or the false-negative proportion for a laboratory. Acceptable FNRs have been proposed, and the number of slides needed to be rescreened to demonstrate a statistically significant difference in FNRs has been determined. The authors sought to determine the range of FNRs this method can measure and, by implication, the value of this method.

METHODS

A literature review and an analysis of the FNRs this method can generate was performed.

RESULTS

If one assumes that the FNR of review is the same as that of initial screening, the maximum measured FNR is only 25%, even with a true FNR of anywhere from 0-100%. In fact, as a laboratory's FNR increases over 50%, the measured FNR decreases back to zero. This range of FNRs corresponds very closely to the published range of FNRs of 1.6-28%. Because many authorities believe that 5% may be the lowest achievable FNR, the entire possible range of measured FNRs is only 5-25%. In this setting, a statistically significant difference of 20% is meaningless, and a statistically significant difference of 10% can only be achieved by laboratories with an initial FNR of less than 15%, and actual changes in FNR that are much greater than 10%.

CONCLUSIONS

FNRs determined by review of negative smears without abnormal smears generate unreliable and potentially seriously misleading results. Current methodologies exist for more accurately determining the FNR of Pap smear screening by incorporating abnormal smears into the review process. There is little justification for further review of negative Pap smears alone as a method for determining the FNR of a laboratory. Cancer (Cancer Cytopathol)

A practical guide to Papanicolaou smear rescreens: how many slides must be reevaluated to make a statistically valid assessment of screening performance?

BACKGROUND

The question of the minimum number of Papanicolaou (Pap) smear slides that must be rescreened to draw statistically valid conclusions regarding the accuracy of screening often is raised. No method for generating answers in varying laboratory circumstances has achieved widespread application; standard statistical sample size calculations may represent such a resource.

METHODS

A series of tables was constructed to display minimum required numbers of rescreens, with each table representing differing hypothetical laboratory circumstances. To use each table, assumptions must be specified in advance as to prevalence of abnormality, definition of error, baseline false-negative proportions (FNPs) of performance, and a degree of increase in FNPs that is considered a departure from baseline warranting concern, among others.

RESULTS

The authors constructed four sample tables displaying minimum numbers of slides that must be rescreened in differing specified laboratory scenarios. Depending on assumed conditions and predetermined levels of satisfactory and unsatisfactory accuracy, the range of numbers is very broad (38-10,000). One example representing likely conditions indicates that 1040 slides must be reexamined; in another scenario, a sample size of 300 is sufficient.

CONCLUSIONS

The minimum number of rescreened slides needed to draw statistically valid conclusions regarding Pap smear screening accuracy can be calculated using standard statistical methods. However, a number of assumptions must be detailed in advance. The authors offer this as a practical guide and a continuation of a general inquiry regarding Pap smear error rate measurement and display. The use of these tables raises at least as many questions as it answers, but still may represent a significant advance. Future efforts at further numeric characterization of aspects of Pap smear screening performance are warranted to enable rational decision making when performance is examined in the course of quality assurance, and during quality control and regulatory activities. [See editorial on pages 127-9, this issue.]