Flawed external validation study of the ADNEX model to diagnose ovarian cancer

Subjective assessment and IOTA ADNEX model in evaluation of adnexal masses in patients with history of breast cancer

OBJECTIVE

To evaluate the performance of subjective assessment and the Assessment of Different NEoplasias in the adneXa (ADNEX) model in discriminating between benign and malignant adnexal tumors and between metastatic and primary adnexal tumors in patients with a personal history of breast cancer.

METHODS

This was a retrospective single-center study including patients with a history of breast cancer who underwent surgery for an adnexal mass between 2013 and 2020. All patients had been examined with transvaginal or transrectal ultrasound using a standardized examination technique and all ultrasound reports had been stored and were retrieved for the purposes of this study. The specific diagnosis suggested by the original ultrasound examiner in the retrieved report was analyzed. For each mass, the ADNEX model risks were calculated prospectively and the highest relative risk was used to categorize each into one of five categories (benign, borderline, primary Stage I, primary Stages II-IV or metastatic ovarian cancer) for analysis of the ADNEX model in predicting the specific tumor type. The performance of subjective assessment and the ADNEX model in discriminating between benign and malignant adnexal tumors and between primary and metastatic adnexal tumors was evaluated, using final histology as the reference standard.

RESULTS

Included in the study were 202 women with a history of breast cancer who underwent surgery for an adnexal mass. At histology, 93/202 (46.0%) masses were benign, 76/202 (37.6%) were primary malignancies (four borderline and 72 invasive tumors) and 33/202 (16.3%) were metastases. The original ultrasound examiner classified correctly 79/93 (84.9%) benign adnexal masses, 72/76 (94.7%) primary adnexal malignancies and 30/33 (90.9%) metastatic tumors. Subjective ultrasound evaluation had a sensitivity of 93.6%, specificity of 84.9% and accuracy of 89.6%, while the ADNEX model had higher sensitivity (98.2%) but lower specificity (78.5%), with similar accuracy (89.1%), in discriminating between benign and malignant ovarian masses. Subjective evaluation had a sensitivity of 51.5%, specificity of 88.8% and accuracy of 82.7% in distinguishing metastatic and primary tumors (including benign, borderline and invasive tumors), and the ADNEX model had a sensitivity of 63.6%, specificity of 84.6% and similar accuracy (81.2%).

CONCLUSIONS

The performance of subjective assessment and the ADNEX model in discriminating between benign and malignant adnexal masses in this series of patients with history of breast cancer was relatively similar. Both subjective assessment and the ADNEX model demonstrated good accuracy and specificity in discriminating between metastatic and primary tumors, but the sensitivity was low. © 2023 International Society of Ultrasound in Obstetrics and Gynecology.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on preoperative diagnosis of ovarian tumors

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the preoperative diagnosis of ovarian tumors, including imaging techniques, biomarkers and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the preoperative diagnosis of ovarian tumors and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the preoperative diagnosis of ovarian tumors and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on pre-operative diagnosis of ovarian tumors

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group, and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the pre-operative diagnosis of ovarian tumors, including imaging techniques, biomarkers, and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the pre-operative diagnosis of ovarian tumors and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when a consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the pre-operative diagnosis of ovarian tumors and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on preoperative diagnosis of ovarian tumours

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the preoperative diagnosis of ovarian tumours, including imaging techniques, biomarkers and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the preoperative diagnosis of ovarian tumours and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when a consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the preoperative diagnosis of ovarian tumours and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

The ADNEX model to triage adnexal masses: An external validation study and comparison with the IOTA two-step strategy and subjective assessment by an experienced ultrasound operator

OBJECTIVES

The ADNEX (Assessment of Different NEoplasias in the adneXa) model was developed using parameters collected by experienced (level III) ultrasound examiners. Our primary aim was to externally validate the ADNEX model. Then, the discriminatory performance of ADNEX was compared with the two-step strategy and subjective assessment by an experienced ultrasound operator.

METHODS

Between February 2013 and January 2017, all patients who were scheduled for surgery for an adnexal mass at the Sant'Anna Hospital in Turin were enrolled in this study. Preoperative transvaginal sonography was performed, and the two-step strategy was applied for triage of the adnexal mass. Two ultrasound examiners, IOTA certified, applied the ADNEX model to all the collected masses based on the ultrasound reports. Finally, an experienced operator assigned the subjective assessment based on recorded ultrasound images. The discrimination and calibration performance of ADNEX were evaluated. The AUC was calculated for the basic discrimination between benign and malignant tumours. In addition, AUCs were computed for each pair of tumour types using the conditional risk method.

RESULTS

A total of 577 patients were included in the analysis: the overall prevalence of malignancy was 25 %. With ADNEX, the AUC to differentiate between benign and malignant masses was 0.9111 (95 % CI 0. 8788-0.9389). At risk cut-offs of 1%, 10 % and 30 %, sensitivities were 100 %, 89.6 % and 79.2 %, respectively, and specificities were 2.8 %, 76.2 % and 89.6 %, respectively. Discrimination between benign and stage II-IV tumours was good (AUC 0.935). The model had the most difficulties discriminating between borderline and stage I tumours (AUC 0.666), and between stages II-IV invasive and secondary metastatic tumours (AUC 0.736). The polytomous discrimination index (PDI) was 0.61 for ADNEX, whereas PDI for random performance would be 0.25. ADNEX proved to be equally or more accurate than the subjective assessment or the two-step strategy in the discrimination between benign and malignant adnexal masses.

CONCLUSIONS

the ADNEX model could probably be successfully applied when an expert examiner is not available and, therefore both a subjective assessment and the two-step strategy cannot be performed.

Poor performance of clinical prediction models: the harm of commonly applied methods

OBJECTIVE

To evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies.

STUDY DESIGN AND SETTING

A previously published model predicted the likelihood of having a mutation in germline DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of over 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to bias and discriminative ability (concordance index, c) in independent data.

RESULTS

We found over 50% bias for five of six originally selected predictors, unstable model specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model prespecification based on external knowledge and using continuous predictors led to better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively).

CONCLUSION

Prediction models perform poorly if based on small numbers of events and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes.