External validation of diagnostic models to estimate the risk of malignancy in adnexal masses

Clinical Utility and Diagnostic Accuracy of ROMA, RMI, ADNEX, HE4, and CA125 in the Prediction of Malignancy in Adnexal Masses

OBJECTIVE

We aimed to compare the clinical utility and diagnostic accuracy of the ADNEX model, ROMA score, RMI I, and RMI IV, as well as two serum markers (CA125 and HE4) in preoperative discrimination between benign and malignant adnexal masses (AMs).

METHODS

We conducted a retrospective study extracting all consecutive patients with AMs seen at our Institution between January 2015 and December 2020. Accuracy metrics included sensitivity (SE), specificity (SP), and area under the receiver operating characteristic curve (AUC), and their 95% confidence intervals (CI) were calculated for basic discrimination between AMs. Model performance was evaluated in terms of discrimination ability and clinical utility (net benefit, NB).

RESULTS

A total of 581 women were included; 481 (82.8%) had a benign ovarian tumor and 100 (17.2%) had a malignant tumor. The SE and SP of CA125, HE4, ROMA score, RMI I, RMI IV, and ADNEX model were 0.60 (0.54-0.66) and 0.80 (0.76-0.83); 0.39 (0.30-0.49) and 0.96 (0.94-0.98); 0.59 (0.50-0.68) and 0.92 (0.88-0.95); 0.56 (0.46-0.65) and 0.98 (0.96-0.99); 0.54 (0.44-0.63) and 0.96 (0.94-0.98); 0.82 (0.73-0.88) and 0.91 (0.89-0.94), respectively. The overall AUC was 0.76 (0.74-0.79) for CA125, 0.81 (0.78-0.83) for HE4, 0.82 (0.80-0.85) for ROMA, 0.86 (0.84-0.88) for RMI I, 0.83 (0.81-0.86) for RMI IV, and 0.92 (0.90-0.94) for ADNEX. The NB for ADNEX was higher than other biomarkers and models across all decision thresholds between 5% and 50%.

CONCLUSIONS

The ADNEX model showed a better discrimination ability and clinical utility when differentiating malignant from benign Ams, compared to CA125, HE4, ROMA score, RMI I, and RMI IV.

Toward real-world deployment of machine learning for health care: External validation, continual monitoring, and randomized clinical trials

In this commentary, we elucidate three indispensable evaluation steps toward the real-world deployment of machine learning within the healthcare sector and demonstrate referable examples for diagnostic, therapeutic, and prognostic tasks. We encourage researchers to move beyond retrospective and within-sample validation, and step into the practical implementation at the bedside rather than leaving developed machine learning models in the dust of archived literature.

Prospective external validation of IOTA methods for classifying adnexal masses and retrospective assessment of two-step strategy using benign descriptors and ADNEX model: Portuguese multicenter study

OBJECTIVES

To externally and prospectively validate the International Ovarian Tumor Analysis (IOTA) Simple Rules (SRs), Logistic Regression model 2 (LR2) and Assessment of Different NEoplasias in the adneXa (ADNEX) model in a Portuguese population, comparing these approaches with subjective assessment and the risk-of-malignancy index (RMI), as well as with each other. This study also aimed to retrospectively validate the IOTA two-step strategy, using modified benign simple descriptors (MBDs) followed by the ADNEX model in cases in which MBDs were not applicable.

METHODS

This was a prospective multicenter diagnostic accuracy study conducted between January 2016 and December 2021 of consecutive patients with an ultrasound diagnosis of at least one adnexal tumor, who underwent surgery at one of three tertiary referral centers in Lisbon, Portugal. All ultrasound assessments were performed by Level-II or -III sonologists with IOTA certification. Patient clinical data and serum CA 125 levels were collected from hospital databases. Each adnexal mass was classified as benign or malignant using subjective assessment, RMI, IOTA SRs, LR2 and the ADNEX model (with and without CA 125). The reference standard was histopathological diagnosis. In the second phase, all adnexal tumors were classified retrospectively using the two-step strategy (MBDs + ADNEX). Sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios and overall accuracy were determined for all methods. Receiver-operating-characteristics curves were constructed and corresponding areas under the curve (AUC) were determined for RMI, LR2, the ADNEX model and the two-step strategy. The ADNEX model calibration plots were constructed using locally estimated scatterplot smoothing (LOESS).

RESULTS

Of the 571 patients included in the study, 428 had benign disease and 143 had malignant disease (prevalence of malignancy, 25.0%), of which 42 had borderline ovarian tumor, 93 had primary invasive adnexal cancer and eight had metastatic tumors in the adnexa. Subjective assessment had an overall sensitivity of 97.9% and a specificity of 83.6% for distinguishing between benign and malignant lesions. RMI showed high specificity (95.6%) but very low sensitivity (58.7%), with an AUC of 0.913. The IOTA SRs were applicable in 80.0% of patients, with a sensitivity of 94.8% and specificity of 98.6%. The IOTA LR2 had a sensitivity of 84.6%, specificity of 86.9% and an AUC of 0.939, at a malignancy risk cut-off of 10%. At the same cut-off, the sensitivity, specificity and AUC for the ADNEX model with vs without CA 125 were 95.8% vs 98.6%, 82.5% vs 79.7% and 0.962 vs 0.960, respectively. The ADNEX model gave heterogeneous results for distinguishing between benign masses and different subtypes of malignancy, with the highest AUC (0.991) for discriminating benign masses from primary invasive adnexal cancer Stages II-IV, and the lowest AUC (0.696) for discriminating primary invasive adnexal cancer Stage I from metastatic lesion in the adnexa. The calibration plot suggested underestimation of the risk by the ADNEX model compared with the observed proportion of malignancy. The MBDs were applicable in 26.3% (150/571) of cases, of which none was malignant. The two-step strategy using the ADNEX model in the second step only, with and without CA 125, had AUCs of 0.964 and 0.961, respectively, which was similar to applying the ADNEX model in all patients.

CONCLUSIONS

The IOTA methods showed good-to-excellent performance in the Portuguese population, outperforming RMI. The ADNEX model was superior to other methods in terms of accuracy, but interpretation of its ability to distinguish between malignant subtypes was limited by sample size and large differences in the prevalence of tumor subtypes. The IOTA MBDs are reliable in identifying benign disease. The two-step strategy comprising application of MBDs followed by the ADNEX model if MBDs are not applicable, is suitable for daily clinical practice, circumventing the need to calculate the risk of malignancy in all patients. © 2024 International Society of Ultrasound in Obstetrics and Gynecology.

Current and Emerging Strategies for Tubo-Ovarian Cancer Diagnostics

Tubo-ovarian cancer is the most lethal gynaecological cancer. More than 75% of patients are diagnosed at an advanced stage, which is associated with poorer overall survival. Symptoms at presentation are vague and non-specific, contributing to late diagnosis. Multimodal risk models have improved the diagnostic accuracy of adnexal mass assessment based on patient risk factors, coupled with findings on imaging and serum-based biomarker tests. Newly developed ultrasonographic assessment algorithms have standardised documentation and enable stratification of care between local hospitals and cancer centres. So far, no screening test has proven to reduce ovarian cancer mortality in the general population. This review is an update on the evidence behind ovarian cancer diagnostic strategies.

Performance of the IOTA ADNEX model combined with HE4 for identifying early-stage ovarian cancer

Objective

This work was designed to investigate the performance of the International Ovarian Tumor Analysis (IOTA) ADNEX (Assessment of Different NEoplasias in the adneXa) model combined with human epithelial protein 4 (HE4) for early ovarian cancer (OC) detection.

Methods

A total of 376 women who were hospitalized and operated on in Women and Children’s Hospital of Chongqing Medical University were selected. Ultrasonographic images, cancer antigen-125 (CA 125) levels, and HE4 levels were obtained. All cases were analyzed and the histopathological diagnosis serves as the reference standard. Based on the IOTA ADNEX model post-processing software, the risk prediction value was calculated. We analyzed receiver operating characteristic curves to determine whether the IOTA ADNEX model alone or combined with HE4 provided better diagnostic accuracy.

Results

The area under the curve (AUC) of the ADNEX model alone or combined with HE4 in predicting benign and malignant ovarian tumors was 0.914 (95% CI, 0.881–0.941) and 0.916 (95% CI, 0.883–0.942), respectively. With the cutoff risk of 10%, the ADNEX model had a sensitivity of 0.93 (95% CI, 0.87–0.97) and a specificity of 0.73 (95% CI, 0.67–0.78), while combined with HE4, it had a sensitivity of 0.90 (95% CI, 0.84–0.95) and a specificity of 0.81 (95% CI, 0.76–0.86). The IOTA ADNEX model combined with HE4 was better at improving the accuracy of the differential diagnosis between different OCs than the IOTA ADNEX model alone. A significant difference was found in separating borderline masses from Stage II–IV OC (p = 0.0257).

Conclusions

A combination of the IOTA ADNEX model and HE4 can improve the specificity of diagnosis of ovarian benign and malignant tumors and increase the sensitivity and effectiveness of the differential diagnosis of Stage II–IV OC and borderline tumors.

Biomarkers in Gynecologic Tumors

Gynecologic malignancies are one of the most frequent cancers amongst women. Biomarkers are crucial for the differential diagnosis of adnexal masses; however, their potential for diagnosis is limited. In the era of difficulty in ovarian cancer screening, novel biomarkers are defined, but CA125 still remains the most valuable one. Circulating tumor DNAs, DNA hypermethylation, metabolites, microRNAs, and kallikreins have recently turned out as ovarian cancer biomarkers and are being applied to clinical practice. For uterine cancer, genomic classification has now been described, it will be used as a prognostic tool. In this chapter, we describe ovarian, endometrial, and cervical cancer biomarkers in detail.

External Validation of O-RADS US Risk Stratification and Management System

Background The Ovarian-Adnexal Reporting and Data System (O-RADS) US risk stratification and management system (O-RADS US) was designed to improve risk assessment and management of ovarian and adnexal lesions. Validation studies including both surgical and nonsurgical treatment as the reference standard remain lacking. Purpose To externally validate O-RADS US in women who underwent either surgical or nonsurgical treatment and to determine if incorporating acoustic shadowing as a benign finding improves diagnostic performance. Materials and Methods This retrospective study included consecutive women who underwent pelvic US between August 2015 and April 2017 at a tertiary referral oncology center. Two independent readers blinded to clinical and histologic outcome assigned an O-RADS risk category and an International Ovarian Tumor Analysis (IOTA) Assessment of Different NEoplasias in the adneXa (ADNEX) model risk of malignancy score to assessable lesions. Reference standards were surgical histopathology or 2-year imaging follow-up. Receiver operating characteristic (ROC) curve analysis was used to evaluate performance of the O-RADS US, ADNEX, and modified O-RADS models incorporating acoustic shadowing. Results In total, 227 women (mean age, 52 years ± 16 [SD]) with 262 ovarian or adnexal lesions were evaluated. Of these lesions, 187 (71%) were benign and 75 (29%) were malignant. The proportion of malignancy was 0% (0 of 100) for O-RADS 2, 3% (one of 32) for O-RADS 3, 35% (22 of 63) for O-RADS 4, and 78% (52 of 67) for O-RADS 5. The area under the ROC curve (AUC) for O-RADS and ADNEX was 0.91 (95% CI: 0.88, 0.94) and 0.95 (95% CI: 0.92, 0.97; P = .01), respectively. The addition of acoustic shadowing as a benign finding improved O-RADS AUC to 0.94 (95% CI: 0.91, 0.96; P = .01). Use of O-RADS 4 as a threshold yielded a sensitivity of 99% (74 of 75; 95% CI: 96, 100) and a specificity of 70% (131 of 187; 95% CI: 64, 77). Conclusion In a tertiary referral oncology center, the Ovarian-Adnexal Reporting and Data System US risk stratification and management system enabled accurate distinction of benign from malignant ovarian and adnexal lesions. Adding acoustic shadowing as a benign finding improved its diagnostic performance. © RSNA, 2022 See also the editorial by Levine in this issue.

Imaging of Endometriosis: The Role of Ultrasound and Magnetic Resonance

Endometriosis is a chronic gynecological disease characterized by the growth of functional ectopic endometrial glands and stroma outside the uterus. It causes pelvic pain, dysmenorrhea, dyspareunia, or infertility. Diagnosis requires a combination of clinical history, non-invasive and invasive techniques. The aim of the present review was to evaluate the contribution of imaging techniques, mainly transvaginal sonography and magnetic resonance imaging to diagnose different locations and for the most appropriate treatment planning. Endometriosis requires a multidisciplinary teamwork to manage these patients clinically and surgically.

Ultrasonographic evaluation of ovarian mass for predicting malignancy in pregnant women

OBJECTIVE

The purpose of this study is to compare ultrasonographic ovarian mass scoring systems in pregnant women.

STUDY DESIGN

This multicenter study included women with an ovarian mass during pregnancy who were evaluated using ultrasound and underwent surgery in 11 referral hospitals. The ovarian mass was evaluated and scored using three different scoring systems(International Ovarian Tumor Analysis Assessment of Different NEoplasias in the adnexa[IOTA ADNEX], Sassone, and Lerner). The final diagnosis was made histopathologically. Receiver operating characteristic(ROC) curves were generated for each scoring system.

RESULTS

During the study period, 236 pregnant women underwent surgery for an ovarian mass, including 223 women(94.5%) with a benign ovarian mass and 13 women(5.5%) with a malignant ovarian mass. Among 10 ultrasound image findings, six findings were different between benign and ovarian masses(maximal diameter of mass, maximal diameter of solid mass, wall thickness of mass, inner wall structure, thickness of septations, and papillarity). In all three scoring systems, the ovarian mass scores were significantly higher in malignant masses than in benign masses, with the highest area under the ROC curve(AUROC) in the Sassone scoring system(AUROC: 0.831 for Sassone, 0.710 for Lerner vs 0.709 for IOTA ADNEX; p < 0.05, between the Sassone and Lerner/ IOTA ADNEX). A combined model was developed with the six different ultrasound findings, and the AUROC of the combined model was 0.883(p = not significant between the combined model and Sassone).

CONCLUSION

In pregnant women, malignant ovarian tumors can be predicted with high accuracy using either the Sassone scoring system or the combined model.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on preoperative diagnosis of ovarian tumors

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the preoperative diagnosis of ovarian tumors, including imaging techniques, biomarkers and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the preoperative diagnosis of ovarian tumors and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the preoperative diagnosis of ovarian tumors and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on pre-operative diagnosis of ovarian tumors

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group, and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the pre-operative diagnosis of ovarian tumors, including imaging techniques, biomarkers, and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the pre-operative diagnosis of ovarian tumors and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when a consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the pre-operative diagnosis of ovarian tumors and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

Ultrasound evaluation of ovarian masses and assessment of the extension of ovarian malignancy

The current review sums up the literature on the diagnostic performance of models to predict malignancy in adnexal masses and the ability of ultrasound to make a specific diagnosis in adnexal masses. A summary of the role of ultrasound in assessing the extension of malignant ovarian disease is also provided.

ESGO/ISUOG/IOTA/ESGE Consensus Statement on preoperative diagnosis of ovarian tumours

The European Society of Gynaecological Oncology (ESGO), the International Society of Ultrasound in Obstetrics and Gynecology (ISUOG), the International Ovarian Tumour Analysis (IOTA) group and the European Society for Gynaecological Endoscopy (ESGE) jointly developed clinically relevant and evidence-based statements on the preoperative diagnosis of ovarian tumours, including imaging techniques, biomarkers and prediction models. ESGO/ISUOG/IOTA/ESGE nominated a multidisciplinary international group, including expert practising clinicians and researchers who have demonstrated leadership and expertise in the preoperative diagnosis of ovarian tumours and management of patients with ovarian cancer (19 experts across Europe). A patient representative was also included in the group. To ensure that the statements were evidence-based, the current literature was reviewed and critically appraised. Preliminary statements were drafted based on the review of the relevant literature. During a conference call, the whole group discussed each preliminary statement and a first round of voting was carried out. Statements were removed when a consensus among group members was not obtained. The voters had the opportunity to provide comments/suggestions with their votes. The statements were then revised accordingly. Another round of voting was carried out according to the same rules to allow the whole group to evaluate the revised version of the statements. The group achieved consensus on 18 statements. This Consensus Statement presents these ESGO/ISUOG/IOTA/ESGE statements on the preoperative diagnosis of ovarian tumours and the assessment of carcinomatosis, together with a summary of the evidence supporting each statement.

Terms, definitions and measurements to describe sonographic features of lymph nodes: consensus opinion from the Vulvar International Tumor Analysis (VITA) group

In centers with access to high-end ultrasound machines and expert sonologists, ultrasound is used to detect metastases in regional lymph nodes from melanoma, breast cancer and vulvar cancer. There is, as yet, no international consensus on ultrasound assessment of lymph nodes in any disease or medical condition. The lack of standardized ultrasound nomenclature to describe lymph nodes makes it difficult to compare results from different ultrasound studies and to find reliable ultrasound features for distinguishing non-infiltrated lymph nodes from lymph nodes infiltrated by cancer or lymphoma cells. The Vulvar International Tumor Analysis (VITA) collaborative group consists of gynecologists, gynecologic oncologists and radiologists with expertise in gynecologic cancer, particularly in the ultrasound staging and treatment of vulvar cancer. The work herein is a consensus opinion on terms, definitions and measurements which may be used to describe inguinal lymph nodes on grayscale and color/power Doppler ultrasound. The proposed nomenclature need not be limited to the description of inguinal lymph nodes as part of vulvar cancer staging; it can be used to describe peripheral lymph nodes in general, as well as non-peripheral (i.e. parietal or visceral) lymph nodes if these can be visualized clearly. The association between the ultrasound features described here and histopathological diagnosis has not yet been established. VITA terms and definitions lay the foundations for prospective studies aiming to identify ultrasound features typical of metastases and other pathology in lymph nodes and studies to elucidate the role of ultrasound in staging of vulvar and other malignancies. © 2021 International Society of Ultrasound in Obstetrics and Gynecology.

Diagnostic Added-Value of Serum CA-125 on the IOTA Simple Rules and Derivation of Practical Combined Prediction Models (IOTA SR X CA-125)

Background: This study aimed to evaluate the diagnostic added-value of serum CA-125 to the International Ovarian Tumor Analysis (IOTA) Simple Rules in order to facilitate differentiation between malignant and benign ovarian tumors before surgery. Methods: A secondary analysis of a cross-sectional cohort of women scheduled for surgery in Maharaj Nakorn Chiang Mai Hospital between April 2010 and March 2018 was carried out. Demographic and clinical data were prospectively collected. Histopathologic diagnosis was used as the reference standard. Logistic regression was used for development of the model. Evaluation of the diagnostic added-value was based on the increment of the area under the receiver operating characteristic curve (AuROC). Results: One hundred and forty-five women (30.3%) out of a total of 479 with adnexal masses had malignant ovarian tumors. The model that included information from the IOTA Simple Rules and serum CA-125 was significantly more superior to the model that used only information from the IOTA Simple Rules (AuROC 0.95 vs. 0.89, p < 0.001 for pre-menopause and AuROC 0.98 vs 0.83, p < 0.001 for post-menopause). Conclusions: The IOTA SR X CA-125 model showed high discriminative ability and is potentially useful as a decision tool for guiding patient referrals to oncologic specialists.

Transferability of the early-stage ovarian malignancy (EOM) score: an external validation study that includes advanced-stage and metastatic ovarian cancer

PURPOSE

To validate the diagnostic performance of the Early-stage Ovarian Malignancy (EOM) score in an external dataset that includes advanced-stage and metastatic ovarian cancer.

METHODS

The data from two cross-sectional cohorts were used in the statistical analysis. The development dataset of the EOM score was collected in Phrapokklao Hospital between September 2013 and December 2017. The validation dataset was collected in Maharaj Nakorn Chiang Mai Hospital between April 2010 and March 2018. The internal and external performance of the EOM score was evaluated in terms of discrimination via area under the receiver-operating characteristic curve (AuROC) and calibration.

RESULTS

There were 270 and 479 patients included in the development and validation datasets, respectively. The prevalence of ovarian malignancy was 20.0% (54/270) in the development set and 30.3% (145/479) in the validation set. The EOM score had excellent discriminative ability in both the development and validation sets (AuROC 88.0 (95% CI 82.6, 93.9) and 88.0 (95% CI 84.3, 91.4), respectively). The EOM score also showed good calibration in both datasets.

CONCLUSIONS

The EOM score had consistent diagnostic performance in the external validation data. It is recommended for use as a triage tool in patient referrals instead of the RMI in settings where experienced sonographers are not available.

Early-Stage Ovarian Malignancy Score versus Risk of Malignancy Indices: Accuracy and Clinical Utility for Preoperative Diagnosis of Women with Adnexal Masses

Background and objectives: To compare the diagnostic accuracy and clinical utility of the Early-stage Ovarian Malignancy (EOM) score with the Risk of Malignancy Index (RMI) in the presurgical assessment of women presenting with adnexal masses. Materials and Methods: A secondary analysis was carried out in a retrospective cohort of women who presented with an adnexal mass and were scheduled for surgery at Phrapokklao Hospital between September 2013 and December 2017. The clinical characteristics, ultrasonographic features of the masses, and preoperative CA-125 levels were recorded. The EOM and the RMI score were calculated and compared in terms of accuracy and clinical utility. Decision curve analysis (DCA), which examined the net benefit (NB) of applying the EOM and the RMI in practice at a range of threshold probabilities, was presented. Results: In this study, data from 270 patients were analyzed. Fifty-four (20.0%) women in the sample had early-stage ovarian cancer. All four RMI versions demonstrated a lower sensitivity for the detection of patients with early-stage ovarian cancer compared to an EOM score ≥ 15. An EOM ≥ 15 resulted in a higher proportion of net true positive or NB than all versions of the RMIs from a threshold probability of 5% to 30%. Conclusions: It also showed a higher capability to reduce the number of inappropriate referrals than the RMIs at a threshold probability between 5% and 30%. The EOM score showed higher diagnostic sensitivity and has the potential to be clinically more useful than the RMIs to triage women who present with adnexal masses for referral to oncologic gynecologists. Further external validation is required to support our findings.

Comparison of the four malignancy risk indices in the discrimination of malignant ovarian masses: A cross-sectional study

OBJECTIVE

To evaluate the accuracy of the four malignancy risk indices to distinguish benign from malignant ovarian masses MATERIALS AND METHODS: This was an observational cross-sectional study conducted on 155 patients between January 2016 and January 2019. Women with ovarian masses planned for surgical management were recruited from the outpatient Gynecology clinic of the hospital. The risk of malignancy index (RMI 1-4) was calculated for all women with ovarian masses. Biopsies obtained from the ovarian masses after the surgical intervention was sent to the pathology lab for histopathological examination. The histopathologic diagnosis of the ovarian masses was considered the gold standard for diagnosis.

RESULTS

The participants' mean age in the group of patients with benign masses was 33.50 ± 14.53 years versus 45.09 ± 13.67 years in the malignant group. The two most prominent features in the malignant group were solid areas in 85.3 % of malignant masses and about 91.2 % of malignant masses showing size <7 cm in their largest diameter. The RMI's most sensitive individual parameter was the CA-125 level, while the lowest sensitivity was for the menopausal status. RMI 2 had the highest sensitivity of 76.47 %, while RMI 1 and 3 had the highest specificity, 92.56 %. RMI 2 had the highest AUC, 0.83.

CONCLUSIONS

RMI 2 is a simple and reliable tool and had the best performance among all RMIs in benign discrimination from malignant ovarian masses with high sensitivity and accuracy.

Validation of models to diagnose ovarian cancer in patients managed surgically or conservatively: multicentre cohort study

OBJECTIVE

To evaluate the performance of diagnostic prediction models for ovarian malignancy in all patients with an ovarian mass managed surgically or conservatively.

DESIGN

Multicentre cohort study.

SETTING

36 oncology referral centres (tertiary centres with a specific gynaecological oncology unit) or other types of centre.

PARTICIPANTS

Consecutive adult patients presenting with an adnexal mass between January 2012 and March 2015 and managed by surgery or follow-up.

MAIN OUTCOME MEASURES

Overall and centre specific discrimination, calibration, and clinical utility of six prediction models for ovarian malignancy (risk of malignancy index (RMI), logistic regression model 2 (LR2), simple rules, simple rules risk model (SRRisk), assessment of different neoplasias in the adnexa (ADNEX) with or without CA125). ADNEX allows the risk of malignancy to be subdivided into risks of a borderline, stage I primary, stage II-IV primary, or secondary metastatic malignancy. The outcome was based on histology if patients underwent surgery, or on results of clinical and ultrasound follow-up at 12 (±2) months. Multiple imputation was used when outcome based on follow-up was uncertain.

RESULTS

The primary analysis included 17 centres that met strict quality criteria for surgical and follow-up data (5717 of all 8519 patients). 812 patients (14%) had a mass that was already in follow-up at study recruitment, therefore 4905 patients were included in the statistical analysis. The outcome was benign in 3441 (70%) patients and malignant in 978 (20%). Uncertain outcomes (486, 10%) were most often explained by limited follow-up information. The overall area under the receiver operating characteristic curve was highest for ADNEX with CA125 (0.94, 95% confidence interval 0.92 to 0.96), ADNEX without CA125 (0.94, 0.91 to 0.95) and SRRisk (0.94, 0.91 to 0.95), and lowest for RMI (0.89, 0.85 to 0.92). Calibration varied among centres for all models, however the ADNEX models and SRRisk were the best calibrated. Calibration of the estimated risks for the tumour subtypes was good for ADNEX irrespective of whether or not CA125 was included as a predictor. Overall clinical utility (net benefit) was highest for the ADNEX models and SRRisk, and lowest for RMI. For patients who received at least one follow-up scan (n=1958), overall area under the receiver operating characteristic curve ranged from 0.76 (95% confidence interval 0.66 to 0.84) for RMI to 0.89 (0.81 to 0.94) for ADNEX with CA125.

CONCLUSIONS

Our study found the ADNEX models and SRRisk are the best models to distinguish between benign and malignant masses in all patients presenting with an adnexal mass, including those managed conservatively.

TRIAL REGISTRATION

ClinicalTrials.gov NCT01698632.

Validation of IOTA-ADNEX Model in Discriminating Characteristics of Adnexal Masses: A Comparison with Subjective Assessment

(1) Background: The aim of this study is to compare the IOTA-ADNEX (international ovarian tumor analysis–assessment of different neoplasias in the adnexa) model with gynecologic experts in differentiating ovarian diseases. (2) Methods: All participants in this prospective study underwent ultrasonography (US) equipped with the IOTA-ADNEX^TM model and subjective assessment by a sonographic expert. Receiver operating characteristic (ROC) curves were also generated to compare overall accuracies. The optimal cut-off value of the ADNEX model for excluding benign diseases was calculated. (3) Results: Fifty-nine participants were eligible: 54 and 5 underwent surgery and follow-up computed tomography (CT), respectively. Benign and malignant diseases were confirmed in 49 (83.1%) and 10 (16.9%) participants, respectively. The specificity of the ADNEX model was 0.816 (95% confidence interval (CI): 0.680–0.912) in all participants and 0.795 (95% CI, 0.647–0.902) in the surgical group. The area under the ROC curve of the ADNEX model (0.924) was not significantly different from that of subjective assessment (0.953 in all participants, 0.951 in the surgical group; p = 0.391 in all participants, p = 0.407 in the surgical group). The optimal cut-off point using the ADNEX model was 47.3%, with a specificity of 0.977 (95% CI: 0.880–0.999). (4) Conclusions: The IOTA-ADNEX model is equal to gynecologic US experts in excluding benign ovarian tumors. Subsequently, being familiar with this US software may help gynecologic beginners to reduce unnecessary surgery.

Scoring systems for the evaluation of adnexal masses nature: current knowledge and clinical applications

Adnexal masses are a common finding in women, with 20% of them developing at least one pelvic mass during their lifetime. There are more than 30 different subtypes of adnexal tumours, with multiple different subcategories, and the correct characterisation of the pelvic masses is of paramount importance to guide the correct management. On that basis, different algorithms and scoring systems have been developed to guide the clinical assessment. The first scoring system implemented into the clinical practice was the Risk of Malignancy Index, which combines ultrasound evaluation, menopausal status, and serum CA-125 levels. Today, current guidelines regarding female patients with adnexal masses include the application of International Ovarian Tumours Analysis simple rules, logistic regression model 1 (LR1) and LR2, OVERA, cancer ovarii non-invasive assessment of treating strategy, and assessment of Different Neoplasias in the adnexa. In this scenario, the choice of the scoring system for the discrimination between benign and malignant ovarian tumours can be complex when approaching patients with adnexal masses. This review aims to summarise the available evidence regarding the different scoring systems to provide a complete overview of the topic.

Efficacy of HE4, CA125, Risk of Malignancy Index and Risk of Ovarian Malignancy Index to Detect Ovarian Cancer in Women with Presumed Benign Ovarian Tumours: A Prospective, Multicentre Trial

BACKGROUND

Presumed benign ovarian tumours (PBOT) are defined by the International Ovarian Tumour Analysis (IOTA) group, without suspected sonographic criteria of cancer, without ascites or metastasis. The aim is to evaluate the efficacy of human epididymis protein 4 (HE4), cancer antigen 125 (CA125), the risk of malignancy index (RMI) and the risk of ovarian malignancy index (ROMA) to predict ovarian cancer in women with PBOT.

METHODS

It is a prospective, observational, multicentre, laboratory-based study including women with PBOT in four hospitals from 11 May 2015 through 12 May 2016. Preoperative CA125 and HE4 plasma levels were measured for all women. The primary endpoint was the specificity of CA125 and HE4 for diagnosing ovarian cancer. The main secondary endpoints were specificity and likelihood ratio of RMI, ROMA and tumours markers.

RESULTS

Two hundred and fifty patients were initially enrolled and 221 patients were finally analysed, including 209 benign ovarian tumours (94.6%) and 12 malignant ovarian tumours (5.4%). The malignant group had significantly higher mean values of HE4, CA125, RMI and ROMA compared to the benign group (p < 0.001). Specificity was significantly higher using a combination of HE4 and CA125 (99.5%) compared to either HE4 or CA125 alone (90.4% and 91.4%, respectively, p < 0.001). Moreover, the positive likelihood ratio for combination HE4 and CA125 was significantly higher (104.5; 95% CI 13.6-800.0) compared to HE4 alone (5.81; 95% CI 2.83-11.90) or CA125 alone (6.97; 95% CI 3.91-12.41).

CONCLUSIONS

The combination of HE4 and CA125 represents the best tool to predict the risk of ovarian cancer in patients with a PBOT.

Methods of Assessing Ovarian Masses: International Ovarian Tumor Analysis Approach

There are many diagnostic methods to assist clinicians in assessing adnexal masses on ultrasound. After suggesting a standardized terminology and measurement technique to evaluate adnexal masses, the International Ovarian Tumor Analysis (IOTA) group has developed different strategies such as the Simple Rules and Assessment of Different Neoplasias in the adneXa (ADNEX) model, which have been shown to outperform other available methods. Besides differentiating between benign neoplasms and malignancies, the ADNEX model can also give the predicted risk for different subtypes of malignant adnexal masses, which is clinically very relevant for guiding patient management.

Prospective external validation of IOTA three-step strategy for characterizing and classifying adnexal masses and retrospective assessment of alternative two-step strategy using simple-rules risk

OBJECTIVES

To perform an external validation of the diagnostic performance of the three-step strategy proposed by the International Ovarian Tumor Analysis (IOTA) group for classifying adnexal masses as benign or malignant, when ultrasound is performed by non-expert sonographers in the first two steps. The second objective was to assess the diagnostic performance of an alternative strategy using simple-rules risk (SRR), instead of simple rules (SR), in the second step.

METHODS

This was a prospective observational study conducted at two university hospitals, from September 2015 to August 2017, of consecutive patients diagnosed with an adnexal mass. All women were evaluated by ultrasound using the IOTA three-step strategy. Non-expert sonographers performed the first step (use of simple descriptors to classify the masses) and the second step (use of SR if the mass could not be classified in the first step); masses that could not be classified in the first two steps were categorized by an expert sonographer based on their subjective assessment (third step). The reference standard was histological diagnosis in patients who underwent surgery or at least 12 months of follow-up in cases managed expectantly. The sensitivity, specificity, positive (LR+) and negative (LR-) likelihood ratios and overall accuracy of the IOTA three-step strategy were estimated. Furthermore, we evaluated retrospectively an alternative two-step strategy using SRR in the second step to categorize the masses not classifiable with simple descriptors, classifying the lesions as being of low, intermediate or high risk for malignancy. The diagnostic performance of this strategy was estimated by calculating its sensitivity and specificity, assuming surgical intervention for intermediate- or high-risk lesions.

RESULTS

The study included 283 patients (median age, 48 (range, 18-90) years), of whom 165 (58.3%) were premenopausal and 118 (41.7%) postmenopausal. Two hundred and sixteen (76.3%) women underwent surgery (154 benign and 62 malignant masses) and 67 (23.7%) were managed expectantly with serial ultrasound follow-up for at least 12 months. All expectantly managed masses were considered benign because no sonographic changes suggestive of malignancy were observed during follow-up. Simple descriptors could be applied in 126 (44.5%) masses. Of the remaining 157 lesions, 112 (39.6%) could be characterized using SR. Therefore, 238 (84.1%) masses could be classified by non-expert sonographers in the first two steps. Of the remaining 45 (15.9%) masses, all could be classified by an expert sonographer. Overall sensitivity, specificity, LR+ and LR- of the IOTA three-step strategy were 95.2%, 97.7%, 42.1 and 0.05, respectively. The diagnostic accuracy was 97.2%. Following the two-step strategy using SRR in the second step, of the 157 lesions not classified with simple descriptors, 42, 38 and 77 presented low, intermediate or high risk for malignancy, respectively. Based on this method, 210 women would have undergone surgical treatment. The sensitivity and specificity of this two-step strategy were 98.4% and 63.8%, respectively.

CONCLUSIONS

The IOTA three-step strategy shows high accuracy for discriminating between benign and malignant adnexal lesions when used by non-expert sonographers. An alternative strategy using the SRR calculator in the second step might improve on this diagnostic performance by decreasing the number of surgical interventions and increasing sensitivity. Copyright © 2018 ISUOG. Published by John Wiley & Sons Ltd.

Biomarkers and algorithms for diagnosis of ovarian cancer: CA125, HE4, RMI and ROMA, a review

Ovarian cancer is the 5th leading cause of death for women with cancer worldwide. In more than 70% of cases, it is only diagnosed at an advanced stage. Our study aims to give an update on the biological markers for diagnosing ovarian cancer, specifically HE4, CA 125, RMI and ROMA algorithms.

Serum CA125 assay has low sensitivity in the early stages and can be increased in certain conditions such as menstruation or endometriosis. The level of HE4 is overexpressed in ovarian tumors. Its specificity is 94% and its level is not affected by endometriosis cysts. The combined measures of CA125 and HE4 have proved to be highly efficient with an area under the curve (AUC) of up to 0.96. Furthermore, this combined measure of CA125 can correct the variations in HE4 which are due to smoking or contraception combining estrogen plus progestin. While the specificity of RMI sometimes reaches 92%, the rather low AUC of 0.86 does not make it the best diagnostic tool. The specificity of ROMA is lower than HE4 (84% compared to 94%).

To date, the most efficient biological diagnostic tool to diagnose ovarian cancer is the combination of CA125 and HE4.

Random-effects meta-analysis of the clinical utility of tests and prediction models

The use of data from multiple studies or centers for the validation of a clinical test or a multivariable prediction model allows researchers to investigate the test's/model's performance in multiple settings and populations. Recently, meta-analytic techniques have been proposed to summarize discrimination and calibration across study populations. Here, we rather consider performance in terms of net benefit, which is a measure of clinical utility that weighs the benefits of true positive classifications against the harms of false positives. We posit that it is important to examine clinical utility across multiple settings of interest. This requires a suitable meta-analysis method, and we propose a Bayesian trivariate random-effects meta-analysis of sensitivity, specificity, and prevalence. Across a range of chosen harm-to-benefit ratios, this provides a summary measure of net benefit, a prediction interval, and an estimate of the probability that the test/model is clinically useful in a new setting. In addition, the prediction interval and probability of usefulness can be calculated conditional on the known prevalence in a new setting. The proposed methods are illustrated by 2 case studies: one on the meta-analysis of published studies on ear thermometry to diagnose fever in children and one on the validation of a multivariable clinical risk prediction model for the diagnosis of ovarian cancer in a multicenter dataset. Crucially, in both case studies the clinical utility of the test/model was heterogeneous across settings, limiting its usefulness in practice. This emphasizes that heterogeneity in clinical utility should be assessed before a test/model is routinely implemented.

Performance of the IOTA ADNEX model in preoperative discrimination of adnexal masses in a gynecological oncology center

OBJECTIVE

To evaluate the performance of the International Ovarian Tumor Analysis (IOTA) ADNEX model in the preoperative discrimination between benign ovarian (including tubal and para-ovarian) tumors, borderline ovarian tumors (BOT), Stage I ovarian cancer (OC), Stage II-IV OC and ovarian metastasis in a gynecological oncology center in Brazil.

METHODS

This was a diagnostic accuracy study including 131 women with an adnexal mass invited to participate between February 2014 and November 2015. Before surgery, pelvic ultrasound examination was performed and serum levels of tumor marker CA 125 were measured in all women. Adnexal masses were classified according to the IOTA ADNEX model. Histopathological diagnosis was the gold standard. Receiver-operating characteristics (ROC) curve analysis was used to determine the diagnostic accuracy of the model to classify tumors into different histological types.

RESULTS

Of 131 women, 63 (48.1%) had a benign ovarian tumor, 16 (12.2%) had a BOT, 17 (13.0%) had Stage I OC, 24 (18.3%) had Stage II-IV OC and 11 (8.4%) had ovarian metastasis. The area under the ROC curve (AUC) was 0.92 (95% CI, 0.88-0.97) for the basic discrimination between benign vs malignant tumors using the IOTA ADNEX model. Performance was high for the discrimination between benign vs Stage II-IV OC, BOT vs Stage II-IV OC and Stage I OC vs Stage II-IV OC, with AUCs of 0.99, 0.97 and 0.94, respectively. Performance was poor for the differentiation between BOT vs Stage I OC and between Stage I OC vs ovarian metastasis with AUCs of 0.64.

CONCLUSION

The majority of adnexal masses in our study were classified correctly using the IOTA ADNEX model. On the basis of our findings, we would expect the model to aid in the management of women with an adnexal mass presenting to a gynecological oncology center. Copyright © 2016 ISUOG. Published by John Wiley & Sons Ltd.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model: a multicentre external validation study

BACKGROUND

The International Ovarian Tumour Analysis (IOTA) group have developed the ADNEX (The Assessment of Different NEoplasias in the adneXa) model to predict the risk that an ovarian mass is benign, borderline, stage I, stages II-IV or metastatic. We aimed to externally validate the ADNEX model in the hands of examiners with varied training and experience.

METHODS

This was a multicentre cross-sectional cohort study for diagnostic accuracy. Patients were recruited from three cancer centres in Europe. Patients who underwent transvaginal ultrasonography and had a histological diagnosis of surgically removed tissue were included. The diagnostic performance of the ADNEX model with and without the use of CA125 as a predictor was calculated.

RESULTS

Data from 610 women were analysed. The overall prevalence of malignancy was 30%. The area under the receiver operator curve (AUC) for the ADNEX diagnostic performance to differentiate between benign and malignant masses was 0.937 (95% CI: 0.915-0.954) when CA125 was included, and 0.925 (95% CI: 0.902-0.943) when CA125 was excluded. The calibration plots suggest good correspondence between the total predicted risk of malignancy and the observed proportion of malignancies. The model showed good discrimination between the different subtypes.

CONCLUSIONS

The performance of the ADNEX model retains its performance on external validation in the hands of ultrasound examiners with varied training and experience.

Past, present and future ultrasonographic techniques for analyzing ovarian masses

Ultrasonography is today the method of choice for distinguishing between benign and malignant adnexal pathologies. Using pattern recognition several types of tumors can be recognized according to their characteristic appearance on gray-scale imaging. Color Doppler imaging should be used only to perform a semiquantitative color score or evaluate the flow location. International Ovarian Tumor Analysis group had standardized definitions characterizing adnexal masses and suggested the use of 'simple rules' in premenopausal women. Recently, the use of 3D vascular indices has been proposed but its potential use in clinical practice is debated. Also computerized aided diagnosis algorithms showed encouraging results to be confirmed in the future.

Predicting the risk of malignancy in adnexal masses based on the Simple Rules from the International Ovarian Tumor Analysis group

BACKGROUND

Accurate methods to preoperatively characterize adnexal tumors are pivotal for optimal patient management. A recent metaanalysis concluded that the International Ovarian Tumor Analysis algorithms such as the Simple Rules are the best approaches to preoperatively classify adnexal masses as benign or malignant.

OBJECTIVE

We sought to develop and validate a model to predict the risk of malignancy in adnexal masses using the ultrasound features in the Simple Rules.

STUDY DESIGN

This was an international cross-sectional cohort study involving 22 oncology centers, referral centers for ultrasonography, and general hospitals. We included consecutive patients with an adnexal tumor who underwent a standardized transvaginal ultrasound examination and were selected for surgery. Data on 5020 patients were recorded in 3 phases from 2002 through 2012. The 5 Simple Rules features indicative of a benign tumor (B-features) and the 5 features indicative of malignancy (M-features) are based on the presence of ascites, tumor morphology, and degree of vascularity at ultrasonography. Gold standard was the histopathologic diagnosis of the adnexal mass (pathologist blinded to ultrasound findings). Logistic regression analysis was used to estimate the risk of malignancy based on the 10 ultrasound features and type of center. The diagnostic performance was evaluated by area under the receiver operating characteristic curve, sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), positive predictive value (PPV), negative predictive value (NPV), and calibration curves.

RESULTS

Data on 4848 patients were analyzed. The malignancy rate was 43% (1402/3263) in oncology centers and 17% (263/1585) in other centers. The area under the receiver operating characteristic curve on validation data was very similar in oncology centers (0.917; 95% confidence interval, 0.901-0.931) and other centers (0.916; 95% confidence interval, 0.873-0.945). Risk estimates showed good calibration. In all, 23% of patients in the validation data set had a very low estimated risk (<1%) and 48% had a high estimated risk (≥30%). For the 1% risk cutoff, sensitivity was 99.7%, specificity 33.7%, LR+ 1.5, LR- 0.010, PPV 44.8%, and NPV 98.9%. For the 30% risk cutoff, sensitivity was 89.0%, specificity 84.7%, LR+ 5.8, LR- 0.13, PPV 75.4%, and NPV 93.9%.

CONCLUSION

Quantification of the risk of malignancy based on the Simple Rules has good diagnostic performance both in oncology centers and other centers. A simple classification based on these risk estimates may form the basis of a clinical management system. Patients with a high risk may benefit from surgery by a gynecological oncologist, while patients with a lower risk may be managed locally.

Usefulness of diagnostic indices comprising clinical, sonographic, and biomarker data for discriminating benign from malignant ovarian masses

The objective of this study was to review the accuracy of indices combining several diagnostic variables, in comparison to other models, sonography alone, and biomarker assays, for predicting benign or malignant ovarian lesions. Different single modalities were reviewed. The most useful complex models were International Ovarian Tumor Analysis (IOTA) sonographic logistic regression model 2 (area under the curve, 0.949), risk of malignancy index-cancer antigen 125-human epididymis protein 4 (0.950), risk of malignancy algorithm (0.953), pelvic mass score (0.960), non-IOTA logistic regression model (0.970), and histoscanning score logistic regression model (0.970). None of the indices was superior to an expert subjective sonographic assessment (0.968). For women with adnexal tumors, indices with high accuracy are available that are applicable in clinical practice and comparable to an expert subjective sonographic assessment for discriminating benign from malignant masses.

Phenotip - a web-based instrument to help diagnosing fetal syndromes antenatally

Prenatal ultrasound can often reliably distinguish fetal anatomic anomalies, particularly in the hands of an experienced ultrasonographer. Given the large number of existing syndromes and the significant overlap in prenatal findings, antenatal differentiation for syndrome diagnosis is difficult. We constructed a hierarchic tree of 1140 sonographic markers and submarkers, organized per organ system. Subsequently, a database of prenatally diagnosable syndromes was built. An internet-based search engine was then designed to search the syndrome database based on a single or multiple sonographic markers. Future developments will include a database with magnetic resonance imaging findings as well as further refinements in the search engine to allow prioritization based on incidence of syndromes and markers.

Intra- and interobserver agreement with regard to describing adnexal masses using International Ovarian Tumor Analysis terminology: reproducibility study involving seven observers

OBJECTIVES

To estimate intraobserver repeatability and interobserver agreement in assessing the presence of papillary projections in adnexal masses and in classifying adnexal masses using the International Ovarian Tumor Analysis terminology for ultrasound examiners with different levels of experience. We also aimed to identify ultrasound findings that cause confusion and might be interpreted differently by different observers, and to determine if repeatability and agreement change after consensus has been reached on how to interpret 'problematic' ultrasound images.

METHODS

Digital clips (two to eight clips per adnexal mass) with gray-scale and color/power Doppler information of 83 adnexal masses in 80 patients were evaluated independently four times, twice before and twice after a consensus meeting, by four experienced and three less experienced ultrasound observers. The variables analyzed were tumor type (unilocular, unilocular solid, multilocular, multilocular solid, solid) and presence of papillary projections. Intraobserver repeatability was evaluated for each observer (percentage agreement, Cohen's kappa). Interobserver agreement was estimated for all seven observers (percentage agreement, Fleiss kappa, Cohen's kappa).

RESULTS

There was uncertainty about how to define a solid component and a papillary projection, but consensus was reached at the consensus meeting. Interobserver agreement for tumor type was good both before and after the consensus meeting, with no clear improvement after the meeting, mean percentage agreement being 76.0% (Fleiss kappa, 0.695) before the meeting and 75.4% (Fleiss kappa, 0.682) after the meeting. Interobserver agreement with regard to papillary projections was moderate both before and after the consensus meeting, with no clear improvement after the meeting, mean percentage agreement being 86.6% (Fleiss kappa, 0.536) before the meeting and 82.7% (Fleiss kappa, 0.487) after it. There was substantial variability in pairwise agreement for papillary projections (Cohen's kappa, 0.148-0.787). Intraobserver repeatability with regard to tumor type was very good and similar before and after the consensus meeting (agreement 87-95%, kappa, 0.83-0.94). With regard to papillary projections intraobserver repeatability was good or very good both before and after the consensus meeting (agreement 88-100%, kappa, 0.64-1.0).

CONCLUSIONS

Despite uncertainty about how to define solid components, interobserver agreement was good for tumor type. The interobserver agreement for papillary projection was moderate but very variable between observer pairs. The term 'papillary projection' might need a more precise definition. The consensus meeting did not change inter- or intraobserver agreement.

Strategies to diagnose ovarian cancer: new evidence from phase 3 of the multicentre international IOTA study

Background:

To compare different ultrasound-based international ovarian tumour analysis (IOTA) strategies and risk of malignancy index (RMI) for ovarian cancer diagnosis using a meta-analysis approach of centre-specific data from IOTA3.

Methods:

This prospective multicentre diagnostic accuracy study included 2403 patients with 1423 benign and 980 malignant adnexal masses from 2009 until 2012. All patients underwent standardised transvaginal ultrasonography. Test performance of RMI, subjective assessment (SA) of ultrasound findings, two IOTA risk models (LR1 and LR2), and strategies involving combinations of IOTA simple rules (SRs), simple descriptors (SDs) and LR2 with and without SA was estimated using a meta-analysis approach. Reference standard was histology after surgery.

Results:

The areas under the receiver operator characteristic curves of LR1, LR2, SA and RMI were 0.930 (0.917–0.942), 0.918 (0.905–0.930), 0.914 (0.886–0.936) and 0.875 (0.853–0.894). Diagnostic one-step and two-step strategies using LR1, LR2, SR and SD achieved summary estimates for sensitivity 90–96%, specificity 74–79% and diagnostic odds ratio (DOR) 32.8–50.5. Adding SA when IOTA methods yielded equivocal results improved performance (DOR 57.6–75.7). Risk of Malignancy Index had sensitivity 67%, specificity 91% and DOR 17.5.

Conclusions:

This study shows all IOTA strategies had excellent diagnostic performance in comparison with RMI. The IOTA strategy chosen may be determined by clinical preference.

An ultrasound algorithm for identification of endometrial cancer

OBJECTIVE

To propose a scoring system to predict endometrial cancer using different ultrasound image characteristics at gray-scale, with and without enhancement by gel infusion, and Doppler transvaginal sonography (TVS) and to evaluate intra- and interobserver variability in assessment of these characteristics.

METHOD

Unenhanced TVS, Doppler examinations and gel infusion sonography (GIS) were performed prospectively in 174 consecutive postmenopausal women with endometrial thickness ≥ 5 mm. The reference standard in all women was hysteroscopy or hysterectomy with pathological evaluation of the malignancy. The presence of various ultrasound pattern characteristics indicative of endometrial malignancy and intra- and interobserver variability in their assessment were evaluated. Multivariate logistic regression was used to correlate image and clinical parameters to presence of endometrial cancer.

RESULTS

A simple Doppler flow score (which considered only presence of vascularity and not presence of single/double dominant vessel, multiple vessels, large vessels, color splash or densely packed vessels) had an area under the receiver-operating characteristics curve (AUC) of 0.83 in the prediction of endometrial cancer. Models including endometrial thickness, Doppler score and interrupted endomyometrial junction on unenhanced TVS predicted endometrial cancer with an AUC of 0.95 (95% CI, 0.92-0.99) and, with addition of irregular surface on GIS, the AUC was 0.97 (95% CI, 0.94-0.99). A risk of endometrial cancer (REC) scoring system based on body mass index, Doppler score, endometrial thickness and interrupted endomyometrial junction on unenhanced TVS and irregular surface at GIS performed very well at identifying endometrial cancer; at a REC-score of ≥ 4 the sensitivity for detection of endometrial cancer was 91% and specificity was 94%. Observers agreed in 82.3% of cases (kappa, 0.63 (0.48-0.78)) when subjective parameters were analyzed in stored videoclips.

CONCLUSION

Our observer-dependent proposed scoring system seems to perform well in the prediction of endometrial cancer and should be tested in future studies.

Imaging techniques for the pre-surgical diagnosis of adnexal tumours

A correct diagnosis of any adnexal mass is essential to triage women to appropriate treatment pathways. Several imaging techniques are available that may be used to provide an assessment of a mass before treatment, such as transvaginal ultrasonography, magnetic resonance imaging, computed tomography, and positron emission tomography combined with computed tomography. In this chapter, we focus in depth on the role of transvaginal ultrasonography, as current evidence suggests it is the most appropriate initial imaging investigation to identify and characterise any mass if present in women suspected of having adnexal pathology. Subjective assessment by an experienced ultrasound examiner is the optimal approach to diagnose masses, followed by risk models and rules developed by the International Ovarian Tumor Analysis study. A group of tumours has proven difficult to classify with transvaginal ultrasound, and remain a diagnostic challenge for which accurate second-stage tests would be of value. Some studies suggest that magnetic resonance imaging (MRI), compared with other imaging modalities, may play a role in the assessment of this cohort of 'difficult to classify' adnexal masses. These studies, however, did not report quality of transvaginal ultrasonography (i.e. experience level of the examiner) and lacked uniformity in describing the criteria used to define such 'difficult' masses. On the basis of standardised terminology developed by the International Ovarian Tumor Analysis study to describe adnexal masses, as well as prediction models and rules developed in the course of the study, we propose new criteria that we can use to clearly define complex or 'difficult to classify' adnexal masses to focus the role for second-line imaging tests, such as conventional magnetic resonance imaging combined with dynamic contrast-enhanced or diffusion-weighted sequences on masses where further tests other than ultrasonography would be of value.

Sensitivity and Specificity Can Change in Opposite Directions When New Predictive Markers Are Added to Risk Models

When comparing prediction models, it is essential to estimate the magnitude of change in performance rather than rely solely on statistical significance. In this paper we investigate measures that estimate change in classification performance, assuming 2-group classification based on a single risk threshold. We study the value of a new biomarker when added to a baseline risk prediction model. First, simulated data are used to investigate the change in sensitivity and specificity (ΔSe and ΔSp). Second, the influence of ΔSe and ΔSp on the net reclassification improvement (NRI; sum of ΔSe and ΔSp) and on decision-analytic measures (net benefit or relative utility) is studied. We assume normal distributions for the predictors and assume correctly specified models such that the extended model has a dominating receiver operating characteristic curve relative to the baseline model. Remarkably, we observe that even when a strong marker is added it is possible that either sensitivity (for thresholds below the event rate) or specificity (for thresholds above the event rate) decreases. In these cases, decision-analytic measures provide more modest support for improved classification than NRI, even though all measures confirm that adding the marker improved classification accuracy. Our results underscore the necessity of reporting ΔSe and ΔSp separately. When a single summary is desired, decision-analytic measures allow for a simple incorporation of the misclassification costs.

Presurgical diagnosis of adnexal tumours using mathematical models and scoring systems: a systematic review and meta-analysis

BACKGROUND Characterizing ovarian pathology is fundamental to optimizing management in both pre- and post-menopausal women. Inappropriate referral to oncology services can lead to unnecessary surgery or overly radical interventions compromising fertility in young women, whilst the consequences of failing to recognize cancer significantly impact on prognosis. By reflecting on recent developments of new diagnostic tests for preoperative identification of malignant disease in women with adnexal masses, we aimed to update a previous systematic review and meta-analysis. METHODS An extended search was performed in MEDLINE (PubMed) and EMBASE (OvidSp) from March 2008 to October 2013. Eligible studies provided information on diagnostic test performance of models, designed to predict ovarian cancer in a preoperative setting, that contained at least two variables. Study selection and extraction of study characteristics, types of bias, and test performance was performed independently by two reviewers. Quality was assessed using a modified version of the QUADAS assessment tool. A bivariate hierarchical random effects model was used to produce summary estimates of sensitivity and specificity with 95% confidence intervals or plot summary ROC curves for all models considered. RESULTS Our extended search identified a total of 1542 new primary articles. In total, 195 studies were eligible for qualitative data synthesis, and 96 validation studies reporting on 19 different prediction models met the predefined criteria for quantitative data synthesis. These models were tested on 26 438 adnexal masses, including 7199 (27%) malignant and 19 239 (73%) benign masses. The Risk of Malignancy Index (RMI) was the most frequently validated model. The logistic regression model LR2 with a risk cut-off of 10% and Simple Rules (SR), both developed by the International Ovarian Tumor Analysis (IOTA) study, performed better than all other included models with a pooled sensitivity and specificity, respectively, of 0.92 [95% CI 0.88-0.95] and 0.83 [95% CI 0.77-0.88] for LR2 and 0.93 [95% CI 0.89-0.95] and 0.81 [95% CI 0.76-0.85] for SR. A meta-analysis of centre-specific results stratified for menopausal status of two multicentre cohorts comparing LR2, SR and RMI-1 (using a cut-off of 200) showed a pooled sensitivity and specificity in premenopausal women for LR2 of 0.85 [95% CI 0.75-0.91] and 0.91 [95% CI 0.83-0.96] compared with 0.93 [95% CI 0.84-0.97] and 0.83 [95% CI 0.73-0.90] for SR and 0.44 [95% CI 0.28-0.62] and 0.95 [95% CI 0.90-0.97] for RMI-1. In post-menopausal women, sensitivity and specificity of LR2, SR and RMI-1 were 0.94 [95% CI 0.89-0.97] and 0.70 [95% CI 0.62-0.77], 0.93 [95% CI 0.88-0.96] and 0.76 [95% CI 0.69-0.82], and 0.79 [95% CI 0.72-0.85] and 0.90 [95% CI 0.84-0.94], respectively. CONCLUSIONS An evidence-based approach to the preoperative characterization of any adnexal mass should incorporate the use of IOTA Simple Rules or the LR2 model, particularly for women of reproductive age.

The special role of ultrasound for screening, staging and surveillance of malignant ovarian tumors: distinction from other methods of diagnostic imaging

Ovarian cancer is the most aggressive gynecologic malignancy, with a 5-year survival rate ranging around 40%. A crucial factor influencing the prognosis is early detection of a suspicious mass and referral to a gynecologic oncology center for further diagnosis, staging and debulking surgery. Here, we present the different imaging methods ultrasound (US), magnetic resonance imaging, computer tomography (CT) and 18F-fluoro-deoxyglucose positron emission tomography (PET)/CT that are used for the characterization, diagnosis, staging and surveillance of ovarian cancer. In this review, we focus on US and discuss in detail the advantages and the limitations, as well as the appropriate indications for each of the individual imaging techniques.

Screening for data clustering in multicenter studies: the residual intraclass correlation

Background

In multicenter studies, center-specific variations in measurements may arise for various reasons, such as low interrater reliability, differences in equipment, deviations from the protocol, sociocultural characteristics, and differences in patient populations due to e.g. local referral patterns. The aim of this research is to derive measures for the degree of clustering. We present a method to detect heavily clustered variables and to identify physicians with outlying measurements.

Methods

We use regression models with fixed effects to account for patient case-mix and a random cluster intercept to study clustering by physicians. We propose to use the residual intraclass correlation (RICC), the proportion of residual variance that is situated at the cluster level, to detect variables that are influenced by clustering. An RICC of 0 indicates that the variance in the measurements is not due to variation between clusters. We further suggest, where appropriate, to evaluate RICC in combination with R², the proportion of variance that is explained by the fixed effects. Variables with a high R² may have benefits that outweigh the disadvantages of clustering in terms of statistical analysis. We apply the proposed methods to a dataset collected for the development of models for ovarian tumor diagnosis. We study the variability in 18 tumor characteristics collected through ultrasound examination, 4 patient characteristics, and the serum marker CA-125 measured by 40 physicians on 2407 patients.

Results

The RICC showed large variation between variables: from 2.2% for age to 25.1% for the amount of fluid in the pouch of Douglas. Seven variables had an RICC above 15%, indicating that a considerable part of the variance is due to systematic differences at the physician level, rather than random differences at the patient level. Accounting for differences in ultrasound machine quality reduced the RICC for a number of blood flow measurements.

Conclusions

We recommend that the degree of data clustering is addressed during the monitoring and analysis of multicenter studies. The RICC is a useful tool that expresses the degree of clustering as a percentage. Specific applications are data quality monitoring and variable screening prior to the development of a prediction model.

Multicentre external validation of IOTA prediction models and RMI by operators with varied training

Background:

Correct characterisation of ovarian tumours is critical to optimise patient care. The purpose of this study is to evaluate the diagnostic performance of the International Ovarian Tumour Analysis (IOTA) logistic regression model (LR2), ultrasound Simple Rules (SR), the Risk of Malignancy Index (RMI) and subjective assessment (SA) for preoperative characterisation of adnexal masses, when ultrasonography is performed by examiners with different background training and experience.

Methods:

A 2-year prospective multicentre cross-sectional study. Thirty-five level II ultrasound examiners contributed in three UK hospitals. Transvaginal ultrasonography was performed using a standardised approach. The final outcome was the surgical findings and histological diagnosis. To characterise the adnexal masses, the six-variable prediction model (LR2) with a cutoff of 0.1, the RMI with cutoff of 200, ten SR (five rules for malignancy and five rules for benignity) and SA were applied. The area under the curves (AUCs) for performance of LR2 and RMI were calculated. Diagnostic performance measures for all models assessed were sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR−), and the diagnostic odds ratio (DOR).

Results:

Nine-hundred and sixty-two women with adnexal masses underwent transvaginal ultrasonography, whereas 255 had surgery. Prevalence of malignancy was 29% (49 primary invasive epithelial ovarian cancers, 18 borderline ovarian tumours, and 7 metastatic tumours). The AUCs for LR2 and RMI for all masses were 0.94 (95% confidence interval (CI): 0.89–0.97) and 0.90 (95% CI: 0.83–0.94), respectively. In premenopausal women, LR2−RMI difference was 0.09 (95% CI: 0.03–0.15) compared with −0.02 (95% CI: −0.08 to 0.04) in postmenopausal women. For all masses, the DORs for LR2, RMI, SR+SA (using SA when SR inapplicable), SR+MA (assuming malignancy when SR inapplicable), and SA were 62 (95% CI: 27–142), 43 (95% CI: 19–97), 109 (95% CI: 44–274), 66 (95% CI: 27–158), and 70 (95% CI: 30–163), respectively.

Conclusion:

Overall, the test performance of IOTA prediction models and rules as well as the RMI was maintained in examiners with varying levels of training and experience.

A multicenter prospective external validation of the diagnostic performance of IOTA simple descriptors and rules to characterize ovarian masses

OBJECTIVES

To evaluate the diagnostic performance of the IOTA (International Ovarian Tumor Analysis group) (clinically oriented three-step strategy for preoperative characterization of ovarian masses when ultrasonography is performed by examiners with different background training and experience.

METHODS

A 27-month prospective multicenter cross-sectional study was performed. 36 level II ultrasound examiners contributed in three UK hospitals. Transvaginal ultrasonography was performed using a standardized approach. Step one uses simple descriptors (SD), step two ultrasound simple rules (SR) and step three subjective assessment of ultrasound images (SA) by examiners. The final outcome was findings at surgery and the histological diagnosis of surgically removed masses.

RESULTS

1165 women with adnexal masses underwent transvaginal ultrasonography, 301 had surgery. Prevalence of malignancy was 31% (n=92). SD were able to classify 46% of the masses into benign or malignant (step one), with a sensitivity of 93% and specificity of 97%. Applying SD followed by SR to residual unclassified masses by SD enabled 89% of all masses (n=268) to be classified with a sensitivity 95% of and specificity of 95%. SA was then used to evaluate the rest of the masses. Compared to the risk of malignancy index (RMI), the sensitivity and specificity for the three-step (SD+SR+SA) strategy were 93% (95% CI: 86-97%) and 92% (95% CI: 87-95%) vs. 72% (95% CI: 62-80%) and 95% (95% CI: 91-97%) for RMI, respectively.

CONCLUSION

The IOTA three-step strategy shows good test performance on external validation in the hands of ultrasonography examiners with different background training and experience. This performance is considerably better than the RMI.