Do clinical prediction models improve concordance of treatment decisions in reproductive medicine?

Development and validation of a nomogram for predicting ongoing pregnancy in single vitrified-warmed blastocyst embryo transfer cycles

Introduction

The global adoption of the "freeze-all strategy" has led to a continuous increase in utilization of single vitrified-warmed blastocyst embryo transfer (SVBT) owing to its clinical effectiveness. Accurate prediction of clinical pregnancy is crucial from a patient-centered perspective. However, this remains challenging, with inherent limitations due to the absence of precise and user-friendly prediction tools. Thus, this study primarily aimed to develop and assess a nomogram based on quantitative clinical data to optimize the efficacy of personalized prognosis assessment.

Materials and methods

We conducted a retrospective cohort analysis of ongoing pregnancy data from 658 patients with infertility who underwent SVBT at our center between October 17, 2017, and December 18, 2021. Patients were randomly assigned to the training (n=461) or validation (n=197) cohort for nomogram development and testing, respectively. A nomogram was constructed using the results of the multivariable logistic regression (MLR), which included clinical covariates that were assessed for their association with ongoing pregnancy.

Results

The MLR identified eight significant variables that independently predicted ongoing pregnancy outcomes in the study population. These predictors encompassed maternal physiology, including maternal age at oocyte retrieval and serum anti-Müllerian hormone levels; uterine factors, such as adenomyosis; and various embryo assessment parameters, including the number of fertilized embryos, blastocyst morphology, blastulation day, blastocyst re-expansion speed, and presence of embryo string. The area under the receiver operating characteristic curve in our prediction model was 0.675 (95% confidence interval [CI], 0.622-0.729) and 0.656 (95% CI, 0.573-0.739) in the training and validation cohorts, respectively, indicating good discrimination performance in both cohorts.

Conclusions

Our individualized nomogram is a practical and user-friendly tool that can provide accurate and useful SVBT information for patients and clinicians. By offering this model to patients, clinical stakeholders can alleviate uncertainty and confusion about fertility treatment options and enhance patients' confidence in making informed decisions.

Real-world evidence analysis of the follicle-stimulating hormone use in male idiopathic infertility

Male idiopathic infertility remains a therapeutic challenge in the couple infertility management. In this setting, an empirical treatment with follicle-stimulating hormone (FSH) is allowed, although not recommended. Twenty-one clinical trials and four meta-analyses highlighted an overall increased pregnancy rate in case of FSH administration, but the indiscriminate FSH prescription is still unsupported by clinical evidence in idiopathic infertility. This context could represent an example in which real-world data (RWD) could add useful information. From a nationwide clinical practice survey performed in Italy, emerged the clinicians' attitude to prescribe FSH in the case of impaired semen with a significant improvement of semen parameters, identifying FSH treatment as a therapeutic card in the real-life management. Although more robust data are still needed to optimize FSH treatment in male idiopathic infertility, RWD should be included in the body of evidence considered in healthcare decision-making.

Predicting cumulative live birth for couples beginning their second complete cycle of in vitro fertilization treatment

STUDY QUESTION

Can we develop an IVF prediction model to estimate individualized chances of a live birth over multiple complete cycles of IVF in couples embarking on their second complete cycle of treatment?

SUMMARY ANSWER

Yes, our prediction model can estimate individualized chances of cumulative live birth over three additional complete cycles of IVF.

WHAT IS KNOWN ALREADY

After the completion of a first complete cycle of IVF, couples who are unsuccessful may choose to undergo further treatment to have their first child, while those who have had a live birth may decide to have more children. Existing prediction models can estimate the overall chances of success in couples before commencing IVF but are unable to revise these chances on the basis of the couple’s response to a first treatment cycle in terms of the number of eggs retrieved and pregnancy outcome. This makes it difficult for couples to plan and prepare emotionally and financially for the next step in their treatment.

STUDY DESIGN, SIZE, DURATION

For model development, a population-based cohort was used of 49 314 women who started their second cycle of IVF including ICSI in the UK from 1999 to 2008 using their own oocytes and their partners’ sperm. External validation was performed on data from 39 442 women who underwent their second cycle from 2010 to 2016.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Data about all UK IVF treatments were obtained from the Human Fertilisation and Embryology Authority (HFEA) database. Using a discrete time logistic regression model, we predicted the cumulative probability of live birth from the second up to and including the fourth complete cycles of IVF. Inverse probability weighting was used to account for treatment discontinuation. Discrimination was assessed using c-statistic and calibration was assessed using calibration-in-the-large and calibration slope.

MAIN RESULTS AND THE ROLE OF CHANCE

Following exclusions, 49 314 women with 73 053 complete cycles were included. 12 408 (25.2%) had a live birth resulting from their second complete cycle. Cumulatively, 17 394 (35.3%) had a live birth over complete cycles two to four. The model showed moderate discriminative ability (c-statistic: 0.65, 95% CI: 0.64 to 0.65) and evidence of overprediction (calibration-in-the-large = −0.08) and overfitting (calibration slope 0.85, 95% CI: 0.81 to 0.88) in the validation cohort. However, after recalibration the fit was much improved. The recalibrated model identified the following key predictors of live birth: female age (38 versus 32 years—adjusted odds ratio: 0.59, 95% CI: 0.57 to 0.62), number of eggs retrieved in the first complete cycle (12 versus 4 eggs; 1.34, 1.30 to 1.37) and outcome of the first complete cycle (live birth versus no pregnancy; 1.78, 1.66 to 1.91; live birth versus pregnancy loss; 1.29, 1.23 to 1.36). As an example, a 32-year-old with 2 years of non-tubal infertility who had 12 eggs retrieved from her first stimulation and had a live birth during her first complete cycle has a 46% chance of having a further live birth from the second complete cycle of IVF and an 81% chance over a further three cycles.

LIMITATIONS, REASONS FOR CAUTION

The developed model was updated using validation data that was 6 to 12 years old. IVF practice continues to evolve over time, which may affect the accuracy of predictions from the model. We were unable to adjust for some potentially important predictors, e.g. BMI, smoking and alcohol intake in women, as well as measures of ovarian reserve such as antral follicle count. These were not available in the linked HFEA dataset.

WIDER IMPLICATIONS OF THE FINDINGS

By appropriately adjusting for couples who discontinue treatment, our novel prediction model will provide more realistic chances of live birth in couples starting a second complete cycle of IVF. Clinicians can use these predictions to inform discussion with couples who wish to plan ahead. This prediction tool will enable couples to prepare emotionally, financially and logistically for IVF treatment.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by an Elphinstone scholarship scheme at the University of Aberdeen and Aberdeen Fertility Centre, University of Aberdeen. The authors have no conflict of interest.

TRIAL REGISTRATION NUMBER

N/A.

The (decision) tree of fertility: an innovative decision-making algorithm in assisted reproduction technique

PURPOSE

Several mathematical models have been developed to estimate individualized chances of assisted reproduction techniques (ART) success, although with limited clinical application. Our study aimed to develop a decisional algorithm able to predict pregnancy and live birth rates after controlled ovarian stimulation (COS) phase, helping the physician to decide whether to perform oocytes pick-up continuing the ongoing ART path.

METHODS

A single-center retrospective analysis of real-world data was carried out including all fresh ART cycles performed in 1998-2020. Baseline characteristics, ART parameters and biochemical/clinical pregnancies and live birth rates were collected. A seven-steps systematic approach for model development, combining linear regression analyses and decision trees (DT), was applied for biochemical, clinical pregnancy, and live birth rates.

RESULTS

Of fresh ART cycles, 12,275 were included. Linear regression analyses highlighted a relationship between number of ovarian follicles > 17 mm detected at ultrasound before pick-up (OF17), embryos number and fertilization rate, and biochemical and clinical pregnancy rates (p < 0.001), but not live birth rate. DT were created for biochemical pregnancy (statistical power-SP:80.8%), clinical pregnancy (SP:85.4%), and live birth (SP:87.2%). Thresholds for OF17 entered in all DT, while sperm motility entered the biochemical pregnancy's model, and female age entered the clinical pregnancy and live birth DT. In case of OF17 < 3, the chance of conceiving was < 6% for all DT.

CONCLUSION

A systematic approach allows to identify OF17, female age, and sperm motility as pre-retrieval predictors of ART outcome, possibly reducing the socio-economic burden of ART failure, allowing the clinician to perform or not the oocytes pick-up.

Nomogram for the cumulative live birth in women undergoing the first IVF cycle: Base on 26, 689 patients in China

OBJECTIVE

Predictive models of the cumulative live birth (CLB) in women undergoing in vitro fertilization (IVF) treatment are limited. The aim of this study was to develop and validate a nomogram for the CLB in women undergoing the first IVF cycle.

METHODS

Based on a cross-sectional study in assisted reproduction center of Northwest Women's and Children's Hospital, 26,689 Chinese patients who underwent IVF treatment was used to develop and validate a prediction model for the CLB. Among those participants, 70% were randomly assigned to the training set (18,601 patients), while the remaining 30% were assigned to the validation set (8,088 patients). A nomogram was constructed based on the results of the multivariate logistic regression analysis. The model performance was evaluated using the C statistic and the calibration performance was assessed by Hosmer-Lemeshow (HL) χ² statistics and calibration plots.

RESULTS

Multivariate logistic regression analyses revealed that female age, female body mass index (BMI), tubal factor infertility, male infertility, uterine factor infertility, unexplained infertility, antral follicle count (AFC) and basal serum follicle stimulating hormone (FSH) were significant factors for CLB in women undergoing the first IVF cycle. An area under the receiver operating characteristic curve (AUC) in the prediction model was 0.676 (95% CI 0.668 to 0.684) in the training group. The validation set showed possibly helpful discrimination with an AUC of 0.672 (95% CI 0.660 to 0.684). Additionally, the prediction model had a good calibration (HL χ² = 8.240, P=0.410).

CONCLUSIONS

We developed and validated a nomogram to predict CLB in women undergoing the first IVF cycle using a single center database in China. The validated nomogram to predict CLB could be a potential tool for IVF counselling.

A systematic review of the quality of clinical prediction models in in vitro fertilisation

STUDY QUESTION

What are the best-quality clinical prediction models in IVF (including ICSI) treatment to inform clinicians and their patients of their chance of success?

SUMMARY ANSWER

The review recommends the McLernon post-treatment model for predicting the cumulative chance of live birth over and up to six complete cycles of IVF.

WHAT IS KNOWN ALREADY

Prediction models in IVF have not found widespread use in routine clinical practice. This could be due to their limited predictive accuracy and clinical utility. A previous systematic review of IVF prediction models, published a decade ago and which has never been updated, did not assess the methodological quality of existing models nor provided recommendations for the best-quality models for use in clinical practice.

STUDY DESIGN, SIZE, DURATION

The electronic databases OVID MEDLINE, OVID EMBASE and Cochrane library were searched systematically for primary articles published from 1978 to January 2019 using search terms on the development and/or validation (internal and external) of models in predicting pregnancy or live birth. No language or any other restrictions were applied.

PARTICIPANTS/MATERIALS, SETTING, METHODS

The PRISMA flowchart was used for the inclusion of studies after screening. All studies reporting on the development and/or validation of IVF prediction models were included. Articles reporting on women who had any treatment elements involving donor eggs or sperm and surrogacy were excluded. The CHARMS checklist was used to extract and critically appraise the methodological quality of the included articles. We evaluated models' performance by assessing their c-statistics and plots of calibration in studies and assessed correct reporting by calculating the percentage of the TRIPOD 22 checklist items met in each study.

MAIN RESULTS AND THE ROLE OF CHANCE

We identified 33 publications reporting on 35 prediction models. Seventeen articles had been published since the last systematic review. The quality of models has improved over time with regard to clinical relevance, methodological rigour and utility. The percentage of TRIPOD score for all included studies ranged from 29 to 95%, and the c-statistics of all externally validated studies ranged between 0.55 and 0.77. Most of the models predicted the chance of pregnancy/live birth for a single fresh cycle. Six models aimed to predict the chance of pregnancy/live birth per individual treatment cycle, and three predicted more clinically relevant outcomes such as cumulative pregnancy/live birth. The McLernon (pre- and post-treatment) models predict the cumulative chance of live birth over multiple complete cycles of IVF per woman where a complete cycle includes all fresh and frozen embryo transfers from the same episode of ovarian stimulation. McLernon models were developed using national UK data and had the highest TRIPOD score, and the post-treatment model performed best on external validation.

LIMITATIONS, REASONS FOR CAUTION

To assess the reporting quality of all included studies, we used the TRIPOD checklist, but many of the earlier IVF prediction models were developed and validated before the formal TRIPOD reporting was published in 2015. It should also be noted that two of the authors of this systematic review are authors of the McLernon model article. However, we feel we have conducted our review and made our recommendations using a fair and transparent systematic approach.

WIDER IMPLICATIONS OF THE FINDINGS

This study provides a comprehensive picture of the evolving quality of IVF prediction models. Clinicians should use the most appropriate model to suit their patients' needs. We recommend the McLernon post-treatment model as a counselling tool to inform couples of their predicted chance of success over and up to six complete cycles. However, it requires further external validation to assess applicability in countries with different IVF practices and policies.

STUDY FUNDING/COMPETING INTEREST(S)

The study was funded by the Elphinstone Scholarship Scheme and the Assisted Reproduction Unit, University of Aberdeen. Both D.J.M. and S.B. are authors of the McLernon model article and S.B. is Editor in Chief of Human Reproduction Open. They have completed and submitted the ICMJE forms for Disclosure of potential Conflicts of Interest. The other co-authors have no conflicts of interest to declare.

REGISTRATION NUMBER

N/A.

Individualized embryo selection strategy developed by stacking machine learning model for better in vitro fertilization outcomes: an application study

BACKGROUND

To minimize the rate of in vitro fertilization (IVF)- associated multiple-embryo gestation, significant efforts have been made. Previous studies related to machine learning in IVF mainly focused on selecting the top-quality embryos to improve outcomes, however, in patients with sub-optimal prognosis or with medium- or inferior-quality embryos, the selection between SET and DET could be perplexing.

METHODS

This was an application study including 9211 patients with 10,076 embryos treated during 2016 to 2018, in Tongji Hospital, Wuhan, China. A hierarchical model was established using the machine learning system XGBoost, to learn embryo implantation potential and the impact of double embryos transfer (DET) simultaneously. The performance of the model was evaluated with the AUC of the ROC curve. Multiple regression analyses were also conducted on the 19 selected features to demonstrate the differences between feature importance for prediction and statistical relationship with outcomes.

RESULTS

For a single embryo transfer (SET) pregnancy, the following variables remained significant: age, attempts at IVF, estradiol level on hCG day, and endometrial thickness. For DET pregnancy, age, attempts at IVF, endometrial thickness, and the newly added P1 + P2 remained significant. For DET twin risk, age, attempts at IVF, 2PN/ MII, and P1 × P2 remained significant. The algorithm was repeated 30 times, and averaged AUC of 0.7945, 0.8385, and 0.7229 were achieved for SET pregnancy, DET pregnancy, and DET twin risk, respectively. The trend of predictive and observed rates both in pregnancy and twin risk was basically identical. XGBoost outperformed the other two algorithms: logistic regression and classification and regression tree.

CONCLUSION

Artificial intelligence based on determinant-weighting analysis could offer an individualized embryo selection strategy for any given patient, and predict clinical pregnancy rate and twin risk, therefore optimizing clinical outcomes.

The predicted probability of live birth in In Vitro Fertilization varies during important stages throughout the treatment: analysis of 114,882 first cycles

RESEARCH QUESTION

How much the variability in patients' response during in vitro fertilization (IVF) may add to the initial predicted prognosis based only on patients' basal characteristics?

DESIGN

Anonymous data were obtained from the Human Fertilization and Embryology Authority (HFEA). Data involving 114,882 stimulated fresh IVF cycles were retrospectively analyzed. Logistic regression was used to develop the models.

RESULTS

Prediction of live birth was feasible with moderate accuracy in all of the three models; discrimination of the model based only on basal patients' characteristics (AUROC 0.61) was markedly improved adding information of number of embryos (AUROC 0.65) and, mostly, number of oocytes (AUROC 0.66).

CONCLUSIONS

The addition to prediction models of parameters such as the number of embryos obtained and especially the number of oocytes retrieved can statistically significantly improve the overall prediction of live birth probabilities when based on only basal patients' characteristics. This seems to be particularly true for women after the first IVF cycle. Since ovarian response affects the probability of live birth in IVF, it is highly recommended to add markers of ovarian response to models based on basal characteristics to increase their predictive ability.

Computational prediction of implantation outcome after embryo transfer

The aim of this study is to develop a computational prediction model for implantation outcome after an embryo transfer cycle. In this study, information of 500 patients and 1360 transferred embryos, including cleavage and blastocyst stages and fresh or frozen embryos, from April 2016 to February 2018, were collected. The dataset containing 82 attributes and a target label (indicating positive and negative implantation outcomes) was constructed. Six dominant machine learning approaches were examined based on their performance to predict embryo transfer outcomes. Also, feature selection procedures were used to identify effective predictive factors and recruited to determine the optimum number of features based on classifiers performance. The results revealed that random forest was the best classifier (accuracy = 90.40% and area under the curve = 93.74%) with optimum features based on a 10-fold cross-validation test. According to the Support Vector Machine-Feature Selection algorithm, the ideal numbers of features are 78. Follicle stimulating hormone/human menopausal gonadotropin dosage for ovarian stimulation was the most important predictive factor across all examined embryo transfer features. The proposed machine learning-based prediction model could predict embryo transfer outcome and implantation of embryos with high accuracy, before the start of an embryo transfer cycle.

A Review of Machine Learning Approaches in Assisted Reproductive Technologies

INTRODUCTION

Assisted reproductive technologies (ART) are recent improvements in infertility treatment. However, there is no significant increase in pregnancy rates with the aid of ART. Costly and complex process of ART's makes them as challenging issues. Computational prediction models could predict treatment outcome, before the start of an ART cycle.

AIM

This review provides an overview on machine learning-based prediction models in ART.

METHODS

This article was executed based on a literature review through scientific databases search such as PubMed, Scopus, Web of Science and Google Scholar.

RESULTS

We identified 20 papers reporting on machine learning-based prediction models in IVF or ICSI settings. All of the models were validated only by internal validation. Therefore, external validation of the models and the impact analysis of them were the missing parts of the all studies.

CONCLUSION

Machine learning-based prediction models provide a clinical decision support tool for both clinicians and patients and lead to improvement in ART success rates.

Risk of operative delivery for intrapartum fetal compromise in small-for-gestational-age fetuses at term: external validation of the IRIS algorithm

Objectives: Small-for-gestational-age fetuses (SGA) are at high risk of intrapartum fetal compromise requiring operative delivery. In a recent study, we developed a model using a combination of three antenatal (gestational age at delivery, parity, cerebroplacental ratio) and three intrapartum (epidural use, labor induction and augmentation using oxytocin) variables for the prediction of operative delivery due to presumed fetal compromise in SGA fetuses - the Individual RIsk aSsessment (IRIS) prediction model. The aim of this study was to test the predictive accuracy of the IRIS prediction model in an external cohort of singleton pregnancies complicated by SGA.Methods: This was an external validation study using a cohort of pregnancies from two tertiary referral centers in Spain and England. The inclusion criteria were singleton pregnancies diagnosed with an SGA fetus, defined as estimated fetal weight (EFW) below the 10th centile for gestational age at 36 weeks or beyond, which had fetal Doppler assessment and available data on their intrapartum care and pregnancy outcomes. The main outcome in this study was the operative delivery for presumed fetal compromise. External validation was performed using the coefficients obtained in the original development cohort. The predictive accuracies of models were investigated with receiver operating characteristics (ROC) curves. The Hosmer-Lemeshow test was used to test the goodness-of-fit of models and calibration plots were also obtained for visual assessment. A mobile application using the combined model algorithm was developed to facilitate clinical use.Results: Four hundred twelve singleton pregnancies with an antenatal diagnosis of SGA were included in the study. The operative delivery rate was 22.8% (n = 94). The group which required operative delivery for presumed fetal compromise had significantly fewer multiparous women (19.1 versus 47.8%, p < .001 in the total study population; 19.0 versus 43.5 and 19.2 versus 49.6%, UK and Spain cohort, respectively), lower cerebroplacental ratio (CPR) multiples of median (MoM) (median: 0.77 versus 0.92, p < .001 in the total study population; 0.77 versus 0.92 and 0.77 versus 0.92, UK and Spain cohort, respectively), more inductions of labor (74.5 versus 60.1%, p = .010 in the total study population; 85.7 versus 77.2 and 71.2% and 53.1, UK and Spain cohort, respectively) and more use of oxytocin augmentation (57.4 versus 39.3%, p = .002 in the total study population; 19.0 versus 12.0 and 68.5 and 50.4%, UK and Spain cohort, respectively) compared to those who did not require operative delivery due to presumed fetal compromise. When the original antenatal model was applied to the present cohort, we observed moderate predictive accuracy (AUC: 0.70, 95% CI: 0.64-0.76), and no signs of poor fit (p = .464). The original combined model, when applied to the external cohort, had moderate predictive accuracy (AUC: 0.72, 95% CI: 0.67-0.77) and also no signs of poor fit (p = .268) without the need for refitting. A statistically significant increase in the predictive accuracy was not achieved via refitting of the combined model (AUC 0.76 versus 0.72, p = .060).Conclusions: Using our recently published model, the predictive accuracy for fetal compromise requiring operative delivery in term fetuses thought to be SGA was modest and showed no signs of poor fit in an external cohort. The IRIS tool for mobile devices has been developed to facilitate wide clinical use of this prediction model.Brief rationaleObjective: To determine the external validity of an intrapartum risk prediction model for suspected small-for-gestational age fetuses.What is already known: Small-for-gestational age fetuses are at increased risk of intrapartum compromise. Fetal weight alone is a poor marker for adverse outcomes and a comprehensive prediction model has been previously suggested.What this study adds: Multivariable prediction model showed good accuracy and calibration in this external validation study. The significance of some variables was different between the original and external validation cohort and there was a small margin for improvement with model refitting. A mobile application has been developed to facilitate clinical use.

Predicting the success of IVF: external validation of the van Loendersloot's model

STUDY QUESTION

Is the predictive model for IVF success proposed by van Loendersloot et al. valid in a different geographical and cultural context?

SUMMARY ANSWER

The model discriminates well but was less accurate than in the original context where it was developed.

WHAT IS ALREADY KNOWN

Several independent groups have developed models that combine different variables with the aim of estimating the chance of pregnancy with IVF but only four of them have been externally validated. One of these four, the van Loendersloot's model, deserves particular attention and further investigation for at least three reasons; (i) the reported area under the receiver operating characteristics curve (c-statistics) in the temporal validation setting was the highest reported to date (0.68), (ii) the perspective of the model is clinically wise since it includes variables obtained from previous failed cycles, if any, so it can be applied to any women entering an IVF cycle, (iii) the model lacks external validation in a geographically different center.

STUDY DESIGN, SIZE, DURATION

Retrospective cohort study of women undergoing oocyte retrieval for IVF between January 2013 and December 2013 at the infertility unit of the Fondazione Ca' Granda, Ospedale Maggiore Policlinico of Milan, Italy. Only the first oocyte retrieval cycle performed during the study period was included in the study. Women with previous IVF cycles were excluded if the last one before the study cycle was in another center. The main outcome was the cumulative live birth rate per oocytes retrieval.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Seven hundred seventy-two women were selected. Variables included in the van Loendersloot's model and the relative weights (beta) were used. The variable resulting from this combination (Y) was transformed into a probability. The discriminatory capacity was assessed using the c-statistics. Calibration was made using a logistic regression that included Y as the unique variable and live birth as the outcome. Data are presented using both the original and the calibrated models. Performance was evaluated correlating the mean predicted chances of live births in the five quintiles and the observed rates.

MAIN RESULTS AND THE ROLE OF CHANCE

Two-hundred-eleven live births (27%) were obtained. The c-statistic was 0.64 (95% CI: 0.61-0.67, P < 0.001). The slope of the linear predictor (calibration slope) expressed as an Odds Ratio was 1.81 (95% CI: 1.46-2.24, P < 0.001), corresponding to a beta of 0.630. The calibration intercept was +0.349 (P = 0.13). While a clear discrepancy exists using the original model, data appear properly distributed with the calibrated model. The Pearson coefficient of the correlation between the mean predicted chances of live births in the five quintiles and the observed rates was 0.99 (P = 0.002).

LIMITATIONS, REASONS FOR CAUTION

Data were collected retrospectively, thus exposing them to potential inaccuracies. The selection criteria for access to IVF adopted in our center might be too stringent, leading to the exclusion of women with a poor, yet acceptable chance of live birth. Therefore, the validity of the model in women with a very low chance of live birth could not be tested.

WIDER IMPLICATIONS OF THE FINDINGS

The van Loendersloot's model can be used in other contexts but it is important that it has local calibration. It may help in counseling couples about their chance of success but it cannot be used to exclude treatments. Further research is needed to improve the discriminatory performance of IVF predictive models.

STUDY FUNDING/COMPETING INTERESTS

None.

TRIAL REGISTRATION NUMBER

Not applicable.

Predicting the chance of live birth for women undergoing IVF: a novel pretreatment counselling tool

STUDY QUESTION

Which pretreatment patient variables have an effect on live birth rates following assisted conception?

SUMMARY ANSWER

The predictors in the final multivariate logistic regression model found to be significantly associated with reduced chances of IVF/ICSI success were increasing age (particularly above 36 years), tubal factor infertility, unexplained infertility and Asian or Black ethnicity.

WHAT IS KNOWN ALREADY

The two most widely recognized prediction models for live birth following IVF were developed on data from 1991 to 2007; pre-dating significant changes in clinical practice. These existing IVF outcome prediction models do not incorporate key pretreatment predictors, such as BMI, ethnicity and ovarian reserve, which are readily available now.

STUDY DESIGN, SIZE, DURATION

In this cohort study a model to predict live birth was derived using data collected from 9915 women who underwent IVF/ICSI treatment at any CARE (Centres for Assisted Reproduction) clinic from 2008 to 2012. Model validation was performed on data collected from 2723 women who underwent treatment in 2013. The primary outcome for the model was live birth, which was defined as any birth event in which at least one baby was born alive and survived for more than 1 month.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Data were collected from 12 fertility clinics within the CARE consortium in the UK. Multivariable logistic regression was used to develop the model. Discriminatory ability was assessed using the area under receiver operating characteristic (AUROC) curve, and calibration was assessed using calibration-in-the-large and the calibration slope test.

MAIN RESULTS AND THE ROLE OF CHANCE

The predictors in the final model were female age, BMI, ethnicity, antral follicle count (AFC), previous live birth, previous miscarriage, cause and duration of infertility. Upon assessing predictive ability, the AUROC curve for the final model and validation cohort was (0.62; 95% confidence interval (CI) 0.61-0.63) and (0.62; 95% CI 0.60-0.64) respectively. Calibration-in-the-large showed a systematic over-estimation of the predicted probability of live birth (Intercept (95% CI) = -0.168 (-0.252 to -0.084), P < 0.001). However, the calibration slope test was not significant (slope (95% CI) = 1.129 (0.893-1.365), P = 0.28). Due to the calibration-in-the-large test being significant we recalibrated the final model. The recalibrated model showed a much-improved calibration.

LIMITATIONS, REASONS FOR CAUTION

Our model is unable to account for factors such as smoking and alcohol that can affect IVF/ICSI outcome and is somewhat restricted to representing the ethnic distribution and outcomes for the UK population only. We were unable to account for socioeconomic status and it may be that by having 75% of the population paying privately for their treatment, the results cannot be generalized to people of all socioeconomic backgrounds. In addition, patients and clinicians should understand this model is designed for use before treatment begins and does not include variables that become available (oocyte, embryo and endometrial) as treatment progresses. Finally, this model is also limited to use prior to first cycle only.

WIDER IMPLICATIONS OF THE FINDINGS

To our knowledge, this is the first study to present a novel, up-to-date model encompassing three readily available prognostic factors; female BMI, ovarian reserve and ethnicity, which have not previously been used in prediction models for IVF outcome. Following geographical validation, the model can be used to build a user-friendly interface to aid decision-making for couples and their clinicians. Thereafter, a feasibility study of its implementation could focus on patient acceptability and quality of decision-making.

STUDY FUNDING/COMPETING INTEREST

None.

External validation and calibration of IVFpredict: a national prospective cohort study of 130,960 in vitro fertilisation cycles

BACKGROUND

Accurately predicting the probability of a live birth after in vitro fertilisation (IVF) is important for patients, healthcare providers and policy makers. Two prediction models (Templeton and IVFpredict) have been previously developed from UK data and are widely used internationally. The more recent of these, IVFpredict, was shown to have greater predictive power in the development dataset. The aim of this study was external validation of the two models and comparison of their predictive ability.

METHODS AND FINDINGS

130,960 IVF cycles undertaken in the UK in 2008-2010 were used to validate and compare the Templeton and IVFpredict models. Discriminatory power was calculated using the area under the receiver-operator curve and calibration assessed using a calibration plot and Hosmer-Lemeshow statistic. The scaled modified Brier score, with measures of reliability and resolution, were calculated to assess overall accuracy. Both models were compared after updating for current live birth rates to ensure that the average observed and predicted live birth rates were equal. The discriminative power of both methods was comparable: the area under the receiver-operator curve was 0.628 (95% confidence interval (CI): 0.625-0.631) for IVFpredict and 0.616 (95% CI: 0.613-0.620) for the Templeton model. IVFpredict had markedly better calibration and higher diagnostic accuracy, with calibration plot intercept of 0.040 (95% CI: 0.017-0.063) and slope of 0.932 (95% CI: 0.839-1.025) compared with 0.080 (95% CI: 0.044-0.117) and 1.419 (95% CI: 1.149-1.690) for the Templeton model. Both models underestimated the live birth rate, but this was particularly marked in the Templeton model. Updating the models to reflect improvements in live birth rates since the models were developed enhanced their performance, but IVFpredict remained superior.

CONCLUSION

External validation in a large population cohort confirms IVFpredict has superior discrimination and calibration for informing patients, clinicians and healthcare policy makers of the probability of live birth following IVF.

Prediction models in in vitro fertilization; where are we? A mini review

Since the introduction of in vitro fertilization (IVF) in 1978, over five million babies have been born worldwide using IVF. Contrary to the perception of many, IVF does not guarantee success. Almost 50% of couples that start IVF will remain childless, even if they undergo multiple IVF cycles. The decision to start or pursue with IVF is challenging due to the high cost, the burden of the treatment, and the uncertain outcome. In optimal counseling on chances of a pregnancy with IVF, prediction models may play a role, since doctors are not able to correctly predict pregnancy chances. There are three phases of prediction model development: model derivation, model validation, and impact analysis. This review provides an overview on predictive factors in IVF, the available prediction models in IVF and provides key principles that can be used to critically appraise the literature on prediction models in IVF. We will address these points by the three phases of model development.

Simple adaptations to the Templeton model for IVF outcome prediction make it current and clinically useful

STUDY QUESTION

What is the validity of the Templeton model (TM) in predicting live birth (LB) for a couple starting an IVF/ICSI cycle?

SUMMARY ANSWER

A centre-specific model based on the original predictors of the TM may reach a sufficient level of accuracy to be used in every day practice, with a few simple adaptations.

WHAT IS KNOWN AND WHAT THIS PAPER ADDS

The TM seems the best predictive model of LB in IVF. However, previous validations of the TM suggest a lack of discrimination and calibration which means that it is not used in regular practice. We confirm this finding, and argue that such results are predictable, and essentially due to a strong centre effect. We provide evidence that the TM constitutes a useful reference reflecting a high proportion of the patient-mix effect since the parameters of the model remain invariant among centres, but also across various cultures, countries and types of hospitals. The only difference was the intercept value, interpreted as the measurement of the global performance of one centre, in particular, for a population of reference.

STUDY DESIGN

The validity of the TM was tested by a retrospective analysis all IVF/ICSI cycles (n = 12 901) in our centre since 2000.

PARTICIPANTS, SETTING AND METHODS

All IVF/ICSI cycles were included in the analysis. The model discrimination was evaluated by C-statistics, calculated as the area under the curve of an ROC curve. The TM was then adjusted for our data and additional variables were assessed.

MAIN RESULTS AND THE ROLE OF CHANCE

Poor calibration and discrimination (C = 0.64) was observed in conformity with previous external validations. Fitting the TM to our centre constituted the first substantial improvement in prediction accuracy of discrimination (C = 0.69) and calibration. We identified an important linear time trend effect and the added value of three other predictors (FSH, smoking habits and BMI) that significantly improved the model (C = 0.71).

BIAS, CONFOUNDING AND OTHER REASONS FOR CAUTION

Bias due to missing data handling was assessed through sensitivity analyses.

GENERALIZABILITY TO OTHER POPULATIONS

Neither the TM nor any other models based on some centres are directly applicable to other centres. However, the TM constitutes a useful basis to build an accurate centre-specific model.

STUDY FUNDING/COMPETING INTEREST(S)

There were no commercial relationships (i.e. consultancies, patent-licensing agreements) that might pose a conflict of interest in connection with the submitted manuscript. The objective of this research was not directed toward any treatment effects.

Tailored expectant management: risk factors for non-adherence

INTRODUCTION

Prediction models for spontaneous pregnancy are useful tools to prevent overtreatment, complications and costs in subfertile couples with a good prognosis. The use of such models and subsequent expectant management in couples with a good prognosis are recommended in the Dutch fertility guidelines, but not fully implemented. In this study, we assess risk factors for non-adherence to tailored expectant management.

METHODS

Couples with mild male, unexplained and cervical subfertility were included in this multicentre prospective cohort study. If the probability of spontaneous pregnancy within 12 months was ≥40%, expectant management for 6-12 months was advised. Multivariable logistic regression was used to identify patient and clinical characteristics associated with non-adherence to tailored expectant management.

RESULTS

We included 3021 couples of whom 1130 (38%) had a ≥40% probability of a spontaneous pregnancy. Follow-up was available for 1020 (90%) couples of whom 214 (21%) had started treatment between 6 and 12 months and 153 (15%) within 6 months. A higher female age and a longer duration of subfertility were associated with treatment within 6 months (OR: 1.06, 95% CI: 1.01-1.1; OR: 1.4; 95% CI: 1.1-1.8). A fertility doctor in a clinical team reduced the risk of treatment within 6 months (OR: 0.62; 95% CI: 0.39-0.99).

CONCLUSIONS

In couples with a favorable prognosis for spontaneous pregnancy, there is considerable overtreatment, especially if the woman is older and duration of the subfertility is longer. The presence of a fertility doctor in a clinic may prevent early treatment.

The use of prediction models of spontaneous pregnancy in in vitro fertilization units reveals differences between the expected results of public and private clinics in Spain

To evaluate the applicability of prediction models (PM) of spontaneous pregnancy (SP) in a population of infertile patients from a university-affiliated private assisted reproductive technology center (Instituto Valenciano de Infertilidad) and in the reproductive medicine section of a public university hospital (La Fe), both belonging to the same city (Valencia, Spain) between January and December 2008. We calculated the probability of SP using the PM developed by Hunault et al. in our two populations, and observed an estimated probability of SP<40% or the PM applicable in approximately 97% of the studied couples, and statistical differences between pregnancy probabilities in the two settings that were mainly a result of different age, sperm quality, and referral policies, leading us to conclude that the usefulness of PMs is limited in our environment.

Prediction of residual disease after primary cytoreductive surgery for advanced-stage ovarian cancer: accuracy of clinical judgment

OBJECTIVES

Treatment of patients with an advanced-stage epithelial ovarian cancer (EOC) is based on cytoreductive surgery and platinum-based chemotherapy. Amount of residual disease after primary cytoreductive surgery is an important prognostic factor. The objectives of the present study were to evaluate the accuracy and reproducibility of preoperative clinical judgment of residual disease after primary cytoreductive surgery and to compare the predictive performance of the offhand assessment to the predictive performance of prediction models.

MATERIALS AND METHODS

Fifteen observers (5 gynecologic oncologists, 5 gynecologists, and 5 senior residents) were offered preoperative data of 20 patients with advanced-stage EOC who underwent primary cytoreductive surgery. The observers were asked to predict residual disease after cytoreductive surgery (<or=1 or >1 cm). Their estimation was compared with the performance of 2 prediction models.

RESULTS

Overall, suboptimal cytoreduction was predicted with a sensitivity of 50% and a specificity of 56%. The intraclass correlation coefficient was 0.27. chi(2) Test showed no significant difference in prediction of suboptimal cytoreduction between the different subgroups and prediction models.

CONCLUSIONS

Clinical judgment of residual disease after primary cytoreductive surgery in patients with advanced-stage EOC shows limited accuracy. Given the poor interobserver reproducibility, prediction models could attribute to uniform treatment decisions and improve counseling.

Prediction models in reproductive medicine: a critical appraisal

BACKGROUND

Prediction models have been developed in reproductive medicine to help assess the chances of a treatment-(in)dependent pregnancy. Careful evaluation is needed before these models can be implemented in clinical practice.

METHODS

We systematically searched the literature for papers reporting prediction models in reproductive medicine for three strategies: expectant management, intrauterine insemination (IUI) or in vitro fertilization (IVF). We evaluated which phases of development these models had passed, distinguishing between (i) model derivation, (ii) internal and/or external validation, and (iii) impact analysis. We summarized their performance at external validation in terms of discrimination and calibration.

RESULTS

We identified 36 papers reporting on 29 prediction models. There were 9 models for the prediction of treatment-independent pregnancy, 3 for the prediction of pregnancy after IUI and 17 for the prediction of pregnancy after IVF. All of the models had completed the phase of model derivation. For six models, the validity of the model was assessed only in the population in which it was developed (internal validation). For eight models, the validity was assessed in populations other than the one in which the model was developed (external validation), and only three of these showed good performance. One model had reached the phase of impact analysis.

CONCLUSIONS

Currently, there are three models with good predictive performance. These models can be used reliably as a guide for making decisions about fertility treatment, in patients similar to the development population. The effects of using these models in patient care have to be further investigated.