Comparison of two models predicting IVF success; the effect of time trends on model performance

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

ESHRE good practice recommendations on recurrent implantation failure

STUDY QUESTION

How should recurrent implantation failure (RIF) in patients undergoing ART be defined and managed?

SUMMARY ANSWER

This is the first ESHRE good practice recommendations paper providing a definition for RIF together with recommendations on how to investigate causes and contributing factors, and how to improve the chances of a pregnancy.

WHAT IS KNOWN ALREADY

RIF is a challenge in the ART clinic, with a multitude of investigations and interventions offered and applied in clinical practice, often without biological rationale or with unequivocal evidence of benefit.

STUDY DESIGN SIZE DURATION

This document was developed according to a predefined methodology for ESHRE good practice recommendations. Recommendations are supported by data from the literature, if available, and the results of a previously published survey on clinical practice in RIF and the expertise of the working group. A literature search was performed in PubMed and Cochrane focussing on 'recurrent reproductive failure', 'recurrent implantation failure', and 'repeated implantation failure'.

PARTICIPANTS/MATERIALS SETTING METHODS

The ESHRE Working Group on Recurrent Implantation Failure included eight members representing the ESHRE Special Interest Groups for Implantation and Early Pregnancy, Reproductive Endocrinology, and Embryology, with an independent chair and an expert in statistics. The recommendations for clinical practice were formulated based on the expert opinion of the working group, while taking into consideration the published data and results of the survey on uptake in clinical practice. The draft document was then open to ESHRE members for online peer review and was revised in light of the comments received.

MAIN RESULTS AND THE ROLE OF CHANCE

The working group recommends considering RIF as a secondary phenomenon of ART, as it can only be observed in patients undergoing IVF, and that the following description of RIF be adopted: 'RIF describes the scenario in which the transfer of embryos considered to be viable has failed to result in a positive pregnancy test sufficiently often in a specific patient to warrant consideration of further investigations and/or interventions'. It was agreed that the recommended threshold for the cumulative predicted chance of implantation to identify RIF for the purposes of initiating further investigation is 60%. When a couple have not had a successful implantation by a certain number of embryo transfers and the cumulative predicted chance of implantation associated with that number is greater than 60%, then they should be counselled on further investigation and/or treatment options. This term defines clinical RIF for which further actions should be considered. Nineteen recommendations were formulated on investigations when RIF is suspected, and 13 on interventions. Recommendations were colour-coded based on whether the investigations/interventions were recommended (green), to be considered (orange), or not recommended, i.e. not to be offered routinely (red).

LIMITATIONS REASONS FOR CAUTION

While awaiting the results of further studies and trials, the ESHRE Working Group on Recurrent Implantation Failure recommends identifying RIF based on the chance of successful implantation for the individual patient or couple and to restrict investigations and treatments to those supported by a clear rationale and data indicating their likely benefit.

WIDER IMPLICATIONS OF THE FINDINGS

This article provides not only good practice advice but also highlights the investigations and interventions that need further research. This research, when well-conducted, will be key to making progress in the clinical management of RIF.

STUDY FUNDING/COMPETING INTERESTS

The meetings and technical support for this project were funded by ESHRE. N.M. declared consulting fees from ArtPRED (The Netherlands) and Freya Biosciences (Denmark); Honoraria for lectures from Gedeon Richter, Merck, Abbott, and IBSA; being co-founder of Verso Biosense. He is Co-Chief Editor of Reproductive Biomedicine Online (RBMO). D.C. declared being an Associate Editor of Human Reproduction Update, and declared honoraria for lectures from Merck, Organon, IBSA, and Fairtility; support for attending meetings from Cooper Surgical, Fujifilm Irvine Scientific. G.G. declared that he or his institution received financial or non-financial support for research, lectures, workshops, advisory roles, or travelling from Ferring, Merck, Gedeon-Richter, PregLem, Abbott, Vifor, Organon, MSD, Coopersurgical, ObsEVA, and ReprodWissen. He is an Editor of the journals Archives of Obstetrics and Gynecology and Reproductive Biomedicine Online, and Editor in Chief of Journal Gynäkologische Endokrinologie. He is involved in guideline developments and quality control on national and international level. G.L. declared he or his institution received honoraria for lectures from Merck, Ferring, Vianex/Organon, and MSD. He is an Associate Editor of Human Reproduction Update, immediate past Coordinator of Special Interest Group for Reproductive Endocrinology of ESHRE and has been involved in Guideline Development Groups of ESHRE and national fertility authorities. D.J.M. declared being an Associate Editor for Human Reproduction Open and statistical Advisor for Reproductive Biomedicine Online. B.T. declared being shareholder of Reprognostics and she or her institution received financial or non-financial support for research, clinical trials, lectures, workshops, advisory roles or travelling from support for attending meetings from Ferring, MSD, Exeltis, Merck Serono, Bayer, Teva, Theramex and Novartis, Astropharm, Ferring. The other authors had nothing to disclose.

DISCLAIMER

This Good Practice Recommendations (GPR) document represents the views of ESHRE, which are the result of consensus between the relevant ESHRE stakeholders and are based on the scientific evidence available at the time of preparation. ESHRE GPRs should be used for information and educational purposes. They should not be interpreted as setting a standard of care or be deemed inclusive of all proper methods of care, or be exclusive of other methods of care reasonably directed to obtaining the same results. They do not replace the need for application of clinical judgement to each individual presentation, or variations based on locality and facility type. Furthermore, ESHRE GPRs do not constitute or imply the endorsement, or favouring, of any of the included technologies by ESHRE.

Nomogram for the cumulative live birth in women undergoing the first IVF cycle: Base on 26, 689 patients in China

OBJECTIVE

Predictive models of the cumulative live birth (CLB) in women undergoing in vitro fertilization (IVF) treatment are limited. The aim of this study was to develop and validate a nomogram for the CLB in women undergoing the first IVF cycle.

METHODS

Based on a cross-sectional study in assisted reproduction center of Northwest Women's and Children's Hospital, 26,689 Chinese patients who underwent IVF treatment was used to develop and validate a prediction model for the CLB. Among those participants, 70% were randomly assigned to the training set (18,601 patients), while the remaining 30% were assigned to the validation set (8,088 patients). A nomogram was constructed based on the results of the multivariate logistic regression analysis. The model performance was evaluated using the C statistic and the calibration performance was assessed by Hosmer-Lemeshow (HL) χ² statistics and calibration plots.

RESULTS

Multivariate logistic regression analyses revealed that female age, female body mass index (BMI), tubal factor infertility, male infertility, uterine factor infertility, unexplained infertility, antral follicle count (AFC) and basal serum follicle stimulating hormone (FSH) were significant factors for CLB in women undergoing the first IVF cycle. An area under the receiver operating characteristic curve (AUC) in the prediction model was 0.676 (95% CI 0.668 to 0.684) in the training group. The validation set showed possibly helpful discrimination with an AUC of 0.672 (95% CI 0.660 to 0.684). Additionally, the prediction model had a good calibration (HL χ² = 8.240, P=0.410).

CONCLUSIONS

We developed and validated a nomogram to predict CLB in women undergoing the first IVF cycle using a single center database in China. The validated nomogram to predict CLB could be a potential tool for IVF counselling.

Updating Clinical Prediction Models: An Illustrative Case Study

The performance of clinical prediction models tends to deteriorate over time. Researchers often develop a new prediction if an existing model performs poorly at external validation. Model updating is an efficient technique and promising alternative to the de novo development of clinical prediction models. Model updating has been recommended by the TRIPOD guidelines. To illustrate several model updating techniques, a case study is provided for the development and updating of a clinical prediction model assessing postoperative anxiety in data coming from two double-blinded placebo-controlled randomized controlled trials with a very similar methodological framework. Note that the developed model and updated model are for didactic purposes only. This paper discusses some common considerations and caveats for researchers to be aware of when planning or applying updating of a prediction model.

Factors associated with livebirth in couples undergoing their first in vitro fertilization cycle: An internally validated prediction model

Objective:

The aim of the study is to create a new model to predict successful outcome in assisted reproductive techniques.

Materials and Methods:

A retrospective cohort study was conducted in tertiary fertility center between 2010 and 2017. Nulliparous women younger than 45 years-old undergoing in vitro fertilization/intracytoplasmic sperm injection (IVF/ICSI) for the first time were included; frozen embryo transfers, canceled induction cycles, freeze-all cycles were excluded. Two prediction models were built using multivariate logistic regression with a subset of the dataset and then were internally validated using bootstrapping methods.

Results:

Four hundred eighty eight women were included with 136 (27.9%) live births. The basal model was built using variable age, antral follicle count (AFC), and basal luteinizing hormone (LH) levels. Age over 37 years [odds ratio (OR): 0.07, 95% confidence interval (CI): 0.00-0.36] and AFC below 5 (OR: 0.15, 95% CI: 0.02-0.53) was associated with poorer outcomes whereas an LH level above 6 mIU/mL (OR: 2.24, 95% CI: 1.27-3.94) was associated with better outcomes. Optimism adjusted area under the curve (AUC) of this model was 0.68 (95% CI: 0.62-0.74). Combined model in addition to basal model variables included the length of induction cycle, the endometrial thickness at the day of transfer, grade and count of the transferred embryo. Cycles lasting more than ten days (OR: 2.23, 95% CI: 1.17-4.42), an endometrial thickness greater than 9 mm (OR: 2.07, 95% CI: 1.00-4.53) were associated with better outcomes. Optimism adjusted AUC of this model was 0.76 (95% CI: 0.70-0.81). Calibration of both models was good according to Hosmer Lemeshow test (p=0.979 and p=0.848, respectively).

Conclusion:

This internally validated prediction model has good calibration and can be used predicting outcomes in first time IVF/ICSI cycles with modest sensitivity.

A systematic review of the quality of clinical prediction models in in vitro fertilisation

STUDY QUESTION

What are the best-quality clinical prediction models in IVF (including ICSI) treatment to inform clinicians and their patients of their chance of success?

SUMMARY ANSWER

The review recommends the McLernon post-treatment model for predicting the cumulative chance of live birth over and up to six complete cycles of IVF.

WHAT IS KNOWN ALREADY

Prediction models in IVF have not found widespread use in routine clinical practice. This could be due to their limited predictive accuracy and clinical utility. A previous systematic review of IVF prediction models, published a decade ago and which has never been updated, did not assess the methodological quality of existing models nor provided recommendations for the best-quality models for use in clinical practice.

STUDY DESIGN, SIZE, DURATION

The electronic databases OVID MEDLINE, OVID EMBASE and Cochrane library were searched systematically for primary articles published from 1978 to January 2019 using search terms on the development and/or validation (internal and external) of models in predicting pregnancy or live birth. No language or any other restrictions were applied.

PARTICIPANTS/MATERIALS, SETTING, METHODS

The PRISMA flowchart was used for the inclusion of studies after screening. All studies reporting on the development and/or validation of IVF prediction models were included. Articles reporting on women who had any treatment elements involving donor eggs or sperm and surrogacy were excluded. The CHARMS checklist was used to extract and critically appraise the methodological quality of the included articles. We evaluated models' performance by assessing their c-statistics and plots of calibration in studies and assessed correct reporting by calculating the percentage of the TRIPOD 22 checklist items met in each study.

MAIN RESULTS AND THE ROLE OF CHANCE

We identified 33 publications reporting on 35 prediction models. Seventeen articles had been published since the last systematic review. The quality of models has improved over time with regard to clinical relevance, methodological rigour and utility. The percentage of TRIPOD score for all included studies ranged from 29 to 95%, and the c-statistics of all externally validated studies ranged between 0.55 and 0.77. Most of the models predicted the chance of pregnancy/live birth for a single fresh cycle. Six models aimed to predict the chance of pregnancy/live birth per individual treatment cycle, and three predicted more clinically relevant outcomes such as cumulative pregnancy/live birth. The McLernon (pre- and post-treatment) models predict the cumulative chance of live birth over multiple complete cycles of IVF per woman where a complete cycle includes all fresh and frozen embryo transfers from the same episode of ovarian stimulation. McLernon models were developed using national UK data and had the highest TRIPOD score, and the post-treatment model performed best on external validation.

LIMITATIONS, REASONS FOR CAUTION

To assess the reporting quality of all included studies, we used the TRIPOD checklist, but many of the earlier IVF prediction models were developed and validated before the formal TRIPOD reporting was published in 2015. It should also be noted that two of the authors of this systematic review are authors of the McLernon model article. However, we feel we have conducted our review and made our recommendations using a fair and transparent systematic approach.

WIDER IMPLICATIONS OF THE FINDINGS

This study provides a comprehensive picture of the evolving quality of IVF prediction models. Clinicians should use the most appropriate model to suit their patients' needs. We recommend the McLernon post-treatment model as a counselling tool to inform couples of their predicted chance of success over and up to six complete cycles. However, it requires further external validation to assess applicability in countries with different IVF practices and policies.

STUDY FUNDING/COMPETING INTEREST(S)

The study was funded by the Elphinstone Scholarship Scheme and the Assisted Reproduction Unit, University of Aberdeen. Both D.J.M. and S.B. are authors of the McLernon model article and S.B. is Editor in Chief of Human Reproduction Open. They have completed and submitted the ICMJE forms for Disclosure of potential Conflicts of Interest. The other co-authors have no conflicts of interest to declare.

REGISTRATION NUMBER

N/A.

Predicting the chance on live birth per cycle at each step of the IVF journey: external validation and update of the van Loendersloot multivariable prognostic model

OBJECTIVE

To study the performance of the 'van Loendersloot' prognostic model for our clinic's in vitro fertilisation (IVF) in its original version, the refitted version and in an adapted version replacing previous by current cycle IVF laboratory variables.

METHODS

This retrospective cohort study in our academic tertiary fertility clinic analysed 1281 IVF cycles of 591 couples, who completed at least one 2nd-6th IVF cycle with own fresh gametes after a previous IVF cycle with the same partner in our clinic between 2010 and 2018. The outcome of interest was the chance on a live birth after one complete IVF cycle (including all fresh and frozen embryo transfers from the same episode of ovarian stimulation). Model performance was expressed in terms of discrimination (c-statistics) and calibration (calibration model, comparison of prognosis to observed ratios of five disjoint groups formed by the quintiles of the IVF prognoses and a calibration plot).

RESULTS

A total of 344 live births were obtained (26.9%). External validation of the original van Loendersloot model showed a poor c-statistic of 0.64 (95% CI: 0.61 to 0.68) and an underestimation of IVF success. The refitted and the adapted models showed c-statistics of respectively 0.68 (95% CI: 0.65 to 0.71) and 0.74 (95% CI: 0.70 to 0.77). Similar c-statistics were found with cross-validation. Both models showed a good calibration model; refitted model: intercept=0.00 (95% CI: -0.23 to 0.23) and slope=1.00 (95% CI: 0.79 to 1.21); adapted model: intercept=0.00 (95% CI: -0.18 to 0.18) and slope=1.00 (95% CI: 0.83 to 1.17). Prognoses and observed success rates of the disjoint groups matched well for the refitted model and even better for the adapted model.

CONCLUSION

External validation of the original van Loendersloot model indicated that model updating was recommended. The good performance of the refitted and adapted models allows informing couples about their IVF prognosis prior to an IVF cycle and at the time of embryo transfer. Whether this has an impact on couple's expected success rates, distress and IVF discontinuation can now be studied.

What is the prognosis for a live birth after unexplained recurrent implantation failure following IVF/ICSI?

STUDY QUESTION

What is the cumulative incidence of live birth and mean time to pregnancy (by conception after IVF/ICSI or natural conception) in women experiencing unexplained recurrent implantation failure (RIF) following IVF/ICSI treatment?

SUMMARY ANSWER

In 118 women who had experienced RIF, the reported cumulative incidence of live birth during a maximum of 5.5 years follow-up period was 49%, with a calculated median time to pregnancy leading to live birth of 9 months after diagnosis of RIF.

WHAT IS KNOWN ALREADY

Current definitions of RIF include failure to achieve a pregnancy following IVF/ICSI and undergoing three or more fresh embryo transfer procedures of one or two high quality embryos or more than 10 embryos transferred in fresh or frozen cycles. The causes and optimal management of this distressing condition remain uncertain and a range of empirical and often expensive adjuvant therapies is often advocated. Little information is available regarding the long-term prognosis for achieving a pregnancy.

STUDY DESIGN, SIZE, DURATION

Two hundred and twenty-three women under 39 years of age who had experienced RIF without a known cause after IVF/ICSI treatment in two tertiary referral university hospitals between January 2008 and December 2012 were invited to participate in this retrospective cohort follow up study.

PARTICIPANTS/MATERIALS, SETTING, METHODS

All eligible women were sent a letter requesting their consent to the anonymous use of their medical file data and were asked to complete a questionnaire enquiring about treatments and pregnancies subsequent to experiencing RIF. Medical files and questionnaires were examined and results were analysed to determine the subsequent cumulative incidence of live birth and time to pregnancy within a maximum 5.5 year follow-up period using Kaplan Meier analysis. Clinical predictors for achieving a live birth were investigated using a Cox hazard model.

MAIN RESULTS AND THE ROLE OF CHANCE

One hundred and twenty-seven women responded (57%) and data from 118 women (53%) were available for analysis. During the maximum 5.5 year follow up period the overall cumulative incidence of live birth was 49% (95% CI 39-59%). Among women who gave birth, the calculated median time to pregnancy was 9 months after experiencing RIF, where 18% arose from natural conceptions.

LIMITATIONS, REASONS FOR CAUTION

Since only 57% of the eligible study cohort completed the questionnaire, the risk of response bias limits the applicability of the study findings.

WIDER IMPLICATIONS OF THE FINDINGS

This study reports a favorable overall prognosis for achieving live birth in women who have previously experienced RIF, especially in those who continue with further IVF/ICSI treatments. However since 51% did not achieve a live birth during the follow-up period, there is a need to distinguish those most likely to benefit from further treatment. In this study, no clinical factors were found to be predictive of those achieving a subsequent live birth.

STUDY FUNDING/COMPETING INTEREST(S)

This study was funded by the University Medical Center Utrecht, in Utrecht and the Academic Medical Centre, in Amsterdam. NSM has received consultancy and speaking fees and research funding from Ferring, MSD, Merck Serono, Abbott, IBSA, Gedion Richter, and Clearblue. During the most recent 5-year period BCJMF has received fees or grant support from the following organizations (in alphabetic order); Actavis/Watson/Uteron, Controversies in Obstetrics & Gynecology (COGI), Dutch Heart Foundation, Dutch Medical Research Counsel (ZonMW), Euroscreen/Ogeda, Ferring, London Womens Clinic (LWC), Merck Serono, Myovant, Netherland Genomic Initiative (NGI), OvaScience, Pantharei Bioscience, PregLem/Gedeon Richter/Finox, Reproductive Biomedicine Online (RBMO), Roche, Teva, World Health Organisation (WHO).None of the authors have disclosures to make in relation to this manuscript.

Can Ratios Between Prognostic Factors Predict the Clinical Pregnancy Rate in an IVF/ICSI Program with a GnRH Agonist-FSH/hMG Protocol? An Assessment of 2421 Embryo Transfers, and a Review of the Literature

None of the models developed in in vitro fertilization/intracytoplasmic sperm injection (IVF/ICSI) is sufficiently good predictors of pregnancy. The aim of this study was to determine whether ratios between prognostic factors could predict the clinical pregnancy rate in IVF/ICSI. We analyzed IVF/ICSI cycles (based on long GnRH agonist-FSH protocols) at two ART centers (the second to validate externally the data). The ratios studied were (i) the total FSH dose divided by the serum estradiol level on the hCG trigger day, (ii) the total FSH dose divided by the number of mature oocytes, (iii) the serum estradiol level on the trigger day divided by the number of mature oocytes, (iv) the serum estradiol level on the trigger day divided by the endometrial thickness on the trigger day, (v) the serum estradiol level on the trigger day divided by the number of mature oocytes and then by the number of grade 1 or 2 embryos obtained, and (vi) the serum estradiol level on the trigger day divided by the endometrial thickness on the trigger day and then by the number of grade 1 or 2 embryos obtained. The analysis covered 2421 IVF/ICSI cycles with an embryo transfer, leading to 753 clinical pregnancies (31.1% per transfer). Four ratios were significantly predictive in both centers; their discriminant power remained moderate (area under the receiver operating characteristic curve between 0.574 and 0.610). In contrast, the models' calibration was excellent (coefficients: 0.943-0.978; p < 0.001). Our ratios were no better than existing models in IVF/ICSI programs. In fact, a strongly discriminant predictive model will be probably never be obtained, given the many factors that influence the occurrence of a pregnancy.

Calibration: the Achilles heel of predictive analytics

BACKGROUND

The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.

MAIN TEXT

Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.

CONCLUSION

Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

Can we predict the IVF/ICSI live birth rate?

Objectives:

To find a pretreatment predictor for achieving a live birth. Assisted reproduction technology with IVF/ICSI is the ultimate chance for some couples to conceive a child. The expectations are high and it is important to give them a realistic perspective about the chances of achieving a live birth.

Methods:

A retrospective cohort study of all IVF/ICSI cycles performed in our center between 2012 and 2016. We considered only those cycles with a live birth delivery after 24 weeks, or cycles with no surplus embryos left. The following data was evaluated: AMH; AFC; age; BMI; previous diagnosis; type of treatment; number of previous deliveries; ethnicity, smoking status. Univariate and multivariate analysis were used to examine the association of live birth with baseline patient characteristics. We determined the odds-ratio for all the statistically significant variables (p<0.05), in a multivariate model. The results are presented according to the predictors founded.

Results:

739 cycles were evaluated: 9.1% were canceled; 10.2% did not have oocytes; 15.6% did not have D2 embryos; 31.4% achieved a live birth. The univariate analysis revealed statistically significant differences regarding AMH, AFC and women’s age between couples with and without a live birth (p<0.001), and the cause of infertility. We found no association with live births in other variables. These variables were categorized and used in a multivariate analysis.

Conclusion:

Age, AMH, AFC and cause, when sub-classified, are independently associated with the results of an IVF/ICSI treatment. These results enable couples to face real expectations in their particular scenario.

Predicting the cumulative chance of live birth over multiple complete cycles of in vitro fertilization: an external validation study

STUDY QUESTION

Are the published pre-treatment and post-treatment McLernon models, predicting cumulative live birth rates (LBR) over multiple complete IVF cycles, valid in a different context?

SUMMARY ANSWER

With minor recalibration of the pre-treatment model, both McLernon models accurately predict cumulative LBR in a different geographical context and a more recent time period.

WHAT IS KNOWN ALREADY

Previous IVF prediction models have estimated the chance of a live birth after a single fresh embryo transfer, thereby excluding the important contribution of embryo cryopreservation and subsequent IVF cycles to cumulative LBR. In contrast, the recently developed McLernon models predict the cumulative chance of a live birth over multiple complete IVF cycles at two certain time points: (i) before initiating treatment using baseline characteristics (pre-treatment model) and (ii) after the first IVF cycle adding treatment related information to update predictions (post-treatment model). Before implementation of these models in clinical practice, their predictive performance needs to be validated in an independent cohort.

STUDY DESIGN, SIZE, DURATION

External validation study in an independent prospective cohort of 1515 Dutch women who participated in the OPTIMIST study (NTR2657) and underwent their first IVF treatment between 2011 and 2014. Participants underwent a total of 2881 complete treatment cycles, with a complete cycle defined as all fresh and frozen thawed embryo transfers resulting from one episode of ovarian stimulation. The follow up duration was 18 months after inclusion, and the primary outcome was ongoing pregnancy leading to live birth.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Model performance was externally validated up to three complete treatment cycles, using the linear predictor as described by McLernon et al. to calculate the probability of a live birth. Discrimination was expressed by the c-statistic and calibration was depicted graphically in a calibration plot. In contrast to the original model development cohort, anti-Müllerian hormone (AMH), antral follicle count (AFC) and body weight were available in the OPTIMIST cohort, and evaluated as potential additional predictors for model improvement.

MAIN RESULTS AND THE ROLE OF CHANCE

Applying the McLernon models to the OPTIMIST cohort, the c-statistic of the pre-treatment model was 0.62 (95% CI: 0.59-0.64) and of the post-treatment model 0.71 (95% CI: 0.69-0.74). The calibration plot of the pre-treatment model indicated a slight overestimation of the cumulative LBR. To improve calibration, the pre-treatment model was recalibrated by subtracting 0.35 from the intercept. The post-treatment model calibration plot revealed accurate cumulative LBR predictions. After addition of AMH, AFC and body weight to the McLernon models, the c-statistic of the updated pre-treatment model improved slightly to 0.66 (95% CI: 0.64-0.68), and of the updated post-treatment model remained at the previous level of 0.71 (95% CI: 0.69-0.73). Using the recalibrated pre-treatment model, a woman aged 30 years with 2 years of primary infertility who starts ICSI treatment for male factor infertility has a chance of 40% of a live birth from the first complete cycle, increasing to 72% over three complete cycles. If this woman weighs 70 kg, has an AMH of 1.5 ng/mL and an AFC of 10 measured at the beginning of her treatment, the updated pre-treatment model revises the estimated chance of a live birth to 30% in the first complete cycle and 59% over three complete cycles. If this woman then has five retrieved oocytes, no embryos cryopreserved and a single fresh cleavage stage embryo transfer in her first ICSI cycle, the post-treatment model estimates the chances of a live birth at 28 and 58%, respectively.

LIMITATIONS, REASONS FOR CAUTION

Two randomized controlled trials (RCT) evaluating the effectiveness of gonadotropin dose individualization on basis of the AFC were nested within the OPTIMIST study. The strict dosing regimens, the RCT in- and exclusion criteria and the limited follow up time of 18 months might have influenced model performance in this independent cohort. Also, consistent with the original model development study, external validation was performed using the optimistic assumption that the cumulative LBR in couples who discontinue treatment without a live birth would have been equal to that of those who continue treatment.

WIDER IMPLICATIONS OF THE FINDINGS

After national recalibration to account for geographical differences in IVF treatment, the McLernon prediction models can be introduced as new counselling tools in clinical practice to inform patients and to complement clinical reasoning. These models are the first to offer an objective and personalized estimate of the cumulative probability of a live birth over multiple complete IVF cycles.

STUDY FUNDING/COMPETING INTEREST(S)

No external funds were obtained for this study. M.J.C.E., D.J.M. and S.B. have nothing to disclose. J.A.L, S.C.O, T.C.v.T. and H.LT. received an unrestricted personal grant from Merck BV. B.W.M. is supported by a NHMRC Practitioner Fellowship (GNT1082548) and reports consultancy for ObsEva, Merck and Guerbet. F.J.M.B. receives monetary compensation as a member of the external advisory board for Merck BV (the Netherlands) and Ferring pharmaceutics BV (the Netherlands), for consultancy work for Gedeon Richter (Belgium) and Roche Diagnostics on automated AMH assay development, and for a research cooperation with Ansh Labs (USA).

TRIAL REGISTRATION NUMBER

Not applicable.

Making IVF more effective through the evolution of prediction models: is prognosis the missing piece of the puzzle?

Assisted reproductive technology has evolved tremendously since the emergence of in vitro fertilization (IVF). In the course of the recent decade, there have been significant efforts in order to minimize multiple gestations, while improving percentages of singleton pregnancies and offering individualized services in IVF, in line with the trend of personalized medicine. Patients as well as clinicians and the entire IVF team benefit majorly from 'knowing what to expect' from an IVF cycle. Hereby, the question that has emerged is to what extent prognosis could facilitate toward the achievement of the above goal. In the current review, we present prediction models based on patients' characteristics and IVF data, as well as models based on embryo morphology and biomarkers during culture shaping a complication free and cost-effective personalized treatment. The starting point for the implementation of prediction models was initiated by the aspiration of moving toward optimal practice. Thus, prediction models could serve as useful tools that could safely set the expectations involved during this journey guiding and making IVF treatment more effective. The aim and scope of this review is to thoroughly present the evolution and contribution of prediction models toward an efficient IVF treatment.

ABBREVIATIONS

IVF: In vitro fertilization; ART: assisted reproduction techniques; BMI: body mass index; OHSS: ovarian hyperstimulation syndrome; eSET: elective single embryo transfer; ESHRE: European Society of Human Reproduction and Embryology; mtDNA: mitochondrial DNA; nDNA: nuclear DNA; ICSI: intracytoplasmic sperm injection; MBR: multiple birth rates; LBR: live birth rates; SART: Society for Assisted Reproductive Technology Clinic Outcome Reporting System; AFC: antral follicle count; GnRH: gonadotrophin releasing hormone; FSH: follicle stimulating hormone; LH: luteinizing hormone; AMH: anti-Müllerian hormone; DHEA: dehydroepiandrosterone; PCOS: polycystic ovarian syndrome; NPCOS: non-polycystic ovarian syndrome; CE: cost-effectiveness; CC: clomiphene citrate; ORT: ovarian reserve test; EU: embryo-uterus; DET: double embryo transfer; CES: Cumulative Embryo Score; GES: Graduated Embryo Score; CSS: Combined Scoring System; MSEQ: Mean Score of Embryo Quality; IMC: integrated morphology cleavage; EFNB2: ephrin-B2; CAMK1D: calcium/calmodulin-dependent protein kinase 1D; GSTA4: glutathione S-transferase alpha 4; GSR: glutathione reductase; PGR: progesterone receptor; AMHR2: anti-Müllerian hormone receptor 2; LIF: leukemia inhibitory factor; sHLA-G: soluble human leukocyte antigen G.

Predicting the chances of a live birth after one or more complete cycles of in vitro fertilisation: population based study of linked cycle data from 113 873 women

OBJECTIVE

 To develop a prediction model to estimate the chances of a live birth over multiple complete cycles of in vitro fertilisation (IVF) based on a couple's specific characteristics and treatment information.

DESIGN

 Population based cohort study.

SETTING

 All licensed IVF clinics in the UK. National data from the Human Fertilisation and Embryology Authority register.

PARTICIPANTS

 All 253 417 women who started IVF (including intracytoplasmic sperm injection) treatment in the UK from 1999 to 2008 using their own eggs and partner's sperm.

MAIN OUTCOME MEASURE

 Two clinical prediction models were developed to estimate the individualised cumulative chance of a first live birth over a maximum of six complete cycles of IVF-one model using information available before starting treatment and the other based on additional information collected during the first IVF attempt. A complete cycle is defined as all fresh and frozen-thawed embryo transfers arising from one episode of ovarian stimulation.

RESULTS

 After exclusions, 113 873 women with 184 269 complete cycles were included, of whom 33 154 (29.1%) had a live birth after their first complete cycle and 48 925 (43.0%) after six complete cycles. Key pretreatment predictors of live birth were the woman's age (31 v 37 years; adjusted odds ratio 1.66, 95% confidence interval 1.62 to 1.71) and duration of infertility (3 v 6 years; 1.09, 1.08 to 1.10). Post-treatment predictors included number of eggs collected (13 v 5 eggs; 1.29, 1.27 to 1.32), cryopreservation of embryos (1.91, 1.86 to 1.96), the woman's age (1.53, 1.49 to 1.58), and stage of embryos transferred (eg, double blastocyst v double cleavage; 1.79, 1.67 to 1.91). Pretreatment, a 30 year old woman with two years of unexplained primary infertility has a 46% chance of having a live birth from the first complete cycle of IVF and a 79% chance over three complete cycles. If she then has five eggs collected in her first complete cycle followed by a single cleavage stage embryo transfer (with no embryos left for freezing) her chances change to 28% and 56%, respectively.

CONCLUSIONS

 This study provides an individualised estimate of a couple's cumulative chances of having a baby over a complete package of IVF both before treatment and after the first fresh embryo transfer. This novel resource may help couples plan their treatment and prepare emotionally and financially for their IVF journey.

Definition by FSH, AMH and embryo numbers of good-, intermediate- and poor-prognosis patients suggests previously unknown IVF outcome-determining factor associated with AMH

BACKGROUND

Though outcome models have been proposed previously, it is unknown whether cutoffs in clinical pregnancy and live birth rates at all ages are able to classify in vitro fertilization (IVF) patients into good-, intermediate- and poor prognosis.

METHODS

We here in 3 infertile patient cohorts, involving 1247, 1514 and 632 women, built logistic regression models based on 3 functional ovarian reserve (FOR) parameters, including (1) number of good quality embryos, (2) follicle stimulating hormone (FSH, mIU/mL) and (3) anti-Müllerian hormone (AMH, ng/mL), determining whether clinical pregnancy and live birth rates can discriminate between good, intermediate and poor prognosis patients.

RESULTS

All models, indeed, allowed at all ages for separation by prognosis, though cut offs changed with age. In the embryo model, increasing embryo production resulted in linear improvement of IVF outcomes despite transfer of similar embryo numbers; in the FSH model outcomes and FSH levels related inversely, while the association of AMH followed a bell-shaped polynomial pattern, demonstrating "best" outcomes at mid-ranges. All 3 models demonstrated increasingly poor outcomes with advancing ages, though "best" AMH even above age 43 was still associated with unexpectedly good pregnancy and delivery outcomes. Excessively high AMH, in contrast, was at all ages associated with spiking miscarriage rates.

CONCLUSIONS

At varying peripheral serum concentrations, AMH, thus, demonstrates hithero unknown and contradictory effects on IVF outcomes, deserving at different concentrations investigation as a potential therapeutic agent, with pregnancy-supporting and pregnancy-interrupting properties.

Predicting the success of IVF: external validation of the van Loendersloot's model

STUDY QUESTION

Is the predictive model for IVF success proposed by van Loendersloot et al. valid in a different geographical and cultural context?

SUMMARY ANSWER

The model discriminates well but was less accurate than in the original context where it was developed.

WHAT IS ALREADY KNOWN

Several independent groups have developed models that combine different variables with the aim of estimating the chance of pregnancy with IVF but only four of them have been externally validated. One of these four, the van Loendersloot's model, deserves particular attention and further investigation for at least three reasons; (i) the reported area under the receiver operating characteristics curve (c-statistics) in the temporal validation setting was the highest reported to date (0.68), (ii) the perspective of the model is clinically wise since it includes variables obtained from previous failed cycles, if any, so it can be applied to any women entering an IVF cycle, (iii) the model lacks external validation in a geographically different center.

STUDY DESIGN, SIZE, DURATION

Retrospective cohort study of women undergoing oocyte retrieval for IVF between January 2013 and December 2013 at the infertility unit of the Fondazione Ca' Granda, Ospedale Maggiore Policlinico of Milan, Italy. Only the first oocyte retrieval cycle performed during the study period was included in the study. Women with previous IVF cycles were excluded if the last one before the study cycle was in another center. The main outcome was the cumulative live birth rate per oocytes retrieval.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Seven hundred seventy-two women were selected. Variables included in the van Loendersloot's model and the relative weights (beta) were used. The variable resulting from this combination (Y) was transformed into a probability. The discriminatory capacity was assessed using the c-statistics. Calibration was made using a logistic regression that included Y as the unique variable and live birth as the outcome. Data are presented using both the original and the calibrated models. Performance was evaluated correlating the mean predicted chances of live births in the five quintiles and the observed rates.

MAIN RESULTS AND THE ROLE OF CHANCE

Two-hundred-eleven live births (27%) were obtained. The c-statistic was 0.64 (95% CI: 0.61-0.67, P < 0.001). The slope of the linear predictor (calibration slope) expressed as an Odds Ratio was 1.81 (95% CI: 1.46-2.24, P < 0.001), corresponding to a beta of 0.630. The calibration intercept was +0.349 (P = 0.13). While a clear discrepancy exists using the original model, data appear properly distributed with the calibrated model. The Pearson coefficient of the correlation between the mean predicted chances of live births in the five quintiles and the observed rates was 0.99 (P = 0.002).

LIMITATIONS, REASONS FOR CAUTION

Data were collected retrospectively, thus exposing them to potential inaccuracies. The selection criteria for access to IVF adopted in our center might be too stringent, leading to the exclusion of women with a poor, yet acceptable chance of live birth. Therefore, the validity of the model in women with a very low chance of live birth could not be tested.

WIDER IMPLICATIONS OF THE FINDINGS

The van Loendersloot's model can be used in other contexts but it is important that it has local calibration. It may help in counseling couples about their chance of success but it cannot be used to exclude treatments. Further research is needed to improve the discriminatory performance of IVF predictive models.

STUDY FUNDING/COMPETING INTERESTS

None.

TRIAL REGISTRATION NUMBER

Not applicable.

Predicting the chance of live birth for women undergoing IVF: a novel pretreatment counselling tool

STUDY QUESTION

Which pretreatment patient variables have an effect on live birth rates following assisted conception?

SUMMARY ANSWER

The predictors in the final multivariate logistic regression model found to be significantly associated with reduced chances of IVF/ICSI success were increasing age (particularly above 36 years), tubal factor infertility, unexplained infertility and Asian or Black ethnicity.

WHAT IS KNOWN ALREADY

The two most widely recognized prediction models for live birth following IVF were developed on data from 1991 to 2007; pre-dating significant changes in clinical practice. These existing IVF outcome prediction models do not incorporate key pretreatment predictors, such as BMI, ethnicity and ovarian reserve, which are readily available now.

STUDY DESIGN, SIZE, DURATION

In this cohort study a model to predict live birth was derived using data collected from 9915 women who underwent IVF/ICSI treatment at any CARE (Centres for Assisted Reproduction) clinic from 2008 to 2012. Model validation was performed on data collected from 2723 women who underwent treatment in 2013. The primary outcome for the model was live birth, which was defined as any birth event in which at least one baby was born alive and survived for more than 1 month.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Data were collected from 12 fertility clinics within the CARE consortium in the UK. Multivariable logistic regression was used to develop the model. Discriminatory ability was assessed using the area under receiver operating characteristic (AUROC) curve, and calibration was assessed using calibration-in-the-large and the calibration slope test.

MAIN RESULTS AND THE ROLE OF CHANCE

The predictors in the final model were female age, BMI, ethnicity, antral follicle count (AFC), previous live birth, previous miscarriage, cause and duration of infertility. Upon assessing predictive ability, the AUROC curve for the final model and validation cohort was (0.62; 95% confidence interval (CI) 0.61-0.63) and (0.62; 95% CI 0.60-0.64) respectively. Calibration-in-the-large showed a systematic over-estimation of the predicted probability of live birth (Intercept (95% CI) = -0.168 (-0.252 to -0.084), P < 0.001). However, the calibration slope test was not significant (slope (95% CI) = 1.129 (0.893-1.365), P = 0.28). Due to the calibration-in-the-large test being significant we recalibrated the final model. The recalibrated model showed a much-improved calibration.

LIMITATIONS, REASONS FOR CAUTION

Our model is unable to account for factors such as smoking and alcohol that can affect IVF/ICSI outcome and is somewhat restricted to representing the ethnic distribution and outcomes for the UK population only. We were unable to account for socioeconomic status and it may be that by having 75% of the population paying privately for their treatment, the results cannot be generalized to people of all socioeconomic backgrounds. In addition, patients and clinicians should understand this model is designed for use before treatment begins and does not include variables that become available (oocyte, embryo and endometrial) as treatment progresses. Finally, this model is also limited to use prior to first cycle only.

WIDER IMPLICATIONS OF THE FINDINGS

To our knowledge, this is the first study to present a novel, up-to-date model encompassing three readily available prognostic factors; female BMI, ovarian reserve and ethnicity, which have not previously been used in prediction models for IVF outcome. Following geographical validation, the model can be used to build a user-friendly interface to aid decision-making for couples and their clinicians. Thereafter, a feasibility study of its implementation could focus on patient acceptability and quality of decision-making.

STUDY FUNDING/COMPETING INTEREST

None.

Antimüllerian hormone: prediction of cumulative live birth in gonadotropin-releasing hormone antagonist treatment for in vitro fertilization

OBJECTIVE

To assess the accuracy of antimüllerian hormone (AMH) in predicting cumulative live birth rate (CLBR) within 1 year after treatment initiation in GnRH antagonist treatment cycles for in vitro fertilization (IVF).

DESIGN

Observational (retrospective) substudy as part of an ongoing prospective cohort study.

SETTING

University medical center.

PATIENT(S)

A total of 487 patients scheduled for IVF/intracytoplasmic sperm injection (ICSI).

INTERVENTION(S)

Patients starting their first IVF/ICSI cycle with 150 or 225 IU recombinant FSH and GnRH antagonist cotreatment were included. Serum samples collected before the first IVF treatment were used to determine AMH. Treatment data after treatment initiation for a maximum of 1 year were recorded.

MAIN OUTCOME MEASURE(S)

Prediction of CLBR with the use of AMH.

RESULT(S)

The model for predicting CLBR within 1 year included age at first treatment, AMH, type of infertility, and previous assisted reproductive technology treatment leading to live birth. The accuracy in discriminating between women who did or did not achieve a live birth was only 59%. AMH had intermediate added value in the prediction of CLBR as demonstrated by the net reclassification improvement (total 29.8). A nomogram based on age and AMH was developed by which a subgroup of patients could be identified with the poorest pregnancy prospects.

CONCLUSION(S)

The predictive accuracy of AMH for 1-year CLBR in GnRH antagonist treatment cycles was limited and did not yield much additional value on top of age. Withholding treatment based on predictors such as age and AMH, or a combination, remains problematic.

CLINICAL TRIAL REGISTRATION NUMBER

www.clinicaltrials.gov, NCT02309073.

Antimüllerian hormone levels and antral follicle count as prognostic indicators in a personalized prediction model of live birth

OBJECTIVE

To compare antimüllerian hormone (AMH) and antral follicle count (AFC) separately and in combination with clinical characteristics for the prediction of live birth after controlled ovarian stimulation.

DESIGN

Retrospective development and temporal external validation of prediction model.

SETTING

Outpatient IVF clinic.

PATIENT(S)

We applied the boosted tree method to develop three prediction models incorporating clinical characteristics plus AMH or AFC or the combination on 2,124 linked IVF cycles from 2006 to 2010 and temporally externally validated predicted live-birth probabilities with an independent data set comprising 1,121 cycles from 2011 to 2012.

INTERVENTION(S)

None.

MAIN OUTCOME MEASURE(S)

Predictive power (posterior log of odds ratio compared to age, or PLORA), reclassification, receiver operator characteristic analysis, calibration, dynamic range.

RESULT(S)

Predictive power, was highest for the AMH model (PLORA = 29.1), followed by the AMH-AFC model (PLORA = 28.3) and AFC model (PLORA = 22.5). The prediction errors were 1% to <5% in each prognostic tier for all three models, except for the predicted live-birth probabilities of <10% in the AFC model, where the prediction error was 8%. The improvement in predictive power was highest for the AMH model: 76.2% improvement over age alone relative to 59% improvement for AFC and 73.3% for the combined model. Receiver operating characteristic analysis demonstrated that the AMH and the combined model had comparable discrimination (area under the curve = 0.716) and similar prediction error for high and low strata of live-birth prediction, with an improvement of 6.3% over age alone.

CONCLUSION(S)

The validated prediction model confirmed that AMH when combined with clinical characteristics can accurately identify the likelihood of live birth with a low prediction error. AFC provided no added predictive value beyond AMH.

External validation and calibration of IVFpredict: a national prospective cohort study of 130,960 in vitro fertilisation cycles

BACKGROUND

Accurately predicting the probability of a live birth after in vitro fertilisation (IVF) is important for patients, healthcare providers and policy makers. Two prediction models (Templeton and IVFpredict) have been previously developed from UK data and are widely used internationally. The more recent of these, IVFpredict, was shown to have greater predictive power in the development dataset. The aim of this study was external validation of the two models and comparison of their predictive ability.

METHODS AND FINDINGS

130,960 IVF cycles undertaken in the UK in 2008-2010 were used to validate and compare the Templeton and IVFpredict models. Discriminatory power was calculated using the area under the receiver-operator curve and calibration assessed using a calibration plot and Hosmer-Lemeshow statistic. The scaled modified Brier score, with measures of reliability and resolution, were calculated to assess overall accuracy. Both models were compared after updating for current live birth rates to ensure that the average observed and predicted live birth rates were equal. The discriminative power of both methods was comparable: the area under the receiver-operator curve was 0.628 (95% confidence interval (CI): 0.625-0.631) for IVFpredict and 0.616 (95% CI: 0.613-0.620) for the Templeton model. IVFpredict had markedly better calibration and higher diagnostic accuracy, with calibration plot intercept of 0.040 (95% CI: 0.017-0.063) and slope of 0.932 (95% CI: 0.839-1.025) compared with 0.080 (95% CI: 0.044-0.117) and 1.419 (95% CI: 1.149-1.690) for the Templeton model. Both models underestimated the live birth rate, but this was particularly marked in the Templeton model. Updating the models to reflect improvements in live birth rates since the models were developed enhanced their performance, but IVFpredict remained superior.

CONCLUSION

External validation in a large population cohort confirms IVFpredict has superior discrimination and calibration for informing patients, clinicians and healthcare policy makers of the probability of live birth following IVF.

To what extent does Anti-Mullerian Hormone contribute to a better prediction of live birth after IVF?

OBJECTIVE

We assessed the predictive value added by Anti-Mullerian Hormone (AMH) to currently validated live birth (LB) prediction models.

METHODS

Based on recent data from our center, we compared the external validity of the Templeton Model (TM) and its recent improvement (TMA) to select our model of reference. The added predictive value of AMH was assessed in testing the likelihood ratio significance and the Net Reclassification Index (NRI). The surrogate utility of AMH was tested by conducting an exploratory stepwise logistic regression.

RESULTS

Based on 715 cycles, the original TM had poor performances (auROC C = 0.61 [0.58, 0.66], improving by fitting TM to our data (C = 0.71[0.66, 0.75]. TMA fitting proved better (C = 0.76; 95 %CI: 0.71, 0.80) and was selected as model of reference. Adding AMH to TMA or TM had no effect on discrimination (C = 0.76; 95 %CI: 0.72, 0.80), the likelihood ratio test was significant (p = 0.023), but the NRI was not (6.7 %; p = 0.055). A stepwise exploratory logistic regression identified the effects of age, previous IVF resulting in LB, time trend and AMH, leading to a prediction model reduced to four predictors (C = 0.75 [0.70, 0.81]).

CONCLUSION

The added predictive value of AMH is limited. A possible surrogate/simplifying effect of AMH was found in eliminating 9/13 predictors from the model of reference. We conclude that whereas AMH does not add significant predictive value to the existing model, it contributes to simplifying the equation to reliable, easy to collect, and available in all databases predictors: age, AMH, time trend and female previous fertility history.