Evaluation of deep convolutional neural networks in classifying human embryo images based on their morphological quality

Visual interpretability of image-based classification models by generative latent space disentanglement applied to in vitro fertilization

The success of deep learning in identifying complex patterns exceeding human intuition comes at the cost of interpretability. Non-linear entanglement of image features makes deep learning a "black box" lacking human meaningful explanations for the models' decision. We present DISCOVER, a generative model designed to discover the underlying visual properties driving image-based classification models. DISCOVER learns disentangled latent representations, where each latent feature encodes a unique classification-driving visual property. This design enables "human-in-the-loop" interpretation by generating disentangled exaggerated counterfactual explanations. We apply DISCOVER to interpret classification of in vitro fertilization embryo morphology quality. We quantitatively and systematically confirm the interpretation of known embryo properties, discover properties without previous explicit measurements, and quantitatively determine and empirically verify the classification decision of specific embryo instances. We show that DISCOVER provides human-interpretable understanding of "black box" classification models, proposes hypotheses to decipher underlying biomedical mechanisms, and provides transparency for the classification of individual predictions.

Current applications of artificial intelligence in assisted reproductive technologies through the perspective of a patient's journey

PURPOSE OF REVIEW

This review highlights the timely relevance of artificial intelligence in enhancing assisted reproductive technologies (ARTs), particularly in-vitro fertilization (IVF). It underscores artificial intelligence's potential in revolutionizing patient outcomes and operational efficiency by addressing challenges in fertility diagnoses and procedures.

RECENT FINDINGS

Recent advancements in artificial intelligence, including machine learning and predictive modeling, are making significant strides in optimizing IVF processes such as medication dosing, scheduling, and embryological assessments. Innovations include artificial intelligence augmented diagnostic testing, predictive modeling for treatment outcomes, scheduling optimization, dosing and protocol selection, follicular and hormone monitoring, trigger timing, and improved embryo selection. These developments promise to refine treatment approaches, enhance patient engagement, and increase the accuracy and scalability of fertility treatments.

SUMMARY

The integration of artificial intelligence into reproductive medicine offers profound implications for clinical practice and research. By facilitating personalized treatment plans, standardizing procedures, and improving the efficiency of fertility clinics, artificial intelligence technologies pave the way for value-based, accessible, and efficient fertility services. Despite the promise, the full potential of artificial intelligence in ART will require ongoing validation and ethical considerations to ensure equitable and effective implementation.

Deep learning pipeline reveals key moments in human embryonic development predictive of live birth after in vitro fertilization

Demand for in vitro fertilization (IVF) treatment is growing; however, success rates remain low partly due to difficulty in selecting the best embryo to be transferred. Current manual assessments are subjective and may not take advantage of the most informative moments in embryo development. Here, we apply convolutional neural networks (CNNs) to identify key windows in pre-implantation human development that can be linked to embryo viability and are therefore suitable for the early grading of IVF embryos. We show how machine learning models trained at these developmental time points can be used to refine overall embryo viability assessment. Exploiting the well-known capabilities of transfer learning, we illustrate the performance of CNN models for very limited datasets, paving the way for the use on a clinic-by-clinic basis, catering for local data heterogeneity.

Machine learning in time-lapse imaging to differentiate embryos from young vs old mice†

Time-lapse microscopy for embryos is a non-invasive technology used to characterize early embryo development. This study employs time-lapse microscopy and machine learning to elucidate changes in embryonic growth kinetics with maternal aging. We analyzed morphokinetic parameters of embryos from young and aged C57BL6/NJ mice via continuous imaging. Our findings show that aged embryos accelerated through cleavage stages (from 5-cells) to morula compared to younger counterparts, with no significant differences observed in later stages of blastulation. Unsupervised machine learning identified two distinct clusters comprising of embryos from aged or young donors. Moreover, in supervised learning, the extreme gradient boosting algorithm successfully predicted the age-related phenotype with 0.78 accuracy, 0.81 precision, and 0.83 recall following hyperparameter tuning. These results highlight two main scientific insights: maternal aging affects embryonic development pace, and artificial intelligence can differentiate between embryos from aged and young maternal mice by a non-invasive approach. Thus, machine learning can be used to identify morphokinetics phenotypes for further studies. This study has potential for future applications in selecting human embryos for embryo transfer, without or in complement with preimplantation genetic testing.

Improving Deep Learning-Based Algorithm for Ploidy Status Prediction Through Combined U-NET Blastocyst Segmentation and Sequential Time-Lapse Blastocysts Images

Background

Several approaches have been proposed to optimize the construction of an artificial intelligence-based model for assessing ploidy status. These encompass the investigation of algorithms, refining image segmentation techniques, and discerning essential patterns throughout embryonic development. The purpose of the current study was to evaluate the effectiveness of using U-NET architecture for embryo segmentation and time-lapse embryo image sequence extraction, three and ten hr before biopsy to improve model accuracy for prediction of embryonic ploidy status.

Methods

A total of 1.020 time-lapse videos of blastocysts with known ploidy status were used to construct a convolutional neural network (CNN)-based model for ploidy detection. Sequential images of each blastocyst were extracted from the time-lapse videos over a period of three and ten hr prior to the biopsy, generating 31.642 and 99.324 blastocyst images, respectively. U-NET architecture was applied for blastocyst image segmentation before its implementation in CNN-based model development.

Results

The accuracy of ploidy prediction model without applying the U-NET segmented sequential embryo images was 0.59 and 0.63 over a period of three and ten hr before biopsy, respectively. Improved model accuracy of 0.61 and 0.66 was achieved, respectively with the implementation of U-NET architecture for embryo segmentation on the current model. Extracting blastocyst images over a 10 hr period yields higher accuracy compared to a three-hr extraction period prior to biopsy.

Conclusion

Combined implementation of U-NET architecture for blastocyst image segmentation and the sequential compilation of ten hr of time-lapse blastocyst images could yield a CNN-based model with improved accuracy in predicting ploidy status.

The prospect of artificial intelligence to personalize assisted reproductive technology

Infertility affects 1-in-6 couples, with repeated intensive cycles of assisted reproductive technology (ART) required by many to achieve a desired live birth. In ART, typically, clinicians and laboratory staff consider patient characteristics, previous treatment responses, and ongoing monitoring to determine treatment decisions. However, the reproducibility, weighting, and interpretation of these characteristics are contentious, and highly operator-dependent, resulting in considerable reliance on clinical experience. Artificial intelligence (AI) is ideally suited to handle, process, and analyze large, dynamic, temporal datasets with multiple intermediary outcomes that are generated during an ART cycle. Here, we review how AI has demonstrated potential for optimization and personalization of key steps in a reproducible manner, including: drug selection and dosing, cycle monitoring, induction of oocyte maturation, and selection of the most competent gametes and embryos, to improve the overall efficacy and safety of ART.

Artificial intelligence in time-lapse system: advances, applications, and future perspectives in reproductive medicine

With the rising demand for in vitro fertilization (IVF) cycles, there is a growing need for innovative techniques to optimize procedure outcomes. One such technique is time-lapse system (TLS) for embryo incubation, which minimizes environmental changes in the embryo culture process. TLS also significantly advances predicting embryo quality, a crucial determinant of IVF cycle success. However, the current subjective nature of embryo assessments is due to inter- and intra-observer subjectivity, resulting in highly variable results. To address this challenge, reproductive medicine has gradually turned to artificial intelligence (AI) to establish a standardized and objective approach, aiming to achieve higher success rates. Extensive research is underway investigating the utilization of AI in TLS to predict multiple outcomes. These studies explore the application of popular AI algorithms, their specific implementations, and the achieved advancements in TLS. This review aims to provide an overview of the advances in AI algorithms and their particular applications within the context of TLS and the potential challenges and opportunities for further advancements in reproductive medicine.

Accurate staging of chick embryonic tissues via deep learning of salient features

Recent work shows that the developmental potential of progenitor cells in the HH10 chick brain changes rapidly, accompanied by subtle changes in morphology. This demands increased temporal resolution for studies of the brain at this stage, necessitating precise and unbiased staging. Here, we investigated whether we could train a deep convolutional neural network to sub-stage HH10 chick brains using a small dataset of 151 expertly labelled images. By augmenting our images with biologically informed transformations and data-driven preprocessing steps, we successfully trained a classifier to sub-stage HH10 brains to 87.1% test accuracy. To determine whether our classifier could be generally applied, we re-trained it using images (269) of randomised control and experimental chick wings, and obtained similarly high test accuracy (86.1%). Saliency analyses revealed that biologically relevant features are used for classification. Our strategy enables training of image classifiers for various applications in developmental biology with limited microscopy data.

Deep learning for embryo evaluation using time-lapse: a systematic review of diagnostic test accuracy

OBJECTIVE

This study aimed to investigate the accuracy of convolutional neural network models in the assessment of embryos using time-lapse monitoring.

DATA SOURCES

A systematic search was conducted in PubMed and Web of Science databases from January 2016 to December 2022. The search strategy was carried out by using key words and MeSH (Medical Subject Headings) terms.

STUDY ELIGIBILITY CRITERIA

Studies were included if they reported the accuracy of convolutional neural network models for embryo evaluation using time-lapse monitoring. The review was registered with PROSPERO (International Prospective Register of Systematic Reviews; identification number CRD42021275916).

METHODS

Two reviewer authors independently screened results using the Covidence systematic review software. The full-text articles were reviewed when studies met the inclusion criteria or in any uncertainty. Nonconsensus was resolved by a third reviewer. Risk of bias and applicability were evaluated using the QUADAS-2 tool and the modified Joanna Briggs Institute or JBI checklist.

RESULTS

Following a systematic search of the literature, 22 studies were identified as eligible for inclusion. All studies were retrospective. A total of 522,516 images of 222,998 embryos were analyzed. Three main outcomes were evaluated: successful in vitro fertilization, blastocyst stage classification, and blastocyst quality. Most studies reported >80% accuracy, and embryologists were outperformed in some. Ten studies had a high risk of bias, mostly because of patient bias.

CONCLUSION

The application of artificial intelligence in time-lapse monitoring has the potential to provide more efficient, accurate, and objective embryo evaluation. Models that examined blastocyst stage classification showed the best predictions. Models that predicted live birth had a low risk of bias, used the largest databases, and had external validation, which heightens their relevance to clinical application. Our systematic review is limited by the high heterogeneity among the included studies. Researchers should share databases and standardize reporting.

Embryo ploidy status classification through computer-assisted morphology assessment

BACKGROUND

Preimplantation genetic testing for aneuploidy has been proven to be effective in determining the embryo's chromosomal or ploidy status. The test requires a biopsy of embryonic cells on day 3, 5, or 6 from which complete information on the chromosomes would be obtained. The main drawbacks of preimplantation genetic testing for aneuploidy include its relatively invasive approach and the lack of research studies on the long-term effects of preimplantation genetic testing for aneuploidy.

OBJECTIVE

Computer-assisted predictive modeling through machine learning and deep learning algorithms has been proposed to minimize the use of invasive preimplantation genetic testing for aneuploidy. The capability to predict morphologic characteristics of embryo ploidy status creates a meaningful support system for decision-making before further treatment.

STUDY DESIGN

Image processing is a component in developing a predictive model specialized in image classification through which a model is able to differentiate images based on unique features. Image processing is obtained through image augmentation to capture segmented embryos and perform feature extraction. Furthermore, multiple machine learning and deep learning algorithms were used to create prediction-based modeling, and all of the prediction models undergo similar model performance assessments to determine the best model prediction algorithm.

RESULTS

An efficient artificial intelligence model that can predict embryo ploidy status was developed using image processing through a histogram of oriented gradient and then followed by principal component analysis. The gradient boosting algorithm showed an advantage against other algorithms and yielded an accuracy of 0.74, an aneuploid precision of 0.83, and an aneuploid predictive value (recall) of 0.84.

CONCLUSION

This research study proved that machine-assisted technology perceives the embryo differently than human observation and determined that further research on in vitro fertilization is needed. The study finding serves as a basis for developing a better computer-assisted prediction model.

Embryo Selection for IVF using Machine Learning Techniques Based on Light Microscopic Images of Embryo and Additional Factors

The current process of embryo selection in In Vitro Fertilization (IVF) process is based on morphological criteria, e.g., Istanbul scoring system and manually evaluated by embryologists; consequently, the assessment can be subjective. In the case of multiple embryos that have the same morphological grading, there is no guidance on how embryos should be prioritized to be transferred. This work aims to develop a deep learning-based model to classify viable and non-viable embryos using light microscopic images of an embryo. Additional features according to Istanbul grading system and the patients' age is also included in the model. Various models are evaluated and the best model based on the fusion of embryo images and additional features provides accuracy, sensitivity, and area under curve (AUC) of 65%, 74.29% and 0.72, respectively. The distributions of the prediction score corresponding to each additional feature are analysed and compared with pregnant and non-pregnant ground truths. We have found that the additional factors, such as age, embryo development stage, the quality of inner cell mass (ICM), and trophectoderm (TE) have a positive impact and enhanced the model prediction of implantation potential.

An annotated human blastocyst dataset to benchmark deep learning architectures for in vitro fertilization

Medical Assisted Reproduction proved its efficacy to treat the vast majority forms of infertility. One of the key procedures in this treatment is the selection and transfer of the embryo with the highest developmental potential. To assess this potential, clinical embryologists routinely work with static images (morphological assessment) or short video sequences (time-lapse annotation). Recently, Artificial Intelligence models were utilized to support the embryo selection procedure. Even though they have proven their great potential in different in vitro fertilization settings, there is still considerable room for improvement. To support the advancement of algorithms in this research field, we built a dataset consisting of static blastocyst images and additional annotations. As such, Gardner criteria annotations, depicting a morphological blastocyst rating scheme, and collected clinical parameters are provided. The presented dataset is intended to be used to train deep learning models on static morphological images to predict Gardner's criteria and clinical outcomes such as live birth. A benchmark of human expert's performance in annotating Gardner criteria is provided.

The use of voting ensembles to improve the accuracy of deep neural networks as a non-invasive method to predict embryo ploidy status

PURPOSE

To determine if creating voting ensembles combining convolutional neural networks (CNN), support vector machine (SVM), and multi-layer neural networks (NN) alongside clinical parameters improves the accuracy of artificial intelligence (AI) as a non-invasive method for predicting aneuploidy.

METHODS

A cohort of 699 day 5 PGT-A tested blastocysts was used to train, validate, and test a CNN to classify embryos as euploid/aneuploid. All embryos were analyzed using a modified FAST-SeqS next-generation sequencing method. Patient characteristics such as maternal age, AMH level, paternal sperm quality, and total number of normally fertilized (2PN) embryos were processed using SVM and NN. To improve model performance, we created voting ensembles using CNN, SVM, and NN to combine our imaging data with clinical parameter variations. Statistical significance was evaluated with a one-sample t-test with 2 degrees of freedom.

RESULTS

When assessing blastocyst images alone, the CNN test accuracy was 61.2% (± 1.32% SEM, n = 3 models) in correctly classifying euploid/aneuploid embryos (n = 140 embryos). When the best CNN model was assessed as a voting ensemble, the test accuracy improved to 65.0% (AMH; p = 0.1), 66.4% (maternal age; p = 0.06), 65.7% (maternal age, AMH; p = 0.08), 66.4% (maternal age, AMH, number of 2PNs; p = 0.06), and 71.4% (maternal age, AMH, number of 2PNs, sperm quality; p = 0.02) (n = 140 embryos).

CONCLUSIONS

By combining CNNs with patient characteristics, voting ensembles can be created to improve the accuracy of classifying embryos as euploid/aneuploid from CNN alone, allowing for AI to serve as a potential non-invasive method to aid in karyotype screening and selection of embryos.

Using artificial intelligence to avoid human error in identifying embryos: a retrospective cohort study

PURPOSE

To determine whether convolutional neural networks (CNN) can be used to accurately ascertain the patient identity (ID) of cleavage and blastocyst stage embryos based on image data alone.

METHODS

A CNN model was trained and validated over three replicates on a retrospective cohort of 4889 time-lapse embryo images. The algorithm processed embryo images for each patient and produced a unique identification key that was associated with the patient ID at a timepoint on day 3 (~ 65 hours post-insemination (hpi)) and day 5 (~ 105 hpi) forming our data library. When the algorithm evaluated embryos at a later timepoint on day 3 (~ 70 hpi) and day 5 (~ 110 hpi), it generates another key that was matched with the patient's unique key available in the library. This approach was tested using 400 patient embryo cohorts on day 3 and day 5 and number of correct embryo identifications with the CNN algorithm was measured.

RESULTS

CNN technology matched the patient identification within random pools of 8 patient embryo cohorts on day 3 with 100% accuracy (n = 400 patients; 3 replicates). For day 5 embryo cohorts, the accuracy within random pools of 8 patients was 100% (n = 400 patients; 3 replicates).

CONCLUSIONS

This study describes an artificial intelligence-based approach for embryo identification. This technology offers a robust witnessing step based on unique morphological features of each embryo. This technology can be integrated with existing imaging systems and laboratory protocols to improve specimen tracking.

Automatic characterization of human embryos at day 4 post-insemination from time-lapse imaging using supervised contrastive learning and inductive transfer learning techniques

BACKGROUND

Embryo morphology is a predictive marker for implantation success and ultimately live births. Viability evaluation and quality grading are commonly used to select the embryo with the highest implantation potential. However, the traditional method of manual embryo assessment is time-consuming and highly susceptible to inter- and intra-observer variability. Automation of this process results in more objective and accurate predictions.

METHOD

In this paper, we propose a novel methodology based on deep learning to automatically evaluate the morphological appearance of human embryos from time-lapse imaging. A supervised contrastive learning framework is implemented to predict embryo viability at day 4 and day 5, and an inductive transfer approach is applied to classify embryo quality at both times.

RESULTS

Results showed that both methods outperformed conventional approaches and improved state-of-the-art embryology results for an independent test set. The viability result achieved an accuracy of 0.8103 and 0.9330 and the quality results reached values of 0.7500 and 0.8001 for day 4 and day 5, respectively. Furthermore, qualitative results kept consistency with the clinical interpretation.

CONCLUSIONS

The proposed methods are up to date with the artificial intelligence literature and have been proven to be promising. Furthermore, our findings represent a breakthrough in the field of embryology in that they study the possibilities of embryo selection at day 4. Moreover, the grad-CAMs findings are directly in line with embryologists' decisions. Finally, our results demonstrated excellent potential for the inclusion of the models in clinical practice.

Artificial intelligence in the embryology laboratory: a review

The goal of an IVF cycle is a healthy live-born baby. Despite the many advances in the field of assisted reproductive technologies, accurately predicting the outcome of an IVF cycle has yet to be achieved. One reason for this is the method of selecting an embryo for transfer. Morphological assessment of embryos is the traditional method of evaluating embryo quality and selecting which embryo to transfer. However, this subjective method of assessing embryos leads to inter- and intra-observer variability, resulting in less than optimal IVF success rates. To overcome this, it is common practice to transfer more than one embryo, potentially resulting in high-risk multiple pregnancies. Although time-lapse incubators and preimplantation genetic testing for aneuploidy have been introduced to help increase the chances of live birth, the outcomes remain less than ideal. Utilization of artificial intelligence (AI) has become increasingly popular in the medical field and is increasingly being leveraged in the embryology laboratory to help improve IVF outcomes. Many studies have been published investigating the use of AI as an unbiased, automated approach to embryo assessment. This review summarizes recent AI advancements in the embryology laboratory.

Adaptive adversarial neural networks for the analysis of lossy and domain-shifted datasets of medical images

In machine learning for image-based medical diagnostics, supervised convolutional neural networks are typically trained with large and expertly annotated datasets obtained using high-resolution imaging systems. Moreover, the network's performance can degrade substantially when applied to a dataset with a different distribution. Here, we show that adversarial learning can be used to develop high-performing networks trained on unannotated medical images of varying image quality. Specifically, we used low-quality images acquired using inexpensive portable optical systems to train networks for the evaluation of human embryos, the quantification of human sperm morphology and the diagnosis of malarial infections in the blood, and show that the networks performed well across different data distributions. We also show that adversarial learning can be used with unlabelled data from unseen domain-shifted datasets to adapt pretrained supervised networks to new distributions, even when data from the original distribution are not available. Adaptive adversarial networks may expand the use of validated neural-network models for the evaluation of data collected from multiple imaging systems of varying quality without compromising the knowledge stored in the network.

Deep learning early warning system for embryo culture conditions and embryologist performance in the ART laboratory

Staff competency is a crucial component of the in vitro fertilization (IVF) laboratory quality management system because it impacts clinical outcomes and informs the key performance indicators (KPIs) used to continuously monitor and assess culture conditions. Contemporary quality control and assurance in the IVF lab can be automated (collect, store, retrieve, and analyze), to elevate quality control and assurance beyond the cursory monthly review. Here we demonstrate that statistical KPI monitoring systems for individual embryologist performance and culture conditions can be detected by artificial intelligence systems to provide systemic, early detection of adverse outcomes, and identify clinically relevant shifts in pregnancy rates, providing critical validation for two statistical process controls proposed in the Vienna Consensus Document; intracytoplasmic sperm injection (ICSI) fertilization rate and day 3 embryo quality.

Review of computer vision application in in vitro fertilization: the application of deep learning-based computer vision technology in the world of IVF

In vitro fertilization has been regarded as a forefront solution in treating infertility for over four decades, yet its effectiveness has remained relatively low. This could be attributed to the lack of advancements for the method of observing and selecting the most viable embryos for implantation. The conventional morphological assessment of embryos exhibits inevitable drawbacks which include time- and effort-consuming, and imminent risks of bias associated with subjective assessments performed by individual embryologists. A combination of these disadvantages, undeterred by the introduction of the time-lapse incubator technology, has been considered as a prominent contributor to the less preferable success rate of IVF cycles. Nonetheless, a recent surge of AI-based solutions for tasks automation in IVF has been observed. An AI-powered assistant could improve the efficiency of performing certain tasks in addition to offering accurate algorithms that behave as baselines to minimize the subjectivity of the decision-making process. Through a comprehensive review, we have discovered multiple approaches of implementing deep learning technology, each with varying degrees of success, for constructing the automated systems in IVF which could evaluate and even annotate the developmental stages of an embryo.