Predicting venous thromboembolism risk from exomes in the Critical Assessment of Genome Interpretation (CAGI) challenges

CAGI, the Critical Assessment of Genome Interpretation, establishes progress and prospects for computational genetic variant interpretation methods

BACKGROUND

The Critical Assessment of Genome Interpretation (CAGI) aims to advance the state-of-the-art for computational prediction of genetic variant impact, particularly where relevant to disease. The five complete editions of the CAGI community experiment comprised 50 challenges, in which participants made blind predictions of phenotypes from genetic data, and these were evaluated by independent assessors.

RESULTS

Performance was particularly strong for clinical pathogenic variants, including some difficult-to-diagnose cases, and extends to interpretation of cancer-related variants. Missense variant interpretation methods were able to estimate biochemical effects with increasing accuracy. Assessment of methods for regulatory variants and complex trait disease risk was less definitive and indicates performance potentially suitable for auxiliary use in the clinic.

CONCLUSIONS

Results show that while current methods are imperfect, they have major utility for research and clinical applications. Emerging methods and increasingly large, robust datasets for training and assessment promise further progress ahead.

Updated benchmarking of variant effect predictors using deep mutational scanning

The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top-performing VEPs are unsupervised methods including EVE, DeepSequence and ESM-1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking.

A Clinical-Genetic Risk Score for Predicting Cancer-Associated Venous Thromboembolism: A Development and Validation Study Involving Two Independent Prospective Cohorts

PURPOSE

Venous thromboembolism (VTE) is a leading cause of death among patients with cancer. The Khorana score was developed for assessing the risk of VTE in outpatients with cancer receiving chemotherapy, but its accuracy in identifying patients at high risk has been questioned. The aim of this study was to develop and validate a clinical-genetic score that improves the assessment of VTE risk in oncology outpatients within 6 months of diagnosis.

METHODS

The new score was developed using the data of 364 outpatients belonging to the Spanish ONCOTHROMB 12-01 population. In this cohort, clinical data associated with the risk of VTE were collected at the time of diagnosis, including the Khorana score. These patients were also genotyped for the 51 genetic variants known to be associated with VTE. Multivariate logistic regression was performed to determine the weight of each genetic and clinical variable in relation to VTE risk, allowing a clinical-genetic risk score (the ONCOTHROMB score) to be developed. The Khorana and the ONCOTHROMB scores were then compared via the area under the receiver operating characteristic curve (AUC), calibration, and the number of patients needed to treat. The new score was then validated in a study of 263 patients in the Vienna Cancer and Thrombosis Study population.

RESULTS

Nine genetic variants, tumor site, TNM stage, and a body mass index of > 25 kg/m2 were found to be associated with VTE and were used to build the ONCOTHROMB score, which better predicted the overall risk of VTE than did the Khorana score (AUC, 0.781 v 0.580; P < .001). Similar AUC results were recorded in the validation study the Vienna Cancer and Thrombosis Study cohort involving patients with the same type of tumor (AUC for the ONCOTHROMB score v the Khorana score: 0.686 v 0.577; P < .001) and with all type of tumors (AUC for the ONCOTHROMB score v the Khorana score: 0.720 v 0.561; P < .0001).

CONCLUSION

The ONCOTHROMB score for VTE risk in outpatients with cancer, which takes into account both clinical and genetic variables, better identifies patients who might benefit from primary thromboprophylaxis than does the Khorana score.

Genome interpretation using in silico predictors of variant impact

Estimating the effects of variants found in disease driver genes opens the door to personalized therapeutic opportunities. Clinical associations and laboratory experiments can only characterize a tiny fraction of all the available variants, leaving the majority as variants of unknown significance (VUS). In silico methods bridge this gap by providing instant estimates on a large scale, most often based on the numerous genetic differences between species. Despite concerns that these methods may lack reliability in individual subjects, their numerous practical applications over cohorts suggest they are already helpful and have a role to play in genome interpretation when used at the proper scale and context. In this review, we aim to gain insights into the training and validation of these variant effect predicting methods and illustrate representative types of experimental and clinical applications. Objective performance assessments using various datasets that are not yet published indicate the strengths and limitations of each method. These show that cautious use of in silico variant impact predictors is essential for addressing genome interpretation challenges.

New approaches to predict the effect of co-occurring variants on protein characteristics

Predicting the effect of a mutated gene before the onset of symptoms of genetic diseases would greatly facilitate diagnosis and potentiate early intervention. There have been myriad attempts to predict the effects of single-nucleotide variants. However, the applicability of these efforts does not scale to co-occurring variants. Furthermore, an increasing number of protein therapeutics contain co-occurring nucleotide variations, adding uncertainty during development to the safety and efficiency of these drugs. Co-occurring nucleotide variants may often have synergistic, additive, or antagonistic effects on protein attributes, further complicating the task of outcome prediction. We tested four models based on the cooperative and antagonistic effects of co-occurring variants to predict pathogenicity and effectiveness of protein therapeutics. A total of 30 attributes, including amino acid and nucleotide features, as well as existing single-variant effect prediction tools, were considered on the basis of previous studies on single-nucleotide variants. Importantly, the effects of synonymous variants, often seen in protein therapeutics, were also included in our models. We used 12 datasets of people with monogenic diseases and controls with co-occurring genetic variants to evaluate the accuracy of our models, accomplishing a degree of accuracy comparable to that of prediction tools for single-nucleotide variants. More importantly, our framework is generalizable to new, well-curated datasets of monogenic diseases and new variant scoring tools. This approach successfully assists in addressing the challenging task of predicting the effect of co-occurring variants on pathogenicity and protein effectiveness and is applicable for a wide range of protein therapeutics and genetic diseases.

The effects of anticoagulant therapy re-initiation after gastrointestinal bleeding: A systematic review and meta-analysis

WHAT IS KNOWN AND OBJECTIVE

Long-term anticoagulant/antithrombotic therapy is widely used for the management of thromboembolic conditions. Gastrointestinal bleeding is a common collateral manifestation of anticoagulant/antithrombotic therapy that complicates its administration. The continuation or discontinuation of anticoagulant/antithrombotic therapy after an episode of gastrointestinal bleeding has been a matter of debate. Despite recent positive reports from retrospective cohort studies suggesting a reduction in morbidity- and mortality-related outcomes with continued administration of anticoagulant/antithrombotic agents (even after gastrointestinal bleeding), no consensus or comparisons about the efficacies of continued or discontinued antithrombotic administration exist. Therefore, we developed this current state-of-evidence analysis evaluating the comparative effects of continuation and discontinuation of anticoagulant/antithrombotic drugs after gastrointestinal bleeding on the overall incidences of gastrointestinal bleeding, thromboembolic events and mortality events.

METHODS

We performed a systematic academic literature search according to the PRISMA guidelines across five databases: Web of Science, Embase, CENTRAL, Scopus and MEDLINE. Moreover, we conducted a random effect meta-analysis to compare the effects of continuation and discontinuation of anticoagulant/antithrombotic drugs after an event of gastrointestinal bleeding on the overall incidences of gastrointestinal bleeding, thromboembolic events and mortality events.

RESULTS

We found seven eligible studies (from 1397 candidates) with 2532 participants (mean age, 73.1 ± 4.1 years). Our meta-analysis revealed lower odds of thromboembolic events (OR, -0.21), mortality outcomes (OR, -0.39) and an increase in the incidence of gastrointestinal bleeding (OR, 2.4) in the group with continued anticoagulant/antithrombotic therapy than in the group discontinuing the therapy.

WHAT IS NEW AND CONCLUSION

We provide an updated evidence on the comparative effects between continuation and discontinuation of anticoagulant/antithrombotic drugs after gastrointestinal bleeding events based on the overall incidences of gastrointestinal bleeding, thromboembolic events and mortality events. This study reports confirm an overall lower incidence of thromboembolic events and mortality outcomes for the continuation group than for the discontinuation group.

An Artificial Neural Network Approach and a Data Augmentation Algorithm to Systematize the Diagnosis of Deep-Vein Thrombosis by Using Wells’ Criteria

The use of a back-propagation artificial neural network (ANN) to systematize the reliability of a Deep Vein Thrombosis (DVT) diagnostic by using Wells’ criteria is introduced herein. In this paper, a new ANN model is proposed to improve the Accuracy when dealing with a highly unbalanced dataset. To create the training dataset, a new data augmentation algorithm based on statistical data known as the prevalence of DVT of real cases reported in literature and from the public hospital is proposed. The above is used to generate one dataset of 10,000 synthetic cases. Each synthetic case has nine risk factors according to Wells’ criteria and also the use of two additional factors, such as gender and age, is proposed. According to interviews with medical specialists, a training scheme was established. In addition, a new algorithm is presented to improve the Accuracy and Sensitivity/Recall. According to the proposed algorithm, two thresholds of decision were found, the first one is 0.484, which is to improve Accuracy. The other one is 0.138 to improve Sensitivity/Recall. The Accuracy achieved is 90.99%, which is greater than that obtained with other related machine learning methods. The proposed ANN model was validated performing the k-fold cross validation technique using a dataset with 10,000 synthetic cases. The test was performed by using 59 real cases obtained from a regional hospital, achieving an Accuracy of 98.30%.

Reports from the fifth edition of CAGI: The Critical Assessment of Genome Interpretation

Interpretation of genomic variation plays an essential role in the analysis of cancer and monogenic disease, and increasingly also in complex trait disease, with applications ranging from basic research to clinical decisions. Many computational impact prediction methods have been developed, yet the field lacks a clear consensus on their appropriate use and interpretation. The Critical Assessment of Genome Interpretation (CAGI, /'kā-jē/) is a community experiment to objectively assess computational methods for predicting the phenotypic impacts of genomic variation. CAGI participants are provided genetic variants and make blind predictions of resulting phenotype. Independent assessors evaluate the predictions by comparing with experimental and clinical data. CAGI has completed five editions with the goals of establishing the state of art in genome interpretation and of encouraging new methodological developments. This special issue (https://onlinelibrary.wiley.com/toc/10981004/2019/40/9) comprises reports from CAGI, focusing on the fifth edition that culminated in a conference that took place 5 to 7 July 2018. CAGI5 was comprised of 14 challenges and engaged hundreds of participants from a dozen countries. This edition had a notable increase in splicing and expression regulatory variant challenges, while also continuing challenges on clinical genomics, as well as complex disease datasets and missense variants in diseases ranging from cancer to Pompe disease to schizophrenia. Full information about CAGI is at https://genomeinterpretation.org.

CAGI5: Objective performance assessments of predictions based on the Evolutionary Action equation

Many computational approaches estimate the effect of coding variants, but their predictions often disagree with each other. These contradictions confound users and raise questions regarding reliability. Performance assessments can indicate the expected accuracy for each method and highlight advantages and limitations. The Critical Assessment of Genome Interpretation (CAGI) community aims to organize objective and systematic assessments: They challenge predictors on unpublished experimental and clinical data and assign independent assessors to evaluate the submissions. We participated in CAGI experiments as predictors, using the Evolutionary Action (EA) method to estimate the fitness effect of coding mutations. EA is untrained, uses homology information, and relies on a formal equation: The fitness effect equals the functional sensitivity to residue changes multiplied by the magnitude of the substitution. In previous CAGI experiments (between 2011 and 2016), our submissions aimed to predict the protein activity of single mutants. In 2018 (CAGI5), we also submitted predictions regarding clinical associations, folding stability, and matching genomic data with phenotype. For all these diverse challenges, we used EA to predict the fitness effect of variants, adjusted to specifically address each question. Our submissions had consistently good performance, suggesting that EA predicts reliably the effects of genetic variants.