In Silico Proficiency Testing for Clinical Next-Generation Sequencing

Resource requirements to accelerate clinical applications of next-generation sequencing and radiomics: workshop commentary and review

The National Institutes of Health-US Food and Drug Administration Joint Leadership Council Next-Generation Sequencing and Radiomics Working Group was formed by the National Institutes of Health-Food and Drug Administration Joint Leadership Council to promote the development and validation of innovative next-generation sequencing tests, radiomic tools, and associated data analysis and interpretation enhanced by artificial intelligence and machine learning technologies. A 2-day workshop was held on September 29-30, 2021, to convene members of the scientific community to discuss how to overcome the "ground truth" gap that has frequently been acknowledged as 1 of the limiting factors impeding high-quality research, development, validation, and regulatory science in these fields. This report provides a summary of the resource gaps identified by the working group and attendees, highlights existing resources and the ways they can potentially be employed to accelerate growth in these fields, and presents opportunities to support next-generation sequencing and radiomic tool development and validation using technologies such as artificial intelligence and machine learning.

Molecular Descriptors and QSAR Models for Sedative Activity of Sesquiterpenes Administered to Mice via Inhalation

Essential oils are often utilized for therapeutic purposes and are composed of complex structural molecules, including sesquiterpenes, with high molecular weight and potential for stereochemistry. A detailed study on the properties of selected sesquiterpenes was conducted as part of a broader investigation on the effects of sesquiterpenes on the central nervous system. A set of 18 sesquiterpenes, rigorously selected from an original list of 114, was divided into 2 groups i.e., the training and test sets, with each containing 9 compounds. The training set was evaluated for the sedative activity in mice through inhalation, and all compounds were sedatives at any dose in the range of 4 × 10-4-4 × 10-2 mg/cage, except for curzerene. Molecular determinants of the sedative activities of sesquiterpenes were evaluated using quantitative structure-activity relationship (QSAR) and structure-activity relationship (SAR) analyses. An additional test set of six compounds obtained from the literature was utilized for validating the QSAR model. The parental carbonyl cation and an oxygen-containing groups are possible determinants of sedative activity. The QSAR study using multiple regression models could reasonably predict the sedative activity of sesquiterpenes with statistical parameters such as the correlation coefficient r2 = 0.82 > 0.6 and q2 LOO = 0.71 > 0.5 obtained using the leave-one-out cross-validation technique. Molar refractivity and the number of hydrogen bond acceptors were statistically important in predicting the activities. The present study could help predict the sedative activity of additional sesquiterpenes, thus accelerating the process of drug development.

Four-Year Laboratory Performance of the First College of American Pathologists In Silico Next-Generation Sequencing Bioinformatics Proficiency Testing Surveys

CONTEXT.—

In 2016, the College of American Pathologists (CAP) launched the first next-generation sequencing (NGS) in silico bioinformatics proficiency testing survey to evaluate the performance of clinical laboratory bioinformatics pipelines for the detection of oncology-associated variants at varying allele fractions. This survey focused on 2 commonly used oncology panels, the Illumina TruSeq Amplicon Cancer Panel and the Thermo Fisher Ion AmpliSeq Cancer Hotspot v2 Panel.

OBJECTIVE.—

To review the analytical performance of laboratories participating in the CAP NGS bioinformatics (NGSB) surveys, comprising NGSB1 for Illumina users and NGSB2 for Thermo Fisher Ion Torrent users, between 2016 and 2019.

DESIGN.—

Responses from 78 laboratories were analyzed for accuracy and associated performance characteristics.

RESULTS.—

The analytical sensitivity was 90.0% (1901 of 2112) for laboratories using the Illumina platform and 94.8% (2153 of 2272) for Thermo Fisher Ion Torrent users. Variant type and variant allele fraction were significantly associated with performance. False-negative results were seen mostly for multi-nucleotide variants and variants engineered at variant allele fractions of less than 25%. Analytical specificity for all participating laboratories was 99.8% (9303 of 9320). There was no statistically significant association between deletion-insertion length and detection rate.

CONCLUSIONS.—

These results demonstrated high analytical sensitivity and specificity, supporting the feasibility and utility of using in silico mutagenized NGS data sets as a supplemental challenge to CAP surveys for oncology-associated variants based on physical samples. This program demonstrates the opportunity and challenges that can guide future surveys inclusive of customized in silico programs.

Reference Samples to Compare Next-Generation Sequencing Test Performance for Oncology Therapeutics and Diagnostics

OBJECTIVES

Diversity of laboratory-developed tests (LDTs) using next-generation sequencing (NGS) raises concerns about their accuracy for selection of targeted therapies. A working group developed a pilot study of traceable reference samples to measure NGS LDT performance among a cohort of clinical laboratories.

METHODS

Human cell lines were engineered via CRISPR/Cas9 and prepared as formalin-fixed, paraffin-embedded cell pellets ("wet" samples) to assess the entire NGS test cycle. In silico mutagenized NGS sequence files ("dry" samples) were used to assess the bioinformatics component of the NGS test cycle. Single and multinucleotide variants (n = 36) of KRAS and NRAS were tested at 5% or 15% variant allele fraction to determine eligibility for therapy with the EGFR inhibitor panitumumab in the setting of metastatic colorectal cancer.

RESULTS

Twenty-one (21/21) laboratories tested wet samples; 19 of 21 analyzed dry samples. Of the laboratories that tested both the wet and dry samples, 7 (37%) of 19 laboratories correctly reported all variants, 3 (16%) of 19 had fewer than five errors, and 9 (47%) of 19 had five or more errors. Most errors were false negatives.

CONCLUSIONS

Genetically engineered cell lines and mutagenized sequence files are complementary reference samples for evaluating NGS test performance among clinical laboratories using LDTs. Variable accuracy in detection of genetic variants among some LDTs may identify different patient populations for targeted therapy.

Creation of an Expert Curated Variant List for Clinical Genomic Test Development and Validation: A ClinGen and GeT-RM Collaborative Project

Modern genomic sequencing tests often interrogate large numbers of genes. Identification of appropriate reference materials for development, validation studies, and quality assurance of these tests poses a significant challenge for laboratories. It is difficult to develop and maintain expert knowledge to identify all variants that must be validated to ensure analytic and clinical validity. Additionally, it is usually not possible to procure appropriate and characterized genomic DNA reference materials containing the number and scope of variants required. To address these challenges, the Centers for Disease Control and Prevention's Genetic Testing Reference Material Program (GeT-RM) has partnered with the Clinical Genome Resource (ClinGen) to develop a publicly available list of expert curated, clinically important variants. ClinGen Variant Curation Expert Panels nominated 546 variants found in 84 disease-associated genes, including common pathogenic and difficult-to-detect variants. Variant types nominated included 346 single nucleotide variants, 104 deletions, 37 copy number variants, 25 duplications, 18 deletion-insertions, 5 inversions, 4 insertions, 2 complex rearrangements, 3 difficult-to-sequence regions, and 2 fusions. This expert-curated variant list is a resource that provides a foundation for designing comprehensive validation studies and for creating in silico reference materials for clinical genomic test development and validation.

The Science and Art of Clinical Genetic Variant Classification and Its Impact on Test Accuracy

Clinical genetic variant classification science is a growing subspecialty of clinical genetics and genomics. The field's continued improvement is essential for the success of precision medicine in both germline (hereditary) and somatic (oncology) contexts. This review focuses on variant classification for DNA next-generation sequencing tests. We first summarize current limitations in variant discovery and definition, and then describe the current five- and four-tier classification systems outlined in dominant standards and guideline publications for germline and somatic tests, respectively. We then discuss measures of variant classification discordance and the field's bias for positive results, as well as considerations for panel size and population screening in the context of estimates of positive predictive value thatincorporate estimated variant classification imperfections. Finally, we share opinions on the current state of variant classification from some of the authors of the most widely used standards and guideline publications and from other domain experts.

Assembling and Validating Bioinformatic Pipelines for Next-Generation Sequencing Clinical Assays

CONTEXT.—

Clinical next-generation sequencing (NGS) is being rapidly adopted, but analysis and interpretation of large data sets prompt new challenges for a clinical laboratory setting. Clinical NGS results rely heavily on the bioinformatics pipeline for identifying genetic variation in complex samples. The choice of bioinformatics algorithms, genome assembly, and genetic annotation databases are important for determining genetic alterations associated with disease. The analysis methods are often tuned to the assay to maximize accuracy. Once a pipeline has been developed, it must be validated to determine accuracy and reproducibility for samples similar to real-world cases. In silico proficiency testing or institutional data exchange will ensure consistency among clinical laboratories.

OBJECTIVE.—

To provide molecular pathologists a step-by-step guide to bioinformatics analysis and validation design in order to navigate the regulatory and validation standards of implementing a bioinformatic pipeline as a part of a new clinical NGS assay.

DATA SOURCES.—

This guide uses published studies on genomic analysis, bioinformatics methods, and methods comparison studies to inform the reader on what resources, including open source software tools and databases, are available for genetic variant detection and interpretation.

CONCLUSIONS.—

This review covers 4 key concepts: (1) bioinformatic analysis design for detecting genetic variation, (2) the resources for assessing genetic effects, (3) analysis validation assessment experiments and data sets, including a diverse set of samples to mimic real-world challenges that assess accuracy and reproducibility, and (4) if concordance between clinical laboratories will be improved by proficiency testing designed to test bioinformatic pipelines.

Prospects and challenges of implementing DNA metabarcoding for high-throughput insect surveillance

Trap-based surveillance strategies are widely used for monitoring of invasive insect species, aiming to detect newly arrived exotic taxa as well as track the population levels of established or endemic pests. Where these surveillance traps have low specificity and capture non-target endemic species in excess of the target pests, the need for extensive specimen sorting and identification creates a major diagnostic bottleneck. While the recent development of standardized molecular diagnostics has partly alleviated this requirement, the single specimen per reaction nature of these methods does not readily scale to the sheer number of insects trapped in surveillance programmes. Consequently, target lists are often restricted to a few high-priority pests, allowing unanticipated species to avoid detection and potentially establish populations. DNA metabarcoding has recently emerged as a method for conducting simultaneous, multi-species identification of complex mixed communities and may lend itself ideally to rapid diagnostics of bulk insect trap samples. Moreover, the high-throughput nature of recent sequencing platforms could enable the multiplexing of hundreds of diverse trap samples on a single flow cell, thereby providing the means to dramatically scale up insect surveillance in terms of both the quantity of traps that can be processed concurrently and number of pest species that can be targeted. In this review of the metabarcoding literature, we explore how DNA metabarcoding could be tailored to the detection of invasive insects in a surveillance context and highlight the unique technical and regulatory challenges that must be considered when implementing high-throughput sequencing technologies into sensitive diagnostic applications.

The ICR639 CPG NGS validation series: A resource to assess analytical sensitivity of cancer predisposition gene testing

The analytical sensitivity of a next generation sequencing (NGS) test reflects the ability of the test to detect real sequence variation. The evaluation of analytical sensitivity relies on the availability of gold-standard, validated, benchmarking datasets. For NGS analysis the availability of suitable datasets has been limited. Most laboratories undertake small scale evaluations using in-house data, and/or rely on in silico generated datasets to evaluate the performance of NGS variant detection pipelines. Cancer predisposition genes (CPGs), such as BRCA1 and BRCA2, are amongst the most widely tested genes in clinical practice today. Hundreds of providers across the world are now offering CPG testing using NGS methods. Validating and comparing the analytical sensitivity of CPG tests has proved difficult, due to the absence of comprehensive, orthogonally validated, benchmarking datasets of CPG pathogenic variants. To address this we present the ICR639 CPG NGS validation series. This dataset comprises data from 639 individuals. Each individual has sequencing data generated using the TruSight Cancer Panel (TSCP), a targeted NGS assay for the analysis of CPGs, together with orthogonally generated data showing the presence of at least one CPG pathogenic variant per individual. The set consists of 645 pathogenic variants in total. There is strong representation of the most challenging types of variants to detect, with 339 indels, including 16 complex indels and 24 with length greater than five base pairs and 74 exon copy number variations (CNVs) including 23 single exon CNVs. The series includes pathogenic variants in 31 CPGs, including 502 pathogenic variants in BRCA1 or BRCA2, making this an important comprehensive validation dataset for providers of BRCA1 and BRCA2 NGS testing. We have deposited the TSCP FASTQ files of the ICR639 series in the European Genome-phenome Archive (EGA) under accession number EGAD00001004134.

Quality Control of Next-generation Sequencing-based In vitro Diagnostic Test for Onco-relevant Mutations Using Multiplex Reference Materials in Plasma

Background: Widespread clinical implementation of next-generation sequencing (NGS)-based cancer in vitro diagnostic tests (IVDs) highlighted the urgency to establish reference materials which could provide full control of the process from nucleic acid extraction to test report generation. The formalin-fixed, paraffin-embedded (FFPE) tissue and blood plasma containing circulating tumor deoxyribonucleic acid (ctDNA) were mostly used for clinically detecting onco-relevant mutations.

Methods: We respectively developed multiplex FFPE and plasma reference materials covering three clinically onco-relevant mutations within the epidermal growth factor receptor (EGFR) gene at serial allelic frequencies. All reference materials were quantified and validated via droplet digital polymerase chain reaction (ddPCR), and then were distributed to eight domestic manufacturers for the collaborative evaluation of the performance of several domestic NGS-based cancer IVDs covering four major NGS platforms (NextSeq, HiSeq, Ion Proton and BGISEQ).

Results: All expected mutations except one at extremely low allelic frequencies were detected, despite some differences in coefficient of variation (CV) which increased with the decrease of allelic frequency (CVs ranging from 18% to 106%). It was worth noting that the CV value seemed to correlate with a particular mutation as well. The repeatability of determination of different mutations was L858R>T790M>19del.

Conclusions: The results indicated our reference materials would be pivotal for quality control of NGS-based cancer IVDs and would guide the further development of reference materials covering more onco-relevant mutations.

The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies

Next-Generation Sequencing (NGS) technologies are expected to play a crucial role in the surveillance of infectious diseases, with their unprecedented capabilities for the characterisation of genetic information underlying the virulence and antimicrobial resistance (AMR) properties of microorganisms.  In the implementation of any novel technology for regulatory purposes, important considerations such as harmonisation, validation and quality assurance need to be addressed.  NGS technologies pose unique challenges in these regards, in part due to their reliance on bioinformatics for the processing and proper interpretation of the data produced.  Well-designed benchmark resources are thus needed to evaluate, validate and ensure continued quality control over the bioinformatics component of the process.  This concept was explored as part of a workshop on "Next-generation sequencing technologies and antimicrobial resistance" held October 4-5 2017.   Challenges involved in the development of such a benchmark resource, with a specific focus on identifying the molecular determinants of AMR, were identified. For each of the challenges, sets of unsolved questions that will need to be tackled for them to be properly addressed were compiled. These take into consideration the requirement for monitoring of AMR bacteria in humans, animals, food and the environment, which is aligned with the principles of a “One Health” approach.

The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies

Next-Generation Sequencing (NGS) technologies are expected to play a crucial role in the surveillance of infectious diseases, with their unprecedented capabilities for the characterisation of genetic information underlying the virulence and antimicrobial resistance (AMR) properties of microorganisms.  In the implementation of any novel technology for regulatory purposes, important considerations such as harmonisation, validation and quality assurance need to be addressed.  NGS technologies pose unique challenges in these regards, in part due to their reliance on bioinformatics for the processing and proper interpretation of the data produced.  Well-designed benchmark resources are thus needed to evaluate, validate and ensure continued quality control over the bioinformatics component of the process.  This concept was explored as part of a workshop on "Next-generation sequencing technologies and antimicrobial resistance" held October 4-5 2017.   Challenges involved in the development of such a benchmark resource, with a specific focus on identifying the molecular determinants of AMR, were identified. For each of the challenges, sets of unsolved questions that will need to be tackled for them to be properly addressed were compiled. These take into consideration the requirement for monitoring of AMR bacteria in humans, animals, food and the environment, which is aligned with the principles of a "One Health" approach.

Validation of a Customized Bioinformatics Pipeline for a Clinical Next-Generation Sequencing Test Targeting Solid Tumor-Associated Variants

Bioinformatic analysis is an integral and critical part of clinical next-generation sequencing. It is especially challenging for some pipelines to consistently identify insertions and deletions. We present the validation of an open source tumor amplicon pipeline (OTA-pipeline) for clinical next-generation sequencing targeting solid tumor-associated variants. Raw data generated from 557 TruSight Tumor 26 samples and in silico data were analyzed by the OTA-pipeline and legacy pipeline and compared. Discrepant results were confirmed by orthogonal methods. The OTA-pipeline reported 22 variants that were not detected by the previously validated pipeline, including seven synonymous or intronic single-nucleotide variants, five single-nucleotide variants at frequency <5%, one insertion, and nine deletions. Variant allele frequencies reported by the two pipelines were highly concordant, although a few significant discrepancies were present. Analysis of in silico FASTQ files demonstrated a higher sensitivity of detecting complex insertions and deletions with the OTA-pipeline. The higher sensitivity came at a cost, because false-positive calls were increased in difficult-to-sequence regions. However, these calls were all flagged by our strand bias filter, distinguishing them from true variants. Our validation process provides a model for laboratories that want to establish an in-house bioinformatics pipeline for clinical next-generation sequencing.

Whole-Genome Sequencing of Bacterial Pathogens: the Future of Nosocomial Outbreak Analysis

Outbreaks of multidrug-resistant bacteria present a frequent threat to vulnerable patient populations in hospitals around the world. Intensive care unit (ICU) patients are particularly susceptible to nosocomial infections due to indwelling devices such as intravascular catheters, drains, and intratracheal tubes for mechanical ventilation. The increased vulnerability of infected ICU patients demonstrates the importance of effective outbreak management protocols to be in place. Understanding the transmission of pathogens via genotyping methods is an important tool for outbreak management. Recently, whole-genome sequencing (WGS) of pathogens has become more accessible and affordable as a tool for genotyping. Analysis of the entire pathogen genome via WGS could provide unprecedented resolution in discriminating even highly related lineages of bacteria and revolutionize outbreak analysis in hospitals. Nevertheless, clinicians have long been hesitant to implement WGS in outbreak analyses due to the expensive and cumbersome nature of early sequencing platforms. Recent improvements in sequencing technologies and analysis tools have rapidly increased the output and analysis speed as well as reduced the overall costs of WGS. In this review, we assess the feasibility of WGS technologies and bioinformatics analysis tools for nosocomial outbreak analyses and provide a comparison to conventional outbreak analysis workflows. Moreover, we review advantages and limitations of sequencing technologies and analysis tools and present a real-world example of the implementation of WGS for antimicrobial resistance analysis. We aimed to provide health care professionals with a guide to WGS outbreak analysis that highlights its benefits for hospitals and assists in the transition from conventional to WGS-based outbreak analysis.

Review of Clinical Next-Generation Sequencing

CONTEXT

- Next-generation sequencing (NGS) is a technology being used by many laboratories to test for inherited disorders and tumor mutations. This technology is new for many practicing pathologists, who may not be familiar with the uses, methodology, and limitations of NGS.

OBJECTIVE

- To familiarize pathologists with several aspects of NGS, including current and expanding uses; methodology including wet bench aspects, bioinformatics, and interpretation; validation and proficiency; limitations; and issues related to the integration of NGS data into patient care.

DATA SOURCES

- The review is based on peer-reviewed literature and personal experience using NGS in a clinical setting at a major academic center.

CONCLUSIONS

- The clinical applications of NGS will increase as the technology, bioinformatics, and resources evolve to address the limitations and improve quality of results. The challenge for clinical laboratories is to ensure testing is clinically relevant, cost-effective, and can be integrated into clinical care.

Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists

Next-generation sequencing (NGS) methods for cancer testing have been rapidly adopted by clinical laboratories. To establish analytical validation best practice guidelines for NGS gene panel testing of somatic variants, a working group was convened by the Association of Molecular Pathology with liaison representation from the College of American Pathologists. These joint consensus recommendations address NGS test development, optimization, and validation, including recommendations on panel content selection and rationale for optimization and familiarization phase conducted before test validation; utilization of reference cell lines and reference materials for evaluation of assay performance; determining of positive percentage agreement and positive predictive value for each variant type; and requirements for minimal depth of coverage and minimum number of samples that should be used to establish test performance characteristics. The recommendations emphasize the role of laboratory director in using an error-based approach that identifies potential sources of errors that may occur throughout the analytical process and addressing these potential errors through test design, method validation, or quality controls so that no harm comes to the patient. The recommendations contained herein are intended to assist clinical laboratories with the validation and ongoing monitoring of NGS testing for detection of somatic variants and to ensure high quality of sequencing results.

The Case for Laboratory Developed Procedures: Quality and Positive Impact on Patient Care

An explosion of knowledge and technology is revolutionizing medicine and patient care. Novel testing must be brought to the clinic with safety and accuracy, but also in a timely and cost-effective manner, so that patients can benefit and laboratories can offer testing consistent with current guidelines. Under the oversight provided by the Clinical Laboratory Improvement Amendments, laboratories have been able to develop and optimize laboratory procedures for use in-house. Quality improvement programs, interlaboratory comparisons, and the ability of laboratories to adjust assays as needed to improve results, utilize new sample types, or incorporate new mutations, information, or technologies are positive aspects of Clinical Laboratory Improvement Amendments oversight of laboratory-developed procedures. Laboratories have a long history of successful service to patients operating under Clinical Laboratory Improvement Amendments. A series of detailed clinical examples illustrating the quality and positive impact of laboratory-developed procedures on patient care is provided. These examples also demonstrate how Clinical Laboratory Improvement Amendments oversight ensures accurate, reliable, and reproducible testing in clinical laboratories.