Fast, parallel, and cache-friendly suffix array construction

PURPOSE

String indexes such as the suffix array (SA) and the closely related longest common prefix (LCP) array are fundamental objects in bioinformatics and have a wide variety of applications. Despite their importance in practice, few scalable parallel algorithms for constructing these are known, and the existing algorithms can be highly non-trivial to implement and parallelize.

METHODS

In this paper we present CAPS-SA, a simple and scalable parallel algorithm for constructing these string indexes inspired by samplesort and utilizing an LCP-informed mergesort. Due to its design, CAPS-SA has excellent memory-locality and thus incurs fewer cache misses and achieves strong performance on modern multicore systems with deep cache hierarchies.

RESULTS

We show that despite its simple design, CAPS-SA outperforms existing state-of-the-art parallel SA and LCP-array construction algorithms on modern hardware. Finally, motivated by applications in modern aligners where the query strings have bounded lengths, we introduce the notion of a bounded-context SA and show that CAPS-SA can easily be extended to exploit this structure to obtain further speedups. We make our code publicly available at https://github.com/jamshed/CaPS-SA .

Suffix sorting via matching statistics

We introduce a new algorithm for constructing the generalized suffix array of a collection of highly similar strings. As a first step, we construct a compressed representation of the matching statistics of the collection with respect to a reference string. We then use this data structure to distribute suffixes into a partial order, and subsequently to speed up suffix comparisons to complete the generalized suffix array. Our experimental evidence with a prototype implementation (a tool we call sacamats) shows that on string collections with highly similar strings we can construct the suffix array in time competitive with or faster than the fastest available methods. Along the way, we describe a heuristic for fast computation of the matching statistics of two strings, which may be of independent interest.

Markov Chain Models for Cardiac Rhythm Dynamics in Patients Undergoing Catheter Ablation of Atrial Fibrillation

We have developed a novel Markov Chain modeling system that considers vectors of patients with atrial fibrillation (AF) by their AF status over a period of time. Our model examines the impact of catheter ablation of AF upon the dynamics of a patient's AF status and their potential return to sinus rhythm. We prove several theorems to determine the probabilities of patients achieving sinus rhythm or progressing to permanent AF. Additionally, we observed aggregation of patients within the paroxysmal AF state in simulation. The aggregating property of Markov chains illustrated the potential benefits of catheter ablation on healthcare resource allocation.

Scalable, ultra-fast, and low-memory construction of compacted de Bruijn graphs with Cuttlefish 2

The de Bruijn graph is a key data structure in modern computational genomics, and construction of its compacted variant resides upstream of many genomic analyses. As the quantity of genomic data grows rapidly, this often forms a computational bottleneck. We present Cuttlefish 2, significantly advancing the state-of-the-art for this problem. On a commodity server, it reduces the graph construction time for 661K bacterial genomes, of size 2.58Tbp, from 4.5 days to 17-23 h; and it constructs the graph for 1.52Tbp white spruce reads in approximately 10 h, while the closest competitor requires 54-58 h, using considerably more memory.

Indexing labeled sequences

BACKGROUND

Labels are a way to add some information on a text, such as functional annotations such as genes on a DNA sequences. V(D)J recombinations are DNA recombinations involving two or three short genes in lymphocytes. Sequencing this short region (500 bp or less) produces labeled sequences and brings insight in the lymphocyte repertoire for onco-hematology or immunology studies.

METHODS

We present two indexes for a text with non-overlapping labels. They store the text in a Burrows-Wheeler transform (BWT) and a compressed label sequence in a Wavelet Tree. The label sequence is taken in the order of the text (TL-index) or in the order of the BWT (TL_BW-index). Both indexes need a space related to the entropy of the labeled text.

RESULTS

These indexes allow efficient text-label queries to count and find labeled patterns. The TL_BW-index has an overhead on simple label queries but is very efficient on combined pattern-label queries. We implemented the indexes in C++ and compared them against a baseline solution on pseudo-random as well as on V(D)J labeled texts.

DISCUSSION

New indexes such as the ones we proposed improve the way we index and query labeled texts as, for instance, lymphocyte repertoire for hematological and immunological studies.

Finding Pairwise Intersections Inside a Query Range

We study the following problem: preprocess a set O of objects into a data structure that allows us to efficiently report all pairs of objects from O that intersect inside an axis-aligned query range Q . We present data structures of size O ( n · polylog n ) and with query time O ( ( k + 1 ) · polylog n ) time, where k is the number of reported pairs, for two classes of objects in R 2 : axis-aligned rectangles and objects with small union complexity. For the 3-dimensional case where the objects and the query range are axis-aligned boxes in  R 3 , we present a data structure of size O ( n n · polylog n ) and query time O ( ( n + k ) · polylog n ) . When the objects and query are fat, we obtain O ( ( k + 1 ) · polylog n ) query time using O ( n · polylog n ) storage.

The SeqAn C++ template library for efficient sequence analysis: A resource for programmers

BACKGROUND

The use of novel algorithmic techniques is pivotal to many important problems in life science. For example the sequencing of the human genome (Venter et al., 2001) would not have been possible without advanced assembly algorithms and the development of practical BWT based read mappers have been instrumental for NGS analysis. However, owing to the high speed of technological progress and the urgent need for bioinformatics tools, there was a widening gap between state-of-the-art algorithmic techniques and the actual algorithmic components of tools that are in widespread use. We previously addressed this by introducing the SeqAn library of efficient data types and algorithms in 2008 (Döring et al., 2008).

RESULTS

The SeqAn library has matured considerably since its first publication 9 years ago. In this article we review its status as an established resource for programmers in the field of sequence analysis and its contributions to many analysis tools.

CONCLUSIONS

We anticipate that SeqAn will continue to be a valuable resource, especially since it started to actively support various hardware acceleration techniques in a systematic manner.

Reactome pathway analysis: a high-performance in-memory approach

Background

Reactome aims to provide bioinformatics tools for visualisation, interpretation and analysis of pathway knowledge to support basic research, genome analysis, modelling, systems biology and education. Pathway analysis methods have a broad range of applications in physiological and biomedical research; one of the main problems, from the analysis methods performance point of view, is the constantly increasing size of the data samples.

Results

Here, we present a new high-performance in-memory implementation of the well-established over-representation analysis method. To achieve the target, the over-representation analysis method is divided in four different steps and, for each of them, specific data structures are used to improve performance and minimise the memory footprint. The first step, finding out whether an identifier in the user’s sample corresponds to an entity in Reactome, is addressed using a radix tree as a lookup table. The second step, modelling the proteins, chemicals, their orthologous in other species and their composition in complexes and sets, is addressed with a graph. The third and fourth steps, that aggregate the results and calculate the statistics, are solved with a double-linked tree.

Conclusion

Through the use of highly optimised, in-memory data structures and algorithms, Reactome has achieved a stable, high performance pathway analysis service, enabling the analysis of genome-wide datasets within seconds, allowing interactive exploration and analysis of high throughput data. The proposed pathway analysis approach is available in the Reactome production web site either via the AnalysisService for programmatic access or the user submission interface integrated into the PathwayBrowser. Reactome is an open data and open source project and all of its source code, including the one described here, is available in the AnalysisTools repository in the Reactome GitHub (https://github.com/reactome/).

Comparing efficient data structures to represent geometric models for three-dimensional virtual medical training

Data structures have been explored for several domains of computer applications in order to ensure efficiency in the data store and retrieval. However, data structures can present different behavior depending on applications that they are being used. Three-dimensional interactive environments offered by techniques of Virtual Reality require operations of loading and manipulating objects in real time, where realism and response time are two important requirements. Efficient representation of geometrical models plays an important part so that the simulation may become real. In this paper, we present the implementation and the comparison of two topologically efficient data structures - Compact Half-Edge and Mate-Face - for the representation of objects for three-dimensional interactive environments. The structures have been tested at different conditions of processors and RAM memories. The results show that both these structures can be used in an efficient manner. Mate-Face structure has shown itself to be more efficient for the manipulation of neighborhood relationships and the Compact Half-Edge was more efficient for loading of the geometric models. We also evaluated the data structures embedded in applications of biopsy simulation using virtual reality, considering a deformation simulation method applied in virtual human organs. The results showed that their use allows the building of applications considering objects with high resolutions (number of vertices), without significant impact in the time spent in the simulation. Therefore, their use contributes for the construction of more realistic simulators.

Compile- and run-time approaches for the selection of efficient data structures for dynamic graph analysis

Graphs are used to model a wide range of systems from different disciplines including social network analysis, biology, and big data processing. When analyzing these constantly changing dynamic graphs at a high frequency, performance is the main concern. Depending on the graph size and structure, update frequency, and read accesses of the analysis, the use of different data structures can yield great performance variations. Even for expert programmers, it is not always obvious, which data structure is the best choice for a given scenario. In previous work, we presented an approach for handling the selection of the most efficient data structures automatically using a compile-time approach well-suited for constant workloads. We extend this work with a measurement study of seven data structures and use the results to fit actual cost estimation functions. In addition, we evaluate our approach for the computations of seven different graph metrics. In analyses of real-world dynamic graphs with a constant workload, our approach achieves a speedup of up to 5.4× compared to basic data structure configurations. Such a compile-time based approach cannot yield optimal results when the behavior of the system changes later and the workload becomes non-constant. To close this gap we present a run-time approach which provides live profiling and facilitates automatic exchanges of data structures during execution. We analyze the performance of this approach using an artificial, non-constant workload where our approach achieves speedups of up to 7.3× compared to basic configurations.