QuaMeter: multivendor performance metrics for LC-MS/MS proteomics instrumentation

A Framework for Quality Control in Quantitative Proteomics

A thorough evaluation of the quality, reproducibility, and variability of bottom-up proteomics data is necessary at every stage of a workflow, from planning to analysis. We share vignettes applying adaptable quality control (QC) measures to assess sample preparation, system function, and quantitative analysis. System suitability samples are repeatedly measured longitudinally with targeted methods, and we share examples where they are used on three instrument platforms to identify severe system failures and track function over months to years. Internal QCs incorporated at the protein and peptide levels allow our team to assess sample preparation issues and to differentiate system failures from sample-specific issues. External QC samples prepared alongside our experimental samples are used to verify the consistency and quantitative potential of our results during batch correction and normalization before assessing biological phenotypes. We combine these controls with rapid analysis (Skyline), longitudinal QC metrics (AutoQC), and server-based data deposition (PanoramaWeb). We propose that this integrated approach to QC is a useful starting point for groups to facilitate rapid quality control assessment to ensure that valuable instrument time is used to collect the best quality data possible. Data are available on Panorama Public and ProteomeXchange under the identifier PXD051318.

A framework for quality control in quantitative proteomics

A thorough evaluation of the quality, reproducibility, and variability of bottom-up proteomics data is necessary at every stage of a workflow from planning to analysis. We share vignettes applying adaptable quality control (QC) measures to assess sample preparation, system function, and quantitative analysis. System suitability samples are repeatedly measured longitudinally with targeted methods, and we share examples where they are used on three instrument platforms to identify severe system failures and track function over months to years. Internal QCs incorporated at protein and peptide-level allow our team to assess sample preparation issues and to differentiate system failures from sample-specific issues. External QC samples prepared alongside our experimental samples are used to verify the consistency and quantitative potential of our results during batch correction and normalization before assessing biological phenotypes. We combine these controls with rapid analysis (Skyline), longitudinal QC metrics (AutoQC), and server-based data deposition (PanoramaWeb). We propose that this integrated approach to QC is a useful starting point for groups to facilitate rapid quality control assessment to ensure that valuable instrument time is used to collect the best quality data possible. Data are available on Panorama Public and on ProteomeXchange under the identifier PXD051318.

Communicating Mass Spectrometry Quality Information in mzQC with Python, R, and Java

Mass spectrometry is a powerful technique for analyzing molecules in complex biological samples. However, inter- and intralaboratory variability and bias can affect the data due to various factors, including sample handling and preparation, instrument calibration and performance, and data acquisition and processing. To address this issue, the Quality Control (QC) working group of the Human Proteome Organization's Proteomics Standards Initiative has established the standard mzQC file format for reporting and exchanging information relating to data quality. mzQC is based on the JavaScript Object Notation (JSON) format and provides a lightweight yet versatile file format that can be easily implemented in software. Here, we present open-source software libraries to process mzQC data in three programming languages: Python, using pymzqc; R, using rmzqc; and Java, using jmzqc. The libraries follow a common data model and provide shared functionalities, including the (de)serialization and validation of mzQC files. We demonstrate use of the software libraries in a workflow for extracting, analyzing, and visualizing QC metrics from different sources. Additionally, we show how these libraries can be integrated with each other, with existing software tools, and in automated workflows for the QC of mass spectrometry data. All software libraries are available as open source under the MS-Quality-Hub organization on GitHub (https://github.com/MS-Quality-Hub).

MS-PyCloud: A Cloud Computing-Based Pipeline for Proteomic and Glycoproteomic Data Analyses

Rapid development and wide adoption of mass spectrometry-based glycoproteomic technologies have empowered scientists to study proteins and protein glycosylation in complex samples on a large scale. This progress has also created unprecedented challenges for individual laboratories to store, manage, and analyze proteomic and glycoproteomic data, both in the cost for proprietary software and high-performance computing and in the long processing time that discourages on-the-fly changes of data processing settings required in explorative and discovery analysis. We developed an open-source, cloud computing-based pipeline, MS-PyCloud, with graphical user interface (GUI), for proteomic and glycoproteomic data analysis. The major components of this pipeline include data file integrity validation, MS/MS database search for spectral assignments to peptide sequences, false discovery rate estimation, protein inference, quantitation of global protein levels, and specific glycan-modified glycopeptides as well as other modification-specific peptides such as phosphorylation, acetylation, and ubiquitination. To ensure the transparency and reproducibility of data analysis, MS-PyCloud includes open-source software tools with comprehensive testing and versioning for spectrum assignments. Leveraging public cloud computing infrastructure via Amazon Web Services (AWS), MS-PyCloud scales seamlessly based on analysis demand to achieve fast and efficient performance. Application of the pipeline to the analysis of large-scale LC-MS/MS data sets demonstrated the effectiveness and high performance of MS-PyCloud. The software can be downloaded at https://github.com/huizhanglab-jhu/ms-pycloud.

SpectiCal: m/z Calibration of MS2 Peptide Spectra Using Known Low Mass Ions

Most tandem mass spectrometry fragmentation spectra have small calibration errors that can lead to suboptimal interpretation and annotation. We developed SpectiCal, a software tool that can read mzML files from data-dependent acquisition proteomics experiments in parallel, compute m/z calibrations for each file prior to identification analysis based on known low-mass ions, and produce information about frequently observed peaks and their explanations. Using calibration coefficients, the data can be corrected to generate new calibrated mzML files. SpectiCal was tested using five public data sets, creating a table of commonly observed low-mass ions and their identifications. Information about the calibration and individual peaks is written in PDF and TSV files. This includes information for each peak, such as the number of runs in which it appears, the percentage of spectra in which it appears, and a plot of the aggregated region surrounding each peak. SpectiCal can be used to compute MS run calibrations, examine MS runs for artifacts that might hinder downstream analysis, and generate tables of detected low-mass ions for further analysis. SpectiCal is freely available at https://github.com/PlantProteomes/SpectiCal.

SEAOP: a statistical ensemble approach for outlier detection in quantitative proteomics data

Quality control in quantitative proteomics is a persistent challenge, particularly in identifying and managing outliers. Unsupervised learning models, which rely on data structure rather than predefined labels, offer potential solutions. However, without clear labels, their effectiveness might be compromised. Single models are susceptible to the randomness of parameters and initialization, which can result in a high rate of false positives. Ensemble models, on the other hand, have shown capabilities in effectively mitigating the impacts of such randomness and assisting in accurately detecting true outliers. Therefore, we introduced SEAOP, a Python toolbox that utilizes an ensemble mechanism by integrating multi-round data management and a statistics-based decision pipeline with multiple models. Specifically, SEAOP uses multi-round resampling to create diverse sub-data spaces and employs outlier detection methods to identify candidate outliers in each space. Candidates are then aggregated as confirmed outliers via a chi-square test, adhering to a 95% confidence level, to ensure the precision of the unsupervised approaches. Additionally, SEAOP introduces a visualization strategy, specifically designed to intuitively and effectively display the distribution of both outlier and non-outlier samples. Optimal hyperparameter models of SEAOP for outlier detection were identified by using a gradient-simulated standard dataset and Mann-Kendall trend test. The performance of the SEAOP toolbox was evaluated using three experimental datasets, confirming its reliability and accuracy in handling quantitative proteomics.

MsQuality: an interoperable open-source package for the calculation of standardized quality metrics of mass spectrometry data

MOTIVATION

Multiple factors can impact accuracy and reproducibility of mass spectrometry data. There is a need to integrate quality assessment and control into data analytic workflows.

RESULTS

The MsQuality package calculates 43 low-level quality metrics based on the controlled mzQC vocabulary defined by the HUPO-PSI on a single mass spectrometry-based measurement of a sample. It helps to identify low-quality measurements and track data quality. Its use of community-standard quality metrics facilitates comparability of quality assessment and control (QA/QC) criteria across datasets.

AVAILABILITY AND IMPLEMENTATION

The R package MsQuality is available through Bioconductor at https://bioconductor.org/packages/MsQuality.

Evaluating cPILOT Data toward Quality Control Implementation

Multiplexing enables the monitoring of hundreds to thousands of proteins in quantitative proteomics analyses and increases sample throughput. In most mass-spectrometry-based proteomics workflows, multiplexing is achieved by labeling biological samples with heavy isotopes via precursor isotopic labeling or isobaric tagging. Enhanced multiplexing strategies, such as combined precursor isotopic labeling and isobaric tagging (cPILOT), combine multiple technologies to afford an even higher sample throughput. Critical to enhanced multiplexing analyses is ensuring that analytical performance is optimal and that missingness of sample channels is minimized. Automation of sample preparation steps and use of quality control (QC) metrics can be incorporated into multiplexing analyses and reduce the likelihood of missing information, thus maximizing the amount of usable quantitative data. Here, we implemented QC metrics previously developed in our laboratory to evaluate a 36-plex cPILOT experiment that encompassed 144 mouse samples of various tissue types, time points, genotypes, and biological replicates. The evaluation focuses on the use of a sample pool generated from all samples in the experiment to monitor the daily instrument performance and to provide a means for data normalization across sample batches. Our results show that tracking QC metrics enabled the quantification of ∼7000 proteins in each sample batch, of which ∼70% had minimal missing values across up to 36 sample channels. Implementation of QC metrics for future cPILOT studies as well as other enhanced multiplexing strategies will help yield high-quality data sets.

Quality Control—A Stepchild in Quantitative Proteomics: A Case Study for the Human CSF Proteome

Proteomic studies using mass spectrometry (MS)-based quantification are a main approach to the discovery of new biomarkers. However, a number of analytical conditions in front and during MS data acquisition can affect the accuracy of the obtained outcome. Therefore, comprehensive quality assessment of the acquired data plays a central role in quantitative proteomics, though, due to the immense complexity of MS data, it is often neglected. Here, we address practically the quality assessment of quantitative MS data, describing key steps for the evaluation, including the levels of raw data, identification and quantification. With this, four independent datasets from cerebrospinal fluid, an important biofluid for neurodegenerative disease biomarker studies, were assessed, demonstrating that sample processing-based differences are already reflected at all three levels but with varying impacts on the quality of the quantitative data. Specifically, we provide guidance to critically interpret the quality of MS data for quantitative proteomics. Moreover, we provide the free and open source quality control tool MaCProQC, enabling systematic, rapid and uncomplicated data comparison of raw data, identification and feature detection levels through defined quality metrics and a step-by-step quality control workflow.

RawBeans: A Simple, Vendor-Independent, Raw-Data Quality-Control Tool

Every laboratory performing mass-spectrometry-based proteomics strives to generate high-quality data. Among the many factors that impact the outcome of any experiment in proteomics is the LC–MS system performance, which should be monitored within each specific experiment and also long term. This process is termed quality control (QC). We present an easy-to-use tool that rapidly produces a visual, HTML-based report that includes the key parameters needed to monitor the LC–MS system performance, with a focus on monitoring the performance within an experiment. The tool, named RawBeans, generates a report for individual files or for a set of samples from a whole experiment. We anticipate that it will help proteomics users and experts evaluate raw data quality independent of data processing. The tool is available at https://bitbucket.org/incpm/prot-qc/downloads. The mass-spectrometry proteomics data have been deposited to the ProteomeXchange Consortium via the PRIDE partner repository with the data set identifier PXD022816.

Calculating Glycoprotein Similarities From Mass Spectrometric Data

Complex protein glycosylation occurs through biosynthetic steps in the secretory pathway that create macro- and microheterogeneity of structure and function. Required for all life forms, glycosylation diversifies and adapts protein interactions with binding partners that underpin interactions at cell surfaces and pericellular and extracellular environments. Because these biological effects arise from heterogeneity of structure and function, it is necessary to measure their changes as part of the quest to understand nature. Quite often, however, the assumption behind proteomics that posttranslational modifications are discrete additions that can be modeled using the genome as a template does not apply to protein glycosylation. Rather, it is necessary to quantify the glycosylation distribution at each glycosite and to aggregate this information into a population of mature glycoproteins that exist in a given biological system. To date, mass spectrometric methods for assigning singly glycosylated peptides are well-established. But it is necessary to quantify glycosylation heterogeneity accurately in order to gauge the alterations that occur during biological processes. The task is to quantify the glycosylated peptide forms as accurately as possible and then apply appropriate bioinformatics algorithms to the calculation of micro- and macro-similarities. In this review, we summarize current approaches for protein quantification as they apply to this glycoprotein similarity problem.

Mass Spectrometry Advances and Perspectives for the Characterization of Emerging Adoptive Cell Therapies

Adoptive cell therapy is an emerging anti-cancer modality, whereby the patient's own immune cells are engineered to express T-cell receptor (TCR) or chimeric antigen receptor (CAR). CAR-T cell therapies have advanced the furthest, with recent approvals of two treatments by the Food and Drug Administration of Kymriah (trisagenlecleucel) and Yescarta (axicabtagene ciloleucel). Recent developments in proteomic analysis by mass spectrometry (MS) make this technology uniquely suited to enable the comprehensive identification and quantification of the relevant biochemical architecture of CAR-T cell therapies and fulfill current unmet needs for CAR-T product knowledge. These advances include improved sample preparation methods, enhanced separation technologies, and extension of MS-based proteomic to single cells. Innovative technologies such as proteomic analysis of raw material quality attributes (MQA) and final product quality attributes (PQA) may provide insights that could ultimately fuel development strategies and lead to broad implementation.

Review of Issues and Solutions to Data Analysis Reproducibility and Data Quality in Clinical Proteomics

In any analytical discipline, data analysis reproducibility is closely interlinked with data quality. In this book chapter focused on mass spectrometry-based proteomics approaches, we introduce how both data analysis reproducibility and data quality can influence each other and how data quality and data analysis designs can be used to increase robustness and improve reproducibility. We first introduce methods and concepts to design and maintain robust data analysis pipelines such that reproducibility can be increased in parallel. The technical aspects related to data analysis reproducibility are challenging, and current ways to increase the overall robustness are multifaceted. Software containerization and cloud infrastructures play an important part.We will also show how quality control (QC) and quality assessment (QA) approaches can be used to spot analytical issues, reduce the experimental variability, and increase confidence in the analytical results of (clinical) proteomics studies, since experimental variability plays a substantial role in analysis reproducibility. Therefore, we give an overview on existing solutions for QC/QA, including different quality metrics, and methods for longitudinal monitoring. The efficient use of both types of approaches undoubtedly provides a way to improve the experimental reliability, reproducibility, and level of consistency in proteomics analytical measurements.

Glycosylation of Tetrabromobisphenol A in Pumpkin

Tetrabromobisphenol A (TBBPA) is the most widely used brominated flame retardant (BFR), and it bioaccumulates throughout the food chains. Its fate in the first trophic level, plants, is of special interest. In this study, a four-day hydroponic exposure of TBBPA at a concentration of 1 μmol L^-1 to pumpkin seedlings was conducted. A nontarget screening method for hydrophilic bromine-containing metabolites was modified, based on both typical isotope patterns of bromine and mass defect, and used to process mass spectra data. A total of 20 glycosylation and malonyl glycosylation metabolites were found for TBBPA in the pumpkin plants. Representative glycosyl TBBPA reference standards were synthesized to evaluate the contribution of this glycosylation process. Approximately 86% of parent TBBPA was metabolized to form those 20 glycosyl TBBPAs, showing that glycosylation was the most dominant metabolism pathway for TBBPA in pumpkin at the tested exposure concentration.

QCMAP: An Interactive Web-Tool for Performance Diagnosis and Prediction of LC-MS Systems

The increasing role played by liquid chromatography-mass spectrometry (LC-MS)-based proteomics in biological discovery has led to a growing need for quality control (QC) on the LC-MS systems. While numerous quality control tools have been developed to track the performance of LC-MS systems based on a pre-defined set of performance factors (e.g., mass error, retention time), the precise influence and contribution of the performance factors and their generalization property to different biological samples are not as well characterized. Here, a web-based application (QCMAP) is developed for interactive diagnosis and prediction of the performance of LC-MS systems across different biological sample types. Leveraging on a standardized HeLa cell sample run as QC within a multi-user facility, predictive models are trained on a panel of commonly used performance factors to pinpoint the precise conditions to a (un)satisfactory performance in three LC-MS systems. It is demonstrated that the learned model can be applied to predict LC-MS system performance for brain samples generated from an independent study. By compiling these predictive models into our web-application, QCMAP allows users to benchmark the performance of their LC-MS systems using their own samples and identify key factors for instrument optimization. QCMAP is freely available from: http://shiny.maths.usyd.edu.au/QCMAP/.

Quality Control Analysis in Real-time (QC-ART): A Tool for Real-time Quality Control Assessment of Mass Spectrometry-based Proteomics Data

Liquid chromatography-mass spectrometry (LC-MS)-based proteomics studies of large sample cohorts can easily require from months to years to complete. Acquiring consistent, high-quality data in such large-scale studies is challenging because of normal variations in instrumentation performance over time, as well as artifacts introduced by the samples themselves, such as those because of collection, storage and processing. Existing quality control methods for proteomics data primarily focus on post-hoc analysis to remove low-quality data that would degrade downstream statistics; they are not designed to evaluate the data in near real-time, which would allow for interventions as soon as deviations in data quality are detected. In addition to flagging analyses that demonstrate outlier behavior, evaluating how the data structure changes over time can aide in understanding typical instrument performance or identify issues such as a degradation in data quality because of the need for instrument cleaning and/or re-calibration. To address this gap for proteomics, we developed Quality Control Analysis in Real-Time (QC-ART), a tool for evaluating data as they are acquired to dynamically flag potential issues with instrument performance or sample quality. QC-ART has similar accuracy as standard post-hoc analysis methods with the additional benefit of real-time analysis. We demonstrate the utility and performance of QC-ART in identifying deviations in data quality because of both instrument and sample issues in near real-time for LC-MS-based plasma proteomics analyses of a sample subset of The Environmental Determinants of Diabetes in the Young cohort. We also present a case where QC-ART facilitated the identification of oxidative modifications, which are often underappreciated in proteomic experiments.

QCloud: A cloud-based quality control system for mass spectrometry-based proteomics laboratories

The increasing number of biomedical and translational applications in mass spectrometry-based proteomics poses new analytical challenges and raises the need for automated quality control systems. Despite previous efforts to set standard file formats, data processing workflows and key evaluation parameters for quality control, automated quality control systems are not yet widespread among proteomics laboratories, which limits the acquisition of high-quality results, inter-laboratory comparisons and the assessment of variability of instrumental platforms. Here we present QCloud, a cloud-based system to support proteomics laboratories in daily quality assessment using a user-friendly interface, easy setup, automated data processing and archiving, and unbiased instrument evaluation. QCloud supports the most common targeted and untargeted proteomics workflows, it accepts data formats from different vendors and it enables the annotation of acquired data and reporting incidences. A complete version of the QCloud system has successfully been developed and it is now open to the proteomics community (http://qcloud.crg.eu). QCloud system is an open source project, publicly available under a Creative Commons License Attribution-ShareAlike 4.0.

MSstatsQC: Longitudinal System Suitability Monitoring and Quality Control for Targeted Proteomic Experiments

Selected Reaction Monitoring (SRM) is a powerful tool for targeted detection and quantification of peptides in complex matrices. An important objective of SRM is to obtain peptide quantifications that are (1) suitable for the investigation, and (2) reproducible across laboratories and runs. The first objective is achieved by system suitability tests (SST), which verify that mass spectrometric instrumentation performs as specified. The second objective is achieved by quality control (QC), which provides in-process quality assurance of the sample profile. A common aspect of SST and QC is the longitudinal nature of the data. Although SST and QC have received a lot of attention in the proteomic community, the currently used statistical methods are limited. This manuscript improves upon the statistical methodology for SST and QC that is currently used in proteomics. It adapts the modern methods of longitudinal statistical process control, such as simultaneous and time weighted control charts and change point analysis, to SST and QC of SRM experiments, discusses their advantages, and provides practical guidelines. Evaluations on simulated data sets, and on data sets from the Clinical Proteomics Technology Assessment for Cancer (CPTAC) consortium, demonstrated that these methods substantially improve our ability of real time monitoring, early detection and prevention of chromatographic and instrumental problems. We implemented the methods in an open-source R-based software package MSstatsQC and its web-based graphical user interface. They are available for use stand-alone, or for integration with automated pipelines. Although the examples focus on targeted proteomics, the statistical methods in this manuscript apply more generally to quantitative proteomics.

The Human Proteome Organization-Proteomics Standards Initiative Quality Control Working Group: Making Quality Control More Accessible for Biological Mass Spectrometry

To have confidence in results acquired during biological mass spectrometry experiments, a systematic approach to quality control is of vital importance. Nonetheless, until now, only scattered initiatives have been undertaken to this end, and these individual efforts have often not been complementary. To address this issue, the Human Proteome Organization-Proteomics Standards Initiative has established a new working group on quality control at its meeting in the spring of 2016. The goal of this working group is to provide a unifying framework for quality control data. The initial focus will be on providing a community-driven standardized file format for quality control. For this purpose, the previously proposed qcML format will be adapted to support a variety of use cases for both proteomics and metabolomics applications, and it will be established as an official PSI format. An important consideration is to avoid enforcing restrictive requirements on quality control but instead provide the basic technical necessities required to support extensive quality control for any type of mass spectrometry-based workflow. We want to emphasize that this is an open community effort, and we seek participation from all scientists with an interest in this field.

Designing Successful Proteomics Experiments

Because proteomics experiments are so complex they can readily fail, and do so without clear cause. Using standard experimental design techniques and incorporating quality control can greatly increase the chances of success. This chapter introduces the relevant concepts and provides examples specific to proteomic workflows. Applying these notions to design successful proteomics experiments is straightforward. It can help identify failure causes and greatly increase the likelihood of inter-laboratory reproducibility.

Statistical Assessment of QC Metrics on Raw LC-MS/MS Data

Data quality assessment is important for reproducibility of proteomics experiments and reusability of proteomics data. We describe a set of statistical tools to routinely visualize and examine the quality control (QC) metrics obtained for raw LC-MS/MS data on different instrument types and mass spectrometers. The QC metrics used here are the identification free QuaMeter metrics. Statistical assessments introduced include (a) principal component analysis, (b) dissimilarity measures, (c) T ²-chart for quality control, and (d) change point analysis. We demonstrate the workflow by a step-by-step assessment of a subset of Study 5 for the Clinical Proteomics Technology Assessment for Cancer (CPTAC) using our R functions.

An Automated Pipeline to Monitor System Performance in Liquid Chromatography-Tandem Mass Spectrometry Proteomic Experiments

We report the development of a completely automated pipeline to monitor system suitability in bottom-up proteomic experiments. LC-MS/MS runs are automatically imported into Skyline and multiple identification-free metrics are extracted from targeted peptides. These data are then uploaded to the Panorama Skyline document repository where metrics can be viewed in a web-based interface using powerful process control techniques, including Levey-Jennings and Pareto plots. The interface is versatile and takes user input, which allows the user significant control over the visualization of the data. The pipeline is vendor and instrument-type neutral, supports multiple acquisition techniques (e.g., MS 1 filtering, data-independent acquisition, parallel reaction monitoring, and selected reaction monitoring), can track performance of multiple instruments, and requires no manual intervention aside from initial setup. Data can be viewed from any computer with Internet access and a web browser, facilitating sharing of QC data between researchers. Herein, we describe the use of this pipeline, termed Panorama AutoQC, to evaluate LC-MS/MS performance in a range of scenarios including identification of suboptimal instrument performance, evaluation of ultrahigh pressure chromatography, and identification of the major sources of variation throughout years of peptide data collection.

BatMass: a Java Software Platform for LC-MS Data Visualization in Proteomics and Metabolomics

Mass spectrometry (MS) coupled to liquid chromatography (LC) is a commonly used technique in metabolomic and proteomic research. As the size and complexity of LC-MS-based experiments grow, it becomes increasingly more difficult to perform quality control of both raw data and processing results. In a practical setting, quality control steps for raw LC-MS data are often overlooked, and assessment of an experiment's success is based on some derived metrics such as "the number of identified compounds". The human brain interprets visual data much better than plain text, hence the saying "a picture is worth a thousand words". Here, we present the BatMass software package, which allows for performing quick quality control of raw LC-MS data through its fast visualization capabilities. It also serves as a testbed for developers of LC-MS data processing algorithms by providing a data access library for open mass spectrometry file formats and a means of visually mapping processing results back to the original data. We illustrate the utility of BatMass with several use cases of quality control and data exploration.

Public proteomics data: How the field has evolved from sceptical inquiry to the promise of in silico proteomics

•

Proteomics data sharing moved from validation to re-use.

•

New tools and services make data very easily accessible.

•

Metadata provision can still benefit from improvements.

•

Quality control metrics will soon be reported along with submitted data.

•

Data re-use will enable the advent of actual in silico proteomics.

Unsupervised Quality Assessment of Mass Spectrometry Proteomics Experiments by Multivariate Quality Control Metrics

Despite many technological and computational advances, the results of a mass spectrometry proteomics experiment are still subject to a large variability. For the understanding and evaluation of how technical variability affects the results of an experiment, several computationally derived quality control metrics have been introduced. However, despite the availability of these metrics, a systematic approach to quality control is often still lacking because the metrics are not fully understood and are hard to interpret. Here, we present a toolkit of powerful techniques to analyze and interpret multivariate quality control metrics to assess the quality of mass spectrometry proteomics experiments. We show how unsupervised techniques applied to these quality control metrics can provide an initial discrimination between low-quality experiments and high-quality experiments prior to manual investigation. Furthermore, we provide a technique to obtain detailed information on the quality control metrics that are related to the decreased performance, which can be used as actionable information to improve the experimental setup. Our toolkit is released as open-source and can be downloaded from https://bitbucket.org/proteinspector/qc_analysis/ .

Proteomics Quality Control: Quality Control Software for MaxQuant Results

Mass spectrometry-based proteomics coupled to liquid chromatography has matured into an automatized, high-throughput technology, producing data on the scale of multiple gigabytes per instrument per day. Consequently, an automated quality control (QC) and quality analysis (QA) capable of detecting measurement bias, verifying consistency, and avoiding propagation of error is paramount for instrument operators and scientists in charge of downstream analysis. We have developed an R-based QC pipeline called Proteomics Quality Control (PTXQC) for bottom-up LC-MS data generated by the MaxQuant software pipeline. PTXQC creates a QC report containing a comprehensive and powerful set of QC metrics, augmented with automated scoring functions. The automated scores are collated to create an overview heatmap at the beginning of the report, giving valuable guidance also to nonspecialists. Our software supports a wide range of experimental designs, including stable isotope labeling by amino acids in cell culture (SILAC), tandem mass tags (TMT), and label-free data. Furthermore, we introduce new metrics to score MaxQuant's Match-between-runs (MBR) functionality by which peptide identifications can be transferred across Raw files based on accurate retention time and m/z. Last but not least, PTXQC is easy to install and use and represents the first QC software capable of processing MaxQuant result tables. PTXQC is freely available at https://github.com/cbielow/PTXQC .

The 2012/2013 ABRF Proteomic Research Group Study: Assessing Longitudinal Intralaboratory Variability in Routine Peptide Liquid Chromatography Tandem Mass Spectrometry Analyses

Questions concerning longitudinal data quality and reproducibility of proteomic laboratories spurred the Protein Research Group of the Association of Biomolecular Resource Facilities (ABRF-PRG) to design a study to systematically assess the reproducibility of proteomic laboratories over an extended period of time. Developed as an open study, initially 64 participants were recruited from the broader mass spectrometry community to analyze provided aliquots of a six bovine protein tryptic digest mixture every month for a period of nine months. Data were uploaded to a central repository, and the operators answered an accompanying survey. Ultimately, 45 laboratories submitted a minimum of eight LC-MSMS raw data files collected in data-dependent acquisition (DDA) mode. No standard operating procedures were enforced; rather the participants were encouraged to analyze the samples according to usual practices in the laboratory. Unlike previous studies, this investigation was not designed to compare laboratories or instrument configuration, but rather to assess the temporal intralaboratory reproducibility. The outcome of the study was reassuring with 80% of the participating laboratories performing analyses at a medium to high level of reproducibility and quality over the 9-month period. For the groups that had one or more outlying experiments, the major contributing factor that correlated to the survey data was the performance of preventative maintenance prior to the LC-MSMS analyses. Thus, the Protein Research Group of the Association of Biomolecular Resource Facilities recommends that laboratories closely scrutinize the quality control data following such events. Additionally, improved quality control recording is imperative. This longitudinal study provides evidence that mass spectrometry-based proteomics is reproducible. When quality control measures are strictly adhered to, such reproducibility is comparable among many disparate groups. Data from the study are available via ProteomeXchange under the accession code PXD002114.

Advances in urinary proteome analysis and applications in systems biology

The urinary proteome is the focus of many studies due to the ease of urine collection and the relative proteome stability. Systems biology allows the combination of multiple omics studies, forming a link between proteomics, metabolomics, genomics and transcriptomics. In-depth data interpretation is achieved by bioinformatics analysis of -omics data sets. It is expected that the contribution of systems biology to the study of the urinary proteome will offer novel insights. The main focus of this review is on technical aspects of proteomics studies, available tools for systems biology analysis and the application of urinary proteomics in clinical studies and systems biology.

Multicenter experiment for quality control of peptide-centric LC-MS/MS analysis - A longitudinal performance assessment with nLC coupled to orbitrap MS analyzers

Proteomic technologies based on mass spectrometry (MS) have greatly evolved in the past years, and nowadays it is possible to routinely identify thousands of peptides from complex biological samples in a single LC-MS/MS experiment. Despite the advancements in proteomic technologies, the scientific community still faces important challenges in terms of depth and reproducibility of proteomics analyses. Here, we present a multicenter study designed to evaluate long-term performance of LC-MS/MS platforms within the Spanish Proteomics Facilities Network (ProteoRed-ISCIII). The study was performed under well-established standard operating procedures, and demonstrated that it is possible to attain qualitative and quantitative reproducibility over time. Our study highlights the importance of deploying quality assessment metrics routinely in individual laboratories and in multi-laboratory studies. The mass spectrometry data have been deposited to the ProteomeXchange Consortium with the data set identifier PXD000205.This article is part of a Special Issue entitled: HUPO 2014.

Variation and quantification among a target set of phosphopeptides in human plasma by multiple reaction monitoring and SWATH-MS2 data-independent acquisition

Human plasma contains proteins that reflect overall health and represents a rich source of proteins for identifying and understanding disease pathophysiology. However, few studies have investigated changes in plasma phosphoproteins. In addition, little is known about the normal variations in these phosphoproteins, especially with respect to specific sites of modification. To address these questions, we evaluated variability in plasma protein phosphorylation in healthy individuals using multiple reaction monitoring (MRM) and SWATH-MS2 data-independent acquisition. First, we developed a discovery workflow for phosphopeptide enrichment from plasma and identified targets for MRM assays. Next, we analyzed plasma from healthy donors using an analytical workflow consisting of MRM and SWATH-MS2 that targeted phosphopeptides from 58 and 68 phosphoproteins, respectively. These two methods produced similar results showing low variability in 13 phosphosites from 10 phosphoproteins (CVinter < 30%) and high interpersonal variation of 16 phosphosites from 14 phosphoproteins (CVinter > 30%). Moreover, these phosphopeptides originate from phosphoproteins involved in cellular processes governing homeostasis, immune response, cell-extracellular matrix interactions, lipid and sugar metabolism, and cell signaling. This limited assessment of technical and biological variability in phosphopeptides generated from plasma phosphoproteins among healthy volunteers constitutes a reference for future studies that target protein phosphorylation as biomarkers.

qcML: an exchange format for quality control metrics from mass spectrometry experiments

Quality control is increasingly recognized as a crucial aspect of mass spectrometry based proteomics. Several recent papers discuss relevant parameters for quality control and present applications to extract these from the instrumental raw data. What has been missing, however, is a standard data exchange format for reporting these performance metrics. We therefore developed the qcML format, an XML-based standard that follows the design principles of the related mzML, mzIdentML, mzQuantML, and TraML standards from the HUPO-PSI (Proteomics Standards Initiative). In addition to the XML format, we also provide tools for the calculation of a wide range of quality metrics as well as a database format and interconversion tools, so that existing LIMS systems can easily add relational storage of the quality control data to their existing schema. We here describe the qcML specification, along with possible use cases and an illustrative example of the subsequent analysis possibilities. All information about qcML is available at http://code.google.com/p/qcml.

Implementation of statistical process control for proteomic experiments via LC MS/MS

Statistical process control (SPC) is a robust set of tools that aids in the visualization, detection, and identification of assignable causes of variation in any process that creates products, services, or information. A tool has been developed termed Statistical Process Control in Proteomics (SProCoP) which implements aspects of SPC (e.g., control charts and Pareto analysis) into the Skyline proteomics software. It monitors five quality control metrics in a shotgun or targeted proteomic workflow. None of these metrics require peptide identification. The source code, written in the R statistical language, runs directly from the Skyline interface, which supports the use of raw data files from several of the mass spectrometry vendors. It provides real time evaluation of the chromatographic performance (e.g., retention time reproducibility, peak asymmetry, and resolution), and mass spectrometric performance (targeted peptide ion intensity and mass measurement accuracy for high resolving power instruments) via control charts. Thresholds are experiment- and instrument-specific and are determined empirically from user-defined quality control standards that enable the separation of random noise and systematic error. Finally, Pareto analysis provides a summary of performance metrics and guides the user to metrics with high variance. The utility of these charts to evaluate proteomic experiments is illustrated in two case studies.

Signatures for mass spectrometry data quality

Ensuring data quality and proper instrument functionality is a prerequisite for scientific investigation. Manual quality assurance is time-consuming and subjective. Metrics for describing liquid chromatography mass spectrometry (LC–MS) data have been developed; however, the wide variety of LC–MS instruments and configurations precludes applying a simple cutoff. Using 1150 manually classified quality control (QC) data sets, we trained logistic regression classification models to predict whether a data set is in or out of control. Model parameters were optimized by minimizing a loss function that accounts for the trade-off between false positive and false negative errors. The classifier models detected bad data sets with high sensitivity while maintaining high specificity. Moreover, the composite classifier was dramatically more specific than single metrics. Finally, we evaluated the performance of the classifier on a separate validation set where it performed comparably to the results for the testing/training data sets. By presenting the methods and software used to create the classifier, other groups can create a classifier for their specific QC regimen, which is highly variable lab-to-lab. In total, this manuscript presents 3400 LC–MS data sets for the same QC sample (whole cell lysate of Shewanella oneidensis), deposited to the ProteomeXchange with identifiers PXD000320–PXD000324.

QC metrics from CPTAC raw LC-MS/MS data interpreted through multivariate statistics

Shotgun proteomics experiments integrate a complex sequence of processes, any of which can introduce variability. Quality metrics computed from LC-MS/MS data have relied upon identifying MS/MS scans, but a new mode for the QuaMeter software produces metrics that are independent of identifications. Rather than evaluating each metric independently, we have created a robust multivariate statistical toolkit that accommodates the correlation structure of these metrics and allows for hierarchical relationships among data sets. The framework enables visualization and structural assessment of variability. Study 1 for the Clinical Proteomics Technology Assessment for Cancer (CPTAC), which analyzed three replicates of two common samples at each of two time points among 23 mass spectrometers in nine laboratories, provided the data to demonstrate this framework, and CPTAC Study 5 provided data from complex lysates under Standard Operating Procedures (SOPs) to complement these findings. Identification-independent quality metrics enabled the differentiation of sites and run-times through robust principal components analysis and subsequent factor analysis. Dissimilarity metrics revealed outliers in performance, and a nested ANOVA model revealed the extent to which all metrics or individual metrics were impacted by mass spectrometer and run time. Study 5 data revealed that even when SOPs have been applied, instrument-dependent variability remains prominent, although it may be reduced, while within-site variability is reduced significantly. Finally, identification-independent quality metrics were shown to be predictive of identification sensitivity in these data sets. QuaMeter and the associated multivariate framework are available from http://fenchurch.mc.vanderbilt.edu and http://homepages.uc.edu/~wang2x7/, respectively.

Bringing proteomics into the clinic: the need for the field to finally take itself seriously

Proteomics has fast become a standard tool in the life sciences, with increasingly sophisticated approaches and instruments delivering ever growing numbers of identified and quantified proteins. Yet despite the enormous technological progress, and the triumphant papers published on whole-cell proteomes being collected and analyzed, proteomics has so far failed to enter the clinic for routine applications. This is a peculiar contradiction, and one that warrants some closer study. I here argue that for proteomics to make a difference in the clinic, it needs to stop shirking responsibility, and to mature into an analytical, transparent, and reproducible discipline that also invests in the consolidation of its technology rather than only focusing on the next big leap forward. A key enabling factor in this maturation process is quality control and quality assurance, with bioinformatics, in its least noticeable but most influential form, as a key underlying technology.

Metriculator: quality assessment for mass spectrometry-based proteomics

SUMMARY

Quality control in mass spectrometry-based proteomics remains subjective, labor-intensive and inconsistent between laboratories. We introduce Metriculator, a software designed to facilitate long-term storage of extensive performance metrics as introduced by NIST in 2010. Metriculator features a web interface that generates interactive comparison plots for contextual understanding of metric values and an automated metric generation toolkit. The comparison plots are designed for at-a-glance determination of outliers and trends in the datasets, together with relevant statistical comparisons. Easy-to-use quantitative comparisons and a framework for integration plugins will encourage a culture of quality assurance within the proteomics community.

AVAILABILITY AND IMPLEMENTATION

Available under the MIT license at http://github.com/princelab/metriculator.

IDPQuantify: combining precursor intensity with spectral counts for protein and peptide quantification

Differentiating and quantifying protein differences in complex samples produces significant challenges in sensitivity and specificity. Label-free quantification can draw from two different information sources: precursor intensities and spectral counts. Intensities are accurate for calculating protein relative abundance, but values are often missing due to peptides that are identified sporadically. Spectral counting can reliably reproduce difference lists, but differentiating peptides or quantifying all but the most concentrated protein changes is usually beyond its abilities. Here we developed new software, IDPQuantify, to align multiple replicates using principal component analysis, extract accurate precursor intensities from MS data, and combine intensities with spectral counts for significant gains in differentiation and quantification. We have applied IDPQuantify to three comparative proteomic data sets featuring gold standard protein differences spiked in complicated backgrounds. The software is able to associate peptides with peaks that are otherwise left unidentified to increase the efficiency of protein quantification, especially for low-abundance proteins. By combing intensities with spectral counts from IDPicker, it gains an average of 30% more true positive differences among top differential proteins. IDPQuantify quantifies protein relative abundance accurately in these test data sets to produce good correlations between known and measured concentrations.

An introduction to statistical process control in research proteomics

BACKGROUND

Statistical process control is a well-established and respected method which provides a general purpose, and consistent framework for monitoring and improving the quality of a process. It is routinely used in many industries where the quality of final products is critical and is often required in clinical diagnostic laboratories [1,2]. To date, the methodology has been little utilised in research proteomics. It has been shown to be capable of delivering quantitative QC procedures for qualitative clinical assays [3] making it an ideal methodology to apply to this area of biological research.

OBJECTIVE

To introduce statistical process control as an objective strategy for quality control and show how it could be used to benefit proteomics researchers and enhance the quality of the results they generate.

RESULTS

We demonstrate that rules which provide basic quality control are easy to derive and implement and could have a major impact on data quality for many studies.

CONCLUSIONS

Statistical process control is a powerful tool for investigating and improving proteomics research work-flows. The process of characterising measurement systems and defining control rules forces the exploration of key questions that can lead to significant improvements in performance.

BIOLOGICAL SIGNIFICANCE

This work asserts that QC is essential to proteomics discovery experiments. Every experimenter must know the current capabilities of their measurement system and have an objective means for tracking and ensuring that performance. Proteomic analysis work-flows are complicated and multi-variate. QC is critical for clinical chemistry measurements and huge strides have been made in ensuring the quality and validity of results in clinical biochemistry labs. This work introduces some of these QC concepts and works to bridge their use from single analyte QC to applications in multi-analyte systems. This article is part of a Special Issue entitled: Standardization and Quality Control in Proteomics.

Using R and Bioconductor for proteomics data analysis

This review presents how R, the popular statistical environment and programming language, can be used in the frame of proteomics data analysis. A short introduction to R is given, with special emphasis on some of the features that make R and its add-on packages premium software for sound and reproducible data analysis. The reader is also advised on how to find relevant R software for proteomics. Several use cases are then presented, illustrating data input/output, quality control, quantitative proteomics and data analysis. Detailed code and additional links to extensive documentation are available in the freely available companion package RforProteomics. This article is part of a Special Issue entitled: Computational Proteomics in the Post-Identification Era. Guest Editors: Martin Eisenacher and Christian Stephan.

Automated quality control system for LC-SRM setups

Selected reaction monitoring (SRM) is emerging as a standard tool for high-throughput protein quantification. For reliable and reproducible SRM protein quantification it is essential that system performance is stable. We present here a quality control workflow that is based on repeated analysis of a standard sample to allow insight into the stability of the key properties of a SRM setup. This is supported by automated software to monitor system performance and display information like signal intensities and retention time stability over time, and alert upon deviations from expected metrics. Utilising the software to evaluate 407 repeated injections of a standard sample during half a year, outliers in relative peptide signal intensities and relative peptide fragment ratios are identified, indicating the need for instrument maintenance. We therefore believe that the software could be a vital and powerful tool for any lab regularly performing SRM, increasing the reliability and quality of the SRM platform.

BIOLOGICAL SIGNIFICANCE

Selected reaction monitoring (SRM) mass spectrometry is becoming established as a standard technique for accurate protein quantification. However, to achieve the required quantification reproducibility of the liquid chromatography (LC)-SRM setup, system performance needs to be monitored over time. Here we introduce a workflow with associated software to enable automated monitoring of LC-SRM setups. We believe that usage of the presented concepts will further strengthen the role of SRM as a reliable tool for protein quantification. This article is part of a Special Issue entitled: Standardization and Quality Control in Proteomics.

Quality assessment for clinical proteomics

Proteomics has emerged from the labs of technologists to enter widespread application in clinical contexts. This transition, however, has been hindered by overstated early claims of accuracy, concerns about reproducibility, and the challenges of handling batch effects properly. New efforts have produced sets of performance metrics and measurements of variability that establish sound expectations for experiments in clinical proteomics. As researchers begin incorporating these metrics in a quality by design paradigm, the variability of individual steps in experimental pipelines will be reduced, regularizing overall outcomes. This review discusses the evolution of quality assessment in 2D gel electrophoresis, mass spectrometry-based proteomic profiling, tandem mass spectrometry-based protein inventories, and proteomic quantitation. Taken together, the advances in each of these technologies are establishing databases that will be increasingly useful for decision-making in clinical experimentation.

SIMPATIQCO: a server-based software suite which facilitates monitoring the time course of LC-MS performance metrics on Orbitrap instruments

While the performance of liquid chromatography (LC) and mass spectrometry (MS) instrumentation continues to increase, applications such as analyses of complete or near-complete proteomes and quantitative studies require constant and optimal system performance. For this reason, research laboratories and core facilities alike are recommended to implement quality control (QC) measures as part of their routine workflows. Many laboratories perform sporadic quality control checks. However, successive and systematic longitudinal monitoring of system performance would be facilitated by dedicated automatic or semiautomatic software solutions that aid an effortless analysis and display of QC metrics over time. We present the software package SIMPATIQCO (SIMPle AuTomatIc Quality COntrol) designed for evaluation of data from LTQ Orbitrap, Q-Exactive, LTQ FT, and LTQ instruments. A centralized SIMPATIQCO server can process QC data from multiple instruments. The software calculates QC metrics supervising every step of data acquisition from LC and electrospray to MS. For each QC metric the software learns the range indicating adequate system performance from the uploaded data using robust statistics. Results are stored in a database and can be displayed in a comfortable manner from any computer in the laboratory via a web browser. QC data can be monitored for individual LC runs as well as plotted over time. SIMPATIQCO thus assists the longitudinal monitoring of important QC metrics such as peptide elution times, peak widths, intensities, total ion current (TIC) as well as sensitivity, and overall LC–MS system performance; in this way the software also helps identify potential problems. The SIMPATIQCO software package is available free of charge.