Drug development: Raise standards for preclinical cancer research

Bridging the gap between basic and applied biology: towards preclinical translation

To better translate basic research findings into the clinic, we are moving away from the traditional one-gene–one-phenotype model towards the discovery of complex mechanisms. In this Editorial, the new Editor-in-Chief and Senior Editors of Disease Models & Mechanisms (DMM) discuss the role that the journal will play in this transition. DMM will continue to provide a platform for studies that bridge basic and applied science, and, by demanding the rigorous assessment of animal models of disease, will help drive the establishment of robust standards of preclinical testing for drug development.

Preclinical research in Rett syndrome: setting the foundation for translational success

In September of 2011, the National Institute of Neurological Disorders and Stroke (NINDS), the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), the International Rett Syndrome Foundation (IRSF) and the Rett Syndrome Research Trust (RSRT) convened a workshop involving a broad cross-section of basic scientists, clinicians and representatives from the National Institutes of Health (NIH), the US Food and Drug Administration (FDA), the pharmaceutical industry and private foundations to assess the state of the art in animal studies of Rett syndrome (RTT). The aim of the workshop was to identify crucial knowledge gaps and to suggest scientific priorities and best practices for the use of animal models in preclinical evaluation of potential new RTT therapeutics. This review summarizes outcomes from the workshop and extensive follow-up discussions among participants, and includes: (1) a comprehensive summary of the physiological and behavioral phenotypes of RTT mouse models to date, and areas in which further phenotypic analyses are required to enhance the utility of these models for translational studies; (2) discussion of the impact of genetic differences among mouse models, and methodological differences among laboratories, on the expression and analysis, respectively, of phenotypic traits; and (3) definitions of the standards that the community of RTT researchers can implement for rigorous preclinical study design and transparent reporting to ensure that decisions to initiate costly clinical trials are grounded in reliable preclinical data.

Representation of probabilistic scientific knowledge

The theory of probability is widely used in biomedical research for data analysis and modelling. In previous work the probabilities of the research hypotheses have been recorded as experimental metadata. The ontology HELO is designed to support probabilistic reasoning, and provides semantic descriptors for reporting on research that involves operations with probabilities. HELO explicitly links research statements such as hypotheses, models, laws, conclusions, etc. to the associated probabilities of these statements being true. HELO enables the explicit semantic representation and accurate recording of probabilities in hypotheses, as well as the inference methods used to generate and update those hypotheses. We demonstrate the utility of HELO on three worked examples: changes in the probability of the hypothesis that sirtuins regulate human life span; changes in the probability of hypotheses about gene functions in the S. cerevisiae aromatic amino acid pathway; and the use of active learning in drug design (quantitative structure activity relation learning), where a strategy for the selection of compounds with the highest probability of improving on the best known compound was used. HELO is open source and available at https://github.com/larisa-soldatova/HELO

Power failure: why small sample size undermines the reliability of neuroscience

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.

Is Intelligence Enhanced by Letter Priming? A Failure to Replicate the Results of Ciani and Sheldon (2010)

Previous research has suggested that motivational processes outside an individual's conscious awareness may be primed so as to enhance or impair cognitive performance. The present study involved a conceptual replication of the 2010 study of Ciani and Sheldon (Experiments 1 and 2), employing the same materials and task, to test whether exposure to the letter A before an analogies test improved performance and the letter F impaired it, relative to the neutral letter J. It also examined the effect of pre-exposing participants before testing to a positive or negative verbal passage concerning letter grades. Priming was not found to have any effect: the participants ( N = 116), under both pre-exposure conditions, gave analogies scores which were virtually identical whether they had been primed with A, F, or J, thus contradicting the previous results. It is concluded that there is a pressing need for more replications of priming experiments as well as other studies.

Transparency in the reporting of in vivo pre-clinical pain research: The relevance and implications of the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines

Clear reporting of research is crucial to the scientific process. Poorly designed and reported studies are damaging not only to the efforts of individual researchers, but also to science as a whole. Standardised reporting methods, such as those already established for reporting randomised clinical trials, have led to improved study design and facilitated the processes of clinical systematic review and meta-analysis.

Such standards were lacking in the pre-clinical field until the development of the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines. These were prompted following a survey which highlighted a widespread lack of robust and consistent reporting of pre-clinical in vivo research, with reports frequently omitting basic information required for study replication and quality assessment.

The resulting twenty item checklist in ARRIVE covers all aspects of experimental design with particular emphasis on bias reduction and methodological transparency. Influential publishers and research funders have already adopted ARRIVE. Further dissemination and acknowledgement of the importance of these guidelines is vital to their widespread implementation.

Conclusions and implications

Wide implementation of the ARRIVE guidelines for reporting of in vivo preclinical research, especially pain research, are essential for a much needed increased transparency and quality in publishing such research. ARRIVE will also positively influence improvements in experimental design and quality, assist the conduct of accurate replication studies of important new findings and facilitate meta-analyses of preclinical research.

Improving basic and translational science by accounting for litter-to-litter variation in animal models

Background

Animals from the same litter are often more alike compared with animals from different litters. This litter-to-litter variation, or “litter effects”, can influence the results in addition to the experimental factors of interest. Furthermore, sometimes an experimental treatment can only be applied to whole litters rather than to individual offspring. An example is the valproic acid (VPA) model of autism, where VPA is administered to pregnant females thereby inducing the disease phenotype in the offspring. With this type of experiment the sample size is the number of litters and not the total number of offspring. If such experiments are not appropriately designed and analysed, the results can be severely biased as well as extremely underpowered.

Results

A review of the VPA literature showed that only 9% (3/34) of studies correctly determined that the experimental unit (n) was the litter and therefore made valid statistical inferences. In addition, litter effects accounted for up to 61% (p <0.001) of the variation in behavioural outcomes, which was larger than the treatment effects. In addition, few studies reported using randomisation (12%) or blinding (18%), and none indicated that a sample size calculation or power analysis had been conducted.

Conclusions

Litter effects are common, large, and ignoring them can make replication of findings difficult and can contribute to the low rate of translating preclinical in vivo studies into successful therapies. Only a minority of studies reported using rigorous experimental methods, which is consistent with much of the preclinical in vivo literature.

Protein kinase C, an elusive therapeutic target?

Protein kinase C (PKC) has been a tantalizing target for drug discovery ever since it was first identified as the receptor for the tumour promoter phorbol ester in 1982. Although initial therapeutic efforts focused on cancer, additional indications--including diabetic complications, heart failure, myocardial infarction, pain and bipolar disorder--were targeted as researchers developed a better understanding of the roles of eight conventional and novel PKC isozymes in health and disease. Unfortunately, both academic and pharmaceutical efforts have yet to result in the approval of a single new drug that specifically targets PKC. Why does PKC remain an elusive drug target? This Review provides a short account of some of the efforts, challenges and opportunities in developing PKC modulators to address unmet clinical needs.

Learning from our mistakes: the 'unknown knowns' in fragment screening

In the past 15 years, fragment-based lead discovery (FBLD) has been adopted widely throughout academia and industry. The approach entails discovering very small molecular fragments and growing, merging, or linking them to produce drug leads. Because the affinities of the initial fragments are often low, detection methods are pushed to their limits, leading to a variety of artifacts, false positives, and false negatives that too often go unrecognized. This Digest discusses some of these problems and offers suggestions to avoid them. Although the primary focus is on FBLD, many of the lessons also apply to more established approaches such as high-throughput screening.

Biosafety data as confidential business information

Removal of confidentiality claims on biosafety data is necessary to adhere to standard scientific procedures of quality assurance, to increase transparency, to minimize impacts of conflicts of interests, and ultimately to improve public confidence in GMOs.

Unraveling the Complexities of Life Sciences Data

The life sciences have entered into the realm of big data and data-enabled science, where data can either empower or overwhelm. These data bring the challenges of the 5 Vs of big data: volume, veracity, velocity, variety, and value. Both independently and through our involvement with DELSA Global (Data-Enabled Life Sciences Alliance, DELSAglobal.org), the Kolker Lab ( kolkerlab.org ) is creating partnerships that identify data challenges and solve community needs. We specialize in solutions to complex biological data challenges, as exemplified by the community resource of MOPED (Model Organism Protein Expression Database, MOPED.proteinspire.org ) and the analysis pipeline of SPIRE (Systematic Protein Investigative Research Environment, PROTEINSPIRE.org ). Our collaborative work extends into the computationally intensive tasks of analysis and visualization of millions of protein sequences through innovative implementations of sequence alignment algorithms and creation of the Protein Sequence Universe tool (PSU). Pushing into the future together with our collaborators, our lab is pursuing integration of multi-omics data and exploration of biological pathways, as well as assigning function to proteins and porting solutions to the cloud. Big data have come to the life sciences; discovering the knowledge in the data will bring breakthroughs and benefits.

Git can facilitate greater reproducibility and increased transparency in science

Background

Reproducibility is the hallmark of good science. Maintaining a high degree of transparency in scientific reporting is essential not just for gaining trust and credibility within the scientific community but also for facilitating the development of new ideas. Sharing data and computer code associated with publications is becoming increasingly common, motivated partly in response to data deposition requirements from journals and mandates from funders. Despite this increase in transparency, it is still difficult to reproduce or build upon the findings of most scientific publications without access to a more complete workflow.

Findings

Version control systems (VCS), which have long been used to maintain code repositories in the software industry, are now finding new applications in science. One such open source VCS, Git, provides a lightweight yet robust framework that is ideal for managing the full suite of research outputs such as datasets, statistical code, figures, lab notes, and manuscripts. For individual researchers, Git provides a powerful way to track and compare versions, retrace errors, explore new approaches in a structured manner, while maintaining a full audit trail. For larger collaborative efforts, Git and Git hosting services make it possible for everyone to work asynchronously and merge their contributions at any time, all the while maintaining a complete authorship trail. In this paper I provide an overview of Git along with use-cases that highlight how this tool can be leveraged to make science more reproducible and transparent, foster new collaborations, and support novel uses.

Association of fascin-1 with mortality, disease progression and metastasis in carcinomas: a systematic review and meta-analysis

BACKGROUND

Fascin-1 is an actin-bundling protein expressed in many human carcinomas, although absent from most normal epithelia. Fascin-1 promotes filopodia formation, migration and invasion in carcinoma cells; in mouse xenograft tumor models it contributes to metastasis. Fascin-1 is an interesting candidate biomarker for aggressive, metastatic carcinomas but data from individual studies of human tumors have not yet been pooled systematically.

METHODS

This systematic review was conducted in accordance with PRISMA guidelines, using fixed and random effects models, as appropriate, to undertake meta-analysis.

RESULTS

A total of 26 immunohistochemical studies of 5 prevalent human carcinomas were identified for meta-analysis. Fascin-1 was associated with increased risk of mortality for breast (pooled hazard ratio, (HR) = 2.58; 95% confidence interval (CI) 1.48 to 4.52; P = 0.001), colorectal (HR = 1.60 (1.37 to 1.86; P <0.001) and esophageal carcinomas (HR = 1.35; CI 1.13 to 1.60; P = 0.001). There was no evidence of association of fascin-1 with mortality in gastric and lung carcinomas. Fascin-1 was associated with increased risk of disease progression in breast (HR = 2.48; CI 1.38 to 4.46; P = 0.002) and colorectal carcinomas (HR = 2.12; CI 1.00 to 4.47; P = 0.05), but not with progression of lung carcinomas (HR = 0.95; CI 0.49 to 1.85; P = 0.9). Fascin-1 was associated with increased risk of lymph node metastasis in colorectal (pooled risk ratio (RR) = 1.47; CI 1.26 to 1.71; P <0.001) and gastric carcinomas (RR = 1.43; CI 1.21 to 1.70; P <0.001). There was no evidence of association of fascin-1 with lymph node metastasis in lung or esophageal carcinomas. Fascin-1 was associated with increased risk of distant metastasis in colorectal (RR = 1.70; CI 1.18 to 2.45; P = 0.004) and gastric carcinomas (RR = 1.93; CI 1.21 to 3.33; P = 0.02). No association with distant metastasis in esophageal carcinomas was observed. Pooling across all the carcinomas provided strong evidence for association of fascin-1 with increased risk of mortality (HR = 1.44; CI 1.24 to 1.68; P <0.001; n = 3,645), lymph node metastasis (RR = 1.36; CI 1.18 to 1.55; P <0.001; n = 2,906) and distant metastasis (1.76; 1.34 to 2.32; P <0.001; n = 1,514).

CONCLUSIONS

Fascin-1 is associated consistently with increased risk of mortality in breast, colorectal and esophageal carcinomas and with metastasis in colorectal and gastric carcinomas. The results were stable to various sensitivity analyses and did not vary by predefined subgroups. These data will assist rational decision making for focusing investigations of fascin-1 as a biomarker or therapeutic target onto the most relevant carcinomas.

Meeting report: using stem cells for biological and therapeutics discovery in mental illness, April 2012

This report synthesizes the discussions during a workshop convened April 24-25, 2012, by the National Institute of Mental Health and the Foundation for the NIH in Bethesda, Maryland, that focused on progress and challenges in the use of patient-derived reprogrammed cells for basic biological discovery, target identification, screening, and drug development for mental illnesses such as schizophrenia, bipolar disorder, and autism spectrum disorders. The workshop revealed that the greatest progress has been made in reprogramming methods and agreed-upon standards for validating the resulting induced pluripotent stem cell lines. However, challenges remain in several areas, including efficiently generating and validating specific neural cell types with respect to regional identity, establishing assays with predictive validity to mental illness pathophysiology, and generating sufficient statistical power and data reproducibility across laboratories. A brainstorming session yielded a number of suggestions, including calls to (a) facilitate the replication of results by standardizing protocols and samples used across laboratories; (b) improve technology by generating cheaper/faster targeting methods, reporters, and assays; and (c) improve resource sharing and collaboration, with an emphasis on rapid sharing of new cell lines, technologies, and best practices, possibly incorporated into a public-private partnership. The meeting provided an important venue for academic, government, and private sector scientists to address potential opportunities for translational and clinical applications of reprogrammed cell research. A number of activities since the workshop have reflected the feedback from meeting participants.

Towards more effective advanced drug delivery systems

This position paper discusses progress made and to be made with so-called advanced drug delivery systems, particularly but not exclusively those in the nanometre domain. The paper has resulted from discussions with a number of international experts in the field who shared their views on aspects of the subject, from the nomenclature used for such systems, the sometimes overwrought claims made in the era of nanotechnology, the complex nature of targeting delivery systems to specific destinations in vivo, the need for setting standards for the choice and characterisation of cell lines used in in vitro studies, to attention to the manufacturability, stability and analytical profiling of systems and more relevant studies on toxicology. The historical background to the development of many systems is emphasised. So too is the stochastic nature of many of the steps to successful access to and action in targets. A lacuna in the field is the lack of availability of data on a variety of carrier systems using the same models in vitro and in vivo using standard controls. The paper asserts that greater emphasis must also be paid to the effective levels of active attained in target organs, for without such crucial data it will be difficult for many experimental systems to enter the clinic. This means the use of diagnostic/imaging technologies to monitor targeted drug delivery and stratify patient groups, identifying patients with optimum chances for successful therapy. Last, but not least, the critical importance of the development of science bases for regulatory policies, scientific platforms overseeing the field and new paradigms of financing are discussed.

A comparative antibody analysis of pannexin1 expression in four rat brain regions reveals varying subcellular localizations

Pannexin1 (Panx1) channels release cytosolic ATP in response to signaling pathways. Panx1 is highly expressed in the central nervous system. We used four antibodies with different Panx1 anti-peptide epitopes to analyze four regions of rat brain. These antibodies labeled the same bands in Western blots and had highly similar patterns of immunofluorescence in tissue culture cells expressing Panx1, but Western blots of brain lysates from Panx1 knockout and control mice showed different banding patterns. Localizations of Panx1 in brain slices were generated using automated wide field mosaic confocal microscopy for imaging large regions of interest while retaining maximum resolution for examining cell populations and compartments. We compared Panx1 expression over the cerebellum, hippocampus with adjacent cortex, thalamus, and olfactory bulb. While Panx1 localizes to the same neuronal cell types, subcellular localizations differ. Two antibodies with epitopes against the intracellular loop and one against the carboxy terminus preferentially labeled cell bodies, while an antibody raised against an N-terminal peptide highlighted neuronal processes more than cell bodies. These labeling patterns may be a reflection of different cellular and subcellular localizations of full-length and/or modified Panx1 channels where each antibody is highlighting unique or differentially accessible Panx1 populations. However, we cannot rule out that one or more of these antibodies have specificity issues. All data associated with experiments from these four antibodies are presented in a manner that allows them to be compared and our claims thoroughly evaluated, rather than eliminating results that were questionable. Each antibody is given a unique identifier through the NIF Antibody Registry that can be used to track usage of individual antibodies across papers and all image and metadata are made available in the public repository, the Cell Centered Database, for on-line viewing, and download.

Benchmarking human protein complexes to investigate drug-related systems and evaluate predicted protein complexes

Protein complexes are key entities to perform cellular functions. Human diseases are also revealed to associate with some specific human protein complexes. In fact, human protein complexes are widely used for protein function annotation, inference of human protein interactome, disease gene prediction, and so on. Therefore, it is highly desired to build an up-to-date catalogue of human complexes to support the research in these applications. Protein complexes from different databases are as expected to be highly redundant. In this paper, we designed a set of concise operations to compile these redundant human complexes and built a comprehensive catalogue called CHPC2012 (Catalogue of Human Protein Complexes). CHPC2012 achieves a higher coverage for proteins and protein complexes than those individual databases. It is also verified to be a set of complexes with high quality as its co-complex protein associations have a high overlap with protein-protein interactions (PPI) in various existing PPI databases. We demonstrated two distinct applications of CHPC2012, that is, investigating the relationship between protein complexes and drug-related systems and evaluating the quality of predicted protein complexes. In particular, CHPC2012 provides more insights into drug development. For instance, proteins involved in multiple complexes (the overlapping proteins) are potential drug targets; the drug-complex network is utilized to investigate multi-target drugs and drug-drug interactions; and the disease-specific complex-drug networks will provide new clues for drug repositioning. With this up-to-date reference set of human protein complexes, we believe that the CHPC2012 catalogue is able to enhance the studies for protein interactions, protein functions, human diseases, drugs, and related fields of research. CHPC2012 complexes can be downloaded from http://www1.i2r.a-star.edu.sg/xlli/CHPC2012/CHPC2012.htm.

Context-dependent multifunctionality of galectin-1: a challenge for defining the lectin as therapeutic target

INTRODUCTION

One route of translating the information encoded in the glycan chains of cellular glycoconjugates into physiological effects is via receptor (lectin) binding. A family of endogenous lectins, sharing folding, a distinct sequence signature and affinity for β-galactosides (thus termed galectins), does so effectively in a context-dependent manner.

AREAS COVERED

An overview is given on the multifunctional nature of galectins, with emphasis on galectin-1. The broad range of functions includes vital processes such as adhesion via glycan bridging, glycoconjugate transport or triggering signaling relevant, for example, for growth regulation. Besides distinct glycoconjugates, this lectin can also interact with certain proteins so that it can target counterreceptors at all sites of location, that is, in the cytoplasm and/or nucleus, at both sides of the membrane or extracellularly. Approaches to strategically exploit galectin activities with therapeutic intentions are outlined.

EXPERT OPINION

The wide versatility of sugar coding and the multifunctionality of galectin-1 explain why considering to turn the protein into a therapeutic target is an ambitious aim. Natural pathways shaped by physiologic master regulators (e.g., the tumor suppressor p16(INK4a)) are suggested to teach inspiring lessons as to how the lectin might be recruited to clinical service.

Selecting the best drugs for phase I clinical development and beyond

Attrition rates of drugs from human entry to regulatory approval are far higher in anticancer drugs than those for nononcology indications. In the era of molecular therapeutics that results from a deeper understanding in cancer biology and advancing technologies, the number of compounds available for clinical testing is likely to continue to increase. Although the main objectives of phase I trials are to characterize toxicities of new agents and to determine the recommended dose for phase II development, most phase I studies are now designed to provide some early signal on preliminary efficacy as secondary objectives. The "go-no-go" decision to further develop a drug, or not, is now often pushed forward to the phase I setting. Thus, there is a need for investigators to be able to critically review the preclinical data available in order to determine which drugs should advance on the developmental path. This review highlights the intrinsic characteristics of a drug and the relevant data to be collected during its preclinical assessment, which may maximize the chances of success in clinical testing and eventual regulatory approval.

Progress in using systematic reviews of animal studies to improve translational research

Carlijn Hooijmans and colleagues discuss developments that might improve the quality and translation of animal research, focusing on the importance of systematic reviews, the role of an international register of animal studies, and cooperation across the scientific community. Please see later in the article for the Editors' Summary

Look back in anger - what clinical studies tell us about preclinical work

Misled by animal studies and basic research? Whenever we take a closer look at the outcome of clinical trials in a field such as, most recently, stroke or septic shock, we see how limited the value of our preclinical models was. For all indications, 95% of drugs that enter clinical trials do not make it to the market, despite all promise of the (animal) models used to develop them. Drug development has started already to decrease its reliance on animal models: In Europe, for example, despite increasing R&D expenditure, animal use by pharmaceutical companies dropped by more than 25% from 2005 to 2008. In vitro studies are likewise limited: questionable cell authenticity, over-passaging, mycoplasma infections, and lack of differentiation as well as non-homeostatic and non-physiologic culture conditions endanger the relevance of these models. The standards of statistics and reporting often are poor, further impairing reliability. Alarming studies from industry show miserable reproducibility of landmark studies. This paper discusses factors contributing to the lack of reproducibility and relevance of pre-clinical research.

THE CONCLUSION

Publish less but of better quality and do not rely on the face value of animal studies.

Opening up drug development to everyone

Effective drug therapies are a cornerstone of medical practice. The path from drug discovery to approval is expensive and commonly associated with failure. The cost of drug development exceeds 800 million dollars per product. Late failure due to lack of clinical efficacy is a common cause of high costs. Recent attempts to improve the process of drug development involve the formation of public-private partnerships, which are facilitating the creation of new collaborations among corporate and nonprofit entities to find solutions that will accelerate innovative drug discovery.

Is everything we eat associated with cancer? A systematic cookbook review

BACKGROUND

Nutritional epidemiology is a highly prolific field. Debates on associations of nutrients with disease risk are common in the literature and attract attention in public media.

OBJECTIVE

We aimed to examine the conclusions, statistical significance, and reproducibility in the literature on associations between specific foods and cancer risk.

DESIGN

We selected 50 common ingredients from random recipes in a cookbook. PubMed queries identified recent studies that evaluated the relation of each ingredient to cancer risk. Information regarding author conclusions and relevant effect estimates were extracted. When >10 articles were found, we focused on the 10 most recent articles.

RESULTS

Forty ingredients (80%) had articles reporting on their cancer risk. Of 264 single-study assessments, 191 (72%) concluded that the tested food was associated with an increased (n = 103) or a decreased (n = 88) risk; 75% of the risk estimates had weak (0.05 > P ≥ 0.001) or no statistical (P > 0.05) significance. Statistically significant results were more likely than nonsignificant findings to be published in the study abstract than in only the full text (P < 0.0001). Meta-analyses (n = 36) presented more conservative results; only 13 (26%) reported an increased (n = 4) or a decreased (n = 9) risk (6 had more than weak statistical support). The median RRs (IQRs) for studies that concluded an increased or a decreased risk were 2.20 (1.60, 3.44) and 0.52 (0.39, 0.66), respectively. The RRs from the meta-analyses were on average null (median: 0.96; IQR: 0.85, 1.10).

CONCLUSIONS

Associations with cancer risk or benefits have been claimed for most food ingredients. Many single studies highlight implausibly large effects, even though evidence is weak. Effect sizes shrink in meta-analyses.

Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments

BACKGROUND

The vast majority of medical interventions introduced into clinical development prove unsafe or ineffective. One prominent explanation for the dismal success rate is flawed preclinical research. We conducted a systematic review of preclinical research guidelines and organized recommendations according to the type of validity threat (internal, construct, or external) or programmatic research activity they primarily address.

METHODS AND FINDINGS

We searched MEDLINE, Google Scholar, Google, and the EQUATOR Network website for all preclinical guideline documents published up to April 9, 2013 that addressed the design and conduct of in vivo animal experiments aimed at supporting clinical translation. To be eligible, documents had to provide guidance on the design or execution of preclinical animal experiments and represent the aggregated consensus of four or more investigators. Data from included guidelines were independently extracted by two individuals for discrete recommendations on the design and implementation of preclinical efficacy studies. These recommendations were then organized according to the type of validity threat they addressed. A total of 2,029 citations were identified through our search strategy. From these, we identified 26 guidelines that met our eligibility criteria--most of which were directed at neurological or cerebrovascular drug development. Together, these guidelines offered 55 different recommendations. Some of the most common recommendations included performance of a power calculation to determine sample size, randomized treatment allocation, and characterization of disease phenotype in the animal model prior to experimentation.

CONCLUSIONS

By identifying the most recurrent recommendations among preclinical guidelines, we provide a starting point for developing preclinical guidelines in other disease domains. We also provide a basis for the study and evaluation of preclinical research practice. Please see later in the article for the Editors' Summary.

Bad medicine

In his new book Ben Goldacre argues that the pharmaceutical industry is in poor health and in urgent need of treatment. Richard Smith agrees.

Vitamin e and the healing of bone fracture: the current state of evidence

Background. The effect of vitamin E on health-related conditions has been extensively researched, with varied results. However, to date, there was no published review of the effect of vitamin E on bone fracture healing. Purpose. This paper systematically audited past studies of the effect of vitamin E on bone fracture healing. Methods. Related articles were identified from Medline, CINAHL, and Scopus databases. Screenings were performed based on the criteria that the study must be an original study that investigated the independent effect of vitamin E on bone fracture healing. Data were extracted using standardised forms, followed by evaluation of quality of reporting using ARRIVE Guidelines, plus recalculation procedure for the effect size and statistical power of the results. Results. Six animal studies fulfilled the selection criteria. The study methods were heterogeneous with mediocre reporting quality and focused on the antioxidant-related mechanism of vitamin E. The metasynthesis showed α-tocopherol may have a significant effect on bone formation during the normal bone remodeling phase of secondary bone healing. Conclusion. In general, the effect of vitamin E on bone fracture healing remained inconclusive due to the small number of heterogeneous and mediocre studies included in this paper.

Are we witnessing the decline effect in the Type D personality literature? What can be learned?

After an unbroken series of positive, but underpowered studies seemed to demonstrate Type D personality predicting mortality in cardiovascular disease patients, initial claims now appear at least exaggerated and probably false. Larger studies with consistently null findings are accumulating. Conceptual, methodological, and statistical issues can be raised concerning the construction of Type D personality as a categorical variable, whether Type D is sufficiently distinct from other negative affect variables, and if it could be plausibly assumed to predict mortality independent of depressive symptoms and known biomedical factors, including disease severity. The existing literature concerning negative affect and health suggests a low likelihood of discovering a new negative affect variable that independently predicts mortality better than its many rivals. The apparent decline effect in the Type D literature is discussed in terms of the need to reduce the persistence of false positive findings in the psychosomatic medicine literature, even while preserving a context allowing risk-taking and discovery. Recommendations include greater transparency concerning research design and analytic strategy; insistence on replication with larger samples before accepting "discoveries" from small samples; reduced confirmatory bias; and availability of all relevant data. Such changes would take time to implement, face practical difficulties, and run counter to established practices. An interim solution is for readers to maintain a sense of pre-discovery probabilities, to be sensitized to the pervasiveness of the decline effect, and to be skeptical of claims based on findings reaching significance in small-scale studies that have not been independently replicated.

Alzheimer's therapeutics: continued clinical failures question the validity of the amyloid hypothesis-but what lies beyond?

The worldwide incidence of Alzheimer's disease (AD) is increasing with estimates that 115 million individuals will have AD by 2050, creating an unsustainable healthcare challenge due to a lack of effective treatment options highlighted by multiple clinical failures of agents designed to reduce the brain amyloid burden considered synonymous with the disease. The amyloid hypothesis that has been the overarching focus of AD research efforts for more than two decades has been questioned in terms of its causality but has not been unequivocally disproven despite multiple clinical failures, This is due to issues related to the quality of compounds advanced to late stage clinical trials and the lack of validated biomarkers that allow the recruitment of AD patients into trials before they are at a sufficiently advanced stage in the disease where therapeutic intervention is deemed futile. Pursuit of a linear, reductionistic amyloidocentric approach to AD research, which some have compared to a religious faith, has resulted in other, equally plausible but as yet unvalidated AD hypotheses being underfunded leading to a disastrous roadblock in the search for urgently needed AD therapeutics. Genetic evidence supporting amyloid causality in AD is reviewed in the context of the clinical failures, and progress in tau-based and alternative approaches to AD, where an evolving modus operandi in biomedical research fosters excessive optimism and a preoccupation with unproven, and often ephemeral, biomarker/genome-based approaches that override transparency, objectivity and data-driven decision making, resulting in low probability environments where data are subordinate to self propagating hypotheses.

Bench to bedside translation of antibody drug conjugates using a multiscale mechanistic PK/PD model: a case study with brentuximab-vedotin

To build a multiscale mechanism based pharmacokinetic-pharmacodynamic (PK/PD) model for antibody drug conjugates (ADCs), using brentuximab-vedotin as an example, for preclinical to clinical translation of ADC efficacy. Brentuximab-vedotin experimental data, collected from diverse publications, were employed in the following steps to build and validate the model: (1) characterization of ADC and payload PK at the cellular level, (2) characterization of payload PK in plasma and tumor tissue of xenograft mouse, (3) characterization of ADC PK in mouse plasma, (4) prediction of the tumor payload concentrations in xenograft mouse by integrating parameters obtained from steps 1-3 with the novel tumor disposition model for ADC, (5) characterization of preclinical brentuximab-vedotin tumor growth inhibition data using the novel PK/PD model, (6) characterization of ADC and payload PK in cancer patients, and (7) prediction of clinical responses of brentuximab-vedotin using the PK/PD model, by integrating PK parameters obtained from step 6, and translated mouse parameters from step 5; and comparing them with clinical trial results. The tumor disposition model was able to accurately predict xenograft tumor and plasma payload concentrations. PK/PD model predicted progression free survival rates and complete response rates for brentuximab-vedotin in patients were comparable to the observed clinical results. It is essential to understand and characterize the disposition of ADC and payload, at the cellular and physiological level, to predict the clinical outcome of ADC. A first of its kind mechanistic model has been developed for ADCs, which can integrate preclinical biomeasures and PK/PD data, to predict clinical response.

A Vast Graveyard of Undead Theories

Publication bias remains a controversial issue in psychological science. The tendency of psychological science to avoid publishing null results produces a situation that limits the replicability assumption of science, as replication cannot be meaningful without the potential acknowledgment of failed replications. We argue that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact.

An Agenda for Purely Confirmatory Research

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology’s academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result—a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled “exploratory.” We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

The Rules of the Game Called Psychological Science

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Is the Replicability Crisis Overblown? Three Arguments Examined

We discuss three arguments voiced by scientists who view the current outpouring of concern about replicability as overblown. The first idea is that the adoption of a low alpha level (e.g., 5%) puts reasonable bounds on the rate at which errors can enter the published literature, making false-positive effects rare enough to be considered a minor issue. This, we point out, rests on statistical misunderstanding: The alpha level imposes no limit on the rate at which errors may arise in the literature (Ioannidis, 2005b). Second, some argue that whereas direct replication attempts are uncommon, conceptual replication attempts are common—providing an even better test of the validity of a phenomenon. We contend that performing conceptual rather than direct replication attempts interacts insidiously with publication bias, opening the door to literatures that appear to confirm the reality of phenomena that in fact do not exist. Finally, we discuss the argument that errors will eventually be pruned out of the literature if the field would just show a bit of patience. We contend that there are no plausible concrete scenarios to back up such forecasts and that what is needed is not patience, but rather systematic reforms in scientific practice.

A call for transparent reporting to optimize the predictive value of preclinical research

The US National Institute of Neurological Disorders and Stroke convened major stakeholders in June 2012 to discuss how to improve the methodological reporting of animal studies in grant applications and publications. The main workshop recommendation is that at a minimum studies should report on sample-size estimation, whether and how animals were randomized, whether investigators were blind to the treatment, and the handling of data. We recognize that achieving a meaningful improvement in the quality of reporting will require a concerted effort by investigators, reviewers, funding agencies and journal editors. Requiring better reporting of animal studies will raise awareness of the importance of rigorous study design to accelerate scientific progress.

Next-generation sequencing data interpretation: enhancing reproducibility and accessibility

Areas of life sciences research that were previously distant from each other in ideology, analysis practices and toolkits, such as microbial ecology and personalized medicine, have all embraced techniques that rely on next-generation sequencing instruments. Yet the capacity to generate the data greatly outpaces our ability to analyse it. Existing sequencing technologies are more mature and accessible than the methodologies that are available for individual researchers to move, store, analyse and present data in a fashion that is transparent and reproducible. Here we discuss currently pressing issues with analysis, interpretation, reproducibility and accessibility of these data, and we present promising solutions and venture into potential future developments.

Biospecimens, biomarkers, and burgeoning data: the imperative for more rigorous research standards

Knowledge of the altered molecular landscapes in disease offers great promise for developing biomarker-based tests to improve diagnosis and optimize treatment. Progress in biomarker research has been frustratingly slow due to the poor clinical trial design and the lack of standards for specimen collection, biomarker analysis, and data reporting. The ability of high throughput genomics, proteomics, and other 'omics' platforms to profile a large number of analytes in a single assay, together with the pending prospect of rapid expansion of whole exome and whole genome sequencing for clinical use, is increasing the technical and logistical complexity of biomarker validation. Harnessing these new technologies and improved productivity in biomarker validation will depend on adopting systems-based approaches and require major changes in the organization and funding strategies for biomarker research. A systems approach will require new multi-institution collaborations, the integration of diverse technical and clinical activities, greater engagement of industry, and education of regulators, clinicians, and payers about how to use biomarkers for improved patient management and clinical outcomes.

Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

Autologous olfactory mucosal cell transplants in clinical spinal cord injury: a randomized double-blinded trial in a canine translational model

This study was designed to determine whether an intervention proven effective in the laboratory to ameliorate the effects of experimental spinal cord injury could provide sufficient benefit to be of value to clinical cases. Intraspinal olfactory ensheathing cell transplantation improves locomotor outcome after spinal cord injury in 'proof of principle' experiments in rodents, suggesting the possibility of efficacy in human patients. However, laboratory animal spinal cord injury cannot accurately model the inherent heterogeneity of clinical patient cohorts, nor are all aspects of their spinal cord function readily amenable to objective evaluation. Here, we measured the effects of intraspinal transplantation of cells derived from olfactory mucosal cultures (containing a mean of ~50% olfactory ensheathing cells) in a population of spinal cord-injured companion dogs that accurately model many of the potential obstacles involved in transition from laboratory to clinic. Dogs with severe chronic thoracolumbar spinal cord injuries (equivalent to ASIA grade 'A' human patients at ~12 months after injury) were entered into a randomized double-blinded clinical trial in which they were allocated to receive either intraspinal autologous cells derived from olfactory mucosal cultures or injection of cell transport medium alone. Recipients of olfactory mucosal cell transplants gained significantly better fore-hind coordination than those dogs receiving cell transport medium alone. There were no significant differences in outcome between treatment groups in measures of long tract functionality. We conclude that intraspinal olfactory mucosal cell transplantation improves communication across the damaged region of the injured spinal cord, even in chronically injured individuals. However, we find no evidence for concomitant improvement in long tract function.

The 6th drug discovery for neurodegeneration conference: an intensive course on translating research into drugs

Neurodegenerative disorders are rapidly becoming one of the greatest unmet health needs. This annual workshop facilitates innovation and progress in neurodegenerative disease drug discovery by convening stakeholders from charitable foundations, government, academia and industry who introduce scientists to the drug development and approval process. New to the 2012 workshop were candid discussions about re-visiting the CNS therapeutic development process. The continuing challenge is partly due to the poor forecasting potential of models of CNS diseases, as well as the lack of reproducibility of published studies, and greater need to increase focus on pharmacodynamic end points. Significant discussion centered on how to improve discovery approaches using examples of recent successes in the field. For example, the idea of combining reductionist, single-target strategies with functional-approach logic was suggested by several speakers, and widely discussed in the workshop. The didactic aspects of the workshop highlighted underlying concepts, best practices and trends that have characterized successful campaigns. The technical and scientific guidance was complemented by discussions of practical ways to approach the major funding gaps required for translation of projects from basic science to clinical investigations. This workshop has evolved to serve a critical educational need, with a wide range of investigator participation.

Knowledge integration in cancer: current landscape and future prospects

Knowledge integration includes knowledge management, synthesis, and translation processes. It aims to maximize the use of collected scientific information and accelerate translation of discoveries into individual and population health benefits. Accumulated evidence in cancer epidemiology constitutes a large share of the 2.7 million articles on cancer in PubMed. We examine the landscape of knowledge integration in cancer epidemiology. Past approaches have mostly used retrospective efforts of knowledge management and traditional systematic reviews and meta-analyses. Systematic searches identify 2,332 meta-analyses, about half of which are on genetics and epigenetics. Meta-analyses represent 1:89-1:1162 of published articles in various cancer subfields. Recently, there are more collaborative meta-analyses with individual-level data, including those with prospective collection of measurements [e.g., genotypes in genome-wide association studies (GWAS)]; this may help increase the reliability of inferences in the field. However, most meta-analyses are still done retrospectively with published information. There is also a flurry of candidate gene meta-analyses with spuriously prevalent "positive" results. Prospective design of large research agendas, registration of datasets, and public availability of data and analyses may improve our ability to identify knowledge gaps, maximize and accelerate translational progress or-at a minimum-recognize dead ends in a more timely fashion.