Getting to grips with c-Myc

The transcription factor c-Myc amplifies the transcription of many growth-related genes in cancer cells, but its role as an oncogene is not fully understood.

Replication Study: Transcriptional amplification in tumor cells with elevated c-Myc

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Blum et al., 2015), that described how we intended to replicate selected experiments from the paper 'Transcriptional amplification in tumor cells with elevated c-Myc' (Lin et al., 2012). Here we report the results. We found overexpression of c-Myc increased total levels of RNA in P493-6 Burkitt's lymphoma cells; however, while the effect was in the same direction as the original study (Figure 3E; Lin et al., 2012), statistical significance and the size of the effect varied between the original study and the two different lots of serum tested in this replication. Digital gene expression analysis for a set of genes was also performed on P493-6 cells before and after c-Myc overexpression. Transcripts from genes that were active before c-Myc induction increased in expression following c-Myc overexpression, similar to the original study (Figure 3F; Lin et al., 2012). Transcripts from genes that were silent before c-Myc induction also increased in expression following c-Myc overexpression, while the original study concluded elevated c-Myc had no effect on silent genes (Figure 3F; Lin et al., 2012). Treating the data as paired, we found a statistically significant increase in gene expression for both active and silent genes upon c-Myc induction, with the change in gene expression greater for active genes compared to silent genes. Finally, we report meta-analyses for each result.

Replication Study: Systematic identification of genomic markers of drug sensitivity in cancer cells

In 2016, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Vanden Heuvel et al., 2016), that described how we intended to replicate selected experiments from the paper ‘Systematic identification of genomic markers of drug sensitivity in cancer cells’ (Garnett et al., 2012). Here we report the results. We found Ewing’s sarcoma cell lines, overall, were more sensitive to the PARP inhibitor olaparib than osteosarcoma cell lines; however, while the effect was in the same direction as the original study (Figure 4C; Garnett et al., 2012), it was not statistically significant. Further, mouse mesenchymal cells transformed with either the EWS-FLI1 or FUS-CHOP rearrangement displayed similar sensitivities to olaparib, whereas the Ewing’s sarcoma cell line SK-N-MC had increased olaparib sensitivity. In the original study, mouse mesenchymal cells transformed with the EWS-FLI1 rearrangement and SK-N-MC cells were found to have similar sensitivities to olaparib, whereas mesenchymal cells transformed with the FUS-CHOP rearrangement displayed a reduced sensitivity to olaparib (Figure 4E; Garnett et al., 2012). We also studied another Ewing’s sarcoma cell line, A673: A673 cells depleted of EWS-FLI1 or a negative control both displayed similar sensitivities to olaparib, whereas the original study reported a decreased sensitivity to olaparib when EWS-FLI1 was depleted (Figure 4F; Garnett et al., 2012). Differences between the original study and this replication attempt, such as the use of different sarcoma cell lines and level of knockdown efficiency, are factors that might have influenced the outcomes. Finally, where possible, we report meta-analyses for each result.

The readability of scientific texts is decreasing over time

Clarity and accuracy of reporting are fundamental to the scientific process. Readability formulas can estimate how difficult a text is to read. Here, in a corpus consisting of 709,577 abstracts published between 1881 and 2015 from 123 scientific journals, we show that the readability of science is steadily decreasing. Our analyses show that this trend is indicative of a growing use of general scientific jargon. These results are concerning for scientists and for the wider public, as they impact both the reproducibility and accessibility of research findings.

Replication Study: Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukaemia

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Fung et al., 2015), that described how we intended to replicate selected experiments from the paper "Inhibition of BET recruitment to chromatin as an effective treatment for MLL-fusion leukaemia" (Dawson et al., 2011). Here, we report the results of those experiments. We found treatment of MLL-fusion leukaemia cells (MV4;11 cell line) with the BET bromodomain inhibitor I-BET151 resulted in selective growth inhibition, whereas treatment of leukaemia cells harboring a different oncogenic driver (K-562 cell line) did not result in selective growth inhibition; this is similar to the findings reported in the original study (Figure 2A and Supplementary Figure 11A,B; Dawson et al., 2011). Further, I-BET151 resulted in a statistically significant decrease in BCL2 expression in MV4;11 cells, but not in K-562 cells; again this is similar to the findings reported in the original study (Figure 3D; Dawson et al., 2011). We did not find a statistically significant difference in survival when testing I-BET151 efficacy in a disseminated xenograft MLL mouse model, whereas the original study reported increased survival in I-BET151 treated mice compared to vehicle control (Figure 4B,D; Dawson et al., 2011). Differences between the original study and this replication attempt, such as different conditioning regimens and I-BET151 doses, are factors that might have influenced the outcome. We also found I-BET151 treatment resulted in a lower median disease burden compared to vehicle control in all tissues analyzed, similar to the example reported in the original study (Supplementary Figure 16A; Dawson et al., 2011). Finally, we report meta-analyses for each result.

Replication Study: The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate

In 2016, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Fiehn et al., 2016), that described how we intended to replicate selected experiments from the paper "The common feature of leukemia-associated IDH1 and IDH2 mutations is a neomorphic enzyme activity converting alpha-ketoglutarate to 2-hydroxyglutarate" (Ward et al., 2010). Here, we report the results of those experiments. We found that cells expressing R172K mutant IDH2 did not display isocitrate-dependent NADPH production above vector control levels, in contrast to the increased production observed with wild-type IDH2. Conversely, expression of R172K mutant IDH2 resulted in increased alpha-ketoglutarate-dependent consumption of NADPH compared to wild-type IDH2 or vector control. These results are similar to those reported in the original study (Figure 2; Ward et al., 2010). Further, expression of R172K mutant IDH2 resulted in increased 2HG levels within cells compared to the background levels observed in wild-type IDH2 and vector control, similar to the original study (Figure 3D; Ward et al., 2010). In primary human AML samples, the 2HG levels observed in samples with mutant IDH1 or IDH2 status were higher than those observed in samples without an IDH mutation, similar to what was observed in the original study (Figure 5C; Ward et al., 2010). Finally, we report meta-analyses for each result.

Replication Study: The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Chroscinski et al., 2015) that described how we intended to replicate selected experiments from the paper "The CD47-signal regulatory protein alpha (SIRPa) interaction is a therapeutic target for human solid tumors "(Willingham et al., 2012). Here we report the results of those experiments. We found that treatment of immune competent mice bearing orthotopic breast tumors with anti-mouse CD47 antibodies resulted in short-term anemia compared to controls, consistent with the previously described function of CD47 in normal phagocytosis of aging red blood cells and results reported in the original study (Table S4; Willingham et al., 2012). The weight of tumors after 30 days administration of anti-CD47 antibodies or IgG isotype control were not found to be statistically different, whereas the original study reported inhibition of tumor growth with anti-CD47 treatment (Figure 6A,B; Willingham et al., 2012). However, our efforts to replicate this experiment were confounded because spontaneous regression of tumors occurred in several of the mice. Additionally, the excised tumors were scored for inflammatory cell infiltrates. We found IgG and anti-CD47 treated tumors resulted in minimal to moderate lymphocytic infiltrate, while the original study observed sparse lymphocytic infiltrate in IgG-treated tumors and increased inflammatory cell infiltrates in anti-CD47 treated tumors (Figure 6C; Willingham et al., 2012). Furthermore, we observed neutrophilic infiltration was slightly increased in anti-CD47 treated tumors compared to IgG control. Finally, we report a meta-analysis of the result.

Making sense of replications

The first results from the Reproducibility Project: Cancer Biology suggest that there is scope for improving reproducibility in pre-clinical cancer research.

Replication Study: Discovery and preclinical validation of drug indications using compendia of public gene expression data

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper "Discovery and Preclinical Validation of Drug Indications Using Compendia of Public Gene Expression Data" (Sirota et al., 2011). Here we report the results of those experiments. We found that cimetidine treatment in a xenograft model using A549 lung adenocarcinoma cells resulted in decreased tumor volume compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 4C; Sirota et al., 2011), it was not statistically significant. Cimetidine treatment in a xenograft model using ACHN renal cell carcinoma cells did not differ from vehicle control treatment, similar to the original study (Supplemental Figure 1; Sirota et al., 2011). Doxorubicin treatment in a xenograft model using A549 lung adenocarcinoma cells did not result in a statistically significant difference compared to vehicle control despite tumor volume being reduced to levels similar to those reported in the original study (Figure 4C; Sirota et al., 2011). Finally, we report a random effects meta-analysis for each result. These meta-analyses show that the inhibition of A549 derived tumors by cimetidine resulted in a statistically significant effect, as did the inhibition of A549 derived tumors by doxorubicin. The effect of cimetidine on ACHN derived tumors was not statistically significant, as predicted.

Melanoma mystery

Biological variability has confounded efforts to confirm the role of PREX2 mutations in melanoma.

Small molecules remain on target for c-Myc

Targeting the transcription factor c-Myc via one of its coactivator proteins is a promising strategy for cancer therapy.

The challenges of replication

Interpreting the first results from the Reproducibility Project: Cancer Biology requires a highly nuanced approach.

Mixed outcomes for computational predictions

Experimental efforts to validate the output of a computational model that predicts new uses for existing drugs highlights the inherently complex nature of cancer biology.

Replication Study: BET bromodomain inhibition as a therapeutic strategy to target c-Myc

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper "BET bromodomain inhibition as a therapeutic strategy to target c-Myc" (Delmore et al., 2011). Here we report the results of those experiments. We found that treatment of human multiple myeloma (MM) cells with the small-molecular inhibitor of BET bromodomains, (+)-JQ1, selectively downregulated MYC transcription, which is similar to what was reported in the original study (Figure 3B; Delmore et al., 2011). Efficacy of (+)-JQ1 was evaluated in an orthotopically xenografted model of MM. Overall survival was increased in (+)-JQ1 treated mice compared to vehicle control, similar to the original study (Figure 7E; Delmore et al., 2011). Tumor burden, as determined by bioluminescence, was decreased in (+)-JQ1 treated mice compared to vehicle control; however, while the effect was in the same direction as the original study (Figure 7C-D; Delmore et al., 2011), it was not statistically significant. The opportunity to detect a statistically significant difference was limited though, due to the higher rate of early death in the control group, and increased overall survival in (+)-JQ1 treated mice before the pre-specified tumor burden analysis endpoint. Additionally, we evaluated the (−)-JQ1 enantiomer that is structurally incapable of inhibiting BET bromodomains, which resulted in a minimal impact on MYC transcription, but did not result in a statistically significant difference in tumor burden or survival distributions compared to treatment with (+)-JQ1. Finally, we report meta-analyses for each result.

DOI:http://dx.doi.org/10.7554/eLife.21253.001

Replication Study: Melanoma genome sequencing reveals frequent PREX2 mutations

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Chroscinski et al., 2014) that described how we intended to replicate selected experiments from the paper "Melanoma genome sequencing reveals frequent PREX2 mutations" (Berger et al., 2012). Here we report the results of those experiments. We regenerated cells stably expressing ectopic wild-type and mutant phosphatidylinositol-3,4,5-trisphosphate-dependent Rac exchange factor 2 (PREX2) using the same immortalized human NRAS^G12D melanocytes as the original study. Evaluation of PREX2 expression in these newly generated stable cells revealed varying levels of expression among the PREX2 isoforms, which was also observed in the stable cells made in the original study (Figure S6A; Berger et al., 2012). Additionally, ectopically expressed PREX2 was found to be at least 5 times above endogenous PREX2 expression. The monitoring of tumor formation of these stable cells in vivo resulted in no statistically significant difference in tumor-free survival driven by PREX2 variants, whereas the original study reported that these PREX2 mutations increased the rate of tumor incidence compared to controls (Figure 3B and S6B; Berger et al., 2012). Surprisingly, the median tumor-free survival was 1 week in this replication attempt, while 70% of the control mice were reported to be tumor-free after 9 weeks in the original study. The rapid tumor onset observed in this replication attempt, compared to the original study, makes the detection of accelerated tumor growth in PREX2 expressing NRAS^G12D melanocytes extremely difficult. Finally, we report meta-analyses for each result.

Replication Study: Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs

In 2015, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Kandela et al., 2015) that described how we intended to replicate selected experiments from the paper "Coadministration of a tumor-penetrating peptide enhances the efficacy of cancer drugs" (Sugahara et al., 2010). Here we report the results of those experiments. We found that coadministration with iRGD peptide did not have an impact on permeability of the chemotherapeutic agent doxorubicin (DOX) in a xenograft model of prostate cancer, whereas the original study reported that it increased the penetrance of this cancer drug (Figure 2B; Sugahara et al., 2010). Further, in mice bearing orthotopic 22Rv1 human prostate tumors, we did not find a statistically significant difference in tumor weight for mice treated with DOX and iRGD compared to DOX alone, whereas the original study reported a decrease in tumor weight when DOX was coadministered with iRGD (Figure 2C; Sugahara et al., 2010). In addition, we did not find a statistically significant difference in TUNEL staining in tumor tissue between mice treated with DOX and iRGD compared to DOX alone, while the original study reported an increase in TUNEL positive staining with iRGD coadministration (Figure 2D; Sugahara et al., 2010). Similar to the original study (Supplemental Figure 9A; Sugahara et al., 2010), we did not observe an impact on mouse body weight with DOX and iRGD treatment. Finally, we report meta-analyses for each result.

The natural selection of bad science

Poor research design and data analysis encourage false-positive findings. Such poor methods persist despite perennial calls for improvement, suggesting that they result from something more than just misunderstanding. The persistence of poor methods results partly from incentives that favour them, leading to the natural selection of bad science. This dynamic requires no conscious strategizing-no deliberate cheating nor loafing-by scientists, only that publication is a principal factor for career advancement. Some normative methods of analysis have almost certainly been selected to further publication instead of discovery. In order to improve the culture of science, a shift must be made away from correcting misunderstandings and towards rewarding understanding. We support this argument with empirical evidence and computational modelling. We first present a 60-year meta-analysis of statistical power in the behavioural sciences and show that power has not improved despite repeated demonstrations of the necessity of increasing power. To demonstrate the logical consequences of structural incentives, we then present a dynamic model of scientific communities in which competing laboratories investigate novel or previously published hypotheses using culturally transmitted research methods. As in the real world, successful labs produce more 'progeny,' such that their methods are more often copied and their students are more likely to start labs of their own. Selection for high output leads to poorer methods and increasingly high false discovery rates. We additionally show that replication slows but does not stop the process of methodological deterioration. Improving the quality of research requires change at the institutional level.

Revolutions in Neuroscience: Tool Development

Thomas Kuhn's famous model of the components and dynamics of scientific revolutions is still dominant to this day across science, philosophy, and history. The guiding philosophical theme of this article is that, concerning actual revolutions in neuroscience over the past 60 years, Kuhn's account is wrong. There have been revolutions, and new ones are brewing, but they do not turn on competing paradigms, anomalies, or the like. Instead, they turn exclusively on the development of new experimental tools. I adopt a metascientific approach and examine in detail the development of two recent neuroscience revolutions: the impact of engineered genetically mutated mammals in the search for causal mechanisms of "higher" cognitive functions; and the more recent impact of optogenetics and designer receptors exclusively activated by designer drugs (DREADDs). The two key metascientific concepts, I derive from these case studies are a revolutionary new tool's motivating problem, and its initial and second-phase hook experiments. These concepts hardly exhaust a detailed metascience of tool development experiments in neuroscience, but they get that project off to a useful start and distinguish the subsequent account of neuroscience revolutions clearly from Kuhn's famous model. I close with a brief remark about the general importance of molecular biology for a current philosophical understanding of science, as comparable to the place physics occupied when Kuhn formulated his famous theory of scientific revolutions.