How I Learned to Stop Worrying and Love Replication Failures

Tactics of just, equitable, diverse, and inclusive scientific research

The principles of social justice, equity, diversity, inclusion (JEDI) have received increasing attention in behavior analysis circles, but the conversation has largely centered on implications for applied behavior analysis practice and research. It may be less clear to researchers who conduct basic and translational research how JEDI principles can inform and inspire their work. This article synthesizes publications from behavior analysis and other scientific fields about tactics of JEDI-informed research. We organized this scholarship across five stages of research from developing the research question to sharing findings and curated sources for an audience of behavioral science researchers. We discuss reflexive practice, representation, belongingness, participatory research, quantitative critical theory, and open science, among other topics. Some researchers may have already adopted some of the practices outlined, some may begin new practices, and some may choose to conduct experimental analyses of JEDI problems. Our hope is that those actions will be reinforced by the behavior analysis scientific community. We conclude by encouraging the leadership of this journal to continue to work toward the structural changes necessary to make the experimental analysis of behavior just, equitable, diverse, and inclusive.

The BEHAVE application as a tool to monitor inclusive interventions for subjects with neurodevelopmental disorders

In the last few years, many educational and therapeutic interventions for young people with neurodevelopmental disorders are based on systematic monitoring of the outcomes. These interventions are typically conducted using single-case experimental designs, (SCEDs) a set of methods aimed at testing the effect of an intervention on a single subject or a small number of subjects. In SCEDs, an effective process of decision-making needs accurate, precise, and reliable data but also that caregivers and health professionals can gather information with minimal effort. The use of Information Communication Technologies in SCEDs can support the process of data collection and analysis, facilitating the collection of accurate and reliable data, providing reports accessible also by non-experts, and promoting interactions and sharing among clinicians, educators, and caregivers. The present paper introduces the BEHAVE application, a web-based highly customizable application, designed to implement SCEDs, supporting both data collection and automatic analysis of the datasets. Moreover, the paper will describe two case studies of kindergarten children with neurodevelopmental disorders, highlighting how the BEHAVE application supported the entire process, from data collection in multiple contexts to decision-making based on the analysis provided by the system. In particular, the paper describes the case studies of Carlo and Dario, two children with severe language and communication impairments, and the inclusive education interventions carried out to maximize their participation in a typical home and school setting increasing their mand repertoire. Results revealed an increase in the mand repertoire in both children who become able to generalize the outcomes to multiple life contexts. The active participation of the caregivers played a crucial role in the ability of children to use the learned skills in settings different from the ones they were learned in.

Applied Behavior Analysis in Children and Youth with Autism Spectrum Disorders: A Scoping Review

This manuscript provides a comprehensive overview of the impact of applied behavior analysis (ABA) on children and youth with autism spectrum disorders (ASD). Seven online databases and identified systematic reviews were searched for published, peer-reviewed, English-language studies examining the impact of ABA on health outcomes. Measured outcomes were classified into eight categories: cognitive, language, social/communication, problem behavior, adaptive behavior, emotional, autism symptoms, and quality of life (QoL) outcomes. Improvements were observed across seven of the eight outcome measures. There were no included studies that measured subject QoL. Moreover, of 770 included study records, only 32 (4%) assessed ABA impact, had a comparison to a control or other intervention, and did not rely on mastery of specific skills to mark improvement. Results reinforce the need for large-scale prospective studies that compare ABA with other non-ABA interventions and include measurements of subject QoL to provide policy makers with valuable information on the impacts of ABA and other existing and emerging interventions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40614-022-00338-x.

Reducing impulsive choice: VIII. Effects of delay-exposure training in female rats

Impulsive choice may play an important role in serious health-related decisions, like addiction tendencies. Thus, there is merit in exploring interventions that reduce impulsive choice. Delay-exposure training involves extended experience with delayed reinforcement. Following training, delay-exposed rats make fewer impulsive choices than control rats. The reducing effects of delay exposure training on impulsive choice have been replicated in male rats seven times. For the first time, this study evaluated the effects of delay exposure training in female rats. Thirty-six rats were randomly assigned to either delay-exposure or immediacy-exposure training. Then, rats underwent two impulsive choice assessments in which they chose between one immediate pellet or three delayed pellets. In the first assessment, delays increased within-sessions, across trial blocks from 0, 8, 16, to 32 s. In the second assessment, delays to the larger reward increased between-sessions, from 8, 16, 32, to 4 s. Unlike findings with male rats, delay-exposure training produced a reduction in impulsive choice only in the initial five sessions in female rats. Possible reasons for the lack of lasting effect in female rats are discussed and future research directions are identified.

On the Efficacy of and Preference for Signaling Extinction in a Multiple Schedule

Previous basic research has shown that signaling the extinction component of a compound schedule can be aversive and nonpreferred. However, such discriminative stimuli are common when thinning schedules of reinforcement in practice, and they provide several advantages to clinicians. A limitation of previous applied studies on different arrangements of discriminative stimuli is that researchers have used identical stimuli to signal the availability of reinforcement across conditions that do and do not signal extinction, often doubling exposure to the stimulus signaling the availability of reinforcement. The present experiments corrected this limitation by comparing multiple-schedule arrangements that do and do not signal extinction when unique stimuli signal each component across conditions. Results from three participants indicated that both multiple-schedule arrangements were similarly efficacious when teaching the successive discrimination. However, response patterns differed when testing under a concurrent-operants arrangement, suggesting different patterns of preference across various multiple-schedule arrangements.

The Evolution of Behavior Analysis: Toward a Replication Crisis?

The Open Science Collaboration (Science, 349(6251), 1-8, 2015) produced a massive failure to replicate previous research in psychology-what has been called a "replication crisis in psychology." An important question for behavior scientists is: To what extent is behavior science vulnerable to this type of massive replication failure? That question is addressed by considering the features of a traditional approach to behavior science. Behavior science in its infancy was a natural science, inductive, within-subject approach that encouraged both direct and systematic replication. Each of these features of behavior science increased its resistance to three factors identified as responsible for the alleged replication crisis: (1) failures to replicate procedures, (2) low-power designs, and (3) publication bias toward positive results. As behavior science has evolved, the features of the traditional approach have become less ubiquitous. And if the science continues to evolve as it has, it will likely become more vulnerable to a massive replication failure like that reported by the Open Science Collaboration (Science, 349(6251), 1-8, 2015).

Historically recontextualizing Sidman's Tactics: How behavior analysis avoided psychology's methodological Ouroboros

Psychology is undergoing major cultural changes methodologically, with efforts to redefine how psychologists analyze and report their data. Davidson (2018) argued that psychology's methodological crises stem from mechanical objectivity involving the adoption of an analytic tool as source of dependable knowledge. This has led to institutionalization, and eventually uncritical ritualistic use, such as happened with null hypothesis statistical testing. Davidson invoked the mythological symbol of the Ouroboros to represent the endless churning of statistical fads. Sidman (1960), in his Tactics of Scientific Research provided a shield from these problems in terms of the premium he placed on the experience, expertise, judgement, and decision-making of the scientist, that appear to be absent in psychology's ritualized processes.

Cooperative responding in rats: II. Performance on fixed-ratio schedules of mutual reinforcement

Coordinated responses of 5 dyads of rats were investigated under fixed-ratio (FR) schedules of mutual water reinforcement. Coordinated responding was defined as 2 consecutive lever-presses, 1 from each of 2 rats, occurring <.5 s apart. In the FR schedules, each coordinated episode was defined as 1 response in the FR sequence. The size of FR schedules was parametrically manipulated assuming the values of FR 1, 6, 12, 18, 24, 30, 50, and 9, in this order. Each FR remained in effect until responding reached stability. Under all conditions, pairs of rats received access to water simultaneously (mutual reinforcement). Rates and proportions of coordinated responding showed a bitonic inverted U-shaped function of ratio size. Postreinforcement pauses increased systematically as the interreinforcement interval increased. Local rates and proportions increased as a function of response location within ratios. Results of a control condition with relaxed temporal constraints for mutual reinforcement showed decreases in rates and proportion of coordinated responses, suggesting that the coordinated responses were controlled by the mutual reinforcement contingencies. The present experiment showed that coordinated responding is quantitatively affected by 3 properties of FR schedules: response requirement, reinforcement rates, and proximity to reinforcement.

Cereal Deal: How the Physical Appearance of Others Affects Attention to Healthy Foods

This eye-tracking study investigated whether the physical appearance of another consumer can influence people's visual attention and choice behavior in a grocery shopping context. Participants (N = 96) took part in a lab-based experiment and watched a brief video recording featuring a female consumer standing in front of a supermarket shelf. The appearance and body type of the consumer was manipulated between conditions, such that she was perceived as 1) healthy and of normal weight, 2) unhealthy by means of overweight, or 3) unhealthy through visual signs associated with a potentially unhealthy lifestyle, but not by means of overweight. Next, participants were exposed to a supermarket shelf with cereals and were asked to choose one alternative they could consider buying. Prior exposure to a seemingly unhealthy (vs. healthy) consumer resulted in a relative increase in participants' visual attention towards products perceived to be healthy (vs. unhealthy), which prompted cereal choices deemed to be healthier. This effect was stronger for products that holistically, through their design features, managed to convey the impression that they are healthy rather than products with explicit cues linked to healthiness (i.e., the keyhole label). These results offer important implications regarding packaging design for marketers, brand owners, and policy makers. Moreover, the findings highlight the value of technological tools, such as eye-tracking methodology, for capturing consumers' entire decision-making processes instead of focusing solely on outcome-based metrics, such as choice data or purchase behavior.

Search and Selection Procedures of Literature Reviews in Behavior Analysis

Literature reviews allow professionals to identify effective interventions and assess developments in research and practice. As in other forms of scientific inquiry, the transparency of literature searches enhances the credibility of findings, particularly in regards to intervention research. The current review evaluated the characteristics of search methods employed in literature reviews appearing in publications concerning behavior analysis (n = 28) from 1997 to 2017. Specific aims included determining the frequency of narrative, systematic, and meta-analytic reviews over time; examining the publication of reviews in specific journals; and evaluating author reports of literature search and selection procedures. Narrative reviews (51.30%; n = 630) represented the majority of the total sample (n = 1,228), followed by systematic (31.51%; n = 387) and meta-analytic (17.18%; n = 211) reviews. In contrast to trends in related fields (e.g., special education), narrative reviews continued to represent a large portion of published reviews each year. The evaluated reviews exhibited multiple strengths; nonetheless, issues involving the reporting and execution of searches may limit the validity and replicability of literature reviews. A discussion of implications for research follows an overview of findings.

An Information Theoretic Approach to Model Selection: A Tutorial with Monte Carlo Confirmation

A reliance on null hypothesis significance testing (NHST) and misinterpretations of its results are thought to contribute to the replication crisis while impeding the development of a cumulative science. One solution is a data-analytic approach called Information-Theoretic (I-T) Model Selection, which builds upon Maximum Likelihood estimates. In the I-T approach, the scientist examines a set of candidate models and determines for each one the probability that it is the closer to the truth than all others in the set. Although the theoretical development is subtle, the implementation of I-T analysis is straightforward. Models are sorted according to the probability that they are the best in light of the data collected. It encourages the examination of multiple models, something investigators desire and that NHST discourages. This article is structured to address two objectives. The first is to illustrate the application of I-T data analysis to data from a virtual experiment. A noisy delay-discounting data set is generated and seven quantitative models are examined. In the illustration, it is demonstrated that it is not necessary to know the "truth" is to identify the one that is closest to it and that the most likely models conform to the model that generated the data. Second, we examine claims made by advocates of the I-T approach using Monte Carlo simulations in which 10,000 different data sets are generated and analyzed. The simulations showed that 1) the probabilities associated with each model returned by the single virtual experiment approximated those that resulted from the simulations, 2) models that were deemed close to the truth produced the most precise parameter estimates, and 3) adding a single replicate sharpens the ability to identify the most probable model.

Replication Research, Publication Bias, and Applied Behavior Analysis

The "replication crisis" describes recent difficulties in replicating studies in various scientific fields, most notably psychology. The available evidence primarily documents replication failures for group research designs. However, we argue that contingencies of publication bias that led to the "replication crisis" also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect. The resulting research literature may unjustifiably inflate confidence about intervention effects, limit researchers' ability to delineate intervention boundary conditions, and diminish the credibility of our science. To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research.

Improving Psychological Science through Transparency and Openness: An Overview

The ability to independently verify and replicate observations made by other researchers is a hallmark of science. In this article, we provide an overview of recent discussions concerning replicability and best practices in mainstream psychology with an emphasis on the practical benefists to both researchers and the field as a whole. We first review challenges individual researchers face in producing research that is both publishable and reliable. We then suggest methods for producing more accurate research claims, such as transparently disclosing how results were obtained and analyzed, preregistering analysis plans, and publicly posting original data and materials. We also discuss ongoing changes at the institutional level to incentivize stronger research. These include officially recognizing open science practices at the journal level, disconnecting the publication decision from the results of a study, training students to conduct replications, and publishing replications. We conclude that these open science practices afford exciting low-cost opportunities to improve the quality of psychological science.

An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis

For over a decade, the failure to reproduce findings in several disciplines, including the biomedical, behavioral, and social sciences, have led some authors to claim that there is a so-called "replication (or reproducibility) crisis" in those disciplines. The current article examines: (a) various aspects of the reproducibility of scientific studies, including definitions of reproducibility; (b) published concerns about reproducibility in the scientific literature and public press; (c) variables involved in assessing the success of attempts to reproduce a study; (d) suggested factors responsible for reproducibility failures; (e) types of validity of experimental studies and threats to validity as they relate to reproducibility; and (f) evidence for threats to reproducibility in the behavior science/analysis literature. Suggestions for improving the reproducibility of studies in behavior science and analysis are described throughout.

Predict, Control, and Replicate to Understand: How Statistics Can Foster the Fundamental Goals of Science

Scientists abstract hypotheses from observations of the world, which they then deploy to test their reliability. The best way to test reliability is to predict an effect before it occurs. If we can manipulate the independent variables (the efficient causes) that make it occur, then ability to predict makes it possible to control. Such control helps to isolate the relevant variables. Control also refers to a comparison condition, conducted to see what would have happened if we had not deployed the key ingredient of the hypothesis: scientific knowledge only accrues when we compare what happens in one condition against what happens in another. When the results of such comparisons are not definitive, metrics of the degree of efficacy of the manipulation are required. Many of those derive from statistical inference, and many of those poorly serve the purpose of the cumulation of knowledge. Without ability to replicate an effect, the utility of the principle used to predict or control is dubious. Traditional models of statistical inference are weak guides to replicability and utility of results. Several alternatives to null hypothesis testing are sketched: Bayesian, model comparison, and predictive inference (p rep). Predictive inference shows, for example, that the failure to replicate most results in the Open Science Project was predictable. Replicability is but one aspect of scientific understanding: it establishes the reliability of our data and the predictive ability of our formal models. It is a necessary aspect of scientific progress, even if not by itself sufficient for understanding.

The Meta-Science of Adult Statistical Word Segmentation: Part 1

We report the first set of results in a multi-year project to assess the robustness – and the factors promoting robustness – of the adult statistical word segmentation literature. This includes eight total experiments replicating six different experiments. The purpose of these replications is to assess the reproducibility of reported experiments, examine the replicability of their results, and provide more accurate effect size estimates. Reproducibility was mixed, with several papers either lacking crucial details or containing errors in the description of method, making it difficult to ascertain what was done. Replicability was also mixed: although in every instance we confirmed above-chance statistical word segmentation, many theoretically important moderations of that learning failed to replicate. Moreover, learning success was generally much lower than in the original studies. In the General Discussion, we consider whether these differences are due to differences in subject populations, low power in the original studies, or some combination of these and other factors. We also consider whether these findings are likely to generalize to the broader statistical word segmentation literature.