Using the dual-criteria methods to supplement visual inspection: An analysis of nonsimulated data

Does the choice of a linear trend-assessment technique matter in the context of single-case data?

Trend is one of the data aspects that is an object of assessment in the context of single-case experimental designs. This assessment can be performed both visually and quantitatively. Given that trend, just like other relevant data features such as level, immediacy, or overlap does not have a single operative definition, a comparison among the existing alternatives is necessary. Previous studies have included illustrations of differences between trend-line fitting techniques using real data. In the current study, I carry out a simulation to study the degree to which different trend-line fitting techniques lead to different degrees of bias, mean square error, and statistical power for a variety of quantifications that entail trend lines. The simulation involves generating both continuous and count data, for several phase lengths, degrees of autocorrelation, and effect sizes (change in level and change in slope). The results suggest that, in general, ordinary least squares estimation performs well in terms of relative bias and mean square error. Especially, a quantification of slope change is associated with better statistical results than quantifying an average difference between conditions on the basis of a projected baseline trend. In contrast, the performance of the split-middle (bisplit) technique is less than optimal.

Transition States in Single Case Experimental Designs

The continuation of a baseline pattern of responding into a treatment phase, sometimes referred to as a "transition state," can obscure interpretation of data depicted in single-case experimental designs (SCEDs). For example, when using visual analysis, transition states may lead to the conclusion that the treatment is ineffective. Likewise, the inclusion of overlapping data points in some statistical analyses may lead to conclusions that the treatment had a small effect size and give rise to publication bias. This study reviewed 20 volumes in a journal that publishes primarily SCEDs studies. We defined a transition state as a situation wherein at least the first three consecutive data points of a treatment phase or condition are within the range of the baseline phase or condition. Results indicate that transitions states (a) were present for 7.4% of graphs that met inclusion criteria and (b) occurred for a mean of 4.9 data points before leading to behavior change. We discuss some implications and directions for future research on transition states.

Single-case intervention research design standards: Additional proposed upgrades and future directions

Single-case intervention research design standards have evolved considerably over the past decade. These standards serve the dual role of assisting in single-case design (SCD) intervention research methodology and as guidelines for literature syntheses within a particular research domain. In a recent article (Kratochwill et al., 2021), we argued for a need to clarify key features of these standards. In this article we offer additional recommendations for SCD research and synthesis standards that have been either underdeveloped or missing in the conduct of research and in literature syntheses. Our recommendations are organized into three categories: expanding design standards, expanding evidence standards, and expanding the applications and consistency of SCDs. The recommendations we advance are for consideration for future standards, research design training, and they are especially important to guide the reporting of SCD intervention investigations as they enter the literature-synthesis phase of evidence-based practice initiatives.

Waiting for baseline stability in single-case designs: Is it worth the time and effort?

Researchers and practitioners often use single-case designs (SCDs), or n-of-1 trials, to develop and validate novel treatments. Standards and guidelines have been published to provide guidance as to how to implement SCDs, but many of their recommendations are not derived from the research literature. For example, one of these recommendations suggests that researchers and practitioners should wait for baseline stability prior to introducing an independent variable. However, this recommendation is not strongly supported by empirical evidence. To address this issue, we used Monte Carlo simulations to generate graphs with fixed, response-guided, and random baseline lengths while manipulating trend and variability. Then, our analyses compared the type I error rate and power produced by two methods of analysis: the conservative dual-criteria method (a structured visual aid) and a support vector classifier (a model derived from machine learning). The conservative dual-criteria method produced fewer errors when using response-guided decision-making (i.e., waiting for stability) and random baseline lengths. In contrast, waiting for stability did not reduce decision-making errors with the support vector classifier. Our findings question the necessity of waiting for baseline stability when using SCDs with machine learning, but the study must be replicated with other designs and graph parameters that change over time to support our results.

Agreement between visual inspection and objective analysis methods: A replication and extension

Behavior analysts typically rely on visual inspection of single‐case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual‐criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the topic, we examined agreement between the visual inspection of five raters, the conservative dual‐criteria method, and a machine‐learning algorithm (i.e., the support vector classifier) on 198 AB graphs extracted from clinical data. The results indicated that average agreement between the 3 methods was generally consistent. Mean interrater agreement was 84%, whereas raters agreed with the conservative dual‐criteria method and the support vector classifier on 84% and 85% of graphs, respectively. Our results indicate that both objective methods produce results consistent with visual inspection, which may support their future use.

Further Analysis of Advanced Quantitative Methods and Supplemental Interpretative Aids with Single-Case Experimental Designs

Reliable and accurate visual analysis of graphically depicted behavioral data acquired using single-case experimental designs (SCEDs) is integral to behavior-analytic research and practice. Researchers have developed a range of techniques to increase reliable and objective visual inspection of SCED data including visual interpretive guides, statistical techniques, and nonstatistical quantitative methods to objectify the visual-analytic interpretation of data to guide clinicians, and ensure a replicable data interpretation process in research. These structured data analytic practices are now more frequently used by behavior analysts and the subject of considerable research within the field of quantitative methods and behavior analysis. First, there are contemporaneous analytic methods that have preliminary support with simulated datasets, but have not been thoroughly examined with nonsimulated clinical datasets. There are a number of relatively new techniques that have preliminary support (e.g., fail-safe k), but require additional research. Other analytic methods (e.g., dual-criteria and conservative dual criteria) have more extensive support, but have infrequently been compared against other analytic methods. Across three studies, we examine how these methods corresponded to clinical outcomes (and one another) for the purpose of replicating and extending extant literature in this area. Implications and recommendations for practitioners and researchers are discussed.

Structured visual analysis of single-case experimental design data: Developments and technological advancements

Visual analysis is the primary method used to interpret single-case experimental design (SCED) data in applied behavior analysis. Research shows that agreement between visual analysts can be suboptimal at times. To address the inconsistent interpretations of SCED data, recent structured visual-analysis technological advancements have been developed. To assess the extent to which structured visual analysis is used to guide or supplement applied behavior analysts' interpretation of SCED graphs, a systematic review between the years of 2015 to 2020 in the Journal of Applied Behavior Analysis was conducted. Findings showed that despite recent efforts to develop structured visual-analysis tools and criteria, these methods are rarely used to analyze SCED data. An overview of structured visual-analysis tools is shared, their utility is delineated, common characteristics are brought to light, and future directions for both research and their clinical use are highlighted.

Machine learning to analyze single-case graphs: A comparison to visual inspection

Behavior analysts commonly use visual inspection to analyze single‐case graphs, but studies on its reliability have produced mixed results. To examine this issue, we compared the Type I error rate and power of visual inspection with a novel approach—machine learning. Five expert visual raters analyzed 1,024 simulated AB graphs, which differed on number of points per phase, autocorrelation, trend, variability, and effect size. The ratings were compared to those obtained by the conservative dual‐criteria method and two models derived from machine learning. On average, visual raters agreed with each other on only 75% of graphs. In contrast, both models derived from machine learning showed the best balance between Type I error rate and power while producing more consistent results across different graph characteristics. The results suggest that machine learning may support researchers and practitioners in making fewer errors when analyzing single‐case graphs, but replications remain necessary.

Brief Report: Mobile Technology to Support Parents in Reducing Stereotypy

Although behavioral interventions have been known to effectively reduce stereotypy in children with ASD, these types of interventions are not accessible to all families. In response to this issue, we evaluated the effects of the iSTIM, an iOS application designed to support parents in the reduction of stereotypy in their child with ASD. We used a series of AB designs to determine the effectiveness of the iSTIM on stereotypy using parents as behavior change agents. The use of iSTIM by the parents led to a reduction in stereotypy for six of seven participants. Our results suggest that the use of technology may be a cost effective and easily accessible method for parents to reduce stereotypy in their child with ASD.

Application of automated nonparametric statistical analysis in clinical contexts

Functional analyses (FAs) provide clinicians with results upon which they design behavioral treatments. Unfortunately, interrater reliability of visual analysis of FA results can be inconsistent. Accordingly, researchers have designed quantitative metrics and visual aids to supplement visual analysis. Recently, Hall et al. (2020) provided a proof of concept for using automated nonparametric statistical analysis (ANSA) to interpret FA data. Their results show promise for ANSA as a supplemental tool. However, they evaluated ANSA with only published FA datasets, which may not be representative of FAs commonly encountered in clinical care. Therefore, the purpose of this replication was to compare ANSA to another validated supplemental aid (i.e., the structured criteria method) and investigate its utility with unpublished clinical FA data. Our results were consistent with Hall et al.'s, indicating ANSA may augment clinical interpretation of FA data. Recommendations for clinical applications of ANSA and future directions for researchers are discussed.

How Many Tiers Do We Need? Type I Errors and Power in Multiple Baseline Designs

Design quality guidelines typically recommend that multiple baseline designs include at least three demonstrations of effects. Despite its widespread adoption, this recommendation does not appear grounded in empirical evidence. The main purpose of our study was to address this issue by assessing Type I error rate and power in multiple baseline designs. First, we generated 10,000 multiple baseline graphs, applied the dual-criteria method to each tier, and computed Type I error rate and power for different number of tiers showing a clear change. Second, two raters categorized the tiers for 300 multiple baseline graphs to replicate our analyses using visual inspection. When multiple baseline designs had at least three tiers and two or more of these tiers showed a clear change, the Type I error rate remained adequate (< .05) while power also reached acceptable levels (> .80). In contrast, requiring all tiers to show a clear change resulted in overly stringent conclusions (i.e., unacceptably low power). Therefore, our results suggest that researchers and practitioners should carefully consider limitations in power when requiring all tiers of a multiple baseline design to show a clear change in their analyses.

Correspondence between Fail-Safe k and Dual-Criteria Methods: Analysis of Data Series Stability

Barnard-Brak, Richman, Little, and Yang (Behaviour Research and Therapy, 102, 8-15, 2018) developed a structured-criteria metric, fail-safe k, which quantifies the stability of data series within single-case experimental designs (SCEDs) using published baseline and treatment data. Fail-safe k suggests the optimal point in time to change phases (e.g., move from Phase B to Phase C, reverse back to Phase A). However, this tool has not been tested with clinical data obtained in the course of care. Thus, the purpose of the current study was to replicate the procedures described by Barnard-Brak et al. with clinical data. We also evaluated the correspondence between the fail-safe k metric with outcomes obtained via dual-criteria and conservative-dual criteria methods, which are empirically supported methods for evaluating data-series trends within SCEDs. Our results provide some degree of support for use of this approach as a research tool with clinical data, in particular when evaluating small or medium treatment effect sizes, but further research is needed before this can be used widely by practitioners.

Evaluating sources of baseline data using dual-criteria and conservative dual-criteria methods: A quantitative analysis

Scheithauer et al. (2020) recently demonstrated that differences in the source of baseline data extracted from a functional analysis (FA) may not affect subsequent clinical decision-making in comparison to a standard baseline. These outcomes warrant additional quantitative examination, as correspondence of visual analysis has sometimes been reported to be unreliable. In the current study, we quantified the occurrence of false positives within a dataset of FA and baseline data using the dual-criteria (DC) and conservative dual-criteria (CDC) methods. Results of this quantitative analysis suggest that false positives were more likely when using FA data (rather than original baseline data) as the initial treatment baseline. However, both sources of baseline data may have acceptably low levels of false positives for practical use. Overall, the findings provide preliminary quantitative support for the conclusion that determinations of effective treatment may be easily obtained using different sources of baseline data.

Machine Learning to Analyze Single-Case Data: A Proof of Concept

Visual analysis is the most commonly used method for interpreting data from single-case designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach.

Using Single-Case Designs in Practical Settings: Is Within-Subject Replication Always Necessary?

Behavior analysts have widely adopted and embraced within-subject replication through the use of reversal and multielement designs. However, the withdrawal of treatment, which is central to these designs, may not be desirable, feasible, or even ethical in practical settings. To examine this issue, we extracted 501 ABAB graphs from theses and dissertations to examine to what extent we would have reached correct or incorrect conclusions if we had based our analysis on the initial AB component only. In our first experiment, we examined the proportion of datasets for which the results of the first AB component matched the results of the subsequent phase reversals. In our second experiment, we calculated three effect size estimates for the same datasets to examine whether these measures could predict the relevance of conducting a within-subject replication. Our analyses indicated that the initial effects were successfully replicated at least once in approximately 85% of the cases and that effect size may predict the probability of within-subject replication. Overall, our results support the rather controversial proposition that it may be possible to set threshold values of effect size above which conducting a replication could be considered unnecessary. That said, more research is needed to confirm and examine the generalizability of these results prior to recommending changes in practice.

An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis

For over a decade, the failure to reproduce findings in several disciplines, including the biomedical, behavioral, and social sciences, have led some authors to claim that there is a so-called "replication (or reproducibility) crisis" in those disciplines. The current article examines: (a) various aspects of the reproducibility of scientific studies, including definitions of reproducibility; (b) published concerns about reproducibility in the scientific literature and public press; (c) variables involved in assessing the success of attempts to reproduce a study; (d) suggested factors responsible for reproducibility failures; (e) types of validity of experimental studies and threats to validity as they relate to reproducibility; and (f) evidence for threats to reproducibility in the behavior science/analysis literature. Suggestions for improving the reproducibility of studies in behavior science and analysis are described throughout.

Group Procedures for Decreasing Problem Behavior Displayed by Detained Adolescents

As one component of providing treatment in a residential facility, Brogan, Falligant, and Rapp decreased problem behavior by two groups of detained adolescents using group contingency procedures. The current series of studies evaluated the extent to which group procedures could be extended to other contexts within a residential facility. In Study 1, fixed-time delivery of attention by dormitory staff decreased problem behavior displayed by a group of five to 11 detained adolescents during free periods. In Study 2, rules from a therapist plus contingencies for following those rules increased appropriate line walking during specific transition periods. Subsequently, rules alone maintained appropriate line walking, however, direct training was required to produce appropriate line walking during other transitions. Measures of social validity indicated the procedures and outcomes in both studies were acceptable to facility personnel.

Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide

Single-case experimental designs (SCED) are experimental designs aiming at testing the effect of an intervention using a small number of patients (typically one to three), using repeated measurements, sequential (±randomized) introduction of an intervention and method-specific data analysis, including visual analysis and specific statistics. The aim of this paper is to familiarise professionals working in different fields of rehabilitation with SCEDs and provide practical advice on how to design and implement a SCED in clinical rehabilitation practice. Research questions suitable for SCEDs and the different types of SCEDs (e.g., alternating treatment designs, introduction/withdrawal designs and multiple baseline designs) are reviewed. Practical steps in preparing a SCED design are outlined. Examples from different rehabilitation domains are provided throughout the paper. Challenging issues such as the choice of the repeated measure, assessment of generalisation, randomization, procedural fidelity, replication and generalizability of findings are discussed. Simple rules and resources for data analysis are presented. The utility of SCEDs in physical and rehabilitation medicine (PRM) are discussed.

Using a Visual Structured Criterion for the Analysis of Alternating-Treatment Designs

Although visual inspection remains common in the analysis of single-case designs, the lack of agreement between raters is an issue that may seriously compromise its validity. Thus, the purpose of our study was to develop and examine the properties of a simple structured criterion to supplement the visual analysis of alternating-treatment designs. To this end, we generated simulated data sets with varying number of points, number of conditions, effect sizes, and autocorrelations, and then measured Type I error rates and power produced by the visual structured criterion (VSC) and permutation analyses. We also validated the results for Type I error rates using nonsimulated data. Overall, our results indicate that using the VSC as a supplement for the analysis of systematically alternating-treatment designs with at least five points per condition generally provides adequate control over Type I error rates and sufficient power to detect most behavior changes.

Linear Trend in Single-Case Visual and Quantitative Analyses

The frequently used visual analysis of single-case data focuses on data aspects such as level, trend, variability, overlap, immediacy of effect, and consistency of data patterns; most of these aspects are also commonly quantified besides inspecting them visually. The present text focuses on trend, because even linear trend can be operatively defined in several different ways, while there are also different approaches for controlling for baseline trend. We recommend using a quantitative criterion for choosing a trend line fitting technique and comparing baseline and intervention slopes, instead of detrending. We implement our proposal in a free web-based application created specifically for following the What Works Clearinghouse Standards recommendations for visual analysis. This application is especially destined to applied researchers and provides graphical representation of the data, visual aids, and quantifications of the difference between phases in terms of level, trend, and overlap, as well as two quantifications of the immediate effect. An evaluation of the consistency of effects across replications of the AB sequence is also provided. For methodologists and statisticians, we include formulas and examples of the different straight line fitting and detrending techniques to improve the reproducibility of results and simulations.