Type I error rates and power of two randomization test procedures for the changing criterion design

Single-case experimental design (SCED) data can be analyzed following different approaches. One of the first historically proposed options is randomizations tests, benefiting from the inclusion of randomization in the design: a desirable methodological feature. Randomization tests have become more feasible with the availability of computational resources, and such tests have been proposed for all major types of SCEDs: multiple-baseline, reversal/withdrawal, alternating treatments, and changing criterion designs. The focus of the current text is on the last of these, given that they have not been the subject of any previous simulation study. Specifically, we estimate type I error rates and statistical power for two different randomization procedures applicable to changing criterion designs: the phase change moment randomization and the blocked alternating criterion randomization. We include different series lengths, number of phases, levels of autocorrelation, and random variability. The results suggest that type I error rates are generally controlled and that sufficient power can be achieved with as few as 28-30 measurements for independent data, although more measurements are needed in case of positive autocorrelation. The presence of a reversal to a previous criterion level is beneficial. R code is provided for carrying out randomization tests following the two randomization procedures.

How to assess and take into account trend in single-case experimental design data

One of the data features that are expected to be assessed when analyzing single-case experimental designs (SCED) data is trend. The current text deals with four different questions that applied researchers can ask themselves when assessing trend and especially when dealing with improving baseline trend: (a) What options exist for assessing the presence of trend?; (b) Once assessed, what criterion can be followed for deciding whether it is necessary to control for baseline trend?; (c) What strategy can be followed for controlling for baseline trend?; and (d) How to proceed in case there is baseline trend only in some A-B comparisons? Several options are reviewed for each of these questions in the context of real data, and tentative recommendations are provided. A new user-friendly website is developed to implement the options for fitting a trend line and a criterion for selecting a specific technique for that purpose. Trend-related and more general data analytical recommendations are provided for applied researchers.Trial registration: ClinicalTrials.gov identifier: NCT04560777.

The Permutation Distancing Test for dependent single-case observational AB-phase design data: A Monte Carlo simulation study

The Permutation Distancing Test (PDT) is a nonparametric test for evaluating treatment effects in dependent single-case observational design (SCOD) AB-phase data without linear trends. Monte Carlo methods were used to estimate the PDT power and type I error rate, and to compare them to those of the Single-Case Randomization Test (SCRT) assuming a randomly determined intervention point and the traditional permutation test assuming full exchangeability. Data were simulated without linear trends for five treatment effect levels (- 2, - 1, 0, 1, 2), five autocorrelation levels (0, .15, .30, .45, .60), and four observation number levels (30, 60, 90, 120). The power was calculated multiple times for all combinations of factor levels each generating 1000 replications. With 30 observations, the PDT showed sufficient power (≥ 80%) to detect medium treatment effects up to autocorrelation ≤ .45. Using 60 observations, the PDT showed sufficient power to detect medium treatment effects regardless of autocorrelation. With ≥ 90 observations, the PDT could also detect small treatment effects up to autocorrelation ≤ .30. With 30 observations, the type I error rate was 5-7%. With 60 observations and more, the type I error rate was ≤ 5% with autocorrelation < .60. The PDT outperformed the SCRT regarding power, particularly with a small number of observations. The PDT outperformed the traditional permutation test regarding type I error rate control, especially when autocorrelation increased. In conclusion, the PDT is a useful and promising nonparametric test to evaluate treatment effects in dependent SCOD AB-phase data without linear trends.

Examining the normality assumption of a design-comparable effect size in single-case designs

What Works Clearinghouse (WWC, 2022) recommends a design-comparable effect size (D-CES; i.e., gAB) to gauge an intervention in single-case experimental design (SCED) studies, or to synthesize findings in meta-analysis. So far, no research has examined gAB's performance under non-normal distributions. This study expanded Pustejovsky et al. (2014) to investigate the impact of data distributions, number of cases (m), number of measurements (N), within-case reliability or intra-class correlation (ρ), ratio of variance components (λ), and autocorrelation (ϕ) on gAB in multiple-baseline (MB) design. The performance of gAB was assessed by relative bias (RB), relative bias of variance (RBV), MSE, and coverage rate of 95% CIs (CR). Findings revealed that gAB was unbiased even under non-normal distributions. gAB's variance was generally overestimated, and its 95% CI was over-covered, especially when distributions were normal or nearly normal combined with small m and N. Large imprecision of gAB occurred when m was small and ρ was large. According to the ANOVA results, data distributions contributed to approximately 49% of variance in RB and 25% of variance in both RBV and CR. m and ρ each contributed to 34% of variance in MSE. We recommend gAB for MB studies and meta-analysis with N ≥ 16 and when either (1) data distributions are normal or nearly normal, m = 6, and ρ = 0.6 or 0.8, or (2) data distributions are mildly or moderately non-normal, m ≥ 4, and ρ = 0.2, 0.4, or 0.6. The paper concludes with a discussion of gAB's applicability and design-comparability, and sound reporting practices of ES indices.

Does the choice of a linear trend-assessment technique matter in the context of single-case data?

Trend is one of the data aspects that is an object of assessment in the context of single-case experimental designs. This assessment can be performed both visually and quantitatively. Given that trend, just like other relevant data features such as level, immediacy, or overlap does not have a single operative definition, a comparison among the existing alternatives is necessary. Previous studies have included illustrations of differences between trend-line fitting techniques using real data. In the current study, I carry out a simulation to study the degree to which different trend-line fitting techniques lead to different degrees of bias, mean square error, and statistical power for a variety of quantifications that entail trend lines. The simulation involves generating both continuous and count data, for several phase lengths, degrees of autocorrelation, and effect sizes (change in level and change in slope). The results suggest that, in general, ordinary least squares estimation performs well in terms of relative bias and mean square error. Especially, a quantification of slope change is associated with better statistical results than quantifying an average difference between conditions on the basis of a projected baseline trend. In contrast, the performance of the split-middle (bisplit) technique is less than optimal.

A Visual Aid and Objective Rule Encompassing the Data Features of Visual Analysis

Visual analysis of single-case research is commonly described as a gold standard, but it is often unreliable. Thus, an objective tool for applying visual analysis is necessary, as an alternative to the Conservative Dual Criterion, which presents some drawbacks. The proposed free web-based tool enables assessing change in trend and level between two adjacent phases, while taking data variability into account. The application of the tool results in (a) a dichotomous decision regarding the presence or absence of an immediate effect, a progressive or delayed effect, or an overall effect and (b) a quantification of overlap. The proposal is evaluated by applying it to both real and simulated data, obtaining favorable results. The visual aid and the objective rules are expected to make visual analysis more consistent, but they are not intended as a substitute for the analysts' judgment, as a formal test of statistical significance, or as a tool for assessing social validity.

Meta-Analysis of Single-Case Experimental Design using Multilevel Modeling

Multilevel modeling (MLM) is an approach for meta-analyzing single-case experimental designs (SCED). In this paper, we provide a step-by-step guideline for using the MLM to meta-analyze SCED time-series data. The MLM approach is first presented using a basic three-level model, then gradually extended to represent more realistic situations of SCED data, such as modeling a time variable, moderators representing different design types and multiple outcomes, and heterogeneous within-case variance. The presented approach is then illustrated using real SCED data. Practical recommendations using the MLM approach are also provided for applied researchers based on the current methodological literature. Available free and commercial software programs to meta-analyze SCED data are also introduced, along with several hands-on software codes for applied researchers to implement their own studies. Potential advantages and limitations of using the MLM approach to meta-analyzing SCED are discussed.

Systematic Review of Descriptions and Justifications Provided for Single-Case Quantification Techniques

There are currently a multitude of quantification techniques that have been developed for use with single-case designs. As a result, choosing an appropriate quantification technique can be overwhelming and it can be difficult for researchers to properly describe and justify their use of quantification techniques. However, providing clear descriptions and justifications is important for enhancing the credibility of single-case research, and allowing others to evaluate the appropriateness of the quantification technique used. The aim of this systematic literature review is to provide an overview of the quantification techniques that are used to analyze single-case designs, with a focus on the descriptions and justifications that are provided. A total of 290 quantifications occurred across 218 articles, and the descriptions and justifications that were provided for the quantification techniques that were used are systematically examined. Results show that certain quantification techniques, such as the non-overlap indices, are more commonly used. Descriptions and justifications provided for using the quantification techniques are sometimes vague or subjective. Single-case researchers are encouraged to complement visual analysis with the use of quantification techniques for which they can provide objective and appropriate descriptions and justifications, and are encouraged to use tools to guide their choice of quantification techniques.

The randomized marker method for single-case randomization tests: Handling data missing at random and data missing not at random

Single-case experiments are frequently plagued by missing data problems. In a recent study, the randomized marker method was found to be valid and powerful for single-case randomization tests when the missing data were missing completely at random. However, in real-life experiments, it is difficult for researchers to ascertain the missing data mechanism. For analyzing such experiments, it is essential that the missing data handling method is valid and powerful for various missing data mechanisms. Hence, we examined the performance of the randomized marker method for data that are missing at random and data that are missing not at random. In addition, we compared the randomized marker method with multiple imputation, because the latter is often considered the gold standard among imputation techniques. To compare and evaluate these two methods under various simulation conditions, we calculated the type I error rate and statistical power in single-case randomization tests using these two methods of handling missing data and compared them to the type I error rate and statistical power using complete datasets. The results indicate that while multiple imputation presents an advantage in the presence of strongly correlated covariate data, the randomized marker method remains valid and results in sufficient statistical power for most of the missing data conditions simulated in this study.

Interpreting treatment effect size from single case experimental design data: a preliminary analysis of differential effects of treatments designed to increase or decrease behaviour

BACKGROUND

Estimates of treatment effect size from single case experimental design (SCED) data may be impacted by the direction for treatment effects (i.e. ascending or descending slope for the dependent variable). Estimating effect sizes for treatments designed to decrease behaviour are potentially more restricted because the intended direction for treatment is zero (i.e. an absolute basal). Conversely, effect sizes for interventions that increase behaviour are less restricted due to a relatively unconstrained ceiling from a pure measurement standpoint (i.e. no absolute ceiling). That is, treatments that increase behaviour have a broader range of possible effect size values as the ceiling is only limited by demand characteristics and the learners' skills and motivation to exhibit the behaviour.

METHOD

The current study represents a preliminary analysis of the mean and range of SCED effect sizes for treatments designed to either increase or decrease target behaviour. A within-case Cohen's d measure that was developed for SCED data was used to estimate treatment effect sizes.

RESULTS

Results indicated that the mean and range of effect size values for treatments that increased behaviour were significantly greater compared with treatments that decreased behaviour.

CONCLUSIONS

Results are discussed in terms of developing standards, or best practices, specific to interpreting effect size values and meeting quality control requirements for inclusion of the data set in future SCED meta-analytic studies estimating treatment effect size. Specifically, preliminary results suggest that benchmarks for low, medium and high SCED effect size values need to be developed separately for treatments that increase or decrease levels of the dependent variable.

A Randomized Case Series Approach to Testing Efficacy of Interventions for Minimally Verbal Autistic Children

BACKGROUND

Randomized Controlled Trials (RCTs) are the gold standard for assessing whether an intervention is effective; however, they require large sample sizes in order to detect small effects. For rare or complex populations, we advocate a case series approach as a more realistic and useful first step for intervention evaluation. We consider the importance of randomization to such designs, and advocate for the use of Randomization Tests and Between Case Effect Sizes to provide a robust and statistically powerful evaluation of outcomes. In this tutorial, we describe the method, procedures, and analysis code necessary to conduct robust single case series, using an empirical example with minimally verbal autistic children.

METHOD

We applied a pre-registered (https://osf.io/9gvbs) randomized baseline design with between-case effect size to a case series (n = 19), to test the efficacy of a novel, parent-mediated, app-based speech production intervention (BabbleBooster) for minimally verbal autistic children. Parent-rated probe scores were used to densely sample performance accuracy over time.

RESULTS

Parents were able to reliably code their children's speech productions using BabbleBooster. A non-significant Randomization Test and small Between-Case Effect Size (d = 0.267), suggested there was no evidence that BabbleBooster improved speech production in minimally verbal autistic children, relative to baseline scores, during this brief period of intervention.

CONCLUSION

The current analyses exemplify a more robust approach to examining treatment effects in rare or complex populations, where RCT may be difficult or premature to implement. To facilitate adoption of this method by researchers and practitioners, we provide analysis code that can be adapted using open source R packages. Future studies could use this case series design to evaluate interventions aiming to improve speech and language outcomes for minimally verbal autistic children, and other heterogeneous and hard to reach populations.

Accurate models vs. accurate estimates: A simulation study of Bayesian single-case experimental designs

Although statistical practices to evaluate intervention effects in single-case experimental design (SCEDs) have gained prominence in recent times, models are yet to incorporate and investigate all their analytic complexities. Most of these statistical models incorporate slopes and autocorrelations, both of which contribute to trend in the data. The question that arises is whether in SCED data that show trend, there is indeterminacy between estimating slope and autocorrelation, because both contribute to trend, and the data have a limited number of observations. Using Monte Carlo simulation, we compared the performance of four Bayesian change-point models: (a) intercepts only (IO), (b) slopes but no autocorrelations (SI), (c) autocorrelations but no slopes (NS), and (d) both autocorrelations and slopes (SA). Weakly informative priors were used to remain agnostic about the parameters. Coverage rates showed that for the SA model, either the slope effect size or the autocorrelation credible interval almost always erroneously contained 0, and the type II errors were prohibitively large. Considering the 0-coverage and coverage rates of slope effect size, intercept effect size, mean relative bias, and second-phase intercept relative bias, the SI model outperformed all other models. Therefore, it is recommended that researchers favor the SI model over the other three models. Research studies that develop slope effect sizes for SCEDs should consider the performance of the statistic by taking into account coverage and 0-coverage rates. These helped uncover patterns that were not realized in other simulation studies. We underline the need for investigating the use of informative priors in SCEDs.

A systematic review of applied single-case research published between 2016 and 2018: Study designs, randomization, data aspects, and data analysis

Single-case experimental designs (SCEDs) have become a popular research methodology in educational science, psychology, and beyond. The growing popularity has been accompanied by the development of specific guidelines for the conduct and analysis of SCEDs. In this paper, we examine recent practices in the conduct and analysis of SCEDs by systematically reviewing applied SCEDs published over a period of three years (2016-2018). Specifically, we were interested in which designs are most frequently used and how common randomization in the study design is, which data aspects applied single-case researchers analyze, and which analytical methods are used. The systematic review of 423 studies suggests that the multiple baseline design continues to be the most widely used design and that the difference in central tendency level is by far most popular in SCED effect evaluation. Visual analysis paired with descriptive statistics is the most frequently used method of data analysis. However, inferential statistical methods and the inclusion of randomization in the study design are not uncommon. We discuss these results in light of the findings of earlier systematic reviews and suggest future directions for the development of SCED methodology.

Application of the Within- and Between-Series Estimators to Non-normal Multiple-Baseline Data: Maximum Likelihood and Bayesian Approaches

In single-case research, multiple-baseline (MB) design provides the opportunity to estimate the treatment effect based on not only within-series comparisons of treatment phase to baseline phase observations, but also time-specific between-series comparisons of observations from those that have started treatment to those that are still in the baseline. For analyzing MB studies, two types of linear mixed modeling methods have been proposed: the within- and between-series models. In principle, those models were developed based on normality assumptions, however, normality may not always be found in practical settings. Therefore, this study aimed to investigate the robustness of the within- and between-series models when data were non-normal. A Monte Carlo study was conducted with four statistical approaches. The approaches were defined by the crossing of two analytic decisions: (a) whether to use a within- or between-series estimate of effect and (b) whether to use restricted maximum likelihood or Markov chain Monte Carlo estimations. The results showed the treatment effect estimates of the four approaches had minimal bias, that within-series estimates were more precise than between-series estimates, and that confidence interval coverage was frequently acceptable, but varied across conditions and methods of estimation. Applications and implications were discussed based on the findings.

A randomization test wrapper for synthesizing single-case experiments using multilevel models: A Monte Carlo simulation study

Multilevel models (MLMs) have been proposed in single-case research, to synthesize data from a group of cases in a multiple-baseline design (MBD). A limitation of this approach is that MLMs require several statistical assumptions that are often violated in single-case research. In this article we propose a solution to this limitation by presenting a randomization test (RT) wrapper for MLMs that offers a nonparametric way to evaluate treatment effects, without making distributional assumptions or an assumption of random sampling. We present the rationale underlying the proposed technique and validate its performance (with respect to Type I error rate and power) as compared to parametric statistical inference in MLMs, in the context of evaluating the average treatment effect across cases in an MBD. We performed a simulation study that manipulated the numbers of cases and of observations per case in a dataset, the data variability between cases, the distributional characteristics of the data, the level of autocorrelation, and the size of the treatment effect in the data. The results showed that the power of the RT wrapper is superior to the power of parametric tests based on F distributions for MBDs with fewer than five cases, and that the Type I error rate of the RT wrapper is controlled for bimodal data, whereas this is not the case for traditional MLMs.

Meta-Analysis of Single-Case Research via Multilevel Models: Fundamental Concepts and Methodological Considerations

Multilevel modeling is an approach that can be used to summarize single-case experimental design (SCED) data. Multilevel models were developed to analyze hierarchical structured data with units at a lower level nested within higher level units. SCEDs use time series data collected from multiple cases (or subjects) within a study that allow researchers to investigate intervention effectiveness at the individual level and also to investigate how these individual intervention effects change over time. There is an increased interest in the field regarding how SCEDs can be used to establish an evidence base for interventions by synthesizing data from a series of intervention studies. Although using multilevel models to meta-analyze SCED studies is promising, application is often hampered by being potentially excessively technical. First, this article provides an accessible description and overview of the potential of multilevel meta-analysis to combine SCED data. Second, a summary of the methodological evidence on the performance of multilevel models for meta-analysis is provided, which is useful given that such evidence is currently scattered over multiple technical articles in the literature. Third, the actual steps to perform a multilevel meta-analysis are outlined in a brief practical guide. Fourth, a suggestion for integrating the quantitative results with a visual representation is provided.

The impact of response-guided baseline phase extensions on treatment effect estimates

BACKGROUND

When developmental disabilities researchers use multiple-baseline designs they are encouraged to delay the start of an intervention until the baseline stabilizes or until preceding cases have responded to intervention. Using ongoing visual analyses to guide the timing of the start of the intervention can help to resolve potential ambiguities in the graphical display; however, these forms of response-guided experimentation have been criticized as a potential source of bias in treatment effect estimation and inference.

AIMS AND METHODS

Monte Carlo simulations were used to examine the bias and precision of average treatment effect estimates obtained from multilevel models of four-case multiple-baseline studies with series lengths that varied from 19 to 49 observations per case. We varied the size of the average treatment effect, the factors used to guide intervention decisions (baseline stability, response to intervention, both, or neither), and whether the ongoing analysis was masked or not.

RESULTS

None of the methods of responding to the data led to appreciable bias in the treatment effect estimates. Furthermore, as timing-of-intervention decisions became responsive to more factors, baselines became longer and treatment effect estimates became more precise.

CONCLUSIONS

Although the study was conducted under limited conditions, the response-guided practices did not lead to substantial bias. By extending baseline phases they reduced estimation error and thus improved the treatment effect estimates obtained from multilevel models.

A Gradual Effects Model for Single-Case Designs

Single-case designs are a class of repeated measures experiments used to evaluate the effects of interventions for small or specialized populations, such as individuals with low-incidence disabilities. There has been growing interest in systematic reviews and syntheses of evidence from single-case designs, but there remains a need to further develop appropriate statistical models and effect sizes for data from the designs. We propose a novel model for single-case data that exhibit nonlinear time trends created by an intervention that produces gradual effects, which build up and dissipate over time. The model expresses a structural relationship between a pattern of treatment assignment and an outcome variable, making it appropriate for both treatment reversal and multiple baseline designs. It is formulated as a generalized linear model so that it can be applied to outcomes measured as frequency counts or proportions, both of which are commonly used in single-case research, while providing readily interpretable effect size estimates such as log response ratios or log odds ratios. We demonstrate the gradual effects model by applying it to data from a single-case study and examine the performance of proposed estimation methods in a Monte Carlo simulation of frequency count data.

One by One: Accumulating Evidence by using Meta-Analytical Procedures for Single-Case Experiments

This paper presents a unilevel and multilevel approach for the analysis and meta-analysis of single-case experiments (SCEs). We propose a definition of SCEs and derive the specific features of SCEs’ data that have to be taken into account when analysing and meta-analysing SCEs. We discuss multilevel models of increasing complexity and propose alternative and complementary techniques based on probability combining and randomisation test wrapping. The proposed techniques are demonstrated with real-life data and corresponding R code.

Linear Trend in Single-Case Visual and Quantitative Analyses

The frequently used visual analysis of single-case data focuses on data aspects such as level, trend, variability, overlap, immediacy of effect, and consistency of data patterns; most of these aspects are also commonly quantified besides inspecting them visually. The present text focuses on trend, because even linear trend can be operatively defined in several different ways, while there are also different approaches for controlling for baseline trend. We recommend using a quantitative criterion for choosing a trend line fitting technique and comparing baseline and intervention slopes, instead of detrending. We implement our proposal in a free web-based application created specifically for following the What Works Clearinghouse Standards recommendations for visual analysis. This application is especially destined to applied researchers and provides graphical representation of the data, visual aids, and quantifications of the difference between phases in terms of level, trend, and overlap, as well as two quantifications of the immediate effect. An evaluation of the consistency of effects across replications of the AB sequence is also provided. For methodologists and statisticians, we include formulas and examples of the different straight line fitting and detrending techniques to improve the reproducibility of results and simulations.

The impact of ordinate scaling on the visual analysis of single-case data

Visual analysis is the primary method for detecting the presence of treatment effects in graphically displayed single-case data and it is often referred to as the "gold standard." Although researchers have developed standards for the application of visual analysis (e.g., Horner et al., 2005), over- and underestimation of effect size magnitude is not uncommon among analysts. Several characteristics have been identified as potential contributors to these errors; however, researchers have largely focused on characteristics of the data itself (e.g., autocorrelation), paying less attention to characteristics of the graphic display which are largely in control of the analyst (e.g., ordinate scaling). The current study investigated the impact that differences in ordinate scaling, a graphic display characteristic, had on experts' accuracy in judgments regarding the magnitude of effect present in single-case percentage data. 32 participants were asked to evaluate eight ABAB data sets (2 each presenting null, small, moderate, and large effects) along with three iterations of each (32 graphs in total) in which only the ordinate scale was manipulated. Results suggest that raters are less accurate in their detection of treatment effects as the ordinate scale is constricted. Additionally, raters were more likely to overestimate the size of a treatment effect when the ordinate scale was constricted.

An Improved Rank Correlation Effect Size Statistic for Single-Case Designs: Baseline Corrected Tau

Measuring treatment effects when an individual’s pretreatment performance is improving poses a challenge for single-case experimental designs. It may be difficult to determine whether improvement is due to the treatment or due to the preexisting baseline trend. Tau- U is a popular single-case effect size statistic that purports to control for baseline trend. However, despite its strengths, Tau- U has substantial limitations: Its values are inflated and not bound between −1 and +1, it cannot be visually graphed, and its relatively weak method of trend control leads to unacceptable levels of Type I error wherein ineffective treatments appear effective. An improved effect size statistic based on rank correlation and robust regression, Baseline Corrected Tau, is proposed and field-tested with both published and simulated single-case time series. A web-based calculator for Baseline Corrected Tau is also introduced for use by single-case investigators.

How Can Single-Case Data Be Analyzed? Software Resources, Tutorial, and Reflections on Analysis

The present article aims to present a series of software developments in the quantitative analysis of data obtained via single-case experimental designs (SCEDs), as well as the tutorial describing these developments. The tutorial focuses on software implementations based on freely available platforms such as R and aims to bring statistical advances closer to applied researchers and help them become autonomous agents in the data analysis stage of a study. The range of analyses dealt with in the tutorial is illustrated on a typical single-case dataset, relying heavily on graphical data representations. We illustrate how visual and quantitative analyses can be used jointly, giving complementary information and helping the researcher decide whether there is an intervention effect, how large it is, and whether it is practically significant. To help applied researchers in the use of the analyses, we have organized the data in the different ways required by the different analytical procedures and made these data available online. We also provide Internet links to all free software available, as well as all the main references to the analytical techniques. Finally, we suggest that appropriate and informative data analysis is likely to be a step forward in documenting and communicating results and also for increasing the scientific credibility of SCEDs.

Using Single-Case Experiments to Support Evidence-Based Decisions: How Much Is Enough?

For practitioners, the use of single-case experimental designs (SCEDs) in the research literature raises an important question: How many single-case experiments are enough to have sufficient confidence that an intervention will be effective with an individual from a given population? Although standards have been proposed to address this question, current guidelines do not appear to be strongly grounded in theory or empirical research. The purpose of our article is to address this issue by presenting guidelines to facilitate evidence-based decisions by adopting a simple statistical approach to quantify the support for interventions that have been validated using SCEDs. Specifically, we propose the use of success rates as a supplement to support evidence-based decisions. The proposed methodology allows practitioners to aggregate the results from single-case experiments to estimate the probability that a given intervention will produce a successful outcome. We also discuss considerations and limitations associated with this approach.

Probability and Visual Aids for Assessing Intervention Effectiveness in Single-Case Designs

Single-case data analysis still relies heavily on visual inspection, and, at the same time, it is not clear to what extent the results of different quantitative procedures converge in identifying an intervention effect and its magnitude when applied to the same data; this is the type of evidence provided here for two procedures. One of the procedures, included due to the importance of providing objective criteria to visual analysts, is a visual aid fitting and projecting split-middle trend while taking into account data variability. The other procedure converts several different metrics into probabilities making their results comparable. In the present study, we expore to what extend these two procedures coincide in the magnitude of intervention effect taking place in a set of studies stemming from a recent meta-analysis. The procedures concur to a greater extent with the values of the indices computed and with each other and, to a lesser extent, with our own visual analysis. For distinguishing smaller from larger effects, the probability-based approach seems somewhat better suited. Moreover, the results of the field test suggest that the latter is a reasonably good mechanism for translating different metrics into similar labels. User friendly R code is provided for promoting the use of the visual aid, together with a quantification based on nonoverlap and the label provided by the probability approach.

Assessing Functional Relations in Single-Case Designs

In the context of the evidence-based practices movement, the emphasis on computing effect sizes and combining them via meta-analysis does not preclude the demonstration of functional relations. For the latter aim, we propose to augment the visual analysis to add consistency to the decisions made on the existence of a functional relation without losing sight of the need for a methodological evaluation of what stimuli and reinforcement or punishment are used to control the behavior. Four options for quantification are reviewed, illustrated, and tested with simulated data. These quantifications include comparing the projected baseline with the actual treatment measurements, on the basis of either parametric or nonparametric statistics. The simulated data used to test the quantifications include nine data patterns in terms of the presence and type of effect and comprise ABAB and multiple-baseline designs. Although none of the techniques is completely flawless in terms of detecting a functional relation only when it is present but not when it is absent, an option based on projecting split-middle trend and considering data variability as in exploratory data analysis proves to be the best performer for most data patterns. We suggest that the information on whether a functional relation has been demonstrated should be included in meta-analyses. It is also possible to use as a weight the inverse of the data variability measure used in the quantification for assessing the functional relation. We offer an easy to use code for open-source software for implementing some of the quantifications.