Indexing effects in single-case experimental designs by estimating the percent of goal obtained

Rise of single-case experimental designs: A historical overview of the necessity of single-case methodology

ABSTRACTWindelband ([1894]1980) advocated that two approaches are used for accumulating scientific knowledge. The first is the idiographic approach that derives knowledge from a single unit, and the second is the nomothetic approach that accumulates knowledge of a group. Given these two approaches, the former matches case studies while the latter is more appropriate with experimental group studies. Scientists have criticized both methodologies for their various limitations. Later, the single-case methodology emerged as an alternative that potentially allays these limitations. In this context, this narrative review aims to describe the historical roots of single-case experimental designs (SCEDs) that have emerged to eliminate the tension of nomothetic and idiographic approaches over time. First, the review focuses on the emergence of SCEDs. Second, the strengths and challenges of SCEDs are reviewed, including those to address the limitations of group experimental and case studies. Third, the use and analyses of SCEDs are outlined, considering their current status. Fourth, this narrative review continues to delineate the dissemination of SCEDs in the modern scientific world. As a result, SCEDs can be evaluated as a method that has the potential to overcome the issues encountered in case description and group experimental research. Thus, that helps accumulate nomothetic and idiographic knowledge in determining evidence-based practices.

Evaluating the Effectiveness of Take it Personal!+ in People With Mild Intellectual Disability or Borderline Intellectual Functioning and Substance Use Disorder: A Multiple Baseline Single-Case Experimental Study

Individuals with mild intellectual disabilities or borderline intellectual functioning are at increased risk to develop a substance use disorder-however, effective treatment programs adapted to this target group are scarce. This study evaluated the effectiveness of Take it Personal!+ in individuals with mild intellectual disabilities or borderline intellectual functioning and substance use disorder. Take it Personal!+ is a personalized treatment based on motivational interviewing and cognitive-behavioral therapy supported by an mHealth application. Data were collected in a nonconcurrent multiple baseline single-case experimental design across individuals with four phases (i.e., baseline, treatment, posttreatment, and follow-up). Twelve participants were randomly allocated to baseline lengths varying between 7 and 11 days. Substance use quantity was assessed during baseline, treatment, and posttreatment with a daily survey using a mobile application. Visual analysis was supported with statistical analysis of the daily surveys by calculating three effect size measures in 10 participants (two participants were excluded from this analysis due to a compliance rate below 50%). Secondary, substance use severity was assessed with standardized questionnaires at baseline, posttreatment, and follow-up and analyzed by calculating the Reliable Change Index. Based on visual analysis of the daily surveys, 10 out of 12 participants showed a decrease in mean substance use quantity from baseline to treatment and, if posttreatment data were available, to posttreatment. Statistical analysis showed an effect of Take it Personal!+ in terms of a decrease in daily substance use in 8 of 10 participants from baseline to treatment and if posttreatment data were available, also to posttreatment. In addition, data of the standardized questionnaires showed a decrease in substance use severity in 8 of 12 participants. These results support the effectiveness of Take it Personal!+ in decreasing substance use in individuals with mild intellectual disabilities or borderline intellectual functioning.

Compassion focused therapy for self-stigma and shame in autism: a single case pre-experimental study

Introduction

Exposure to public stigma can lead to the internalization of autism-related stigma (i.e., self-stigma), associated with negative health, occupational and social outcomes. Importantly, self-stigma is linked to shame and social isolation. Although elevated self-stigma has been reported in autistic adults, to the best of our knowledge, interventions designed to target this issue are lacking. Compassion is an effective way to reduce the emotional correlates of self-stigma (i.e., shame) and their impacts on mental health. However, no study has investigated whether compassion focused therapy (CFT) can effectively reduce self-stigma in autistic adults. The present study aims at investigating whether and how self-compassion improvement following CFT may reduce self-stigma and shame in an autistic individual.

Methods

A single case pre-experimental design (SCED) was used with weekly repeated measures during four phases: (i) pure baseline without any intervention (A), (ii) case conceptualization (A'), (iii) intervention (B) where CFT was delivered, (iv) follow-up without intervention (FU). The participant is a 46-year-old autistic man with high self-stigma and shame. Self-report measures of self-compassion and self-stigma and a daily idiographic measure of shame were used.

Results

There was a large increase in self-compassion between pure baseline (A) and the intervention phase (A'B) (Tau-U = 0.99), maintained at follow-up. Similarly, there was a moderate decrease of self-stigma (Tau-U = 0.32). In contrast, when we compared the whole baseline phase AA' (i.e., considering the conceptualisation phase as baseline) to the intervention (B), there was no change in self-stigma (Tau-U = -0.09). There was no change in self-stigma between the intervention (B) and follow-up (Tau-U = -0.19). There was a moderate decrease in daily shame reports between the baseline (AA') and the intervention (B) (Tau-U = 0.31) and a moderate decrease between the pure baseline (A) and intervention phase (A'B) (Tau-U = 0.51).

Conclusion

CFT was feasible for this autistic client and our results show that CFT led to the improvement of self-compassion. Changes on self-stigma measures were moderate. Self-stigma may need more time to change. Because self-stigma is involved in poorer social functioning and mental health in autistic adults, our results are promising and suggesting conducting more large-scale studies on CFT in autistic adults.

Vocabulary Instruction Embedded in Narrative Intervention: A Repeated Acquisition Design Study With First Graders at Risk of Language-Based Reading Difficulty

PURPOSE

The purpose of the current study was to investigate the effect of vocabulary instruction embedded in the narrative intervention on the immediate and retained definitional knowledge of taught words for first graders at risk for language-related reading difficulties.

METHOD

We employed a repeated acquisition design with innovative quality features and supplemental statistics with 11 treatment students and three control students. In the context of the school's multitiered system of supports, treatment students received 30-min small group interventions, 4 days a week for 12 weeks. Intervention involved story retelling and personal story generation lessons, both of which emphasized the learning and practicing of target vocabulary words in each story. Pre- and postprobes of the taught definitions were conducted every week.

RESULTS

According to visual analysis conventions of single-case research, there was a consistent pattern of improvement from pre- to postprobes for all treatment participants, but for none of the control participants. Retention was also consistently observed, when measured at Week 13. Supplemental statistics confirmed that large effects were associated with the intervention.

CONCLUSION

Vocabulary instruction embedded in narrative intervention led to meaningful acquisition and retention of taught vocabulary for students at risk of language-based reading difficulty.

Systematic Review of Descriptions and Justifications Provided for Single-Case Quantification Techniques

There are currently a multitude of quantification techniques that have been developed for use with single-case designs. As a result, choosing an appropriate quantification technique can be overwhelming and it can be difficult for researchers to properly describe and justify their use of quantification techniques. However, providing clear descriptions and justifications is important for enhancing the credibility of single-case research, and allowing others to evaluate the appropriateness of the quantification technique used. The aim of this systematic literature review is to provide an overview of the quantification techniques that are used to analyze single-case designs, with a focus on the descriptions and justifications that are provided. A total of 290 quantifications occurred across 218 articles, and the descriptions and justifications that were provided for the quantification techniques that were used are systematically examined. Results show that certain quantification techniques, such as the non-overlap indices, are more commonly used. Descriptions and justifications provided for using the quantification techniques are sometimes vague or subjective. Single-case researchers are encouraged to complement visual analysis with the use of quantification techniques for which they can provide objective and appropriate descriptions and justifications, and are encouraged to use tools to guide their choice of quantification techniques.

Evaluating Technology-Based Self-Monitoring of Performance with Differential Reinforcement for Students with Disabilities

This study evaluated the use of a technology-based self-monitoring of performance (SMP) with differential reinforcement to increase task completion and reduce off-task behavior for three 5th-grade students with disabilities. A concurrent multiple baseline design across participants was used to examine the impact of the intervention on the targeted behaviors when implemented by a general education teacher and its maintenance effects with a delay of reinforcement. The implementation involved training students to use a mobile app for SMP and providing differential reinforcement contingent on task completion and accuracy of self-monitoring during academic periods. The secondary measure of off-task behavior was included to evaluate the relationship between task completion and engagement. The results demonstrated that the technology-based SMP with differential reinforcement increased task completion and reduced off-task behavior for all students. Furthermore, the gradual fading of the reinforcement, with a 45 min delay, was successful for all students. The efficiency and immediacy of the intervention suggest that technology-based SMP with differential reinforcement holds promise as a practical, efficient, and effective school-based intervention.

Single-case intervention research design standards: Additional proposed upgrades and future directions

Single-case intervention research design standards have evolved considerably over the past decade. These standards serve the dual role of assisting in single-case design (SCD) intervention research methodology and as guidelines for literature syntheses within a particular research domain. In a recent article (Kratochwill et al., 2021), we argued for a need to clarify key features of these standards. In this article we offer additional recommendations for SCD research and synthesis standards that have been either underdeveloped or missing in the conduct of research and in literature syntheses. Our recommendations are organized into three categories: expanding design standards, expanding evidence standards, and expanding the applications and consistency of SCDs. The recommendations we advance are for consideration for future standards, research design training, and they are especially important to guide the reporting of SCD intervention investigations as they enter the literature-synthesis phase of evidence-based practice initiatives.

Effects of Explicit Vocabulary Interventions for Preschoolers: An Exploratory Application of the Percent of Goal Obtained Effect Size Metric

PURPOSE

Systematic reviews of literature are routinely conducted to identify practices that are effective in addressing educational and clinical problems. One complication, however, is how best to combine data from both group experimental design (GED) studies and single-case experimental design (SCED) studies. Percent of Goal Obtained (PoGO) has been developed as a metric to express the size of the effect relative to the distance to a goal, which could have broad applicability. This study sought to augment this descriptive index with estimates of standard errors, which are needed to use PoGO as an effect size metric in meta-analyses of SCED and GED studies. This study investigated the application of PoGO and standard errors to both SCED and GED studies examining a common intervention approach used with a single population.

METHOD

Sixteen articles investigating explicit vocabulary instruction applied to pre-K and kindergarten students were identified. PoGO and standard errors were calculated for variations of explicit vocabulary interventions. Evaluated interventions included six studies using exclusively an SCED, nine studies using a GED, and one that used both.

RESULTS

PoGO was calculated for each treatment condition when applicable (i.e., alternating treatments designs). Standard errors and confidence interval limits also were calculated. PoGO effect size values ranged from 14.4% to 93.6%. PoGO for single-case experiments was 49.2% with a standard error of 7.26, and for group experiments, it was 30.8% with a standard error of 3.71.

CONCLUSION

Despite variation in the percentage of goal obtained across studies, the high degree of overlap in PoGO and standard errors between single-case and group experiments provides an indication that systematic reviews can apply this effect size metric to combine information obtained across experimental designs.

Interpreting treatment effect size from single case experimental design data: a preliminary analysis of differential effects of treatments designed to increase or decrease behaviour

BACKGROUND

Estimates of treatment effect size from single case experimental design (SCED) data may be impacted by the direction for treatment effects (i.e. ascending or descending slope for the dependent variable). Estimating effect sizes for treatments designed to decrease behaviour are potentially more restricted because the intended direction for treatment is zero (i.e. an absolute basal). Conversely, effect sizes for interventions that increase behaviour are less restricted due to a relatively unconstrained ceiling from a pure measurement standpoint (i.e. no absolute ceiling). That is, treatments that increase behaviour have a broader range of possible effect size values as the ceiling is only limited by demand characteristics and the learners' skills and motivation to exhibit the behaviour.

METHOD

The current study represents a preliminary analysis of the mean and range of SCED effect sizes for treatments designed to either increase or decrease target behaviour. A within-case Cohen's d measure that was developed for SCED data was used to estimate treatment effect sizes.

RESULTS

Results indicated that the mean and range of effect size values for treatments that increased behaviour were significantly greater compared with treatments that decreased behaviour.

CONCLUSIONS

Results are discussed in terms of developing standards, or best practices, specific to interpreting effect size values and meeting quality control requirements for inclusion of the data set in future SCED meta-analytic studies estimating treatment effect size. Specifically, preliminary results suggest that benchmarks for low, medium and high SCED effect size values need to be developed separately for treatments that increase or decrease levels of the dependent variable.

Performance criteria-based effect size (PCES) measurement of single-case experimental designs: A real-world data study

Visual analysis and nonoverlap-based effect sizes are predominantly used in analyzing single case experimental designs (SCEDs). Although they are popular analytical methods for SCEDs, they have certain limitations. In this study, a new effect size calculation model for SCEDs, named performance criteria-based effect size (PCES), is proposed considering the limitations of 4 nonoverlap-based effect size measures, widely accepted in the literature and that blend well with visual analysis. In the field test of PCES, actual data from published studies were utilized, and the relations between PCES, visual analysis, and the 4 nonoverlap-based methods were examined. In determining the data to be used in the field test, 1,052 tiers (AB phases) were identified from 6 journals. The results revealed a weak or moderate relation between PCES and nonoverlap-based methods due to its focus on performance criteria. Although PCES has some weaknesses, it promises to eliminate the causes that may create issues in nonoverlap-based methods, using quantitative data to determine socially important changes in behavior and to complement visual analysis.

A proposal for the assessment of replication of effects in single-case experimental designs

In science in general and in the context of single‐case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been replicated or not (or to what extent) are scarce, in the general context of behavioral sciences, and practically null in the single‐case experimental designs context. We propose an extension of the modified Brinley plot for assessing how many of the effects replicate. To make this assessment possible, a definition of replication is suggested, on the basis of expert judgment, rather than on statistical criteria. The definition of replication and its graphical representation are justified, presenting their strengths and limitations, and illustrated with real data. A user‐friendly software is made available for obtaining automatically the graphical representation.

Causal Mediation Analysis in Single Case Experimental Designs: Introduction to the Special Issue

This special issue of Evaluation and the Health Professions is dedicated to methods for causal mediation analysis in Single Case Experimental Designs (SCEDs). Mediation analysis is used to identify intermediate variables that transmit the effect of the independent variable on the outcome. Until recently, mediation analysis was mostly confined to between-subjects designs and panel studies with few exceptions. Consequently, most of the developments in causal mediation analysis have also been restricted to such designs. In applied health research, SCEDs have been used to evaluate total effects of treatments on outcomes of interest. Providing researchers with the methods for evaluating causal indirect effects for individual participants can lead to important improvements in diagnosis, treatment, and prevention. This special issue includes articles that describe advanced quantitative methods for testing mediators in SCEDs, propose and test approaches that allow for relaxing statistical assumptions that may not hold in real data, and illustrate mediation analysis for a single participant in real and simulated SCEDs data.

Single-case design standards: An update and proposed upgrades

In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.

Meta-analyses and effect sizes in applied behavior analysis: A review and discussion

For more than four decades, researchers have used meta-analyses to synthesize data from multiple experimental studies often to draw conclusions that are not supported by individual studies. More recently, single-case experimental design (SCED) researchers have adopted meta-analysis techniques to answer research questions with data gleaned from SCED experiments. Meta-analyses enable researchers to answer questions regarding intervention efficacy, generality, and condition boundaries. Here we discuss meta-analysis techniques, the rationale for their adaptation with SCED studies, and current indices used to quantify the effect of SCED data in applied behavior analysis.

Assessing Consistency in Single-Case Data Features Using Modified Brinley Plots

The current text deals with the assessment of consistency of data features from experimentally similar phases and consistency of effects in single-case experimental designs. Although consistency is frequently mentioned as a critical feature, few quantifications have been proposed so far: namely, under the acronyms CONDAP (consistency of data patterns in similar phases) and CONEFF (consistency of effects). Whereas CONDAP allows assessing the consistency of data patterns, the proposals made here focus on the consistency of data features such as level, trend, and variability, as represented by summary measures (mean, ordinary least squares slope, and standard deviation, respectively). The assessment of consistency of effect is also made in terms of these three data features, while also including the study of the consistency of an immediate effect (if expected). The summary measures are represented as points on a modified Brinley plot and their similarity is assessed via quantifications of distance. Both absolute and relative measures of consistency are proposed: the former expressed in the same measurement units as the outcome variable and the latter as a percentage. Illustrations with real data sets (multiple baseline, ABAB, and alternating treatments designs) show the wide applicability of the proposals. We developed a user-friendly website to offer both the graphical representations and the quantifications.