A Visual Aid and Objective Rule Encompassing the Data Features of Visual Analysis

How to assess and take into account trend in single-case experimental design data

One of the data features that are expected to be assessed when analyzing single-case experimental designs (SCED) data is trend. The current text deals with four different questions that applied researchers can ask themselves when assessing trend and especially when dealing with improving baseline trend: (a) What options exist for assessing the presence of trend?; (b) Once assessed, what criterion can be followed for deciding whether it is necessary to control for baseline trend?; (c) What strategy can be followed for controlling for baseline trend?; and (d) How to proceed in case there is baseline trend only in some A-B comparisons? Several options are reviewed for each of these questions in the context of real data, and tentative recommendations are provided. A new user-friendly website is developed to implement the options for fitting a trend line and a criterion for selecting a specific technique for that purpose. Trend-related and more general data analytical recommendations are provided for applied researchers.Trial registration: ClinicalTrials.gov identifier: NCT04560777.

Effectiveness of CO-OP group intervention for children with developmental coordination disorder: single-case experimental design study protocol

INTRODUCTION

With the Cognitive Orientation to daily Occupational Performance (CO-OP) approach, children with developmental coordination disorder learn to use a problem-solving strategy to deal with their motor difficulties and perform daily activities of their choice. Therapists use guided discovery to enable children to find their own solutions. Although CO-OP is recommended in a group setting, studies are needed to support its effectiveness.

METHODS AND ANALYSIS

A single-case study design with multiple baselines across participants and four systematic replications will be used. In each of the five groups, four children (aged 8-12 years) will be randomly included at the baseline. The baseline includes 5-8 measurements, and the CO-OP intervention stage is comprised of 10 sessions. The follow-up stage includes five measurements. Prior to baseline, each child in each of the five groups will choose five activities of which three will be carried out during the intervention sessions. Children's performance in each of these activities will be scored using the Performance Quality Rating Scale (PQRS) as the main measure. Three secondary measures will be collected: perceived activity performance using the Canadian Occupational Performance Measure, quality of life using the Kidscreen-27 and spontaneous motor rhythm using a computerised typing task. Graphed data will be analysed visually at the individual level with the Visual Aid Implying an Objective Rule (VAIOR) protocol which provides a colour code based on the level and trend of two consecutive phases, facilitating an objective visual analysis. Statistics will be performed for PQRS scores at the individual level and at the group level.

ETHICS AND DISSEMINATION

The protocol has been approved by the Comité de protection des personnes Sud-Est I (CPP 2021070) and the Comité d'éthique de la recherche avec les êtres humains de l'Université du Québec à Trois-Rivières (CER-22-294-07.03). Results will be published in a peer-reviewed scientific journal.

TRIAL REGISTRATION NUMBER

NCT05231486.

Waiting for baseline stability in single-case designs: Is it worth the time and effort?

Researchers and practitioners often use single-case designs (SCDs), or n-of-1 trials, to develop and validate novel treatments. Standards and guidelines have been published to provide guidance as to how to implement SCDs, but many of their recommendations are not derived from the research literature. For example, one of these recommendations suggests that researchers and practitioners should wait for baseline stability prior to introducing an independent variable. However, this recommendation is not strongly supported by empirical evidence. To address this issue, we used Monte Carlo simulations to generate graphs with fixed, response-guided, and random baseline lengths while manipulating trend and variability. Then, our analyses compared the type I error rate and power produced by two methods of analysis: the conservative dual-criteria method (a structured visual aid) and a support vector classifier (a model derived from machine learning). The conservative dual-criteria method produced fewer errors when using response-guided decision-making (i.e., waiting for stability) and random baseline lengths. In contrast, waiting for stability did not reduce decision-making errors with the support vector classifier. Our findings question the necessity of waiting for baseline stability when using SCDs with machine learning, but the study must be replicated with other designs and graph parameters that change over time to support our results.

Defining and assessing immediacy in single-case experimental designs

Immediacy is one of six data aspects (alongside level, trend, variability, overlap, and consistency) that has to be accounted for when visually analyzing single-case data. Given that it is one of the aspects that has received considerably less attention than other data aspects, the current text offers a review of the proposed conceptual definitions of immediacy (i.e., what it refers to) and also of the suggested operational definitions (i.e., how exactly is it assessed and/or quantified). Provided that a variety of conceptual and operational definitions is identified, we propose following a sensitivity analysis using a randomization test for assessing immediate effects in single-case experimental designs, by identifying when changes were most clear. In such a sensitivity analysis, the immediate effects are tested for multiple possible intervention points and for different possible operational definitions. Robust immediate effects can be detected if the results for the different operational definitions converge.

Performance criteria-based effect size (PCES) measurement of single-case experimental designs: A real-world data study

Visual analysis and nonoverlap-based effect sizes are predominantly used in analyzing single case experimental designs (SCEDs). Although they are popular analytical methods for SCEDs, they have certain limitations. In this study, a new effect size calculation model for SCEDs, named performance criteria-based effect size (PCES), is proposed considering the limitations of 4 nonoverlap-based effect size measures, widely accepted in the literature and that blend well with visual analysis. In the field test of PCES, actual data from published studies were utilized, and the relations between PCES, visual analysis, and the 4 nonoverlap-based methods were examined. In determining the data to be used in the field test, 1,052 tiers (AB phases) were identified from 6 journals. The results revealed a weak or moderate relation between PCES and nonoverlap-based methods due to its focus on performance criteria. Although PCES has some weaknesses, it promises to eliminate the causes that may create issues in nonoverlap-based methods, using quantitative data to determine socially important changes in behavior and to complement visual analysis.

Agreement between visual inspection and objective analysis methods: A replication and extension

Behavior analysts typically rely on visual inspection of single‐case experimental designs to make treatment decisions. However, visual inspection is subjective, which has led to the development of supplemental objective methods such as the conservative dual‐criteria method. To replicate and extend a study conducted by Wolfe et al. (2018) on the topic, we examined agreement between the visual inspection of five raters, the conservative dual‐criteria method, and a machine‐learning algorithm (i.e., the support vector classifier) on 198 AB graphs extracted from clinical data. The results indicated that average agreement between the 3 methods was generally consistent. Mean interrater agreement was 84%, whereas raters agreed with the conservative dual‐criteria method and the support vector classifier on 84% and 85% of graphs, respectively. Our results indicate that both objective methods produce results consistent with visual inspection, which may support their future use.

Machine Learning to Support Visual Inspection of Data: A Clinical Application

Practitioners in pediatric feeding programs often rely on single-case experimental designs and visual inspection to make treatment decisions (e.g., whether to change or keep a treatment in place). However, researchers have shown that this practice remains subjective, and there is no consensus yet on the best approach to support visual inspection results. To address this issue, we present the first application of a pediatric feeding treatment evaluation using machine learning to analyze treatment effects. A 5-year-old male with autism spectrum disorder participated in a 2-week home-based, behavior-analytic treatment program. We compared interrater agreement between machine learning and expert visual analysts on the effects of a pediatric feeding treatment within a modified reversal design. Both the visual analyst and the machine learning model generally agreed about the effectiveness of the treatment while overall agreement remained high. Overall, the results suggest that machine learning may provide additional support for the analysis of single-case experimental designs implemented in pediatric feeding treatment evaluations.

Single-case experimental designs: Characteristics, changes, and challenges

Tactics of Scientific Research (Sidman, 1960) provides a visionary treatise on single-case designs, their scientific underpinnings, and their critical role in understanding behavior. Since the foundational base was provided, single-case designs have proliferated especially in areas of application where they have been used to evaluate interventions with an extraordinary range of clients, settings, and target foci. This article highlights core features of single-case experimental designs, how key and ancillary features of the designs have evolved, the special strengths of the designs, and challenges that have impeded their integration in many areas where their contributions are sorely needed. The article ends by placing the methodological approach in the context of other research traditions. In this way, the discussion moves from the specific designs toward foundations and philosophy of science issues in keeping with the strengths of the person and book we are honoring.

Search and Selection Procedures of Literature Reviews in Behavior Analysis

Literature reviews allow professionals to identify effective interventions and assess developments in research and practice. As in other forms of scientific inquiry, the transparency of literature searches enhances the credibility of findings, particularly in regards to intervention research. The current review evaluated the characteristics of search methods employed in literature reviews appearing in publications concerning behavior analysis (n = 28) from 1997 to 2017. Specific aims included determining the frequency of narrative, systematic, and meta-analytic reviews over time; examining the publication of reviews in specific journals; and evaluating author reports of literature search and selection procedures. Narrative reviews (51.30%; n = 630) represented the majority of the total sample (n = 1,228), followed by systematic (31.51%; n = 387) and meta-analytic (17.18%; n = 211) reviews. In contrast to trends in related fields (e.g., special education), narrative reviews continued to represent a large portion of published reviews each year. The evaluated reviews exhibited multiple strengths; nonetheless, issues involving the reporting and execution of searches may limit the validity and replicability of literature reviews. A discussion of implications for research follows an overview of findings.

Three Alternatives for Graphing Behavioral Data: A Comparison of Usability and Acceptability

Making treatment decisions based upon graphed data is important in helping professions. A small amount of research has compared usability between equal-interval and semi-log graphs, but no prior studies have compared different types of semi-log graphs. Using a randomized, cross-over, experimental design with 72 participants, this study examined the relative usability and acceptability of three types of graphs: Regular (equal-interval), Standard Celeration Chart (SCC; semi-log), and Standard Behavior Graph (SBG; semi-log). All participants used each graph across three usability tasks (Plotting Data, Writing Values, and Interpreting Trends). For the Plotting and Writing tasks, the equal-interval graph produced the greatest rate of correct responses. However, for the Interpreting task the SBG produced the greatest rate of corrects, while the equal-interval graph produced the smallest rate. User acceptability mainly favored the equal-interval and SBG graphs. Study findings and implications are discussed with respect to graph usability and acceptability during day-to-day practice.

How Many Tiers Do We Need? Type I Errors and Power in Multiple Baseline Designs

Design quality guidelines typically recommend that multiple baseline designs include at least three demonstrations of effects. Despite its widespread adoption, this recommendation does not appear grounded in empirical evidence. The main purpose of our study was to address this issue by assessing Type I error rate and power in multiple baseline designs. First, we generated 10,000 multiple baseline graphs, applied the dual-criteria method to each tier, and computed Type I error rate and power for different number of tiers showing a clear change. Second, two raters categorized the tiers for 300 multiple baseline graphs to replicate our analyses using visual inspection. When multiple baseline designs had at least three tiers and two or more of these tiers showed a clear change, the Type I error rate remained adequate (< .05) while power also reached acceptable levels (> .80). In contrast, requiring all tiers to show a clear change resulted in overly stringent conclusions (i.e., unacceptably low power). Therefore, our results suggest that researchers and practitioners should carefully consider limitations in power when requiring all tiers of a multiple baseline design to show a clear change in their analyses.

Machine Learning to Analyze Single-Case Data: A Proof of Concept

Visual analysis is the most commonly used method for interpreting data from single-case designs, but levels of interrater agreement remain a concern. Although structured aids to visual analysis such as the dual-criteria (DC) method may increase interrater agreement, the accuracy of the analyses may still benefit from improvements. Thus, the purpose of our study was to (a) examine correspondence between visual analysis and models derived from different machine learning algorithms, and (b) compare the accuracy, Type I error rate and power of each of our models with those produced by the DC method. We trained our models on a previously published dataset and then conducted analyses on both nonsimulated and simulated graphs. All our models derived from machine learning algorithms matched the interpretation of the visual analysts more frequently than the DC method. Furthermore, the machine learning algorithms outperformed the DC method on accuracy, Type I error rate, and power. Our results support the somewhat unorthodox proposition that behavior analysts may use machine learning algorithms to supplement their visual analysis of single-case data, but more research is needed to examine the potential benefits and drawbacks of such an approach.