Analysis of single-case data: Randomisation tests for measures of effect size

Testing delayed, gradual, and temporary treatment effects in randomized single-case experiments: A general response function framework

Randomization tests represent a class of significance tests to assess the statistical significance of treatment effects in randomized single-case experiments. Most applications of single-case randomization tests concern simple treatment effects: immediate, abrupt, and permanent changes in the level of the outcome variable. However, researchers are confronted with delayed, gradual, and temporary treatment effects; in general, with "response functions" that are markedly different from single-step functions. We here introduce a general framework that allows specifying a test statistic for a randomization test based on predicted response functions that is sensitive to a wide variety of data patterns beyond immediate and sustained changes in level: different latencies (degrees of delay) of effect, abrupt versus gradual effects, and different durations of the effect (permanent or temporary). There may be reasonable expectations regarding the kind of effect (abrupt or gradual), entailing a different focal data feature (e.g., level or slope). However, the exact amount of latency and the exact duration of a temporary effect may not be known a priori, justifying an exploratory approach studying the effect of specifying different latencies or delayed effects and different durations for temporary effects. We provide illustrations of the proposal with real data, and we present a user-friendly freely available web application implementing it.

Typicality of Level Change (TLC) as an Additional Effect Measure to NAP and Tau-U in Single Case Research

Single case research is a viable way to obtain evidence for social and psychological interventions on an individual level. Across single case research studies various analysis strategies are employed, varying from visual analysis to the calculation of effect sizes. To calculate effect sizes in studies with few measurements per time period (<40 data points with a minimum of five data points in each phase), non-parametric indices such as Nonoverlap of All Pairs (NAP) and Tau-U are recommended. However, both indices have restrictions. This article discusses the restrictions of NAP and Tau-U and presents the description, calculation, and benefits of an additional effect size, called the Typicality of Level Change (TLC) index. In comparison to NAP and Tau-U, the TLC index is more aligned to visual analysis, not restricted by a ceiling effect, and does not overcompensate for problematic trends in data. The TLC index is also sensitive to the typicality of an effect. TLC is an important addition to ease the restrictions of current nonoverlap methods when comparing effect sizes between cases and studies.

Acceptance and commitment therapy for individuals with depressive and anxiety symptoms following acquired brain injury: A non-concurrent multiple baseline design across four cases

Patients with acquired brain injury (ABI) often experience symptoms of anxiety and depression. Until now, evidence-based treatment is scarce. This study aimed to investigate the effectiveness of Acceptance and Commitment Therapy (ACT) for patients with ABI. To evaluate the effect of ACT for people with ABI, a non-concurrent multiple baseline design across four cases was used. Participants were randomly assigned to a baseline period, followed by treatment and then follow-up phases. Anxiety and depressive symptoms were repeatedly measured. During six measurement moments over a year, participants filled in questionnaires measuring anxiety, depression, stress, participation, quality of life, and ACT-related processes. Randomization tests and NAP scores were used to calculate the level of change across phases. Clinically significant change was defined with the Reliable Change Index. Three out of four participants showed medium to large decreases in anxiety and depressive symptoms (NAP = 0.85 till 0.99). Furthermore, participants showed improvements regarding stress, cognitive fusion, and quality of life. There were no improvements regarding psychological flexibility, value-driven behaviour, or social participation. This study shows that ACT is possibly an effective treatment option for people experiencing ABI-related anxiety and depression symptoms. Replication with single case or large scale group studies is needed to confirm these findings.

Single-case experimental designs for child neurological rehabilitation and developmental disability research

Single-case experimental designs (SCEDs) are a group of methodologies of growing interest, aiming to test the effectiveness of an intervention at the single-participant level, using a rigorous and prospective methodology. SCEDs may promote flexibility on how we design research protocols and inform clinical decision-making, especially for personalized outcome measures, inclusion of families with challenging needs, measurement of children's progress in relation to parental implementation of interventions, and focus on personal goals. Design options for SCEDs are discussed in relation to an expected on/off effect of the intervention (e.g. school/environmental adaptation, assistive technology devices) or, alternatively, on an expected carry-on/maintenance of effects (interventions aiming to develop or restore a function). Randomization in multiple-baseline designs and 'power' calculations are explained. The most frequent reasons for not detecting an intervention effect in SCEDs are also presented, especially in relation to baseline length, trend, and instability. The use of SCEDs on the front and back ends of randomized controlled trials is discussed.

Emotionally focused couple therapy in cancer survivor couples with marital and sexual problems: a replicated single-case experimental design

Objective

The current research examined the effect of Emotionally Focused Couples Therapy (EFCT) on perceived intimacy, affect, and dyadic connection in cancer survivor couples with relationship challenges.

Method

In this longitudinal replicated single-case study, positive and negative affect, intimacy, partner responsiveness, and expression of attachment-based emotional needs were reported every 3 days before and during treatment. Thirteen couples, with one partner having survived colorectal cancer or breast cancer, participated for the full duration of the study. Statistical analysis of the data was performed using randomization tests, piecewise regression, and multilevel analyses.

Results

Adherence to the therapeutic protocol was tested and found adequate. Compared with baseline, significant positive effects on affect variables were found during the therapeutic process. Positive affect increased and negative affect decreased. Partner responsiveness, perceived intimacy, and the expression of attachment-based emotional needs improved, but only in the later phase of treatment. Results at the group level were statistically significant, whereas effects at the individual level were not.

Discussion

This study found positive group-level effects of EFCT on affect and dyadic outcome measures in cancer survivors. The positive results warrant further research, including randomized clinical trials, to replicate these effects of EFCT in cancer survivor couples experiencing marital and sexual problems.

Defining and assessing immediacy in single-case experimental designs

Immediacy is one of six data aspects (alongside level, trend, variability, overlap, and consistency) that has to be accounted for when visually analyzing single-case data. Given that it is one of the aspects that has received considerably less attention than other data aspects, the current text offers a review of the proposed conceptual definitions of immediacy (i.e., what it refers to) and also of the suggested operational definitions (i.e., how exactly is it assessed and/or quantified). Provided that a variety of conceptual and operational definitions is identified, we propose following a sensitivity analysis using a randomization test for assessing immediate effects in single-case experimental designs, by identifying when changes were most clear. In such a sensitivity analysis, the immediate effects are tested for multiple possible intervention points and for different possible operational definitions. Robust immediate effects can be detected if the results for the different operational definitions converge.

A proposal for the assessment of replication of effects in single-case experimental designs

In science in general and in the context of single‐case experimental designs, replication of the effects of the intervention within and/or across participants or experiments is crucial for establishing causality and for assessing the generality of the intervention effect. Specific developments and proposals for assessing whether an effect has been replicated or not (or to what extent) are scarce, in the general context of behavioral sciences, and practically null in the single‐case experimental designs context. We propose an extension of the modified Brinley plot for assessing how many of the effects replicate. To make this assessment possible, a definition of replication is suggested, on the basis of expert judgment, rather than on statistical criteria. The definition of replication and its graphical representation are justified, presenting their strengths and limitations, and illustrated with real data. A user‐friendly software is made available for obtaining automatically the graphical representation.

Applied hybrid single-case experiments published between 2016 and 2020: A systematic review

Single-case experimental designs (SCEDs) are frequently used research designs in psychology, (special) education, and related fields. Hybrid designs are formed by combining two or more of the basic SCED forms (i.e. phase designs, alternation designs, multiple baseline designs, and changing criterion designs). Hybrid designs have the potential to tackle complex research questions and increase internal validity, but relatively little is known about their use in actual research practice. Therefore, we systematically reviewed SCED hybrid designs published between 2016 and 2020. The systematic review of 67 studies indicates that a hybrid of phase designs and multiple baseline designs is most popular. Hybrid designs are most frequently analyzed by means of visual analysis paired with descriptive statistics. Randomization in the study design is common only for one particular kind of hybrid design. Examples of hybrid studies reveal that these designs are particularly popular in educational research. We compare some of the results of the systematic review to those obtained by Hammond and Gast, Shadish and Sullivan, and Tanious and Onghena. Finally, we discuss the results of the present systematic review in light of the need for specific guidelines for hybrid designs, including analytical methods, design specific randomization and reporting, and the need for terminological clarification.

Optimal number of baseline sessions before changing phases within single-case experimental designs

Recommendations vary considerably for the minimum or optimal number of baseline sessions to conduct within single-case experimental design clinical analyses or research studies. We examined the optimal number of baseline sessions that produced minimal bias. First, we examined the relation between the number of baseline sessions and the degree of bias in calculating estimates of treatment effect size. As the number of baseline sessions increased, the bias in effect size estimates decreased, r = -0.36, p < 0.001. s, we examined what would be the minimum number of baseline sessions associated with varying levels of bias. Bias of approximately ten percent was associated with four to five baseline sessions. Bias of about five percent was associated with six to seven baseline sessions. Third, we examined the relation between standard deviation and varying levels of bias. As the number of baseline sessions increases, the standard deviation for the phase decreased, r = -0.89, p < 0.001. Fourth, we examined what value of standard deviation in the baseline phase was associated with equal to or more than five versus ten percent bias. When considering five or ten percent bias, the optimal level of standard deviation was 0.59 or less.

A systematic review of applied single-case research published between 2016 and 2018: Study designs, randomization, data aspects, and data analysis

Single-case experimental designs (SCEDs) have become a popular research methodology in educational science, psychology, and beyond. The growing popularity has been accompanied by the development of specific guidelines for the conduct and analysis of SCEDs. In this paper, we examine recent practices in the conduct and analysis of SCEDs by systematically reviewing applied SCEDs published over a period of three years (2016-2018). Specifically, we were interested in which designs are most frequently used and how common randomization in the study design is, which data aspects applied single-case researchers analyze, and which analytical methods are used. The systematic review of 423 studies suggests that the multiple baseline design continues to be the most widely used design and that the difference in central tendency level is by far most popular in SCED effect evaluation. Visual analysis paired with descriptive statistics is the most frequently used method of data analysis. However, inferential statistical methods and the inclusion of randomization in the study design are not uncommon. We discuss these results in light of the findings of earlier systematic reviews and suggest future directions for the development of SCED methodology.

An unperceived acoustic stimulus decreases reaction time to visual information in a patient with cortical deafness

Responding to multiple stimuli of different modalities has been shown to reduce reaction time (RT), yet many different processes can potentially contribute to multisensory response enhancement. To investigate the neural circuits involved in voluntary response initiation, an acoustic stimulus of varying intensities (80, 105, or 120 dB) was presented during a visual RT task to a patient with profound bilateral cortical deafness and an intact auditory brainstem response. Despite being unable to consciously perceive sound, RT was reliably shortened (~100 ms) on trials where the unperceived acoustic stimulus was presented, confirming the presence of multisensory response enhancement. Although the exact locus of this enhancement is unclear, these results cannot be attributed to involvement of the auditory cortex. Thus, these data provide new and compelling evidence that activation from subcortical auditory processing circuits can contribute to other cortical or subcortical areas responsible for the initiation of a response, without the need for conscious perception.

Randomized Single-Case Experimental Designs in Healthcare Research: What, Why, and How?

Health problems are often idiosyncratic in nature and therefore require individualized diagnosis and treatment. In this paper, we show how single-case experimental designs (SCEDs) can meet the requirement to find and evaluate individually tailored treatments. We give a basic introduction to the methodology of SCEDs and provide an overview of the available design options. For each design, we show how an element of randomization can be incorporated to increase the internal and statistical conclusion validity and how the obtained data can be analyzed using visual tools, effect size measures, and randomization inference. We illustrate each design and data analysis technique using applied data sets from the healthcare literature.

Assessing Consistency in Single-Case A-B-A-B Phase Designs

Previous research has introduced several effect size measures (ESMs) to quantify data aspects of single-case experimental designs (SCEDs): level, trend, variability, overlap, and immediacy. In the current article, we extend the existing literature by introducing two methods for quantifying consistency in single-case A-B-A-B phase designs. The first method assesses the consistency of data patterns across phases implementing the same condition, called CONsistency of DAta Patterns (CONDAP). The second measure assesses the consistency of the five other data aspects when changing from baseline to experimental phase, called CONsistency of the EFFects (CONEFF). We illustrate the calculation of both measures for four A-B-A-B phase designs from published literature and demonstrate how CONDAP and CONEFF can supplement visual analysis of SCED data. Finally, we discuss directions for future research.

Results reporting in single case experiments and single case meta-analysis

Single Case Experimental Design is a discipline grounded in applied behavior analysis where the needs of individual clients and the application of scientific inquiry are fundamental tenets. These two principles remain tantamount in the conduct of research using this methodology and the expansion of the method into evidence-based practice determinations. Although recommendations for quality indicators are widespread, implementation is not. Concurrent to the rise of quality indicators is an increasing interest in analysis methodology. Visual analysis has a history of application and validity, newer forms of analysis less so. While some argue for concordance between the two, it may be the differences that are worth exploration in understanding characteristics of trend and variability in much of the published literature. Design choice and visual analysis decisions are rarely fully articulated. Statistical analyses are likewise inadequately justified or described. Recommendations for the explicit language of reporting as derived from prior meta-analysis and a current review of two leading journals provides a scaffold consistent with existing guidelines but additive in detail, exemplars, and justification. This is intended to improve reporting of results for individual studies and their potential use in future meta-analytic work.

Quantifying differences between conditions in single-case designs: Possible analysis and meta-analysis

The current paper is a call for and illustration of a way of closing the gap between basic research and professional practice in the field of neurorehabilitation. Methodologically, single-case experimental designs and the guidelines created regarding their conduct are highlighted. Statistically, we review two data analytical options, namely (a) indices quantifying the difference between pairs of conditions in the same metric as the target behavior and (b) a formal statistical procedure offering a standardized overall quantification. The paper provides guidance in the analysis and suggests free software in order to illustrate, in the context of data from behavioral interventions with children with developmental disorders, that informative analyses are feasible. We also show how the results of individual studies can be made eligible for meta-analyses, which are useful for establishing the evidence basis of interventions. Nevertheless, we also point at decisions that need to be made during the process of data analysis.

Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide

Single-case experimental designs (SCED) are experimental designs aiming at testing the effect of an intervention using a small number of patients (typically one to three), using repeated measurements, sequential (±randomized) introduction of an intervention and method-specific data analysis, including visual analysis and specific statistics. The aim of this paper is to familiarise professionals working in different fields of rehabilitation with SCEDs and provide practical advice on how to design and implement a SCED in clinical rehabilitation practice. Research questions suitable for SCEDs and the different types of SCEDs (e.g., alternating treatment designs, introduction/withdrawal designs and multiple baseline designs) are reviewed. Practical steps in preparing a SCED design are outlined. Examples from different rehabilitation domains are provided throughout the paper. Challenging issues such as the choice of the repeated measure, assessment of generalisation, randomization, procedural fidelity, replication and generalizability of findings are discussed. Simple rules and resources for data analysis are presented. The utility of SCEDs in physical and rehabilitation medicine (PRM) are discussed.

Analytical Options for Single-Case Experimental Designs: Review and Application to Brain Impairment

Single-case experimental designs meeting evidence standards are useful for identifying empirically-supported practices. Part of the research process entails data analysis, which can be performed both visually and numerically. In the current text, we discuss several statistical techniques focusing on the descriptive quantifications that they provide on aspects such as overlap, difference in level and in slope. In both cases, the numerical results are interpreted in light of the characteristics of the data as identified via visual inspection. Two previously published data sets from patients with traumatic brain injury are re-analysed, illustrating several analytical options and the data patterns for which each of these analytical techniques is especially useful, considering their assumptions and limitations. In order to make the current review maximally informative for applied researchers, we point to free user-friendly web applications of the analytical techniques. Moreover, we offer up-to-date references to the potentially useful analytical techniques not illustrated in the article. Finally, we point to some analytical challenges and offer tentative recommendations about how to deal with them.

Recommendations for Choosing Single-Case Data Analytical Techniques

The current paper responds to the need to provide guidance to applied single-case researchers regarding the possibilities of data analysis. The amount of available single-case data analytical techniques has been growing during recent years and a general overview, comparing the possibilities of these techniques, is missing. Such an overview is provided that refers to techniques that yield results in terms of a raw or standardized difference and procedures related to regression analysis, as well as nonoverlap and percentage change indices. The comparison is provided in terms of the type of quantification provided, data features taken into account, conditions in which the techniques are appropriate, possibilities for meta-analysis, and evidence available on their performance. Moreover, we provide a set of recommendations for choosing appropriate analysis techniques, pointing at specific situations (aims, types of data, researchers' resources) and the data analytical techniques that are most appropriate in these situations. The recommendations are contextualized using a variety of published single-case data sets in order to illustrate a range of realistic situations that researchers have faced and may face in their investigations.

How Can Single-Case Data Be Analyzed? Software Resources, Tutorial, and Reflections on Analysis

The present article aims to present a series of software developments in the quantitative analysis of data obtained via single-case experimental designs (SCEDs), as well as the tutorial describing these developments. The tutorial focuses on software implementations based on freely available platforms such as R and aims to bring statistical advances closer to applied researchers and help them become autonomous agents in the data analysis stage of a study. The range of analyses dealt with in the tutorial is illustrated on a typical single-case dataset, relying heavily on graphical data representations. We illustrate how visual and quantitative analyses can be used jointly, giving complementary information and helping the researcher decide whether there is an intervention effect, how large it is, and whether it is practically significant. To help applied researchers in the use of the analyses, we have organized the data in the different ways required by the different analytical procedures and made these data available online. We also provide Internet links to all free software available, as well as all the main references to the analytical techniques. Finally, we suggest that appropriate and informative data analysis is likely to be a step forward in documenting and communicating results and also for increasing the scientific credibility of SCEDs.

Using Single-Case Experiments to Support Evidence-Based Decisions: How Much Is Enough?

For practitioners, the use of single-case experimental designs (SCEDs) in the research literature raises an important question: How many single-case experiments are enough to have sufficient confidence that an intervention will be effective with an individual from a given population? Although standards have been proposed to address this question, current guidelines do not appear to be strongly grounded in theory or empirical research. The purpose of our article is to address this issue by presenting guidelines to facilitate evidence-based decisions by adopting a simple statistical approach to quantify the support for interventions that have been validated using SCEDs. Specifically, we propose the use of success rates as a supplement to support evidence-based decisions. The proposed methodology allows practitioners to aggregate the results from single-case experiments to estimate the probability that a given intervention will produce a successful outcome. We also discuss considerations and limitations associated with this approach.

Further developments in summarising and meta-analysing single-case data: An illustration with neurobehavioural interventions in acquired brain injury

Data analysis for single-case designs is an issue that has prompted many researchers to propose a variety of alternatives, including use of randomisation tests, regression-based procedures, and standardised mean difference. Another option consists in computing unstandardised or raw differences between conditions: the changes in slope and in level, or the difference between the projected baseline (including trend) and the actual treatment phase measurements. Apart from the strengths of these procedures (potentially easier interpretation clinically, separate estimations and an overall quantification of effects, reasonable performance), they require further development, such as (a) creating extensions for dealing with methodologically strong designs such as multiple baseline, (b) achieving comparability across studies and making possible meta-analytical integrations, and (c) implementing software for the extensions. The proposals are illustrated herein in the context of a meta-analysis of 28 studies on (neuro)behavioural interventions in adults who have challenging behaviours after acquired brain injury.

Probability and Visual Aids for Assessing Intervention Effectiveness in Single-Case Designs

Single-case data analysis still relies heavily on visual inspection, and, at the same time, it is not clear to what extent the results of different quantitative procedures converge in identifying an intervention effect and its magnitude when applied to the same data; this is the type of evidence provided here for two procedures. One of the procedures, included due to the importance of providing objective criteria to visual analysts, is a visual aid fitting and projecting split-middle trend while taking into account data variability. The other procedure converts several different metrics into probabilities making their results comparable. In the present study, we expore to what extend these two procedures coincide in the magnitude of intervention effect taking place in a set of studies stemming from a recent meta-analysis. The procedures concur to a greater extent with the values of the indices computed and with each other and, to a lesser extent, with our own visual analysis. For distinguishing smaller from larger effects, the probability-based approach seems somewhat better suited. Moreover, the results of the field test suggest that the latter is a reasonably good mechanism for translating different metrics into similar labels. User friendly R code is provided for promoting the use of the visual aid, together with a quantification based on nonoverlap and the label provided by the probability approach.

Single-case experimental designs: reflections on conduct and analysis

In this editorial discussion we reflect on the issues addressed by, and arising from, the papers in this special issue on Single-Case Experimental Design (SCED) study methodology. We identify areas of consensus and disagreement regarding the conduct and analysis of SCED studies. Despite the long history of application of SCEDs in studies of interventions in clinical and educational settings, the field is still developing. There is an emerging consensus on methodological quality criteria for many aspects of SCEDs, but disagreement on what are the most appropriate methods of SCED data analysis. Our aim is to stimulate this ongoing debate and highlight issues requiring further attention from applied researchers and methodologists. In addition we offer tentative criteria to support decision-making in relation to the selection of analytical techniques in SCED studies. Finally, we stress that large-scale interdisciplinary collaborations, such as the current Special Issue, are necessary if SCEDs are going to play a significant role in the development of the evidence base for clinical practice.

Single case experimental designs: introduction to a special issue of Neuropsychological Rehabilitation

This paper introduces the Special Issue of Neuropsychological Rehabilitation on Single Case Experimental Design (SCED) methodology. SCED studies have a long history of use in evaluating behavioural and psychological interventions, but in recent years there has been a resurgence of interest in SCED methodology, driven in part by the development of standards for conducting and reporting SCED studies. Although there is consensus on some aspects of SCED methodology, the question of how SCED data should be analysed remains unresolved. This Special Issues includes two papers discussing aspects of conducting SCED studies, five papers illustrating use of SCED methodology in clinical practice, and nine papers that present different methods of SCED data analysis. A final Discussion paper summarises points of agreement, highlights areas where further clarity is needed, and ends with a set of resources that will assist researchers conduct and analyse SCED studies.