1
|
Manolov R, Onghena P. Testing delayed, gradual, and temporary treatment effects in randomized single-case experiments: A general response function framework. Behav Res Methods 2024; 56:3915-3936. [PMID: 37749426 PMCID: PMC11133040 DOI: 10.3758/s13428-023-02230-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/29/2023] [Indexed: 09/27/2023]
Abstract
Randomization tests represent a class of significance tests to assess the statistical significance of treatment effects in randomized single-case experiments. Most applications of single-case randomization tests concern simple treatment effects: immediate, abrupt, and permanent changes in the level of the outcome variable. However, researchers are confronted with delayed, gradual, and temporary treatment effects; in general, with "response functions" that are markedly different from single-step functions. We here introduce a general framework that allows specifying a test statistic for a randomization test based on predicted response functions that is sensitive to a wide variety of data patterns beyond immediate and sustained changes in level: different latencies (degrees of delay) of effect, abrupt versus gradual effects, and different durations of the effect (permanent or temporary). There may be reasonable expectations regarding the kind of effect (abrupt or gradual), entailing a different focal data feature (e.g., level or slope). However, the exact amount of latency and the exact duration of a temporary effect may not be known a priori, justifying an exploratory approach studying the effect of specifying different latencies or delayed effects and different durations for temporary effects. We provide illustrations of the proposal with real data, and we present a user-friendly freely available web application implementing it.
Collapse
Affiliation(s)
- Rumen Manolov
- Department of Social Psychology and Quantitative Psychology, Faculty of Psychology, University of Barcelona, Passeig de la Vall d'Hebron 171, 08035, Barcelona, Spain.
| | - Patrick Onghena
- Faculty of Psychology and Educational Sciences, Methodology of Educational Sciences Research Group, KU Leuven, Tiensestraat 102, 3000, Leuven, Belgium
| |
Collapse
|
2
|
Manolov R. Does the choice of a linear trend-assessment technique matter in the context of single-case data? Behav Res Methods 2023; 55:4200-4221. [PMID: 36622560 DOI: 10.3758/s13428-022-02013-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/22/2022] [Indexed: 01/10/2023]
Abstract
Trend is one of the data aspects that is an object of assessment in the context of single-case experimental designs. This assessment can be performed both visually and quantitatively. Given that trend, just like other relevant data features such as level, immediacy, or overlap does not have a single operative definition, a comparison among the existing alternatives is necessary. Previous studies have included illustrations of differences between trend-line fitting techniques using real data. In the current study, I carry out a simulation to study the degree to which different trend-line fitting techniques lead to different degrees of bias, mean square error, and statistical power for a variety of quantifications that entail trend lines. The simulation involves generating both continuous and count data, for several phase lengths, degrees of autocorrelation, and effect sizes (change in level and change in slope). The results suggest that, in general, ordinary least squares estimation performs well in terms of relative bias and mean square error. Especially, a quantification of slope change is associated with better statistical results than quantifying an average difference between conditions on the basis of a projected baseline trend. In contrast, the performance of the split-middle (bisplit) technique is less than optimal.
Collapse
Affiliation(s)
- Rumen Manolov
- Department of Social Psychology and Quantitative Psychology, Faculty of Psychology, University of Barcelona, Passeig de la Vall d'Hebron 171, 08035, Barcelona, Spain.
| |
Collapse
|
3
|
De TK, Onghena P. The randomized marker method for single-case randomization tests: Handling data missing at random and data missing not at random. Behav Res Methods 2022; 54:2905-2938. [PMID: 35132582 DOI: 10.3758/s13428-021-01781-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/22/2021] [Indexed: 12/16/2022]
Abstract
Single-case experiments are frequently plagued by missing data problems. In a recent study, the randomized marker method was found to be valid and powerful for single-case randomization tests when the missing data were missing completely at random. However, in real-life experiments, it is difficult for researchers to ascertain the missing data mechanism. For analyzing such experiments, it is essential that the missing data handling method is valid and powerful for various missing data mechanisms. Hence, we examined the performance of the randomized marker method for data that are missing at random and data that are missing not at random. In addition, we compared the randomized marker method with multiple imputation, because the latter is often considered the gold standard among imputation techniques. To compare and evaluate these two methods under various simulation conditions, we calculated the type I error rate and statistical power in single-case randomization tests using these two methods of handling missing data and compared them to the type I error rate and statistical power using complete datasets. The results indicate that while multiple imputation presents an advantage in the presence of strongly correlated covariate data, the randomized marker method remains valid and results in sufficient statistical power for most of the missing data conditions simulated in this study.
Collapse
Affiliation(s)
- Tamal Kumar De
- Methodology of Educational Sciences Research Group, Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102 Box 3762, 3000, Leuven, Belgium.
| | - Patrick Onghena
- Methodology of Educational Sciences Research Group, Faculty of Psychology and Educational Sciences, KU Leuven, Tiensestraat 102 Box 3762, 3000, Leuven, Belgium
| |
Collapse
|
4
|
Manolov R, Onghena P. Defining and assessing immediacy in single-case experimental designs. J Exp Anal Behav 2022; 118:462-492. [PMID: 36106573 PMCID: PMC9825864 DOI: 10.1002/jeab.799] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Revised: 08/24/2022] [Accepted: 08/28/2022] [Indexed: 01/11/2023]
Abstract
Immediacy is one of six data aspects (alongside level, trend, variability, overlap, and consistency) that has to be accounted for when visually analyzing single-case data. Given that it is one of the aspects that has received considerably less attention than other data aspects, the current text offers a review of the proposed conceptual definitions of immediacy (i.e., what it refers to) and also of the suggested operational definitions (i.e., how exactly is it assessed and/or quantified). Provided that a variety of conceptual and operational definitions is identified, we propose following a sensitivity analysis using a randomization test for assessing immediate effects in single-case experimental designs, by identifying when changes were most clear. In such a sensitivity analysis, the immediate effects are tested for multiple possible intervention points and for different possible operational definitions. Robust immediate effects can be detected if the results for the different operational definitions converge.
Collapse
Affiliation(s)
- Rumen Manolov
- Department of Social Psychology and Quantitative Psychology, Faculty of PsychologyUniversity of Barcelona
| | - Patrick Onghena
- Faculty of Psychology and Educational Sciences, Methodology of Educational Sciences Research GroupKU Leuven – University of LeuvenLeuvenBelgium
| |
Collapse
|
5
|
Kratochwill TR, Horner RH, Levin JR, Machalicek W, Ferron J, Johnson A. Single-case design standards: An update and proposed upgrades. J Sch Psychol 2021; 89:91-105. [PMID: 34836578 DOI: 10.1016/j.jsp.2021.10.006] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2020] [Revised: 08/17/2021] [Accepted: 10/29/2021] [Indexed: 11/26/2022]
Abstract
In this paper, we provide a critique focused on the What Works Clearinghouse (WWC) Standards for Single-Case Research Design (Standards 4.1). Specifically, we (a) recommend the use of visual-analysis to verify a single-case intervention study's design standards and to examine the study's operational issues, (b) identify limitations of the design-comparable effect-size measure and discuss related statistical matters, (c) review the applicability and practicality of Standards 4.1 to single-case designs (SCDs), and (d) recommend inclusion of content pertaining to diversity, equity, and inclusion in future standards. Within the historical context of the WWC Pilot Standards for Single-Case Design (1.0), we suggest that Standards 4.1 may best serve as standards for meta-analyses of SCDs but will need to make clear distinctions among the various types of SCD studies that are included in any research synthesis. In this regard, we argue for transparency in SCD studies that meet design standards and those that do not meet design standards in any meta-analysis emanating from the WWC. The intent of these recommendations is to advance the science of SCD research both in research synthesis and in promoting evidence-based practices.
Collapse
Affiliation(s)
| | | | | | | | - John Ferron
- University of South Florida, United States of America
| | - Austin Johnson
- University of California, Riverside, United States of America
| |
Collapse
|
6
|
Manolov R, Moeyaert M, Fingerhut JE. A Priori Justification for Effect Measures in Single-Case Experimental Designs. Perspect Behav Sci 2021; 45:153-186. [DOI: 10.1007/s40614-021-00282-2] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/27/2021] [Indexed: 01/31/2023] Open
|
7
|
Abstract
Due to repeated observations of an outcome behavior in N-of-1 or single-case design (SCD) intervention studies, the occurrence of missing scores is inevitable in such studies. Approximately 21% of SCD articles published in five reputable journals between 2015 and 2019 exhibited evidence of missing scores. Missing rates varied by designs, with the highest rate (24%) found in multiple baseline/probe designs. Missing scores cause difficulties in data analysis. And inappropriate treatments of missing scores lead to consequences that threaten internal validity and weaken generalizability of intervention effects reported in SCD research. In this paper, we comprehensively review nine methods for treating missing SCD data: the available data method, six single imputations, and two model-based methods. The strengths, weaknesses, assumptions, and examples of these methods are summarized. The available data method and three single imputation methods are further demonstrated in assessing an intervention effect at the class and students’ levels. Assessment results are interpreted in terms of effect sizes, statistical significances, and visual analysis of data. Differences in results among the four methods are noted and discussed. The extensive review of problems caused by missing scores and possible treatments should empower researchers and practitioners to account for missing scores effectively and to support evidence-based interventions vigorously. The paper concludes with a discussion of contingencies for implementing the nine methods and practical strategies for managing missing scores in single-case intervention studies.
Collapse
|
8
|
Manolov R, Tanious R. Assessing Consistency in Single-Case Data Features Using Modified Brinley Plots. Behav Modif 2020; 46:581-627. [PMID: 33371723 DOI: 10.1177/0145445520982969] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
The current text deals with the assessment of consistency of data features from experimentally similar phases and consistency of effects in single-case experimental designs. Although consistency is frequently mentioned as a critical feature, few quantifications have been proposed so far: namely, under the acronyms CONDAP (consistency of data patterns in similar phases) and CONEFF (consistency of effects). Whereas CONDAP allows assessing the consistency of data patterns, the proposals made here focus on the consistency of data features such as level, trend, and variability, as represented by summary measures (mean, ordinary least squares slope, and standard deviation, respectively). The assessment of consistency of effect is also made in terms of these three data features, while also including the study of the consistency of an immediate effect (if expected). The summary measures are represented as points on a modified Brinley plot and their similarity is assessed via quantifications of distance. Both absolute and relative measures of consistency are proposed: the former expressed in the same measurement units as the outcome variable and the latter as a percentage. Illustrations with real data sets (multiple baseline, ABAB, and alternating treatments designs) show the wide applicability of the proposals. We developed a user-friendly website to offer both the graphical representations and the quantifications.
Collapse
|
9
|
Abstract
Single-case experimental design (SCED) research plays an important role in establishing and confirming evidence-based practices. Due to multiple measures of a target behavior in such studies, missing information is common in their data. The expectation-maximization (EM) algorithm has been successfully applied to deal with missing data in between-subjects designs, but only in a handful of SCED studies. The present study extends the findings from Smith, Borckardt, and Nash (2012) and Velicer and Colby (2005b, Study 2) by systematically examining the performance of EM in a baseline-intervention (or AB) design under various missing rates, autocorrelations, intervention phase lengths, and magnitudes of effects, as well as two fitted models. Three indicators of an intervention effect (baseline slope, level shift, and slope change) were estimated. The estimates' relative bias, root-mean squared error, and relative bias of the estimated standard error were used to assess EM's performance. The findings revealed that autocorrelation impacted the estimates' qualities most profoundly. Autocorrelation interacted with missing rate in impacting the relative bias of the estimates, impacted the root-mean squared error nonlinearly, and interacted with the fitted model in impacting the relative bias of the estimated standard errors. A simpler model without autocorrelation can be used to estimate baseline slope and slope change in time-series data. EM is recommended as a principled method to handle missing data in SCED studies. Two decision trees are presented to assist researchers and practitioners in applying EM. Emerging research directions are identified for treating missing data in SCED studies.
Collapse
|
10
|
Investigation of Single-Case Multiple-Baseline Randomization Tests of Trend and Variability. EDUCATIONAL PSYCHOLOGY REVIEW 2020. [DOI: 10.1007/s10648-020-09549-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
|
11
|
Chen LT, Wu PJ, Peng CYJ. Accounting for baseline trends in intervention studies: Methods, effect sizes, and software. COGENT PSYCHOLOGY 2019. [DOI: 10.1080/23311908.2019.1679941] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022] Open
Affiliation(s)
- Li-Ting Chen
- Counseling and Educational Psychology, University of Nevada, Reno, NV, USA
| | - Po-Ju Wu
- Department of Statistics, Indiana University Bloomington, IN, USA
| | | |
Collapse
|
12
|
Gana K, Gallé-Tessonneau M, Broc G. Le protocole individuel en psychologie : tutoriel à l’usage des psychologues praticiens. PRAT PSYCHOL 2019. [DOI: 10.1016/j.prps.2018.11.002] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
13
|
Onghena P, Tanious R, De TK, Michiels B. Randomization tests for changing criterion designs. Behav Res Ther 2019; 117:18-27. [DOI: 10.1016/j.brat.2019.01.005] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2018] [Revised: 01/07/2019] [Accepted: 01/10/2019] [Indexed: 11/29/2022]
|
14
|
De Young KP, Bottera AR. A summary of reporting guidelines and evaluation domains for using single-case experimental designs and recommendations for the study of eating disorders. Int J Eat Disord 2018; 51:617-628. [PMID: 29846014 DOI: 10.1002/eat.22887] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/15/2018] [Revised: 04/17/2018] [Accepted: 04/18/2018] [Indexed: 11/08/2022]
Abstract
The insights and activities that regularly occur in clinical settings are an underutilized wellspring of potential scientific advances. Single-case experimental designs (SCEDs) include a collection of methods for testing hypotheses in one individual (or a small number of individuals) at a time that are particularly well suited to clinical settings. Their strengths include their potential for maximizing both internal and external validity by way of tight experimental control over threats to internal validity and flexibility in application and inclusion criteria. Despite these advantages, the eating disorder field has not made use of SCEDs to a similar extent as other research areas, providing an opportunity for expansion. Recent advances in organizing SCED standards have resulted in the publication of two sets of reporting guidelines relevant to eating disorders research. These guidelines are integrated and summarized alongside descriptions of domains for evaluating their methodological rigor. Example research questions of potential interest to the eating disorder field are illustrated with SCEDs, and several issues to consider when using SCEDs in eating disorders research are described. It is recommended that researchers aiming to use SCEDs refer to this paper and the resources cited herein when designing and reporting their work. Finally, it is hoped that individuals with access to clinical samples use this paper as a source of inspiration for testing their unique clinical insights.
Collapse
Affiliation(s)
- Kyle P De Young
- Department of Psychology, University of Wyoming, Laramie, Wyoming
| | | |
Collapse
|
15
|
One by One: Accumulating Evidence by using Meta-Analytical Procedures for Single-Case Experiments. BRAIN IMPAIR 2017. [DOI: 10.1017/brimp.2017.25] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
This paper presents a unilevel and multilevel approach for the analysis and meta-analysis of single-case experiments (SCEs). We propose a definition of SCEs and derive the specific features of SCEs’ data that have to be taken into account when analysing and meta-analysing SCEs. We discuss multilevel models of increasing complexity and propose alternative and complementary techniques based on probability combining and randomisation test wrapping. The proposed techniques are demonstrated with real-life data and corresponding R code.
Collapse
|
16
|
Rosenberg DE, Kadokura E, Morris ME, Renz A, Vilardaga RM. Application of N-of-1 Experiments to Test the Efficacy of Inactivity Alert Features in Fitness Trackers to Increase Breaks from Sitting in Older Adults. Methods Inf Med 2017; 56:16-02-0043. [PMID: 28816343 PMCID: PMC5997558 DOI: 10.3414/me16-02-0043] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2016] [Accepted: 07/24/2017] [Indexed: 12/28/2022]
Abstract
BACKGROUND Frequent breaks from sitting could improve health. Many commercially available fitness trackers deliver vibration alerts that could be used to cue sitting breaks. As a potentially pragmatic approach to promote frequent breaks from sitting, we tested the efficacy of inactivity alerts among obese older adults, a highly sedentary population. METHODS We conducted 10 sequential N-of-1 (single-case) experimental ABA trials. Participants (mean age = 68, mean BMI = 35) were monitored for a baseline phase ("A1") followed by an intervention phase ("B"). The intervention was then removed and participants were monitored to test an experimental effect (reversal "A2" phase). Total time in the study was limited to 25 days. During the intervention phase ("B"), participants used fitness trackers to stand up or move every time they received an alert (every 15 or 20 minutes of inactivity). Participants wore activPAL devices to measure breaks from sitting each day. Randomization tests were used to determine whether the number of breaks was significantly higher during the "B" phase than the two "A" phases. RESULTS Breaks were higher by 7.2 breaks per day during the "B" phase compared to the mean of the "A" phases. Seven out of 10 participants had more sitting breaks during the intervention phase which subsequently decreased during the reversal "A2" phase (combined p-value < .05). CONCLUSION Inactivity alert features within commercially available devices are efficacious for promoting modest improvements in breaks from sitting among older adults with obesity and could be a simple health-promoting strategy in this population.
Collapse
Affiliation(s)
- Dori E Rosenberg
- Dori E. Rosenberg, PhD, MPH, Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Suite 1600, Seattle, WA 98101, USA, E-mail:
| | | | | | | | | |
Collapse
|
17
|
McDonald S, Quinn F, Vieira R, O’Brien N, White M, Johnston DW, Sniehotta FF. The state of the art and future opportunities for using longitudinal n-of-1 methods in health behaviour research: a systematic literature overview. Health Psychol Rev 2017; 11:307-323. [DOI: 10.1080/17437199.2017.1316672] [Citation(s) in RCA: 72] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Suzanne McDonald
- Institute of Health & Society, Newcastle University, Newcastle upon Tyne, UK
| | - Francis Quinn
- School of Applied Social Studies, The Robert Gordon University, Aberdeen, UK
| | - Rute Vieira
- Institute of Health & Society, Newcastle University, Newcastle upon Tyne, UK
| | - Nicola O’Brien
- Institute of Health & Society, Newcastle University, Newcastle upon Tyne, UK
| | - Martin White
- UKCRC Centre for Diet and Activity Research (CEDAR), MRC Epidemiology Unit, School of Clinical Medicine, University of Cambridge, Cambridge, UK
| | | | - Falko F. Sniehotta
- Fuse, UKCRC Centre for Translational Research in Public Health, Newcastle University, Newcastle upon Tyne, UK
| |
Collapse
|
18
|
Manolov R, Moeyaert M. Recommendations for Choosing Single-Case Data Analytical Techniques. Behav Ther 2017; 48:97-114. [PMID: 28077224 DOI: 10.1016/j.beth.2016.04.008] [Citation(s) in RCA: 55] [Impact Index Per Article: 7.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/23/2015] [Revised: 04/13/2016] [Accepted: 04/30/2016] [Indexed: 11/29/2022]
Abstract
The current paper responds to the need to provide guidance to applied single-case researchers regarding the possibilities of data analysis. The amount of available single-case data analytical techniques has been growing during recent years and a general overview, comparing the possibilities of these techniques, is missing. Such an overview is provided that refers to techniques that yield results in terms of a raw or standardized difference and procedures related to regression analysis, as well as nonoverlap and percentage change indices. The comparison is provided in terms of the type of quantification provided, data features taken into account, conditions in which the techniques are appropriate, possibilities for meta-analysis, and evidence available on their performance. Moreover, we provide a set of recommendations for choosing appropriate analysis techniques, pointing at specific situations (aims, types of data, researchers' resources) and the data analytical techniques that are most appropriate in these situations. The recommendations are contextualized using a variety of published single-case data sets in order to illustrate a range of realistic situations that researchers have faced and may face in their investigations.
Collapse
|
19
|
Manolov R, Moeyaert M. How Can Single-Case Data Be Analyzed? Software Resources, Tutorial, and Reflections on Analysis. Behav Modif 2016; 41:179-228. [DOI: 10.1177/0145445516664307] [Citation(s) in RCA: 39] [Impact Index Per Article: 4.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
The present article aims to present a series of software developments in the quantitative analysis of data obtained via single-case experimental designs (SCEDs), as well as the tutorial describing these developments. The tutorial focuses on software implementations based on freely available platforms such as R and aims to bring statistical advances closer to applied researchers and help them become autonomous agents in the data analysis stage of a study. The range of analyses dealt with in the tutorial is illustrated on a typical single-case dataset, relying heavily on graphical data representations. We illustrate how visual and quantitative analyses can be used jointly, giving complementary information and helping the researcher decide whether there is an intervention effect, how large it is, and whether it is practically significant. To help applied researchers in the use of the analyses, we have organized the data in the different ways required by the different analytical procedures and made these data available online. We also provide Internet links to all free software available, as well as all the main references to the analytical techniques. Finally, we suggest that appropriate and informative data analysis is likely to be a step forward in documenting and communicating results and also for increasing the scientific credibility of SCEDs.
Collapse
|