Practical implications of data reliability and treatment integrity monitoring

A Practitioner's Guide to Measuring Procedural Fidelity

Ensuring high levels of procedural fidelity during behavior-analytic interventions is a crucial component of providing effective behavior-analytic services. However, few resources are available to help guide practitioners through measuring procedural fidelity. In fact, most published behavior-analytic research on procedural fidelity analyzes a single treatment procedure, which might not completely reflect the process of monitoring and addressing the procedural fidelity of a robust treatment package that might be necessary in clinical settings. The purpose of this article is to guide behavior analysts through the process of creating and using procedural fidelity measurement systems, with a focus on direct observation of implementation as a means of fidelity data collection. This process consists of six steps: (1) task analyze treatment procedures into measurable units; (2) assign measures to each treatment component; (3) plan the direct observation; (4) collect procedural fidelity data; (5) analyze and interpret procedural fidelity data; and (6) take action to improve procedural fidelity. Each step is described and discussed in the article.

Comparison of Enhanced and Standard Data Sheets on Treatment Fidelity and Data Collection for Tact Training

Specifically designed data sheets have been recommended to assist with the fidelity of implementation of treatment procedures. The present study extended previous research (e.g., Bottini et al. Behavior Analysis: Research & Practice 21(2), 140-152, 2021; LeBlanc et al. Behavior Analysis in Practice 13(1), 53-62, 2020) by comparing an enhanced data sheet (i.e., the inclusion of randomized targets, prompts for treatment components of securing attending and reinforcement) to a standard data sheet (i.e., targets not preset, no prompts for treatment components) on the fidelity of tact training of features. Ten behavior therapists participated in each condition (n = 20). Participants first watched a brief instructional video explaining the teaching procedure and their assigned data sheet, followed by conducting a treatment session with a confederate serving as a child with autism spectrum disorder. The enhanced data sheet resulted in higher fidelity on multiple variables including randomizing of targets and data collection.

Tutorial: Lessons Learned for Behavior Analysts from Data Scientists

Big data is a computing term used to refer to large and complex data sets, typically consisting of terabytes or more of diverse data that is produced rapidly. The analysis of such complex data sets requires advanced analysis techniques with the capacity to identify patterns and abstract meanings from the vast data. The field of data science combines computer science with mathematics/statistics and leverages artificial intelligence, in particular machine learning, to analyze big data. This field holds great promise for behavior analysis, where both clinical and research studies produce large volumes of diverse data at a rapid pace (i.e., big data). This article presents basic lessons for the behavior analytic researchers and clinicians regarding integration of data science into the field of behavior analysis. We provide guidance on how to collect, protect, and process the data, while highlighting the importance of collaborating with data scientists to select a proper machine learning model that aligns with the project goals and develop models with input from human experts. We hope this serves as a guide to support the behavior analysts interested in the field of data science to advance their practice or research, and helps them avoid some common pitfalls.

Effects of omission and commission errors during tact instruction

The purpose of the current study was to compare the effects of omission and commission errors of reinforcement during tact instruction via telehealth with three children, 6 to 7 years of age, who were diagnosed with an autism spectrum disorder. We used an adapted alternating treatment design to evaluate skill acquisition of target stimuli across high-integrity, commission errors, and omission errors conditions. The high-integrity condition produced mastery criteria in fewer sessions compared with the integrity-error conditions in four of six comparisons, and the omission condition reached mastery criteria in fewer sessions than the commission condition in five of six comparisons.

A detailed examination of reporting procedural fidelity in the Journal of Applied Behavior Analysis

Few reviews on procedural fidelity-the degree to which procedures are implemented as designed-provide details to gauge the quality of fidelity reporting in behavior-analytic research. This review focused on experiments in the Journal of Applied Behavior Analysis (2006-2021) with "integrity" or "fidelity" in the abstract or body. When fidelity data were collected, the coders characterized measurement details (e.g., description of calculation, report of single or multiple values, frequency of fidelity checks, checklist use). The researchers found increasing trends in describing the calculation(s), reporting multiple values, and stating the frequency of measurement. Few studies described using a checklist. Most studies reported fidelity as a percentage, with high obtained values (M = 97%). When not collecting fidelity data was stated as a limitation, authors were unlikely to provide a rationale for the omission. We discuss recommendations for reporting procedural fidelity to increase the quality of and transparency in behavior-analytic research.

Peter CC, Feng J, Long BP. Trends in Reporting Procedural Integrity: A Comparison

Procedural integrity refers to the extent to which an independent variable is implemented as described. Measuring procedural integrity is one important factor when considering internal and external validity of experiments. Experimental articles in behavior-analytic journals have rarely reported procedural-integrity data. The purpose of this study was to update previous reviews of whether articles published in the Journal of Applied Behavior Analysis reported procedural integrity, spanning a period from 1980 to 2020, and compare reporting in JABA to recent reviews of studies published in Behavior Analysis in Practice (2008-2019) and the Journal of Organizational Behavior Management (2000-2020). Procedural integrity continues to be underreported across all three journals, but an increasing trend in reporting procedural integrity is evident in the Journal of Applied Behavior Analysis and Behavior Analysis in Practice. In addition to our recommendations and implications for research and practice, we provide examples and resources to assist researchers and practitioners with recording and reporting integrity data.

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

Outcomes Accountability Systems for Early Childhood Disruptive Behaviors: A Scoping Review of Availability

Early childhood disruptive behaviors are common mental health problems among American youth, and if poorly-managed, pose costly psychological and societal burdens. Outcomes accountability systems in clinical practice are vital opportunities to optimize early intervention for common mental health problems; however, such systems seem rare. A scoping review was conducted to summarize the current availability of outcomes accountability systems in clinical programs addressing early childhood disruptive behaviors, particularly in the US. We used PsycINFO to identify peer-reviewed literature published in English from 2005 to 2021, from which we selected 23 publications from the US, UK, and Netherlands on outcomes accountability systems within clinical programs treating common childhood mental health problems. Only 3 out of 23 publications described outcomes accountability efforts specifically for early childhood problems. Within the 3 studies, only one UK-based study specifically targeted early childhood disruptive behaviors. We did not find publications specifically describing outcomes accountability efforts in US-based clinical programs to treat early childhood disruptive behaviors. There are multi-level challenges preventing changes to the prevalent US model of paying a fee for each unit of child mental healthcare, with little regard for patient outcomes. However, opportunities exist to improve US-based accountability efforts; from top-down expansion of financial incentives, accountability initiatives, and PDT evidence-based practices to an iterative, bottom-up development of meaningful outcomes measurement by providers. Greater adoption of outcomes monitoring in US clinical practice for common mental health problems can optimize management of early childhood disruptive behaviors and mitigate long-term societal and economic burdens.

Nominally acceptable integrity failures negatively affect interventions involving intermittent reinforcement

The finding that differential reinforcement of alternative behavior (DRA) is efficacious at 80% integrity when continuous reinforcement is programmed for alternative responding may have contributed to a perception that integrity at 80% or above is acceptable. However, research also suggests that other interventions (e.g., noncontingent reinforcement) may not remain effective at 80% integrity. The conditions under which 80% integrity is acceptable for common behavioral interventions remains unclear. Therefore, we conducted two human-operant studies to evaluate effects of 80% integrity for interventions with contingent or noncontingent intermittent reinforcement schedules. During Experiment 1, we compared noncontingent reinforcement (NCR) and DRA when implemented with 80% integrity. During Experiment 2, we compared 2 variations of DRA, which included either a ratio or interval schedule to reinforce alternative behavior. Results replicated previous research showing that DRA with a FR-1 schedule programmed for alternative responding resulted in consistent target response suppression, even when integrity was reduced to 80%. In contrast, neither NCR nor interval-based DRA were consistently effective when implemented at 80% integrity. These results demonstrate that 80% integrity is not a uniformly acceptable minimal level of integrity.

Strengthening the procedural fidelity research-to-practice loop in animal behavior

Procedural fidelity is the extent to which components of an intervention are implemented as designed. Procedural fidelity is measured as a dependent variable and manipulated as an independent variable. In research and practice, procedural-fidelity data should be collected, monitored, and reported. Procedural fidelity as an independent variable has been investigated in humans using parametric analyses, and the current article summarizes some of the research conducted on the effects of procedural-fidelity errors in behavior-reduction and skill-acquisition interventions. Connections were drawn to applied animal researchers and the work of animal behavior practitioners to produce implications for practice with human and animal clients and suggestions for future research. Further, there are multiple ways to measure procedural fidelity, and different conclusions can be drawn based on the measure and computation method. The current article describes procedural-fidelity measures that are most applicable to animal behavior researchers and professionals.

The mitigating effects of enhanced reinforcer magnitude and quality on treatment degradation

Differential reinforcement of alternative behavior is a common intervention for problem behavior in persons with neurodevelopmental disorders, but it is susceptible to integrity errors that can degrade treatment effects. Manipulating reinforcement parameters to favor alternative behavior might make it more persistent in the face of integrity errors. We devised an analog of differential reinforcement of alternative behavior to examine if enhanced reinforcer magnitude or quality for the alternative response could protect against treatment degradation. Across 2 experiments, reinforcer magnitude or quality was manipulated to favor the alternative response in 1 condition but kept constant across both alternative and target responses in a second condition. Comparisons of the 2 conditions indicated that higher-magnitude or higher-quality reinforcement for alternative behavior can mitigate against treatment degradation when treatment errors occur and provided support for the utility of considering parameters of reinforcement when developing behavioral interventions for problem behavior.

Planning Positive Reinforcement Cycles in Behavior Intervention Plans

Behavior analysts partner with educators in schools to support the creation of behavior intervention plans (BIPs). Assessment and intervention planning often focuses on the relational contingencies between the student and their environment, with little attention paid to the relational contingencies contacted by the educator. In this article, we posit that planning should simultaneously include contingencies for both the student and the educator as BIPs are created. Specifically, we aim to explore a dual-pathway intervention plan in which student and educator access to reinforcement is simultaneously designed to increase both educators' implementation of high-quality instruction and students' engagement and performance. Procedural steps outlining the duality of intervention planning for both the student and the educator, as well as a theoretical model for considering contextual and reinforcement contingencies for both parties, will be detailed in a step-by-step guide to support readers' creation and implementation of plans to support improved educator and student performance. Planning for supporting both the educator and student may increase sustained, high-quality instruction and improved student outcomes for students with behavioral support needs.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40617-021-00663-8.

Further examination of the treatment of multiply controlled inappropriate mealtime behavior

We systematically replicated Bachmeyer et al. (2009) by examining extinction procedures matched to each function, individually and in combination, to treat the food or liquid refusal of 4 children diagnosed with a feeding disorder whose inappropriate mealtime behavior was maintained by multiple functions (i.e., escape and attention). Previous research suggests that adding differential reinforcement to extinction procedures may result in better treatment outcomes. Therefore, we added differential reinforcement to extinction procedures matched to each function. Differential reinforcement and extinction matched only to escape or attention resulted in low rates of inappropriate mealtime behavior and high, stable levels of acceptance for only 1 child. Consistent with Bachmeyer et al., inappropriate mealtime behavior decreased, and acceptance increased for the remaining 3 children only after we matched differential reinforcement and extinction procedures to both escape and attention.

Negative reinforcer magnitude manipulations for treating escape-maintained problem behavior

The purpose of the current study was to evaluate the effects of different magnitudes of escape for compliance relative to the magnitudes of escape for problem behavior in a concurrent-schedule arrangement. Three individuals who exhibited escape-maintained problem behavior participated. A large differential magnitude condition (240-s escape for compliance, 10-s escape for problem behavior) was compared to equal (30-s escape for compliance and problem behavior) and moderate differential magnitude (90-s escape for compliance, 10-s escape for problem behavior) conditions. The authors also evaluated the impact of correcting for reinforcer access time (i.e., time on escape intervals) on intervention interpretation. For all participants, problem behavior decreased during only the large differential magnitude condition, and including reinforcer access time in the overall session time did not affect interpretation of treatment outcomes. Providing larger escape magnitudes for compliance relative to problem behavior may facilitate treatment involving concurrent-reinforcement schedules for escape-maintained problem behavior.

Improving procedural fidelity of behavioural interventions for people with intellectual and developmental disabilities: A systematic review

BACKGROUND

Despite its importance within behavioural intervention, it remains unclear how best to achieve high procedural fidelity. This paper reviewed studies on improving procedural fidelity of behavioural interventions for individuals with intellectual and developmental disabilities (IDD).

METHOD

A systematic literature search was conducted, which identified 20 studies meeting inclusion criteria. Data were extracted on study design, participant characteristics, intervention, target behaviours, effect sizes, maintenance, generalization and social validity. A quality rating was also applied.

RESULTS

A total of 100 participants took part in the included studies. Most participants were teachers working with children in school settings. There was a significant positive correlation between level of procedural fidelity and client outcomes. Feedback was the most commonly employed intervention to improve procedural fidelity.

CONCLUSIONS

More research should be conducted in environments with high levels of variability such as community homes to determine how to reach and maintain high levels of procedural fidelity.

Improving Accuracy of Data Collection on a Psychiatric Unit for Children Diagnosed With Intellectual and Developmental Disabilities

Data collection is a hallmark of effective behavior-analytic therapy. Collecting accurate data permits a behavior analyst to evaluate the effectiveness of behavioral treatment. The current study evaluated the use of a clicker, simplified observation, and a timer to improve accuracy of data collection on a psychiatric unit for children diagnosed with intellectual and developmental disabilities. Experiment 1, conducted within a combined multiple-baseline across-participants and reversal design, was an evaluation to identify an intervention package for four participants employed by the psychiatric unit. Interventions yielding the highest interobserver agreement (IOA) were highly individualized. Thus, we selected the most comprehensive intervention and exposed four additional participants to this intervention during Experiment 2. Results showed that this intervention improved IOA for these additional participants as evaluated within a multiple-baseline across-participants design. Results of the current study will be discussed to assist other behavior analysts in improving data-collection practices in hospital or school settings.

Applying the Taxonomy of Validity Threats from Mainstream Research Design to Single-Case Experiments in Applied Behavior Analysis

Mainstream research design in the social and behavioral sciences has often been conceptualized using a taxonomy of threats to experimental validity first articulated by Campbell and his colleagues (Campbell & Stanley, 1966; Cook & Campbell, 1979). The most recent update of this framework was published by Shadish, Cook, and Campbell (2002), in which the authors describe different types of validity and numerous threats to each primarily in terms of group-design experiments. In the present article, we apply Shadish et al.'s analysis of threats to internal, external, statistical conclusion, and construct validity to single-case experimental research as it is typically conducted in applied behavior analysis. In doing so, we hope to provide researchers and educators in the field with a translation of the validity-threats taxonomy into terms and considerations relevant to the design and interpretation of applied behavior-analytic research for the purposes of more careful research design and the ability to communicate our designs to individuals outside of behavior analysis, using their own vocabulary.

Naturalistic language sampling to characterize the language abilities of 3-year-olds with autism spectrum disorder

Characterization of language in naturalistic settings in autism spectrum disorder has been lacking, particularly at young ages, but such information is important for parents, teachers, and clinicians to better support language development in real-world settings. Factors contributing to this lack of clarity include conflicting definitions of language abilities, use of non-naturalistic standardized assessments, and restricted samples. The current study examined one of the largest datasets of naturalistic language samples in toddlers with autism spectrum disorder, and language delay and typically developing contrast groups at age 3. A range of indices including length of phrase, grammatical markings, and social use of language was assayed during a naturalistic observation of a parent-child play session. In contrast to historical estimates, results indicated only 3.7% of children with autism spectrum disorder used no words, and 34% were minimally verbal. Children with autism spectrum disorder and language delay exhibited similar usage of grammatical markings, although both were reduced compared to typically developing children. The greatest difference between autism spectrum disorder and language delay groups was the quantity of social language. Overall, findings highlight a range of language deficits in autism spectrum disorder, but also illustrate that the most severe level of impairments is not as common in naturalistic settings as previously estimated by standardized assessments.

Teacher-Implemented Response Interruption and Redirection: Training, Evaluation, and Descriptive Analysis of Treatment Integrity

Response interruption and redirection (RIRD) is an effective intervention for decreasing stereotypy. During RIRD, contingent on occurrences of stereotypy, therapists interrupt the behavior and prompt the participant to complete an alternative response. Although RIRD has been implemented by teachers in classrooms, it requires continuous monitoring of participants to be implemented with fidelity and may be difficult for teachers to manage. The present study evaluated the effectiveness of RIRD when implemented in classrooms. In addition, we evaluated if novice teaching assistants could be trained to implement RIRD. Finally, a descriptive analysis of treatment integrity errors during RIRD was conducted. Three children and teaching assistants participated. Following a written instructions baseline, the teaching assistants were trained to implement RIRD using modeling, rehearsal, and feedback. The training increased the accuracy of RIRD implementation for all participants. Incorrectly initiating and terminating RIRD were the most common treatment integrity errors observed.

Function-Altering Effects of Rule Phrasing in the Modulation of Instructional Control

This study evaluated the effects of four instructional variants on instruction following under changing reinforcement schedules using an operant task based on Hackenberg and Joker's Journal of the Experimental Analysis of Behavior, 62, 367-383 (1994) experimental preparation. Sixteen college-aged adults served as participants and were randomly assigned to one of four instruction conditions (directive, generic, non-directive, and control). Results suggest textual verbal behavior modulated instruction following. Specifically, directive and generic instructions produced greater levels of instructional control and relatively lower levels of schedule control compared to non-directive instructions. Thus, participants in the directive and generic groups responded in accordance with the instructions even when schedules of reinforcement favored deviation from the instructed pattern. In contrast, participants in the non-directive group responded toward the optimal pattern. In the control condition, participant responding was variable but toward the optimal pattern. Findings are interpreted within the framework of Skinner's analysis of verbal behavior and formulation of rule governance.

Essential components of written behavior treatment plans

For the last 25 years, the only empirically determined system to evaluate the content of written behavior analysis plans was developed by Vollmer et al. (1992). For the current study, the content of that earlier system was revised by the first author and submitted to 48 members of the editorial board of the Journal of Applied Behavior Analysis and seven (7) other acknowledged experts on the editorial boards of Behavioral Interventions and Research in Developmental Disabilities. Of 55 recipients, 36 responded. The thirty-six (36) respondents rated each of 28 items from essential to non-essential using a five-point Likert scale. After reviewing the expert panel members' evaluations, we reduced the 28 items to 20 essential components of written behavior treatment plans. The implications of the results were discussed.

Behavior description effect on accuracy and reliability

The clinical and scientific efficacy of behavioral analysis is dependent upon interveners' accurate and reliable detection and measurement of target behaviors. This study compared the accuracy and reliability of observers' detection and recording of a designated target behavior when different forms of a target behavior description were used. Using an intra-subject design, undergraduate college students were asked to count the number of target behaviors depicted on a videotape under each of two conditions. Conditions differed only to the extent that each contained a different description of the target behavior. Results showed that participants' detection and recording of the target behavior was more accurate and reliable when the target behavior description used a verb (in the present tense, active voice) depicting an action with an observable and discrete beginning and end and omitted modifiers requiring observers to make subjective or relative judgments. Analysis of the data using methods developed by Signal Detection Theory demonstrated the potential utility of this approach for studying observer detection of target behaviors.

A microsoft excel(®) 2010 based tool for calculating interobserver agreement

This technical report provides detailed information on the rationale for using a common computer spreadsheet program (Microsoft Excel(®)) to calculate various forms of interobserver agreement for both continuous and discontinuous data sets. In addition, we provide a brief tutorial on how to use an Excel spreadsheet to automatically compute traditional total count, partial agreement-within-intervals, exact agreement, trial-by-trial, interval-by-interval, scored-interval, unscored-interval, total duration, and mean duration-per-interval interobserver agreement algorithms. We conclude with a discussion of how practitioners may integrate this tool into their clinical work.

A preliminary analysis of the effects of coaching feedback on teacher implementation fidelity of first step to success

First Step to Success (FSS) is a "manualized" intervention with documented effectiveness in reducing problem behaviors for young children at risk for school failure. This study is a preliminary analysis focusing on the role of performance feedback from FSS coaches on the fidelity with which teachers implemented FSS. Three typically developing students (2 kindergarten and 1 first grade) and their respective teachers served as participants. Student behavior and teacher implementation fidelity were assessed using a nonconcurrent multiple baseline design across student/teacher dyads, in which a no coaching baseline was followed by a performance feedback condition. The baseline phase showed modest levels of implementation fidelity and initial low levels of problem behavior followed by gradually increasing trends. A relation was demonstrated between coaching feedback and improved implementation fidelity. In addition, improved fidelity was associated with improvements in student problem behavior. Implications are drawn for clinical application of FSS and other "manualized" interventions.

Improving safe consumer transfers in a day treatment setting using training and feedback

An intervention package that included employee training, supervisory feedback, and graphic feedback was developed to increase employees' safe patient-transfers at a day treatment center for adults with disabilities. The intervention was developed based on the center's results from a Performance Diagnostic Checklist (PDC), which focused on antecedents, equipment and processes, knowledge and skills, and consequences related to patient-transfers. A multiple baseline (MBL) across two lifts (pivot and trunk), with one lift (side) remaining in baseline was used to evaluate the effects of the treatment package on three lifts commonly used by three health-care workers. The results indicated a substantial increase in the overall safe performance of the three lifts. The mean increase for group safety performance following intervention was 34% and 29% over baseline measures for the two target transfers, and 28% over baseline measures for the nontargeted transfer. The implications of these findings suggest that in settings where patient transfers are frequent and injuries are likely to occur (e.g., hospitals, day treatment centers), safe lifting and transferring behaviors can improve with an efficient and cost-effective intervention.

Operations analysis of behavioral observation procedures: a taxonomy for modeling in an expert training system

This article introduces a taxonomy based on a procedural operations analysis (Verplanck, 1996) of various method descriptions found in the behavior observation research literature. How these alternative procedures impact the recording and subsequent analysis of behavioral events on the basis of the type of time and behavior recordings made is also discussed. The taxonomy was generated as a foundation for the continuing development of an expert training system called Train-to-Code (TTC; J. M. Ray & Ray, (Behavior Research Methods 40:673-693, 2008)). Presently in its second version, TTC V2.0 is software designed for errorless training (Terrace, (Journal of the Experimental Analysis of Behavior 6:1-27, 1963)) of student accuracy and fluency in the direct observation and coding of behavioral or verbal events depicted via digital video. Two of 16 alternative procedures classified by the taxonomy are presently modeled in TTC's structural interface and functional services. These two models are presented as illustrations of how the taxonomy guides software user interface and algorithm development. The remaining 14 procedures are described in sufficient operational detail to allow similar model-oriented translation.