A detailed examination of reporting procedural fidelity in the Journal of Applied Behavior Analysis

Few reviews on procedural fidelity-the degree to which procedures are implemented as designed-provide details to gauge the quality of fidelity reporting in behavior-analytic research. This review focused on experiments in the Journal of Applied Behavior Analysis (2006-2021) with "integrity" or "fidelity" in the abstract or body. When fidelity data were collected, the coders characterized measurement details (e.g., description of calculation, report of single or multiple values, frequency of fidelity checks, checklist use). The researchers found increasing trends in describing the calculation(s), reporting multiple values, and stating the frequency of measurement. Few studies described using a checklist. Most studies reported fidelity as a percentage, with high obtained values (M = 97%). When not collecting fidelity data was stated as a limitation, authors were unlikely to provide a rationale for the omission. We discuss recommendations for reporting procedural fidelity to increase the quality of and transparency in behavior-analytic research.

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

A simulated comparison of behavioural observation sampling methods

Behavioural research requires the use of sampling methods to document the occurrence of responses observed. Sampling/recording methods include ad libitum, continuous, pinpoint (instantaneous), and one-zero (interval) sampling. Researchers have questioned the utility of each sampling method under different contexts. Our study compared computerized simulations of both pinpoint and one-zero sampling to continuous recordings. Two separate computer simulations were generated, one for response frequency and one for response duration, with three different response frequencies (high, medium, or low) and response durations (short, medium, and long) in each simulation, respectively. Similarly, three different observation intervals (5, 50, and 500 s) were used to record responses as both pinpoint and one-zero sampling methods in the simulations. Under both simulations, pinpoint sampling outperformed one-zero sampling, with pinpoint sampling producing less statistical bias in error rates under all frequencies, durations, and observation intervals. As observation intervals increased, both mean error rates and variability in error rates increased for one-zero sampling, while only variability in error rate increased for pinpoint sampling. The results suggest that pinpoint sampling techniques are effective for measuring both frequency (event) and duration (state) behaviours, and that pinpoint sampling is a less statistically biased behavioural observation method than one-zero sampling.

Procedures and Accuracy of Discontinuous Measurement of Problem Behavior in Common Practice of Applied Behavior Analysis

Discontinuous measurement involves dividing an observation into intervals and recording whether a behavior occurred during some or all of each interval (i.e., interval recording) or at the exact time of observation (i.e., momentary time sampling; MTS). Collecting discontinuous data is often easier for observers than collecting continuous data, but it also produces more measurement error. Smaller intervals (e.g., 5 s, 10 s, 15 s) tend to produce less error but may not be used in everyday practice. This study examined the most common intervals used by a large sample of data collectors and evaluated the effect of these intervals on measurement error. The most commonly used intervals fell between 2 and 5 min. We then analyzed over 800 sessions to evaluate the correspondence between continuous and discontinuous data at each commonly used interval. Intervals of 3 min or less produced the greatest correspondence, and MTS outperformed interval recording.

Using Mobile Technology to Reduce Engagement in Stereotypy: A Validation of Decision-Making Algorithms

We developed an iOS app, the iSTIM, designed to support parents of children with autism spectrum disorders (ASD) in reducing common repetitive vocal and motor behavior (i.e., stereotypy). The purpose of our study was to preliminarily test the decision-making algorithms of the iSTIM using trained university students to implement the assessments and interventions. Specifically, we examined the effects of the iSTIM on stereotypy and functional engagement in 11 children with ASD within alternating treatment designs. Using the iSTIM reduced engagement in stereotypy for eight participants and increased functional engagement for four of those participants. Our results indicate that the iSTIM may decrease engagement in stereotypy but that some of the decision-making algorithms may benefit from modifications prior to testing with parents.

Using Conditional Percentages During Free-Operant Stimulus Preference Assessments to Predict the Effects of Preferred Items on Stereotypy: Preliminary Findings

To date, researchers have not identified an efficient methodology for selecting items that will compete with automatically reinforced behavior. In the present study, we identified high preference, high stereotypy (HP-HS), high preference, low stereotypy (HP-LS), low preference, high stereotypy (LP-HS), and low preference, low stereotypy (LP-LS) items based on response allocation to items and engagement in stereotypy during one to three, 30-min free-operant competing stimulus assessments (CSAs). The results showed that access to HP-LS items decreased stereotypy for all four participants; however, the results for other items were only predictive for one participant. Reanalysis of the CSA results revealed that the HP-LS item was typically identified by (a) the combined results of the first 10 min of the three 30-min assessments or (b) the results of one 30-min assessment. The clinical implications for the use of this method, as well as future directions for research, are briefly discussed.

Four Methods for Analyzing Partial Interval Recording Data, with Application to Single-Case Research

Partial interval recording (PIR) is a procedure for collecting measurements during direct observation of behavior. It is used in several areas of educational and psychological research, particularly in connection with single-case research. Measurements collected using partial interval recording suffer from construct invalidity because they are not readily interpretable in terms of the underlying characteristics of the behavior. Using an alternating renewal process model for the behavior under observation, we demonstrate that ignoring the construct invalidity of PIR data can produce misleading inferences, such as inferring that an intervention reduces the prevalence of an undesirable behavior when in fact it has the opposite effect. We then propose four different methods for analyzing PIR summary measurements, each of which can be used to draw inferences about interpretable behavioral parameters. We demonstrate the methods by applying them to data from two single-case studies of problem behavior.

Interval sampling methods and measurement error: a computer simulation

A simulation study was conducted to provide a more thorough account of measurement error associated with interval sampling methods. A computer program simulated the application of momentary time sampling, partial-interval recording, and whole-interval recording methods on target events randomly distributed across an observation period. The simulation yielded measures of error for multiple combinations of observation period, interval duration, event duration, and cumulative event duration. The simulations were conducted up to 100 times to yield measures of error variability. Although the present simulation confirmed some previously reported characteristics of interval sampling methods, it also revealed many new findings that pertain to each method's inherent strengths and weaknesses. The analysis and resulting error tables can help guide the selection of the most appropriate sampling method for observation-based behavioral assessments.

Use of discontinuous methods of data collection in behavioral intervention: guidelines for practitioners

Over the past three decades, researchers have examined the sensitivity and accuracy of discontinuous data-collection methods. Momentary-time sampling (MTS) and partial-interval recording (PIR) have received particular attention in regards to their ability to estimate the occurrence of behavior and their sensitivity to behavior change compared to continuous data collection. In this article, we summarize these findings and provide recommendations for designing a discontinuous measurement system with consideration of the dimensions of behavior to be measured and the expected direction of behavior change.

Detecting False Positives in Multielement Designs

The authors assessed the extent to which multielement designs produced false positives using continuous duration recording (CDR) and interval recording with 10-s and 1-min interval sizes. Specifically, they created 6,000 graphs with multielement designs that varied in the number of data paths, and the number of data points per data path, using a random number generator. In Experiment 1, the authors visually analyzed the graphs for the occurrence of false positives. Results indicated that graphs depicting only two sessions for each condition (e.g., a control condition plotted with multiple test conditions) produced the highest percentage of false positives for CDR and interval recording with 10-s and 1-min intervals. Conversely, graphs with four or five sessions for each condition produced the lowest percentage of false positives for each method. In Experiment 2, they applied two new rules, which were intended to decrease false positives, to each graph that depicted a false positive in Experiment 1. Results showed that application of new rules decreased false positives to less than 5% for all of the graphs except for those with two data paths and two data points per data path. Implications for brief assessments are discussed.

A Comparison of Reliability Measures for Continuous and Discontinuous Recording Methods: Inflated Agreement Scores With Partial Interval Recording and Momentary Time Sampling for Duration Events

The authors evaluated the extent to which interobserver agreement (IOA) scores, using the block-by-block method for events scored with continuous duration recording (CDR), were higher when the data from the same sessions were converted to discontinuous methods. Sessions with IOA scores of 89% or less with CDR were rescored using 10-s partial interval recording (PIR) and 10-s momentary time sampling (MTS). Results indicated that IOA scores for 10-s PIR and 10-s MTS were consistently higher than IOA scores based on CDR for the same sessions. Specifically, 10-s MTS provided higher overestimations for low-duration events, whereas 10-s PIR produced higher overestimations for moderate- and high-duration events. Implications for researchers and clinicians are briefly discussed.

Promoting mouth-drying responses to reduce drooling effects by persons with intellectual and multiple disabilities: a study of two cases

This study assessed the use of microswitch technology to promote mouth-drying responses and thereby reduce the effects of drooling by two adults with severe intellectual and multiple disabilities. Mouth-drying responses were performed via a special napkin that contained pressure sensors, a microprocessor and an MP3 to monitor the responses and ensure positive stimulation contingent on them. Initially, the responses produced 10 or 15 s of preferred stimulation. Subsequently, preferred stimulation was supplemented with matching periods of lower-grade stimulation to extend the inter-response intervals. Results showed that both participants (a) learned to dry their mouth consistently and reduce their chin wetness during the intervention, (b) stabilized their responding at lower frequencies as the lower-grade stimulation was added to the preferred stimulation, and (c) maintained the latter levels at a 3-month follow-up. Procedure and response conditions and outcome implications are discussed.

Instability of physical anxiety symptoms in daily life of patients with panic disorder and patients with posttraumatic stress disorder

The present study examined severity as well as degree and temporal pattern of instability of DSM-IV-based bodily symptoms of anxiety (BSA) in daily life of 26 panic disorder (PD) patients, 17 posttraumatic stress disorder (PTSD) patients, and 28 healthy controls (HC) during 1 week, using electronic diaries. The ecological momentary assessment around every 3h during wake times was accepted well by patients. Compared to HC, patient groups exhibited elevated instability of BSA. BSA instability was more pronounced in PTSD than PD (p<0.005), even after controlling for mean symptom level. Numbers of symptomatic episodes were comparable in PTSD and PD, but the duration of symptom-free episodes was shorter in PTSD than PD. Results indicate that PTSD patients are particularly burdened by fluctuations in somatic symptoms of anxiety, implying perceived unpredictability and uncontrollability. Electronic diaries can be applied in innovative ways to provide novel insights into the phenomenology of anxiety disorders that may not be captured well by retrospective interviews and questionnaires.

Two persons with multiple disabilities use a mouth-drying response to reduce the effects of their drooling

These two studies involved a boy and a man with multiple disabilities, who were taught to use a mouth-drying response to reduce the effects of their drooling. Both studies relied on microswitch technology to monitor the drying response and follow it with positive stimulation (i.e., during intervention). In Study I, the boy performed the drying response via a special napkin. The microswitch technology consisted of touch/pressure sensors and a radio transmitter hidden inside the napkin. Drying responses led the boy to 8s of preferred stimulation. In Study II, the man performed the drying response via a handkerchief. The microswitch technology consisted of an optic sensor and a radio transmitter at the man's chest. Drying responses led the man to 8-10s of preferred stimulation. The stimulation time/conditions were subsequently modified to promote a reduction in the man's response frequency. The experimental design involved an ABAB sequence (Study I) or an ABABB(1)B(2) sequence (Study II), with the second B or the B(1)B(2) combination spreading over periods of about 3 months. The results indicated vast increases in drying responses and decreases in chin wetness during the intervention phases. The frequencies of the drying response remained consistent for the boy and stabilized at a lower level (i.e., in line with the manipulation of the stimulation conditions) for the man. Implications of the findings and limitations of the studies are discussed.

Evaluating progress in behavioral programs for children with autism spectrum disorders via continuous and discontinuous measurement

We evaluated the influence of two different frequencies of data collection on skill acquisition and maintenance within behavioral treatment programs for children with autism spectrum disorders. Six children were taught multiple skills in up to four different behavioral programs. Half of the skills were measured continuously (i.e., trial by trial), and the other half were measured discontinuously (i.e., first trial only). When differences were detected, quicker acquisition was typically associated with discontinuous measurement, and stronger maintenance was typically associated with continuous measurement.