Fidelity Assessment in Community Programs: An Approach to Validating Simplified Methodology

Technology-based interactive communication simulation addressing challenging communication situations for nursing students

BACKGROUND

Simulation-based interventions for nursing students addressing challenging communication situations involving geriatric patients and end-of-life care are limited.

PURPOSE

This study evaluated the effects of technology-based interactive communication simulations on nursing students' communication knowledge, self-efficacy, skills, compassion, and program satisfaction.

METHOD

A randomized controlled repeated-measures design was used with third- and fourth-year nursing students enrolled in five nursing colleges located in five regions in Korea as participants. Participants were randomly assigned to either a technology-based interactive communication simulation or an attention control group. Changes in communication knowledge, self-efficacy, skills, compassion, and program satisfaction were assessed using three self-reported measures and communication skills were measured by the raters. Statistical analyses included descriptive analyses, chi-square tests, t-tests, and a generalized estimating equation model.

RESULTS

Eighty students participated in one of the two programs, and 77 in the four-week follow-up test. The intervention group indicated significant improvements in communication knowledge, self-efficacy, skills, and compassion, as well as higher program satisfaction compared with the attention control group. Communication skills as assessed by raters also showed significant change at all assessment time points.

CONCLUSION

The technology-based interactive communication simulation program is effective in improving communication skills among nursing students managing geriatric patients and end-of-life care.

A Practitioner's Guide to Measuring Procedural Fidelity

Ensuring high levels of procedural fidelity during behavior-analytic interventions is a crucial component of providing effective behavior-analytic services. However, few resources are available to help guide practitioners through measuring procedural fidelity. In fact, most published behavior-analytic research on procedural fidelity analyzes a single treatment procedure, which might not completely reflect the process of monitoring and addressing the procedural fidelity of a robust treatment package that might be necessary in clinical settings. The purpose of this article is to guide behavior analysts through the process of creating and using procedural fidelity measurement systems, with a focus on direct observation of implementation as a means of fidelity data collection. This process consists of six steps: (1) task analyze treatment procedures into measurable units; (2) assign measures to each treatment component; (3) plan the direct observation; (4) collect procedural fidelity data; (5) analyze and interpret procedural fidelity data; and (6) take action to improve procedural fidelity. Each step is described and discussed in the article.

The Case for Assessing and Reporting on Facilitator Fidelity: Introducing the Fidelity of Implementation in Parenting Programs Guideline

The sizeable body of evidence indicating that parenting programs have a positive impact on children and families highlights the potential public health benefits of their implementation on a large scale. Despite evidence and global attention, beyond the highly controlled delivery of parenting programs via randomized trials, little is known about program effectiveness or how to explain the poorer results commonly observed when implemented in community settings. Researchers, practitioners, and policymakers must work together to identify what is needed to spur adoption and sustainment of evidence-based parenting programs in real-world service systems and how to enhance program effectiveness when delivered via these systems. Collecting, analyzing, and using facilitator fidelity data is an important frontier through which researchers and practitioners can contribute. In this commentary, we outline the value of assessing facilitator fidelity and utilizing the data generated from these assessments; describe gaps in research, knowledge, and practice; and recommend directions for research and practice. In making recommendations, we describe a collaborative process to develop a preliminary guideline-the Fidelity of Implementation in Parenting Programs Guideline or FIPP-to use when reporting on facilitator fidelity. Readers are invited to complete an online survey to provide comments and feedback on the first draft of the guideline.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43477-023-00092-5.

Peter CC, Feng J, Long BP. Trends in Reporting Procedural Integrity: A Comparison

Procedural integrity refers to the extent to which an independent variable is implemented as described. Measuring procedural integrity is one important factor when considering internal and external validity of experiments. Experimental articles in behavior-analytic journals have rarely reported procedural-integrity data. The purpose of this study was to update previous reviews of whether articles published in the Journal of Applied Behavior Analysis reported procedural integrity, spanning a period from 1980 to 2020, and compare reporting in JABA to recent reviews of studies published in Behavior Analysis in Practice (2008-2019) and the Journal of Organizational Behavior Management (2000-2020). Procedural integrity continues to be underreported across all three journals, but an increasing trend in reporting procedural integrity is evident in the Journal of Applied Behavior Analysis and Behavior Analysis in Practice. In addition to our recommendations and implications for research and practice, we provide examples and resources to assist researchers and practitioners with recording and reporting integrity data.

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

Predictors of expert providers' familiarity with intervention practices for school- and transition-age youth with autism spectrum disorder

LAY ABSTRACT

School-age children, adolescents, and young adults with autism spectrum disorder encounter many different types of providers in their pursuit of treatment for anxiety, behavior problems, and social difficulties. These providers may all be familiar with different types of intervention practices. However, research has not yet investigated patterns in expert providers' familiarity with different practices nor how these patterns are related to the characteristics of providers (years in practice, academic discipline, setting) and the youth (age and intellectual disability) they typically support. A panel of 53 expert transdisciplinary providers rated their familiarity with 55 intervention practices (derived from research and expert nominations) via an online Delphi poll. Advanced statistical methods were used to identify types of intervention practices with which providers were familiar, which included two approaches (cognitive and behavioral) and two strategies (engagement and accessibility). Providers who practiced outside a school setting or treated clients without intellectual disability were more familiar with cognitive approaches. Clinical psychologists, behavior analysts, and school-based providers were more familiar with behavioral approaches. Providers practicing outside school settings were also more familiar with engagement strategies, and providers with more years in practice were more familiar with accessibility strategies. These results may help families and researchers to better anticipate how services may vary depending on the types of autism spectrum disorder providers seen and work to reduce disparities in care that may result.

Strengthening the procedural fidelity research-to-practice loop in animal behavior

Procedural fidelity is the extent to which components of an intervention are implemented as designed. Procedural fidelity is measured as a dependent variable and manipulated as an independent variable. In research and practice, procedural-fidelity data should be collected, monitored, and reported. Procedural fidelity as an independent variable has been investigated in humans using parametric analyses, and the current article summarizes some of the research conducted on the effects of procedural-fidelity errors in behavior-reduction and skill-acquisition interventions. Connections were drawn to applied animal researchers and the work of animal behavior practitioners to produce implications for practice with human and animal clients and suggestions for future research. Further, there are multiple ways to measure procedural fidelity, and different conclusions can be drawn based on the measure and computation method. The current article describes procedural-fidelity measures that are most applicable to animal behavior researchers and professionals.

Measuring parent strategy use in early intervention: Reliability and validity of the Naturalistic Developmental Behavioral Intervention Fidelity Rating Scale across strategy types

LAY ABSTRACT

Children with autism spectrum disorder benefit from early intervention to improve social communication, and parent-implemented interventions are a feasible and family-centered way to increase the amount of treatment they receive. For these treatments to be effective, it is important for the parent to implement the strategies as intended. However, measurement of parent strategy use is inconsistent across studies of parent-implemented interventions. This study evaluates the quality of the NDBI-Fi, an efficient measure, compared to a more time-consuming measure that is known to be precise. Videos of parents playing with their children were used to compare these two measurement methods. Results demonstrated that the NDBI-Fi was of good quality: scorers had high levels of agreement, the NDBI-Fi was similar to the more precise measure in rating parents after intervention, it detected changes from before to after intervention, and it detected differences when parents learned different types of intervention strategies. The NDBI-Fi was not as precise as the other measure across all strategies before parents learned intervention. Taken together, the findings of this study support the use of the NDBI-Fi as a high-quality outcome measure.

Training Teachers to Implement Classroom Pivotal Response Teaching during Small-group Instruction: A Pilot Study

Background: Providing effective education to students with autism spectrum disorder (ASD) poses a significant challenge to educators. Although several evidence-based practices (EBPs) have been developed, few have been systematically implemented in educational settings. Pivotal response treatment (PRT) is a naturalistic behavioral intervention that has been adapted for implementation in the school context. Methods: This pilot study used a concurrent multiple baseline design across seven teachers and students with ASD to examine the effectiveness of teacher training in classroom pivotal response teaching (CPRT) on teacher fidelity of implementation during small-group instruction and students' communication skills and maladaptive behaviors in schools for special education in the Netherlands. Results: Results indicated no replicated effect of CPRT training on teachers' fidelity of implementation or children's communications skills and maladaptive behavior, although teachers reported high satisfaction with the CPRT training. Discussion: Implications for clinical practice and directions for future research are discussed.

ELABORATION AND VALIDATION OF AN EDUCATIONAL VIDEO ON BLOOD PRESSURE MEASUREMENT IN SCREENING PROGRAMS

ABSTRACT Objective: to produce and validate an educational video on the procedure of indirect blood pressure measurement with the oscillometric technique to support actions of blood pressure screening programs in Brazil. Method: a methodological study consisting in three stages: 1) pre-production of a script/storyboard; 2) content validation by a committee of 16 experts recruited according to their degrees and training in the area of interest of the study; and 3) final production, recording and edition of the educational video. For data collection, three validated instruments were used that allowed the video to be assessed as to its functionality, usability, efficiency, relevance, verbal language, audiovisual technique, environment, content and proposed objectives. The analysis of the quantitative variables was performed by calculating absolute and relative frequencies, and the categorical variables were measured using means and standard deviations. Items that obtained a CVI > 0.75 were considered valid. Results: script validation was performed by eight experts and obtained a CVI of 0.93, while the technical assessment of the educational video and storyboard was performed by three experts and obtained a CVI of 0.97. After accepting the experts' suggestions, the educational video was produced and validated by eight experts, who considered the material valid for application (CVI = 0.94). Conclusion: the educational video produced and validated in this study was characterized as an appropriate strategy for teaching the indirect measurement of blood pressure with the oscillometric technique among health professionals who volunteer to participate in blood pressure screening programs in Brazil.