Self-Report After Randomly Assigned Supervision Does not Predict Ability to Practice Motivational Interviewing

Measuring a motivational interviewing practice orientation in criminal justice practitioners: initial validation of the response style screening questionnaire

Introduction

The use of collaborative communication techniques by criminal justice practitioners has been identified as a component of core correctional practices (CCPs). Criminal justice agencies and programs are investing in motivational interviewing (MI) training for their staff with varying intensity, follow-up coaching, and expectations for integration into practice. The present article describes the development and initial validation of the Response Style Screening Questionnaire (RSSQ), a practitioner completed tool assessing an MI-consistent practice orientation. Over three studies, we examined the factor structure, reliability, and correlations between the scale and self-reported and behavioral validity indicators.

Method

Study 1 examined the factor structure of the RSSQ with a sample of 825 criminal justice practitioners. In Study 2, data from 350 probation officers completing the RSSQ alongside measures of work-orientation and attitudes was used to conduct a confirmatory factor analysis and an initial assessment of its construct validity. In Study 3, correlations between the RSSQ and audio recorded office visits of 33 probation officers coded for MI and relationship building skills examined the scale's criterion validity.

Results

In Study 1, an exploratory factor analysis with an oblique rotation yielded 18 items on four factors. In Study 2, four and five factor models were tested, with the 4-factor model of Study 1 yielding the best fit. Two of the 4 factor-derived subscales reflect styles inconsistent with MI: (1) Confrontational style, and (2) Sustain Talk style; while the remaining two reflect styles consistent with MI: (3) Eliciting style, and (4) Change Talk style. Confrontational style scores were correlated with a work-orientation reflecting probation as a law enforcement endeavor, while Eliciting and Change Talk scores were correlated with a behavior change and resource broker work-orientations. In Study 3, Confrontational and Sustain Talk style scores were negatively correlated with a variety of MI skills and CCPs displayed on audio recordings, while Change Talk style scores were positively correlated with use of such skills.

Discussion

Overall, the findings suggest the RSSQ is a potentially useful new practitioner self-report tool for assessing an MI practice orientation.

Pilot Trial of Online Measurement Training and Feedback in Family Therapy for Adolescent Behavior Problems

OBJECTIVE

Pragmatic procedures for sustaining high-fidelity delivery of evidence-based interventions are needed to support implementation in usual care. This study tested an online therapist training system, featuring observational coder training and self-report fidelity feedback, to promote self-report acumen and routine use of family therapy (FT) techniques for adolescent behavior problems.

METHOD

Therapists (N = 84) from nine substance use and mental health treatment sites reported on 185 adolescent clients. Therapists submitted baseline data on FT technique use with clients, completed a workshop introducing the 32-week training system, and were randomly assigned by site to Core Training versus Core Training + Consultation. Core Training included a therapist coder training course (didactic instruction and mock session coding exercises in 13 FT techniques) and fidelity feedback procedures depicting therapist-report data on FT use. Consultation convened therapists and supervisors for one-hour monthly sessions with an external FT expert. During the 32 weeks of training, therapists submitted self-report data on FT use along with companion session audiotapes subsequently coded by observational raters.

RESULTS

Therapist self-report reliability and accuracy both increased substantially during training. Observers reported no increase over time in FT use; therapists self-reported a decrease in FT use, likely an artifact of their improved self-report accuracy. Consultation did not enhance therapist self-report acumen or increase FT use.

CONCLUSIONS

Online training methods that improve therapist-report reliability and accuracy for FT use may confer important advantages for treatment planning and fidelity monitoring. More intensive and/or different training interventions appear needed to increase routine FT delivery.

The Status of SBIRT Training in Health Professions Education: A Cross-Discipline Review and Evaluation of SBIRT Curricula and Educational Research

PURPOSE

To assess the quality of curricular research on the Screening Brief Intervention and Referral to Treatment (SBIRT) approach and determine the presence of useful training modalities, particularly motivational interviewing (MI) training, across health care training curricula.

METHOD

The authors conducted a systematic review of published, peer-reviewed studies in PubMed, ERIC, CINAHL, Ovid HealthSTAR, and PsycINFO databases through March 2021 for English-language studies describing SBIRT, a curriculum for health care trainees, and curricular intervention outcomes. After the records were independently assessed, data were extracted and 20% of the studies were double-coded for interrater reliability.

RESULTS

Of 1,856 studies, 95 were included in the review; 22 had overlapping samples and were consolidated into 10 nested studies, leaving 83 total. Interrater reliability ranged from moderate (κ = .74, P < .001) to strong (κ = .91, P < .001) agreement. SBIRT training was delivered to trainees across many professions, including nursing (n = 34, 41%), medical residency (n = 28, 34%), and social work (n = 24, 29%). Nearly every study described SBIRT training methods (n = 80, 96%), and most reported training in MI (n = 54, 65%). On average, studies reported 4.06 (SD = 1.64) different SBIRT training methods and 3.31 (SD = 1.59) MI training methods. Their mean design score was 1.92 (SD = 0.84) and mean measurement score was 1.89 (SD = 1.05). A minority of studies measured SBIRT/MI skill (n = 23, 28%), and 4 studies (5%) set a priori benchmarks for their curricula.

CONCLUSIONS

SBIRT training has been delivered to a wide range of health care trainees and often includes MI. Rigor scores for the studies were generally low due to limited research designs and infrequent use of objective skill measurement. Future work should include predefined training benchmarks and validated skills measurement.

Correspondence between practitioners’ self-assessment and independent motivational interviewing treatment integrity ratings

As evaluation of practitioners’ competence is largely based on self-report, accuracy in practitioners’ self-assessment is essential for ensuring high quality treatment-delivery. The aim of this study was to assess the relationship between independent observers’ ratings and practitioners’ self-reported treatment integrity ratings of Motivational interviewing (MI). Practitioners (N = 134) were randomized to two types of supervision [i.e., regular institutional group supervision, or individual telephone supervision based on the MI Treatment Integrity (MITI) code]. The mean age was 43.2 years (SD = 10.2), and 62.7 percent were females. All sessions were recorded and evaluated with the MITI, and the MI skills were self-assessed with a questionnaire over a period of 12 months. The associations between self-reported and objectively assessed MI skills were overall weak, but increased slightly from baseline to the 12-months assessment. However, the self-ratings from the group that received monthly objective feedback were not more accurate than those participating in regular group supervision. These results expand findings from previous studies and have important implications for assessment of practitioners’ treatment fidelity: Practitioners may learn to improve the accuracy of self-assessment of competence, but to ensure that patients receive intended care, adherence and competence should be assessed objectively.

“Sometimes It Felt Great, and Sometimes It Just Went Pear-Shaped”: Experiences and Perceptions of School Nurses’ Motivational Interviewing Competence: A Convergent Mixed-Methods Study

In this convergent mixed-methods study, the aim was to explore how objective and subjective quality ratings of school nurses’ motivational interviewing (MI) correlate whilst also considering the perceptions of delivering and participating in the same MI sessions. Quantitative and qualitative data were derived from seven intervention schools participating in the Healthy School Start Plus parenting support intervention. School nurses were trained in MI and conducted an MI session with parents of 6–7-year-old children to discuss children’s physical health and development. Quantitative data comprised objective ratings of school nurses’ MI competence using the Motivational Interviewing Treatment Integrity 4.2 [MITI-4] protocol, as well as parents’ and school nurses’ subjective ratings of the MI sessions. Qualitative data comprised semi-structured interviews with parents and school nurses about their perceptions of the MI sessions. First, quantitative data were analysed using Spearman’s rank correlation, and qualitative data were analysed using content analysis. Next, quantitative and qualitative findings were merged. Our findings suggest that school nurses’ MI performances were rated and perceived as valuable and family-centred by both school nurses and parents who had left the meeting feeling motivated and empowered to promote their children’s healthy behaviours. Nonetheless, school nurses were critical to their own MI technical performance, and they found that reflections were easier to deliver and to self-rate. Overall, MITI ratings were the lowest and parents’ ratings were the highest. Future studies should focus on relating clients’ subjective ratings of MI with clients’ behavioural outcomes.

In Search of the Common Elements of Clinical Supervision: A Systematic Review

The importance of clinical supervision for supporting effective implementation of evidence-based treatments (EBTs) is widely accepted; however, very little is known about which supervision practice elements contribute to implementation effectiveness. This systematic review aimed to generate a taxonomy of empirically-supported supervision practice elements that have been used in treatment trials and shown to independently predict improved EBT implementation. Supervision practice elements were identified using a two-phase, empirically-validated distillation process. In Phase I, a systematic review identified supervision protocols that had evidence of effectiveness based on (a) inclusion in one or more EBT trials, and (b) independent association with improved EBT implementation in one or more secondary studies. In Phase II, a hybrid deductive-inductive coding process was applied to the supervision protocols to characterize the nature and frequency of supervision practice elements across EBTs. Twenty-one of the 876 identified articles assessed the associations of supervision protocols with implementation or clinical outcomes, representing 13 separate studies. Coding and distillation of the supervision protocols resulted in a taxonomy of 21 supervision practice elements. The most frequently used elements were: reviewing supervisees' practice (92%; n = 12), clinical suggestions (85%; n = 11), behavioral rehearsal (77%; n = 10), elicitation (77%; n = 10), and fidelity assessment (77%; n = 10). This review identified supervision practice elements that could be targets for future research testing which elements are necessary and sufficient to support effective EBT implementation. Discrepancies between supervision practice elements observed in trials as compared to routine practice highlights the importance of research addressing supervision-focused implementation strategies.

Comparison of Patient Versus Trained Observer Assessments of Healthcare Providers' Use of Motivational Interviewing Techniques for Patients Experiencing Depression and Anxiety in the Dominican Republic

In settings with limited mental health system capacity, integrated care and the improvement of patient-provider communication surrounding common mental disorders is critical to advancing treatment outcomes. We trained primary care providers in the Dominican Republic in motivational interviewing (MI) to improve communication with patients experiencing depression and anxiety. Providers were randomized to an intervention group, which received MI training, or a control group. To evaluate the training’s effectiveness, patients assessed their clinical encounters using the Motivational Interviewing Measure of Staff Interaction (MIMSI). Trained research assistants (RAs) rated a sub-set of those interactions using an adapted MIMSI instrument. Overall, patients (n = 36) perceived their interactions with providers (n = 10) very positively; however, the RAs’ ratings strongly indicated that providers’ application of MI behaviors was insufficient. Patients generally could not distinguish between intervention and control providers. Findings underscore the need to carefully consider optimal training delivery and cultural influences surrounding the implementation of MI mental health interventions in settings where directive communication is highly valued.

Counselors' acceptability of adherence counseling session recording, fidelity monitoring, and feedback in a multi-site HIV prevention study in four African countries

Biomedical HIV prevention trials increasingly include evidence-based adherence counseling to encourage product use. To retain effectiveness, interventions must contain key components. Monitoring counseling fidelity ensures inclusion of components but is challenging in multinational contexts with different languages and scarce local supervision. The MTN-025/HOPE Study, a Phase 3b open-label trial to assess continued safety of and adherence to the dapivirine vaginal ring for HIV prevention, was the largest such trial to integrate fidelity monitoring using audio recordings of counseling sessions. We describe the monitoring process, along with counselor and participants' perceptions of it, which were collected via quantitative online survey (counselors only N = 42) and in-depth interviews with a subset of counselors (N = 22) and participants (N = 10). Sessions were conducted in five languages across 14 study sites in four countries. In total, 1238 sessions (23% of submitted sessions) were randomly selected and rated. Assessment of interrater reliability was essential to address drift in ratings. Counselors were apprehensive about being monitored, but appreciated clear guidance and found ratings very helpful (mean = 6.64 out of 7). Some participants perceived sessions as time-consuming; others found monitoring improved counseling quality. Fidelity monitoring of counseling sessions in mult-isite biomedical HIV studies is feasible and supportive for counselors.

Assessment of motivational interviewing with the VASE-(Mental) Healthcare: Mixed-methods study to examine feasibility and validity in the general practice setting

OBJECTIVE

The VASE-(M)HC is an instrument to evaluate Motivational Interviewing (MI) skills. We adjusted the previous version for use in the broader (mental) healthcare context, incorporated new MI insights, expanded the scoring system and created a parallel version. Feasibility and validity evidence in the general practice setting was explored.

METHODS

The teaching staff of the GP specialty-training, GP-, and PN-trainees (N = 156) completed the VASE-(M)HC. In this mixed-methods study, we examined psychometric characteristics, compared parallel versions, and interviewed assessors.

RESULTS

Our adjustments enable assessment of a wider range of MI skills, and allow differentiation of basic and advanced skills. Inter-rater reliability was excellent and internal consistency of the total scale was good for both versions. The parallel versions are comparable in terms of difficulty.

CONCLUSION

The VASE-(M)HC is improved by our revisions and adds multiple advantages to the domain of available MI assessment tools.

PRACTICE IMPLICATIONS

Due to the adjustments, the instrument can be used in the GP setting (instead of sole focus on substance abuse). The parallel version is useful for research (pretest/posttest) and selective assessment (retake of a test). It is promising to further explore its applicability in the broader (mental) healthcare context and as training material.

Assessing competence in health professionals' use of motivational interviewing: A systematic review of training and supervision tools

OBJECTIVE

To conduct a systematic review of instruments for assessing the competency of therapists in motivational interviewing (MI) for training purposes.

METHODS

A search of Medline, Emcare, CINAHL, Scopus, Proquest, and Web of Science databases yielded 20,313 articles, of which 105 were included in the review. Data were summarised in terms of the instruments' development, adherence to MI principles, administration characteristics, psychometric properties, advantages, and disadvantages.

RESULTS

Twelve instruments were identified. Tools tended to be better at covering simpler MI techniques. Differences in administration burden allow users to choose between briefer but cheaper and more detailed yet costly tools. Psychometric testing was often limited, and even if more extensive, the quality was often inconsistent. Although each tool tended to have relatively unique advantages (e.g. use of client ratings), they shared common disadvantages (e.g. lack of psychometric testing).

CONCLUSION

A number of tools can be used to assess MI competency, each with their own strengths but notable weaknesses that should be considered by potential users.

PRACTICE IMPLICATIONS

There is a need to further test existing tools before creating new ones, due to the repetition of the same limitations. Standardised guidelines should also be created to ensure each tool meets the same quality standards.

Concordance between clinician, supervisor and observer ratings of therapeutic competence in CBT and treatment as usual: does clinician competence or supervisor session observation improve agreement?

BACKGROUND

Lowering the cost of assessing clinicians' competence could promote the scalability of evidence-based treatments such as cognitive behavioral therapy (CBT).

AIMS

This study examined the concordance between clinicians', supervisors' and independent observers' session-specific ratings of clinician competence in school-based CBT and treatment as usual (TAU). It also investigated the association between clinician competence and supervisory session observation and rater agreement.

METHOD

Fifty-nine school-based clinicians (90% female, 73% Caucasian) were randomly assigned to implement TAU or modular CBT for youth anxiety. Clinicians rated their confidence after each therapy session (n = 1898), and supervisors rated clinicians' competence after each supervision session (n = 613). Independent observers rated clinicians' competence from audio recordings (n = 395).

RESULTS

Patterns of rater discrepancies differed between the TAU and CBT groups. Correlations with independent raters were low across groups. Clinician competence and session observation were associated with higher agreement among TAU, but not CBT, supervisors and clinicians.

CONCLUSIONS

These results support the gold standard practice of obtaining independent ratings of adherence and competence in implementation contexts. Further development of measures and/or rater training methods for clinicians and supervisors is needed.

Motivational interviewing training for physiotherapy and occupational therapy students: Effect on confidence, knowledge and skills

OBJECTIVE

To examine the effectiveness of a three-day training programme on knowledge, confidence and fidelity to Motivational Interviewing (MI) delivery in an undergraduate occupational therapy and physiotherapy cohort (n = 25).

METHODS

Training outcomes were assessed pre-training, post-training and following a subsequent clinical placement. The Motivational Interviewing Knowledge and Attitudes Test (MIKAT) and an 8-item survey assessed knowledge, attitudes and confidence respectively. MI fidelity was evaluated by a simulated patient interview rated with the Motivational Interviewing Treatment Integrity scale (MITI). Analysis was by one-way repeated measures ANOVA.

RESULTS

Self-report measurements indicated increased confidence but no effect on knowledge or attitude. MITI analysis showed superior performance in all four global criteria and an increased frequency of MI adherent behaviours post-training. Positive changes were maintained following clinical placement. MITI summary scores indicated an improvement in question to reflection ratio in line with beginner competency.

CONCLUSION(S)

Participation in a three-day MI training programme significantly improved student confidence and MI skilfulness.

PRACTICE IMPLICATIONS

Where feasible, MI training should be embedded within the curriculum. Further research is needed elucidate the best practices to incorporate teaching this skill set within the curriculum in order to best prepare students to counsel clients in behaviour change in their applied settings.

Effectiveness of Motivational Interviewing Training for Primary Care Dentists and Dental Health Technicians: Results from a Community Clinical Trial

Motivational interviewing (MI) is an approach that uses dialogue about behavioral change to encourage a constructive relationship between professionals and patients. The aim of this study was to evaluate the maintenance of basic MI skills in the daily practice of dentists and oral health technicians (OHTs) after a training course and for two years of follow-up in the context of primary health care (PHC). A randomized community trial, from September 2012 to September 2014, was conducted at the Community Health Service of Conceição Hospital Group in Porto Alegre, RS, Brazil. The experimental group consisted of all 41 dentists and OHTs who received an intensive eight-hour training course in basic MI principles; the control group consisted of 31 dentists and OHTs who did not receive MI training. The follow-up assessments were at one and two years using three instruments validated for MI. In the one- and two-year follow-ups, improvement was maintained in responses with a statistically significant difference for use of open questioning, reflective listening, and total percentage of correct answers (p<0.001), with a large effect size (ES=1.12). On the Helpful Responses Questionnaire, the participants continued using open questioning and reflective listening (p<0.001), maintaining an increase in the percentage of responses compatible with MI (p<0.001). Likewise, the effect size remained large (ES=1.33) over time. These results suggest that the training course with dentists and OHTs of the PHC oral health staff was effective over the two-year follow-up in enabling them to act in the spirit and techniques of MI.

Measurement Training and Feedback System for Implementation of family-based services for adolescent substance use: protocol for a cluster randomized trial of two implementation strategies

BACKGROUND

This article describes a study protocol for testing the Measurement Training and Feedback System for Implementation (MTFS-I) and comparing two implementation strategies for MTFS-I delivery. MTFS-I is a web-based treatment quality improvement system designed to increase the delivery of evidence-based interventions for behavioral health problems in routine care settings. This version of MTFS-I focuses on family-based services (FBS) for adolescent substance use. FBS, comprising both family participation in treatment and family therapy technique use, have achieved the strongest evidence base for adolescent substance use and are a prime candidate for upgrading treatment quality in outpatient care. For FBS to fulfill their potential for widespread dissemination, FBS implementation must be bolstered by effective quality procedures that support sustainable delivery in usual care.

METHODS/DESIGN

Adapted from measurement feedback systems for client outcomes, MTFS-I contains three synergistic components: (a) weekly reporter training modules to instruct therapists in reliable post-session self-reporting on FBS utilization; (b) weekly mock session videos of FBS interventions (5-8 min) for supportive training in, and practice coding of, high-quality FBS; and (c) monthly feedback reports to therapists and supervisors displaying aggregated data on therapist-reported FBS use. MTFS-I is hosted online and requires approximately 20 min per week to complete. The study will experimentally compare two well-established implementation strategies designed to foster ongoing MTFS-I usage: Core Training, consisting of two 3-h training sessions focused on FBS site mapping, selecting FBS improvement goals, and sustaining MTFS-I, followed by routine remote technical assistance; and Core + Facilitation, which boosts Core Training sessions with collaborative phone-based clinical consultation and on-site facilitation meetings for 1 year to promote FBS goal achievement. The study design is a cluster randomized trial testing Core Training versus Core + Facilitation in ten substance use treatment clinics. Study aims will compare conditions on MTFS-I uptake, FBS delivery (based on therapist-report and observational data), and 1-year client outcomes.

DISCUSSION

Study contributions to implementation science and considerations of MTFS-I sustainability are discussed.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03342872 . Registered 10 November 2017.

Assessment of the relationship between a written measure of empathy and an independently rated interview of Motivational Interviewing

Motivational Interviewing (MI) is an evidence-based practice shown to be effective when working with people in treatment for substance use disorders. However, MI is a complex treatment modality optimized by training with feedback. Feedback, assessment and monitoring of treatment fidelity require measurement, which is typically done using audiotaped sessions. The gold standard for such measurement of MI skill has been an audiotaped interview, scored by a rater with a detailed structured instrument such as the Motivational Interviewing Treatment Integrity 2.0 (MITI 2.0) Coding System (Moyers, et al., 2005). The Helpful Responses Questionnaire (HRQ) (Miller, Hedrick, & Orlofsky, 1991) is a pen-and-paper test of empathy (a foundational MI skill) that does not require an audiotaped session. A randomized trial of three different regimens for training counselors in MI (live supervision using Teleconferencing, Tape-based supervision and Workshop only) (Smith et al., 2012) offered the opportunity to evaluate the performance of the HRQ as a measure of MI ability, compared to the several MITI 2.0 global scores and subscales. Participants were counselors (N=97) working at community-based substance use treatment programs, whose MI proficiency was measured at four time points: baseline (before an initial 2-day MI workshop), post-workshop, 8weeks post-workshop (i.e., post-supervision), and 20weeks post-workshop with both MITI 2.0 and HRQ. HRQ total scores correlated significantly with the Reflection to Question Ratio from the MITI 2.0 at post-workshop (r=0.33), week 8 (r=0.34), and week 20 (r=0.38), and with the Spirit (r=0.32) and Empathy (r=0.32) global scores at week 20. Correlations of HRQ with other MITI 2.0 subscales and time points after workshop were small and not significant. As predicted, HRQ scores differed between training conditions (X²(2)=7.88, p=0.02), with counselors assigned to live supervision achieving better HRQ scores than those in Workshop only. In summary, HRQ is a modestly accurate measure, mainly of the Reflection to Question Ratio, considered a core marker of MI skill. It is sensitive to training effects and may help identify counselors needing more intensive supervision. Given its ease of administration and scoring, HRQ may be a useful marker of MI skill during training efforts.