Development and psychometric validation of a novel patient survey to assess perceived quality of substance abuse treatment in South Africa

Psychometric evaluation of the Dutch version of the patient-reported experience measure for addiction treatment (PREMAT-NL)

INTRODUCTION

Patient-centred care has become increasingly important in health care. Patient-reported experience measures (PREM) are used to measure patient experiences in health care, but the availability of psychometrically validated PREMs is limited. The Patient Reported Experience Measure in Addiction Treatment (PREMAT) is a PREM developed with extensive service user input to assess the experiences of people in residential addiction treatment services. In this study we aimed to evaluate the psychometric properties of the Dutch translation of the PREMAT, the PREMAT-NL.

METHODS

Ninety-three participants completed the PREMAT-NL approximately 45 days after starting addiction treatment as part of a naturalistic prospective multi-centre study in Belgium. We examined the factorial structure using principal component analysis with Promax oblique rotation and assessed the internal consistencies of the subscales and total score using Cronbach's α. Additionally, we explored the relationship of PREMAT-NL scores with demographic and clinical variables.

RESULTS

The PREMAT-NL had a four-factor structure, with good internal consistencies of the subscales (Cronbach's α >0.70) and excellent internal consistency of the total score (Cronbach's α = 0.94). The PREMAT-NL total score was negatively skewed, and four score categories were proposed based on z-scores. PREMAT-NL scores correlated weakly with the type of treatment centre (r = 0.21, p < 0.05) and with previously received treatment for addiction (r = -0.25, p < 0.05).

DISCUSSION AND CONCLUSIONS

Although the factor structure and thus the appropriate use of subscales need further investigation, the findings of this study support the use of the PREMAT-NL total score as a valid and reliable PREM to evaluate residential addiction treatment services.

Involvement of people who use alcohol and other drug services in the development of patient-reported measures of experience: A scoping review

INTRODUCTION

Patient-reported measures that assess satisfaction and experience are increasingly utilised in healthcare sectors, including the alcohol and other drug (AOD) sector. This scoping review identifies how and to what extent people accessing AOD services have been involved in the development of satisfaction and experience measures to date.

METHODS

PubMed, EMBASE, CINAHL, Scopus, ProQuest, Google and Google Scholar were searched. Included papers described the development and/or implementation of a multiple-item measure of patient-reported experience or satisfaction specifically for people accessing AOD treatment and/or harm reduction programmes. If there was more than one paper, key papers were chosen that described each measure. The method of development, including service user involvement, was assessed against a framework generated for this review. Two reviewers were involved at each stage.

RESULTS

Thirty measures-23 satisfaction and 7 experience-were identified. Sixteen measures reported some level of involvement by people accessing AOD services in their development, although, for most measures, at a relatively low level. This involvement increased over the time span of the review becoming more frequent in later years. Only four measures were developed for use in harm reduction-specific settings, and fewer than half reported undertaking analysis of underlying scale structure and constructs.

CONCLUSION

Several gaps could be addressed to enhance the measurement of patient-centred care in the AOD sector, including: developing experience measures for use in harm reduction settings and across various AOD settings in a service system; improved reporting of psychometric properties of these measures and increasing commitment to the meaningful involvement of AOD service users in measure development.

PATIENT OR PUBLIC CONTRIBUTION

This scoping review is part of a broader codesign project that involves a partnership between the peak organisation for AOD services and the peer-based AOD consumer organisation in the Australian Capital Territory, Australia. These organisations are working closely together to engage with AOD service users, service providers and policy makers in this codesign project. As such, the Executive Director of the peer-based AOD consumer organisation is involved as a co-author of this scoping review.

Development, acceptability, and perceived effectiveness of a trauma-informed adolescent self-regulation intervention

Youth who have experienced trauma often face challenges with self-regulation and can have diminished health outcomes, including those related to sexual and reproductive health. We developed a 12-session blended e-learning intervention for youth involved in juvenile justice systems to encourage healthy sexual decision-making by improving self-regulation skills. This paper describes the development of the program, e-Practice Self-Regulation, as well as an assessment of program acceptability and perceived effectiveness among youth participants. Data were collected using a feedback survey from youth who both completed (n = 95) and did not complete (n = 80) the intervention. Program acceptability was high among all participants, and over 70 % of both completers and non-completers would recommend e-Practice Self-Regulation to peers. Youth reported positive interactions with facilitators and described sessions as helpful, interesting, and educational. The most common criticism was online session length. A majority in both groups reported perceived effectiveness related to sexual health and self-regulation, which increased with more sessions completed. Results suggest e-Practice Self-Regulation is acceptable to youth participants, who reported learning skills necessary for self-regulation and sexual health and intended to use these skills to avoid unplanned pregnancy.

Substance use referral, treatment utilization, and patient costs associated with problematic substance use in people living with HIV in Cape Town, South Africa

Introduction

Despite efforts to detect and treat problematic substance use (SU) among people living with HIV (PLWH) in South Africa, integration of HIV and SU services is limited. We sought to understand whether PLWH and problematic SU were: (a) routinely referred to SU treatment, a co-located Matrix clinic, (b) used SU treatment services when referred, and (c) the individual amount spent on SU.

Methods

Guided by the RE-AIM implementation science framework, we examined patient-level quantitative screening and baseline data from a pilot clinical trial for medication adherence and problematic SU. Qualitative data came from semi-structured interviews with HIV care providers (N = 8), supplemented by patient interviews (N = 15).

Results

None of the screened patient participants (N = 121) who were seeking HIV care and had problematic SU were engaged in SU treatment, despite the freely available co-located SU treatment program. Only 1.5% of the enrolled patient study sample (N = 66) reported lifetime referral to SU treatment. On average, patients with untreated SU spent 33.3% (SD=34.5%) of their monthly household income on substances. HIV care providers reported a lack of clarity about the SU referral process and a lack of direct communication with patients about patients' needs or interest in receiving an SU referral.

Discussion

SU treatment referrals and uptake were rare among PLWH reporting problematic SU, despite the high proportion of individual resources allocated to substances and the co-located Matrix site. A standardized referral policy between the HIV and Matrix sites may improve communication and uptake of SU referrals.

Factors associated with patient-reported experiences and outcomes of substance use disorder treatment in Cape Town, South Africa

Background

Interventions are needed to improve the quality of South Africa’s substance use disorder (SUD) treatment system. This study aimed to identify factors associated with patient-reported suboptimal access, quality, and outcomes of SUD treatment to guide the design of targeted quality improvement initiatives.

Method

We analysed clinical record and patient survey data routinely collected by SUD services in the Western Cape Province, South Africa. The sample included 1097 treatment episodes, representing 32% of all episodes in 2019. Using multivariate logistic regression, we modelled socio-demographic, substance use and treatment correlates of patient-reported suboptimal access to, quality and outcomes of SUD treatment.

Results

Overall, 37.9% of patients reported substantial difficulties in accessing treatment, 28.8% reported suboptimal quality treatment, and 31.1% reported suboptimal SUD outcomes. The odds of reporting poor access were elevated for patients identifying as Black/African, in residential treatment, with comorbid mental health problems, and longer histories of substance use. Length of substance use, comorbid mental health problems, and prior SUD treatment were associated with greater likelihood of reporting suboptimal quality treatment. Patients with comorbid mental health problems, polysubstance use, who did not complete treatment, and who perceived treatment to be of poor quality were more likely to report suboptimal outcomes.

Conclusion

This study is among the first to use patient-reported experiences and outcome measures to identify targets for SUD treatment improvement. Findings suggest substantial room to improve South African SUD treatment services, with targeted efforts needed to reduce disparities in outcomes for patients of Black/African descent, for those with comorbid mental health problems, and for patients who have chronic substance use difficulties. Interventions to enhance the relevance, appropriateness, and acceptability of SUD services for these patient sub-groups are needed to improve system performance.

Factors predicting satisfaction in outpatient substance abuse treatment: a prospective follow-up study

BACKGROUND

While treatment satisfaction has been associated with better outcomes in substance abuse treatment, there is an obvious need for a more profound understanding of what predicts client's satisfaction with treatment. This study elucidates factors relevant to treatment outcome measured at follow-up in terms of satisfaction with the treatment received.

METHODS

The research was implemented as a multisite study in outpatient clinics (N = 7) in southern and western Finland. Data consists of therapists (N = 33) and their clients (N = 327). Each consenting client beginning a treatment period was accepted as a research subject and all therapists at the clinics in question participated. The study was conducted as part of the clinic's normal activity. Clients were allocated to therapists according to a randomization list drawn up in advance. Apart from the randomisation and the completion of questionnaires, it did not interfere with the progress of treatment. Follow-up lasted 6 months. Multiple Classification Analysis (MCA) was used through combinations of variables organized by content, e.g. client demographics, previous substance use, therapist's characteristics and client's expectations. The analyses were based in part on conventional statistical testing (t -test, χ2-test, ANOVA).

RESULTS

Among 37 independent variables few were statistically significant in the final model. The results suggest that high treatment expectations at baseline are a strong predictor of satisfaction at follow-up. Also, previous substance use predicted treatment satisfaction; people using multiple substances were less satisfied than those taking only one substance. Stronger predictors reduced the statistical significance of those independent variables that were statistically significant in the first analyses. Therefore, therapist's role in recovery and readiness to change should be also seen as antecedents to treatment satisfaction.

CONCLUSIONS

It seems that treatment expectations are fulfilled among those participating in follow-up. Yet many are lost during treatment and by follow-up. Service users have experiential knowledge that differs from professionals' and policymakers' knowledge. It is clinically relevant to understand what factors affect client's satisfaction. Hence, it is possible to identify the population whose treatment should receive the most attention, how the client's experience, their commitment to treatment, and treatment effectiveness could be improved.

South African service users' perceptions of patient-reported outcome and experience measures for adolescent substance use treatment: A qualitative study

INTRODUCTION AND AIMS

Patient-reported outcome measures (PROM) and experience measures (PREM) for substance use disorder (SUD) treatment exist for adults but have unknown relevance for adolescents. This study aimed to explore adolescents' perceptions of effective SUD treatment and possible barriers to completing PROMs and PREMs to guide efforts to adapt the South African Addiction Treatment Services Assessment (SAATSA) for adolescents.

DESIGN AND METHODS

Five focus groups were conducted with 38 adolescent service users recruited from residential and outpatient SUD treatment facilities in South Africa. Group discussions explored perceptions of treatment components necessary for desired SUD treatment outcomes, treatment experiences that support engagement in care, and perceptions of PROM and PREM completion.

RESULTS

Participants viewed treatment elements that enhance motivation for change, coping and emotional regulation; provide recreational alternatives to substance use; and improve family relationships and home environments as critical to positive treatment outcomes. They reflected that provider characteristics and developmentally and culturally appropriate services facilitated engagement in treatment. PROM and PREM completion seemed acceptable, with participants suggesting ways to enhance their appeal.

DISCUSSION AND CONCLUSION

Findings confirm that adult-oriented PROMs and PREMs require adaptation for adolescents. Service user inputs identified ways to expand the content of the SAATSA to better reflect adolescents' treatment priorities. These inputs have also guided changes to item formulation and administration procedures to enhance the SAATSA's acceptability for adolescents. Ensuring the SAATSA addresses treatment outcomes and experiences that matter to adolescents is vital for generating information to guide improvements to adolescent SUD services.

Performance Measurement in Mental Health and Addictions Systems: A Scoping Review

OBJECTIVE

The purpose of this study was to evaluate how performance is defined, conceptualized, and measured in mental health and addiction service systems around the world.

METHOD

We conducted a systematic scoping review of English-language scientific and gray literature published from 2005 to 2015. Eligible documents (n = 222) described performance measurement systems and outlined the theory or empirical evidence for indicators. We used a structured approach for data extraction and descriptive and thematic analysis, supplemented with stakeholder consultation.

RESULTS

We identified seven themes in the literature: similarity in performance domains across frameworks; the ability of frameworks to inform care quality at client, program/facility, and system levels; the predominance of indicators of process and outcome, over structure; the lack of evidence on the links between domains and/or indicators; common, but limited, evaluation of family/caregiver involvement; equity as a cross-cutting domain of performance; and limited attention to performance measurement in peer support services.

CONCLUSIONS

The literature on performance measurement in mental health and addictions services is vast, and a wide variety of indicators is available to those designing a measurement system. Evaluations of commonly used performance indicators have yielded mixed evidence on their ability to discriminate high- and low-performing service providers, and their sensitivity to changes in policies and practices. As performance measurement efforts grow in scope and complexity, work will be needed to ensure that indicators are fair, appropriate, and suited to support quality improvement in services of different types.

A Mixed-Methods Evaluation of the Implementation of a Performance Measurement System for South Africa's Substance Use Treatment Services

OBJECTIVE

Minimal knowledge exists on the factors that affect implementation of performance measurement systems, particularly in low- and middle-income countries (LMICs). To address this, we describe the implementation of a performance measurement system for South Africa's substance abuse treatment services known as the Service Quality Measures (SQM) initiative.

METHOD

We conducted a mixed-methods evaluation of system implementation. We surveyed 81 providers about the extent of system implementation within their agencies and the factors that facilitated implementation. We conducted 26 in-depth interviews of providers' perceived barriers and facilitators to implementation.

RESULTS

The overall penetration of this system was high. Almost all providers viewed the system as feasible to implement, acceptable, appropriate for use in their context, and useful for guiding service improvements. However, the extent of implementation varied significantly across sites (p < .05). Leadership support (p < .05) was associated with increased implementation in multivariable analyses. Providers reflected that high rates of patient attrition, variability in willingness to implement the system, and limited capacity for interpreting performance feedback affected the extent of system implementation.

CONCLUSIONS

It is feasible to implement a performance measurement system in LMICs if the system is acceptable, appropriate, and useful to providers. To ensure the utility of this system for treatment service strengthening, system implementation must be optimized. Efforts to enhance target population coverage, strengthen leadership support for performance measurement, and build capacity for performance feedback utilization may enhance the implementation of this performance measurement system.

Patient experiences with interdisciplinary treatment for substance dependence: an assessment of quality indicators based on two national surveys in Norway

Purpose: The quality of health care is often measured using quality indicators, which can be utilized to compare the performance of health-care providers. Conducting comparisons in a meaningful and fair way requires the quality indicators to be adjusted for patient characteristics and other individual-level factors. The aims of the study were to develop and test a case-mix adjustment model for quality indicators based on patient-experience surveys among inpatients receiving interdisciplinary treatment for substance dependence, and to establish whether the quality indicators discriminate between health care providers. Patients and methods: Data were collected through two national surveys involving inpatients receiving residential treatment in Norway in 2013 and 2014. The same questionnaire was used in both surveys, and comprised three patient-experience scales. The scales are reported as national quality indicators, and associations between the scales and patient characteristics were tested through multilevel modeling to establish a case-mix model. The intraclass correlation coefficient was computed to assess the amount of variation at the hospital-trust level. Results: The intraclass correlation coefficient for the patient-reported experience scales varied from 2.3% for "treatment and personnel" to 8.1% for "milieu". Multivariate multilevel regression analyses showed that alcohol reported as the most frequently used substance, gender and age were significantly associated with two of the three scales. The length of stay at the institution, pressure to be admitted for treatment, and self-perceived health were significantly related to all three scales. Explained variance at the individual level was approximately 7% for all three scales. Conclusion: This study identified several important case-mix variables for the patient-based quality indicators and systematic variations at the hospital-trust level. Future research should assess the association between patient-based quality indicators and other quality indicators, and the predictive validity of patient-experience indicators based on on-site measurements.

Protocol for development and validation of a context-appropriate tool for assessing organisational readiness for change in primary health clinics in South Africa

INTRODUCTION

A large treatment gap for common mental disorders (such as depression) exists in South Africa. Comorbidity with other chronic diseases, including HIV and diseases of lifestyle, is an increasing public health concern globally. Currently, primary health facilities as points of care for those with chronic disease provide limited services for common mental disorders. Assessing organisational readiness for change (ORC) towards adopting health innovations (such as mental health services) using contextually appropriate measures is needed to facilitate implementation of these services. This study aims to investigate the validity of the Texas Christian University Organisational Readiness for Change (TCU-ORC) scale in the South African context. Subsequently, we will develop a shortened version of this scale. This study is nested within Project MIND, a multiyear randomised controlled trial that is testing two different approaches for integrating counselling for common mental disorders into chronic disease care. Although the modified, contextually appropriate ORC measure resulting from the proposed study will be developed in the context of integrating mental health into primary healthcare services, the potential for the tool to be generalised to further understanding barriers to any change being implemented in primary care settings is high.

METHODS AND ANALYSIS

We will establish internal consistency (Cronbach's alpha coefficients), test-retest reliability (intraclass correlation coefficient) and construct validity of the long-form TCU-ORC questionnaire. Survey data will be collected from 288 clinical, management and operational staff from 24 primary health facilities where the Project MIND trial is implemented. A modified Delphi approach will assess the content validity of the TCU-ORC items and identify areas for potential adaptation and item reduction.

ETHICS AND DISSEMINATION

Ethical approval has been granted by the South African Medical Research Council (Protocol ID EC004-2-2015, amendment of 20 August 2017). Results will be submitted to peer-reviewed journals relevant to implementation and health systems strengthening.

Substance abuse treatment engagement, completion and short-term outcomes in the Western Cape province, South Africa: Findings from the Service Quality Measures Initiative

BACKGROUND

Optimizing the effectiveness of substance use disorder (SUD) treatment is critical in low-and middle-income countries (LMICs) with limited opportunities for SUD treatment. This is the first study to identify targets for interventions to improve the quality of SUD treatment in a LMIC.

METHOD

We explored correlates of three indicators of treatment quality (treatment engagement, completion and abstinence at treatment exit) using data from a SUD performance measurement system implemented in the Western Cape Province of South Africa. The sample included data from 1094 adult treatment episodes representing 53% of the treatment episodes in 2016. Using multivariate logistic regression analyses, we modeled socio-demographic, substance use and program correlates of treatment engagement, completion, and abstinence at treatment exit.

RESULTS

Overall, 59% of patients completed treatment (48% of patients from outpatient services). Treatment completion was associated with greater likelihood of abstinence at treatment exit. Patients were more likely to complete treatment if they engaged in treatment, were older, and had more severe drug problems (characterized by daily drug use and heroin problems) and attended programs of shorter duration. Residential treatment was associated with greater likelihood of treatment engagement, completion, and abstinence at treatment exit.

CONCLUSION

Improving rates of outpatient treatment completion will enhance the effectiveness of South Africa's SUD treatment system. Interventions that promote engagement in treatment, particularly among younger patients; reduce program length through referral to step-down continuing care; and ensure better matching of drug problem to treatment level and type could improve rates of treatment completion.

Openness to adopting evidence-based practice in public substance use treatment in South Africa using task shifting: Caseload size matters

BACKGROUND

In response to the lack of coverage for substance use treatment in the Western Cape province of South Africa, the local government expanded funding for evidence-based practices (EBPs) for treating substance use. Yet, little is known about provider and staff attitudes towards adopting EBPs in this setting, which is particularly relevant in this context where task shifting clinical care increases demands on paraprofessional providers. This study aimed to (1) assess attitudes towards adopting EBPs among a range of staff working in substance use treatment in Cape Town using a task shifting model; and (2) evaluate factors associated with openness towards adopting EBPs in this setting.

METHODS

Staff (n = 87) were recruited from 11 substance use treatment clinics. Demographics and job-related characteristics were assessed. Staff perceptions of organizational factors were assessed using the TCU Organizational Readiness for Change (ORC) scale. The dependent variable, attitudes towards adopting EBPs, was assessed using the Evidence-Based Practice Attitude Scale (EBPAS).

RESULTS

This study is one of the first to administer the EBPAS in South Africa and found good internal consistency (total score: α = .82). In a multivariable model adjusting for site and factors associated with EBPAS total score at the bivariate level, only smaller caseload size was associated with greater openness to adopting EBPs (B = 1.61, SE = .73; t = 2.21; p<.05).

CONCLUSIONS

As pressure to scale up implementation of EBPs in South African substance use treatment services intensifies, additional efforts are needed to understand barriers to adopt EBPs in this setting. Supporting staff adoption of EBPs in resource-limited settings may require additional resources to limit staff caseloads in the context of task shifting.