A prospective meta-analytic approach in a multisite study of homelessness prevention

Large-scale evaluation efforts and their implications for the field

The BUILD initiative is part of the Diversity Program Consortium, which the National Institutes of Health funded to increase diversity in biomedical research. This chapter aims to identify implications for the field from the multisite evaluation of BUILD initiative programs by reviewing the work undertaken by the authors of the other chapters in this issue. Given the complexities involved in multisite evaluations, innovative approaches and methods were used to balance the needs of each site with the overall objectives of the broader initiative. These approaches included a flexible orientation to the evaluation, mixed-methods designs that prioritized understanding the context before measuring it, and innovative analytic techniques (e.g., meta-analysis) to recognize the uniqueness of each site while providing insights about their cumulative impact. The BUILD initiative evaluation also offered many other valuable lessons about engaging stakeholders, focusing on use, and responding to changing priorities over time.

Homelessness and health-related outcomes: an umbrella review of observational studies and randomized controlled trials

BACKGROUND

Homelessness has been associated with multiple detrimental health outcomes across observational studies. However, relatively few randomized controlled trials (RCTs) have been conducted on people who experience homelessness (PEH). Thus, this umbrella review ranked the credibility of evidence derived from systematic reviews (SRs) and meta-analyses (MAs) of observational studies investigating the associations between homelessness and any health outcome as well as RCTs targeting health needs in this population.

METHODS

Several databases were systematically searched from inception through April 28, 2021. Any SR and/or MA reporting quantitative data and providing a control group were eligible for inclusion. The credibility of the evidence derived from observational studies was appraised by considering the significance level of the association and the largest study, the degree of heterogeneity, the presence of small-study effects as well as excess significance bias. The credibility of evidence was then ranked in five classes. For SRs and/or MAs of RCTs, we considered the level of significance and whether the prediction interval crossed the null. The AMSTAR-2 and AMSTAR-plus instruments were adopted to further assess the methodological quality of SRs and/or MAs. The Newcastle-Ottawa Scale (NOS) was employed to further appraise the methodological quality of prospective cohort studies only; a sensitivity analysis limited to higher quality studies was conducted.

RESULTS

Out of 1549 references, 8 MAs and 2 SRs were included. Among those considering observational studies, 23 unique associations were appraised. Twelve of them were statistically significant at the p≤0.005 level. Included cases had worst health-related outcomes than controls, but only two associations reached a priori-defined criteria for convincing (class I) evidence namely hospitalization due to any cause among PEH diagnosed with HIV infection, and the occurrence of falls within the past year among PEH. According to the AMSTAR-2 instrument, the methodological quality of all included SRs and/or MAs was "critically low." Interventional studies were scant.

CONCLUSION

While homelessness has been repeatedly associated with detrimental health outcomes, only two associations met the criteria for convincing evidence. Furthermore, few RCTs were appraised by SRs and/or MAs. Our umbrella review also highlights the need to standardize definitions of homelessness to be incorporated by forthcoming studies to improve the external validity of the findings in this vulnerable population.

Child Sexual Abuse and Problems Reported by Survivors of CSA: A Meta-Review

A systematic meta-review of reviews and meta-analyses on problems reported by survivors of Child Sexual Abuse (CSA) was conducted. The aim was to comprise a comprehensive overview of 1) problems reported by survivors of CSA, and 2) variables moderating these relations. Two raters independently conducted a search through PubMed, PsychINFO, Campbell Library, Cochrane Library and Web of Science. Twenty-five reviews and meta-analyses reporting on 53 problems related to CSA were analyzed. All fell into five domains: medical, psychological, sexual, repeated (self-)harm, and a final category of other problems. Thirty-six of all problems (68%) were consistently and significantly more commonly reported by CSA survivors as compared to individuals without a history of CSA. Most moderator analyses did not significantly influence these relationships. In conclusion, CSA is associated with various problems across different domains and overall, these problems are prevalent independent of specific characteristics of and circumstances surrounding the abuse.

Review of organizational effects on the outcome of mental health treatments

OBJECTIVE

As there are theoretical, clinical, and "common sense" reasons to expect a relationship between organizational factors and outcome in clinics providing psychotherapy and other mental health treatments, a review of empirical research in this area was undertaken with the aim of finding empirical evidence for organizational effects.

METHODS

A structured search for studies on organizational differences in patient mental health outcomes was performed using EBSCO host, Cochrane Library Database, and the Health Systems Evidence database at McMasters University. Finished studies published in English were included if they presented data from more than one mental health service and used change in symptom, level of functioning, or quality of life as outcome.

RESULTS

The search yielded not more than 19 studies fulfilling inclusion criteria. All studies showed some evidence for organization effects, and there was some evidence for organizational climate and culture explaining differences in outcome.

CONCLUSION

Given that mental health treatments are likely to be especially susceptive to organizational effects, it is remarkable that not more research has been devoted to this. Clearly, more research is needed to study the consequences of work organization for the outcome of psychotherapy. Methodological issues in organizational studies are discussed.

Efficacy of a process improvement intervention on delivery of HIV services to offenders: a multisite trial

OBJECTIVES

We tested a modified Network for the Improvement of Addiction Treatment (NIATx) process improvement model to implement improved HIV services (prevention, testing, and linkage to treatment) for offenders under correctional supervision.

METHODS

As part of the Criminal Justice Drug Abuse Treatment Studies, Phase 2, the HIV Services and Treatment Implementation in Corrections study conducted 14 cluster-randomized trials in 2011 to 2013 at 9 US sites, where one correctional facility received training in HIV services and coaching in a modified NIATx model and the other received only HIV training. The outcome measure was the odds of successful delivery of an HIV service.

RESULTS

The results were significant at the .05 level, and the point estimate for the odds ratio was 2.14. Although overall the results were heterogeneous, the experiments that focused on implementing HIV prevention interventions had a 95% confidence interval that exceeded the no-difference point.

CONCLUSIONS

Our results demonstrate that a modified NIATx process improvement model can effectively implement improved rates of delivery of some types of HIV services in correctional environments.

Developing an instrument for assessing fidelity of motivational care planning: The Aboriginal and Islander Mental health initiative adherence scale

Purpose

The aim of this study was to design and trial an Adherence Scale to measure fidelity of Motivational Care Planning (MCP) within a clinical trial. This culturally adapted therapy MCP uses a client centered holistic approach that emphasises family and culture to motivate healthy life style changes.

Methods

The Motivational Care Planning-Adherence Scale (MCP-AS) was developed through consultation with Aboriginal and Islander Mental Health Initiative (AIMhi) Indigenous and non-Indigenous trainers, and review of MCP training resources. The resultant ten-item scale incorporates a 9-Point Likert Scale with a supporting protocol manual and uses objective, behaviourally anchored criteria for each scale point. A fidelity assessor piloted the tool through analysis of four audio-recordings of MCP (conducted by Indigenous researchers within a study in remote communities in Northern Australia). File audits of the remote therapy sessions were utilised as an additional source of information. A Gold Standard Motivational Care Planning training video was also assessed using the MCP-AS.

Results

The Motivational Care Planning-Adherence Scale contains items measuring both process and content of therapy sessions. This scale was used successfully to assess therapy through observation of audio or video-recorded sessions and review of clinical notes. Treatment fidelity measured by the MCP-AS within the pilot study indicated high fidelity ratings. Ratings were high across the three domains of rapport, motivation, and self-management with especially high ratings for positive feedback and engagement, review of stressors and goal setting.

Conclusions

The Motivational Care Planning-Adherence Scale has the potential to provide a measure of quality of delivery of Motivation Care Planning. The pilot findings suggest that despite challenges within the remote Indigenous community setting, Indigenous therapists delivered therapy that was of high fidelity. While developed as a research tool, the scale has the potential to support fidelity of delivery of Motivation Care Planning in clinical, supervision and training settings. Larger studies are needed to establish inter-rater reliability and internal and external validity.

Predictors of employment for people with severe mental illness: results of an international six-centre randomised controlled trial

BACKGROUND

An international six-centre randomised controlled trial comparing individual placement and support (IPS) with usual vocational rehabilitation for people with serious mental illness found IPS to be more effective for all vocational outcomes.

AIMS

To determine which patients with severe mental illness do well in vocational services and which process and service factors are associated with better outcomes.

METHOD

Patient characteristics and early process variables were tested as predictors of employment outcomes. Service characteristics were explored as predictors of the effectiveness of IPS.

RESULTS

Patients with previous work history, fewer met social needs and better relationships with their vocational workers were more likely to obtain employment and work for longer. Remission and swifter service uptake were associated with working more. Having an IPS service closer to the original IPS model was the only service characteristic associated with greater effectiveness.

CONCLUSIONS

The IPS service was found to be more effective for all vocational outcomes. In addition, maintaining high IPS fidelity and targeting relational skills would be a valuable focus for all vocational interventions, leading to improved employment outcomes. Motivation to find work may be decreased by satisfaction with current life circumstances.

The effectiveness of supported employment for people with severe mental illness: a randomised controlled trial

BACKGROUND

The value of the individual placement and support (IPS) programme in helping people with severe mental illness gain open employment is unknown in Europe. Our aim was to assess the effectiveness of IPS, and to examine whether its effect is modified by local labour markets and welfare systems.

METHODS

312 patients with severe mental illness were randomly assigned in six European centres to receive IPS (n=156) or vocational services (n=156). Patients were followed up for 18 months. The primary outcome was the difference between the proportions of people entering competitive employment in the two groups. The heterogeneity of IPS effectiveness was explored with prospective meta-analyses to establish the effect of local welfare systems and labour markets. Analysis was by intention to treat. This study is registered with ClinicalTrials.gov, with the number NCT00461318.

FINDINGS

IPS was more effective than vocational services for every vocational outcome, with 85 (55%) patients assigned to IPS working for at least 1 day compared with 43 (28%) patients assigned to vocational services (difference 26.9%, 95% CI 16.4-37.4). Patients assigned to vocational services were significantly more likely to drop out of the service and to be readmitted to hospital than were those assigned to IPS (drop-out 70 [45%] vs 20 [13%]; difference -32.1% [95% CI -41.5 to -22.7]; readmission 42 [31%] vs 28 [20%]; difference -11.2% [-21.5 to -0.90]). Local unemployment rates accounted for a substantial amount of the heterogeneity in IPS effectiveness.

INTERPRETATION

Our demonstration of the effectiveness of IPS in widely differing labour market and welfare contexts confirms this service to be an effective approach for vocational rehabilitation in mental health that deserves investment and further investigation.

Outcomes for women with co-occurring disorders and trauma: program-level effects

Program-level effects at 6 months are reported from meta-analysis of a nine-site quasi-experimental study of comprehensive, integrated, trauma-informed, and consumer-involved services for women who have mental health problems, substance use disorders, and who have experienced interpersonal violence. The average weighted effect size is significant for the treatment condition for improved post-traumatic symptoms (p < 0.02), drug use problem severity (p < 0.02), and nearly significant for mental health symptoms (p < 0.06). There is significant heterogeneity in effect sizes across sites. Program-level variables were examined in an effort to explain this heterogeneity. The findings indicate that sites which provided significantly more integrated counseling produced more favorable results in mental health symptoms (p < 0.01) and both alcohol (p < 0.001) and drug use problem severity (p < 0.001). The same trend is observable for reductions in post-traumatic stress symptoms, although the difference does not attain statistical significance.

Outcomes for women with co-occurring disorders and trauma: program and person-level effects

Six-month outcomes are evaluated from a 9-site quasi-experimental study of women with mental health and substance use disorders who have experienced physical or sexual abuse who enrolled in either comprehensive, integrated, trauma-informed, and consumer/survivor/recovering person-involved services (N = 1023) or usual care (N = 983). Mental health, post-traumatic stress symptoms, and substance use outcomes are assessed with multilevel regression models, controlling for program and personal characteristics. Person-level variables predict outcomes independent of intervention condition and, to a small extent, moderate intervention and program effects. In sites where the intervention condition provided more integrated counseling than the comparison condition, there are increased effects on mental health and substance use outcomes; these effects are partially mediated by person-level variables. These results encourage further research to identify the longer-term effects of integrated counseling for women with co-occurring disorders and trauma histories.

Women, Co-occurring Disorders, and Violence Study: Evaluation design and study population

The Women, Co-occurring Disorders, and Violence Study (WCDVS) was a multi-site cooperative study to evaluate new service models for women with co-occurring mental health and substance use disorders and a history of physical and/or sexual abuse. Despite common features in the service interventions and evaluation procedures, diversity across the nine sites plus differences introduced by non-random assignment led to numerous methodological challenges. This article describes the design, measurement, and analysis decisions behind the WCDVS and lays the foundation for understanding participant-level outcomes and service costs. This article also describes the study population, as recruited and following attrition at the 6-month follow-up, in order to address the threat of selection bias to inferences drawn from this multi-site study.