Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Goal-Directed Treatment of Patients With Anxiety and Mood Disorders in a Regular Curative Mental Health Care Setting

This study examined whether goal-directed treatment leads to improved treatment outcomes for patients with a primary mood or anxiety disorder and whether beneficial outcomes are achieved sooner compared to treatment as usual. In a quasi-experimental controlled study with a nested design, 17 therapists received training in goal-directed treatment and treated 105 patients with anxiety or mood disorders using principles of goal-directed treatment. Treatment results on a generic self-report instrument were compared with two control groups: a historical control group consisting of 16 of the 17 participating therapists, who provided treatment as usual to 97 patients before having received training in goal-directed treatment, and a parallel control group consisting of various therapists, who provided treatment as usual to 105 patients. Symptom reduction on a self-report measure was compared using multilevel analysis. A survival analysis was performed to assess whether a satisfactory end state had been reached sooner after goal-directed treatment. The results of this study show that goal-directed treatment only led to a significantly better overall treatment outcome compared to the parallel treatment as usual group. Furthermore, goal-directed treatment was significantly shorter than both treatment as usual groups. In conclusion, this research suggest that goal-directed treatment led to a similar or better treatment outcome in a shorter amount of time.

Causal feature selection using a knowledge graph combining structured knowledge from the biomedical literature and ontologies: A use case studying depression as a risk factor for Alzheimer's disease

BACKGROUND

Causal feature selection is essential for estimating effects from observational data. Identifying confounders is a crucial step in this process. Traditionally, researchers employ content-matter expertise and literature review to identify confounders. Uncontrolled confounding from unidentified confounders threatens validity, conditioning on intermediate variables (mediators) weakens estimates, and conditioning on common effects (colliders) induces bias. Additionally, without special treatment, erroneous conditioning on variables combining roles introduces bias. However, the vast literature is growing exponentially, making it infeasible to assimilate this knowledge. To address these challenges, we introduce a novel knowledge graph (KG) application enabling causal feature selection by combining computable literature-derived knowledge with biomedical ontologies. We present a use case of our approach specifying a causal model for estimating the total causal effect of depression on the risk of developing Alzheimer's disease (AD) from observational data.

METHODS

We extracted computable knowledge from a literature corpus using three machine reading systems and inferred missing knowledge using logical closure operations. Using a KG framework, we mapped the output to target terminologies and combined it with ontology-grounded resources. We translated epidemiological definitions of confounder, collider, and mediator into queries for searching the KG and summarized the roles played by the identified variables. We compared the results with output from a complementary method and published observational studies and examined a selection of confounding and combined role variables in-depth.

RESULTS

Our search identified 128 confounders, including 58 phenotypes, 47 drugs, 35 genes, 23 collider, and 16 mediator phenotypes. However, only 31 of the 58 confounder phenotypes were found to behave exclusively as confounders, while the remaining 27 phenotypes played other roles. Obstructive sleep apnea emerged as a potential novel confounder for depression and AD. Anemia exemplified a variable playing combined roles.

CONCLUSION

Our findings suggest combining machine reading and KG could augment human expertise for causal feature selection. However, the complexity of causal feature selection for depression with AD highlights the need for standardized field-specific databases of causal variables. Further work is needed to optimize KG search and transform the output for human consumption.

Does loss to follow-up lead to an overestimation of treatment success? Findings from a spine surgery registry of over 15,000 patients

PURPOSE

Patient-reported outcome measures (PROMs) are integral to the assessment of treatment success, but loss to follow-up (attrition) may lead to bias in the results reported. We sought to evaluate the extent, nature and implications of attrition in a long-established, single-centre spine registry.

METHODS

The registry contained the data of 15,264 consecutive spine surgery patients. PROMs included the Core Outcome Measures Index (COMI) and a rating of the Global Treatment Outcome (GTO) and Satisfaction with Care. Baseline characteristics associated with returning a 12-month PROM (= "responder") were analysed (logistic regression). The 3-month outcomes of 12-month responders versus 12-month non-responders were compared (ANOVA and Chi-square).

RESULTS

In total, 14,758/15,264 (97%) patients (60 ± 17y; 46% men) had consented to the use of their registry data for research. Preoperative, 3-month post-operative and 12-month post-operative PROMs were returned by 91, 90 and 86%, respectively. Factors associated with being a 12-month responder included: greater age, born in the country of the study, no private/semi-private insurance, better baseline status (lower COMI score), fewer previous surgeries, less comorbidity and no perioperative medical complications. 12-month non-responders had shown significantly worse outcomes in their 3-month PROMs than had 12-month responders (respectively, 66% vs 80% good GTO ("treatment helped/helped a lot"); 77% vs 88% satisfied/very satisfied; and 49% vs 63% achieved MCIC on COMI).

CONCLUSION

Although attrition in this cohort was relatively low, 12-month non-responders displayed distinctive characteristics and their early outcomes were significantly worse than those of 12-month responders. If loss to follow-up is not addressed, treatment success will likely be overestimated, with erroneously optimistic results being reported.

Comparing the effectiveness and predictors of cognitive behavioural therapy-enhanced between patients with various eating disorder diagnoses: a naturalistic study

Cognitive behaviour therapy-enhanced (CBT-E) is an effective treatment for non-underweight patients with eating disorders. Its efficacy and effectiveness is investigated mostly among transdiagnostic samples and remains unknown for binge eating disorder. The aim of the present study was to assess several treatment outcome predictors and to compare effectiveness of CBT-E among adult out-patients with bulimia nervosa (n=370), binge eating disorder (n=113), and those with a restrictive food pattern diagnosed with other specified feeding and eating disorders (n=139). Effectiveness of CBT-E was assessed in routine clinical practice in a specialised eating disorders centre. Eating disorder pathology was measured with the EDEQ pre- and post-treatment, and at 20 weeks follow-up. Linear mixed model analyses with fixed effect were performed to compare treatment outcome among the eating disorder groups. Several predictors of treatment completion and outcome were examined with a regression analysis. No predictors for drop-out were found, except the diagnosis of bulimia nervosa. Eating disorder pathology decreased among all groups with effect sizes between 1.43 and 1.70 on the EDE-Q total score. There were no differences in remission rates between the three groups at end of treatment or at follow-up. Eating disorder severity at baseline affected treatment response. The results can be generalised to other specialised treatment centres. No subgroup of patients differentially benefited from CBT-E supporting the transdiagnostic perspective for the treatment of eating disorders. Longer-term follow-up data are necessary to measure persistence of treatment benefits.

Key learning aims

(1)

Impact of COVID‐19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands

Objective

The uptake of digital interventions in mental health care (MHC) has been slow, as many therapists and patients believe that in‐person contact is essential for establishing a good working relationship and good outcomes in treatment. The public health policies regarding social distancing during the coronavirus disease‐2019 (COVID‐19) pandemic forced an abrupt transformation of MHC provisions for outpatients: Since mid‐March 2020, nearly all in‐person contact was replaced with videoconferencing. The COVID‐19 crisis offered a unique opportunity to investigate whether MHC with videoconferencing yields inferior results as compared to in‐person interventions.

Method

In a large urban MHC facility in the Netherlands, measurement‐based care is routine practice. Outcome data are regularly collected to support shared decision making and monitor patient progress. For this study, pretest and post‐test data were used to compare outcomes for three cohorts: treatments performed prior to, partially during and entirely during the COVID‐19 lockdown. Outcomes were compared in two large data sets: Basic MHC (N = 1392) and Specialized MHC (N = 1040).

Results

Therapeutic outcomes appeared robust for COVID‐19 conditions across the three cohorts: No differences in outcomes were found between treatments that were conducted during lockdown compared to in‐person treatments prior to COVID‐19, or treatments which started in‐person, but needed to be continued by means of videoconferencing.

Discussion

Videoconferencing care during the COVID‐19 pandemic had similar outcomes compared to traditional in‐person care. These real‐world results corroborate findings of previous randomized controlled studies and meta‐analyses in which videoconferencing and in‐person care has been directly compared in terms of clinical effectiveness.

Feasibility of a Mobile Health App for Routine Outcome Monitoring and Feedback in SMART Recovery Mutual Support Groups: Stage 1 Mixed Methods Pilot Study

Background

Mutual support groups are an important source of long-term help for people impacted by addictive behaviors. Routine outcome monitoring (ROM) and feedback are yet to be implemented in these settings. SMART Recovery mutual support groups focus on self-empowerment and use evidence-based techniques (eg, motivational and behavioral strategies). Trained facilitators lead all SMART Recovery groups, providing an opportunity to implement ROM.

Objective

The aim of this stage 1 pilot study is to explore the feasibility, acceptability, and preliminary outcomes of a novel, purpose-built mobile health ROM and feedback app (SMART Track) in mutual support groups coordinated by SMART Recovery Australia (SRAU) over 8 weeks.

Methods

SMART Track was developed during phase 1 of this study using participatory design methods and an iterative development process. During phase 2, 72 SRAU group participants were recruited to a nonrandomized, prospective, single-arm trial of the SMART Track app. Four modes of data collection were used: ROM data directly entered by participants into the app; app data analytics captured by Amplitude Analytics (number of visits, number of unique users, visit duration, time of visit, and user retention); baseline, 2-, and 8-week follow-up assessments conducted through telephone; and qualitative telephone interviews with a convenience sample of study participants (20/72, 28%) and facilitators (n=8).

Results

Of the 72 study participants, 68 (94%) created a SMART Track account, 64 (88%) used SMART Track at least once, and 42 (58%) used the app for more than 5 weeks. During week 1, 83% (60/72) of participants entered ROM data for one or more outcomes, decreasing to 31% (22/72) by the end of 8 weeks. The two main screens designed to provide personal feedback data (Urges screen and Overall Progress screen) were the most frequently visited sections of the app. Qualitative feedback from participants and facilitators supported the acceptability of SMART Track and the need for improved integration into the SRAU groups. Participants reported significant reductions between the baseline and 8- week scores on the Severity of Dependence Scale (mean difference 1.93, SD 3.02; 95% CI 1.12-2.73) and the Kessler Psychological Distress Scale-10 (mean difference 3.96, SD 8.31; 95% CI 1.75-6.17), but no change on the Substance Use Recovery Evaluator (mean difference 0.11, SD 7.97; 95% CI –2.02 to 2.24) was reported.

Conclusions

Findings support the feasibility, acceptability, and utility of SMART Track. Given that sustained engagement with mobile health apps is notoriously difficult to achieve, our findings are promising. SMART Track offers a potential solution for ROM and personal feedback, particularly for people with substance use disorders who attend mutual support groups.

Trial Registration

Australian New Zealand Clinical Trials Registry ACTRN12619000686101; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377336

International Registered Report Identifier (IRRID)

RR2-10.2196/15113

The effect of home visits as an additional recruitment step on the composition of the final sample: a cross-sectional analysis in two study centers of the German National Cohort (NAKO)

Background

Participation in epidemiologic studies has been declining over the last decades. In addition to postal invitations and phone calls, home visits can be conducted to increase participation. The aim of this study was therefore to evaluate the effects of home visits in terms of response increase and composition of the additionally recruited and final sample.

Methods

In the framework of the German National Cohort (NAKO) recruitment process, two of 18 study centers, Halle (Saale) and Berlin-Center, performed home visits as additional recruitment step after postal invitation and reminders. Response increase was calculated and differences between participants recruited via home visits and standard recruitment were examined. Proportions are presented as percentages with 95%-confidence intervals.

Results

In the general population in Halle, 21.3-22.8% participated after postal invitation and two reminders in the five assessed recruitment waves. The increase of the overall response was 2.8 percentage points (95%confidence interval: 1.9-4.0) for home visits compared to 2.4 percentage points (95%CI: 1.7-3.3) for alternatively sent third postal reminder. Participants recruited via home visits had similar characteristics to those recruited via standard recruitment. Among persons of Turkish descent in Berlin-Center site of the NAKO, home visits conducted by native speakers increased the participation of women, persons living together with their partner, were born in Turkey, had lower German language skills, lower-income, lower education, were more often smokers and reported more often diabetes and depression to a degree which changed overall estimates for this subsample.

Conclusions

As an additional recruitment measure in the general population, home visits increased response only marginally, and the through home visits recruited participants did not differ from those already recruited. Among persons with migration background, home visits by a native speaker increased participation of persons not reached by the standard recruitment, but the effects of using a native speaker approach could not be separated from the effect of home visits.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01357-z.

What Gets Measured Gets Done: How Mental Health Agencies can Leverage Measurement-Based Care for Better Patient Care, Clinician Supports, and Organizational Goals

Mental health clinicians and administrators are increasingly asked to collect and report treatment outcome data despite numerous challenges to select and use instruments in routine practice. Measurement-based care (MBC) is an evidence-based practice for improving patient care. We propose that data collected from MBC processes with patients can be strategically leveraged by agencies to also support clinicians and respond to accountability requirements. MBC data elements are outlined using the Precision Mental Health Framework (Bickman et al. in Adm Policy Mental Health Mental Health Serv Res 43:271-276, 2016), practical guidance is provided for agency administrators, and conceptual examples illustrate strategic applications of one or more instruments to meet various needs throughout the organization.

Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Background

Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic.

Methods

The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated.

Results

Provider‐based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r = .50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading.

Conclusions

Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.