Using Machine Learning of Online Expression to Explain Recovery Trajectories: Content Analytic Approach to Studying a Substance Use Disorder Forum

BACKGROUND

Smartphone-based apps are increasingly used to prevent relapse among those with substance use disorders (SUDs). These systems collect a wealth of data from participants, including the content of messages exchanged in peer-to-peer support forums. How individuals self-disclose and exchange social support in these forums may provide insight into their recovery course, but a manual review of a large corpus of text by human coders is inefficient.

OBJECTIVE

The study sought to evaluate the feasibility of applying supervised machine learning (ML) to perform large-scale content analysis of an online peer-to-peer discussion forum. Machine-coded data were also used to understand how communication styles relate to writers' substance use and well-being outcomes.

METHODS

Data were collected from a smartphone app that connects patients with SUDs to online peer support via a discussion forum. Overall, 268 adult patients with SUD diagnoses were recruited from 3 federally qualified health centers in the United States beginning in 2014. Two waves of survey data were collected to measure demographic characteristics and study outcomes: at baseline (before accessing the app) and after 6 months of using the app. Messages were downloaded from the peer-to-peer forum and subjected to manual content analysis. These data were used to train supervised ML algorithms using features extracted from the Linguistic Inquiry and Word Count (LIWC) system to automatically identify the types of expression relevant to peer-to-peer support. Regression analyses examined how each expression type was associated with recovery outcomes.

RESULTS

Our manual content analysis identified 7 expression types relevant to the recovery process (emotional support, informational support, negative affect, change talk, insightful disclosure, gratitude, and universality disclosure). Over 6 months of app use, 86.2% (231/268) of participants posted on the app's support forum. Of these participants, 93.5% (216/231) posted at least 1 message in the content categories of interest, generating 10,503 messages. Supervised ML algorithms were trained on the hand-coded data, achieving F1-scores ranging from 0.57 to 0.85. Regression analyses revealed that a greater proportion of the messages giving emotional support to peers was related to reduced substance use. For self-disclosure, a greater proportion of the messages expressing universality was related to improved quality of life, whereas a greater proportion of the negative affect expressions was negatively related to quality of life and mood.

CONCLUSIONS

This study highlights a method of natural language processing with potential to provide real-time insights into peer-to-peer communication dynamics. First, we found that our ML approach allowed for large-scale content coding while retaining moderate-to-high levels of accuracy. Second, individuals' expression styles were associated with recovery outcomes. The expression types of emotional support, universality disclosure, and negative affect were significantly related to recovery outcomes, and attending to these dynamics may be important for appropriate intervention.

Mental health triage tools: A narrative review

Mental Health Triage (MHT) tools may be defined as any clinician administered scale that specifies psychiatric signs or symptoms, proposes a corresponding service response, and determines priority categories based on the level of perceived acuity. Multiple MHT tools are used across different jurisdictions and care settings. This article summarizes the literature on MHT tools, describes the available tools and the supportive evidence, evaluates the impact and clinical applications, and compares their strengths and weaknesses. This review utilized a systematic review process to identify articles examining MHT tools. Several benefits of using MHT tools are described; however, in general, the supportive evidence for their use is lacking. A modified Australasian Triage Scale has the strongest evidence base for use in emergency settings; however, further data are needed to establish improved outcomes. There is limited evidence for the use of MHT tools in ambulatory or primary care settings. No evidence was found supporting any one tool as effective in guiding service responses across the entire clinical spectrum. Future research could focus on developing and evaluating MHT tools that service all levels of illness presentation. Additionally, more robust studies are required to support the use of MHT tools in emergency settings. Finally, there is an impetus for the development and evaluation of MHT tools in ambulatory, community, and primary care settings.

"I Want to Be Stepping in More" - Professional Online Forum Moderators' Experiences of Supporting Individuals in a Suicide Crisis

INTRODUCTION

Individuals experiencing suicidal crises increasingly turn to online mental health forums for support. Support can come from peers but also from online moderators, many of whom are trained health professionals. Much is known about users' forum experiences; however, the experiences of professional moderators who work to keep users safe has been overlooked. The beneficial nature of online forums cannot be fully realized until there is a clearer understanding of both parties' participation. This study explored the experiences of professional online forum moderators engaged in suicide prevention.

MATERIALS AND METHODS

A purposive sample of professionally qualified moderators was recruited from three online mental health organizations. In-depth semi-structured, video-recorded interviews were conducted with 15 moderators (3 male, 12 female), to explore their experiences and perceptions of working in online suicide prevention spaces. Data was analyzed using inductive thematic analysis.

RESULTS

Five themes were identified related to the experiences and challenges for moderators. These were the sense of the unknown, the scope of the role, limitations of the written word, volume of tasks, and balancing individual vs. community needs.

DISCUSSION

Findings indicate that the professionally qualified moderator role is complex and multifaceted, with organizations failing to recognize these aspects. Organizations restrict moderators from using their full therapeutic skill set, limiting them to only identifying and re-directing at-risk users to crisis services. The benefits of moderated online forums could be enhanced by allowing moderators to use more of their skills. To facilitate this, in-situ research is needed that examines how moderators use their skills to identify at-risk users.

Suicide Risk and Protective Factors in Online Support Forum Posts: Annotation Scheme Development and Validation Study

BACKGROUND

Online communities provide support for individuals looking for help with suicidal ideation and crisis. As community data are increasingly used to devise machine learning models to infer who might be at risk, there have been limited efforts to identify both risk and protective factors in web-based posts. These annotations can enrich and augment computational assessment approaches to identify appropriate intervention points, which are useful to public health professionals and suicide prevention researchers.

OBJECTIVE

This qualitative study aims to develop a valid and reliable annotation scheme for evaluating risk and protective factors for suicidal ideation in posts in suicide crisis forums.

METHODS

We designed a valid, reliable, and clinically grounded process for identifying risk and protective markers in social media data. This scheme draws on prior work on construct validity and the social sciences of measurement. We then applied the scheme to annotate 200 posts from r/SuicideWatch-a Reddit community focused on suicide crisis.

RESULTS

We documented our results on producing an annotation scheme that is consistent with leading public health information coding schemes for suicide and advances attention to protective factors. Our study showed high internal validity, and we have presented results that indicate that our approach is consistent with findings from prior work.

CONCLUSIONS

Our work formalizes a framework that incorporates construct validity into the development of annotation schemes for suicide risk on social media. This study furthers the understanding of risk and protective factors expressed in social media data. This may help public health programming to prevent suicide and computational social science research and investigations that rely on the quality of labels for downstream machine learning tasks.

An analysis of the diagnostic accuracy and peer-to-peer health information provided on online health forums for heart failure

AIMS

To examine the accuracy of diagnostic responses and types of information provided on online health forums.

DESIGN

Qualitative descriptive study.

METHODS

This paper reports the findings of a thematic analysis of peer responses to posts included on heart failure online health forums, to understand the quality and types of information provided. Responses posted between March 2016 and March 2019 were screened, collected and analysed thematically using Braun & Clarke. Themes were conceptually underpinned by Normalization Process Theory. Responses were assessed for quality against the NICE and SIGN guidelines to determine whether they were evidence based or not.

RESULTS

The total number of responses collected for analysis was 639. Five main themes were identified: diagnostic, experiential, informational, peer relations and relationships with healthcare professionals. Out of 298 diagnostic responses, 5% were guideline evidence-based and 6% had information that were partly evidence-based. Non-evidence based and potentially dangerous responses were 10%. Experiential responses were 10%; 23% included advice that was not supported with any clinical evidence; and 46% signposted users to other online references/healthcare professionals.

CONCLUSION

Online health communication largely focuses on provision of experiential responses to assist those in need of pre- or post-diagnosis advice and support. However, there is evidence of inaccurate information provision which suggests the use of a moderator would be beneficial.

IMPACT

This study suggests heart failure online health forums are a source of support, however, there are potential risks. Increasing nurses and other health care professional's awareness of online health forums will be important. Additional training is needed to help them learn more about patient's use of online health forums, to gain a better understanding about the types of information sought, and how best to address such knowledge deficits. Healthcare systems must ensure sufficient time and resources are available to meet information needs for people with heart failure.

Suicidal behaviours and moderator support in online health communities: a scoping review

OBJECTIVES

Online support can be a crucial source of support for individuals experiencing suicidal behaviours, with forum moderators being pivotal in terms of the role they play in times of personal mental health emergencies. This study identified what is empirically known about the professional practices of health professionals who are online mental health forum moderators and provide support to individuals experiencing suicidal behaviours.

DESIGN

The Levac, Colquhoun and O'Brien extension of the Arksey and O'Malley scoping review framework was used.

SEARCH STRATEGY

The Psychology Collection (EBSCO), PsycINFO (EBSCO), Web of Science, Taylor and Francis Online, SAGE Journals and Science Direct databases were searched for articles that featured a result relating to an online forum; included participants who worked as online moderators or facilitators and focused on suicide or self-harm. Results were limited to peer-reviewed articles published in English from 1990 onwards. As a quality assurance measure, grey literature (nonacademic literature) was not included. Reference lists of included articles were hand-searched.

RESULTS

There were 397 articles initially identified after applying inclusion and exclusion criteria, with five articles included for synthesis. All articles received a moderate quality rating. Only one article featured a moderator who was a qualified health professional; the moderators in the remaining articles were volunteers who undertook preservice training. We found that there is little research that examines the professional working practices of online moderators who support individuals experiencing suicidal behaviours.

CONCLUSIONS

The dearth of research focusing on the professional practices of online forum moderators is cause for concern given that individuals experiencing suicidal behaviours are increasingly turning to online forums when in crisis. Future research should focus on online moderators' practice through interviewing moderators about their professional practices and by examining online moderator practice as it occurs in situ.

Methods in predictive techniques for mental health status on social media: a critical review

Social media is now being used to model mental well-being, and for understanding health outcomes. Computer scientists are now using quantitative techniques to predict the presence of specific mental disorders and symptomatology, such as depression, suicidality, and anxiety. This research promises great benefits to monitoring efforts, diagnostics, and intervention design for these mental health statuses. Yet, there is no standardized process for evaluating the validity of this research and the methods adopted in the design of these studies. We conduct a systematic literature review of the state-of-the-art in predicting mental health status using social media data, focusing on characteristics of the study design, methods, and research design. We find 75 studies in this area published between 2013 and 2018. Our results outline the methods of data annotation for mental health status, data collection and quality management, pre-processing and feature selection, and model selection and verification. Despite growing interest in this field, we identify concerning trends around construct validity, and a lack of reflection in the methods used to operationalize and identify mental health status. We provide some recommendations to address these challenges, including a list of proposed reporting standards for publications and collaboration opportunities in this interdisciplinary space.

A systematic literature review of machine learning in online personal health data

OBJECTIVE

User-generated content (UGC) in online environments provides opportunities to learn an individual's health status outside of clinical settings. However, the nature of UGC brings challenges in both data collecting and processing. The purpose of this study is to systematically review the effectiveness of applying machine learning (ML) methodologies to UGC for personal health investigations.

MATERIALS AND METHODS

We searched PubMed, Web of Science, IEEE Library, ACM library, AAAI library, and the ACL anthology. We focused on research articles that were published in English and in peer-reviewed journals or conference proceedings between 2010 and 2018. Publications that applied ML to UGC with a focus on personal health were identified for further systematic review.

RESULTS

We identified 103 eligible studies which we summarized with respect to 5 research categories, 3 data collection strategies, 3 gold standard dataset creation methods, and 4 types of features applied in ML models. Popular off-the-shelf ML models were logistic regression (n = 22), support vector machines (n = 18), naive Bayes (n = 17), ensemble learning (n = 12), and deep learning (n = 11). The most investigated problems were mental health (n = 39) and cancer (n = 15). Common health-related aspects extracted from UGC were treatment experience, sentiments and emotions, coping strategies, and social support.

CONCLUSIONS

The systematic review indicated that ML can be effectively applied to UGC in facilitating the description and inference of personal health. Future research needs to focus on mitigating bias introduced when building study cohorts, creating features from free text, improving clinical creditability of UGC, and model interpretability.

Improving Moderator Responsiveness in Online Peer Support Through Automated Triage

BACKGROUND

Online peer support forums require oversight to ensure they remain safe and therapeutic. As online communities grow, they place a greater burden on their human moderators, which increases the likelihood that people at risk may be overlooked. This study evaluated the potential for machine learning to assist online peer support by directing moderators' attention where it is most needed.

OBJECTIVE

This study aimed to evaluate the accuracy of an automated triage system and the extent to which it influences moderator behavior.

METHODS

A machine learning classifier was trained to prioritize forum messages as green, amber, red, or crisis depending on how urgently they require attention from a moderator. This was then launched as a set of widgets injected into a popular online peer support forum hosted by ReachOut.com, an Australian Web-based youth mental health service that aims to intervene early in the onset of mental health problems in young people. The accuracy of the system was evaluated using a holdout test set of manually prioritized messages. The impact on moderator behavior was measured as response ratio and response latency, that is, the proportion of messages that receive at least one reply from a moderator and how long it took for these replies to be made. These measures were compared across 3 periods: before launch, after an informal launch, and after a formal launch accompanied by training.

RESULTS

The algorithm achieved 84% f-measure in identifying content that required a moderator response. Between prelaunch and post-training periods, response ratios increased by 0.9, 4.4, and 10.5 percentage points for messages labelled as crisis, red, and green, respectively, but decreased by 5.0 percentage points for amber messages. Logistic regression indicated that the triage system was a significant contributor to response ratios for green, amber, and red messages, but not for crisis messages. Response latency was significantly reduced (P<.001), between the same periods, by factors of 80%, 80%, 77%, and 12% for crisis, red, amber, and green messages, respectively. Regression analysis indicated that the triage system made a significant and unique contribution to reducing the time taken to respond to green, amber, and red messages, but not to crisis messages, after accounting for moderator and community activity.

CONCLUSIONS

The triage system was generally accurate, and moderators were largely in agreement with how messages were prioritized. It had a modest effect on response ratios, primarily because moderators were already more likely to respond to high priority content before the introduction of triage. However, it significantly and substantially reduced the time taken for moderators to respond to prioritized content. Further evaluations are needed to assess the impact of mistakes made by the triage algorithm and how changes to moderator responsiveness impact the well-being of forum members.

Social Connection and Online Engagement: Insights From Interviews With Users of a Mental Health Online Forum

BACKGROUND

Over the past 2 decades, online forums for mental health support have emerged as an important tool for improving mental health and well-being. There has been important research that analyzes the content of forum posts, studies on how and why individuals engage with forums, and how extensively forums are used. However, we still lack insights into key questions on how they are experienced from the perspective of their users, especially those in rural and remote settings.

OBJECTIVE

The aim of our study was to investigate the dynamics, benefits, and challenges of a generalized peer-to-peer mental health online forum from a user perspective; in particular, to better explore and understand user perspectives on connection, engagement, and support offered in such forums; information and advice they gained; and what issues they encountered. We studied experiences of the forums from the perspective of both people with lived experience of mental illness and people who care for people with mental illness.

METHODS

To understand the experience of forum users, we devised a qualitative study utilizing semistructured interviews with 17 participants (12 women and 5 men). Data were transcribed, and a thematic analysis was undertaken.

RESULTS

The study identified 3 key themes: participants experienced considerable social and geographical isolation, which the forums helped to address; participants sought out the forums to find a social connection that was lacking in their everyday lives; and participants used the forums to both find and provide information and practical advice.

CONCLUSIONS

The study suggests that online peer support provides a critical, ongoing role in providing social connection for people with a lived experience of mental ill-health and their carers, especially for those living in rural and remote areas. Forums may offer a way for individuals to develop their own understanding of recovery through reflecting on the recovery experiences and peer support shown by others and individuals enacting peer support themselves. Key to the success of this online forum was the availability of appropriate moderation, professional support, and advice.

Detecting Recovery Problems Just in Time: Application of Automated Linguistic Analysis and Supervised Machine Learning to an Online Substance Abuse Forum

BACKGROUND

Online discussion forums allow those in addiction recovery to seek help through text-based messages, including when facing triggers to drink or use drugs. Trained staff (or "moderators") may participate within these forums to offer guidance and support when participants are struggling but must expend considerable effort to continually review new content. Demands on moderators limit the scalability of evidence-based digital health interventions.

OBJECTIVE

Automated identification of recovery problems could allow moderators to engage in more timely and efficient ways with participants who are struggling. This paper aimed to investigate whether computational linguistics and supervised machine learning can be applied to successfully flag, in real time, those discussion forum messages that moderators find most concerning.

METHODS

Training data came from a trial of a mobile phone-based health intervention for individuals in recovery from alcohol use disorder, with human coders labeling discussion forum messages according to whether or not authors mentioned problems in their recovery process. Linguistic features of these messages were extracted via several computational techniques: (1) a Bag-of-Words approach, (2) the dictionary-based Linguistic Inquiry and Word Count program, and (3) a hybrid approach combining the most important features from both Bag-of-Words and Linguistic Inquiry and Word Count. These features were applied within binary classifiers leveraging several methods of supervised machine learning: support vector machines, decision trees, and boosted decision trees. Classifiers were evaluated in data from a later deployment of the recovery support intervention.

RESULTS

To distinguish recovery problem disclosures, the Bag-of-Words approach relied on domain-specific language, including words explicitly linked to substance use and mental health ("drink," "relapse," "depression," and so on), whereas the Linguistic Inquiry and Word Count approach relied on language characteristics such as tone, affect, insight, and presence of quantifiers and time references, as well as pronouns. A boosted decision tree classifier, utilizing features from both Bag-of-Words and Linguistic Inquiry and Word Count performed best in identifying problems disclosed within the discussion forum, achieving 88% sensitivity and 82% specificity in a separate cohort of patients in recovery.

CONCLUSIONS

Differences in language use can distinguish messages disclosing recovery problems from other message types. Incorporating machine learning models based on language use allows real-time flagging of concerning content such that trained staff may engage more efficiently and focus their attention on time-sensitive issues.