World Trade Center responders in their own words: predicting PTSD symptom trajectories with AI-based language analyses of interviews

Interdisciplinary approach to identify language markers for post-traumatic stress disorder using machine learning and deep learning

Post-traumatic stress disorder (PTSD) lacks clear biomarkers in clinical practice. Language as a potential diagnostic biomarker for PTSD is investigated in this study. We analyze an original cohort of 148 individuals exposed to the November 13, 2015, terrorist attacks in Paris. The interviews, conducted 5-11 months after the event, include individuals from similar socioeconomic backgrounds exposed to the same incident, responding to identical questions and using uniform PTSD measures. Using this dataset to collect nuanced insights that might be clinically relevant, we propose a three-step interdisciplinary methodology that integrates expertise from psychiatry, linguistics, and the Natural Language Processing (NLP) community to examine the relationship between language and PTSD. The first step assesses a clinical psychiatrist's ability to diagnose PTSD using interview transcription alone. The second step uses statistical analysis and machine learning models to create language features based on psycholinguistic hypotheses and evaluate their predictive strength. The third step is the application of a hypothesis-free deep learning approach to the classification of PTSD in our cohort. Results show that the clinical psychiatrist achieved a diagnosis of PTSD with an AUC of 0.72. This is comparable to a gold standard questionnaire (Area Under Curve (AUC) ≈ 0.80). The machine learning model achieved a diagnostic AUC of 0.69. The deep learning approach achieved an AUC of 0.64. An examination of model error informs our discussion. Importantly, the study controls for confounding factors, establishes associations between language and DSM-5 subsymptoms, and integrates automated methods with qualitative analysis. This study provides a direct and methodologically robust description of the relationship between PTSD and language. Our work lays the groundwork for advancing early and accurate diagnosis and using linguistic markers to assess the effectiveness of pharmacological treatments and psychotherapies.

Robust language-based mental health assessments in time and space through social media

In the most comprehensive population surveys, mental health is only broadly captured through questionnaires asking about "mentally unhealthy days" or feelings of "sadness." Further, population mental health estimates are predominantly consolidated to yearly estimates at the state level, which is considerably coarser than the best estimates of physical health. Through the large-scale analysis of social media, robust estimation of population mental health is feasible at finer resolutions. In this study, we created a pipeline that used ~1 billion Tweets from 2 million geo-located users to estimate mental health levels and changes for depression and anxiety, the two leading mental health conditions. Language-based mental health assessments (LBMHAs) had substantially higher levels of reliability across space and time than available survey measures. This work presents reliable assessments of depression and anxiety down to the county-weeks level. Where surveys were available, we found moderate to strong associations between the LBMHAs and survey scores for multiple levels of granularity, from the national level down to weekly county measurements (fixed effects β = 0.34 to 1.82; p < 0.001). LBMHAs demonstrated temporal validity, showing clear absolute increases after a list of major societal events (+23% absolute change for depression assessments). LBMHAs showed improved external validity, evidenced by stronger correlations with measures of health and socioeconomic status than population surveys. This study shows that the careful aggregation of social media data yields spatiotemporal estimates of population mental health that exceed the granularity achievable by existing population surveys, and does so with generally greater reliability and validity.

Conversational assessment using artificial intelligence is as clinically useful as depression scales and preferred by users

BACKGROUND

Depression is prevalent, chronic, and burdensome. Due to limited screening access, depression often remains undiagnosed. Artificial intelligence (AI) models based on spoken responses to interview questions may offer an effective, efficient alternative to other screening methods.

OBJECTIVE

The primary aim was to use a demographically diverse sample to validate an AI model, previously trained on human-administered interviews, on novel bot-administered interviews, and to check for algorithmic biases related to age, sex, race, and ethnicity.

METHODS

Using the Aiberry app, adults recruited via social media (N = 393) completed a brief bot-administered interview and a depression self-report form. An AI model was used to predict form scores based on interview responses alone. For all meaningful discrepancies between model inference and form score, clinicians performed a masked review to determine which one they preferred.

RESULTS

There was strong concurrent validity between the model predictions and raw self-report scores (r = 0.73, MAE = 3.3). 90 % of AI predictions either agreed with self-report or with clinical expert opinion when AI contradicted self-report. There was no differential model performance across age, sex, race, or ethnicity.

LIMITATIONS

Limitations include access restrictions (English-speaking ability and access to smartphone or computer with broadband internet) and potential self-selection of participants more favorably predisposed toward AI technology.

CONCLUSION

The Aiberry model made accurate predictions of depression severity based on remotely collected spoken responses to a bot-administered interview. This study shows promising results for the use of AI as a mental health screening tool on par with self-report measures.

Beyond rating scales: With targeted evaluation, large language models are poised for psychological assessment

In this narrative review, we survey recent empirical evaluations of AI-based language assessments and present a case for the technology of large language models to be poised for changing standardized psychological assessment. Artificial intelligence has been undergoing a purported "paradigm shift" initiated by new machine learning models, large language models (e.g., BERT, LAMMA, and that behind ChatGPT). These models have led to unprecedented accuracy over most computerized language processing tasks, from web searches to automatic machine translation and question answering, while their dialogue-based forms, like ChatGPT have captured the interest of over a million users. The success of the large language model is mostly attributed to its capability to numerically represent words in their context, long a weakness of previous attempts to automate psychological assessment from language. While potential applications for automated therapy are beginning to be studied on the heels of chatGPT's success, here we present evidence that suggests, with thorough validation of targeted deployment scenarios, that AI's newest technology can move mental health assessment away from rating scales and to instead use how people naturally communicate, in language.

Implicit and explicit COVID-19 associations and mental health in the United States: a large-scale examination and replication

BACKGROUND

Given the sensitive nature of COVID-19 beliefs, evaluating them explicitly and implicitly may provide a fuller picture of how these beliefs vary based on identities and how they relate to mental health.

OBJECTIVE

Three novel brief implicit association tests (BIATs) were created and evaluated: two that measured COVID-19-as-dangerous (vs. safe) and one that measured COVID-19 precautions-as-necessary (vs. unnecessary). Implicit and explicit COVID-19 associations were examined based on individuals' demographic characteristics. Implicit associations were hypothesized to uniquely contribute to individuals' self-reports of mental health.

METHODS

Participants (N = 13,413 US residents; April-November 2020) were volunteers for a COVID-19 study. Participants completed one BIAT and self-report measures. This was a preregistered study with a planned internal replication.

RESULTS

Results revealed older age was weakly associated with stronger implicit and explicit associations of COVID-as-dangerous and precautions-as-necessary. Black and Asian individuals reported greater necessity of taking precautions than White individuals (with small-to-medium effects); greater education was associated with greater explicit reports of COVID-19-as-dangerous and precautions-as-necessary with small effects. Replicated relationships between COVID-as-dangerous explicit associations and mental health had very small effects.

CONCLUSIONS

Implicit associations did not predict mental health but there was evidence that stronger COVID-19-as-dangerous explicit associations are weakly associated with worse mental health.

Natural language processing for mental health interventions: a systematic review and research framework

Neuropsychiatric disorders pose a high societal cost, but their treatment is hindered by lack of objective outcomes and fidelity metrics. AI technologies and specifically Natural Language Processing (NLP) have emerged as tools to study mental health interventions (MHI) at the level of their constituent conversations. However, NLP's potential to address clinical and research challenges remains unclear. We therefore conducted a pre-registered systematic review of NLP-MHI studies using PRISMA guidelines (osf.io/s52jh) to evaluate their models, clinical applications, and to identify biases and gaps. Candidate studies (n = 19,756), including peer-reviewed AI conference manuscripts, were collected up to January 2023 through PubMed, PsycINFO, Scopus, Google Scholar, and ArXiv. A total of 102 articles were included to investigate their computational characteristics (NLP algorithms, audio features, machine learning pipelines, outcome metrics), clinical characteristics (clinical ground truths, study samples, clinical focus), and limitations. Results indicate a rapid growth of NLP MHI studies since 2019, characterized by increased sample sizes and use of large language models. Digital health platforms were the largest providers of MHI data. Ground truth for supervised learning models was based on clinician ratings (n = 31), patient self-report (n = 29) and annotations by raters (n = 26). Text-based features contributed more to model accuracy than audio markers. Patients' clinical presentation (n = 34), response to intervention (n = 11), intervention monitoring (n = 20), providers' characteristics (n = 12), relational dynamics (n = 14), and data preparation (n = 4) were commonly investigated clinical categories. Limitations of reviewed studies included lack of linguistic diversity, limited reproducibility, and population bias. A research framework is developed and validated (NLPxMHI) to assist computational and clinical researchers in addressing the remaining gaps in applying NLP to MHI, with the goal of improving clinical utility, data access, and fairness.

U.S. mothers' appraisals of the race-related events of 2020: Implications for the course of maternal posttraumatic stress disorder symptoms

For parents of color, publicized racial violence can heighten concerns about their children's safety. The goal of this study was to test whether this form of race-related stress exacerbates maternal posttraumatic stress disorder (PTSD) symptoms over a 4-month period in families of color. Participants included 262 U.S. mothers with a lifetime mental health diagnosis (67.6% non-Hispanic White, 15.6% African-American/Black, 16.8% other family of color). Mothers completed online surveys and open-ended questions, including appraisals of the meaning of the 2020 race-related events (i.e., George Floyd's death, subsequent protests) in relation to their children's future. Open-ended responses were quantified using LIWC15 text analysis for emotion word frequency and thematic coding for perceived implications. In ANCOVA, there were significant racial group differences in appraisals, ds = 0.09-0.57. The responses from mothers of Black children included fewer positive and more negative emotion words than mothers of White children; they also included more perceived negative implications than all other mothers but did not vary on perceived positive implications. In regression analyses, there were significant moderating effects of race/ethnicity in the association between appraisals and PTSD symptom course such that negative appraisals predicted a subsequent increase in PTSD symptoms only for mothers of Black children, βs = .26-.37. Variations in event appraisals were unrelated to PTSD symptom course for other mothers. These findings provide longitudinal support for the link between vicarious racism exposure and PTSD symptoms and highlight one potential form of racism-related stress for parents of Black children.

The where and when of COVID-19: Using ecological and Twitter-based assessments to examine impacts in a temporal and community context

In March 2020, residents of the Bronx, New York experienced one of the first significant community COVID-19 outbreaks in the United States. Focusing on intensive longitudinal data from 78 Bronx-based older adults, we used a multi-method approach to (1) examine 2019 to early pandemic (February-June 2020) changes in momentary psychological well-being of Einstein Aging Study (EAS) participants and (2) to contextualize these changes with community distress scores collected from public Twitter posts posted in Bronx County. We found increases in mean loneliness from 2019 to 2020; and participants that were higher in neuroticism had greater increases in thought unpleasantness and feeling depressed. Twitter-based Bronx community scores of anxiety, depressivity, and negatively-valenced affect showed elevated levels in 2020 weeks relative to 2019. Integration of EAS participant data and community data showed week-to-week fluctuations across 2019 and 2020. Results highlight how community-level data can characterize a rapidly changing environment to supplement individual-level data at no additional burden to individual participants.

Detecting Presence of PTSD Using Sentiment Analysis From Text Data

Rates of Post-traumatic stress disorder (PTSD) have risen significantly due to the COVID-19 pandemic. Telehealth has emerged as a means to monitor symptoms for such disorders. This is partly due to isolation or inaccessibility of therapeutic intervention caused from the pandemic. Additional screening tools may be needed to augment identification and diagnosis of PTSD through a virtual medium. Sentiment analysis refers to the use of natural language processing (NLP) to extract emotional content from text information. In our study, we train a machine learning (ML) model on text data, which is part of the Audio/Visual Emotion Challenge and Workshop (AVEC-19) corpus, to identify individuals with PTSD using sentiment analysis from semi-structured interviews. Our sample size included 188 individuals without PTSD, and 87 with PTSD. The interview was conducted by an artificial character (Ellie) over a video-conference call. Our model was able to achieve a balanced accuracy of 80.4% on a held out dataset used from the AVEC-19 challenge. Additionally, we implemented various partitioning techniques to determine if our model was generalizable enough. This shows that learned models can use sentiment analysis of speech to identify the presence of PTSD, even through a virtual medium. This can serve as an important, accessible and inexpensive tool to detect mental health abnormalities during the COVID-19 pandemic.