Towards a temporospatial framework for measurements of disorganization in speech using semantic vectors

Incoherent speech in schizophrenia has long been described as the mind making "leaps" of large distances between thoughts and ideas. Such a view seems intuitive, and for almost two decades, attempts to operationalize these conceptual "leaps" in spoken word meanings have used language-based embedding spaces. An embedding space represents meaning of words as numerical vectors where a greater proximity between word vectors represents more shared meaning. However, there are limitations with word vector-based operationalizations of coherence which can limit their appeal and utility in clinical practice. First, the use of esoteric word embeddings can be conceptually hard to grasp, and this is complicated by several different operationalizations of incoherent speech. This problem can be overcome by a better visualization of methods. Second, temporal information from the act of speaking has been largely neglected since models have been built using written text, yet speech is spoken in real time. This issue can be resolved by leveraging time stamped transcripts of speech. Third, contextual information - namely the situation of where something is spoken - has often only been inferred and never explicitly modeled. Addressing this situational issue opens up new possibilities for models with increased temporal resolution and contextual relevance. In this paper, direct visualizations of semantic distances are used to enable the inspection of examples of incoherent speech. Some common operationalizations of incoherence are illustrated, and suggestions are made for how temporal and spatial contextual information can be integrated in future implementations of measures of incoherence.

Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function

Modern advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this paper we reflect on two decades of experience with the application of language-based assessment to patients' mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. We address the questions by advocating for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to patient clinical states. We further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, we describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders.

Using Automated Speech Processing for Repeated Measurements in a Clinical Setting of the Behavioral Variability in the Stroop Task

The Stroop interference task is indispensable to current neuropsychological practice. Despite this, it is limited in its potential for repeated administration, its sensitivity and its demands on professionals and their clients. We evaluated a digital Stroop deployed using a smart device. Spoken responses were timed using automated speech recognition. Participants included adult nonpatients (N = 113; k = 5 sessions over 5 days) and patients with psychiatric diagnoses (N = 85; k = 3-4 sessions per week over 4 weeks). Traditional interference (difference in response time between color incongruent words vs. color neutral words; M = 0.121 s) and facilitation (neutral vs. color congruent words; M = 0.085 s) effects were robust and temporally stable over testing sessions (ICCs 0.50-0.86). The performance showed little relation to clinical symptoms for a two-week window for either nonpatients or patients but was related to self-reported concentration at the time of testing for both groups. Performance was also related to treatment outcomes in patients. The duration of response word utterances was longer in patients than in nonpatients. Measures of intra-individual variability showed promise for understanding clinical state and treatment outcome but were less temporally stable than measures based solely on average response time latency. This framework of remote assessment using speech processing technology enables the fine-grained longitudinal charting of cognition and verbal behavior. However, at present, there is a problematic lower limit to the absolute size of the effects that can be examined when using voice in such a brief 'out-of-the-laboratory condition' given the temporal resolution of the speech-to-text detection system (in this case, 10 ms). This resolution will limit the parsing of meaningful effect sizes.

Using automated syllable counting to detect missing information in speech transcripts from clinical settings

Speech rate and quantity reflect clinical state; thus automated transcription holds potential clinical applications. We describe two datasets where recording quality and speaker characteristics affected transcription accuracy. Transcripts of low-quality recordings omitted significant portions of speech. An automated syllable counter estimated actual speech output and quantified the amount of missing information. The efficacy of this method differed by audio quality: the correlation between missing syllables and word error rate was only significant when quality was low. Automatically counting syllables could be useful to measure and flag transcription omissions in clinical contexts where speaker characteristics and recording quality are problematic.

Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation

BACKGROUND AND HYPOTHESIS

Despite decades of "proof of concept" findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a "generalized deficit"), and potential biases from demographics and other individual differences.

STUDY DESIGN

This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video "selfies" recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression.

STUDY RESULTS

While test-retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone.

CONCLUSIONS

Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

Predicting self-injurious thoughts in daily life using ambulatory assessment of state cognition

Self-injurious thoughts (SITs) fluctuate considerably from moment to moment. As such, "static" and temporally stable predictors (e.g., demographic variables, prior history) are suboptimal in predicting imminent SITs. This concern is particularly true for "online" cognitive abilities, which are important for understanding SITs, but are typically measured using tests selected for temporal stability. Advances in ambulatory assessments (i.e., real-time assessment in a naturalistic environment) allow for measuring cognition with improved temporal resolution. The present study measured relationships between "state" cognitive performance, measured using an ambulatory-based Trail Making Test, and SITs. Self-reported state hope and social connectedness was also measured. Data were collected using a specially designed mobile application (administered 4x/week up to 28 days) in substance use inpatients (N = 99). Consistent with prior literature, state hope and social connectedness was significantly associated with state SITs. Importantly, poorer state cognitive performance also significantly predicted state SITs, independent of hallmark static and state self-report risk variables. These findings highlight the potential importance of "online" cognition to predict SITs. Ambulatory recording reflects an efficient, sensitive, and ecological valid methodology for evaluating subjective and objectives predictors of imminent SITs.

Extending the usefulness of the verbal memory test: The promise of machine learning

The evaluation of verbal memory is a core component of neuropsychological assessment in a wide range of clinical and research settings. Leveraging story recall to assay neurocognitive function could be made more useful if it were possible to administer frequently (i.e., would allow for the collection of more patient data over time) and automatically assess the recalls with machine learning methods. In the present study, we evaluated a novel story recall test with 24 parallel forms that was deployed using smart devices in 94 psychiatric inpatients and 80 nonpatient adults. Machine learning and vector-based natural language processing methods were employed to automate test scoring, and performance using these methods was evaluated in their incremental validity, criterion validity (i.e., convergence with trained human raters), and parallel forms reliability. Our results suggest moderate to high consistency across the parallel forms, high convergence with human raters (r values ~ 0.89), and high incremental validity for discriminating between groups. While much work remains, the present findings are critical for implementing an automated, neuropsychological test deployable using remote technologies across multiple and frequent administrations.

Validating Biobehavioral Technologies for Use in Clinical Psychiatry

The last decade has witnessed the development of sophisticated biobehavioral and genetic, ambulatory, and other measures that promise unprecedented insight into psychiatric disorders. As yet, clinical sciences have struggled with implementing these objective measures and they have yet to move beyond "proof of concept." In part, this struggle reflects a traditional, and conceptually flawed, application of traditional psychometrics (i.e., reliability and validity) for evaluating them. This paper focuses on "resolution," concerning the degree to which changes in a signal can be detected and quantified, which is central to measurement evaluation in informatics, engineering, computational and biomedical sciences. We define and discuss resolution in terms of traditional reliability and validity evaluation for psychiatric measures, then highlight its importance in a study using acoustic features to predict self-injurious thoughts/behaviors (SITB). This study involved tracking natural language and self-reported symptoms in 124 psychiatric patients: (a) over 5-14 recording sessions, collected using a smart phone application, and (b) during a clinical interview. Importantly, the scope of these measures varied as a function of time (minutes, weeks) and spatial setting (i.e., smart phone vs. interview). Regarding reliability, acoustic features were temporally unstable until we specified the level of temporal/spatial resolution. Regarding validity, accuracy based on machine learning of acoustic features predicting SITB varied as a function of resolution. High accuracy was achieved (i.e., ~87%), but only when the acoustic and SITB measures were "temporally-matched" in resolution was the model generalizable to new data. Unlocking the potential of biobehavioral technologies for clinical psychiatry will require careful consideration of resolution.

Applying speech technologies to assess verbal memory in patients with serious mental illness

Verbal memory deficits are some of the most profound neurocognitive deficits associated with schizophrenia and serious mental illness in general. As yet, their measurement in clinical settings is limited to traditional tests that allow for limited administrations and require substantial resources to deploy and score. Therefore, we developed a digital ambulatory verbal memory test with automated scoring, and repeated self-administration via smart devices. One hundred and four adults participated, comprising 25 patients with serious mental illness and 79 healthy volunteers. The study design was successful with high quality speech recordings produced to 92% of prompts (Patients: 86%, Healthy: 96%). The story recalls were both transcribed and scored by humans, and scores generated using natural language processing on transcriptions were comparable to human ratings (R = 0.83, within the range of human-to-human correlations of R = 0.73-0.89). A fully automated approach that scored transcripts generated by automatic speech recognition produced comparable and accurate scores (R = 0.82), with very high correlation to scores derived from human transcripts (R = 0.99). This study demonstrates the viability of leveraging speech technologies to facilitate the frequent assessment of verbal memory for clinical monitoring purposes in psychiatry.

The importance of loneliness in psychotic-like symptoms: Data from three studies

Poor social connection or loneliness is a prominent feature of schizotypy and may exacerbate psychosis risk. Previous studies have examined the inter-relationships between loneliness and psychosis risk, but critically, they have largely been conducted in non-clinical samples or exclusively used laboratory questionnaires with limited consideration of the heterogeneity within schizotypy (i.e., positive, negative, disorganized factors). The present study examined links between loneliness and psychotic-like symptoms across the dimensions of schizotypy through cross-sectional, laboratory-based questionnaires (Study 1; N = 160), ambulatory assessment (Study 2; N = 118) in undergraduates, and ambulatory assessment in inpatients in a substance abuse treatment program (Study 3; N = 48). Trait positive schizotypy consistently predicted cross-sectional and state psychotic-like symptoms. Loneliness, assessed via cross-sectional and ambulatory means, was largely linked with psychotic-like symptoms. Importantly, psychotic-like symptoms were dynamic: psychotic-like symptoms largely increased with loneliness in individuals with elevated positive and disorganized schizotypal traits, though there were some inconsistency related to disorganized schizotypy and state psychotic-like symptoms. Negative schizotypy and loneliness did not significantly interact to predict psychotic-like symptoms, suggesting specificity to positive schizotypy. Ambulatory approaches provide the opportunity for ecologically valid identification of risk states across psychopathology, thus informing early intervention.

Moving psychological assessment out of the controlled laboratory setting: Practical challenges

Behavioral assessment using smart devices affords novel methods, notably remote self-administration by the individuals themselves. However, this new approach requires navigating complex legal and technical terrain. Given the limited empirical data that currently exists, we provide and discuss anecdotes of the methodological, technical, legal, and cultural issues associated with an implementation in both U.S. and European settings of a mobile software application for regular psychological monitoring purposes. The tasks required participants to listen, watch, speak, and touch to interact with the smart device, thus assessing cognition, motor skill, and language. Four major findings merit mention: First, moving assessment out of the hands of a trained investigator necessitates excellent usability engineering, such that the tool is easily usable by the participant and the resulting data relevant to the investigator. Second, remote assessment requires that the data are transferred safely back to the investigator, and that risk of compromising participant confidentiality is minimized. Third, frequent data collection over long periods of time is associated with a possibility that participants may choose to withdraw consent for participation thus requiring data retraction. Fourth, data collection and analysis across international borders creates new challenges and new opportunities because of important cultural and language issues that may inform the underlying behavioral constructs of interest. In conclusion, the new technological frameworks provide unprecedented opportunities for remote self-administered behavioral assessments but will be most productive in multidisciplinary teams to ensure the highest level of user satisfaction and data quality, and to guarantee the highest level of data protection. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Comparing static and dynamic predictors of risk for hostility in serious mental illness: Preliminary findings

This study compared static predictors of hostility (e.g. demographics, clinician ratings) to subjective (i.e., self-reported affect on slider scales in response to written questions) and objective (i.e., vocal indicators of arousal from speech samples in a story-retelling task) dynamic predictors using ambulatory assessment over five days in a sample of 25 stable outpatients with diagnoses of a serious mental illness. Multilevel modeling showed that both subjective and objective dynamic predictors were significant, but none of the static predictors were. These results suggest that, in predicting hostility, it is more important to account for state variation than static traits.