Improving the Applicability of AI for Psychiatric Applications through Human-in-the-loop Methodologies

Unveiling readability challenges: An extensive analysis of consent document accessibility in clinical trials

Background

Clinical research trials rely on informed consent forms (ICFs) to explain all aspects of the study to potential participants. Despite efforts to ensure the readability of ICFs, concerns about their complexity and participant understanding persist. There is a noted gap between Institutional Review Board (IRB) standards and the actual readability levels of ICFs, which often exceed the recommended 8th-grade reading level. This study evaluates the readability of over five thousand ICFs from ClinicalTrials.gov in the USA to assess their literacy levels.

Methods

We analyzed 5,239 US-based ICFs from ClinicalTrials.gov using readability metrics such as the Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, and the percentage of difficult words. We examined trends in readability levels across studies initiated from 2005 to 2024.

Results

Most ICFs exceeded the recommended 8th-grade reading level, with an average Flesch-Kincaid Grade Level of 10.99. While 91% of the ICFs were written above the 8th-grade level, there was an observable improvement in readability, with fewer studies exceeding a 10th-grade reading level in recent years.

Conclusions

The study reveals a discrepancy between the recommended readability levels and actual ICFs, highlighting a need for simplification. Despite a trend toward improvement in more recent years, ongoing efforts are necessary to ensure ICFs are comprehensible to participants of varied educational backgrounds, reinforcing the ethical integrity of the consent process.

AI and professional liability assessment in healthcare. A revolution in legal medicine?

The adoption of advanced artificial intelligence (AI) systems in healthcare is transforming the healthcare-delivery landscape. Artificial intelligence may enhance patient safety and improve healthcare outcomes, but it presents notable ethical and legal dilemmas. Moreover, as AI streamlines the analysis of the multitude of factors relevant to malpractice claims, including informed consent, adherence to standards of care, and causation, the evaluation of professional liability might also benefit from its use. Beginning with an analysis of the basic steps in assessing professional liability, this article examines the potential new medical-legal issues that an expert witness may encounter when analyzing malpractice cases and the potential integration of AI in this context. These changes related to the use of integrated AI, will necessitate efforts on the part of judges, experts, and clinicians, and may require new legislative regulations. A new expert witness will be likely necessary in the evaluation of professional liability cases. On the one hand, artificial intelligence will support the expert witness; however, on the other hand, it will introduce specific elements into the activities of healthcare workers. These elements will necessitate an expert witness with a specialized cultural background. Examining the steps of professional liability assessment indicates that the likely path for AI in legal medicine involves its role as a collaborative and integrated tool. The combination of AI with human judgment in these assessments can enhance comprehensiveness and fairness. However, it is imperative to adopt a cautious and balanced approach to prevent complete automation in this field.

Reflections on the nature of measurement in language-based automated assessments of patients' mental state and cognitive function

Modern advances in computational language processing methods have enabled new approaches to the measurement of mental processes. However, the field has primarily focused on model accuracy in predicting performance on a task or a diagnostic category. Instead the field should be more focused on determining which computational analyses align best with the targeted neurocognitive/psychological functions that we want to assess. In this paper we reflect on two decades of experience with the application of language-based assessment to patients' mental state and cognitive function by addressing the questions of what we are measuring, how it should be measured and why we are measuring the phenomena. We address the questions by advocating for a principled framework for aligning computational models to the constructs being assessed and the tasks being used, as well as defining how those constructs relate to patient clinical states. We further examine the assumptions that go into the computational models and the effects that model design decisions may have on the accuracy, bias and generalizability of models for assessing clinical states. Finally, we describe how this principled approach can further the goal of transitioning language-based computational assessments to part of clinical practice while gaining the trust of critical stakeholders.

Natural Language Processing Markers for Psychosis and Other Psychiatric Disorders: Emerging Themes and Research Agenda From a Cross-Linguistic Workshop

This workshop summary on natural language processing (NLP) markers for psychosis and other psychiatric disorders presents some of the clinical and research issues that NLP markers might address and some of the activities needed to move in that direction. We propose that the optimal development of NLP markers would occur in the context of research efforts to map out the underlying mechanisms of psychosis and other disorders. In this workshop, we identified some of the challenges to be addressed in developing and implementing NLP markers-based Clinical Decision Support Systems (CDSSs) in psychiatric practice, especially with respect to psychosis. Of note, a CDSS is meant to enhance decision-making by clinicians by providing additional relevant information primarily through software (although CDSSs are not without risks). In psychiatry, a field that relies on subjective clinical ratings that condense rich temporal behavioral information, the inclusion of computational quantitative NLP markers can plausibly lead to operationalized decision models in place of idiosyncratic ones, although ethical issues must always be paramount.

Language abnormalities in schizophrenia: binding core symptoms through contemporary empirical evidence

Both the ability to speak and to infer complex linguistic messages from sounds have been claimed as uniquely human phenomena. In schizophrenia, formal thought disorder (FTD) and auditory verbal hallucinations (AVHs) are manifestations respectively relating to concrete disruptions of those abilities. From an evolutionary perspective, Crow (1997) proposed that "schizophrenia is the price that Homo sapiens pays for the faculty of language". Epidemiological and experimental evidence points to an overlap between FTD and AVHs, yet a thorough investigation examining their shared neural mechanism in schizophrenia is lacking. In this review, we synthesize observations from three key domains. First, neuroanatomical evidence indicates substantial shared abnormalities in language-processing regions between FTD and AVHs, even in the early phases of schizophrenia. Second, neurochemical studies point to a glutamate-related dysfunction in these language-processing brain regions, contributing to verbal production deficits. Third, genetic findings further show how genes that overlap between schizophrenia and language disorders influence neurodevelopment and neurotransmission. We argue that these observations converge into the possibility that a glutamatergic dysfunction in language-processing brain regions might be a shared neural basis of both FTD and AVHs. Investigations of language pathology in schizophrenia could facilitate the development of diagnostic tools and treatments, so we call for multilevel confirmatory analyses focused on modulations of the language network as a therapeutic goal in schizophrenia.

Pragmatic AI-augmentation in mental healthcare: Key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians

The integration of artificial intelligence (AI) technologies into mental health holds the promise of increasing patient access, engagement, and quality of care, and of improving clinician quality of work life. However, to date, studies of AI technologies in mental health have focused primarily on challenges that policymakers, clinical leaders, and data and computer scientists face, rather than on challenges that frontline mental health clinicians are likely to face as they attempt to integrate AI-based technologies into their everyday clinical practice. In this Perspective, we describe a framework for "pragmatic AI-augmentation" that addresses these issues by describing three categories of emerging AI-based mental health technologies which frontline clinicians can leverage in their clinical practice-automation, engagement, and clinical decision support technologies. We elaborate the potential benefits offered by these technologies, the likely day-to-day challenges they may raise for mental health clinicians, and some solutions that clinical leaders and technology developers can use to address these challenges, based on emerging experience with the integration of AI technologies into clinician daily practice in other healthcare disciplines.