Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)

Patient reported experiences of community rehabilitation and/or support services for people with long term neurological conditions: a narrative review of quantitative studies

OBJECTIVES

1. To identify validated quantitative Patient Reported Experience Measures (PREM's) being used in Community Rehabilitation and/or Support services for people with long term neurological conditions (PwLTNC). 2. To explore how data from quantitative PREM's adds to research on patient experiences of Community Rehabilitation and Support for PwLTNC.

METHOD

Eight data bases were searched for peer reviewed studies (2005-2021) which met inclusion criteria. Data extraction and quality assessment for sixteen studies was performed by two reviewers. Narrative synthesis was conducted.

RESULTS

Eleven validated PREM's were identified which captured data for 15,831 PwLTNC. PREM scores indicated positive and negative experiences for people with Multiple Sclerosis (n = 13,123), Parkinson's Disease (n = 2215) and Acquired Brain Injury (n = 493). Negative experiences related to Picker Institute Principles: 1 (accessibility); 3 (coordination/continuity); 4 (involvement/support for family and carers); 5 (information provision), 6 (Involvement in decision making) and 7 (empathy and emotional support).

CONCLUSION

Quantitative PREM's provide evidence of process quality and person-centred care within community rehabilitation and support services across large data sets of heterogeneous neurological conditions and geographical locations. Quality improvement initiatives for people with MS, PD and ABI should target processes relating to Picker Institute Principles 1,3,4,5,6, and 7.

The impact of artificial intelligence in the diagnosis and management of acoustic neuroma: A systematic review

BACKGROUND

Schwann cell sheaths are the source of benign, slowly expanding tumours known as acoustic neuromas (AN). The diagnostic and treatment approaches for AN must be patient-centered, taking into account unique factors and preferences.

OBJECTIVE

The purpose of this study is to investigate how machine learning and artificial intelligence (AI) can revolutionise AN management and diagnostic procedures.

METHODS

A thorough systematic review that included peer-reviewed material from public databases was carried out. Publications on AN, AI, and deep learning up until December 2023 were included in the review's purview.

RESULTS

Based on our analysis, AI models for volume estimation, segmentation, tumour type differentiation, and separation from healthy tissues have been developed successfully. Developments in computational biology imply that AI can be used effectively in a variety of fields, including quality of life evaluations, monitoring, robotic-assisted surgery, feature extraction, radiomics, image analysis, clinical decision support systems, and treatment planning.

CONCLUSION

For better AN diagnosis and treatment, a variety of imaging modalities require the development of strong, flexible AI models that can handle heterogeneous imaging data. Subsequent investigations ought to concentrate on reproducing findings in order to standardise AI approaches, which could transform their use in medical environments.

A review of ophthalmology education in the era of generative artificial intelligence

PURPOSE

To explore the integration of generative AI, specifically large language models (LLMs), in ophthalmology education and practice, addressing their applications, benefits, challenges, and future directions.

DESIGN

A literature review and analysis of current AI applications and educational programs in ophthalmology.

METHODS

Analysis of published studies, reviews, articles, websites, and institutional reports on AI use in ophthalmology. Examination of educational programs incorporating AI, including curriculum frameworks, training methodologies, and evaluations of AI performance on medical examinations and clinical case studies.

RESULTS

Generative AI, particularly LLMs, shows potential to improve diagnostic accuracy and patient care in ophthalmology. Applications include aiding in patient, physician, and medical students' education. However, challenges such as AI hallucinations, biases, lack of interpretability, and outdated training data limit clinical deployment. Studies revealed varying levels of accuracy of LLMs on ophthalmology board exam questions, underscoring the need for more reliable AI integration. Several educational programs nationwide provide AI and data science training relevant to clinical medicine and ophthalmology.

CONCLUSIONS

Generative AI and LLMs offer promising advancements in ophthalmology education and practice. Addressing challenges through comprehensive curricula that include fundamental AI principles, ethical guidelines, and updated, unbiased training data is crucial. Future directions include developing clinically relevant evaluation metrics, implementing hybrid models with human oversight, leveraging image-rich data, and benchmarking AI performance against ophthalmologists. Robust policies on data privacy, security, and transparency are essential for fostering a safe and ethical environment for AI applications in ophthalmology.

Deconstructing subjective unmet healthcare needs: a South Korean case study with policy implications

Background

Despite widespread efforts by many countries to reduce the prevalence of unmet healthcare needs within their populations, there remains a scarcity of research systematically exploring the components of these needs.

Objectives

This study aims to deconstruct subjective unmet healthcare needs into two distinct components: the experience of subjective healthcare needs (the "Needs" component) and the experience of unmet needs contingent on those healthcare needs (the "Unmet" component).

Methods

This analysis utilizes data from 13,359 adults aged 19 or older, collected through the 2018 Korea Health Panel survey, with the aim of minimizing the influence of the coronavirus disease 19 pandemic. The two dependent variables are the experience of subjective healthcare needs and whether these needs have been met. The independent variables include 15 socio-demographic, health, and functional characteristics. The study employs both a population proportion analysis and a multivariable bivariate probit model with sample selection.

Results

In South Korea, 11.6% (CI [confidence interval] = 11.0-12.3%) of the population experienced subjective unmet healthcare needs. Upon deconstructing these, 96.7% (CI = 96.2-97.1%) of the population exhibited the Needs component, and 12.0% (CI = 11.4-12.7%) displayed the Unmet component. Each independent variable showed different associations between the two components. Furthermore, effective interventions targeting the characteristics associated with each component could reduce the proportion of the population experiencing subjective unmet healthcare needs from 11.6 to 4.0%.

Conclusion

South Korea faces a significant challenge due to the considerable prevalence of subjective unmet healthcare needs. To address this challenge effectively, the universal healthcare coverage system should adapt its approach based on the characteristics associated with both the Needs and Unmet components of subjective unmet healthcare needs. To achieve this goal, it is highly recommended that the government prioritize strengthening community-based primary healthcare, which currently suffers from insufficient resources.

Insights into the nutritional prevention of macular degeneration based on a comparative topic modeling approach

Topic modeling and text mining are subsets of natural language processing (NLP) with relevance for conducting meta-analysis (MA) and systematic review (SR). For evidence synthesis, the above NLP methods are conventionally used for topic-specific literature searches or extracting values from reports to automate essential phases of SR and MA. Instead, this work proposes a comparative topic modeling approach to analyze reports of contradictory results on the same general research question. Specifically, the objective is to identify topics exhibiting distinct associations with significant results for an outcome of interest by ranking them according to their proportional occurrence in (and consistency of distribution across) reports of significant effects. Macular degeneration (MD) is a disease that affects millions of people annually, causing vision loss. Augmenting evidence synthesis to provide insight into MD prevention is therefore of central interest in this article. The proposed method was tested on broad-scope studies addressing whether supplemental nutritional compounds significantly benefit macular degeneration. Six compounds were identified as having a particular association with reports of significant results for benefiting MD. Four of these were further supported in terms of effectiveness upon conducting a follow-up literature search for validation (omega-3 fatty acids, copper, zeaxanthin, and nitrates). The two not supported by the follow-up literature search (niacin and molybdenum) also had scores in the lowest range under the proposed scoring system. Results therefore suggest that the proposed method's score for a given topic may be a viable proxy for its degree of association with the outcome of interest, and can be helpful in the systematic search for potentially causal relationships. Further, the compounds identified by the proposed method were not simultaneously captured as salient topics by state-of-the-art topic models that leverage document and word embeddings (Top2Vec) and transformer models (BERTopic). These results underpin the proposed method's potential to add specificity in understanding effects from broad-scope reports, elucidate topics of interest for future research, and guide evidence synthesis in a scalable way. All of this is accomplished while yielding valuable and actionable insights into the prevention of MD.

Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives

Machine learning (ML) solutions are increasingly entering healthcare. They are complex, sociotechnical systems that include data inputs, ML models, technical infrastructure and human interactions. They have promise for improving care across a wide range of clinical applications but if poorly implemented, they may disrupt clinical workflows, exacerbate inequities in care and harm patients. Many aspects of ML solutions are similar to other digital technologies, which have well-established approaches to implementation. However, ML applications present distinct implementation challenges, given that their predictions are often complex and difficult to understand, they can be influenced by biases in the data sets used to develop them, and their impacts on human behaviour are poorly understood. This manuscript summarises the current state of knowledge about implementing ML solutions in clinical care and offers practical guidance for implementation. We propose three overarching questions for potential users to consider when deploying ML solutions in clinical care: (1) Is a clinical or operational problem likely to be addressed by an ML solution? (2) How can an ML solution be evaluated to determine its readiness for deployment? (3) How can an ML solution be deployed and maintained optimally? The Quality Improvement community has an essential role to play in ensuring that ML solutions are translated into clinical practice safely, effectively, and ethically.

The added value of the artificial intelligence patient-reported experience measure (AI-PREM tool) in clinical practise: Deployment in a vestibular schwannoma care pathway

Objectives

Patient-reported experience measures (PREMs) can be used for the improvement of quality of care. In this study, the outcome of an open-ended question PREM combined with computer-assisted analysis is compared to the outcome of a closed-ended PREM questionnaire.

Methods

This survey study assessed the outcome of the open-ended questionnaire PREM and a close-ended question PREM of patients with unilateral vestibular schwannoma in a tertiary vestibular schwannoma expert centre.

Results

The open-ended questions PREM, consisting of five questions, was completed by 507 participants and resulted in 1508 positive and 171 negative comments, categorised into 27 clusters. The close-ended questions PREM results were mainly positive (overall experience graded as 8/10), but did not identify specific action points. Patients who gave high overall scores (>8) on the close-ended question provided points for improvement in the open-ended question PREM, which would have been missed using the close-ended questions only.

Conclusions

Compared to the close-ended question PREM, the open-ended question PREM provides more detailed and specific information about the patient experience in the vestibular schwannoma care pathway.

Innovation

Automated analysis of feedback with the open-ended question PREM revealed relevant insights and identified topics for targeted quality improvement, whereas the close-ended PREM did not.

Is the patient speaking or the nurse? Automatic speaker type identification in patient-nurse audio recordings

OBJECTIVES

Patient-clinician communication provides valuable explicit and implicit information that may indicate adverse medical conditions and outcomes. However, practical and analytical approaches for audio-recording and analyzing this data stream remain underexplored. This study aimed to 1) analyze patients' and nurses' speech in audio-recorded verbal communication, and 2) develop machine learning (ML) classifiers to effectively differentiate between patient and nurse language.

MATERIALS AND METHODS

Pilot studies were conducted at VNS Health, the largest not-for-profit home healthcare agency in the United States, to optimize audio-recording patient-nurse interactions. We recorded and transcribed 46 interactions, resulting in 3494 "utterances" that were annotated to identify the speaker. We employed natural language processing techniques to generate linguistic features and built various ML classifiers to distinguish between patient and nurse language at both individual and encounter levels.

RESULTS

A support vector machine classifier trained on selected linguistic features from term frequency-inverse document frequency, Linguistic Inquiry and Word Count, Word2Vec, and Medical Concepts in the Unified Medical Language System achieved the highest performance with an AUC-ROC = 99.01 ± 1.97 and an F1-score = 96.82 ± 4.1. The analysis revealed patients' tendency to use informal language and keywords related to "religion," "home," and "money," while nurses utilized more complex sentences focusing on health-related matters and medical issues and were more likely to ask questions.

CONCLUSION

The methods and analytical approach we developed to differentiate patient and nurse language is an important precursor for downstream tasks that aim to analyze patient speech to identify patients at risk of disease and negative health outcomes.