KLOSURE: Closing in on open-ended patient questionnaires with text mining

Evaluating Patients' Experiences with Healthcare Services: Extracting Domain and Language-Specific Information from Free-Text Narratives

Evaluating patients’ experience and satisfaction often calls for analyses of free-text data. Language and domain-specific information extraction can reduce costly manual preprocessing and enable the analysis of extensive collections of experience-based narratives. The research aims were to (1) elicit free-text narratives about experiences with health services of international students in Poland, (2) develop domain- and language-specific algorithms for the extraction of information relevant for the evaluation of quality and safety of health services, and (3) test the performance of information extraction algorithms’ on questions about the patients’ experiences with health services. The materials were free-text narratives about health clinic encounters produced by English-speaking foreigners recalling their experiences (n = 104) in healthcare facilities in Poland. A linguistic analysis of the text collection led to constructing a semantic−syntactic lexicon and a set of lexical-syntactic frames. These were further used to develop rule-based information extraction algorithms in the form of Python scripts. The extraction algorithms generated text classifications according to predefined queries. In addition, the narratives were classified by human readers. The algorithm-based and the human readers’ classifications were highly correlated and significant (p < 0.01), indicating an excellent performance of the automatic query algorithms. The study results demonstrate that domain-specific and language-specific information extraction from free-text narratives can be used as an efficient and low-cost method for evaluating patient experiences and satisfaction with health services and built into software solutions for the quality evaluation in health care.

Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)

Background

Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness.

Methods

We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context.

Results

The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic.

Conclusions

The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01923-5.

The Case of Aspect in Sentiment Analysis: Seeking Attention or Co-Dependency?

(1) Background: Aspect-based sentiment analysis (SA) is a natural language processing task, the aim of which is to classify the sentiment associated with a specific aspect of a written text. The performance of SA methods applied to texts related to health and well-being lags behind that of other domains. (2) Methods: In this study, we present an approach to aspect-based SA of drug reviews. Specifically, we analysed signs and symptoms, which were extracted automatically using the Unified Medical Language System. This information was then passed onto the BERT language model, which was extended by two layers to fine-tune the model for aspect-based SA. The interpretability of the model was analysed using an axiomatic attribution method. We performed a correlation analysis between the attribution scores and syntactic dependencies. (3) Results: Our fine-tuned model achieved accuracy of approximately 95% on a well-balanced test set. It outperformed our previous approach, which used syntactic information to guide the operation of a neural network and achieved an accuracy of approximately 82%. (4) Conclusions: We demonstrated that a BERT-based model of SA overcomes the negative bias associated with health-related aspects and closes the performance gap against the state-of-the-art in other domains.

Online Discussions About Tinnitus: What Can We Learn From Natural Language Processing of Reddit Posts?

BACKGROUND

This study was aimed at identifying key topics in online discussions about tinnitus by examining a large data set extracted from Reddit social media using a natural language processing technique.

METHOD

A corpus of 113,215 posts about tinnitus was extracted from Reddit's application programming interface. After cleaning the data for duplications and posts without any text information, the sample was reduced to 101,905 posts, which was subjected to cluster analysis using the open-source IRaMuTeQ software to identify main topics based on the co-occurrence of texts. These clusters were named by a panel of tinnitus experts (n = 9) by reading typical text segments within each cluster.

RESULTS

The cluster analysis identified 16 unique clusters that belong to two topics, which were named "tinnitus causes and consequences" and "tinnitus management and coping." Based on their characteristics, the clusters were named: tinnitus timeline (10%), tinnitus perception (9.7%), medical triggers and modulators (8.8%), hearing research (8.8%), attention and silence (8.6%), social media posts about tinnitus (7.4%), hearing protection (7.3%), interaction with hearing health care providers (6.7%), mental health and coping (5.8%), music listening (5.7%), hope for a cure (5.6%), interactions with people without tinnitus (5.4%), dietary supplements and alternative therapies (3.2%), sleep (3.9%), dietary effects (1.7%), and writing about tinnitus and being thankful to online community (1.4%).

CONCLUSIONS

Despite some limitations, tinnitus posts on Reddit provide rich real-world data to identify various issues and complaints that tinnitus patients and their significant others discuss in online communities. Some of the clusters identified here are novel (e.g., tinnitus timeline, interactions with people without tinnitus) and have not been much discussed in the tinnitus literature. The results suggest that individuals with tinnitus relay on social media for support and highlight the service delivery needs in providing social support through other means (e.g., support groups).

Patient Triage by Topic Modeling of Referral Letters: Feasibility Study

Background

Musculoskeletal conditions are managed within primary care, but patients can be referred to secondary care if a specialist opinion is required. The ever-increasing demand for health care resources emphasizes the need to streamline care pathways with the ultimate aim of ensuring that patients receive timely and optimal care. Information contained in referral letters underpins the referral decision-making process but is yet to be explored systematically for the purposes of treatment prioritization for musculoskeletal conditions.

Objective

This study aims to explore the feasibility of using natural language processing and machine learning to automate the triage of patients with musculoskeletal conditions by analyzing information from referral letters. Specifically, we aim to determine whether referral letters can be automatically assorted into latent topics that are clinically relevant, that is, considered relevant when prescribing treatments. Here, clinical relevance is assessed by posing 2 research questions. Can latent topics be used to automatically predict treatment? Can clinicians interpret latent topics as cohorts of patients who share common characteristics or experiences such as medical history, demographics, and possible treatments?

Methods

We used latent Dirichlet allocation to model each referral letter as a finite mixture over an underlying set of topics and model each topic as an infinite mixture over an underlying set of topic probabilities. The topic model was evaluated in the context of automating patient triage. Given a set of treatment outcomes, a binary classifier was trained for each outcome using previously extracted topics as the input features of the machine learning algorithm. In addition, a qualitative evaluation was performed to assess the human interpretability of topics.

Results

The prediction accuracy of binary classifiers outperformed the stratified random classifier by a large margin, indicating that topic modeling could be used to predict the treatment, thus effectively supporting patient triage. The qualitative evaluation confirmed the high clinical interpretability of the topic model.

Conclusions

The results established the feasibility of using natural language processing and machine learning to automate triage of patients with knee or hip pain by analyzing information from their referral letters.

Using routine referral data for patients with knee and hip pain to improve access to specialist care

Background

Referral letters from primary care contain a large amount of information that could be used to improve the appropriateness of the referral pathway for individuals seeking specialist opinion for knee or hip pain. The primary aim of this study was to evaluate the content of the referral letters to identify information that can independently predict an optimal care pathway.

Methods

Using a prospective longitudinal design, a convenience sample of patients with hip or knee pain were recruited from orthopaedic, specialist general practice and advanced physiotherapy practitioner clinics. Individuals completed a Knee or hip Osteoarthritis Outcome Score at initial consultation and after 6 months. Participant demographics, body mass index, medication and co-morbidity data were extracted from the referral letters. Free text of the referral letters was mapped automatically onto the Unified Medical Language System to identify relevant clinical variables. Treatment outcomes were extracted from the consultation letters. Each outcome was classified as being an optimal or sub-optimal pathway, where an optimal pathway was defined as the one that results in the right treatment at the right time. Logistic regression was used to identify variables that were independently associated with an optimal pathway.

Results

A total of 643 participants were recruited, 419 (66.7%) were classified as having an optimal pathway. Variables independently associated with having an optimal care pathway were lower body mass index (OR 1.0, 95% CI 0.9 to 1.0 p = 0.004), named disease or syndromes (OR 1.8, 95% CI 1.1 to 2.8, p = 0.02) and taking pharmacologic substances (OR 1.8, 95% CI 1.0 to 3.3, p = 0.02). Having a single diagnostic procedure was associated with a suboptimal pathway (OR 0.5, 95% CI 0.3 to 0.9 p < 0.001). Neither Knee nor Hip Osteoarthritis Outcome scores were associated with an optimal pathway. Body mass index was found to be a good predictor of patient rated function (coefficient − 0.8, 95% CI -1.1, − 0.4 p < 0.001).

Conclusion

Over 30% of patients followed sub-optimal care pathway, which represents potential inefficiency and wasted healthcare resource. A core data set including body mass index should be considered as this was a predictor of optimal care and patient rated pain and function.