1
|
Jacennik B, Zawadzka-Gosk E, Moreira JP, Glinkowski WM. Evaluating Patients' Experiences with Healthcare Services: Extracting Domain and Language-Specific Information from Free-Text Narratives. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:10182. [PMID: 36011816 PMCID: PMC9408527 DOI: 10.3390/ijerph191610182] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 08/12/2022] [Accepted: 08/15/2022] [Indexed: 06/15/2023]
Abstract
Evaluating patients’ experience and satisfaction often calls for analyses of free-text data. Language and domain-specific information extraction can reduce costly manual preprocessing and enable the analysis of extensive collections of experience-based narratives. The research aims were to (1) elicit free-text narratives about experiences with health services of international students in Poland, (2) develop domain- and language-specific algorithms for the extraction of information relevant for the evaluation of quality and safety of health services, and (3) test the performance of information extraction algorithms’ on questions about the patients’ experiences with health services. The materials were free-text narratives about health clinic encounters produced by English-speaking foreigners recalling their experiences (n = 104) in healthcare facilities in Poland. A linguistic analysis of the text collection led to constructing a semantic−syntactic lexicon and a set of lexical-syntactic frames. These were further used to develop rule-based information extraction algorithms in the form of Python scripts. The extraction algorithms generated text classifications according to predefined queries. In addition, the narratives were classified by human readers. The algorithm-based and the human readers’ classifications were highly correlated and significant (p < 0.01), indicating an excellent performance of the automatic query algorithms. The study results demonstrate that domain-specific and language-specific information extraction from free-text narratives can be used as an efficient and low-cost method for evaluating patient experiences and satisfaction with health services and built into software solutions for the quality evaluation in health care.
Collapse
Affiliation(s)
| | - Emilia Zawadzka-Gosk
- Multimedia Department, Polish-Japanese Academy of Information Technology, 02-008 Warsaw, Poland
| | - Joaquim Paulo Moreira
- International Healthcare Management Research and Development Center (IHM-RDC), Shandong Provincial Qianfoshan Hospital, Jinan 250014, China
- Gestao em Saude, Atlantica Instituto Universitario, 2730-036 Oeiras, Portugal
| | - Wojciech Michał Glinkowski
- Polish Telemedicine and eHealth Society, 03-728 Warsaw, Poland
- Center of Excellence “TeleOrto” for Telediagnostics and Treatment of Disorders and Injuries of the Locomotor System, Department of Medical Informatics and Telemedicine, Medical University of Warsaw, 00-581 Warsaw, Poland
| |
Collapse
|
2
|
van Buchem MM, Neve OM, Kant IMJ, Steyerberg EW, Boosman H, Hensen EF. Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM). BMC Med Inform Decis Mak 2022; 22:183. [PMID: 35840972 PMCID: PMC9284859 DOI: 10.1186/s12911-022-01923-5] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2022] [Accepted: 06/23/2022] [Indexed: 11/12/2022] Open
Abstract
Background Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness.
Methods We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context.
Results The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic. Conclusions The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01923-5.
Collapse
Affiliation(s)
- Marieke M van Buchem
- Information Technology & Digital Innovation Department, Leiden University Medical Center, Leiden, the Netherlands. .,Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands. .,Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands.
| | - Olaf M Neve
- Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands
| | - Ilse M J Kant
- Information Technology & Digital Innovation Department, Leiden University Medical Center, Leiden, the Netherlands.,Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands.,Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands
| | - Ewout W Steyerberg
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, the Netherlands.,Clinical Artificial Intelligence Implementation and Research Lab (CAIRELab), Leiden University Medical Center, Leiden, the Netherlands
| | | | - Erik F Hensen
- Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center, Leiden, the Netherlands
| |
Collapse
|
3
|
The Case of Aspect in Sentiment Analysis: Seeking Attention or Co-Dependency? MACHINE LEARNING AND KNOWLEDGE EXTRACTION 2022. [DOI: 10.3390/make4020021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
(1) Background: Aspect-based sentiment analysis (SA) is a natural language processing task, the aim of which is to classify the sentiment associated with a specific aspect of a written text. The performance of SA methods applied to texts related to health and well-being lags behind that of other domains. (2) Methods: In this study, we present an approach to aspect-based SA of drug reviews. Specifically, we analysed signs and symptoms, which were extracted automatically using the Unified Medical Language System. This information was then passed onto the BERT language model, which was extended by two layers to fine-tune the model for aspect-based SA. The interpretability of the model was analysed using an axiomatic attribution method. We performed a correlation analysis between the attribution scores and syntactic dependencies. (3) Results: Our fine-tuned model achieved accuracy of approximately 95% on a well-balanced test set. It outperformed our previous approach, which used syntactic information to guide the operation of a neural network and achieved an accuracy of approximately 82%. (4) Conclusions: We demonstrated that a BERT-based model of SA overcomes the negative bias associated with health-related aspects and closes the performance gap against the state-of-the-art in other domains.
Collapse
|
4
|
Manchaiah V, Londero A, Deshpande AK, Revel M, Palacios G, Boyd RL, Ratinaud P. Online Discussions About Tinnitus: What Can We Learn From Natural Language Processing of Reddit Posts? Am J Audiol 2022; 31:993-1002. [PMID: 35130042 DOI: 10.1044/2021_aja-21-00158] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022] Open
Abstract
BACKGROUND This study was aimed at identifying key topics in online discussions about tinnitus by examining a large data set extracted from Reddit social media using a natural language processing technique. METHOD A corpus of 113,215 posts about tinnitus was extracted from Reddit's application programming interface. After cleaning the data for duplications and posts without any text information, the sample was reduced to 101,905 posts, which was subjected to cluster analysis using the open-source IRaMuTeQ software to identify main topics based on the co-occurrence of texts. These clusters were named by a panel of tinnitus experts (n = 9) by reading typical text segments within each cluster. RESULTS The cluster analysis identified 16 unique clusters that belong to two topics, which were named "tinnitus causes and consequences" and "tinnitus management and coping." Based on their characteristics, the clusters were named: tinnitus timeline (10%), tinnitus perception (9.7%), medical triggers and modulators (8.8%), hearing research (8.8%), attention and silence (8.6%), social media posts about tinnitus (7.4%), hearing protection (7.3%), interaction with hearing health care providers (6.7%), mental health and coping (5.8%), music listening (5.7%), hope for a cure (5.6%), interactions with people without tinnitus (5.4%), dietary supplements and alternative therapies (3.2%), sleep (3.9%), dietary effects (1.7%), and writing about tinnitus and being thankful to online community (1.4%). CONCLUSIONS Despite some limitations, tinnitus posts on Reddit provide rich real-world data to identify various issues and complaints that tinnitus patients and their significant others discuss in online communities. Some of the clusters identified here are novel (e.g., tinnitus timeline, interactions with people without tinnitus) and have not been much discussed in the tinnitus literature. The results suggest that individuals with tinnitus relay on social media for support and highlight the service delivery needs in providing social support through other means (e.g., support groups).
Collapse
Affiliation(s)
- Vinaya Manchaiah
- Department of Otolaryngology–Head and Neck Surgery, University of Colorado School of Medicine, Aurora
- UCHealth Hearing and Balance, University of Colorado Hospital, Aurora
- Virtual Hearing Lab, Collaborative Initiative between University of Colorado School of Medicine and University of Pretoria, Aurora, CO
- Department of Speech-Language Pathology and Audiology, University of Pretoria, Gauteng, South Africa
- Department of Speech and Hearing, Manipal College of Health Professions, Manipal Academy of Higher Education, Manipal, India
| | - Alain Londero
- Hôpital Européen Georges Pompidou, Assistance Publique – Hôpitaux de Paris; Faculté de Médecine Paris Descartes – Université de Paris, France
| | - Aniruddha K. Deshpande
- Department of Speech-Language-Hearing Sciences, Hofstra University, Long Island, NY
- Long Island Doctor of Audiology Consortium, Garden City, NY
| | - Manon Revel
- Institute for Data, Systems and Society, Massachusetts Institute of Technology, Cambridge
| | | | - Ryan L. Boyd
- Department of Psychology, Lancaster University, United Kingdom
- Security Lancaster, Lancaster University, United Kingdom
- Data Science Institute, Lancaster University, United Kingdom
| | - Pierre Ratinaud
- Laboratory of Applied Studies and Research in Social Sciences, University of Toulouse, France
| |
Collapse
|
5
|
Spasic I, Button K. Patient Triage by Topic Modeling of Referral Letters: Feasibility Study. JMIR Med Inform 2020; 8:e21252. [PMID: 33155985 PMCID: PMC7679210 DOI: 10.2196/21252] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Revised: 09/17/2020] [Accepted: 10/05/2020] [Indexed: 01/22/2023] Open
Abstract
Background Musculoskeletal conditions are managed within primary care, but patients can be referred to secondary care if a specialist opinion is required. The ever-increasing demand for health care resources emphasizes the need to streamline care pathways with the ultimate aim of ensuring that patients receive timely and optimal care. Information contained in referral letters underpins the referral decision-making process but is yet to be explored systematically for the purposes of treatment prioritization for musculoskeletal conditions. Objective This study aims to explore the feasibility of using natural language processing and machine learning to automate the triage of patients with musculoskeletal conditions by analyzing information from referral letters. Specifically, we aim to determine whether referral letters can be automatically assorted into latent topics that are clinically relevant, that is, considered relevant when prescribing treatments. Here, clinical relevance is assessed by posing 2 research questions. Can latent topics be used to automatically predict treatment? Can clinicians interpret latent topics as cohorts of patients who share common characteristics or experiences such as medical history, demographics, and possible treatments? Methods We used latent Dirichlet allocation to model each referral letter as a finite mixture over an underlying set of topics and model each topic as an infinite mixture over an underlying set of topic probabilities. The topic model was evaluated in the context of automating patient triage. Given a set of treatment outcomes, a binary classifier was trained for each outcome using previously extracted topics as the input features of the machine learning algorithm. In addition, a qualitative evaluation was performed to assess the human interpretability of topics. Results The prediction accuracy of binary classifiers outperformed the stratified random classifier by a large margin, indicating that topic modeling could be used to predict the treatment, thus effectively supporting patient triage. The qualitative evaluation confirmed the high clinical interpretability of the topic model. Conclusions The results established the feasibility of using natural language processing and machine learning to automate triage of patients with knee or hip pain by analyzing information from their referral letters.
Collapse
Affiliation(s)
- Irena Spasic
- School of Computer Science & Informatics, Cardiff University, Cardiff, United Kingdom
| | - Kate Button
- School of Healthcare Sciences, Cardiff University, Cardiff, United Kingdom
| |
Collapse
|
6
|
Button K, Spasić I, Playle R, Owen D, Lau M, Hannaway L, Jones S. Using routine referral data for patients with knee and hip pain to improve access to specialist care. BMC Musculoskelet Disord 2020; 21:66. [PMID: 32013997 PMCID: PMC6998102 DOI: 10.1186/s12891-020-3087-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/13/2019] [Accepted: 01/22/2020] [Indexed: 11/29/2022] Open
Abstract
Background Referral letters from primary care contain a large amount of information that could be used to improve the appropriateness of the referral pathway for individuals seeking specialist opinion for knee or hip pain. The primary aim of this study was to evaluate the content of the referral letters to identify information that can independently predict an optimal care pathway. Methods Using a prospective longitudinal design, a convenience sample of patients with hip or knee pain were recruited from orthopaedic, specialist general practice and advanced physiotherapy practitioner clinics. Individuals completed a Knee or hip Osteoarthritis Outcome Score at initial consultation and after 6 months. Participant demographics, body mass index, medication and co-morbidity data were extracted from the referral letters. Free text of the referral letters was mapped automatically onto the Unified Medical Language System to identify relevant clinical variables. Treatment outcomes were extracted from the consultation letters. Each outcome was classified as being an optimal or sub-optimal pathway, where an optimal pathway was defined as the one that results in the right treatment at the right time. Logistic regression was used to identify variables that were independently associated with an optimal pathway. Results A total of 643 participants were recruited, 419 (66.7%) were classified as having an optimal pathway. Variables independently associated with having an optimal care pathway were lower body mass index (OR 1.0, 95% CI 0.9 to 1.0 p = 0.004), named disease or syndromes (OR 1.8, 95% CI 1.1 to 2.8, p = 0.02) and taking pharmacologic substances (OR 1.8, 95% CI 1.0 to 3.3, p = 0.02). Having a single diagnostic procedure was associated with a suboptimal pathway (OR 0.5, 95% CI 0.3 to 0.9 p < 0.001). Neither Knee nor Hip Osteoarthritis Outcome scores were associated with an optimal pathway. Body mass index was found to be a good predictor of patient rated function (coefficient − 0.8, 95% CI -1.1, − 0.4 p < 0.001). Conclusion Over 30% of patients followed sub-optimal care pathway, which represents potential inefficiency and wasted healthcare resource. A core data set including body mass index should be considered as this was a predictor of optimal care and patient rated pain and function.
Collapse
Affiliation(s)
- Kate Button
- School of Healthcare Sciences, Cardiff University, Eastgate House, Newport Road, Cardiff, CF24 0AB, UK. .,Physiotherapy Department, Cardiff and Vale University Health Board, Cardiff, UK.
| | - Irena Spasić
- School of Computer Science & Informatics, Cardiff University, Cardiff, UK
| | - Rebecca Playle
- Centre for Trials Research, Cardiff University, Cardiff, UK
| | - David Owen
- School of Computer Science & Informatics, Cardiff University, Cardiff, UK
| | - Mandy Lau
- Centre for Trials Research, Cardiff University, Cardiff, UK
| | | | - Stephen Jones
- Trauma and Orthopaedics, Cardiff and Vale Orthopaedic Centre, University Hospital Llandough, Cardiff and Vale UHB, Cardiff, UK
| |
Collapse
|