Patient Triage by Topic Modeling of Referral Letters: Feasibility Study

Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review

INTRODUCTION

Patient-operated digital triage systems with AI components are becoming increasingly common. However, previous reviews have found a limited amount of research on such systems' accuracy. This systematic review of the literature aimed to identify the main challenges in determining the accuracy of patient-operated digital AI-based triage systems.

METHODS

A systematic review was designed and conducted in accordance with PRISMA guidelines in October 2021 using PubMed, Scopus and Web of Science. Articles were included if they assessed the accuracy of a patient-operated digital triage system that had an AI-component and could triage a general primary care population. Limitations and other pertinent data were extracted, synthesized and analysed. Risk of bias was not analysed as this review studied the included articles' limitations (rather than results). Results were synthesized qualitatively using a thematic analysis.

RESULTS

The search generated 76 articles and following exclusion 8 articles (6 primary articles and 2 reviews) were included in the analysis. Articles' limitations were synthesized into three groups: epistemological, ontological and methodological limitations. Limitations varied with regards to intractability and the level to which they can be addressed through methodological choices. Certain methodological limitations related to testing triage systems using vignettes can be addressed through methodological adjustments, whereas epistemological and ontological limitations require that readers of such studies appraise the studies with limitations in mind.

DISCUSSION

The reviewed literature highlights recurring limitations and challenges in studying the accuracy of patient-operated digital triage systems with AI components. Some of these challenges can be addressed through methodology whereas others are intrinsic to the area of inquiry and involve unavoidable trade-offs. Future studies should take these limitations in consideration in order to better address the current knowledge gaps in the literature.

A Bibliometric Analysis and Visualization of Decision Support Systems for Healthcare Referral Strategies

BACKGROUND

The referral process is an important research focus because of the potential consequences of delays, especially for patients with serious medical conditions that need immediate care, such as those with metastatic cancer. Thus, a systematic literature review of recent and influential manuscripts is critical to understanding the current methods and future directions in order to improve the referral process.

METHODS

A hybrid bibliometric-structured review was conducted using both quantitative and qualitative methodologies. Searches were conducted of three databases, Web of Science, Scopus, and PubMed, in addition to the references from the eligible papers. The papers were considered to be eligible if they were relevant English articles or reviews that were published from January 2010 to June 2021. The searches were conducted using three groups of keywords, and bibliometric analysis was performed, followed by content analysis.

RESULTS

A total of 163 papers that were published in impactful journals between January 2010 and June 2021 were selected. These papers were then reviewed, analyzed, and categorized as follows: descriptive analysis (n = 77), cause and effect (n = 12), interventions (n = 50), and quality management (n = 24). Six future research directions were identified.

CONCLUSIONS

Minimal attention was given to the study of the primary referral of blood cancer cases versus those with solid cancer types, which is a gap that future studies should address. More research is needed in order to optimize the referral process, specifically for suspected hematological cancer patients.

Impact of digital interdisciplinary consultation on secondary care referrals by general practitioners: a protocol for a stepped-wedge cluster randomised controlled trial

INTRODUCTION

Optimal collaboration between general practice and hospital care is crucial to maintain affordable and sustainable access to healthcare for the entire population. General practitioners (GPs) are the gatekeepers to specialist care and patients will visit hospitals mostly only after referral. However, a substantial part of these referrals may be inappropriate, as communication between GPs and medical specialists can be challenging and referring patients may be the most obvious action for a GP to perform.A new digital platform (Prisma) connects GPs and specialists in interdisciplinary groups and facilitates asynchronous, accessible and fast teleconsultation within the group. No previous research has been done to evaluate the impact of this new platform on the referral rates to the hospital.

METHODS AND ANALYSIS

A stepped-wedge randomised controlled trial (RCT) will be performed in Zwolle region in the Netherlands to analyse the effect of introduction of the platform on rate of inappropriate referrals to orthopaedic surgery. In four steps, GPs in the region will be given access to the platform. GPs will be part of the control condition until randomisation to the intervention. According to our sample size calculation, we need to include 18 practices with 1008 patients presenting with hip and knee symptoms. Routine care data of hospital registrations will be analysed to calculate the rate of inappropriate referrals (primary outcome). Secondary outcome are costs, primary and secondary care workload, posted cases and user satisfaction. Alongside this quantitative analysis, we will evaluate patient experience, facilitators and barriers for use of the platform.

ETHICS AND DISSEMINATION

The medical ethics review board of University Medical Center Groningen (UMCG), the Netherlands (METc-number: 2021/288) has confirmed that the Medical Research Involving Human Subjects Act (WMO) does not apply to the process evaluation because the study does not involve randomisation of patients or different medical treatments (letter number: M21.275351).

TRIAL REGISTRATION NUMBER

NL9704.

Analyzing patient experiences using natural language processing: development and validation of the artificial intelligence patient reported experience measure (AI-PREM)

Background

Evaluating patients’ experiences is essential when incorporating the patients’ perspective in improving healthcare. Experiences are mainly collected using closed-ended questions, although the value of open-ended questions is widely recognized. Natural language processing (NLP) can automate the analysis of open-ended questions for an efficient approach to patient-centeredness.

Methods

We developed the Artificial Intelligence Patient-Reported Experience Measures (AI-PREM) tool, consisting of a new, open-ended questionnaire, an NLP pipeline to analyze the answers using sentiment analysis and topic modeling, and a visualization to guide physicians through the results. The questionnaire and NLP pipeline were iteratively developed and validated in a clinical context.

Results

The final AI-PREM consisted of five open-ended questions about the provided information, personal approach, collaboration between healthcare professionals, organization of care, and other experiences. The AI-PREM was sent to 867 vestibular schwannoma patients, 534 of which responded. The sentiment analysis model attained an F1 score of 0.97 for positive texts and 0.63 for negative texts. There was a 90% overlap between automatically and manually extracted topics. The visualization was hierarchically structured into three stages: the sentiment per question, the topics per sentiment and question, and the original patient responses per topic.

Conclusions

The AI-PREM tool is a comprehensive method that combines a validated, open-ended questionnaire with a well-performing NLP pipeline and visualization. Thematically organizing and quantifying patient feedback reduces the time invested by healthcare professionals to evaluate and prioritize patient experiences without being confined to the limited answer options of closed-ended questions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01923-5.

Triaging Medical Referrals Based on Clinical Prioritisation Criteria Using Machine Learning Techniques

Triaging of medical referrals can be completed using various machine learning techniques, but trained models with historical datasets may not be relevant as the clinical criteria for triaging are regularly updated and changed. This paper proposes the use of machine learning techniques coupled with the clinical prioritisation criteria (CPC) of Queensland (QLD), Australia, to deliver better triaging for referrals in accordance with the CPC’s updates. The unique feature of the proposed model is its non-reliance on the past datasets for model training. Medical Natural Language Processing (NLP) was applied in the proposed approach to process the medical referrals, which are unstructured free text. The proposed multiclass classification approach achieved a Micro F1 score = 0.98. The proposed approach can help in the processing of two million referrals that the QLD health service receives annually; therefore, they can deliver better and more efficient health services.

Comparison of the Erectile Dysfunction Drugs Sildenafil and Tadalafil Using Patient Medication Reviews: Topic Modeling Study

BACKGROUND

Topic modeling of patient medication reviews of erectile dysfunction (ED) drugs can help identify patient preferences regarding ED treatment options. The identification of a set of topics important to the patient from social network service drug reviews would inform the design of patient-centered medication counseling.

OBJECTIVE

This study aimed to (1) identify the distinctive topics from patient medication reviews unique to tadalafil versus sildenafil; (2) determine if the primary topics are distributed differently for each drug and for each patient characteristic (age and time on ED drug therapy); and (3) test if the primary topics affect satisfaction with ED drug therapy controlling for patient characteristics.

METHODS

Data were collected from the patient medication reviews of sildenafil and tadalafil posted on WebMD and Ask a Patient. The latent Dirichlet allocation method of natural language processing was used to identify 5 distinctive topics from the patient medication reviews on each drug. Analysis of variance and a 2-sample t test were conducted to compare the topic distribution and assess whether patient satisfaction varies with the primary topics, age, and time on medication for each ED drug. Statistical significance was tested at an alpha of .05.

RESULTS

The patient medication reviews of sildenafil (N=463) had 2 topics on treatment benefit and 1 each on medication safety, marketing claim, and treatment comparison, while the patient medication reviews of tadalafil (N=919) had 2 topics on medication safety and 1 each on the remaining subjects. Sildenafil's reviewers quite frequently (94/463, 20.4%) mentioned erection sustainability as their primary topic, whereas tadalafil's reviewers were more concerned about severe medication safety. Those who mentioned erection sustainability as their primary topic were quite satisfied with their treatment as opposed to those who mentioned severe medication safety as their primary topic (score 3.85 vs 2.44). The discovered topics reflected the marketing claims of blue magic and amber romance for sildenafil and tadalafil, respectively. The topic of blue magic was preferred among younger patients, while the topic of amber romance was preferred among older patients. The topic alternative choices, which appeared for both the ED drugs, reflected patient interest in the comparative effectiveness and price outside the drug labeling information.

CONCLUSIONS

The patient medication reviews of ED drugs reflect patient preferences regarding drug labeling information, marketing claims, and alternative treatment choices. The patient preferences concerning ED treatment attributes inform the design of patient-centered communication for improved ED drug therapy.