AI Chatbots in Clinical Laboratory Medicine: Foundations and Trends

Advancing Clinical Chatbot Validation Using AI-Powered Evaluation With a New 3-Bot Evaluation System: Instrument Validation Study

Background

The health care sector faces a projected shortfall of 10 million workers by 2030. Artificial intelligence (AI) automation in areas such as patient education and initial therapy screening presents a strategic response to mitigate this shortage and reallocate medical staff to higher-priority tasks. However, current methods of evaluating early-stage health care AI chatbots are highly limited due to safety concerns and the amount of time and effort that goes into evaluating them.

Objective

This study introduces a novel 3-bot method for efficiently testing and validating early-stage AI health care provider chatbots. To extensively test AI provider chatbots without involving real patients or researchers, various AI patient bots and an evaluator bot were developed.

Methods

Provider bots interacted with AI patient bots embodying frustrated, anxious, or depressed personas. An evaluator bot reviewed interaction transcripts based on specific criteria. Human experts then reviewed each interaction transcript, and the evaluator bot's results were compared to human evaluation results to ensure accuracy.

Results

The patient-education bot's evaluations by the AI evaluator and the human evaluator were nearly identical, with minimal variance, limiting the opportunity for further analysis. The screening bot's evaluations also yielded similar results between the AI evaluator and human evaluator. Statistical analysis confirmed the reliability and accuracy of the AI evaluations.

Conclusions

The innovative evaluation method ensures a safe, adaptable, and effective means to test and refine early versions of health care provider chatbots without risking patient safety or investing excessive researcher time and effort. Our patient-education evaluator bots could have benefitted from larger evaluation criteria, as we had extremely similar results from the AI and human evaluators, which could have arisen because of the small number of evaluation criteria. We were limited in the amount of prompting we could input into each bot due to the practical consideration that response time increases with larger and larger prompts. In the future, using techniques such as retrieval augmented generation will allow the system to receive more information and become more specific and accurate in evaluating the chatbots. This evaluation method will allow for rapid testing and validation of health care chatbots to automate basic medical tasks, freeing providers to address more complex tasks.

Advancing Laboratory Medicine Practice With Machine Learning: Swift yet Exact

Machine learning (ML) is currently being widely studied and applied in data analysis and prediction in various fields, including laboratory medicine. To comprehensively evaluate the application of ML in laboratory medicine, we reviewed the literature on ML applications in laboratory medicine published between February 2014 and March 2024. A PubMed search using a search string yielded 779 articles on the topic, among which 144 articles were selected for this review. These articles were analyzed to extract and categorize related fields within laboratory medicine, research objectives, specimen types, data types, ML models, evaluation metrics, and sample sizes. Sankey diagrams and pie charts were used to illustrate the relationships between categories and the proportions within each category. We found that most studies involving the application of ML in laboratory medicine were designed to improve efficiency through automation or expand the roles of clinical laboratories. The most common ML models used are convolutional neural networks, multilayer perceptrons, and tree-based models, which are primarily selected based on the type of input data. Our findings suggest that, as the technology evolves, ML will rise in prominence in laboratory medicine as a tool for expanding research activities. Nonetheless, expertise in ML applications should be improved to effectively utilize this technology.

Automatic kidney stone identification: an adaptive feature-weighted LSTM model based on urine and blood routine analysis

Kidney stones are the most common urinary system diseases, and early identification is of great significance. The purpose of this study was to use routine urine and blood detection indices to build a deep learning (DL) model to identify the presence of kidney stones in the early stage. A retrospective analysis was conducted on patients with kidney stones who were treated at West China Hospital of Sichuan University from January 2020 to June 2023. A total of 1130 individuals presenting with kidney stones and 1230 healthy subjects were enrolled. The first blood and urine laboratory data of participants at our hospital were collected, and the data were divided into a training dataset (80%) and a verification dataset (20%). Additionally, a long short-term memory (LSTM)-based adaptive feature weighting model was trained for the early identification of kidney stones, and the results were compared with those of other models. The performance of the model was evaluated by the area under the subject working characteristic curve (AUC). The important predictive factors are determined by ranking the characteristic importance of the predictive factors. A total of 17 variables were screened; among the top 4 characteristics according to the weight coefficient in this model, urine WBC, urine occult blood, qualitative urinary protein, and microcyte percentage had high predictive value for kidney stones in patients. The accuracy of the kidney stone (KS-LSTM) learning model was 89.5%, and the AUC was 0.95. Compared with other models, it has better performance. The results show that the KS-LSTM model based on routine urine and blood tests can accurately identify the presence of kidney stones. And provide valuable assistance for clinicians to identify kidney stones in the early stage.

ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles

The attainment of academic superiority relies heavily upon the accessibility of scholarly resources and the expression of research findings through faultless language usage. Although modern tools, such as the Publish or Perish software program, are proficient in sourcing academic papers based on specific keywords, they often fall short of extracting comprehensive content, including crucial references. The challenge of linguistic precision remains a prominent issue, particularly for research papers composed by non-native English speakers who may encounter word usage errors. This manuscript serves a twofold purpose: firstly, it reassesses the effectiveness of ChatGPT-4 in the context of retrieving pertinent references tailored to specific research topics. Secondly, it introduces a suite of language editing services that are skilled in rectifying word usage errors, ensuring the refined presentation of research outcomes. The article also provides practical guidelines for formulating precise queries to mitigate the risks of erroneous language usage and the inclusion of spurious references. In the ever-evolving realm of academic discourse, leveraging the potential of advanced AI, such as ChatGPT-4, can significantly enhance the quality and impact of scientific publications.

Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge-Attitudes-Practices Survey

BACKGROUND

Pharmacists need up-to-date knowledge and decision-making support in HIV care. We aim to develop MARVIN-Pharma, an adapted artificial intelligence-based chatbot initially for people with HIV, to assist pharmacists in considering evidence-based needs.

METHODS

From December 2022 to December 2023, an online needs-assessment survey evaluated Québec pharmacists' knowledge, attitudes, involvement, and barriers relative to HIV care, alongside perceptions relevant to the usability of MARVIN-Pharma. Recruitment involved convenience and snowball sampling, targeting National HIV and Hepatitis Mentoring Program affiliates.

RESULTS

Forty-one pharmacists (28 community, 13 hospital-based) across 15 Québec municipalities participated. Participants perceived their HIV knowledge as moderate (M = 3.74/6). They held largely favorable attitudes towards providing HIV care (M = 4.02/6). They reported a "little" involvement in the delivery of HIV care services (M = 2.08/5), most often ART adherence counseling, refilling, and monitoring. The most common barriers reported to HIV care delivery were a lack of time, staff resources, clinical tools, and HIV information/training, with pharmacists at least somewhat agreeing that they experienced each (M ≥ 4.00/6). On average, MARVIN-Pharma's acceptability and compatibility were in the 'undecided' range (M = 4.34, M = 4.13/7, respectively), while pharmacists agreed to their self-efficacy to use online health services (M = 5.6/7).

CONCLUSION

MARVIN-Pharma might help address pharmacists' knowledge gaps and barriers to HIV treatment and care, but pharmacist engagement in the chatbot's development seems vital for its future uptake and usability.

Evolution of Chatbots in Nursing Education: Narrative Review

Background

The integration of chatbots in nursing education is a rapidly evolving area with potential transformative impacts. This narrative review aims to synthesize and analyze the existing literature on chatbots in nursing education.

Objective

This study aims to comprehensively examine the temporal trends, international distribution, study designs, and implications of chatbots in nursing education.

Methods

A comprehensive search was conducted across 3 databases (PubMed, Web of Science, and Embase) following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.

Results

A total of 40 articles met the eligibility criteria, with a notable increase of publications in 2023 (n=28, 70%). Temporal analysis revealed a notable surge in publications from 2021 to 2023, emphasizing the growing scholarly interest. Geographically, Taiwan province made substantial contributions (n=8, 20%), followed by the United States (n=6, 15%) and South Korea (n=4, 10%). Study designs varied, with reviews (n=8, 20%) and editorials (n=7, 18%) being predominant, showcasing the richness of research in this domain.

Conclusions

Integrating chatbots into nursing education presents a promising yet relatively unexplored avenue. This review highlights the urgent need for original research, emphasizing the importance of ethical considerations.

Applications of Large Language Models in Pathology

Large language models (LLMs) are transformer-based neural networks that can provide human-like responses to questions and instructions. LLMs can generate educational material, summarize text, extract structured data from free text, create reports, write programs, and potentially assist in case sign-out. LLMs combined with vision models can assist in interpreting histopathology images. LLMs have immense potential in transforming pathology practice and education, but these models are not infallible, so any artificial intelligence generated content must be verified with reputable sources. Caution must be exercised on how these models are integrated into clinical practice, as these models can produce hallucinations and incorrect results, and an over-reliance on artificial intelligence may lead to de-skilling and automation bias. This review paper provides a brief history of LLMs and highlights several use cases for LLMs in the field of pathology.

The performance of artificial intelligence chatbot large language models to address skeletal biology and bone health queries

Artificial intelligence (AI) chatbots utilizing large language models (LLMs) have recently garnered significant interest due to their ability to generate humanlike responses to user inquiries in an interactive dialog format. While these models are being increasingly utilized to obtain medical information by patients, scientific and medical providers, and trainees to address biomedical questions, their performance may vary from field to field. The opportunities and risks these chatbots pose to the widespread understanding of skeletal health and science are unknown. Here we assess the performance of 3 high-profile LLM chatbots, Chat Generative Pre-Trained Transformer (ChatGPT) 4.0, BingAI, and Bard, to address 30 questions in 3 categories: basic and translational skeletal biology, clinical practitioner management of skeletal disorders, and patient queries to assess the accuracy and quality of the responses. Thirty questions in each of these categories were posed, and responses were independently graded for their degree of accuracy by four reviewers. While each of the chatbots was often able to provide relevant information about skeletal disorders, the quality and relevance of these responses varied widely, and ChatGPT 4.0 had the highest overall median score in each of the categories. Each of these chatbots displayed distinct limitations that included inconsistent, incomplete, or irrelevant responses, inappropriate utilization of lay sources in a professional context, a failure to take patient demographics or clinical context into account when providing recommendations, and an inability to consistently identify areas of uncertainty in the relevant literature. Careful consideration of both the opportunities and risks of current AI chatbots is needed to formulate guidelines for best practices for their use as source of information about skeletal health and biology.

Revealing the Complexity of Fatigue: A Review of the Persistent Challenges and Promises of Artificial Intelligence

Part I reviews persistent challenges obstructing progress in understanding complex fatigue's biology. Difficulties quantifying subjective symptoms, mapping multi-factorial mechanisms, accounting for individual variation, enabling invasive sensing, overcoming research/funding insularity, and more are discussed. Part II explores how emerging artificial intelligence and machine and deep learning techniques can help address limitations through pattern recognition of complex physiological signatures as more objective biomarkers, predictive modeling to capture individual differences, consolidation of disjointed findings via data mining, and simulation to explore interventions. Conversational agents like Claude and ChatGPT also have potential to accelerate human fatigue research, but they currently lack capacities for robust autonomous contributions. Envisioned is an innovation timeline where synergistic application of enhanced neuroimaging, biosensors, closed-loop systems, and other advances combined with AI analytics could catalyze transformative progress in elucidating fatigue neural circuitry and treating associated conditions over the coming decades.