Clinical Decision Support in the Era of Artificial Intelligence

Automated Lugano Metabolic Response Assessment in 18F-Fluorodeoxyglucose-Avid Non-Hodgkin Lymphoma With Deep Learning on 18F-Fluorodeoxyglucose-Positron Emission Tomography

PURPOSE

Artificial intelligence can reduce the time used by physicians on radiological assessments. For 18F-fluorodeoxyglucose-avid lymphomas, obtaining complete metabolic response (CMR) by end of treatment is prognostic.

METHODS

Here, we present a deep learning-based algorithm for fully automated treatment response assessments according to the Lugano 2014 classification. The proposed four-stage method, trained on a multicountry clinical trial (ClinicalTrials.gov identifier: NCT01287741) and tested in three independent multicenter and multicountry test sets on different non-Hodgkin lymphoma subtypes and different lines of treatment (ClinicalTrials.gov identifiers NCT02257567, NCT02500407; 20% holdout in ClinicalTrials.gov identifier NCT01287741), outputs the detected lesions at baseline and follow-up to enable focused radiologist review.

RESULTS

The method's response assessment achieved high agreement with the adjudicated radiologic responses (eg, agreement for overall response assessment of 93%, 87%, and 85% in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, respectively) similar to inter-radiologist agreement and was strongly prognostic of outcomes with a trend toward higher accuracy for death risk than adjudicated radiologic responses (hazard ratio for end of treatment by-model CMR of 0.123, 0.054, and 0.205 in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, compared with, respectively, 0.226, 0.292, and 0.272 for CMR by the adjudicated responses). Furthermore, a radiologist review of the algorithm's assessments was conducted. The radiologist median review time was 1.38 minutes/assessment, and no statistically significant differences were observed in the level of agreement of the radiologist with the model's response compared with the level of agreement of the radiologist with the adjudicated responses.

CONCLUSION

These results suggest that the proposed method can be incorporated into radiologic response assessment workflows in cancer imaging for significant time savings and with performance similar to trained medical experts.

Appropriateness of Frequently Asked Patient Questions Following Total Hip Arthroplasty From ChatGPT Compared to Arthroplasty-Trained Nurses

BACKGROUND

The use of ChatGPT (Generative Pretrained Transformer), which is a natural language artificial intelligence model, has gained unparalleled attention with the accumulation of over 100 million users within months of launching. As such, we aimed to compare the following: 1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after total hip arthroplasty; and 2) patients' evaluation of ChatGPT and arthroplasty-trained nurses responses to answer their postoperative questions.

METHODS

We prospectively created 60 questions to address the most commonly asked patient questions following total hip arthroplasty. We obtained answers from arthroplasty-trained nurses and from the ChatGPT-3.5 version for each of the questions. Surgeons graded each set of responses based on clinical judgment as 1) "appropriate," 2) "inappropriate" if the response contained inappropriate information, or 3) "unreliable" if the responses provided inconsistent content. Each patient was given a randomly selected question from the 60 aforementioned questions, with responses provided by ChatGPT and arthroplasty-trained nurses, using a Research Electronic Data Capture survey hosted at our local hospital.

RESULTS

The 3 fellowship-trained surgeons graded 56 out of 60 (93.3%) responses for the arthroplasty-trained nurses and 57 out of 60 (95.0%) for ChatGPT to be "appropriate." There were 175 out of 252 (69.4%) patients who were more comfortable following the ChatGPT responses and 77 out of 252 (30.6%) who preferred arthroplasty-trained nurses' responses. However, 199 out of 252 patients (79.0%) responded that they were "uncertain" with regard to trusting AI to answer their postoperative questions.

CONCLUSIONS

ChatGPT provided appropriate answers from a physician perspective. Patients were also more comfortable with the ChatGPT responses than those from arthroplasty-trained nurses. Inevitably, its successful implementation is dependent on its ability to provide credible information that is consistent with the goals of the physician and patient alike.

Exploring explainable AI features in the vocal biomarkers of lung disease

This review delves into the burgeoning field of explainable artificial intelligence (XAI) in the detection and analysis of lung diseases through vocal biomarkers. Lung diseases, often elusive in their early stages, pose a significant public health challenge. Recent advancements in AI have ushered in innovative methods for early detection, yet the black-box nature of many AI models limits their clinical applicability. XAI emerges as a pivotal tool, enhancing transparency and interpretability in AI-driven diagnostics. This review synthesizes current research on the application of XAI in analyzing vocal biomarkers for lung diseases, highlighting how these techniques elucidate the connections between specific vocal features and lung pathology. We critically examine the methodologies employed, the types of lung diseases studied, and the performance of various XAI models. The potential for XAI to aid in early detection, monitor disease progression, and personalize treatment strategies in pulmonary medicine is emphasized. Furthermore, this review identifies current challenges, including data heterogeneity and model generalizability, and proposes future directions for research. By offering a comprehensive analysis of explainable AI features in the context of lung disease detection, this review aims to bridge the gap between advanced computational approaches and clinical practice, paving the way for more transparent, reliable, and effective diagnostic tools.

Comparative Analysis of Accuracy, Readability, Sentiment, and Actionability: Artificial Intelligence Chatbots (ChatGPT and Google Gemini) versus Traditional Patient Information Leaflets for Local Anesthesia in Eye Surgery

Background and Aim

Eye surgeries often evoke strong negative emotions in patients, including fear and anxiety. Patient education material plays a crucial role in informing and empowering individuals. Traditional sources of medical information may not effectively address individual patient concerns or cater to varying levels of understanding. This study aims to conduct a comparative analysis of the accuracy, completeness, readability, tone, and understandability of patient education material generated by AI chatbots versus traditional Patient Information Leaflets (PILs), focusing on local anesthesia in eye surgery.

Methods

Expert reviewers evaluated responses generated by AI chatbots (ChatGPT and Google Gemini) and a traditional PIL (Royal College of Anaesthetists' PIL) based on accuracy, completeness, readability, sentiment, and understandability. Statistical analyses, including ANOVA and Tukey HSD tests, were conducted to compare the performance of the sources.

Results

Readability analysis showed variations in complexity among the sources, with AI chatbots offering simplified language and PILs maintaining better overall readability and accessibility. Sentiment analysis revealed differences in emotional tone, with Google Gemini exhibiting the most positive sentiment. AI chatbots demonstrated superior understandability and actionability, while PILs excelled in completeness. Overall, ChatGPT showed slightly higher accuracy (scores expressed as mean ± standard deviation) (4.71 ± 0.5 vs 4.61 ± 0.62) and completeness (4.55 ± 0.58 vs 4.47 ± 0.58) compared to Google Gemini, but PILs performed best (4.84 ± 0.37 vs 4.88 ± 0.33) in terms of both accuracy and completeness (p-value for completeness <0.05).

Conclusion

AI chatbots show promise as innovative tools for patient education, complementing traditional PILs. By leveraging the strengths of both AI-driven technologies and human expertise, healthcare providers can enhance patient education and empower individuals to make informed decisions about their health and medical care.

ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

OBJECTIVE

To evaluate the performance of an artificial intelligence (AI) large language model, ChatGPT (version 4.0), for common retinal diseases, in accordance with the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) guidelines.

DESIGN

A cross-sectional survey study design was employed to compare the responses made by ChatGPT to established clinical guidelines.

PARTICIPANTS

Responses by the AI were reviewed by a panel of three vitreoretinal specialists for evaluation.

METHODS

To investigate ChatGPT's comprehension of clinical guidelines, we designed 130 questions covering a broad spectrum of topics within 12 AAO PPP domains of retinal disease These questions were crafted to encompass diagnostic criteria, treatment guidelines, and management strategies, including both medical and surgical aspects of retinal care. A panel of 3 retinal specialists independently evaluated responses on a Likert scale from 1 to 5 based on their relevance, accuracy, and adherence to AAO PPP guidelines. Response readability was evaluated using Flesch Readability Ease and Flesch-Kincaid grade level scores.

RESULTS

ChatGPT achieved an overall average score of 4.9/5.0, suggesting high alignment with the AAO PPP guidelines. Scores varied across domains, with the lowest in the surgical management of disease. The responses had a low reading ease score and required a college-to-graduate level of comprehension. Identified errors were related to diagnostic criteria, treatment options, and methodological procedures.

CONCLUSION

ChatGPT 4.0 demonstrated significant potential in generating guideline-concordant responses, particularly for common medical retinal diseases. However, its performance slightly decreased in surgical retina, highlighting the ongoing need for clinician input, further model refinement, and improved comprehensibility.

Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians

BACKGROUND

ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education.

METHODS

150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses.

RESULTS

ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150).

DISCUSSION

ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.

The Role of Artificial Intelligence in Early Diagnosis and Molecular Classification of Head and Neck Skin Cancers: A Multidisciplinary Approach

Cancer remains a significant global health concern, with increasing genetic and metabolic irregularities linked to its onset. Among various forms of cancer, skin cancer, including squamous cell carcinoma, basal cell carcinoma, and melanoma, is on the rise worldwide, often triggered by ultraviolet (UV) radiation. The propensity of skin cancer to metastasize highlights the importance of early detection for successful treatment. This narrative review explores the evolving role of artificial intelligence (AI) in diagnosing head and neck skin cancers from both radiological and pathological perspectives. In the past two decades, AI has made remarkable progress in skin cancer research, driven by advances in computational capabilities, digitalization of medical images, and radiomics data. AI has shown significant promise in image-based diagnosis across various medical domains. In dermatology, AI has played a pivotal role in refining diagnostic and treatment strategies, including genomic risk assessment. This technology offers substantial potential to aid primary clinicians in improving patient outcomes. Studies have demonstrated AI's effectiveness in identifying skin lesions, categorizing them, and assessing their malignancy, contributing to earlier interventions and better prognosis. The rising incidence and mortality rates of skin cancer, coupled with the high cost of treatment, emphasize the need for early diagnosis. Further research and integration of AI into clinical practice are warranted to maximize its benefits in skin cancer diagnosis and treatment.

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.

Preferences for Management of Pediatric Pneumonia: A Clinician Survey of Artificially Generated Patient Cases

BACKGROUND

It is unknown which factors are associated with chest radiograph (CXR) and antibiotic use for suspected community-acquired pneumonia (CAP) in children. We evaluated factors associated with CXR and antibiotic preferences among clinicians for children with suspected CAP using case scenarios generated through artificial intelligence (AI).

METHODS

We performed a survey of general pediatric, pediatric emergency medicine, and emergency medicine attending physicians employed by a private physician contractor. Respondents were given 5 unique, AI-generated case scenarios. We used generalized estimating equations to identify factors associated with CXR and antibiotic use. We evaluated the cluster-weighted correlation between clinician suspicion and clinical prediction model risk estimates for CAP using 2 predictive models.

RESULTS

A total of 172 respondents provided responses to 839 scenarios. Factors associated with CXR acquisition (OR, [95% CI]) included presence of crackles (4.17 [2.19, 7.95]), prior pneumonia (2.38 [1.32, 4.20]), chest pain (1.90 [1.18, 3.05]) and fever (1.82 [1.32, 2.52]). The decision to use antibiotics before knowledge of CXR results included past hospitalization for pneumonia (4.24 [1.88, 9.57]), focal decreased breath sounds (3.86 [1.98, 7.52]), and crackles (3.45 [2.15, 5.53]). After revealing CXR results to clinicians, these results were the sole predictor associated with antibiotic decision-making. Suspicion for CAP correlated with one of 2 prediction models for CAP (Spearman's rho = 0.25). Factors associated with a greater suspicion of pneumonia included prior pneumonia, duration of illness, worsening course of illness, shortness of breath, vomiting, decreased oral intake or urinary output, respiratory distress, head nodding, focal decreased breath sounds, focal rhonchi, fever, and crackles, and lower pulse oximetry.

CONCLUSIONS

Ordering preferences for CXRs demonstrated similarities and differences with evidence-based risk models for CAP. Clinicians relied heavily on CXR findings to guide antibiotic ordering. These findings can be used within decision support systems to promote evidence-based management practices for pediatric CAP.

Development of a machine learning model for predicting pneumothorax risk in coaxial core needle biopsy (≤3 cm)

PURPOSE

The aim is to devise a machine learning algorithm exploiting preoperative clinical data to forecast the hazard of pneumothorax post-coaxial needle lung biopsy (CCNB), thereby informing clinical decision-making and enhancing perioperative care.

METHOD

This retrospective analysis aggregated clinical and imaging data from patients with lung nodules (≤3 cm) biopsies. Variable selection was done using univariate analysis and LASSO regression, with the dataset subsequently divided into training (80 %) and validation (20 %) subsets. Various machine learning (ML) classifiers were employed in a consolidated approach to ascertain the paramount model, which was followed by individualized risk profiling showcased through Shapley Additive eXplanations (SHAP).

RESULTS

Out of the 325 patients included in the study, 19.6% (64/325) experienced postoperative pneumothorax. High-risk factors determined were Cancer, Lesion_type, GOLD, Size, and Depth. The Gaussian Naive Bayes (GNB) classifier demonstrated superior prediction with an Area Under the Curve (AUC) of 0.82 (95% CI 0.71-0.94), complemented by an accuracy rate of 0.8, sensitivity of 0.71, specificity of 0.84, and an F1 score of 0.61 in the test cohort.

CONCLUSION

The formulated prognostic algorithm exhibited commendable efficacy in preoperatively prognosticating CCNB-induced pneumothorax, harboring the potential to refine personalized risk appraisals, steer clinical judgment, and ameliorate perioperative patient stewardship.

Assessing the Utility, Impact, and Adoption Challenges of an Artificial Intelligence-Enabled Prescription Advisory Tool for Type 2 Diabetes Management: Qualitative Study

BACKGROUND

The clinical management of type 2 diabetes mellitus (T2DM) presents a significant challenge due to the constantly evolving clinical practice guidelines and growing array of drug classes available. Evidence suggests that artificial intelligence (AI)-enabled clinical decision support systems (CDSSs) have proven to be effective in assisting clinicians with informed decision-making. Despite the merits of AI-driven CDSSs, a significant research gap exists concerning the early-stage implementation and adoption of AI-enabled CDSSs in T2DM management.

OBJECTIVE

This study aimed to explore the perspectives of clinicians on the use and impact of the AI-enabled Prescription Advisory (APA) tool, developed using a multi-institution diabetes registry and implemented in specialist endocrinology clinics, and the challenges to its adoption and application.

METHODS

We conducted focus group discussions using a semistructured interview guide with purposively selected endocrinologists from a tertiary hospital. The focus group discussions were audio-recorded and transcribed verbatim. Data were thematically analyzed.

RESULTS

A total of 13 clinicians participated in 4 focus group discussions. Our findings suggest that the APA tool offered several useful features to assist clinicians in effectively managing T2DM. Specifically, clinicians viewed the AI-generated medication alterations as a good knowledge resource in supporting the clinician's decision-making on drug modifications at the point of care, particularly for patients with comorbidities. The complication risk prediction was seen as positively impacting patient care by facilitating early doctor-patient communication and initiating prompt clinical responses. However, the interpretability of the risk scores, concerns about overreliance and automation bias, and issues surrounding accountability and liability hindered the adoption of the APA tool in clinical practice.

CONCLUSIONS

Although the APA tool holds great potential as a valuable resource for improving patient care, further efforts are required to address clinicians' concerns and improve the tool's acceptance and applicability in relevant contexts.

Chinese Oncologists' Perspectives on Integrating AI into Clinical Practice: Cross-Sectional Survey Study

BACKGROUND

The rapid development of artificial intelligence (AI) has brought significant interest to its potential applications in oncology. Although AI-powered tools are already being implemented in some Chinese hospitals, their integration into clinical practice raises several concerns for Chinese oncologists.

OBJECTIVE

This study aims to explore the concerns of Chinese oncologists regarding the integration of AI into clinical practice and to identify the factors influencing these concerns.

METHODS

A total of 228 Chinese oncologists participated in a cross-sectional web-based survey from April to June in 2023 in mainland China. The survey gauged their worries about AI with multiple-choice questions. The survey evaluated their views on the statements of "The impact of AI on the doctor-patient relationship" and "AI will replace doctors." The data were analyzed using descriptive statistics, and variate analyses were used to find correlations between the oncologists' backgrounds and their concerns.

RESULTS

The study revealed that the most prominent concerns were the potential for AI to mislead diagnosis and treatment (163/228, 71.5%); an overreliance on AI (162/228, 71%); data and algorithm bias (123/228, 54%); issues with data security and patient privacy (123/228, 54%); and a lag in the adaptation of laws, regulations, and policies in keeping up with AI's development (115/228, 50.4%). Oncologists with a bachelor's degree expressed heightened concerns related to data and algorithm bias (34/49, 69%; P=.03) and the lagging nature of legal, regulatory, and policy issues (32/49, 65%; P=.046). Regarding AI's impact on doctor-patient relationships, 53.1% (121/228) saw a positive impact, whereas 35.5% (81/228) found it difficult to judge, 9.2% (21/228) feared increased disputes, and 2.2% (5/228) believed that there is no impact. Although sex differences were not significant (P=.08), perceptions varied-male oncologists tended to be more positive than female oncologists (74/135, 54.8% vs 47/93, 50%). Oncologists with a bachelor's degree (26/49, 53%; P=.03) and experienced clinicians (≥21 years; 28/56, 50%; P=.054). found it the hardest to judge. Those with IT experience were significantly more positive (25/35, 71%) than those without (96/193, 49.7%; P=.02). Opinions regarding the possibility of AI replacing doctors were diverse, with 23.2% (53/228) strongly disagreeing, 14% (32/228) disagreeing, 29.8% (68/228) being neutral, 16.2% (37/228) agreeing, and 16.7% (38/228) strongly agreeing. There were no significant correlations with demographic and professional factors (all P>.05).

CONCLUSIONS

Addressing oncologists' concerns about AI requires collaborative efforts from policy makers, developers, health care professionals, and legal experts. Emphasizing transparency, human-centered design, bias mitigation, and education about AI's potential and limitations is crucial. Through close collaboration and a multidisciplinary strategy, AI can be effectively integrated into oncology, balancing benefits with ethical considerations and enhancing patient care.

Computerized decision support is an effective approach to select memory clinic patients for amyloid-PET

BACKGROUND

The use of amyloid-PET in dementia workup is upcoming. At the same time, amyloid-PET is costly and limitedly available. While the appropriate use criteria (AUC) aim for optimal use of amyloid-PET, their limited sensitivity hinders the translation to clinical practice. Therefore, there is a need for tools that guide selection of patients for whom amyloid-PET has the most clinical utility. We aimed to develop a computerized decision support approach to select patients for amyloid-PET.

METHODS

We included 286 subjects (135 controls, 108 Alzheimer's disease dementia, 33 frontotemporal lobe dementia, and 10 vascular dementia) from the Amsterdam Dementia Cohort, with available neuropsychology, APOE, MRI and [18F]florbetaben amyloid-PET. In our computerized decision support approach, using supervised machine learning based on the DSI classifier, we first classified the subjects using only neuropsychology, APOE, and quantified MRI. Then, for subjects with uncertain classification (probability of correct class (PCC) < 0.75) we enriched classification by adding (hypothetical) amyloid positive (AD-like) and negative (normal) PET visual read results and assessed whether the diagnosis became more certain in at least one scenario (PPC≥0.75). If this was the case, the actual visual read result was used in the final classification. We compared the proportion of PET scans and patients diagnosed with sufficient certainty in the computerized approach with three scenarios: 1) without amyloid-PET, 2) amyloid-PET according to the AUC, and 3) amyloid-PET for all patients.

RESULTS

The computerized approach advised PET in n = 60(21%) patients, leading to a diagnosis with sufficient certainty in n = 188(66%) patients. This approach was more efficient than the other three scenarios: 1) without amyloid-PET, diagnostic classification was obtained in n = 155(54%), 2) applying the AUC resulted in amyloid-PET in n = 113(40%) and diagnostic classification in n = 156(55%), and 3) performing amyloid-PET in all resulted in diagnostic classification in n = 154(54%).

CONCLUSION

Our computerized data-driven approach selected 21% of memory clinic patients for amyloid-PET, without compromising diagnostic performance. Our work contributes to a cost-effective implementation and could support clinicians in making a balanced decision in ordering additional amyloid PET during the dementia workup.

How Will Artificial Intelligence Shape the Future of Decision-Making in Congenital Heart Disease?

Improvements in medical technology have significantly changed the management of congenital heart disease (CHD), offering novel tools to predict outcomes and personalize follow-up care. By using sophisticated imaging modalities, computational models and machine learning algorithms, clinicians can experiment with unprecedented insights into the complex anatomy and physiology of CHD. These tools enable early identification of high-risk patients, thus allowing timely, tailored interventions and improved outcomes. Additionally, the integration of genetic testing offers valuable prognostic information, helping in risk stratification and treatment optimisation. The birth of telemedicine platforms and remote monitoring devices facilitates customised follow-up care, enhancing patient engagement and reducing healthcare disparities. Taking into consideration challenges and ethical issues, clinicians can make the most of the full potential of artificial intelligence (AI) to further refine prognostic models, personalize care and improve long-term outcomes for patients with CHD. This narrative review aims to provide a comprehensive illustration of how AI has been implemented as a new technological method for enhancing the management of CHD.

Consumer satisfaction, palliative care and artificial intelligence (AI)

The scope of artificial intelligence (AI) in healthcare is promising, and AI has the potential to revolutionise the field of palliative care services also. Consumer satisfaction in palliative care is a critical aspect of providing high-quality end-of-life support. It encompasses various elements that contribute to a positive experience for both patients and their families. AI-based tools and technologies can help in early identification of the beneficiaries, reduce the cost, improve the quality of care and satisfaction to the patients with chronic life-limiting illnesses. However, it is essential to ensure that AI is used ethically and in a way that complements, rather than replaces, the human touch and compassionate care, which are the core components of palliative care. This article tries to analyse the scope and challenges of improving consumer satisfaction through AI-based technology in palliative care services.

Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls

OBJECTIVE

We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance.

MATERIALS AND METHODS

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

RESULTS

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

DISCUSSION

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

CONCLUSION

The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.

Personalized anesthesia and precision medicine: a comprehensive review of genetic factors, artificial intelligence, and patient-specific factors

Precision medicine, characterized by the personalized integration of a patient's genetic blueprint and clinical history, represents a dynamic paradigm in healthcare evolution. The emerging field of personalized anesthesia is at the intersection of genetics and anesthesiology, where anesthetic care will be tailored to an individual's genetic make-up, comorbidities and patient-specific factors. Genomics and biomarkers can provide more accurate anesthetic protocols, while artificial intelligence can simplify anesthetic procedures and reduce anesthetic risks, and real-time monitoring tools can improve perioperative safety and efficacy. The aim of this paper is to present and summarize the applications of these related fields in anesthesiology by reviewing them, exploring the potential of advanced technologies in the implementation and development of personalized anesthesia, realizing the future integration of new technologies into clinical practice, and promoting multidisciplinary collaboration between anesthesiology and disciplines such as genomics and artificial intelligence.

Development and validation of risk prediction model for recurrent cardiovascular events among Chinese: the Personalized CARdiovascular DIsease risk Assessment for Chinese model

Aims

Cardiovascular disease (CVD) is a leading cause of mortality, especially in developing countries. This study aimed to develop and validate a CVD risk prediction model, Personalized CARdiovascular DIsease risk Assessment for Chinese (P-CARDIAC), for recurrent cardiovascular events using machine learning technique.

Methods and results

Three cohorts of Chinese patients with established CVD were included if they had used any of the public healthcare services provided by the Hong Kong Hospital Authority (HA) since 2004 and categorized by their geographical locations. The 10-year CVD outcome was a composite of diagnostic or procedure codes with specific International Classification of Diseases, Ninth Revision, Clinical Modification. Multivariate imputation with chained equations and XGBoost were applied for the model development. The comparison with Thrombolysis in Myocardial Infarction Risk Score for Secondary Prevention (TRS-2°P) and Secondary Manifestations of ARTerial disease (SMART2) used the validation cohorts with 1000 bootstrap replicates. A total of 48 799, 119 672 and 140 533 patients were included in the derivation and validation cohorts, respectively. A list of 125 risk variables were used to make predictions on CVD risk, of which 8 classes of CVD-related drugs were considered interactive covariates. Model performance in the derivation cohort showed satisfying discrimination and calibration with a C statistic of 0.69. Internal validation showed good discrimination and calibration performance with C statistic over 0.6. The P-CARDIAC also showed better performance than TRS-2°P and SMART2.

Conclusion

Compared with other risk scores, the P-CARDIAC enables to identify unique patterns of Chinese patients with established CVD. We anticipate that the P-CARDIAC can be applied in various settings to prevent recurrent CVD events, thus reducing the related healthcare burden.

Should the Use of Adaptive Machine Learning Systems in Medicine be Classified as Research?

A novel advantage of the use of machine learning (ML) systems in medicine is their potential to continue learning from new data after implementation in clinical practice. To date, considerations of the ethical questions raised by the design and use of adaptive machine learning systems in medicine have, for the most part, been confined to discussion of the so-called "update problem," which concerns how regulators should approach systems whose performance and parameters continue to change even after they have received regulatory approval. In this paper, we draw attention to a prior ethical question: whether the continuous learning that will occur in such systems after their initial deployment should be classified, and regulated, as medical research? We argue that there is a strong prima facie case that the use of continuous learning in medical ML systems should be categorized, and regulated, as research and that individuals whose treatment involves such systems should be treated as research subjects.

Applying Machine Learning Techniques to Implementation Science

Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.

Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial

OBJECTIVES

We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2).

MATERIALS AND METHODS

A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected.

RESULTS

Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians.

DISCUSSION

Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved.

CONCLUSION

The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.

Clinical decision support system in emergency telephone triage: A scoping review of technical design, implementation and evaluation

OBJECTIVES

Emergency department overcrowding could be improved by upstream telephone triage. Emergency telephone triage aims at managing and orientating adequately patients as early as possible and distributing limited supply of staff and materials. This complex task could be improved with the use of Clinical decision support systems (CDSS). The aim of this scoping review was to identify literature gaps for the future development and evaluation of CDSS for Emergency telephone triage.

MATERIALS AND METHODS

We present here a scoping review of CDSS designed for emergency telephone triage, and compared them in terms of functional characteristics, technical design, health care implementation and methodologies used for evaluation, following the PRISMA-ScR guidelines.

RESULTS

Regarding design, 19 CDSS were retrieved: 12 were knowledge based CDSS (decisional algorithms built according to guidelines or clinical expertise) and 7 were data driven (statistical, machine learning, or deep learning models). Most of them aimed at assisting nurses or non-medical staff by providing patient orientation and/or severity/priority assessment. Eleven were implemented in real life, and only three were connected to the Electronic Health Record. Regarding evaluation, CDSS were assessed through various aspects: intrinsic characteristics, impact on clinical practice or user apprehension. Only one pragmatic trial and one randomized controlled trial were conducted.

CONCLUSION

This review highlights the potential of a hybrid system, user tailored, flexible, connected to the electronic health record, which could work with oral, video and digital data; and the need to evaluate CDSS on intrinsic characteristics and impact on clinical practice, iteratively at each distinct stage of the IT lifecycle.

Global Trends and Hotspots in Nursing Research on Decision Support Systems: A Bibliometric Analysis in CiteSpace

Decision support systems have been widely used in healthcare in recent years; however, there is lack of evidence on global trends and hotspots. This descriptive bibliometric study aimed to analyze bibliometric patterns of decision support systems in nursing. Data were extracted from the Web of Science Core Collection. Published research articles on decision support systems in nursing were identified. Co-occurrence and co-citation analysis was performed using CiteSpace version 6.1.R2. In total, 165 articles were analyzed. A total of 358 authors and 257 institutions from 20 countries contributed to this research field. The most productive authors were Andrew Johnson, Suzanne Bakken, Alessandro Febretti, Eileen S. O'Neill, and Kathryn H. Bowles. The most productive country and institution were the United States and Duke University, respectively. The top 10 keywords were "care," "clinical decision support," "clinical decision support system," "decision support system," "electronic health record," "system," "nursing informatics," "guideline," "decision support," and "outcomes." Common themes on keywords were planning intervention, national health information infrastructure, and methodological challenge. This study will help to find potential partners, countries, and institutions for future researchers, practitioners, and scholars. Additionally, it will contribute to health policy development, evidence-based practice, and further studies for researchers, practitioners, and scholars.

Applications of Artificial Intelligence in Helicopter Emergency Medical Services: A Scoping Review

OBJECTIVE

Recent systematic reviews of acute care medicine applications of artificial intelligence (AI) have focused on hospital and general prehospital uses. The purpose of this scoping review was to identify and describe the literature on AI use with a focus on applications in helicopter emergency medical services (HEMS).

METHODS

A literature search was performed with specific inclusion and exclusion criteria. Articles were grouped by characteristics such as publication year and general subject matter with categoric and temporal trend analyses.

RESULTS

We identified 21 records focused on the use of AI in HEMS. These applications included both clinical and triage uses and nonclinical uses. The earliest study appeared in 2006, but over one third of the identified studies have been published in 2021 or later. The passage of time has seen an increased likelihood of HEMS AI studies focusing on nonclinical issues; for each year, the likelihood of a nonclinical focus had an odds ratio of 1.3.

CONCLUSION

This scoping review provides overview and hypothesis-generating information regarding AI applications specific to HEMS. HEMS AI may be ultimately deployed in nonclinical arenas as much as or more than for clinical decision support. Future studies will inform future decisions as to how AI may improve HEMS systems design, asset deployment, and clinical care.

Evaluating clinical decision support software (CDSS): challenges for robust evidence generation

OBJECTIVES

Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

METHODS

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

RESULTS

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

CONCLUSION

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

A novel score for predicting 1-year mortality of intensive care patients

BACKGROUND

We aimed to develop a simple scoring table for predicting probability of death within 1-year after admission to an intensive care unit. We analysed data on emergency admissions from the nationwide Finnish intensive care quality registry.

METHODS

We included first admissions of adult patients with data available on 1-year vital status (dead or alive) and all five variables included in a premorbid functional status score, which is the number of activities the person can manage independently of the following five: get out of bed, move indoors, dress, climb stairs and walk 400 m. We analysed data on patient characteristics and admission-associated factors from 2012 to 2014 to find predictors of 1-year mortality and to develop a score for predicting probability of death. We tested the performance of this score in data from 2015. We assessed the 1-year functional status score of survivors with data available.

RESULTS

Out of 25,261 patients, 20,628 (81.7%) patients were able to perform all five functional activities independently prior to the intensive care unit admission. At 1-year post admission, 19,625 (77.7%) patients were alive. 1-year functional status score was known for 11,011 patients and 8970 (81.5%) patients achieved functional status score 5, managing all five activities independently. The score based on age, sex, preceding functional status, type of intensive care unit admission, severity of acute illness and the most significant diagnoses predicted 1-year mortality with an area under the receiver operating characteristic curve 0.78 (95% CI, 0.76-0.79). The calibration of our prediction model was good, with calibration intercept -0.01 (-0.07 to 0.05) and calibration slope 0.96 (0.90 to 1.02).

CONCLUSION

Our score based on data available at intensive care unit admission predicted 1-year mortality with fairly good discrimination. Most survivors achieved good functional recovery.

Large-Scale assessment of ChatGPT's performance in benign and malignant bone tumors imaging report diagnosis and its potential for clinical applications

Objective

This study was designed to delve into the complexities involved in diagnosing of benign and malignant bone tumors and to assess the potential of AI technologies like ChatGPT in improving diagnostic accuracy and efficiency. The study also explores the few-shot learning as a method to optimize ChatGPT's performance in specialized medical domains such as benign and malignant bone tumors diagnosis.

Methods

A total of 1366 benign and malignant bone tumors-related imaging reports were collected and diagnosed by 25 experienced physicians. The gold standard of diagnosis was established by combining clinical, imaging and pathological principles.These reports were then input into the ChatGPT model which underwent a few-shot learning method to generate diagnostic results. The diagnostic results of the physicians and the AI model were compared to evaluate the performance of ChatGPT. An experiment was conducted to assess the influence of different radiologist's reporting styles on the model's diagnostic performance. Furthermore, in-depth analysis of misdiagnosed cases was carried out, categorizing diagnostic errors and exploring possible causes.

Results

The diagnostic results generated by ChatGPT showed an accuracy of 0.73, sensitivity of 0.95, and specificity of 0.58. After few-shot learning, ChatGPT demonstrated significant improvement, achieving an accuracy of 0.87, sensitivity of 0.99, and specificity of 0.73, bringing it much closer to the level of physician diagnostics. In an experiment analyzing the influence of the radiologist's reporting style, the model demonstrated higher sensitivity when interpreting reports written by high-level radiologists. In 56 benign cases, ChatGPT misdiagnosed them as malignant. Among these, 35 benign lesions- fibrous dysplasia and osteofibrous dysplasia- were incorrectly identified as metastatic tumors or osteosarcomas; 8 cases of myositis ossificans were wrongly diagnosed as extraosseous osteosarcoma. 7 cases of giant cell tumor of bone at the end of long bone were misdiagnosed as osteosarcoma by intermediate doctors. Chondroblastoma was misdiagnosed as malignant tumor in 6 cases -2 osteosarcoma and 4 chondrosarcoma-In this study, 23 osteosarcoma cases were misdiagnosed by ChatGPT as osteomyelitis; Chondrosarcoma was misdiagnosed as fibrous dysplasia or aneurysmal bone cyst in 8 cases. Four cases of spinal chordoma were misdiagnosed as spinal tuberculosis.

Conclusion

Our findings highlight the potential of ChatGPT in the diagnosis of benign and malignant bone tumors, offering advantages like enhanced efficiency and a reduction in missed diagnoses. However, the necessity of collaborative interactions between physicians and ChatGPT in practical settings was underscored. With an examination into AI's capacity in benign and malignant bone tumors diagnosis, this study lays the groundwork for future AI advancements in medicine. Additionally, the benefits of few-shot learning in fine-tuning ChatGPT applications in specialized fields were also demonstrated.

A Patient Similarity Network (CHDmap) to Predict Outcomes After Congenital Heart Surgery: Development and Validation Study

Background

Although evidence-based medicine proposes personalized care that considers the best evidence, it still fails to address personal treatment in many real clinical scenarios where the complexity of the situation makes none of the available evidence applicable. "Medicine-based evidence" (MBE), in which big data and machine learning techniques are embraced to derive treatment responses from appropriately matched patients in real-world clinical practice, was proposed. However, many challenges remain in translating this conceptual framework into practice.

Objective

This study aimed to technically translate the MBE conceptual framework into practice and evaluate its performance in providing general decision support services for outcomes after congenital heart disease (CHD) surgery.

Methods

Data from 4774 CHD surgeries were collected. A total of 66 indicators and all diagnoses were extracted from each echocardiographic report using natural language processing technology. Combined with some basic clinical and surgical information, the distances between each patient were measured by a series of calculation formulas. Inspired by structure-mapping theory, the fusion of distances between different dimensions can be modulated by clinical experts. In addition to supporting direct analogical reasoning, a machine learning model can be constructed based on similar patients to provide personalized prediction. A user-operable patient similarity network (PSN) of CHD called CHDmap was proposed and developed to provide general decision support services based on the MBE approach.

Results

Using 256 CHD cases, CHDmap was evaluated on 2 different types of postoperative prognostic prediction tasks: a binary classification task to predict postoperative complications and a multiple classification task to predict mechanical ventilation duration. A simple poll of the k-most similar patients provided by the PSN can achieve better prediction results than the average performance of 3 clinicians. Constructing logistic regression models for prediction using similar patients obtained from the PSN can further improve the performance of the 2 tasks (best area under the receiver operating characteristic curve=0.810 and 0.926, respectively). With the support of CHDmap, clinicians substantially improved their predictive capabilities.

Conclusions

Without individual optimization, CHDmap demonstrates competitive performance compared to clinical experts. In addition, CHDmap has the advantage of enabling clinicians to use their superior cognitive abilities in conjunction with it to make decisions that are sometimes even superior to those made using artificial intelligence models. The MBE approach can be embraced in clinical practice, and its full potential can be realized.

Antimicrobial learning systems: an implementation blueprint for artificial intelligence to tackle antimicrobial resistance

The proliferation of various forms of artificial intelligence (AI) brings many opportunities to improve health care. AI models can harness complex evolving data, inform and augment human actions, and learn from health outcomes such as morbidity and mortality. The global public health challenge of antimicrobial resistance (AMR) needs large-scale optimisation of antimicrobial use and wider infection care, which could be enabled by carefully constructed AI models. As AI models become increasingly useful and robust, health-care systems remain challenging places for their deployment. An implementation gap exists between the promise of AI models and their use in patient and population care. Here, we outline an adaptive implementation and maintenance framework for AI models to improve antimicrobial use and infection care as a learning system. The roles of AMR problem identification, law and regulation, organisational support, data processing, and AI development, assessment, maintenance, and scalability in the implementation of AMR-targeted AI models are considered.

Validity of the Espiro Mobile Application in the Interpretation of Spirometric Patterns: An App Accuracy Study

Spirometry is a pulmonary function test where correct interpretation of the results is crucial for accurate diagnosis of disease. There are online tools to assist in the interpretation of spirometry results; however, as yet none are validated. We evaluated the interpretation accuracy of the Espiro app using pulmonologist interpretations as the gold standard. This is an observational descriptive study in which 118 spirometry results were interpreted by the Espiro app, two pulmonologists, two primary care physicians, and two residents of a primary care training program. We determined the interpretation accuracy of the Espiro app and the concordance of the pattern and severity interpretation between the Espiro app and each of the observers using Cohen's kappa coefficient (k). We obtained a sensitivity and specificity for the Espiro app of 97.5% (95% confidence interval (CI): 86.8-99.9%) and 94.9% (95%CI: 87.4-98.6%) with pulmonologist 1 and 100% (95%CI: 91.6-100%) and 98.7% (95%CI: 92.9-99.9%) with pulmonologist 2. The concordance for the pattern interpretation was greater than k 0.907, representing almost perfect agreement. The concordance of the severity interpretation was greater than k 0.807, representing substantial to almost perfect agreement. We concluded that the Espiro app is a valid tool for spirometry interpretation.

Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach

OBJECTIVES

To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design.

MATERIALS AND METHODS

Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements.

RESULTS

Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist.

CONCLUSION

User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

Early experiences of integrating an artificial intelligence-based diagnostic decision support system into radiology settings: a qualitative study

OBJECTIVES

Artificial intelligence (AI)-based clinical decision support systems to aid diagnosis are increasingly being developed and implemented but with limited understanding of how such systems integrate with existing clinical work and organizational practices. We explored the early experiences of stakeholders using an AI-based imaging software tool Veye Lung Nodules (VLN) aiding the detection, classification, and measurement of pulmonary nodules in computed tomography scans of the chest.

MATERIALS AND METHODS

We performed semistructured interviews and observations across early adopter deployment sites with clinicians, strategic decision-makers, suppliers, patients with long-term chest conditions, and academics with expertise in the use of diagnostic AI in radiology settings. We coded the data using the Technology, People, Organizations, and Macroenvironmental factors framework.

RESULTS

We conducted 39 interviews. Clinicians reported VLN to be easy to use with little disruption to the workflow. There were differences in patterns of use between experts and novice users with experts critically evaluating system recommendations and actively compensating for system limitations to achieve more reliable performance. Patients also viewed the tool positively. There were contextual variations in tool performance and use between different hospital sites and different use cases. Implementation challenges included integration with existing information systems, data protection, and perceived issues surrounding wider and sustained adoption, including procurement costs.

DISCUSSION

Tool performance was variable, affected by integration into workflows and divisions of labor and knowledge, as well as technical configuration and infrastructure.

CONCLUSION

The socio-organizational factors affecting performance of diagnostic AI are under-researched and require attention and further research.

Development and Evaluation of a Digital Scribe: Conversation Summarization Pipeline for Emergency Department Counseling Sessions towards Reducing Documentation Burden

Objective

We present a proof-of-concept digital scribe system as an ED clinical conversation summarization pipeline and report its performance.

Materials and Methods

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

Results

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

Discussion

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

Conclusion

The study provides evidence towards the potential of AI-assisted tools in reducing clinical documentation burden. Future work is suggested on expanding the research scope with larger language models, and comparative analysis to measure documentation efforts and time.

Digital health interventions in dermatology-Mapping technology and study parameters of systematically identified publications

Digital health interventions (DHI) potentially improve the efficiency and effectiveness of dermatological care. Currently, an overview clustering and characterizing the evidence on DHIs is missing. This systematic mapping of the literature aims to analyse published research on DHIs in dermatology to identify trends and gaps in research. For this purpose, a systematic search of the MEDLINE database was conducted in August 2022 to identify original publications on DHIs in dermatology. Data on country, targeted audience, DHI category, indication, outcome parameter and study design were extracted. Out of 12,009 records identified in MEDLINE, 403 studies were included in the final analysis. Studies on DHIs mainly performed in western countries, headed by the United States (n = 133), Germany (n = 32) and Spain (n = 23). Of all identified DHIs, 261 targeted healthcare providers (HCP), 66 clients (e.g. patients, caregivers, healthy individuals) and 67 both clients and HCPs. A majority of DHIs focussed on establishing a diagnosis (n = 254). Every other study analysed store-and-forward teledermatology (n = 187), followed by artificial intelligence applications for image analysis (n = 65). The most often analysed DHI category for clients was a support of health behaviour change (n = 31). Monitoring of clients was targeted by 77 studies. Skin cancer (n = 148), wounds (n = 29) and psoriasis (n = 29) were the most targeted indications by DHIs. Most studies analysed diagnostic performance (n = 166), fewer studies analysed acceptance (n = 92) and effectiveness (n = 98). Usability (n = 32) and efficiency (n = 36) were investigated only to a small extent. Studies on DHIs in dermatology have focused on teledermatology and AI applications, with an emphasis on skin cancer diagnosis. Apart from that, a range of DHIs for different user groups, purposes and indications were identified, demonstrating the broad potential for DHIs in dermatology. Further research with a wider set of outcome parameters is needed to fully understand the potential of DHIs and ensure their sustainable implementation into dermatological care.

Analysis of Devices Authorized by the FDA for Clinical Decision Support in Critical Care

This case series study examines the clinical evidence cited for US Food and Drug Administration–approved clinical decision support devices for use in the critical care setting.

The Ethics of Risk Prediction for Psychosis and Suicide Attempt in Youth Mental Health

OBJECTIVE

To identify potential clinical utility of polygenic risk scores (PRS) and exposomic risk scores (ERS) for psychosis and suicide attempt in youth and assess the ethical implications of these tools.

STUDY DESIGN

We conducted a narrative literature review of emerging findings on PRS and ERS for suicide and psychosis as well as a literature review on the ethics of PRS. We discuss the ethical implications of the emerging findings for the clinical potential of PRS and ERS.

RESULTS

Emerging evidence suggests that PRS and ERS may offer clinical utility in the relatively near future but that this utility will be limited to specific, narrow clinical questions, in contrast to the suggestion that population-level screening will have sweeping impact. Combining PRS and ERS might optimize prediction. This clinical utility would change the risk-benefit balance of PRS, and further empirical assessment of proposed risks would be necessary. Some concerns for PRS, such as those about counseling, privacy, and inequities, apply to ERS. ERS raise distinct ethical challenges as well, including some that involve informed consent and direct-to-consumer advertising. Both raise questions about the ethics of machine-learning/artificial intelligence approaches.

CONCLUSIONS

Predictive analytics using PRS and ERS may soon play a role in youth mental health settings. Our findings help educate clinicians about potential capabilities, limitations, and ethical implications of these tools. We suggest that a broader discussion with the public is needed to avoid overenthusiasm and determine regulations and guidelines for use of predictive scores.

Evaluation of clinical decision support systems in oncology: An updated systematic review

With increasing reliance on technology in oncology, the impact of digital clinical decision support (CDS) tools needs to be examined. A systematic review update was conducted and peer-reviewed literature from 2016 to 2022 were included if CDS tools were used for live decision making and comparatively assessed quantitative outcomes. 3369 studies were screened and 19 were included in this updated review. Combined with a previous review of 24 studies, a total of 43 studies were analyzed. Improvements in outcomes were observed in 42 studies, and 34 of these were of statistical significance. Computerized physician order entry and clinical practice guideline systems comprise the greatest number of evaluated CDS tools (13 and 10 respectively), followed by those that utilize patient-reported outcomes (8), clinical pathway systems (8) and prescriber alerts for best-practice advisories (4). Our review indicates that CDS can improve guideline adherence, patient-centered care, and care delivery processes in oncology.

Harnessing the power of clinical decision support systems: challenges and opportunities

Clinical decision support systems (CDSSs) are increasingly integrated into healthcare settings to improve patient outcomes, reduce medical errors and enhance clinical efficiency by providing clinicians with evidence-based recommendations at the point of care. However, the adoption and optimisation of these systems remain a challenge. This review aims to provide an overview of the current state of CDSS, discussing their development, implementation, benefits, limitations and future directions. We also explore the potential for enhancing their effectiveness and provide an outlook for future developments in this field. There are several challenges in CDSS implementation, including data privacy concerns, system integration and clinician acceptance. While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge.

Machine-learning assisted swallowing assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia

Introduction

Post-stroke dysphagia is common and associated with significant morbidity and mortality, rendering bedside screening of significant clinical importance. Using voice as a biomarker coupled with deep learning has the potential to improve patient access to screening and mitigate the subjectivity associated with detecting voice change, a component of several validated screening protocols.

Methods

In this single-center study, we developed a proof-of-concept model for automated dysphagia screening and evaluated the performance of this model on training and testing cohorts. Patients were admitted to a comprehensive stroke center, where primary English speakers could follow commands without significant aphasia and participated on a rolling basis. The primary outcome was classification either as a pass or fail equivalent using a dysphagia screening test as a label. Voice data was recorded from patients who spoke a standardized set of vowels, words, and sentences from the National Institute of Health Stroke Scale. Seventy patients were recruited and 68 were included in the analysis, with 40 in training and 28 in testing cohorts, respectively. Speech from patients was segmented into 1,579 audio clips, from which 6,655 Mel-spectrogram images were computed and used as inputs for deep-learning models (DenseNet and ConvNext, separately and together). Clip-level and participant-level swallowing status predictions were obtained through a voting method.

Results

The models demonstrated clip-level dysphagia screening sensitivity of 71% and specificity of 77% (F1 = 0.73, AUC = 0.80 [95% CI: 0.78-0.82]). At the participant level, the sensitivity and specificity were 89 and 79%, respectively (F1 = 0.81, AUC = 0.91 [95% CI: 0.77-1.05]).

Discussion

This study is the first to demonstrate the feasibility of applying deep learning to classify vocalizations to detect post-stroke dysphagia. Our findings suggest potential for enhancing dysphagia screening in clinical settings. https://github.com/UofTNeurology/masa-open-source.

Diagnostic suspicion bias and machine learning: Breaking the awareness deadlock for sepsis detection

Many early warning algorithms are downstream of clinical evaluation and diagnostic testing, which means that they may not be useful when clinicians fail to suspect illness and fail to order appropriate tests. Depending on how such algorithms handle missing data, they could even indicate "low risk" simply because the testing data were never ordered. We considered predictive methodologies to identify sepsis at triage, before diagnostic tests are ordered, in a busy Emergency Department (ED). One algorithm used "bland clinical data" (data available at triage for nearly every patient). The second algorithm added three yes/no questions to be answered after the triage interview. Retrospectively, we studied adult patients from a single ED between 2014-16, separated into training (70%) and testing (30%) cohorts, and a final validation cohort of patients from four EDs between 2016-2018. Sepsis was defined per the Rhee criteria. Investigational predictors were demographics and triage vital signs (downloaded from the hospital EMR); past medical history; and the auxiliary queries (answered by chart reviewers who were blinded to all data except the triage note and initial HPI). We developed L2-regularized logistic regression models using a greedy forward feature selection. There were 1164, 499, and 784 patients in the training, testing, and validation cohorts, respectively. The bland clinical data model yielded ROC AUC's 0.78 (0.76-0.81) and 0.77 (0.73-0.81), for training and testing, respectively, and ranged from 0.74-0.79 in four hospital validation. The second model which included auxiliary queries yielded 0.84 (0.82-0.87) and 0.83 (0.79-0.86), and ranged from 0.78-0.83 in four hospital validation. The first algorithm did not require clinician input but yielded middling performance. The second showed a trend towards superior performance, though required additional user effort. These methods are alternatives to predictive algorithms downstream of clinical evaluation and diagnostic testing. For hospital early warning algorithms, consideration should be given to bias and usability of various methods.

Humans inherit artificial intelligence biases

Artificial intelligence recommendations are sometimes erroneous and biased. In our research, we hypothesized that people who perform a (simulated) medical diagnostic task assisted by a biased AI system will reproduce the model's bias in their own decisions, even when they move to a context without AI support. In three experiments, participants completed a medical-themed classification task with or without the help of a biased AI system. The biased recommendations by the AI influenced participants' decisions. Moreover, when those participants, assisted by the AI, moved on to perform the task without assistance, they made the same errors as the AI had made during the previous phase. Thus, participants' responses mimicked AI bias even when the AI was no longer making suggestions. These results provide evidence of human inheritance of AI bias.

Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach

OBJECTIVE

To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators.

DESIGN, SETTING AND PARTICIPANTS

This study was conducted in Alberta, Canada during 2018-2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded.

EXPOSURE

Each BZRA dispensation from a community pharmacy served as the unit of analysis.

MAIN OUTCOMES AND MEASURES

ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data.

RESULTS

65,063 study participants were included which represented 633,333 BZRA dispensation during 2018-2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance.

CONCLUSION

Developing ML models using only administrative health data may not provide health regulators with sufficient informative predictions to use as decision aids for potential interventions, especially if considering daily or quarterly classifications of BZRA risks in older adults. ML models may be informative for this context if yearly classifications are preferred. Health regulators should have access to other types of data to improve ML prediction.

Sensitivity of an AI method for [18F]FDG PET/CT outcome prediction of diffuse large B-cell lymphoma patients to image reconstruction protocols

BACKGROUND

Convolutional neural networks (CNNs), applied to baseline [18F]-FDG PET/CT maximum intensity projections (MIPs), show potential for treatment outcome prediction in diffuse large B-cell lymphoma (DLBCL). The aim of this study is to investigate the robustness of CNN predictions to different image reconstruction protocols. Baseline [18F]FDG PET/CT scans were collected from 20 DLBCL patients. EARL1, EARL2 and high-resolution (HR) protocols were applied per scan, generating three images with different image qualities. Image-based transformation was applied by blurring EARL2 and HR images to generate EARL1 compliant images using a Gaussian filter of 5 and 7 mm, respectively. MIPs were generated for each of the reconstructions, before and after image transformation. An in-house developed CNN predicted the probability of tumor progression within 2 years for each MIP. The difference in probabilities per patient was then calculated between both EARL2 and HR with respect to EARL1 (delta probabilities or ΔP). We compared these to the probabilities obtained after aligning the data with ComBat using the difference in median and interquartile range (IQR).

RESULTS

CNN probabilities were found to be sensitive to different reconstruction protocols (EARL2 ΔP: median = 0.09, interquartile range (IQR) = [0.06, 0.10] and HR ΔP: median = 0.1, IQR = [0.08, 0.16]). Moreover, higher resolution images (EARL2 and HR) led to higher probability values. After image-based and ComBat transformation, an improved agreement of CNN probabilities among reconstructions was found for all patients. This agreement was slightly better after image-based transformation (transformed EARL2 ΔP: median = 0.022, IQR = [0.01, 0.02] and transformed HR ΔP: median = 0.029, IQR = [0.01, 0.03]).

CONCLUSION

Our CNN-based outcome predictions are affected by the applied reconstruction protocols, yet in a predictable manner. Image-based harmonization is a suitable approach to harmonize CNN predictions across image reconstruction protocols.

Revolutionizing Cancer Research: The Impact of Artificial Intelligence in Digital Biobanking

BACKGROUND

Biobanks are vital research infrastructures aiming to collect, process, store, and distribute biological specimens along with associated data in an organized and governed manner. Exploiting diverse datasets produced by the biobanks and the downstream research from various sources and integrating bioinformatics and "omics" data has proven instrumental in advancing research such as cancer research. Biobanks offer different types of biological samples matched with rich datasets comprising clinicopathologic information. As digital pathology and artificial intelligence (AI) have entered the precision medicine arena, biobanks are progressively transitioning from mere biorepositories to integrated computational databanks. Consequently, the application of AI and machine learning on these biobank datasets holds huge potential to profoundly impact cancer research.

METHODS

In this paper, we explore how AI and machine learning can respond to the digital evolution of biobanks with flexibility, solutions, and effective services. We look at the different data that ranges from specimen-related data, including digital images, patient health records and downstream genetic/genomic data and resulting "Big Data" and the analytic approaches used for analysis.

RESULTS

These cutting-edge technologies can address the challenges faced by translational and clinical research, enhancing their capabilities in data management, analysis, and interpretation. By leveraging AI, biobanks can unlock valuable insights from their vast repositories, enabling the identification of novel biomarkers, prediction of treatment responses, and ultimately facilitating the development of personalized cancer therapies.

CONCLUSIONS

The integration of biobanking with AI has the potential not only to expand the current understanding of cancer biology but also to pave the way for more precise, patient-centric healthcare strategies.

Perceptions of Artificial Intelligence-Assisted Care for Children With a Respiratory Complaint

OBJECTIVES

To evaluate caregiver opinions on the use of artificial intelligence (AI)-assisted medical decision-making for children with a respiratory complaint in the emergency department (ED).

METHODS

We surveyed a sample of caregivers of children presenting to a pediatric ED with a respiratory complaint. We assessed caregiver opinions with respect to AI, defined as "specialized computer programs" that "help make decisions about the best way to care for children." We performed multivariable logistic regression to identify factors associated with discomfort with AI-assisted decision-making.

RESULTS

Of 279 caregivers who were approached, 254 (91.0%) participated. Most indicated they would want to know if AI was being used for their child's health care (93.5%) and were extremely or somewhat comfortable with the use of AI in deciding the need for blood (87.9%) and viral testing (87.6%), interpreting chest radiography (84.6%), and determining need for hospitalization (78.9%). In multivariable analysis, caregiver age of 30 to 37 years (adjusted odds ratio [aOR] 3.67, 95% confidence interval [CI] 1.43-9.38; relative to 18-29 years) and a diagnosis of bronchospasm (aOR 5.77, 95% CI 1.24-30.28 relative to asthma) were associated with greater discomfort with AI. Caregivers with children being admitted to the hospital (aOR 0.23, 95% CI 0.09-0.50) had less discomfort with AI.

CONCLUSIONS

Caregivers were receptive toward the use of AI-assisted decision-making. Some subgroups (caregivers aged 30-37 years with children discharged from the ED) demonstrated greater discomfort with AI. Engaging with these subgroups should be considered when developing AI applications for acute care.

lab: an R package for generating analysis-ready data from laboratory records

Background

Electronic health records (EHRs) play a crucial role in healthcare decision-making by giving physicians insights into disease progression and suitable treatment options. Within EHRs, laboratory test results are frequently utilized for predicting disease progression. However, processing laboratory test results often poses challenges due to variations in units and formats. In addition, leveraging the temporal information in EHRs can improve outcomes, prognoses, and diagnosis predication. Nevertheless, the irregular frequency of the data in these records necessitates data preprocessing, which can add complexity to time-series analyses.

Methods

To address these challenges, we developed an open-source R package that facilitates the extraction of temporal information from laboratory records. The proposed lab package generates analysis-ready time series data by segmenting the data into time-series windows and imputing missing values. Moreover, users can map local laboratory codes to the Logical Observation Identifier Names and Codes (LOINC), an international standard. This mapping allows users to incorporate additional information, such as reference ranges and related diseases. Moreover, the reference ranges provided by LOINC enable us to categorize results into normal or abnormal. Finally, the analysis-ready time series data can be further summarized using descriptive statistics and utilized to develop models using machine learning technologies.

Results

Using the lab package, we analyzed data from MIMIC-III, focusing on newborns with patent ductus arteriosus (PDA). We extracted time-series laboratory records and compared the differences in test results between patients with and without 30-day in-hospital mortality. We then identified significant variations in several laboratory test results 7 days after PDA diagnosis. Leveraging the time series-analysis-ready data, we trained a prediction model with the long short-term memory algorithm, achieving an area under the receiver operating characteristic curve of 0.83 for predicting 30-day in-hospital mortality in model training. These findings demonstrate the lab package's effectiveness in analyzing disease progression.

Conclusions

The proposed lab package simplifies and expedites the workflow involved in laboratory records extraction. This tool is particularly valuable in assisting clinical data analysts in overcoming the obstacles associated with heterogeneous and sparse laboratory records.

Predicting falls-related admissions in older adults in Alberta, Canada: a machine-learning falls prevention tool developed using population administrative health data

OBJECTIVE

To construct a machine-learning (ML) model for health systems with organised falls prevention programmes to identify older adults at risk for fall-related admissions.

DESIGN

This prognostic study used population-level administrative health data to develop an ML prediction model.

SETTING

This study took place in Alberta, Canada during 2018-2019.

PARTICIPANTS

Albertans aged 65 and older with at least one prior admission. Those with palliative conditions or emigrated out of Alberta were excluded.

EXPOSURE

Unit of analysis was the individual person.

MAIN OUTCOMES/MEASURES

We identified fall-related admissions. A CatBoost model was developed on 2018 data to predict risk of fall-related emergency department visits or hospitalisations. Temporal validation was done using 2019 data to evaluate model performance. We reported discrimination, calibration and other relevant metrics measured at the end of 2019 on both ranked predictions and predicted probability thresholds. A cost-savings simulation was performed using 2019 data.

RESULTS

Final number of study participants was 224 445. The validation set had 203 584 participants with 19 389 fall-related events (9.5% pretest probability) and an ML model c-statistic of 0.70. The highest ranked predictions had post-test probabilities ranging from 40% to 50%. Net benefit analysis presented mixed results with some net benefit using the ML model in the 6%-30% range. The top 50 percentile of predicted risks represented nearly $C60 million in health system costs related to falls. Intervening on the top 25 or 50 percentiles of predicted risk could realise substantial (up to $C16 million) savings.

CONCLUSION

ML prediction models based on population-level administrative data can assist health systems with fall prevention programmes identify older adults at risk of fall-related admissions and reduce costs. ML predictions based on ranked predictions or probability thresholds could guide subsequent interventions to mitigate fall risks. Increased access to diverse forms of data could improve ML performance and further reduce costs.

An artificial intelligence method using FDG PET to predict treatment outcome in diffuse large B cell lymphoma patients

Convolutional neural networks (CNNs) may improve response prediction in diffuse large B-cell lymphoma (DLBCL). The aim of this study was to investigate the feasibility of a CNN using maximum intensity projection (MIP) images from 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET) baseline scans to predict the probability of time-to-progression (TTP) within 2 years and compare it with the International Prognostic Index (IPI), i.e. a clinically used score. 296 DLBCL 18F-FDG PET/CT baseline scans collected from a prospective clinical trial (HOVON-84) were analysed. Cross-validation was performed using coronal and sagittal MIPs. An external dataset (340 DLBCL patients) was used to validate the model. Association between the probabilities, metabolic tumour volume and Dmaxbulk was assessed. Probabilities for PET scans with synthetically removed tumors were also assessed. The CNN provided a 2-year TTP prediction with an area under the curve (AUC) of 0.74, outperforming the IPI-based model (AUC = 0.68). Furthermore, high probabilities (> 0.6) of the original MIPs were considerably decreased after removing the tumours (< 0.4, generally). These findings suggest that MIP-based CNNs are able to predict treatment outcome in DLBCL.

Acceptance of a digital therapy recommender system for psoriasis

BACKGROUND

About 2% of the German population are affected by psoriasis. A growing number of cost-intensive systemic treatments are available. Surveys have shown high proportions of patients with moderate to severe psoriasis are not adequately treated despite a high disease burden. Digital therapy recommendation systems (TRS) may help implement guideline-based treatment. However, little is known about the acceptance of such clinical decision support systems (CDSSs). Therefore, the aim of the study was to access the acceptance of a prototypical TRS demonstrator.

METHODS

Three scenarios (potential test patients with psoriasis but different sociodemographic and clinical characteristics, previous treatments, desire to have children, and multiple comorbidities) were designed in the demonstrator. The TRS demonstrator and test patients were presented to a random sample of 76 dermatologists attending a national dermatology conference in a cross-sectional face-to-face survey with case vignettes. The dermatologist were asked to rate the demonstrator by system usability scale (SUS), whether they would use it for certain patients populations and barriers of usage. Reasons for potential usage of the TRS demonstrator were tested via a Poisson regression with robust standard errors.

RESULTS

Acceptance of the TRS was highest for patients eligible for systemic therapy (82%). 50% of participants accepted the system for patients with additional comorbidities and 43% for patients with special subtypes of psoriasis. Dermatologists in the outpatient sector or with many patients per week were less willing to use the TRS for patients with special psoriasis-subtypes. Dermatologists rated the demonstrator as acceptable with an mean SUS of 76.8. Participants whose SUS was 10 points above average were 27% more likely to use TRS for special psoriasis-subtypes. The main barrier in using the TRS was time demand (47.4%). Participants who perceived time as an obstacle were 22.3% less willing to use TRS with systemic therapy patients. 27.6% of physicians stated that they did not understand exactly how the recommendation was generated by the TRS, with no effect on the preparedness to use the system.

CONCLUSION

The considerably high acceptance and the preparedness to use the psoriasis CDSS suggests that a TRS appears to be implementable in routine healthcare and may improve clinical care. Main barrier is the additional time demand posed on dermatologists in a busy clinical setting. Therefore, it will be a major challenge to identify a limited set of variables that still allows a valid recommendation with precise prediction of the patient-individual benefits and harms.