Clinical Decision Support in the Era of Artificial Intelligence

Clinician Perspectives on Decision Support and AI-based Decision Support in a Pediatric ED

BACKGROUND

Clinical decision support (CDS) systems offer the potential to improve pediatric care through enhanced test ordering, prescribing, and standardization of care. Its augmentation with artificial intelligence (AI-CDS) may help address current limitations with CDS implementation regarding alarm fatigue and accuracy of recommendations. We sought to evaluate strengths and perceptions of CDS, with a focus on AI-CDS, through semistructured interviews of clinician partners.

METHODS

We conducted a qualitative study using semistructured interviews of physicians, nurse practitioners, and nurses at a single quaternary-care pediatric emergency department to evaluate clinician perceptions of CDS and AI-CDS. We used reflexive thematic analysis to identify themes and purposive sampling to complete recruitment with the goal of reaching theoretical sufficiency.

RESULTS

We interviewed 20 clinicians. Participants demonstrated a variable understanding of CDS and AI, with some lacking a clear definition. Most recognized the potential benefits of AI-CDS in clinical contexts, such as data summarization and interpretation. Identified themes included the potential of AI-CDS to improve diagnostic accuracy, standardize care, and improve efficiency, while also providing educational benefits to clinicians. Participants raised concerns about the ability of AI-based tools to appreciate nuanced pediatric care, accurately interpret data, and about tensions between AI recommendations and clinician autonomy.

CONCLUSIONS

AI-CDS tools have a promising role in pediatric emergency medicine but require careful integration to address clinicians' concerns about autonomy, nuance recognition, and interpretability. A collaborative approach to development and implementation, informed by clinicians' insights and perspectives, will be pivotal for their successful adoption and efficacy in improving patient care.

Should the Use of Adaptive Machine Learning Systems in Medicine be Classified as Research?

A novel advantage of the use of machine learning (ML) systems in medicine is their potential to continue learning from new data after implementation in clinical practice. To date, considerations of the ethical questions raised by the design and use of adaptive machine learning systems in medicine have, for the most part, been confined to discussion of the so-called "update problem," which concerns how regulators should approach systems whose performance and parameters continue to change even after they have received regulatory approval. In this paper, we draw attention to a prior ethical question: whether the continuous learning that will occur in such systems after their initial deployment should be classified, and regulated, as medical research? We argue that there is a strong prima facie case that the use of continuous learning in medical ML systems should be categorized, and regulated, as research and that individuals whose treatment involves such systems should be treated as research subjects.

Comparative evaluation of interpretation methods in surface-based age prediction for neonates

Significant changes in brain morphology occur during the third trimester of gestation. The capability of deep learning in leveraging these morphological features has enhanced the accuracy of brain age predictions for this critical period. Yet, the opaque nature of deep learning techniques, often described as "black box" approaches, limits their interpretability, posing challenges in clinical applications. Traditional interpretable methods developed for computer vision and natural language processing may not directly translate to the distinct demands of neuroimaging. In response, our research evaluates the effectiveness and adaptability of two interpretative methods-regional age prediction and the perturbation-based saliency map approach-for predicting the brain age of neonates. Analyzing 664 T1 MRI scans with the NEOCIVET pipeline to extract brain surface and cortical features, we assess how these methods illuminate key brain regions for age prediction, focusing on technical analysis with clinical insight. Through a comparative analysis of the saliency index (SI) with relative brain age (RBA) and the examination of structural covariance networks, we uncover the saliency index's enhanced ability to pinpoint regions vital for accurate indication of clinical factors. Our results highlight the advantages of perturbation techniques in addressing the complexities of medical data, steering clinical interventions for premature neonates towards more personalized and interpretable approaches. This study not only reveals the promise of these methods in complex medical scenarios but also offers a blueprint for implementing more interpretable and clinically relevant deep learning models in healthcare settings.

Artificial Intelligence-Augmented Clinical Decision Support Systems for Pregnancy Care: Systematic Review

BACKGROUND

Despite the emerging application of clinical decision support systems (CDSS) in pregnancy care and the proliferation of artificial intelligence (AI) over the last decade, it remains understudied regarding the role of AI in CDSS specialized for pregnancy care.

OBJECTIVE

To identify and synthesize AI-augmented CDSS in pregnancy care, CDSS functionality, AI methodologies, and clinical implementation, we reported a systematic review based on empirical studies that examined AI-augmented CDSS in pregnancy care.

METHODS

We retrieved studies that examined AI-augmented CDSS in pregnancy care using database queries involved with titles, abstracts, keywords, and MeSH (Medical Subject Headings) terms. Bibliographic records from their inception to 2022 were retrieved from PubMed/MEDLINE (n=206), Embase (n=101), and ACM Digital Library (n=377), followed by eligibility screening and literature review. The eligibility criteria include empirical studies that (1) developed or tested AI methods, (2) developed or tested CDSS or CDSS components, and (3) focused on pregnancy care. Data of studies used for review and appraisal include title, abstract, keywords, MeSH terms, full text, and supplements. Publications with ancillary information or overlapping outcomes were synthesized as one single study. Reviewers independently reviewed and assessed the quality of selected studies.

RESULTS

We identified 30 distinct studies of 684 studies from their inception to 2022. Topics of clinical applications covered AI-augmented CDSS from prenatal, early pregnancy, obstetric care, and postpartum care. Topics of CDSS functions include diagnostic support, clinical prediction, therapeutics recommendation, and knowledge base.

CONCLUSIONS

Our review acknowledged recent advances in CDSS studies including early diagnosis of prenatal abnormalities, cost-effective surveillance, prenatal ultrasound support, and ontology development. To recommend future directions, we also noted key gaps from existing studies, including (1) decision support in current childbirth deliveries without using observational data from consequential fetal or maternal outcomes in future pregnancies; (2) scarcity of studies in identifying several high-profile biases from CDSS, including social determinants of health highlighted by the American College of Obstetricians and Gynecologists; and (3) chasm between internally validated CDSS models, external validity, and clinical implementation.

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.

Automated Lugano Metabolic Response Assessment in 18F-Fluorodeoxyglucose-Avid Non-Hodgkin Lymphoma With Deep Learning on 18F-Fluorodeoxyglucose-Positron Emission Tomography

PURPOSE

Artificial intelligence can reduce the time used by physicians on radiological assessments. For 18F-fluorodeoxyglucose-avid lymphomas, obtaining complete metabolic response (CMR) by end of treatment is prognostic.

METHODS

Here, we present a deep learning-based algorithm for fully automated treatment response assessments according to the Lugano 2014 classification. The proposed four-stage method, trained on a multicountry clinical trial (ClinicalTrials.gov identifier: NCT01287741) and tested in three independent multicenter and multicountry test sets on different non-Hodgkin lymphoma subtypes and different lines of treatment (ClinicalTrials.gov identifiers NCT02257567, NCT02500407; 20% holdout in ClinicalTrials.gov identifier NCT01287741), outputs the detected lesions at baseline and follow-up to enable focused radiologist review.

RESULTS

The method's response assessment achieved high agreement with the adjudicated radiologic responses (eg, agreement for overall response assessment of 93%, 87%, and 85% in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, respectively) similar to inter-radiologist agreement and was strongly prognostic of outcomes with a trend toward higher accuracy for death risk than adjudicated radiologic responses (hazard ratio for end of treatment by-model CMR of 0.123, 0.054, and 0.205 in ClinicalTrials.gov identifiers NCT01287741, NCT02500407, and NCT02257567, compared with, respectively, 0.226, 0.292, and 0.272 for CMR by the adjudicated responses). Furthermore, a radiologist review of the algorithm's assessments was conducted. The radiologist median review time was 1.38 minutes/assessment, and no statistically significant differences were observed in the level of agreement of the radiologist with the model's response compared with the level of agreement of the radiologist with the adjudicated responses.

CONCLUSION

These results suggest that the proposed method can be incorporated into radiologic response assessment workflows in cancer imaging for significant time savings and with performance similar to trained medical experts.

Appropriateness of Frequently Asked Patient Questions Following Total Hip Arthroplasty From ChatGPT Compared to Arthroplasty-Trained Nurses

BACKGROUND

The use of ChatGPT (Generative Pretrained Transformer), which is a natural language artificial intelligence model, has gained unparalleled attention with the accumulation of over 100 million users within months of launching. As such, we aimed to compare the following: 1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after total hip arthroplasty; and 2) patients' evaluation of ChatGPT and arthroplasty-trained nurses responses to answer their postoperative questions.

METHODS

We prospectively created 60 questions to address the most commonly asked patient questions following total hip arthroplasty. We obtained answers from arthroplasty-trained nurses and from the ChatGPT-3.5 version for each of the questions. Surgeons graded each set of responses based on clinical judgment as 1) "appropriate," 2) "inappropriate" if the response contained inappropriate information, or 3) "unreliable" if the responses provided inconsistent content. Each patient was given a randomly selected question from the 60 aforementioned questions, with responses provided by ChatGPT and arthroplasty-trained nurses, using a Research Electronic Data Capture survey hosted at our local hospital.

RESULTS

The 3 fellowship-trained surgeons graded 56 out of 60 (93.3%) responses for the arthroplasty-trained nurses and 57 out of 60 (95.0%) for ChatGPT to be "appropriate." There were 175 out of 252 (69.4%) patients who were more comfortable following the ChatGPT responses and 77 out of 252 (30.6%) who preferred arthroplasty-trained nurses' responses. However, 199 out of 252 patients (79.0%) responded that they were "uncertain" with regard to trusting AI to answer their postoperative questions.

CONCLUSIONS

ChatGPT provided appropriate answers from a physician perspective. Patients were also more comfortable with the ChatGPT responses than those from arthroplasty-trained nurses. Inevitably, its successful implementation is dependent on its ability to provide credible information that is consistent with the goals of the physician and patient alike.

Exploring explainable AI features in the vocal biomarkers of lung disease

This review delves into the burgeoning field of explainable artificial intelligence (XAI) in the detection and analysis of lung diseases through vocal biomarkers. Lung diseases, often elusive in their early stages, pose a significant public health challenge. Recent advancements in AI have ushered in innovative methods for early detection, yet the black-box nature of many AI models limits their clinical applicability. XAI emerges as a pivotal tool, enhancing transparency and interpretability in AI-driven diagnostics. This review synthesizes current research on the application of XAI in analyzing vocal biomarkers for lung diseases, highlighting how these techniques elucidate the connections between specific vocal features and lung pathology. We critically examine the methodologies employed, the types of lung diseases studied, and the performance of various XAI models. The potential for XAI to aid in early detection, monitor disease progression, and personalize treatment strategies in pulmonary medicine is emphasized. Furthermore, this review identifies current challenges, including data heterogeneity and model generalizability, and proposes future directions for research. By offering a comprehensive analysis of explainable AI features in the context of lung disease detection, this review aims to bridge the gap between advanced computational approaches and clinical practice, paving the way for more transparent, reliable, and effective diagnostic tools.

The Role of Artificial Intelligence and Machine Learning in Cardiovascular Imaging and Diagnosis

Cardiovascular diseases remain the leading cause of global mortality, underscoring the critical need for accurate and timely diagnosis. This narrative review examines the current applications and future potential of artificial intelligence (AI) and machine learning (ML) in cardiovascular imaging. We discuss the integration of these technologies across various imaging modalities, including echocardiography, computed tomography, magnetic resonance imaging, and nuclear imaging techniques. The review explores AI-assisted diagnosis in key areas such as coronary artery disease detection, valve disorders assessment, cardiomyopathy classification, arrhythmia detection, and prediction of cardiovascular events. AI demonstrates promise in improving diagnostic accuracy, efficiency, and personalized care. However, significant challenges persist, including data quality standardization, model interpretability, regulatory considerations, and clinical workflow integration. We also address the limitations of current AI applications and the ethical implications of their implementation in clinical practice. Future directions point towards advanced AI architectures, multimodal imaging integration, and applications in precision medicine and population health management. The review emphasizes the need for ongoing collaboration between clinicians, data scientists, and policymakers to realize the full potential of AI in cardiovascular imaging while ensuring ethical and equitable implementation. As the field continues to evolve, addressing these challenges will be crucial for the successful integration of AI technologies into cardiovascular care, potentially revolutionizing diagnostic capabilities and improving patient outcomes.

Advancing Chinese biomedical text mining with community challenges

OBJECTIVE

This study aims to review the recent advances in community challenges for biomedical text mining in China.

METHODS

We collected information of evaluation tasks released in community challenges of biomedical text mining, including task description, dataset description, data source, task type and related links. A systematic summary and comparative analysis were conducted on various biomedical natural language processing tasks, such as named entity recognition, entity normalization, attribute extraction, relation extraction, event extraction, text classification, text similarity, knowledge graph construction, question answering, text generation, and large language model evaluation.

RESULTS

We identified 39 evaluation tasks from 6 community challenges that spanned from 2017 to 2023. Our analysis revealed the diverse range of evaluation task types and data sources in biomedical text mining. We explored the potential clinical applications of these community challenge tasks from a translational biomedical informatics perspective. We compared with their English counterparts, and discussed the contributions, limitations, lessons and guidelines of these community challenges, while highlighting future directions in the era of large language models.

CONCLUSION

Community challenge evaluation competitions have played a crucial role in promoting technology innovation and fostering interdisciplinary collaboration in the field of biomedical text mining. These challenges provide valuable platforms for researchers to develop state-of-the-art solutions.

Comparative Analysis of Accuracy, Readability, Sentiment, and Actionability: Artificial Intelligence Chatbots (ChatGPT and Google Gemini) versus Traditional Patient Information Leaflets for Local Anesthesia in Eye Surgery

Background and Aim

Eye surgeries often evoke strong negative emotions in patients, including fear and anxiety. Patient education material plays a crucial role in informing and empowering individuals. Traditional sources of medical information may not effectively address individual patient concerns or cater to varying levels of understanding. This study aims to conduct a comparative analysis of the accuracy, completeness, readability, tone, and understandability of patient education material generated by AI chatbots versus traditional Patient Information Leaflets (PILs), focusing on local anesthesia in eye surgery.

Methods

Expert reviewers evaluated responses generated by AI chatbots (ChatGPT and Google Gemini) and a traditional PIL (Royal College of Anaesthetists' PIL) based on accuracy, completeness, readability, sentiment, and understandability. Statistical analyses, including ANOVA and Tukey HSD tests, were conducted to compare the performance of the sources.

Results

Readability analysis showed variations in complexity among the sources, with AI chatbots offering simplified language and PILs maintaining better overall readability and accessibility. Sentiment analysis revealed differences in emotional tone, with Google Gemini exhibiting the most positive sentiment. AI chatbots demonstrated superior understandability and actionability, while PILs excelled in completeness. Overall, ChatGPT showed slightly higher accuracy (scores expressed as mean ± standard deviation) (4.71 ± 0.5 vs 4.61 ± 0.62) and completeness (4.55 ± 0.58 vs 4.47 ± 0.58) compared to Google Gemini, but PILs performed best (4.84 ± 0.37 vs 4.88 ± 0.33) in terms of both accuracy and completeness (p-value for completeness <0.05).

Conclusion

AI chatbots show promise as innovative tools for patient education, complementing traditional PILs. By leveraging the strengths of both AI-driven technologies and human expertise, healthcare providers can enhance patient education and empower individuals to make informed decisions about their health and medical care.

ChatGPT and retinal disease: a cross-sectional study on AI comprehension of clinical guidelines

OBJECTIVE

To evaluate the performance of an artificial intelligence (AI) large language model, ChatGPT (version 4.0), for common retinal diseases, in accordance with the American Academy of Ophthalmology (AAO) Preferred Practice Pattern (PPP) guidelines.

DESIGN

A cross-sectional survey study design was employed to compare the responses made by ChatGPT to established clinical guidelines.

PARTICIPANTS

Responses by the AI were reviewed by a panel of three vitreoretinal specialists for evaluation.

METHODS

To investigate ChatGPT's comprehension of clinical guidelines, we designed 130 questions covering a broad spectrum of topics within 12 AAO PPP domains of retinal disease These questions were crafted to encompass diagnostic criteria, treatment guidelines, and management strategies, including both medical and surgical aspects of retinal care. A panel of 3 retinal specialists independently evaluated responses on a Likert scale from 1 to 5 based on their relevance, accuracy, and adherence to AAO PPP guidelines. Response readability was evaluated using Flesch Readability Ease and Flesch-Kincaid grade level scores.

RESULTS

ChatGPT achieved an overall average score of 4.9/5.0, suggesting high alignment with the AAO PPP guidelines. Scores varied across domains, with the lowest in the surgical management of disease. The responses had a low reading ease score and required a college-to-graduate level of comprehension. Identified errors were related to diagnostic criteria, treatment options, and methodological procedures.

CONCLUSION

ChatGPT 4.0 demonstrated significant potential in generating guideline-concordant responses, particularly for common medical retinal diseases. However, its performance slightly decreased in surgical retina, highlighting the ongoing need for clinician input, further model refinement, and improved comprehensibility.

Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians

BACKGROUND

ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education.

METHODS

150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses.

RESULTS

ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150).

DISCUSSION

ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.

The Role of Artificial Intelligence in Early Diagnosis and Molecular Classification of Head and Neck Skin Cancers: A Multidisciplinary Approach

Cancer remains a significant global health concern, with increasing genetic and metabolic irregularities linked to its onset. Among various forms of cancer, skin cancer, including squamous cell carcinoma, basal cell carcinoma, and melanoma, is on the rise worldwide, often triggered by ultraviolet (UV) radiation. The propensity of skin cancer to metastasize highlights the importance of early detection for successful treatment. This narrative review explores the evolving role of artificial intelligence (AI) in diagnosing head and neck skin cancers from both radiological and pathological perspectives. In the past two decades, AI has made remarkable progress in skin cancer research, driven by advances in computational capabilities, digitalization of medical images, and radiomics data. AI has shown significant promise in image-based diagnosis across various medical domains. In dermatology, AI has played a pivotal role in refining diagnostic and treatment strategies, including genomic risk assessment. This technology offers substantial potential to aid primary clinicians in improving patient outcomes. Studies have demonstrated AI's effectiveness in identifying skin lesions, categorizing them, and assessing their malignancy, contributing to earlier interventions and better prognosis. The rising incidence and mortality rates of skin cancer, coupled with the high cost of treatment, emphasize the need for early diagnosis. Further research and integration of AI into clinical practice are warranted to maximize its benefits in skin cancer diagnosis and treatment.

Preferences for Management of Pediatric Pneumonia: A Clinician Survey of Artificially Generated Patient Cases

BACKGROUND

It is unknown which factors are associated with chest radiograph (CXR) and antibiotic use for suspected community-acquired pneumonia (CAP) in children. We evaluated factors associated with CXR and antibiotic preferences among clinicians for children with suspected CAP using case scenarios generated through artificial intelligence (AI).

METHODS

We performed a survey of general pediatric, pediatric emergency medicine, and emergency medicine attending physicians employed by a private physician contractor. Respondents were given 5 unique, AI-generated case scenarios. We used generalized estimating equations to identify factors associated with CXR and antibiotic use. We evaluated the cluster-weighted correlation between clinician suspicion and clinical prediction model risk estimates for CAP using 2 predictive models.

RESULTS

A total of 172 respondents provided responses to 839 scenarios. Factors associated with CXR acquisition (OR, [95% CI]) included presence of crackles (4.17 [2.19, 7.95]), prior pneumonia (2.38 [1.32, 4.20]), chest pain (1.90 [1.18, 3.05]) and fever (1.82 [1.32, 2.52]). The decision to use antibiotics before knowledge of CXR results included past hospitalization for pneumonia (4.24 [1.88, 9.57]), focal decreased breath sounds (3.86 [1.98, 7.52]), and crackles (3.45 [2.15, 5.53]). After revealing CXR results to clinicians, these results were the sole predictor associated with antibiotic decision-making. Suspicion for CAP correlated with one of 2 prediction models for CAP (Spearman's rho = 0.25). Factors associated with a greater suspicion of pneumonia included prior pneumonia, duration of illness, worsening course of illness, shortness of breath, vomiting, decreased oral intake or urinary output, respiratory distress, head nodding, focal decreased breath sounds, focal rhonchi, fever, and crackles, and lower pulse oximetry.

CONCLUSIONS

Ordering preferences for CXRs demonstrated similarities and differences with evidence-based risk models for CAP. Clinicians relied heavily on CXR findings to guide antibiotic ordering. These findings can be used within decision support systems to promote evidence-based management practices for pediatric CAP.

Development of a machine learning model for predicting pneumothorax risk in coaxial core needle biopsy (≤3 cm)

PURPOSE

The aim is to devise a machine learning algorithm exploiting preoperative clinical data to forecast the hazard of pneumothorax post-coaxial needle lung biopsy (CCNB), thereby informing clinical decision-making and enhancing perioperative care.

METHOD

This retrospective analysis aggregated clinical and imaging data from patients with lung nodules (≤3 cm) biopsies. Variable selection was done using univariate analysis and LASSO regression, with the dataset subsequently divided into training (80 %) and validation (20 %) subsets. Various machine learning (ML) classifiers were employed in a consolidated approach to ascertain the paramount model, which was followed by individualized risk profiling showcased through Shapley Additive eXplanations (SHAP).

RESULTS

Out of the 325 patients included in the study, 19.6% (64/325) experienced postoperative pneumothorax. High-risk factors determined were Cancer, Lesion_type, GOLD, Size, and Depth. The Gaussian Naive Bayes (GNB) classifier demonstrated superior prediction with an Area Under the Curve (AUC) of 0.82 (95% CI 0.71-0.94), complemented by an accuracy rate of 0.8, sensitivity of 0.71, specificity of 0.84, and an F1 score of 0.61 in the test cohort.

CONCLUSION

The formulated prognostic algorithm exhibited commendable efficacy in preoperatively prognosticating CCNB-induced pneumothorax, harboring the potential to refine personalized risk appraisals, steer clinical judgment, and ameliorate perioperative patient stewardship.

Assessing the Utility, Impact, and Adoption Challenges of an Artificial Intelligence-Enabled Prescription Advisory Tool for Type 2 Diabetes Management: Qualitative Study

BACKGROUND

The clinical management of type 2 diabetes mellitus (T2DM) presents a significant challenge due to the constantly evolving clinical practice guidelines and growing array of drug classes available. Evidence suggests that artificial intelligence (AI)-enabled clinical decision support systems (CDSSs) have proven to be effective in assisting clinicians with informed decision-making. Despite the merits of AI-driven CDSSs, a significant research gap exists concerning the early-stage implementation and adoption of AI-enabled CDSSs in T2DM management.

OBJECTIVE

This study aimed to explore the perspectives of clinicians on the use and impact of the AI-enabled Prescription Advisory (APA) tool, developed using a multi-institution diabetes registry and implemented in specialist endocrinology clinics, and the challenges to its adoption and application.

METHODS

We conducted focus group discussions using a semistructured interview guide with purposively selected endocrinologists from a tertiary hospital. The focus group discussions were audio-recorded and transcribed verbatim. Data were thematically analyzed.

RESULTS

A total of 13 clinicians participated in 4 focus group discussions. Our findings suggest that the APA tool offered several useful features to assist clinicians in effectively managing T2DM. Specifically, clinicians viewed the AI-generated medication alterations as a good knowledge resource in supporting the clinician's decision-making on drug modifications at the point of care, particularly for patients with comorbidities. The complication risk prediction was seen as positively impacting patient care by facilitating early doctor-patient communication and initiating prompt clinical responses. However, the interpretability of the risk scores, concerns about overreliance and automation bias, and issues surrounding accountability and liability hindered the adoption of the APA tool in clinical practice.

CONCLUSIONS

Although the APA tool holds great potential as a valuable resource for improving patient care, further efforts are required to address clinicians' concerns and improve the tool's acceptance and applicability in relevant contexts.

Chinese Oncologists' Perspectives on Integrating AI into Clinical Practice: Cross-Sectional Survey Study

BACKGROUND

The rapid development of artificial intelligence (AI) has brought significant interest to its potential applications in oncology. Although AI-powered tools are already being implemented in some Chinese hospitals, their integration into clinical practice raises several concerns for Chinese oncologists.

OBJECTIVE

This study aims to explore the concerns of Chinese oncologists regarding the integration of AI into clinical practice and to identify the factors influencing these concerns.

METHODS

A total of 228 Chinese oncologists participated in a cross-sectional web-based survey from April to June in 2023 in mainland China. The survey gauged their worries about AI with multiple-choice questions. The survey evaluated their views on the statements of "The impact of AI on the doctor-patient relationship" and "AI will replace doctors." The data were analyzed using descriptive statistics, and variate analyses were used to find correlations between the oncologists' backgrounds and their concerns.

RESULTS

The study revealed that the most prominent concerns were the potential for AI to mislead diagnosis and treatment (163/228, 71.5%); an overreliance on AI (162/228, 71%); data and algorithm bias (123/228, 54%); issues with data security and patient privacy (123/228, 54%); and a lag in the adaptation of laws, regulations, and policies in keeping up with AI's development (115/228, 50.4%). Oncologists with a bachelor's degree expressed heightened concerns related to data and algorithm bias (34/49, 69%; P=.03) and the lagging nature of legal, regulatory, and policy issues (32/49, 65%; P=.046). Regarding AI's impact on doctor-patient relationships, 53.1% (121/228) saw a positive impact, whereas 35.5% (81/228) found it difficult to judge, 9.2% (21/228) feared increased disputes, and 2.2% (5/228) believed that there is no impact. Although sex differences were not significant (P=.08), perceptions varied-male oncologists tended to be more positive than female oncologists (74/135, 54.8% vs 47/93, 50%). Oncologists with a bachelor's degree (26/49, 53%; P=.03) and experienced clinicians (≥21 years; 28/56, 50%; P=.054). found it the hardest to judge. Those with IT experience were significantly more positive (25/35, 71%) than those without (96/193, 49.7%; P=.02). Opinions regarding the possibility of AI replacing doctors were diverse, with 23.2% (53/228) strongly disagreeing, 14% (32/228) disagreeing, 29.8% (68/228) being neutral, 16.2% (37/228) agreeing, and 16.7% (38/228) strongly agreeing. There were no significant correlations with demographic and professional factors (all P>.05).

CONCLUSIONS

Addressing oncologists' concerns about AI requires collaborative efforts from policy makers, developers, health care professionals, and legal experts. Emphasizing transparency, human-centered design, bias mitigation, and education about AI's potential and limitations is crucial. Through close collaboration and a multidisciplinary strategy, AI can be effectively integrated into oncology, balancing benefits with ethical considerations and enhancing patient care.

Computerized decision support is an effective approach to select memory clinic patients for amyloid-PET

BACKGROUND

The use of amyloid-PET in dementia workup is upcoming. At the same time, amyloid-PET is costly and limitedly available. While the appropriate use criteria (AUC) aim for optimal use of amyloid-PET, their limited sensitivity hinders the translation to clinical practice. Therefore, there is a need for tools that guide selection of patients for whom amyloid-PET has the most clinical utility. We aimed to develop a computerized decision support approach to select patients for amyloid-PET.

METHODS

We included 286 subjects (135 controls, 108 Alzheimer's disease dementia, 33 frontotemporal lobe dementia, and 10 vascular dementia) from the Amsterdam Dementia Cohort, with available neuropsychology, APOE, MRI and [18F]florbetaben amyloid-PET. In our computerized decision support approach, using supervised machine learning based on the DSI classifier, we first classified the subjects using only neuropsychology, APOE, and quantified MRI. Then, for subjects with uncertain classification (probability of correct class (PCC) < 0.75) we enriched classification by adding (hypothetical) amyloid positive (AD-like) and negative (normal) PET visual read results and assessed whether the diagnosis became more certain in at least one scenario (PPC≥0.75). If this was the case, the actual visual read result was used in the final classification. We compared the proportion of PET scans and patients diagnosed with sufficient certainty in the computerized approach with three scenarios: 1) without amyloid-PET, 2) amyloid-PET according to the AUC, and 3) amyloid-PET for all patients.

RESULTS

The computerized approach advised PET in n = 60(21%) patients, leading to a diagnosis with sufficient certainty in n = 188(66%) patients. This approach was more efficient than the other three scenarios: 1) without amyloid-PET, diagnostic classification was obtained in n = 155(54%), 2) applying the AUC resulted in amyloid-PET in n = 113(40%) and diagnostic classification in n = 156(55%), and 3) performing amyloid-PET in all resulted in diagnostic classification in n = 154(54%).

CONCLUSION

Our computerized data-driven approach selected 21% of memory clinic patients for amyloid-PET, without compromising diagnostic performance. Our work contributes to a cost-effective implementation and could support clinicians in making a balanced decision in ordering additional amyloid PET during the dementia workup.

How Will Artificial Intelligence Shape the Future of Decision-Making in Congenital Heart Disease?

Improvements in medical technology have significantly changed the management of congenital heart disease (CHD), offering novel tools to predict outcomes and personalize follow-up care. By using sophisticated imaging modalities, computational models and machine learning algorithms, clinicians can experiment with unprecedented insights into the complex anatomy and physiology of CHD. These tools enable early identification of high-risk patients, thus allowing timely, tailored interventions and improved outcomes. Additionally, the integration of genetic testing offers valuable prognostic information, helping in risk stratification and treatment optimisation. The birth of telemedicine platforms and remote monitoring devices facilitates customised follow-up care, enhancing patient engagement and reducing healthcare disparities. Taking into consideration challenges and ethical issues, clinicians can make the most of the full potential of artificial intelligence (AI) to further refine prognostic models, personalize care and improve long-term outcomes for patients with CHD. This narrative review aims to provide a comprehensive illustration of how AI has been implemented as a new technological method for enhancing the management of CHD.

Consumer satisfaction, palliative care and artificial intelligence (AI)

The scope of artificial intelligence (AI) in healthcare is promising, and AI has the potential to revolutionise the field of palliative care services also. Consumer satisfaction in palliative care is a critical aspect of providing high-quality end-of-life support. It encompasses various elements that contribute to a positive experience for both patients and their families. AI-based tools and technologies can help in early identification of the beneficiaries, reduce the cost, improve the quality of care and satisfaction to the patients with chronic life-limiting illnesses. However, it is essential to ensure that AI is used ethically and in a way that complements, rather than replaces, the human touch and compassionate care, which are the core components of palliative care. This article tries to analyse the scope and challenges of improving consumer satisfaction through AI-based technology in palliative care services.

Evaluation of a Digital Scribe: Conversation Summarization for Emergency Department Consultation Calls

OBJECTIVE

We present a proof-of-concept digital scribe system as an Emergency Department (ED) consultation call-based clinical conversation summarization pipeline to support clinical documentation, and report its performance.

MATERIALS AND METHODS

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

RESULTS

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

DISCUSSION

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

CONCLUSION

The study provides evidence towards the potential of AI-assisted tools in assisting clinical documentation. Future work is suggested on expanding the research scope with additional language models and hybrid approaches, and comparative analysis to measure documentation burden and human factors.

Personalized anesthesia and precision medicine: a comprehensive review of genetic factors, artificial intelligence, and patient-specific factors

Precision medicine, characterized by the personalized integration of a patient's genetic blueprint and clinical history, represents a dynamic paradigm in healthcare evolution. The emerging field of personalized anesthesia is at the intersection of genetics and anesthesiology, where anesthetic care will be tailored to an individual's genetic make-up, comorbidities and patient-specific factors. Genomics and biomarkers can provide more accurate anesthetic protocols, while artificial intelligence can simplify anesthetic procedures and reduce anesthetic risks, and real-time monitoring tools can improve perioperative safety and efficacy. The aim of this paper is to present and summarize the applications of these related fields in anesthesiology by reviewing them, exploring the potential of advanced technologies in the implementation and development of personalized anesthesia, realizing the future integration of new technologies into clinical practice, and promoting multidisciplinary collaboration between anesthesiology and disciplines such as genomics and artificial intelligence.

Development and validation of risk prediction model for recurrent cardiovascular events among Chinese: the Personalized CARdiovascular DIsease risk Assessment for Chinese model

Aims

Cardiovascular disease (CVD) is a leading cause of mortality, especially in developing countries. This study aimed to develop and validate a CVD risk prediction model, Personalized CARdiovascular DIsease risk Assessment for Chinese (P-CARDIAC), for recurrent cardiovascular events using machine learning technique.

Methods and results

Three cohorts of Chinese patients with established CVD were included if they had used any of the public healthcare services provided by the Hong Kong Hospital Authority (HA) since 2004 and categorized by their geographical locations. The 10-year CVD outcome was a composite of diagnostic or procedure codes with specific International Classification of Diseases, Ninth Revision, Clinical Modification. Multivariate imputation with chained equations and XGBoost were applied for the model development. The comparison with Thrombolysis in Myocardial Infarction Risk Score for Secondary Prevention (TRS-2°P) and Secondary Manifestations of ARTerial disease (SMART2) used the validation cohorts with 1000 bootstrap replicates. A total of 48 799, 119 672 and 140 533 patients were included in the derivation and validation cohorts, respectively. A list of 125 risk variables were used to make predictions on CVD risk, of which 8 classes of CVD-related drugs were considered interactive covariates. Model performance in the derivation cohort showed satisfying discrimination and calibration with a C statistic of 0.69. Internal validation showed good discrimination and calibration performance with C statistic over 0.6. The P-CARDIAC also showed better performance than TRS-2°P and SMART2.

Conclusion

Compared with other risk scores, the P-CARDIAC enables to identify unique patterns of Chinese patients with established CVD. We anticipate that the P-CARDIAC can be applied in various settings to prevent recurrent CVD events, thus reducing the related healthcare burden.

Applying Machine Learning Techniques to Implementation Science

Machine learning (ML) approaches could expand the usefulness and application of implementation science methods in clinical medicine and public health settings. The aim of this viewpoint is to introduce a roadmap for applying ML techniques to address implementation science questions, such as predicting what will work best, for whom, under what circumstances, and with what predicted level of support, and what and when adaptation or deimplementation are needed. We describe how ML approaches could be used and discuss challenges that implementation scientists and methodologists will need to consider when using ML throughout the stages of implementation.

Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial

OBJECTIVES

We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2).

MATERIALS AND METHODS

A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected.

RESULTS

Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians.

DISCUSSION

Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved.

CONCLUSION

The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.

Clinical decision support system in emergency telephone triage: A scoping review of technical design, implementation and evaluation

OBJECTIVES

Emergency department overcrowding could be improved by upstream telephone triage. Emergency telephone triage aims at managing and orientating adequately patients as early as possible and distributing limited supply of staff and materials. This complex task could be improved with the use of Clinical decision support systems (CDSS). The aim of this scoping review was to identify literature gaps for the future development and evaluation of CDSS for Emergency telephone triage.

MATERIALS AND METHODS

We present here a scoping review of CDSS designed for emergency telephone triage, and compared them in terms of functional characteristics, technical design, health care implementation and methodologies used for evaluation, following the PRISMA-ScR guidelines.

RESULTS

Regarding design, 19 CDSS were retrieved: 12 were knowledge based CDSS (decisional algorithms built according to guidelines or clinical expertise) and 7 were data driven (statistical, machine learning, or deep learning models). Most of them aimed at assisting nurses or non-medical staff by providing patient orientation and/or severity/priority assessment. Eleven were implemented in real life, and only three were connected to the Electronic Health Record. Regarding evaluation, CDSS were assessed through various aspects: intrinsic characteristics, impact on clinical practice or user apprehension. Only one pragmatic trial and one randomized controlled trial were conducted.

CONCLUSION

This review highlights the potential of a hybrid system, user tailored, flexible, connected to the electronic health record, which could work with oral, video and digital data; and the need to evaluate CDSS on intrinsic characteristics and impact on clinical practice, iteratively at each distinct stage of the IT lifecycle.

Global Trends and Hotspots in Nursing Research on Decision Support Systems: A Bibliometric Analysis in CiteSpace

Decision support systems have been widely used in healthcare in recent years; however, there is lack of evidence on global trends and hotspots. This descriptive bibliometric study aimed to analyze bibliometric patterns of decision support systems in nursing. Data were extracted from the Web of Science Core Collection. Published research articles on decision support systems in nursing were identified. Co-occurrence and co-citation analysis was performed using CiteSpace version 6.1.R2. In total, 165 articles were analyzed. A total of 358 authors and 257 institutions from 20 countries contributed to this research field. The most productive authors were Andrew Johnson, Suzanne Bakken, Alessandro Febretti, Eileen S. O'Neill, and Kathryn H. Bowles. The most productive country and institution were the United States and Duke University, respectively. The top 10 keywords were "care," "clinical decision support," "clinical decision support system," "decision support system," "electronic health record," "system," "nursing informatics," "guideline," "decision support," and "outcomes." Common themes on keywords were planning intervention, national health information infrastructure, and methodological challenge. This study will help to find potential partners, countries, and institutions for future researchers, practitioners, and scholars. Additionally, it will contribute to health policy development, evidence-based practice, and further studies for researchers, practitioners, and scholars.

Applications of Artificial Intelligence in Helicopter Emergency Medical Services: A Scoping Review

OBJECTIVE

Recent systematic reviews of acute care medicine applications of artificial intelligence (AI) have focused on hospital and general prehospital uses. The purpose of this scoping review was to identify and describe the literature on AI use with a focus on applications in helicopter emergency medical services (HEMS).

METHODS

A literature search was performed with specific inclusion and exclusion criteria. Articles were grouped by characteristics such as publication year and general subject matter with categoric and temporal trend analyses.

RESULTS

We identified 21 records focused on the use of AI in HEMS. These applications included both clinical and triage uses and nonclinical uses. The earliest study appeared in 2006, but over one third of the identified studies have been published in 2021 or later. The passage of time has seen an increased likelihood of HEMS AI studies focusing on nonclinical issues; for each year, the likelihood of a nonclinical focus had an odds ratio of 1.3.

CONCLUSION

This scoping review provides overview and hypothesis-generating information regarding AI applications specific to HEMS. HEMS AI may be ultimately deployed in nonclinical arenas as much as or more than for clinical decision support. Future studies will inform future decisions as to how AI may improve HEMS systems design, asset deployment, and clinical care.

Evaluating clinical decision support software (CDSS): challenges for robust evidence generation

OBJECTIVES

Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

METHODS

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

RESULTS

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

CONCLUSION

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

A novel score for predicting 1-year mortality of intensive care patients

BACKGROUND

We aimed to develop a simple scoring table for predicting probability of death within 1-year after admission to an intensive care unit. We analysed data on emergency admissions from the nationwide Finnish intensive care quality registry.

METHODS

We included first admissions of adult patients with data available on 1-year vital status (dead or alive) and all five variables included in a premorbid functional status score, which is the number of activities the person can manage independently of the following five: get out of bed, move indoors, dress, climb stairs and walk 400 m. We analysed data on patient characteristics and admission-associated factors from 2012 to 2014 to find predictors of 1-year mortality and to develop a score for predicting probability of death. We tested the performance of this score in data from 2015. We assessed the 1-year functional status score of survivors with data available.

RESULTS

Out of 25,261 patients, 20,628 (81.7%) patients were able to perform all five functional activities independently prior to the intensive care unit admission. At 1-year post admission, 19,625 (77.7%) patients were alive. 1-year functional status score was known for 11,011 patients and 8970 (81.5%) patients achieved functional status score 5, managing all five activities independently. The score based on age, sex, preceding functional status, type of intensive care unit admission, severity of acute illness and the most significant diagnoses predicted 1-year mortality with an area under the receiver operating characteristic curve 0.78 (95% CI, 0.76-0.79). The calibration of our prediction model was good, with calibration intercept -0.01 (-0.07 to 0.05) and calibration slope 0.96 (0.90 to 1.02).

CONCLUSION

Our score based on data available at intensive care unit admission predicted 1-year mortality with fairly good discrimination. Most survivors achieved good functional recovery.

Large-Scale assessment of ChatGPT's performance in benign and malignant bone tumors imaging report diagnosis and its potential for clinical applications

Objective

This study was designed to delve into the complexities involved in diagnosing of benign and malignant bone tumors and to assess the potential of AI technologies like ChatGPT in improving diagnostic accuracy and efficiency. The study also explores the few-shot learning as a method to optimize ChatGPT's performance in specialized medical domains such as benign and malignant bone tumors diagnosis.

Methods

A total of 1366 benign and malignant bone tumors-related imaging reports were collected and diagnosed by 25 experienced physicians. The gold standard of diagnosis was established by combining clinical, imaging and pathological principles.These reports were then input into the ChatGPT model which underwent a few-shot learning method to generate diagnostic results. The diagnostic results of the physicians and the AI model were compared to evaluate the performance of ChatGPT. An experiment was conducted to assess the influence of different radiologist's reporting styles on the model's diagnostic performance. Furthermore, in-depth analysis of misdiagnosed cases was carried out, categorizing diagnostic errors and exploring possible causes.

Results

The diagnostic results generated by ChatGPT showed an accuracy of 0.73, sensitivity of 0.95, and specificity of 0.58. After few-shot learning, ChatGPT demonstrated significant improvement, achieving an accuracy of 0.87, sensitivity of 0.99, and specificity of 0.73, bringing it much closer to the level of physician diagnostics. In an experiment analyzing the influence of the radiologist's reporting style, the model demonstrated higher sensitivity when interpreting reports written by high-level radiologists. In 56 benign cases, ChatGPT misdiagnosed them as malignant. Among these, 35 benign lesions- fibrous dysplasia and osteofibrous dysplasia- were incorrectly identified as metastatic tumors or osteosarcomas; 8 cases of myositis ossificans were wrongly diagnosed as extraosseous osteosarcoma. 7 cases of giant cell tumor of bone at the end of long bone were misdiagnosed as osteosarcoma by intermediate doctors. Chondroblastoma was misdiagnosed as malignant tumor in 6 cases -2 osteosarcoma and 4 chondrosarcoma-In this study, 23 osteosarcoma cases were misdiagnosed by ChatGPT as osteomyelitis; Chondrosarcoma was misdiagnosed as fibrous dysplasia or aneurysmal bone cyst in 8 cases. Four cases of spinal chordoma were misdiagnosed as spinal tuberculosis.

Conclusion

Our findings highlight the potential of ChatGPT in the diagnosis of benign and malignant bone tumors, offering advantages like enhanced efficiency and a reduction in missed diagnoses. However, the necessity of collaborative interactions between physicians and ChatGPT in practical settings was underscored. With an examination into AI's capacity in benign and malignant bone tumors diagnosis, this study lays the groundwork for future AI advancements in medicine. Additionally, the benefits of few-shot learning in fine-tuning ChatGPT applications in specialized fields were also demonstrated.

A Patient Similarity Network (CHDmap) to Predict Outcomes After Congenital Heart Surgery: Development and Validation Study

Background

Although evidence-based medicine proposes personalized care that considers the best evidence, it still fails to address personal treatment in many real clinical scenarios where the complexity of the situation makes none of the available evidence applicable. "Medicine-based evidence" (MBE), in which big data and machine learning techniques are embraced to derive treatment responses from appropriately matched patients in real-world clinical practice, was proposed. However, many challenges remain in translating this conceptual framework into practice.

Objective

This study aimed to technically translate the MBE conceptual framework into practice and evaluate its performance in providing general decision support services for outcomes after congenital heart disease (CHD) surgery.

Methods

Data from 4774 CHD surgeries were collected. A total of 66 indicators and all diagnoses were extracted from each echocardiographic report using natural language processing technology. Combined with some basic clinical and surgical information, the distances between each patient were measured by a series of calculation formulas. Inspired by structure-mapping theory, the fusion of distances between different dimensions can be modulated by clinical experts. In addition to supporting direct analogical reasoning, a machine learning model can be constructed based on similar patients to provide personalized prediction. A user-operable patient similarity network (PSN) of CHD called CHDmap was proposed and developed to provide general decision support services based on the MBE approach.

Results

Using 256 CHD cases, CHDmap was evaluated on 2 different types of postoperative prognostic prediction tasks: a binary classification task to predict postoperative complications and a multiple classification task to predict mechanical ventilation duration. A simple poll of the k-most similar patients provided by the PSN can achieve better prediction results than the average performance of 3 clinicians. Constructing logistic regression models for prediction using similar patients obtained from the PSN can further improve the performance of the 2 tasks (best area under the receiver operating characteristic curve=0.810 and 0.926, respectively). With the support of CHDmap, clinicians substantially improved their predictive capabilities.

Conclusions

Without individual optimization, CHDmap demonstrates competitive performance compared to clinical experts. In addition, CHDmap has the advantage of enabling clinicians to use their superior cognitive abilities in conjunction with it to make decisions that are sometimes even superior to those made using artificial intelligence models. The MBE approach can be embraced in clinical practice, and its full potential can be realized.

Antimicrobial learning systems: an implementation blueprint for artificial intelligence to tackle antimicrobial resistance

The proliferation of various forms of artificial intelligence (AI) brings many opportunities to improve health care. AI models can harness complex evolving data, inform and augment human actions, and learn from health outcomes such as morbidity and mortality. The global public health challenge of antimicrobial resistance (AMR) needs large-scale optimisation of antimicrobial use and wider infection care, which could be enabled by carefully constructed AI models. As AI models become increasingly useful and robust, health-care systems remain challenging places for their deployment. An implementation gap exists between the promise of AI models and their use in patient and population care. Here, we outline an adaptive implementation and maintenance framework for AI models to improve antimicrobial use and infection care as a learning system. The roles of AMR problem identification, law and regulation, organisational support, data processing, and AI development, assessment, maintenance, and scalability in the implementation of AMR-targeted AI models are considered.

Validity of the Espiro Mobile Application in the Interpretation of Spirometric Patterns: An App Accuracy Study

Spirometry is a pulmonary function test where correct interpretation of the results is crucial for accurate diagnosis of disease. There are online tools to assist in the interpretation of spirometry results; however, as yet none are validated. We evaluated the interpretation accuracy of the Espiro app using pulmonologist interpretations as the gold standard. This is an observational descriptive study in which 118 spirometry results were interpreted by the Espiro app, two pulmonologists, two primary care physicians, and two residents of a primary care training program. We determined the interpretation accuracy of the Espiro app and the concordance of the pattern and severity interpretation between the Espiro app and each of the observers using Cohen's kappa coefficient (k). We obtained a sensitivity and specificity for the Espiro app of 97.5% (95% confidence interval (CI): 86.8-99.9%) and 94.9% (95%CI: 87.4-98.6%) with pulmonologist 1 and 100% (95%CI: 91.6-100%) and 98.7% (95%CI: 92.9-99.9%) with pulmonologist 2. The concordance for the pattern interpretation was greater than k 0.907, representing almost perfect agreement. The concordance of the severity interpretation was greater than k 0.807, representing substantial to almost perfect agreement. We concluded that the Espiro app is a valid tool for spirometry interpretation.

Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach

OBJECTIVES

To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design.

MATERIALS AND METHODS

Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements.

RESULTS

Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist.

CONCLUSION

User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

Early experiences of integrating an artificial intelligence-based diagnostic decision support system into radiology settings: a qualitative study

OBJECTIVES

Artificial intelligence (AI)-based clinical decision support systems to aid diagnosis are increasingly being developed and implemented but with limited understanding of how such systems integrate with existing clinical work and organizational practices. We explored the early experiences of stakeholders using an AI-based imaging software tool Veye Lung Nodules (VLN) aiding the detection, classification, and measurement of pulmonary nodules in computed tomography scans of the chest.

MATERIALS AND METHODS

We performed semistructured interviews and observations across early adopter deployment sites with clinicians, strategic decision-makers, suppliers, patients with long-term chest conditions, and academics with expertise in the use of diagnostic AI in radiology settings. We coded the data using the Technology, People, Organizations, and Macroenvironmental factors framework.

RESULTS

We conducted 39 interviews. Clinicians reported VLN to be easy to use with little disruption to the workflow. There were differences in patterns of use between experts and novice users with experts critically evaluating system recommendations and actively compensating for system limitations to achieve more reliable performance. Patients also viewed the tool positively. There were contextual variations in tool performance and use between different hospital sites and different use cases. Implementation challenges included integration with existing information systems, data protection, and perceived issues surrounding wider and sustained adoption, including procurement costs.

DISCUSSION

Tool performance was variable, affected by integration into workflows and divisions of labor and knowledge, as well as technical configuration and infrastructure.

CONCLUSION

The socio-organizational factors affecting performance of diagnostic AI are under-researched and require attention and further research.

Development and Evaluation of a Digital Scribe: Conversation Summarization Pipeline for Emergency Department Counseling Sessions towards Reducing Documentation Burden

Objective

We present a proof-of-concept digital scribe system as an ED clinical conversation summarization pipeline and report its performance.

Materials and Methods

We use four pre-trained large language models to establish the digital scribe system: T5-small, T5-base, PEGASUS-PubMed, and BART-Large-CNN via zero-shot and fine-tuning approaches. Our dataset includes 100 referral conversations among ED clinicians and medical records. We report the ROUGE-1, ROUGE-2, and ROUGE-L to compare model performance. In addition, we annotated transcriptions to assess the quality of generated summaries.

Results

The fine-tuned BART-Large-CNN model demonstrates greater performance in summarization tasks with the highest ROUGE scores (F1ROUGE-1=0.49, F1ROUGE-2=0.23, F1ROUGE-L=0.35) scores. In contrast, PEGASUS-PubMed lags notably (F1ROUGE-1=0.28, F1ROUGE-2=0.11, F1ROUGE-L=0.22). BART-Large-CNN's performance decreases by more than 50% with the zero-shot approach. Annotations show that BART-Large-CNN performs 71.4% recall in identifying key information and a 67.7% accuracy rate.

Discussion

The BART-Large-CNN model demonstrates a high level of understanding of clinical dialogue structure, indicated by its performance with and without fine-tuning. Despite some instances of high recall, there is variability in the model's performance, particularly in achieving consistent correctness, suggesting room for refinement. The model's recall ability varies across different information categories.

Conclusion

The study provides evidence towards the potential of AI-assisted tools in reducing clinical documentation burden. Future work is suggested on expanding the research scope with larger language models, and comparative analysis to measure documentation efforts and time.

Digital health interventions in dermatology-Mapping technology and study parameters of systematically identified publications

Digital health interventions (DHI) potentially improve the efficiency and effectiveness of dermatological care. Currently, an overview clustering and characterizing the evidence on DHIs is missing. This systematic mapping of the literature aims to analyse published research on DHIs in dermatology to identify trends and gaps in research. For this purpose, a systematic search of the MEDLINE database was conducted in August 2022 to identify original publications on DHIs in dermatology. Data on country, targeted audience, DHI category, indication, outcome parameter and study design were extracted. Out of 12,009 records identified in MEDLINE, 403 studies were included in the final analysis. Studies on DHIs mainly performed in western countries, headed by the United States (n = 133), Germany (n = 32) and Spain (n = 23). Of all identified DHIs, 261 targeted healthcare providers (HCP), 66 clients (e.g. patients, caregivers, healthy individuals) and 67 both clients and HCPs. A majority of DHIs focussed on establishing a diagnosis (n = 254). Every other study analysed store-and-forward teledermatology (n = 187), followed by artificial intelligence applications for image analysis (n = 65). The most often analysed DHI category for clients was a support of health behaviour change (n = 31). Monitoring of clients was targeted by 77 studies. Skin cancer (n = 148), wounds (n = 29) and psoriasis (n = 29) were the most targeted indications by DHIs. Most studies analysed diagnostic performance (n = 166), fewer studies analysed acceptance (n = 92) and effectiveness (n = 98). Usability (n = 32) and efficiency (n = 36) were investigated only to a small extent. Studies on DHIs in dermatology have focused on teledermatology and AI applications, with an emphasis on skin cancer diagnosis. Apart from that, a range of DHIs for different user groups, purposes and indications were identified, demonstrating the broad potential for DHIs in dermatology. Further research with a wider set of outcome parameters is needed to fully understand the potential of DHIs and ensure their sustainable implementation into dermatological care.

Analysis of Devices Authorized by the FDA for Clinical Decision Support in Critical Care

This case series study examines the clinical evidence cited for US Food and Drug Administration–approved clinical decision support devices for use in the critical care setting.

The Ethics of Risk Prediction for Psychosis and Suicide Attempt in Youth Mental Health

OBJECTIVE

To identify potential clinical utility of polygenic risk scores (PRS) and exposomic risk scores (ERS) for psychosis and suicide attempt in youth and assess the ethical implications of these tools.

STUDY DESIGN

We conducted a narrative literature review of emerging findings on PRS and ERS for suicide and psychosis as well as a literature review on the ethics of PRS. We discuss the ethical implications of the emerging findings for the clinical potential of PRS and ERS.

RESULTS

Emerging evidence suggests that PRS and ERS may offer clinical utility in the relatively near future but that this utility will be limited to specific, narrow clinical questions, in contrast to the suggestion that population-level screening will have sweeping impact. Combining PRS and ERS might optimize prediction. This clinical utility would change the risk-benefit balance of PRS, and further empirical assessment of proposed risks would be necessary. Some concerns for PRS, such as those about counseling, privacy, and inequities, apply to ERS. ERS raise distinct ethical challenges as well, including some that involve informed consent and direct-to-consumer advertising. Both raise questions about the ethics of machine-learning/artificial intelligence approaches.

CONCLUSIONS

Predictive analytics using PRS and ERS may soon play a role in youth mental health settings. Our findings help educate clinicians about potential capabilities, limitations, and ethical implications of these tools. We suggest that a broader discussion with the public is needed to avoid overenthusiasm and determine regulations and guidelines for use of predictive scores.

Evaluation of clinical decision support systems in oncology: An updated systematic review

With increasing reliance on technology in oncology, the impact of digital clinical decision support (CDS) tools needs to be examined. A systematic review update was conducted and peer-reviewed literature from 2016 to 2022 were included if CDS tools were used for live decision making and comparatively assessed quantitative outcomes. 3369 studies were screened and 19 were included in this updated review. Combined with a previous review of 24 studies, a total of 43 studies were analyzed. Improvements in outcomes were observed in 42 studies, and 34 of these were of statistical significance. Computerized physician order entry and clinical practice guideline systems comprise the greatest number of evaluated CDS tools (13 and 10 respectively), followed by those that utilize patient-reported outcomes (8), clinical pathway systems (8) and prescriber alerts for best-practice advisories (4). Our review indicates that CDS can improve guideline adherence, patient-centered care, and care delivery processes in oncology.

Harnessing the power of clinical decision support systems: challenges and opportunities

Clinical decision support systems (CDSSs) are increasingly integrated into healthcare settings to improve patient outcomes, reduce medical errors and enhance clinical efficiency by providing clinicians with evidence-based recommendations at the point of care. However, the adoption and optimisation of these systems remain a challenge. This review aims to provide an overview of the current state of CDSS, discussing their development, implementation, benefits, limitations and future directions. We also explore the potential for enhancing their effectiveness and provide an outlook for future developments in this field. There are several challenges in CDSS implementation, including data privacy concerns, system integration and clinician acceptance. While CDSS have demonstrated significant potential, their adoption and optimisation remain a challenge.

Machine-learning assisted swallowing assessment: a deep learning-based quality improvement tool to screen for post-stroke dysphagia

Introduction

Post-stroke dysphagia is common and associated with significant morbidity and mortality, rendering bedside screening of significant clinical importance. Using voice as a biomarker coupled with deep learning has the potential to improve patient access to screening and mitigate the subjectivity associated with detecting voice change, a component of several validated screening protocols.

Methods

In this single-center study, we developed a proof-of-concept model for automated dysphagia screening and evaluated the performance of this model on training and testing cohorts. Patients were admitted to a comprehensive stroke center, where primary English speakers could follow commands without significant aphasia and participated on a rolling basis. The primary outcome was classification either as a pass or fail equivalent using a dysphagia screening test as a label. Voice data was recorded from patients who spoke a standardized set of vowels, words, and sentences from the National Institute of Health Stroke Scale. Seventy patients were recruited and 68 were included in the analysis, with 40 in training and 28 in testing cohorts, respectively. Speech from patients was segmented into 1,579 audio clips, from which 6,655 Mel-spectrogram images were computed and used as inputs for deep-learning models (DenseNet and ConvNext, separately and together). Clip-level and participant-level swallowing status predictions were obtained through a voting method.

Results

The models demonstrated clip-level dysphagia screening sensitivity of 71% and specificity of 77% (F1 = 0.73, AUC = 0.80 [95% CI: 0.78-0.82]). At the participant level, the sensitivity and specificity were 89 and 79%, respectively (F1 = 0.81, AUC = 0.91 [95% CI: 0.77-1.05]).

Discussion

This study is the first to demonstrate the feasibility of applying deep learning to classify vocalizations to detect post-stroke dysphagia. Our findings suggest potential for enhancing dysphagia screening in clinical settings. https://github.com/UofTNeurology/masa-open-source.

Diagnostic suspicion bias and machine learning: Breaking the awareness deadlock for sepsis detection

Many early warning algorithms are downstream of clinical evaluation and diagnostic testing, which means that they may not be useful when clinicians fail to suspect illness and fail to order appropriate tests. Depending on how such algorithms handle missing data, they could even indicate "low risk" simply because the testing data were never ordered. We considered predictive methodologies to identify sepsis at triage, before diagnostic tests are ordered, in a busy Emergency Department (ED). One algorithm used "bland clinical data" (data available at triage for nearly every patient). The second algorithm added three yes/no questions to be answered after the triage interview. Retrospectively, we studied adult patients from a single ED between 2014-16, separated into training (70%) and testing (30%) cohorts, and a final validation cohort of patients from four EDs between 2016-2018. Sepsis was defined per the Rhee criteria. Investigational predictors were demographics and triage vital signs (downloaded from the hospital EMR); past medical history; and the auxiliary queries (answered by chart reviewers who were blinded to all data except the triage note and initial HPI). We developed L2-regularized logistic regression models using a greedy forward feature selection. There were 1164, 499, and 784 patients in the training, testing, and validation cohorts, respectively. The bland clinical data model yielded ROC AUC's 0.78 (0.76-0.81) and 0.77 (0.73-0.81), for training and testing, respectively, and ranged from 0.74-0.79 in four hospital validation. The second model which included auxiliary queries yielded 0.84 (0.82-0.87) and 0.83 (0.79-0.86), and ranged from 0.78-0.83 in four hospital validation. The first algorithm did not require clinician input but yielded middling performance. The second showed a trend towards superior performance, though required additional user effort. These methods are alternatives to predictive algorithms downstream of clinical evaluation and diagnostic testing. For hospital early warning algorithms, consideration should be given to bias and usability of various methods.

Humans inherit artificial intelligence biases

Artificial intelligence recommendations are sometimes erroneous and biased. In our research, we hypothesized that people who perform a (simulated) medical diagnostic task assisted by a biased AI system will reproduce the model's bias in their own decisions, even when they move to a context without AI support. In three experiments, participants completed a medical-themed classification task with or without the help of a biased AI system. The biased recommendations by the AI influenced participants' decisions. Moreover, when those participants, assisted by the AI, moved on to perform the task without assistance, they made the same errors as the AI had made during the previous phase. Thus, participants' responses mimicked AI bias even when the AI was no longer making suggestions. These results provide evidence of human inheritance of AI bias.

Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach

OBJECTIVE

To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators.

DESIGN, SETTING AND PARTICIPANTS

This study was conducted in Alberta, Canada during 2018-2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded.

EXPOSURE

Each BZRA dispensation from a community pharmacy served as the unit of analysis.

MAIN OUTCOMES AND MEASURES

ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data.

RESULTS

65,063 study participants were included which represented 633,333 BZRA dispensation during 2018-2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance.

CONCLUSION

Developing ML models using only administrative health data may not provide health regulators with sufficient informative predictions to use as decision aids for potential interventions, especially if considering daily or quarterly classifications of BZRA risks in older adults. ML models may be informative for this context if yearly classifications are preferred. Health regulators should have access to other types of data to improve ML prediction.

Sensitivity of an AI method for [18F]FDG PET/CT outcome prediction of diffuse large B-cell lymphoma patients to image reconstruction protocols

BACKGROUND

Convolutional neural networks (CNNs), applied to baseline [18F]-FDG PET/CT maximum intensity projections (MIPs), show potential for treatment outcome prediction in diffuse large B-cell lymphoma (DLBCL). The aim of this study is to investigate the robustness of CNN predictions to different image reconstruction protocols. Baseline [18F]FDG PET/CT scans were collected from 20 DLBCL patients. EARL1, EARL2 and high-resolution (HR) protocols were applied per scan, generating three images with different image qualities. Image-based transformation was applied by blurring EARL2 and HR images to generate EARL1 compliant images using a Gaussian filter of 5 and 7 mm, respectively. MIPs were generated for each of the reconstructions, before and after image transformation. An in-house developed CNN predicted the probability of tumor progression within 2 years for each MIP. The difference in probabilities per patient was then calculated between both EARL2 and HR with respect to EARL1 (delta probabilities or ΔP). We compared these to the probabilities obtained after aligning the data with ComBat using the difference in median and interquartile range (IQR).

RESULTS

CNN probabilities were found to be sensitive to different reconstruction protocols (EARL2 ΔP: median = 0.09, interquartile range (IQR) = [0.06, 0.10] and HR ΔP: median = 0.1, IQR = [0.08, 0.16]). Moreover, higher resolution images (EARL2 and HR) led to higher probability values. After image-based and ComBat transformation, an improved agreement of CNN probabilities among reconstructions was found for all patients. This agreement was slightly better after image-based transformation (transformed EARL2 ΔP: median = 0.022, IQR = [0.01, 0.02] and transformed HR ΔP: median = 0.029, IQR = [0.01, 0.03]).

CONCLUSION

Our CNN-based outcome predictions are affected by the applied reconstruction protocols, yet in a predictable manner. Image-based harmonization is a suitable approach to harmonize CNN predictions across image reconstruction protocols.