Dissecting racial bias in an algorithm used to manage the health of populations

The Impact of Race, Ethnicity, and Sex on Fairness in Artificial Intelligence for Glaucoma Prediction Models

Objective

Despite advances in artificial intelligence (AI) in glaucoma prediction, most works lack multicenter focus and do not consider fairness concerning sex, race, or ethnicity. This study aims to examine the impact of these sensitive attributes on developing fair AI models that predict glaucoma progression to necessitating incisional glaucoma surgery.

Design

Database study.

Participants

Thirty-nine thousand ninety patients with glaucoma, as identified by International Classification of Disease codes from 7 academic eye centers participating in the Sight OUtcomes Research Collaborative.

Methods

We developed XGBoost models using 3 approaches: (1) excluding sensitive attributes as input features, (2) including them explicitly as input features, and (3) training separate models for each group. Model input features included demographic details, diagnosis codes, medications, and clinical information (intraocular pressure, visual acuity, etc.), from electronic health records. The models were trained on patients from 5 sites (N = 27 999) and evaluated on a held-out internal test set (N = 3499) and 2 external test sets consisting of N = 1550 and N = 2542 patients.

Main Outcomes and Measures

Area under the receiver operating characteristic curve (AUROC) and equalized odds on the test set and external sites.

Results

Six thousand six hundred eighty-two (17.1%) of 39 090 patients underwent glaucoma surgery with a mean age of 70.1 (standard deviation 14.6) years, 54.5% female, 62.3% White, 22.1% Black, and 4.7% Latinx/Hispanic. We found that not including the sensitive attributes led to better classification performance (AUROC: 0.77-0.82) but worsened fairness when evaluated on the internal test set. However, on external test sites, the opposite was true: including sensitive attributes resulted in better classification performance (AUROC: external #1 - [0.73-0.81], external #2 - [0.67-0.70]), but varying degrees of fairness for sex and race as measured by equalized odds.

Conclusions

Artificial intelligence models predicting whether patients with glaucoma progress to surgery demonstrated bias with respect to sex, race, and ethnicity. The effect of sensitive attribute inclusion and exclusion on fairness and performance varied based on internal versus external test sets. Prior to deployment, AI models should be evaluated for fairness on the target population.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

From theory to practice: Harmonizing taxonomies of trustworthy AI

The increasing capabilities of AI pose new risks and vulnerabilities for organizations and decision makers. Several trustworthy AI frameworks have been created by U.S. federal agencies and international organizations to outline the principles to which AI systems must adhere for their use to be considered responsible. Different trustworthy AI frameworks reflect the priorities and perspectives of different stakeholders, and there is no consensus on a single framework yet. We evaluate the leading frameworks and provide a holistic perspective on trustworthy AI values, allowing federal agencies to create agency-specific trustworthy AI strategies that account for unique institutional needs and priorities. We apply this approach to the Department of Veterans Affairs, an entity with largest health care system in US. Further, we contextualize our framework from the perspective of the federal government on how to leverage existing trustworthy AI frameworks to develop a set of guiding principles that can provide the foundation for an agency to design, develop, acquire, and use AI systems in a manner that simultaneously fosters trust and confidence and meets the requirements of established laws and regulations.

Advancing patient-centered clinical decision support in today's health care ecosystem: key themes from the Clinical Decision Support Innovation Collaborative's 2023 Annual Meeting

Objective

This perspective summarizes key themes that arose from stakeholder discussions at the inaugural Clinical Decision Support Innovation Collaborative (CDSiC) 2023 Annual Meeting. The CDSiC is an Agency for Healthcare Research and Quality (AHRQ)-funded innovation hub for patient-centered clinical decision support (PC CDS).

Materials and Methods

The meeting took place on May 16-17, 2023, and engaged 73 participants that represented a range of stakeholder groups including researchers, informaticians, federal representatives, clinicians, patients, and electronic health record developers. Each meeting session was recorded and had 2 notetakers. CDSiC leadership analyzed the compiled meeting notes to synthesize key themes.

Results

Participants discussed 7 key opportunities to advance PC CDS: (1) establish feedback loops between patients and clinicians; (2) develop new workflows; (3) expand the evidence base; (4) adapt the CDS Five Rights for the patient perspective; (5) advance health equity; (6) explore perceptions on the use of artificial intelligence; and (7) encourage widespread use and scalability of PC CDS.

Discussion and Conclusion

Innovative approaches are needed to ensure patients' and caregivers' voices are meaningfully included to advance PC CDS.

Visualizing Inequities: A Step Toward Equitable Student Outcomes

The underrepresentation and underperformance of low-income, first-generation, gender minoritized, Black, Latine, and Indigenous students in Science, Technology, Engineering, and Mathematics (STEM) occurs for a variety of reasons, including, that students in these groups experience opportunity gaps in STEM classes. A critical approach to disrupting persistent inequities is implementing policies and practices that no longer systematically disadvantage students from minoritized groups. To do this, instructors must use data-informed reflection to interrogate their course outcomes. However, these data can be hard to access, process, and visualize in ways that make patterns of inequities clear. To address this need, we developed an R-Shiny application that allows authenticated users to visualize inequities in student performance. An explorable example can be found here: https://theobaldlab.shinyapps.io/visualizinginequities/. In this essay, we use publicly retrieved data as an illustrative example to detail 1) how individual instructors, groups of instructors, and institutions might use this tool for guided self-reflection and 2) how to adapt the code to accommodate data retrieved from local sources. All of the code is freely available here: https://github.com/TheobaldLab/VisualizingInequities. We hope faculty, administrators, and higher-education policymakers will make visible the opportunity gaps in college courses, with the explicit goal of creating transformative, equitable education through self-reflection, group discussion, and structured support.

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

This paper discusses some overlooked challenges faced when working with machine learning models for histopathology and presents a novel opportunity to support "Learning Health Systems" with them. Initially, the authors elaborate on these challenges after separating them according to their mitigation strategies: those that need innovative approaches, time, or future technological capabilities and those that require a conceptual reappraisal from a critical perspective. Then, a novel opportunity to support "Learning Health Systems" by integrating hidden information extracted by ML models from digitalized histopathology slides with other healthcare big data is presented.

EDAI Framework for Integrating Equity, Diversity, and Inclusion Throughout the Lifecycle of AI to Improve Health and Oral Health Care: Qualitative Study

BACKGROUND

Recent studies have identified significant gaps in equity, diversity, and inclusion (EDI) considerations within the lifecycle of artificial intelligence (AI), spanning from data collection and problem definition to implementation stages. Despite the recognized need for integrating EDI principles, there is currently no existing guideline or framework to support this integration in the AI lifecycle.

OBJECTIVE

This study aimed to address this gap by identifying EDI principles and indicators to be integrated into the AI lifecycle. The goal was to develop a comprehensive guiding framework to guide the development and implementation of future AI systems.

METHODS

This study was conducted in 3 phases. In phase 1, a comprehensive systematic scoping review explored how EDI principles have been integrated into AI in health and oral health care settings. In phase 2, a multidisciplinary team was established, and two 2-day, in-person international workshops with over 60 representatives from diverse backgrounds, expertise, and communities were conducted. The workshops included plenary presentations, round table discussions, and focused group discussions. In phase 3, based on the workshops' insights, the EDAI framework was developed and refined through iterative feedback from participants. The results of the initial systematic scoping review have been published separately, and this paper focuses on subsequent phases of the project, which is related to framework development.

RESULTS

In this study, we developed the EDAI framework, a comprehensive guideline that integrates EDI principles and indicators throughout the entire AI lifecycle. This framework addresses existing gaps at various stages, from data collection to implementation, and focuses on individual, organizational, and systemic levels. Additionally, we identified both the facilitators and barriers to integrating EDI within the AI lifecycle in health and oral health care.

CONCLUSIONS

The developed EDAI framework provides a comprehensive, actionable guideline for integrating EDI principles into AI development and deployment. By facilitating the systematic incorporation of these principles, the framework supports the creation and implementation of AI systems that are not only technologically advanced but also sensitive to EDI principles.

Machine learning for environmental justice: Dissecting an algorithmic approach to predict drinking water quality in California

The potential for machine learning to answer questions of environmental science, monitoring, and regulatory enforcement is evident, but there is cause for concern regarding potential embedded bias: algorithms can codify discrimination and exacerbate systematic gaps. This paper, organized into two halves, underscores the importance of vetting algorithms for bias when used for questions of environmental science and justice. In the first half, we present a case study of using machine learning for environmental justice-motivated research: prediction of drinking water quality. While performance varied across models and contaminants, some performed well. Multiple models had overall accuracy rates at or above 90 % and F2 scores above 0.60 on their respective test sets. In the second half, we dissect this algorithmic approach to examine how modeling decisions affect modeling outcomes - and not only how these decisions change whether the model is correct or incorrect, but for whom. We find that multiple decision points in the modeling process can lead to different predictive outcomes. More importantly, we find that these choices can result in significant differences in demographic characteristics of false negatives. We conclude by proposing a set of practices for researchers and policy makers to follow (and improve upon) when applying machine learning to questions of environmental science, management, and justice.

Economics and Equity of Large Language Models: Health Care Perspective

Large language models (LLMs) continue to exhibit noteworthy capabilities across a spectrum of areas, including emerging proficiencies across the health care continuum. Successful LLM implementation and adoption depend on digital readiness, modern infrastructure, a trained workforce, privacy, and an ethical regulatory landscape. These factors can vary significantly across health care ecosystems, dictating the choice of a particular LLM implementation pathway. This perspective discusses 3 LLM implementation pathways-training from scratch pathway (TSP), fine-tuned pathway (FTP), and out-of-the-box pathway (OBP)-as potential onboarding points for health systems while facilitating equitable adoption. The choice of a particular pathway is governed by needs as well as affordability. Therefore, the risks, benefits, and economics of these pathways across 4 major cloud service providers (Amazon, Microsoft, Google, and Oracle) are presented. While cost comparisons, such as on-demand and spot pricing across the cloud service providers for the 3 pathways, are presented for completeness, the usefulness of managed services and cloud enterprise tools is elucidated. Managed services can complement the traditional workforce and expertise, while enterprise tools, such as federated learning, can overcome sample size challenges when implementing LLMs using health care data. Of the 3 pathways, TSP is expected to be the most resource-intensive regarding infrastructure and workforce while providing maximum customization, enhanced transparency, and performance. Because TSP trains the LLM using enterprise health care data, it is expected to harness the digital signatures of the population served by the health care system with the potential to impact outcomes. The use of pretrained models in FTP is a limitation. It may impact its performance because the training data used in the pretrained model may have hidden bias and may not necessarily be health care-related. However, FTP provides a balance between customization, cost, and performance. While OBP can be rapidly deployed, it provides minimal customization and transparency without guaranteeing long-term availability. OBP may also present challenges in interfacing seamlessly with downstream applications in health care settings with variations in pricing and use over time. Lack of customization in OBP can significantly limit its ability to impact outcomes. Finally, potential applications of LLMs in health care, including conversational artificial intelligence, chatbots, summarization, and machine translation, are highlighted. While the 3 implementation pathways discussed in this perspective have the potential to facilitate equitable adoption and democratization of LLMs, transitions between them may be necessary as the needs of health systems evolve. Understanding the economics and trade-offs of these onboarding pathways can guide their strategic adoption and demonstrate value while impacting health care outcomes favorably.

Why we should talk about institutional (dis)trustworthiness and medical machine learning

The principle of trust has been placed at the centre as an attitude for engaging with clinical machine learning systems. However, the notions of trust and distrust remain fiercely debated in the philosophical and ethical literature. In this article, we proceed on a structural level ex negativo as we aim to analyse the concept of "institutional distrustworthiness" to achieve a proper diagnosis of how we should not engage with medical machine learning. First, we begin with several examples that hint at the emergence of a climate of distrust in the context of medical machine learning. Second, we introduce the concept of institutional trustworthiness based on an expansion of Hawley's commitment account. Third, we argue that institutional opacity can undermine the trustworthiness of medical institutions and can lead to new forms of testimonial injustices. Finally, we focus on possible building blocks for repairing institutional distrustworthiness.

Acceptance of Digital Health Care Technology and the Role of Nursing Education

BACKGROUND

As digital health care technology develops, the use of technology in the nursing field has become an essential part of nursing education. However, few studies have linked nurses' awareness of digital health care with nursing education. This study examines the direct effects and mediating effects of factors in technology acceptance on nurses' intention to accept digital health care technology.

METHOD

To empirically investigate these relationships, a survey was conducted among nurses in South Korea.

RESULTS

This study emphasizes the importance of a multifaceted approach that considers personal, organizational, and innovation-related factors in understanding nurses' intentions toward acceptance of digital health care technologies.

CONCLUSION

The findings confirm that performance expectancy and facilitating conditions play crucial roles in nurses' acceptance of digital health care technologies. The mediating effects of performance expectancy and facilitating conditions on intentions to accept technologies suggest that these factors can also play vital indirect roles. [J Contin Educ Nurs. 202x;5x(x):xx-xx.].

Visualizing radiological data bias through persistence images

Persistence images, derived from topological data analysis, emerge as a powerful tool for visualizing and mitigating biases in radiological data interpretation and AI model development. This technique transforms complex topological features into stable, interpretable representations, offering unique insights into medical imaging data structure. By providing intuitive visualizations, persistence images enable the identification of subtle structural differences and potential biases in data acquisition, interpretation, and AI model training. Persistence images can also facilitate stratified sampling, matching statistics, and noise filtration, enhancing the accuracy and equity of radiological analysis. Despite challenges in computational complexity and workflow integration, persistence images show promise in developing more accurate, equitable, and trustworthy AI systems in radiology, potentially improving patient outcomes and personalized healthcare delivery.

Recommendations for Clinicians, Technologists, and Healthcare Organizations on the Use of Generative Artificial Intelligence in Medicine: A Position Statement from the Society of General Internal Medicine

Generative artificial intelligence (generative AI) is a new technology with potentially broad applications across important domains of healthcare, but serious questions remain about how to balance the promise of generative AI against unintended consequences from adoption of these tools. In this position statement, we provide recommendations on behalf of the Society of General Internal Medicine on how clinicians, technologists, and healthcare organizations can approach the use of these tools. We focus on three major domains of medical practice where clinicians and technology experts believe generative AI will have substantial immediate and long-term impacts: clinical decision-making, health systems optimization, and the patient-physician relationship. Additionally, we highlight our most important generative AI ethics and equity considerations for these stakeholders. For clinicians, we recommend approaching generative AI similarly to other important biomedical advancements, critically appraising its evidence and utility and incorporating it thoughtfully into practice. For technologists developing generative AI for healthcare applications, we recommend a major frameshift in thinking away from the expectation that clinicians will "supervise" generative AI. Rather, these organizations and individuals should hold themselves and their technologies to the same set of high standards expected of the clinical workforce and strive to design high-performing, well-studied tools that improve care and foster the therapeutic relationship, not simply those that improve efficiency or market share. We further recommend deep and ongoing partnerships with clinicians and patients as necessary collaborators in this work. And for healthcare organizations, we recommend pursuing a combination of both incremental and transformative change with generative AI, directing resources toward both endeavors, and avoiding the urge to rapidly displace the human clinical workforce with generative AI. We affirm that the practice of medicine remains a fundamentally human endeavor which should be enhanced by technology, not displaced by it.

Persistence landscapes: Charting a path to unbiased radiological interpretation

Persistence landscapes, a sophisticated tool from topological data analysis, offer a promising approach to address biases in radiological interpretation and AI model development. By transforming complex topological features into statistically analyzable functions, they enable robust comparisons between populations and datasets. Persistence landscapes excel in noise filtration, fusion bias mitigation, and enhancing machine learning models. Despite challenges in computation and integration, they show potential to improve the accuracy and equity of radiological analysis, particularly in multi-modal imaging and AI-assisted interpretation.

Artificial Intelligence in Cardiovascular Clinical Trials

Randomized clinical trials are the gold standard for establishing the efficacy and safety of cardiovascular therapies. However, current pivotal trials are expensive, lengthy, and insufficiently diverse. Emerging artificial intelligence (AI) technologies can potentially automate and streamline clinical trial operations. This review describes opportunities to integrate AI throughout a trial's life cycle, including designing the trial, identifying eligible patients, obtaining informed consent, ascertaining physiological and clinical event outcomes, interpreting imaging, and analyzing or disseminating the results. Nevertheless, AI poses risks, including generating inaccurate results, amplifying biases against underrepresented groups, and violating patient privacy. Medical journals and regulators are developing new frameworks to evaluate AI research tools and the data they generate. Given the high-stakes role of randomized trials in medical decision making, AI must be integrated carefully and transparently to protect the validity of trial results.

Persistence barcodes: A novel approach to reducing bias in radiological analysis

Persistence barcodes emerge as a promising tool in radiological analysis, offering a novel approach to reduce bias and uncover hidden patterns in medical imaging. By leveraging topological data analysis, this technique provides a robust, multi-scale perspective on image features, potentially overcoming limitations in traditional methods and Graph Neural Networks. While challenges in interpretation and implementation remain, persistence barcodes show significant potential for improving diagnostic accuracy, standardization, and ultimately, patient outcomes in the evolving field of radiology.

Representation of intensivists' race/ethnicity, sex, and age by artificial intelligence: a cross-sectional study of two text-to-image models

BACKGROUND

Integrating artificial intelligence (AI) into intensive care practices can enhance patient care by providing real-time predictions and aiding clinical decisions. However, biases in AI models can undermine diversity, equity, and inclusion (DEI) efforts, particularly in visual representations of healthcare professionals. This work aims to examine the demographic representation of two AI text-to-image models, Midjourney and ChatGPT DALL-E 2, and assess their accuracy in depicting the demographic characteristics of intensivists.

METHODS

This cross-sectional study, conducted from May to July 2024, used demographic data from the USA workforce report (2022) and intensive care trainees (2021) to compare real-world intensivist demographics with images generated by two AI models, Midjourney v6.0 and ChatGPT 4.0 DALL-E 2. A total of 1,400 images were generated across ICU subspecialties, with outcomes being the comparison of sex, race/ethnicity, and age representation in AI-generated images to the actual workforce demographics.

RESULTS

The AI models demonstrated noticeable biases when compared to the actual U.S. intensive care workforce data, notably overrepresenting White and young doctors. ChatGPT-DALL-E2 produced less female (17.3% vs 32.2%, p < 0.0001), more White (61% vs 55.1%, p = 0.002) and younger (53.3% vs 23.9%, p < 0.001) individuals. While Midjourney depicted more female (47.6% vs 32.2%, p < 0.001), more White (60.9% vs 55.1%, p = 0.003) and younger intensivist (49.3% vs 23.9%, p < 0.001). Substantial differences between the specialties within both models were observed. Finally when compared together, both models showed significant differences in the Portrayal of intensivists.

CONCLUSIONS

Significant biases in AI images of intensivists generated by ChatGPT DALL-E 2 and Midjourney reflect broader cultural issues, potentially perpetuating stereotypes of healthcare worker within the society. This study highlights the need for an approach that ensures fairness, accountability, transparency, and ethics in AI applications for healthcare.

Machine Learning for Mental Health: Applications, Challenges, and the Clinician's Role

PURPOSE OF REVIEW

This review aims to evaluate the current psychiatric applications and limitations of machine learning (ML), defined as techniques used to train algorithms to improve performance at a task based on data. The review emphasizes the clinician's role in ensuring equitable and effective patient care and seeks to inform mental health providers about the importance of clinician involvement in these technologies.

RECENT FINDINGS

ML in psychiatry has advanced through electronic health record integration, disease phenotyping, and remote monitoring through mobile applications. However, these applications face challenges related to health equity, privacy, translation to practice, and validation. Clinicians play crucial roles in ensuring data quality, mitigating biases, promoting algorithm transparency, guiding clinical implementation, and advocating for ethical and patient-centered use of ML tools. Clinicians are essential in addressing the challenges of ML, ensuring its ethical application, and promoting equitable care, thus improving the effectiveness of ML in practice.

A new era for stress research: supporting user performance and experience in the digital age

Stress is both a driver of objective performance impairments and a source of negative user experience of technology. This review addresses future directions for research on stress and ergonomics in the digital age. The review is structured around three levels of analysis. At the individual user level, stress is elicited by novel technologies and tasks including interaction with AI and robots, working in Virtual Reality, and operating autonomous vehicles. At the organisational level, novel, potentially stressful challenges include maintaining cybersecurity, surveillance and monitoring of employees supported by technology, and addressing bias and discrimination in the workplace. At the sociocultural level, technology, values and norms are evolving symbiotically, raising novel demands illustrated with respect to interactions with social media and new ethical challenges. We also briefly review the promise of neuroergonomics and emotional design to support stress mitigation. We conclude with seven high-level principles that may guide future work.

Evaluating machine learning model bias and racial disparities in non-small cell lung cancer using SEER registry data

BACKGROUND

Despite decades of pursuing health equity, racial and ethnic disparities persist in healthcare in America. For cancer specifically, one of the leading observed disparities is worse mortality among non-Hispanic Black patients compared to non-Hispanic White patients across the cancer care continuum. These real-world disparities are reflected in the data used to inform the decisions made to alleviate such inequities. Failing to account for inherently biased data underlying these observations could intensify racial cancer disparities and lead to misguided efforts that fail to appropriately address the real causes of health inequity.

OBJECTIVE

Estimate the racial/ethnic bias of machine learning models in predicting two-year survival and surgery treatment recommendation for non-small cell lung cancer (NSCLC) patients.

METHODS

A Cox survival model, and a LOGIT model as well as three other machine learning models for predicting surgery recommendation were trained using SEER data from NSCLC patients diagnosed from 2000-2018. Models were trained with a 70/30 train/test split (both including and excluding race/ethnicity) and evaluated using performance and fairness metrics. The effects of oversampling the training data were also evaluated.

RESULTS

The survival models show disparate impact towards non-Hispanic Black patients regardless of whether race/ethnicity is used as a predictor. The models including race/ethnicity amplified the disparities observed in the data. The exclusion of race/ethnicity as a predictor in the survival and surgery recommendation models improved fairness metrics without degrading model performance. Stratified oversampling strategies reduced disparate impact while reducing the accuracy of the model.

CONCLUSION

NSCLC disparities are complex and multifaceted. Yet, even when accounting for age and stage at diagnosis, non-Hispanic Black patients with NSCLC are less often recommended to have surgery than non-Hispanic White patients. Machine learning models amplified the racial/ethnic disparities across the cancer care continuum (which are reflected in the data used to make model decisions). Excluding race/ethnicity lowered the bias of the models but did not affect disparate impact. Developing analytical strategies to improve fairness would in turn improve the utility of machine learning approaches analyzing population-based cancer data.

Validating, Implementing, and Monitoring Machine Learning Solutions in the Clinical Laboratory Safely and Effectively

BACKGROUND

Machine learning solutions offer tremendous promise for improving clinical and laboratory operations in pathology. Proof-of-concept descriptions of these approaches have become commonplace in laboratory medicine literature, but only a scant few of these have been implemented within clinical laboratories, owing to the often substantial barriers in validating, implementing, and monitoring these applications in practice. This mini-review aims to highlight the key considerations in each of these steps.

CONTENT

Effective and responsible applications of machine learning in clinical laboratories require robust validation prior to implementation. A comprehensive validation study involves a critical evaluation of study design, data engineering and interoperability, target label definition, metric selection, generalizability and applicability assessment, algorithmic fairness, and explainability. While the main text highlights these concepts in broad strokes, a supplementary code walk-through is also provided to facilitate a more practical understanding of these topics using a real-world classification task example, the detection of saline-contaminated chemistry panels.Following validation, the laboratorian's role is far from over. Implementing machine learning solutions requires an interdisciplinary effort across several roles in an organization. We highlight the key roles, responsibilities, and terminologies for successfully deploying a validated solution into a live production environment. Finally, the implemented solution must be routinely monitored for signs of performance degradation and updated if necessary.

SUMMARY

This mini-review aims to bridge the gap between theory and practice by highlighting key concepts in validation, implementation, and monitoring machine learning solutions effectively and responsibly in the clinical laboratory.

Enhancing neuro-oncology care through equity-driven applications of artificial intelligence

The disease course and clinical outcome for brain tumor patients depend not only on the molecular and histological features of the tumor but also on the patient's demographics and social determinants of health. While current investigations in neuro-oncology have broadly utilized artificial intelligence (AI) to enrich tumor diagnosis and more accurately predict treatment response, postoperative complications, and survival, equity-driven applications of AI have been limited. However, AI applications to advance health equity in the broader medical field have the potential to serve as practical blueprints to address known disparities in neuro-oncologic care. In this consensus review, we will describe current applications of AI in neuro-oncology, postulate viable AI solutions for the most pressing inequities in neuro-oncology based on broader literature, propose a framework for the effective integration of equity into AI-based neuro-oncology research, and close with the limitations of AI.

The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities - a scoping review

This scoping review examined racial and ethnic bias in artificial intelligence health algorithms (AIHA), the role of stakeholders in oversight, and the consequences of AIHA for health equity. Using the PRISMA-ScR guidelines, databases were searched between 2020 and 2024 using the terms racial and ethnic bias in health algorithms resulting in a final sample of 23 sources. Suggestions for how to mitigate algorithmic bias were compiled and evaluated, roles played by stakeholders were identified, and governance and stewardship plans for AIHA were examined. While AIHA represent a significant breakthrough in predictive analytics and treatment optimization, regularly outperforming humans in diagnostic precision and accuracy, they also present serious challenges to patient privacy, data security, institutional transparency, and health equity. Evidence from extant sources including those in this review showed that AIHA carry the potential to perpetuate health inequities. While the current study considered AIHA in the US, the use of AIHA carries implications for global health equity.

Artificial intelligence in abdominal and pelvic ultrasound imaging: current applications

BACKGROUND

In recent years, the integration of artificial intelligence (AI) techniques into medical imaging has shown great potential to transform the diagnostic process. This review aims to provide a comprehensive overview of current state-of-the-art applications for AI in abdominal and pelvic ultrasound imaging.

METHODS

We searched the PubMed, FDA, and ClinicalTrials.gov databases for applications of AI in abdominal and pelvic ultrasound imaging.

RESULTS

A total of 128 titles were identified from the database search and were eligible for screening. After screening, 57 manuscripts were included in the final review. The main anatomical applications included multi-organ detection (n = 16, 28%), gynecology (n = 15, 26%), hepatobiliary system (n = 13, 23%), and musculoskeletal (n = 8, 14%). The main methodological applications included deep learning (n = 37, 65%), machine learning (n = 13, 23%), natural language processing (n = 5, 9%), and robots (n = 2, 4%). The majority of the studies were single-center (n = 43, 75%) and retrospective (n = 56, 98%). We identified 17 FDA approved AI ultrasound devices, with only a few being specifically used for abdominal/pelvic imaging (infertility monitoring and follicle development).

CONCLUSION

The application of AI in abdominal/pelvic ultrasound shows promising early results for disease diagnosis, monitoring, and report refinement. However, the risk of bias remains high because very few of these applications have been prospectively validated (in multi-center studies) or have received FDA clearance.

Mitigating Racial Bias in Health Care Algorithms: Improving Fairness in Access to Supportive Housing

Algorithms for guiding health care decisions have come under increasing scrutiny for being unfair to certain racial and ethnic groups. The authors describe their multistep process, using data from 3,465 individuals, to reduce racial and ethnic bias in an algorithm developed to identify state Medicaid beneficiaries experiencing homelessness and chronic health needs who were eligible for coordinated health care and housing supports. Through an iterative process of adjusting inputs, reviewing outputs with diverse stakeholders, and performing quality assurance, the authors developed an algorithm that achieved racial and ethnic parity in the selection of eligible Medicaid beneficiaries.

Bias in medical AI: Implications for clinical decision-making

Biases in medical artificial intelligence (AI) arise and compound throughout the AI lifecycle. These biases can have significant clinical consequences, especially in applications that involve clinical decision-making. Left unaddressed, biased medical AI can lead to substandard clinical decisions and the perpetuation and exacerbation of longstanding healthcare disparities. We discuss potential biases that can arise at different stages in the AI development pipeline and how they can affect AI algorithms and clinical decision-making. Bias can occur in data features and labels, model development and evaluation, deployment, and publication. Insufficient sample sizes for certain patient groups can result in suboptimal performance, algorithm underestimation, and clinically unmeaningful predictions. Missing patient findings can also produce biased model behavior, including capturable but nonrandomly missing data, such as diagnosis codes, and data that is not usually or not easily captured, such as social determinants of health. Expertly annotated labels used to train supervised learning models may reflect implicit cognitive biases or substandard care practices. Overreliance on performance metrics during model development may obscure bias and diminish a model's clinical utility. When applied to data outside the training cohort, model performance can deteriorate from previous validation and can do so differentially across subgroups. How end users interact with deployed solutions can introduce bias. Finally, where models are developed and published, and by whom, impacts the trajectories and priorities of future medical AI development. Solutions to mitigate bias must be implemented with care, which include the collection of large and diverse data sets, statistical debiasing methods, thorough model evaluation, emphasis on model interpretability, and standardized bias reporting and transparency requirements. Prior to real-world implementation in clinical settings, rigorous validation through clinical trials is critical to demonstrate unbiased application. Addressing biases across model development stages is crucial for ensuring all patients benefit equitably from the future of medical AI.

Demographic reporting in biosignal datasets: a comprehensive analysis of the PhysioNet open access database

The PhysioNet open access database (PND) is one of the world's largest and most comprehensive repositories of biosignal data and is widely used by researchers to develop, train, and validate algorithms. To contextualise the results of such algorithms, understanding the underlying demographic distribution of the data is crucial-specifically, the race, ethnicity, sex or gender, and age of study participants. We sought to understand the underlying reporting patterns and characteristics of the demographic data of the datasets available on PND. Of the 181 unique datasets present in the PND as of July 6, 2023, 175 involved human participants, with less than 7% of studies reporting on all four of the key demographic variables. Furthermore, we found a higher rate of reporting sex or gender and age than race and ethnicity. In the studies that did include participant sex or gender, the samples were mostly male. Additionally, we found that most studies were done in North America, particularly in the USA. These imbalances and poor reporting of representation raise concerns regarding potential embedded biases in the algorithms that rely on these datasets. They also underscore the need for universal and comprehensive reporting practices to ensure equitable development and deployment of artificial intelligence and machine learning tools in medicine.

An Approach for Combining Clinical Judgment with Machine Learning to Inform Medical Decision Making: Analysis of Nonemergency Surgery Strategies for Acute Appendicitis in Patients with Multiple Long-Term Conditions

BACKGROUND

Machine learning (ML) methods can identify complex patterns of treatment effect heterogeneity. However, before ML can help to personalize decision making, transparent approaches must be developed that draw on clinical judgment. We develop an approach that combines clinical judgment with ML to generate appropriate comparative effectiveness evidence for informing decision making.

METHODS

We motivate this approach in evaluating the effectiveness of nonemergency surgery (NES) strategies, such as antibiotic therapy, for people with acute appendicitis who have multiple long-term conditions (MLTCs) compared with emergency surgery (ES). Our 4-stage approach 1) draws on clinical judgment about which patient characteristics and morbidities modify the relative effectiveness of NES; 2) selects additional covariates from a high-dimensional covariate space (P > 500) by applying an ML approach, least absolute shrinkage and selection operator (LASSO), to large-scale administrative data (N = 24,312); 3) generates estimates of comparative effectiveness for relevant subgroups; and 4) presents evidence in a suitable form for decision making.

RESULTS

This approach provides useful evidence for clinically relevant subgroups. We found that overall NES strategies led to increases in the mean number of days alive and out-of-hospital compared with ES, but estimates differed across subgroups, ranging from 21.2 (95% confidence interval: 1.8 to 40.5) for patients with chronic heart failure and chronic kidney disease to -10.4 (-29.8 to 9.1) for patients with cancer and hypertension. Our interactive tool for visualizing ML output allows for findings to be customized according to the specific needs of the clinical decision maker.

CONCLUSIONS

This principled approach of combining clinical judgment with an ML approach can improve trust, relevance, and usefulness of the evidence generated for clinical decision making.

HIGHLIGHTS

Machine learning (ML) methods have many potential applications in medical decision making, but the lack of model interpretability and usability constitutes an important barrier for the wider adoption of ML evidence in practice.We develop a 4-stage approach for integrating clinical judgment into the way an ML approach is used to estimate and report comparative effectiveness.We illustrate the approach in undertaking an evaluation of nonemergency surgery (NES) strategies for acute appendicitis in patients with multiple long-term conditions and find that NES strategies lead to better outcomes compared with emergency surgery and that the effects differ across subgroups.We develop an interactive tool for visualizing the results of this study that allows findings to be customized according to the user's preferences.

Artificial intelligence in oncology: ensuring safe and effective integration of language models in clinical practice

In this Personal View, we address the latest advancements in automatic text analysis with artificial intelligence (AI) in medicine, with a focus on its implications in aiding treatment decisions in medical oncology. Acknowledging that a majority of hospital medical content is embedded in narrative format, natural language processing has become one of the most dynamic research fields for developing clinical decision support tools. In addition, large language models have recently reached unprecedented performance, notably when answering medical questions. Emerging applications include prognosis estimation, treatment recommendations, multidisciplinary tumor board recommendations and matching patients to recruiting clinical trials. Altogether, we advocate for a forward-looking approach in which the community efficiently initiates global prospective clinical evaluations of promising AI-based decision support systems. Such assessments will be essential to validate and evaluate potential biases, ensuring these innovations can be effectively and safely translated into practical tools for oncological practice. We are at a pivotal moment, where continued advancements in patient care must be pursued with scientific rigor.

Derivation of a clinical decision rule for termination of resuscitation in non-traumatic pediatric out-of-hospital cardiac arrest

AIM

Prehospital termination of resuscitation (ToR) rules are used to predict medical futility in adult out-of-hospital cardiac arrest (OHCA), however, the available evidence for pediatric patients is limited. The primary aim of this study is to derive a Pediatric Termination of Resuscitation (PToR) prediction rule for use in pediatric non-traumatic OHCA patients.

METHODS

We analyzed a retrospective cohort of pediatric OHCA patients within the CARES database over a 10-year period (2013-2022). We split the dataset into training and test datasets and fit logistic regressions with Least Absolute Shrinkage and Selection Operator (LASSO) to select predictor variables and estimate predictive test characteristics for the primary outcome of death and a secondary composite outcome of death or survival to hospital discharge with unfavorable neurologic status.

RESULTS

We analyzed a sample of 21,240 children where 2,326 (11.0%) survived to hospital discharge, and 1,894 (8.9%) survived to hospital discharge with favorable neurologic status. We derived a PToR rule for death demonstrating a specificity of 99.1% and a positive predictive value (PPV) of 99.8% and a PToR rule for death or survival with poor neurologic status with a specificity of 99.7% and PPV of 99.9% within the test dataset.

CONCLUSION

We derived a clinical prediction rule with high specificity and positive predictive value in prehospital settings utilizing Advanced Life Support (ALS) providers which may inform termination of resuscitation considerations in pediatric patients. Further prospective and validation studies will be necessary to define the appropriateness and applicability of these PToR criteria for routine use.

White privilege, ethnic disadvantage, and stigmatized linguistic capital: COVID-19 infection rates and lockdown law enforcement in Hong Kong

The COVID-19 pandemic has exposed and exacerbated existing disparities in various societies. This study investigates disparities among racial, ethnic, and linguistic groups in Hong Kong's society in COVID-19 infection rates and lockdown enforcement practices that was imposed 545 times from January 2021 to September 2022 and affected 9% of the population. It is found that neighborhoods with more white individuals had lower infection rates than the overall population, while those with more ethnically minoritized groups had higher infection rates. Furthermore, hit rate tests reveal that the government targeted more neighborhoods with a higher share of individuals from linguistically minoritized groups. This novel finding suggests that not only race, but linguistic difference of the same ethnicity can cause bias. The study highlights the positive impact of providing ethnic support services on health outcomes in neighborhoods with a higher share of individuals from ethnically minoritized groups.

Revealing transparency gaps in publicly available COVID-19 datasets used for medical artificial intelligence development-a systematic review

During the COVID-19 pandemic, artificial intelligence (AI) models were created to address health-care resource constraints. Previous research shows that health-care datasets often have limitations, leading to biased AI technologies. This systematic review assessed datasets used for AI development during the pandemic, identifying several deficiencies. Datasets were identified by screening articles from MEDLINE and using Google Dataset Search. 192 datasets were analysed for metadata completeness, composition, data accessibility, and ethical considerations. Findings revealed substantial gaps: only 48% of datasets documented individuals' country of origin, 43% reported age, and under 25% included sex, gender, race, or ethnicity. Information on data labelling, ethical review, or consent was frequently missing. Many datasets reused data with inadequate traceability. Notably, historical paediatric chest x-rays appeared in some datasets without acknowledgment. These deficiencies highlight the need for better data quality and transparent documentation to lessen the risk that biased AI models are developed in future health emergencies.

Fairness gaps in Machine learning models for hospitalization and emergency department visit risk prediction in home healthcare patients with heart failure

OBJECTIVES

This study aims to evaluate the fairness performance metrics of Machine Learning (ML) models to predict hospitalization and emergency department (ED) visits in heart failure patients receiving home healthcare. We analyze biases, assess performance disparities, and propose solutions to improve model performance in diverse subpopulations.

METHODS

The study used a dataset of 12,189 episodes of home healthcare collected between 2015 and 2017, including structured (e.g., standard assessment tool) and unstructured data (i.e., clinical notes). ML risk prediction models, including Light Gradient-boosting model (LightGBM) and AutoGluon, were developed using demographic information, vital signs, comorbidities, service utilization data, and the area deprivation index (ADI) associated with the patient's home address. Fairness metrics, such as Equal Opportunity, Predictive Equality, Predictive Parity, and Statistical Parity, were calculated to evaluate model performance across subpopulations.

RESULTS

Our study revealed significant disparities in model performance across diverse demographic subgroups. For example, the Hispanic, Male, High-ADI subgroup excelled in terms of Equal Opportunity with a metric value of 0.825, which was 28% higher than the lowest-performing Other, Female, Low-ADI subgroup, which scored 0.644. In Predictive Parity, the gap between the highest and lowest-performing groups was 29%, and in Statistical Parity, the gap reached 69%. In Predictive Equality, the difference was 45%.

DISCUSSION AND CONCLUSION

The findings highlight substantial differences in fairness metrics across diverse patient subpopulations in ML risk prediction models for heart failure patients receiving home healthcare services. Ongoing monitoring and improvement of fairness metrics are essential to mitigate biases.

Core Concepts in Pharmacoepidemiology: Principled Use of Artificial Intelligence and Machine Learning in Pharmacoepidemiology and Healthcare Research

Artificial intelligence (AI) and machine learning (ML) are important tools across many fields of health and medical research. Pharmacoepidemiologists can bring essential methodological rigor and study design expertise to the design and use of these technologies within healthcare settings. AI/ML-based tools also play a role in pharmacoepidemiology research, as we may apply them to answer our own research questions, take responsibility for evaluating medical devices with AI/ML components, or participate in interdisciplinary research to create new AI/ML algorithms. While epidemiologic expertise is essential to deploying AI/ML responsibly and ethically, the rapid advancement of these technologies in the past decade has resulted in a knowledge gap for many in the field. This article provides a brief overview of core AI/ML concepts, followed by a discussion of potential applications of AI/ML in pharmacoepidemiology research, and closes with a review of important concepts across application areas, including interpretability and fairness. This review is intended to provide an accessible, practical overview of AI/ML for pharmacoepidemiology research, with references to further, more detailed resources on fundamental topics.

Tailoring Risk Prediction Models to Local Populations

Importance

Risk estimation is an integral part of cardiovascular care. Local recalibration of guideline-recommended models could address the limitations of existing tools.

Objective

To provide a machine learning (ML) approach to augment the performance of the American Heart Association's Predicting Risk of Cardiovascular Disease Events (AHA-PREVENT) equations when applied to a local population while preserving clinical interpretability.

Design, Setting, and Participants

This cohort study used a New England-based electronic health record cohort of patients without prior atherosclerotic cardiovascular disease (ASCVD) who had the data necessary to calculate the AHA-PREVENT 10-year risk of developing ASCVD in the event period (2007-2016). Patients with prior ASCVD events, death prior to 2007, or age 79 years or older in 2007 were subsequently excluded. The final study population of 95 326 patients was split into 3 nonoverlapping subsets for training, testing, and validation. The AHA-PREVENT model was adapted to this local population using the open-source ML model (MLM) Extreme Gradient Boosting model (XGBoost) with minimal predictor variables, including age, sex, and AHA-PREVENT. The MLM was monotonically constrained to preserve known associations between risk factors and ASCVD risk. Along with sex, race and ethnicity data from the electronic health record were collected to validate the performance of ASCVD risk prediction in subgroups. Data were analyzed from August 2021 to February 2024.

Main Outcomes and Measures

Consistent with the AHA-PREVENT model, ASCVD events were defined as the first occurrence of either nonfatal myocardial infarction, coronary artery disease, ischemic stroke, or cardiovascular death. Cardiovascular death was coded via government registries. Discrimination, calibration, and risk reclassification were assessed using the Harrell C index, a modified Hosmer-Lemeshow goodness-of-fit test and calibration curves, and reclassification tables, respectively.

Results

In the test set of 38 137 patients (mean [SD] age, 64.8 [6.9] years, 22 708 [59.5]% women and 15 429 [40.5%] men; 935 [2.5%] Asian, 2153 [5.6%] Black, 1414 [3.7%] Hispanic, 31 400 [82.3%] White, and 2235 [5.9%] other, including American Indian, multiple races, unspecified, and unrecorded, consolidated owing to small numbers), MLM-PREVENT had improved calibration (modified Hosmer-Lemeshow P > .05) compared to the AHA-PREVENT model across risk categories in the overall cohort (χ23 = 2.2; P = .53 vs χ23 > 16.3; P < .001) and sex subgroups (men: χ23 = 2.1; P = .55 vs χ23 > 16.3; P < .001; women: χ23 = 6.5; P = .09 vs. χ23 > 16.3; P < .001), while also surpassing a traditional recalibration approach. MLM-PREVENT maintained or improved AHA-PREVENT's calibration in Asian, Black, and White individuals. Both MLM-PREVENT and AHA-PREVENT performed equally well in discriminating risk (approximate ΔC index, ±0.01). Using a clinically significant 7.5% risk threshold, MLM-PREVENT reclassified a total of 11.5% of patients. We visualize the recalibration through MLM-PREVENT ASCVD risk charts that highlight preserved risk associations of the original AHA-PREVENT model.

Conclusions and Relevance

The interpretable ML approach presented in this article enhanced the accuracy of the AHA-PREVENT model when applied to a local population while still preserving the risk associations found by the original model. This method has the potential to recalibrate other established risk tools and is implementable in electronic health record systems for improved cardiovascular risk assessment.

Racial differences in laboratory testing as a potential mechanism for bias in AI: A matched cohort analysis in emergency department visits

AI models are often trained using available laboratory test results. Racial differences in laboratory testing may bias AI models for clinical decision support, amplifying existing inequities. This study aims to measure the extent of racial differences in laboratory testing in adult emergency department (ED) visits. We conducted a retrospective 1:1 exact-matched cohort study of Black and White adult patients seen in the ED, matching on age, biological sex, chief complaint, and ED triage score, using ED visits at two U.S. teaching hospitals: Michigan Medicine, Ann Arbor, MI (U-M, 2015-2022), and Beth Israel Deaconess Medical Center, Boston, MA (BIDMC, 2011-2019). Post-matching, White patients had significantly higher testing rates than Black patients for complete blood count (BIDMC difference: 1.7%, 95% CI: 1.1% to 2.4%, U-M difference: 2.0%, 95% CI: 1.6% to 2.5%), metabolic panel (BIDMC: 1.5%, 95% CI: 0.9% to 2.1%, U-M: 1.9%, 95% CI: 1.4% to 2.4%), and blood culture (BIDMC: 0.9%, 95% CI: 0.5% to 1.2%, U-M: 0.7%, 95% CI: 0.4% to 1.1%). Black patients had significantly higher testing rates for troponin than White patients (BIDMC: -2.1%, 95% CI: -2.6% to -1.6%, U-M: -2.2%, 95% CI: -2.7% to -1.8%). The observed racial testing differences may impact AI models trained using available laboratory results. The findings also motivate further study of how such differences arise and how to mitigate potential impacts on AI models.

Perceptions Toward Using Artificial Intelligence and Technology for Asthma Attack Risk Prediction: Qualitative Exploration of Māori Views

BACKGROUND

Asthma is a significant global health issue, impacting over 500,000 individuals in New Zealand and disproportionately affecting Māori communities in New Zealand, who experience worse asthma symptoms and attacks. Digital technologies, including artificial intelligence (AI) and machine learning (ML) models, are increasingly popular for asthma risk prediction. However, these AI models may underrepresent minority ethnic groups and introduce bias, potentially exacerbating disparities.

OBJECTIVE

This study aimed to explore the views and perceptions that Māori have toward using AI and ML technologies for asthma self-management, identify key considerations for developing asthma attack risk prediction models, and ensure Māori are represented in ML models without worsening existing health inequities.

METHODS

Semistructured interviews were conducted with 20 Māori participants with asthma, 3 male and 17 female, aged 18-76 years. All the interviews were conducted one-on-one, except for 1 interview, which was conducted with 2 participants. Altogether, 10 web-based interviews were conducted, while the rest were kanohi ki te kanohi (face-to-face). A thematic analysis was conducted to identify the themes. Further, sentiment analysis was carried out to identify the sentiments using a pretrained Bidirectional Encoder Representations from Transformers model.

RESULTS

We identified four key themes: (1) concerns about AI use, (2) interest in using technology to support asthma, (3) desired characteristics of AI-based systems, and (4) experience with asthma management and opportunities for technology to improve care. AI was relatively unfamiliar to many participants, and some of them expressed concerns about whether AI technology could be trusted, kanohi ki te kanohi interaction, and inadequate knowledge of AI and technology. These concerns are exacerbated by the Māori experience of colonization. Most of the participants were interested in using technology to support their asthma management, and we gained insights into user preferences regarding computer-based health care applications. Participants discussed their experiences, highlighting problems with health care quality and limited access to resources. They also mentioned the factors that trigger their asthma control level.

CONCLUSIONS

The exploration revealed that there is a need for greater information about AI and technology for Māori communities and a need to address trust issues relating to the use of technology. Expectations in relation to computer-based applications for health purposes were expressed. The research outcomes will inform future investigations on AI and technology to enhance the health of people with asthma, in particular those designed for Indigenous populations in New Zealand.

Leveraging Artificial Intelligence and Data Science for Integration of Social Determinants of Health in Emergency Medicine: Scoping Review

Background

Social determinants of health (SDOH) are critical drivers of health disparities and patient outcomes. However, accessing and collecting patient-level SDOH data can be operationally challenging in the emergency department (ED) clinical setting, requiring innovative approaches.

Objective

This scoping review examines the potential of AI and data science for modeling, extraction, and incorporation of SDOH data specifically within EDs, further identifying areas for advancement and investigation.

Methods

We conducted a standardized search for studies published between 2015 and 2022, across Medline (Ovid), Embase (Ovid), CINAHL, Web of Science, and ERIC databases. We focused on identifying studies using AI or data science related to SDOH within emergency care contexts or conditions. Two specialized reviewers in emergency medicine (EM) and clinical informatics independently assessed each article, resolving discrepancies through iterative reviews and discussion. We then extracted data covering study details, methodologies, patient demographics, care settings, and principal outcomes.

Results

Of the 1047 studies screened, 26 met the inclusion criteria. Notably, 9 out of 26 (35%) studies were solely concentrated on ED patients. Conditions studied spanned broad EM complaints and included sepsis, acute myocardial infarction, and asthma. The majority of studies (n=16) explored multiple SDOH domains, with homelessness/housing insecurity and neighborhood/built environment predominating. Machine learning (ML) techniques were used in 23 of 26 studies, with natural language processing (NLP) being the most commonly used approach (n=11). Rule-based NLP (n=5), deep learning (n=2), and pattern matching (n=4) were the most commonly used NLP techniques. NLP models in the reviewed studies displayed significant predictive performance with outcomes, with F1-scores ranging between 0.40 and 0.75 and specificities nearing 95.9%.

Conclusions

Although in its infancy, the convergence of AI and data science techniques, especially ML and NLP, with SDOH in EM offers transformative possibilities for better usage and integration of social data into clinical care and research. With a significant focus on the ED and notable NLP model performance, there is an imperative to standardize SDOH data collection, refine algorithms for diverse patient groups, and champion interdisciplinary synergies. These efforts aim to harness SDOH data optimally, enhancing patient care and mitigating health disparities. Our research underscores the vital need for continued investigation in this domain.

Conceptual scaffolding for the philosophy of medicine

This paper consists of two parts. In the first part, I will introduce a philosophical toolbox that I call 'conceptual scaffolding,' which helps to reflect holistically on phenomena and concepts. I situate this framework within the landscape of conceptual analysis and conceptual engineering, exemplified by the debate about the concept of disease. Within the framework of conceptual scaffolding, I develop the main idea of the paper, which is 'the binocular model of plural medicine', a holistic framework for analyzing medical concepts and phenomena. In the second part, I demonstrate the use and value of the binocular model by analyzing, through the lenses of the model, the phenomenon of health wearable devices and their effects on the concept of diagnosis.

Advancing AI Data Ethics in Nursing: Future Directions for Nursing Practice, Research, and Education

Unlabelled

The ethics of artificial intelligence (AI) are increasingly recognized due to concerns such as algorithmic bias, opacity, trust issues, data security, and fairness. Specifically, machine learning algorithms, central to AI technologies, are essential in striving for ethically sound systems that mimic human intelligence. These technologies rely heavily on data, which often remain obscured within complex systems and must be prioritized for ethical collection, processing, and usage. The significance of data ethics in achieving responsible AI was first highlighted in the broader context of health care and subsequently in nursing. This viewpoint explores the principles of data ethics, drawing on relevant frameworks and strategies identified through a formal literature review. These principles apply to real-world and synthetic data in AI and machine-learning contexts. Additionally, the data-centric AI paradigm is briefly examined, emphasizing its focus on data quality and the ethical development of AI solutions that integrate human-centered domain expertise. The ethical considerations specific to nursing are addressed, including 4 recommendations for future directions in nursing practice, research, and education and 2 hypothetical nurse-focused ethical case studies. The primary objectives are to position nurses to actively participate in AI and data ethics, thereby contributing to creating high-quality and relevant data for machine learning applications.

Navigating algorithm bias in AI: ensuring fairness and trust in Africa

This article presents a perspective on the impact of algorithmic bias on information fairness and trust in artificial intelligence (AI) systems within the African context. The author's personal experiences and observations, combined with relevant literature, formed the basis of this article. The authors demonstrate why algorithm bias poses a substantial challenge in Africa, particularly regarding fairness and the integrity of AI applications. This perspective underscores the urgent need to address biases that compromise the fairness of information dissemination and undermine public trust. The authors advocate for the implementation of strategies that promote inclusivity, enhance cultural sensitivity, and actively engage local communities in the development of AI systems. By prioritizing ethical practices and transparency, stakeholders can mitigate the risks associated with bias, thereby fostering trust and ensuring equitable access to technology. Additionally, the article explores the potential consequences of inaction, including exacerbated social disparities, diminished confidence in public institutions, and economic stagnation. Ultimately, this work argues for a collaborative approach to AI that positions Africa as a leader in responsible development, ensuring that technology serves as a catalyst for sustainable development and social justice.

Guidance for unbiased predictive information for healthcare decision-making and equity (GUIDE): considerations when race may be a prognostic factor

Clinical prediction models (CPMs) are tools that compute the risk of an outcome given a set of patient characteristics and are routinely used to inform patients, guide treatment decision-making, and resource allocation. Although much hope has been placed on CPMs to mitigate human biases, CPMs may potentially contribute to racial disparities in decision-making and resource allocation. While some policymakers, professional organizations, and scholars have called for eliminating race as a variable from CPMs, others raise concerns that excluding race may exacerbate healthcare disparities and this controversy remains unresolved. The Guidance for Unbiased predictive Information for healthcare Decision-making and Equity (GUIDE) provides expert guidelines for model developers and health system administrators on the transparent use of race in CPMs and mitigation of algorithmic bias across contexts developed through a 5-round, modified Delphi process from a diverse 14-person technical expert panel (TEP). Deliberations affirmed that race is a social construct and that the goals of prediction are distinct from those of causal inference, and emphasized: the importance of decisional context (e.g., shared decision-making versus healthcare rationing); the conflicting nature of different anti-discrimination principles (e.g., anticlassification versus antisubordination principles); and the importance of identifying and balancing trade-offs in achieving equity-related goals with race-aware versus race-unaware CPMs for conditions where racial identity is prognostically informative. The GUIDE, comprising 31 key items in the development and use of CPMs in healthcare, outlines foundational principles, distinguishes between bias and fairness, and offers guidance for examining subgroup invalidity and using race as a variable in CPMs. This GUIDE presents a living document that supports appraisal and reporting of bias in CPMs to support best practice in CPM development and use.

The Revised Declaration of Helsinki-Considerations for the Future of Artificial Intelligence in Health and Medical Research

This Viewpoint summarizes recent updates to the Declaration of Helsinki, discusses its relevance in the context of artificial intelligence (AI) in health research, and highlights issues that could affect its future implementation as the use of AI in research increases.

Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings

BACKGROUND

Integrating advanced machine-learning (ML) algorithms into clinical practice is challenging and requires interdisciplinary collaboration to develop transparent, interpretable, and ethically sound clinical decision support (CDS) tools. We aimed to design a ML-driven CDS tool to predict opioid overdose risk and gather feedback for its integration into the University of Florida Health (UFHealth) electronic health record (EHR) system.

METHODS

We used user-centered design methods to integrate the ML algorithm into the EHR system. The backend and UI design sub-teams collaborated closely, both informed by user feedback sessions. We conducted seven user feedback sessions with five UF Health primary care physicians (PCPs) to explore aspects of CDS tools, including workflow, risk display, and risk mitigation strategies. After customizing the tool based on PCPs' feedback, we held two rounds of one-on-one usability testing sessions with 8 additional PCPs to gather feedback on prototype alerts. These sessions informed iterative UI design and backend processes, including alert frequency and reappearance circumstances.

RESULTS

The backend process development identified needs and requirements from our team, information technology, UFHealth, and PCPs. Thirteen PCPs (male = 62%, White = 85%) participated across 7 user feedback sessions and 8 usability testing sessions. During the user feedback sessions, PCPs (n = 5) identified flaws such as the term "high risk" of overdose potentially leading to unintended consequences (e.g., immediate addiction services referrals), offered suggestions, and expressed trust in the tool. In the first usability testing session, PCPs (n = 4) emphasized the need for natural risk presentation (e.g., 1 in 200) and suggested displaying the alert multiple times yearly for at-risk patients. Another 4 PCPs in the second usability testing session valued the UFHealth-specific alert for managing new or unfamiliar patients, expressed concerns about PCPs' workload when prescribing to high-risk patients, and recommended incorporating the details page into training sessions to enhance usability.

CONCLUSIONS

The final backend process for our CDS alert aligns with PCP needs and UFHealth standards. Integrating feedback from PCPs in the early development phase of our ML-driven CDS tool helped identify barriers and facilitators in the CDS integration process. This collaborative approach yielded a refined prototype aimed at minimizing unintended consequences and enhancing usability.

Shaping the future: perspectives on the Integration of Artificial Intelligence in health profession education: a multi-country survey

BACKGROUND

Artificial intelligence (AI) is transforming health profession education (HPE) through personalized learning technologies. HPE students must also learn about AI to understand its impact on healthcare delivery. We examined HPE students' AI-related knowledge and attitudes, and perceived challenges in integrating AI in HPE.

METHODS

This cross-sectional included medical, nursing, physiotherapy, and clinical nutrition students from four public universities in Jordan, the Kingdom of Saudi Arabia (KSA), the United Arab Emirates (UAE), and Egypt. Data were collected between February and October 2023 via an online survey that covered five main domains: benefits of AI in healthcare, negative impact on patient trust, negative impact on the future of healthcare professionals, inclusion of AI in HPE curricula, and challenges hindering integration of AI in HPE.

RESULTS

Of 642 participants, 66.4% reported low AI knowledge levels. The UAE had the largest proportion of students with low knowledge (72.7%). The majority (54.4%) of participants had learned about AI outside their curriculum, mainly through social media (66%). Overall, 51.2% expressed positive attitudes toward AI, with Egypt showing the largest proportion of positive attitudes (59.1%). Although most participants viewed AI in healthcare positively (91%), significant variations were observed in other domains. The majority (77.6%) supported integrating AI in HPE, especially in Egypt (82.3%). A perceived negative impact of AI on patient trust was expressed by 43.5% of participants, particularly in Egypt (54.7%). Only 18.1% of participants were concerned about the impact of AI on future healthcare professionals, with the largest proportion from Egypt (33.0%). Some participants (34.4%) perceived AI integration as challenging, notably in the UAE (47.6%). Common barriers included lack of expert training (53%), awareness (50%), and interest in AI (41%).

CONCLUSION

This study clarified key considerations when integrating AI in HPE. Enhancing students' awareness and fostering innovation in an AI-driven medical landscape are crucial for effectively incorporating AI in HPE curricula.

Digital phenotyping from wearables using AI characterizes psychiatric disorders and identifies genetic associations

Psychiatric disorders are complex and influenced by both genetic and environmental factors. However, studying the full spectrum of these disorders is hindered by practical limitations on measuring human behavior. This highlights the need for novel technologies that can measure behavioral changes at an intermediate level between diagnosis and genotype. Wearable devices are a promising tool in precision medicine, since they can record physiological measurements over time in response to environmental stimuli and do so at low cost and minimal invasiveness. Here we analyzed wearable and genetic data from a cohort of the Adolescent Brain Cognitive Development study. We generated >250 wearable-derived features and used them as intermediate phenotypes in an interpretable AI modeling framework to assign risk scores and classify adolescents with psychiatric disorders. Our model identifies key physiological processes and leverages their temporal patterns to achieve a higher performance than has been previously possible. To investigate how these physiological processes relate to the underlying genetic architecture of psychiatric disorders, we also utilized these intermediate phenotypes in univariate and multivariate GWAS. We identified a total of 29 significant genetic loci and 52 psychiatric-associated genes, including ELFN1 and ADORA3. These results show that wearable-derived continuous features enable a more precise representation of psychiatric disorders and exhibit greater detection power compared to categorical diagnostic labels. In summary, we demonstrate how consumer wearable technology can facilitate dimensional approaches in precision psychiatry and uncover etiological linkages between behavior and genetics.

Artificial General Intelligence for the Detection of Neurodegenerative Disorders

Parkinson's disease and Alzheimer's disease are among the most common neurodegenerative disorders. These diseases are correlated with advancing age and are hence increasingly becoming prevalent in developed countries due to an increasingly aging demographic. Several tools are used to predict and diagnose these diseases, including pathological and genetic tests, radiological scans, and clinical examinations. Artificial intelligence is evolving to artificial general intelligence, which mimics the human learning process. Large language models can use an enormous volume of online and offline resources to gain knowledge and use it to perform different types of tasks. This work presents an understanding of two major neurodegenerative disorders, artificial general intelligence, and the efficacy of using artificial general intelligence in detecting and predicting these neurodegenerative disorders. A detailed discussion on detecting these neurodegenerative diseases using artificial general intelligence by analyzing diagnostic data is presented. An Internet of Things-based ubiquitous monitoring and treatment framework is presented. An outline for future research opportunities based on the challenges in this area is also presented.

Developing a suicide risk model for use in the Indian Health Service

We developed and evaluated an electronic health record (EHR)-based model for suicide risk specific to an American Indian patient population. Using EHR data for all patients over 18 with a visit between 1/1/2017 and 10/2/2021, we developed a model for the risk of a suicide attempt or death in the 90 days following a visit. Features included demographics, medications, diagnoses, and scores from relevant screening tools. We compared the predictive performance of logistic regression and random forest models against existing suicide screening, which was augmented to include the history of previous attempts or ideation. During the study, 16,835 patients had 331,588 visits, with 490 attempts and 37 deaths by suicide. The logistic regression and random forest models (area under the ROC (AUROC) 0.83 [0.80-0.86]; both models) performed better than enhanced screening (AUROC 0.64 [0.61-0.67]). These results suggest that an EHR-based suicide risk model can add value to existing practices at Indian Health Service clinics.

Artificial intelligence and predictive models for early detection of acute kidney injury: transforming clinical practice

Acute kidney injury (AKI) presents a significant clinical challenge due to its rapid progression to kidney failure, resulting in serious complications such as electrolyte imbalances, fluid overload, and the potential need for renal replacement therapy. Early detection and prediction of AKI can improve patient outcomes through timely interventions. This review was conducted as a narrative literature review, aiming to explore state-of-the-art models for early detection and prediction of AKI. We conducted a comprehensive review of findings from various studies, highlighting their strengths, limitations, and practical considerations for implementation in healthcare settings. We highlight the potential benefits and challenges of their integration into routine clinical care and emphasize the importance of establishing robust early-detection systems before the introduction of artificial intelligence (AI)-assisted prediction models. Advances in AI for AKI detection and prediction are examined, addressing their clinical applicability, challenges, and opportunities for routine implementation.

Building and Sustaining Public Trust in Health Data Sharing for Musculoskeletal Research: Semistructured Interview and Focus Group Study

BACKGROUND

Although many people are supportive of their deidentified health care data being used for research, concerns about privacy, safety, and security of health care data remain. There is low awareness about how data are used for research and related governance. Transparency about how health data are used for research is crucial for building public trust. One proposed solution is to ensure that affected communities are notified, particularly marginalized communities where there has previously been a lack of engagement and mistrust.

OBJECTIVE

This study aims to explore patient and public perspectives on the use of deidentified data from electronic health records for musculoskeletal research and to explore ways to build and sustain public trust in health data sharing for a research program (known as "the Data Jigsaw") piloting new ways of using and analyzing electronic health data. Views and perspectives about how best to engage with local communities informed the development of a public notification campaign about the research.

METHODS

Qualitative methods data were generated from 20 semistructured interviews and 8 focus groups, comprising 48 participants in total with musculoskeletal conditions or symptoms, including 3 carers. A presentation about the use of health data for research and examples from the specific research projects within the program were used to trigger discussion. We worked in partnership with a patient and public involvement group throughout the research and cofacilitated wider community engagement.

RESULTS

Respondents were supportive of their health care data being shared for research purposes, but there was low awareness about how electronic health records are used for research. Security and governance concerns about data sharing were noted, including collaborations with external companies and accessing social care records. Project examples from the Data Jigsaw program were viewed positively after respondents knew more about how their data were being used to improve patient care. A range of different methods to build and sustain trust were deemed necessary by participants. Information was requested about: data management; individuals with access to the data (including any collaboration with external companies); the National Health Service's national data opt-out; and research outcomes. It was considered important to enable in-person dialogue with affected communities in addition to other forms of information.

CONCLUSIONS

The findings have emphasized the need for transparency and awareness about health data sharing for research, and the value of tailoring this to reflect current and local research where residents might feel more invested in the focus of research and the use of local records. Thus, the provision for targeted information within affected communities with accessible messages and community-based dialogue could help to build and sustain public trust. These findings can also be extrapolated to other conditions beyond musculoskeletal conditions, making the findings relevant to a much wider community.