Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

The potential role for artificial intelligence in fracture risk prediction

Osteoporotic fractures are a major health challenge in older adults. Despite the availability of safe and effective therapies for osteoporosis, these therapies are underused in individuals at high risk for fracture, calling for better case-finding and fracture risk assessment strategies. Artificial intelligence (AI) and machine learning (ML) hold promise for enhancing identification of individuals at high risk for fracture by distilling useful features from high-dimensional data derived from medical records, imaging, and wearable devices. AI-ML could enable automated opportunistic screening for vertebral fractures and osteoporosis, home-based monitoring and intervention targeting lifestyle factors, and integration of multimodal features to leverage fracture prediction, ultimately aiding improved fracture risk assessment and individualised treatment. Optimism must be balanced with consideration for the explainability of AI-ML models, biases (including information inequity in numerically under-represented populations), model limitations, and net clinical benefit and workload impact. Clinical integration of AI-ML algorithms has the potential to transform osteoporosis management, offering a more personalised approach to reduce the burden of osteoporotic fractures.

Fairness in Predicting Cancer Mortality Across Racial Subgroups

Importance

Machine learning has potential to transform cancer care by helping clinicians prioritize patients for serious illness conversations. However, models need to be evaluated for unequal performance across racial groups (ie, racial bias) so that existing racial disparities are not exacerbated.

Objective

To evaluate whether racial bias exists in a predictive machine learning model that identifies 180-day cancer mortality risk among patients with solid malignant tumors.

Design, Setting, and Participants

In this cohort study, a machine learning model to predict cancer mortality for patients aged 21 years or older diagnosed with cancer between January 2016 and December 2021 was developed with a random forest algorithm using retrospective data from the Mount Sinai Health System cancer registry, Social Security Death Index, and electronic health records up to the date when databases were accessed for cohort extraction (February 2022).

Exposure

Race category.

Main Outcomes and Measures

The primary outcomes were model discriminatory performance (area under the receiver operating characteristic curve [AUROC], F1 score) among each race category (Asian, Black, Native American, White, and other or unknown) and fairness metrics (equal opportunity, equalized odds, and disparate impact) among each pairwise comparison of race categories. True-positive rate ratios represented equal opportunity; both true-positive and false-positive rate ratios, equalized odds; and the percentage of predictive positive rate ratios, disparate impact. All metrics were estimated as a proportion or ratio, with variability captured through 95% CIs. The prespecified criterion for the model's clinical use was a threshold of at least 80% for fairness metrics across different racial groups to ensure the model's prediction would not be biased against any specific race.

Results

The test validation dataset included 43 274 patients with balanced demographics. Mean (SD) age was 64.09 (14.26) years, with 49.6% older than 65 years. A total of 53.3% were female; 9.5%, Asian; 18.9%, Black; 0.1%, Native American; 52.2%, White; and 19.2%, other or unknown race; 0.1% had missing race data. A total of 88.9% of patients were alive, and 11.1% were dead. The AUROCs, F1 scores, and fairness metrics maintained reasonable concordance among the racial subgroups: the AUROCs ranged from 0.75 (95% CI, 0.72-0.78) for Asian patients and 0.75 (95% CI, 0.73-0.77) for Black patients to 0.77 (95% CI, 0.75-0.79) for patients with other or unknown race; F1 scores, from 0.32 (95% CI, 0.32-0.33) for White patients to 0.40 (95% CI, 0.39-0.42) for Black patients; equal opportunity ratios, from 0.96 (95% CI, 0.95-0.98) for Black patients compared with White patients to 1.02 (95% CI, 1.00-1.04) for Black patients compared with patients with other or unknown race; equalized odds ratios, from 0.87 (95% CI, 0.85-0.92) for Black patients compared with White patients to 1.16 (1.10-1.21) for Black patients compared with patients with other or unknown race; and disparate impact ratios, from 0.86 (95% CI, 0.82-0.89) for Black patients compared with White patients to 1.17 (95% CI, 1.12-1.22) for Black patients compared with patients with other or unknown race.

Conclusions and Relevance

In this cohort study, the lack of significant variation in performance or fairness metrics indicated an absence of racial bias, suggesting that the model fairly identified cancer mortality risk across racial groups. It remains essential to consistently review the model's application in clinical settings to ensure equitable patient care.

Recognition of Patient Gender: A Machine Learning Preliminary Analysis Using Heart Sounds from Children and Adolescents

Research has shown that X-rays and fundus images can classify gender, age group, and race, raising concerns about bias and fairness in medical AI applications. However, the potential for physiological sounds to classify sociodemographic traits has not been investigated. Exploring this gap is crucial for understanding the implications and ensuring fairness in the field of medical sound analysis. We aimed to develop classifiers to determine gender (men/women) based on heart sound recordings and using machine learning (ML). Data-driven ML analysis. We utilized the open-access CirCor DigiScope Phonocardiogram Dataset obtained from cardiac screening programs in Brazil. Volunteers < 21 years of age. Each participant completed a questionnaire and underwent a clinical examination, including electronic auscultation at four cardiac points: aortic (AV), mitral (MV), pulmonary (PV), and tricuspid (TV). We used Mel-frequency cepstral coefficients (MFCCs) to develop the ML classifiers. From each patient and from each auscultation sound recording, we extracted 10 MFCCs. In sensitivity analysis, we additionally extracted 20, 30, 40, and 50 MFCCs. The most effective gender classifier was developed using PV recordings (AUC ROC = 70.3%). The second best came from MV recordings (AUC ROC = 58.8%). AV and TV recordings produced classifiers with an AUC ROC of 56.4% and 56.1%, respectively. Using more MFCCs did not substantially improve the classifiers. It is possible to classify between males and females using phonocardiogram data. As health-related audio recordings become more prominent in ML applications, research is required to explore if these recordings contain signals that could distinguish sociodemographic features.

Equity in Using Artificial Intelligence Mortality Predictions to Target Goals of Care Documentation

BACKGROUND

Artificial intelligence (AI) algorithms are increasingly used to target patients with elevated mortality risk scores for goals-of-care (GOC) conversations.

OBJECTIVE

To evaluate the association between the presence or absence of AI-generated mortality risk scores with GOC documentation.

DESIGN

Retrospective cross-sectional study at one large academic medical center between July 2021 and December 2022.

PARTICIPANTS

Hospitalized adult patients with AI-defined Serious Illness Risk Indicator (SIRI) scores indicating > 30% 90-day mortality risk (defined as "elevated" SIRI) or no SIRI scores due to insufficient data.

INTERVENTION

A targeted intervention to increase GOC documentation for patients with AI-generated scores predicting elevated risk of mortality.

MAIN MEASURES

Odds ratios comparing GOC documentation for patients with elevated or no SIRI scores with similar severity of illness using propensity score matching and risk-adjusted mixed-effects logistic regression.

KEY RESULTS

Among 13,710 patients with elevated (n = 3643, 27%) or no (n = 10,067, 73%) SIRI scores, the median age was 64 years (SD 18). Twenty-five percent were non-White, 18% had Medicaid, 43% were admitted to an intensive care unit, and 11% died during admission. Patients lacking SIRI scores were more likely to be younger (median 60 vs. 72 years, p < 0.0001), be non-White (29% vs. 13%, p < 0.0001), and have Medicaid (22% vs. 9%, p < 0.0001). Patients with elevated versus no SIRI scores were more likely to have GOC documentation in the unmatched (aOR 2.5, p < 0.0001) and propensity-matched cohorts (aOR 2.1, p < 0.0001).

CONCLUSIONS

Using AI predictions of mortality to target GOC documentation may create differences in documentation prevalence between patients with and without AI mortality prediction scores with similar severity of illness. These finding suggest using AI to target GOC documentation may have the unintended consequence of disadvantaging severely ill patients lacking AI-generated scores from receiving targeted GOC documentation, including patients who are more likely to be non-White and have Medicaid insurance.

Dynamic and Transdiagnostic Risk Calculator Based on Natural Language Processing for the Prediction of Psychosis in Secondary Mental Health Care: Development and Internal-External Validation Cohort Study

BACKGROUND

Automatic transdiagnostic risk calculators can improve the detection of individuals at risk of psychosis. However, they rely on assessment at a single point in time and can be refined with dynamic modeling techniques that account for changes in risk over time.

METHODS

We included 158,139 patients (5007 events) who received a first index diagnosis of a nonorganic and nonpsychotic mental disorder within electronic health records from the South London and Maudsley National Health Service Foundation Trust between January 1, 2008, and October 8, 2021. A dynamic Cox landmark model was developed to estimate the 2-year risk of developing psychosis according to the TRIPOD (Transparent Reporting of a multivariate prediction model for Individual Prognosis or Diagnosis) statement. The dynamic model included 24 predictors extracted at 9 landmark points (baseline, 0, 6, 12, 24, 30, 36, 42, and 48 months): 3 demographic, 1 clinical, and 20 natural language processing-based symptom and substance use predictors. Performance was compared with a static Cox regression model with all predictors assessed at baseline only and indexed via discrimination (C-index), calibration (calibration plots), and potential clinical utility (decision curves) in internal-external validation.

RESULTS

The dynamic model improved discrimination performance from baseline compared with the static model (dynamic: C-index = 0.9; static: C-index = 0.87) and the final landmark point (dynamic: C-index = 0.79; static: C-index = 0.76). The dynamic model was also significantly better calibrated (calibration slope = 0.97-1.1) than the static model at later landmark points (≥24 months). Net benefit was higher for the dynamic than for the static model at later landmark points (≥24 months).

CONCLUSIONS

These findings suggest that dynamic prediction models can improve the detection of individuals at risk for psychosis in secondary mental health care settings.

Artificial Intelligence in Otolaryngology: Topics in Epistemology & Ethics

To fuel artificial intelligence (AI) potential in clinical practice in otolaryngology, researchers must understand its epistemic limitations, which are tightly linked to ethical dilemmas requiring careful consideration. AI tools are fundamentally opaque systems, though there are methods to increase explainability and transparency. Reproducibility and replicability limitations can be overcomed by sharing computing code, raw data, and data processing methodology. The risk of bias can be mitigated via algorithmic auditing, careful consideration of the training data, and advocating for a diverse AI workforce to promote algorithmic pluralism, reflecting our population's diverse values and preferences.

The potential of artificial intelligence to revolutionize health care delivery, research, and education in cardiac electrophysiology

The field of electrophysiology (EP) has benefited from numerous seminal innovations and discoveries that have enabled clinicians to deliver therapies and interventions that save lives and promote quality of life. The rapid pace of innovation in EP may be hindered by several challenges including the aging population with increasing morbidity, the availability of multiple costly therapies that, in many instances, confer minor incremental benefit, the limitations of healthcare reimbursement, the lack of response to therapies by some patients, and the complications of the invasive procedures performed. To overcome these challenges and continue on a steadfast path of transformative innovation, the EP community must comprehensively explore how artificial intelligence (AI) can be applied to healthcare delivery, research, and education and consider all opportunities in which AI can catalyze innovation; create workflow, research, and education efficiencies; and improve patient outcomes at a lower cost. In this white paper, we define AI and discuss the potential of AI to revolutionize the EP field. We also address the requirements for implementing, maintaining, and enhancing quality when using AI and consider ethical, operational, and regulatory aspects of AI implementation. This manuscript will be followed by several perspective papers that will expand on some of these topics.

Preparing for the bedside-optimizing a postpartum depression risk prediction model for clinical implementation in a health system

OBJECTIVE

We developed and externally validated a machine-learning model to predict postpartum depression (PPD) using data from electronic health records (EHRs). Effort is under way to implement the PPD prediction model within the EHR system for clinical decision support. We describe the pre-implementation evaluation process that considered model performance, fairness, and clinical appropriateness.

MATERIALS AND METHODS

We used EHR data from an academic medical center (AMC) and a clinical research network database from 2014 to 2020 to evaluate the predictive performance and net benefit of the PPD risk model. We used area under the curve and sensitivity as predictive performance and conducted a decision curve analysis. In assessing model fairness, we employed metrics such as disparate impact, equal opportunity, and predictive parity with the White race being the privileged value. The model was also reviewed by multidisciplinary experts for clinical appropriateness. Lastly, we debiased the model by comparing 5 different debiasing approaches of fairness through blindness and reweighing.

RESULTS

We determined the classification threshold through a performance evaluation that prioritized sensitivity and decision curve analysis. The baseline PPD model exhibited some unfairness in the AMC data but had a fair performance in the clinical research network data. We revised the model by fairness through blindness, a debiasing approach that yielded the best overall performance and fairness, while considering clinical appropriateness suggested by the expert reviewers.

DISCUSSION AND CONCLUSION

The findings emphasize the need for a thorough evaluation of intervention-specific models, considering predictive performance, fairness, and appropriateness before clinical implementation.

Simulated Misuse of Large Language Models and Clinical Credit Systems

Large language models (LLMs) have been proposed to support many healthcare tasks, including disease diagnostics and treatment personalization. While AI may be applied to assist or enhance the delivery of healthcare, there is also a risk of misuse. LLMs could be used to allocate resources based on unfair, inaccurate, or unjust criteria. For example, a social credit system uses big data to assess "trustworthiness" in society, punishing those who score poorly based on evaluation metrics defined only by a power structure (corporate entity, governing body). Such a system may be amplified by powerful LLMs which can rate individuals based on multimodal data - financial transactions, internet activity, and other behavioural inputs. Healthcare data is perhaps the most sensitive information which can be collected and could potentially be used to violate civil liberty via a "clinical credit system", which may include limiting or rationing access to standard care. This report simulates how clinical datasets might be exploited and proposes strategies to mitigate the risks inherent to the development of AI models for healthcare.

The Impact of Gender Diversity on Excellence in Pathology Research and Education

In this editorial, we inspect the critical role of gender diversity within the domain of pathology and its consequential impact on research innovation and clinical outcomes. The editorial commences with a historical overview of gender disparities in pathology, acknowledging advancements toward gender parity while highlighting persistent impediments to full inclusivity. The discourse emphasizes the intrinsic value of integrating diverse gender perspectives in research, illustrating how such inclusivity catalyzes innovation, mitigates research biases, and elevates the standard of patient care through a more comprehensive understanding of the field of pathology. Key barriers to gender diversity in pathology are systematically analysed, including disproportionate clinical burdens, time allocation conflicts due to societal roles, restricted access to specialized training, financial limitations, inadequate support networks, workplace discrimination, and the challenge of balancing family responsibilities with professional aspirations. We propose strategic interventions to address these barriers, advocating for increased awareness, diversity-focused training programs, and mechanisms for recognizing and rewarding the contributions of underrepresented genders in the field. Furthermore, we highlight exemplary initiatives that have successfully promoted gender diversity, such as the Johns Hopkins Pathology Department's outreach program, and the role of professional organizations, notably the American Society for Investigative Pathology and its "Women in Pathology" community, is discussed as pivotal in celebrating and advancing women's contributions to the field of pathology. In conclusion, we suggest that dismantling gender bias is imperative for realizing the full potential of pathology as a discipline. The editorial argues for a systemic embrace of gender diversity and inclusivity as fundamental to fostering research innovation, enhancing clinical practice, and ultimately improving patient outcomes. This scholarly examination calls for a concerted effort within the pathology community to integrate diverse perspectives, thereby enriching the field and contributing to the advancement of medical science.

Building towards an adolescent neural urbanome: Expanding environmental measures using linked external data (LED) in the ABCD study

Many recent studies have demonstrated that environmental contexts, both social and physical, have an important impact on child and adolescent neural and behavioral development. The adoption of geospatial methods, such as in the Adolescent Brain Cognitive Development (ABCD) Study, has facilitated the exploration of many environmental contexts surrounding participants' residential locations without creating additional burdens for research participants (i.e., youth and families) in neuroscience studies. However, as the number of linked databases increases, developing a framework that considers the various domains related to child and adolescent environments external to their home becomes crucial. Such a framework needs to identify structural contextual factors that may yield inequalities in children's built and natural environments; these differences may, in turn, result in downstream negative effects on children from historically minoritized groups. In this paper, we develop such a framework - which we describe as the "adolescent neural urbanome" - and use it to categorize newly geocoded information incorporated into the ABCD Study by the Linked External Data (LED) Environment & Policy Working Group. We also highlight important relationships between the linked measures and describe possible applications of the Adolescent Neural Urbanome. Finally, we provide a number of recommendations and considerations regarding the responsible use and communication of these data, highlighting the potential harm to historically minoritized groups through their misuse.