Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare

[Feminist perspectives in German-language medical ethics: a review and three hypotheses]

Im internationalen Diskurs sind feministische Perspektiven auf die Medizinethik bereits etabliert. Demgegenüber scheinen diese bislang nur vereinzelt in den deutschsprachigen medizinethischen Diskurs eingebracht zu werden. In diesem Artikel untersuchen wir, welche feministischen Perspektiven im deutschsprachigen medizinethischen Diskurs vertreten sind, und schlagen weitere Ansätze für eine feministische Medizinethik vor.

Zu diesem Zweck zeichnen wir mittels einer systematisierten Literaturrecherche feministische Perspektiven im deutschsprachigen medizinethischen Diskurs seit der Etablierung der Medizinethik als eigenständiger institutionalisierter Disziplin nach. Wir analysieren, welche Themen bereits innerhalb der Medizinethik aus einer feministischen Perspektive untersucht worden sind, und identifizieren Leerstellen. Basierend auf der Literaturrecherche, unseren eigenen Vorarbeiten sowie der Zusammenarbeit in der Arbeitsgruppe in der Akademie für Ethik in der Medizin „Feministische Perspektiven in der Bio- und Medizinethik“ stellen wir drei Thesen vor, die aus unserer Sicht einer Weiterentwicklung des deutschsprachigen medizinethischen Diskurses dienen können. Die erste These bezieht sich auf die Ziele feministischer Medizinethiken und besagt, dass diese (epistemische) Gerechtigkeit anstreben. Die zweite These stellt zentrale Eigenschaften von feministischen Medizinethiken als kritisch und kontext-sensibel heraus. In der dritten These diskutieren wir Intersektionalität und Postkolonialismus als theoretische Ansätze, die zu einer epistemisch gerechten, kritischen und kontext-sensiblen Medizinethik beitragen können. Wir argumentieren, dass feministische Perspektiven grundständig verankert werden sollten. Der Artikel schließt mit einem Ausblick auf die Arbeit der im letzten Jahr gegründeten Arbeitsgruppe in der Akademie für Ethik in der Medizin „Feministische Perspektiven in der Bio- und Medizinethik“.

Artificial intelligence for multimodal data integration in oncology

In oncology, the patient state is characterized by a whole spectrum of modalities, ranging from radiology, histology, and genomics to electronic health records. Current artificial intelligence (AI) models operate mainly in the realm of a single modality, neglecting the broader clinical context, which inevitably diminishes their potential. Integration of different data modalities provides opportunities to increase robustness and accuracy of diagnostic and prognostic models, bringing AI closer to clinical practice. AI models are also capable of discovering novel patterns within and across modalities suitable for explaining differences in patient outcomes or treatment resistance. The insights gleaned from such models can guide exploration studies and contribute to the discovery of novel biomarkers and therapeutic targets. To support these advances, here we present a synopsis of AI methods and strategies for multimodal data fusion and association discovery. We outline approaches for AI interpretability and directions for AI-driven exploration through multimodal data interconnections. We examine challenges in clinical adoption and discuss emerging solutions.

Artificial intelligence in pancreatic cancer

Pancreatic cancer is the deadliest disease, with a five-year overall survival rate of just 11%. The pancreatic cancer patients diagnosed with early screening have a median overall survival of nearly ten years, compared with 1.5 years for those not diagnosed with early screening. Therefore, early diagnosis and early treatment of pancreatic cancer are particularly critical. However, as a rare disease, the general screening cost of pancreatic cancer is high, the accuracy of existing tumor markers is not enough, and the efficacy of treatment methods is not exact. In terms of early diagnosis, artificial intelligence technology can quickly locate high-risk groups through medical images, pathological examination, biomarkers, and other aspects, then screening pancreatic cancer lesions early. At the same time, the artificial intelligence algorithm can also be used to predict the survival time, recurrence risk, metastasis, and therapy response which could affect the prognosis. In addition, artificial intelligence is widely used in pancreatic cancer health records, estimating medical imaging parameters, developing computer-aided diagnosis systems, etc. Advances in AI applications for pancreatic cancer will require a concerted effort among clinicians, basic scientists, statisticians, and engineers. Although it has some limitations, it will play an essential role in overcoming pancreatic cancer in the foreseeable future due to its mighty computing power.

Female Authorship Trends Among Articles About Artificial Intelligence in North American Radiology Journals

Purpose: To examine trends in female authorship of peer-reviewed North American radiology articles centred around artificial intelligence (AI). Method: A bibliographic search was conducted for all AI-related articles published in four North American radiology journals. Collected data included the genders of the first and last (senior) authors, year and country. We compared the trends of female authorship using Pearson chi-square, Fisher exact tests and logistic regression models. Results: 453 articles met the inclusion criteria. Among these, 107 (22.3%) had a female first author and 97 (27.3%) had a female senior author. Female first authors were over three times more likely to publish with a female senior author. Among the four journals, the CARJ had the highest proportion of female senior authors at 45.5%. The only significant temporal trend identified was an increase over the years in female senior authors in Radiology. Twenty-four countries contributed to the included articles, with the largest contributors being the United States (n = 290) and Canada (n = 30). Of the countries contributing more than 15 articles, there were none with above 50% female authorship. Conclusions: Female authors are underrepresented in AI-related radiology literature. However, there has been an encouraging recent increase in female authorship in AI-related radiology articles trending towards significance. There is a great opportunity to improve female representation in AI with intentional mentorship and recruitment. We urge more platforms for female voices in radiology as AI becomes increasingly integrated into the radiology community.

Conformal Prediction in Clinical Medical Sciences

The use of machine learning (ML) and artificial intelligence (AI) applications in medicine has attracted a great deal of attention in the medical literature, but little is known about how to use Conformal Predictions (CP) to assess the accuracy of individual predictions in clinical applications. We performed a comprehensive search in SCOPUS® to find papers reporting the use of CP in clinical applications. We identified 14 papers reporting the use of CP for clinical applications, and we briefly describe the methods and results reported in these papers. The literature reviewed shows that CP methods can be used in clinical applications to provide important insight into the accuracy of individual predictions. Unfortunately, the review also shows that most of the studies have been performed in isolation, without input from practicing clinicians, not providing comparisons among different approaches and not considering important socio-technical considerations leading to clinical adoption.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Foundations for fairness in digital health apps

Digital mental health applications promise scalable and cost-effective solutions to mitigate the gap between the demand and supply of mental healthcare services. However, very little attention is paid on differential impact and potential discrimination in digital mental health services with respect to different sensitive user groups (e.g., race, age, gender, ethnicity, socio-economic status) as the extant literature as well as the market lack the corresponding evidence. In this paper, we outline a 7-step model to assess algorithmic discrimination in digital mental health services, focusing on algorithmic bias assessment and differential impact. We conduct a pilot analysis with 610 users of the model applied on a digital wellbeing service called Foundations that incorporates a rich set of 150 proposed activities designed to increase wellbeing and reduce stress. We further apply the 7-step model on the evaluation of two algorithms that could extend the current service: monitoring step-up model, and a popularity-based activities recommender system. This study applies an algorithmic fairness analysis framework for digital mental health and explores differences in the outcome metrics for the interventions, monitoring model, and recommender engine for the users of different age, gender, type of work, country of residence, employment status and monthly income.

Systematic Review Registration: The study with main hypotheses is registered at: https://osf.io/hvtf8

Sex- and age-specific associations between cardiometabolic risk and white matter brain age in the UK Biobank cohort

Cardiometabolic risk (CMR) factors are associated with accelerated brain aging and increased risk for sex-dimorphic illnesses such as Alzheimer's disease (AD). Yet, it is unknown how CMRs interact with sex and apolipoprotein E-ϵ4 (APOE4), a known genetic risk factor for AD, to influence brain age across different life stages. Using age prediction based on multi-shell diffusion-weighted imaging data in 21,308 UK Biobank participants, we investigated whether associations between white matter Brain Age Gap (BAG) and body mass index (BMI), waist-to-hip ratio (WHR), body fat percentage (BF%), and APOE4 status varied (i) between males and females, (ii) according to age at menopause in females, and (iii) across different age groups in males and females. We report sex differences in associations between BAG and all three CMRs, with stronger positive associations among males compared to females. Independent of APOE4 status, higher BAG (older brain age relative to chronological age) was associated with greater BMI, WHR, and BF% in males, whereas in females, higher BAG was associated with greater WHR, but not BMI and BF%. These divergent associations were most prominent within the oldest group of females (66-81 years), where greater BF% was linked to lower BAG. Earlier menopause transition was associated with higher BAG, but no interactions were found with CMRs. In conclusion, the findings point to sex- and age-specific associations between CMRs and brain age. Incorporating sex as a factor of interest in studies addressing CMR may promote sex-specific precision medicine, consequently improving health care for both males and females.

Sex trouble: Sex/gender slippage, sex confusion, and sex obsession in machine learning using electronic health records

False assumptions that sex and gender are binary, static, and concordant are deeply embedded in the medical system. As machine learning researchers use medical data to build tools to solve novel problems, understanding how existing systems represent sex/gender incorrectly is necessary to avoid perpetuating harm. In this perspective, we identify and discuss three factors to consider when working with sex/gender in research: "sex/gender slippage," the frequent substitution of sex and sex-related terms for gender and vice versa; "sex confusion," the fact that any given sex variable holds many different potential meanings; and "sex obsession," the idea that the relevant variable for most inquiries related to sex/gender is sex assigned at birth. We then explore how these phenomena show up in medical machine learning research using electronic health records, with a specific focus on HIV risk prediction. Finally, we offer recommendations about how machine learning researchers can engage more carefully with questions of sex/gender.

Effect of imbalanced sampling and missing data on associations between gender norms and risk of adolescent HIV

BACKGROUND

Despite strides towards gender equality, inequalities persist or remain unstudied, due potentially to data gaps. Although mapped, the effects of key data gaps remain unknown. This study provides a framework to measure effects of gender- and age-imbalanced and missing covariate data on gender-health research. The framework is demonstrated using a previously studied pathway for effects of pre-marital sex norms among adults on adolescent HIV risk.

METHODS

After identifying gender-age-imbalanced Demographic and Health Survey (DHS) datasets, we resampled responses and restricted covariate data from a relatively complete, balanced dataset derived from the 2007 Zambian DHS to replicate imbalanced gender-age sampling and covariate missingness. Differences in model outcomes due to sampling were measured using tests for interaction. Missing covariate effects were measured by comparing fully-adjusted and reduced model fitness.

FINDINGS

We simulated data from 25 DHS surveys across 20 countries from 2005-2014 on four sex-stratified models for pathways of adult attitude-behaviour discordance regarding pre-marital sex and adolescent risk of HIV. On average, across gender-age-imbalanced surveys, males comprised 29.6% of responses compared to 45.3% in the gender-balanced dataset. Gender-age-imbalanced sampling significantly affected regression coefficients in 40% of model-scenarios (N = 40 of 100) and biased relative-risk estimates away from gender-age-balanced sampling outcomes in 46% (N = 46) of model-scenarios. Model fitness was robust to covariate removal with minor effects on male HIV models. No consistent trends were observed between sampling distribution and risk of biased outcomes.

INTERPRETATION

Gender-health model outcomes may be affected by sampling gender-age-imbalanced data and less-so by missing covariates. Although occasionally attenuated, the effect magnitude of gender-age-imbalanced sampling is variable and may mask true associations, thus misinforming policy dialogue. We recommend future surveys improve balanced gender-age sampling to promote research reliability.

FUNDING

Bill & Melinda Gates Foundation grant OPP1140262 to Stanford University.

Performance of hybrid artificial intelligence in determining candidacy for lumbar stenosis surgery

PURPOSE

Lumbar spinal stenosis (LSS) is a condition affecting several hundreds of thousands of adults in the United States each year and is associated with significant economic burden. The current decision-making practice to determine surgical candidacy for LSS is often subjective and clinician specific. In this study, we hypothesize that the performance of artificial intelligence (AI) methods could prove comparable in terms of prediction accuracy to that of a panel of spine experts.

METHODS

We propose a novel hybrid AI model which computes the probability of spinal surgical recommendations for LSS, based on patient demographic factors, clinical symptom manifestations, and MRI findings. The hybrid model combines a random forest model trained from medical vignette data reviewed by surgeons, with an expert Bayesian network model built from peer-reviewed literature and the expert opinions of a multidisciplinary team in spinal surgery, rehabilitation medicine, interventional and diagnostic radiology. Sets of 400 and 100 medical vignettes reviewed by surgeons were used for training and testing.

RESULTS

The model demonstrated high predictive accuracy, with a root mean square error (RMSE) between model predictions and ground truth of 0.0964, while the average RMSE between individual doctor's recommendations and ground truth was 0.1940. For dichotomous classification, the AUROC and Cohen's kappa were 0.9266 and 0.6298, while the corresponding average metrics based on individual doctor's recommendations were 0.8412 and 0.5659, respectively.

CONCLUSIONS

Our results suggest that AI can be used to automate the evaluation of surgical candidacy for LSS with performance comparable to a multidisciplinary panel of physicians.

Erroneous data: The Achilles' heel of AI and personalized medicine

This paper reviews dilemmas and implications of erroneous data for clinical implementation of AI. It is well-known that if erroneous and biased data are used to train AI, there is a risk of systematic error. However, even perfectly trained AI applications can produce faulty outputs if fed with erroneous inputs. To counter such problems, we suggest 3 steps: (1) AI should focus on data of the highest quality, in essence paraclinical data and digital images, (2) patients should be granted simple access to the input data that feed the AI, and granted a right to request changes to erroneous data, and (3) automated high-throughput methods for error-correction should be implemented in domains with faulty data when possible. Also, we conclude that erroneous data is a reality even for highly reputable Danish data sources, and thus, legal framework for the correction of errors is universally needed.

Ethics of AI in Radiology: A Review of Ethical and Societal Implications

Artificial intelligence (AI) is being applied in medicine to improve healthcare and advance health equity. The application of AI-based technologies in radiology is expected to improve diagnostic performance by increasing accuracy and simplifying personalized decision-making. While this technology has the potential to improve health services, many ethical and societal implications need to be carefully considered to avoid harmful consequences for individuals and groups, especially for the most vulnerable populations. Therefore, several questions are raised, including (1) what types of ethical issues are raised by the use of AI in medicine and biomedical research, and (2) how are these issues being tackled in radiology, especially in the case of breast cancer? To answer these questions, a systematic review of the academic literature was conducted. Searches were performed in five electronic databases to identify peer-reviewed articles published since 2017 on the topic of the ethics of AI in radiology. The review results show that the discourse has mainly addressed expectations and challenges associated with medical AI, and in particular bias and black box issues, and that various guiding principles have been suggested to ensure ethical AI. We found that several ethical and societal implications of AI use remain underexplored, and more attention needs to be paid to addressing potential discriminatory effects and injustices. We conclude with a critical reflection on these issues and the identified gaps in the discourse from a philosophical and STS perspective, underlining the need to integrate a social science perspective in AI developments in radiology in the future.

A bias evaluation checklist for predictive models and its pilot application for 30-day hospital readmission models

OBJECTIVE

Health care providers increasingly rely upon predictive algorithms when making important treatment decisions, however, evidence indicates that these tools can lead to inequitable outcomes across racial and socio-economic groups. In this study, we introduce a bias evaluation checklist that allows model developers and health care providers a means to systematically appraise a model's potential to introduce bias.

MATERIALS AND METHODS

Our methods include developing a bias evaluation checklist, a scoping literature review to identify 30-day hospital readmission prediction models, and assessing the selected models using the checklist.

RESULTS

We selected 4 models for evaluation: LACE, HOSPITAL, Johns Hopkins ACG, and HATRIX. Our assessment identified critical ways in which these algorithms can perpetuate health care inequalities. We found that LACE and HOSPITAL have the greatest potential for introducing bias, Johns Hopkins ACG has the most areas of uncertainty, and HATRIX has the fewest causes for concern.

DISCUSSION

Our approach gives model developers and health care providers a practical and systematic method for evaluating bias in predictive models. Traditional bias identification methods do not elucidate sources of bias and are thus insufficient for mitigation efforts. With our checklist, bias can be addressed and eliminated before a model is fully developed or deployed.

CONCLUSION

The potential for algorithms to perpetuate biased outcomes is not isolated to readmission prediction models; rather, we believe our results have implications for predictive models across health care. We offer a systematic method for evaluating potential bias with sufficient flexibility to be utilized across models and applications.

The Potential and Emerging Role of Quantitative Imaging Biomarkers for Cancer Characterization

Simple Summary

Modern, personalized therapy approaches are increasingly changing advanced cancer into a chronic disease. Compared to imaging, novel omics methodologies in molecular biology have already achieved an individual characterization of cancerous lesions. With quantitative imaging biomarkers, analyzed by radiomics or deep learning, an imaging-based assessment of tumoral biology can be brought into clinical practice. Combining these with other non-invasive methods, e.g., liquid profiling, could allow for more individual decision making regarding therapies and applications.

Abstract

Similar to the transformation towards personalized oncology treatment, emerging techniques for evaluating oncologic imaging are fostering a transition from traditional response assessment towards more comprehensive cancer characterization via imaging. This development can be seen as key to the achievement of truly personalized and optimized cancer diagnosis and treatment. This review gives a methodological introduction for clinicians interested in the potential of quantitative imaging biomarkers, treating of radiomics models, texture visualization, convolutional neural networks and automated segmentation, in particular. Based on an introduction to these methods, clinical evidence for the corresponding imaging biomarkers—(i) dignity and etiology assessment; (ii) tumoral heterogeneity; (iii) aggressiveness and response; and (iv) targeting for biopsy and therapy—is summarized. Further requirements for the clinical implementation of these imaging biomarkers and the synergistic potential of personalized molecular cancer diagnostics and liquid profiling are discussed.

A Review of Artificial Intelligence and Robotics in Transformed Health Ecosystems

Health care is shifting toward become proactive according to the concept of P5 medicine-a predictive, personalized, preventive, participatory and precision discipline. This patient-centered care heavily leverages the latest technologies of artificial intelligence (AI) and robotics that support diagnosis, decision making and treatment. In this paper, we present the role of AI and robotic systems in this evolution, including example use cases. We categorize systems along multiple dimensions such as the type of system, the degree of autonomy, the care setting where the systems are applied, and the application area. These technologies have already achieved notable results in the prediction of sepsis or cardiovascular risk, the monitoring of vital parameters in intensive care units, or in the form of home care robots. Still, while much research is conducted around AI and robotics in health care, adoption in real world care settings is still limited. To remove adoption barriers, we need to address issues such as safety, security, privacy and ethical principles; detect and eliminate bias that could result in harmful or unfair clinical decisions; and build trust in and societal acceptance of AI.

Explanations of Machine Learning Models in Repeated Nested Cross-Validation: An Application in Age Prediction Using Brain Complexity Features

SHAP (Shapley additive explanations) is a framework for explainable AI that makes explanations locally and globally. In this work, we propose a general method to obtain representative SHAP values within a repeated nested cross-validation procedure and separately for the training and test sets of the different cross-validation rounds to assess the real generalization abilities of the explanations. We applied this method to predict individual age using brain complexity features extracted from MRI scans of 159 healthy subjects. In particular, we used four implementations of the fractal dimension (FD) of the cerebral cortex—a measurement of brain complexity. Representative SHAP values highlighted that the most recent implementation of the FD had the highest impact over the others and was among the top-ranking features for predicting age. SHAP rankings were not the same in the training and test sets, but the top-ranking features were consistent. In conclusion, we propose a method—and share all the source code—that allows a rigorous assessment of the SHAP explanations of a trained model in a repeated nested cross-validation setting.

Digital biomarkers and sex impacts in Alzheimer's disease management - potential utility for innovative 3P medicine approach

Digital biomarkers are defined as objective, quantifiable physiological and behavioral data that are collected and measured by means of digital devices. Their use has revolutionized clinical research by enabling high-frequency, longitudinal, and sensitive measurements. In the field of neurodegenerative diseases, an example of a digital biomarker-based technology is instrumental activities of daily living (iADL) digital medical application, a predictive biomarker of conversion from mild cognitive impairment (MCI) due to Alzheimer's disease (AD) to dementia due to AD in individuals aged 55 + . Digital biomarkers show promise to transform clinical practice. Nevertheless, their use may be affected by variables such as demographics, genetics, and phenotype. Among these factors, sex is particularly important in Alzheimer's, where men and women present with different symptoms and progression patterns that impact diagnosis. In this study, we explore sex differences in Altoida's digital medical application in a sample of 568 subjects consisting of a clinical dataset (MCI and dementia due to AD) and a healthy population. We found that a biological sex-classifier, built on digital biomarker features captured using Altoida's application, achieved a 75% ROC-AUC (receiver operating characteristic - area under curve) performance in predicting biological sex in healthy individuals, indicating significant differences in neurocognitive performance signatures between males and females. The performance dropped when we applied this classifier to more advanced stages on the AD continuum, including MCI and dementia, suggesting that sex differences might be disease-stage dependent. Our results indicate that neurocognitive performance signatures built on data from digital biomarker features are different between men and women. These results stress the need to integrate traditional approaches to dementia research with digital biomarker technologies and personalized medicine perspectives to achieve more precise predictive diagnostics, targeted prevention, and customized treatment of cognitive decline.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13167-022-00284-3.

Diagnostic classification of Parkinson’s disease based on non-motor manifestations and machine learning strategies

Non-motor manifestations of Parkinson’s disease (PD) appear early and have a significant impact on the quality of life of patients, but few studies have evaluated their predictive potential with machine learning algorithms. We evaluated 9 algorithms for discriminating PD patients from controls using a wide collection of non-motor clinical PD features from two databases: Biocruces (96 subjects) and PPMI (687 subjects). In addition, we evaluated whether the combination of both databases could improve the individual results. For each database 2 versions with different granularity were created and a feature selection process was performed. We observed that most of the algorithms were able to detect PD patients with high accuracy (>80%). Support Vector Machine and Multi-Layer Perceptron obtained the best performance, with an accuracy of 86.3% and 84.7%, respectively. Likewise, feature selection led to a significant reduction in the number of variables and to better performance. Besides, the enrichment of Biocruces database with data from PPMI moderately benefited the performance of the classification algorithms, especially the recall and to a lesser extent the accuracy, while the precision worsened slightly. The use of interpretable rules obtained by the RIPPER algorithm showed that simply using two variables (autonomic manifestations and olfactory dysfunction), it was possible to achieve an accuracy of 84.4%. Our study demonstrates that the analysis of non-motor parameters of PD through machine learning techniques can detect PD patients with high accuracy and recall, and allows us to select the most discriminative non-motor variables to create potential tools for PD screening.

Regulating the Safety of Health-Related Artificial Intelligence

This article analyzes whether Canada's present approach to regulating health-related artificial intelligence (AI) can address relevant safety-related challenges. Focusing primarily on Health Canada's regulation of medical devices with AI, it examines whether the existing regulatory approach can adequately address general safety concerns, as well as those related to algorithmic bias and challenges posed by the intersections of these concerns with privacy and security interests. It identifies several issues and proposes reforms that aim to ensure that Canadians can access beneficial AI while keeping unsafe products off Canadian markets and motivating safe, effective use of AI products for appropriate purposes and populations.

Does imbalance in chest X-ray datasets produce biased deep learning approaches for COVID-19 screening?

Background

The health crisis resulting from the global COVID-19 pandemic highlighted more than ever the need for rapid, reliable and safe methods of diagnosis and monitoring of respiratory diseases. To study pulmonary involvement in detail, one of the most common resources is the use of different lung imaging modalities (like chest radiography) to explore the possible affected areas.

Methods

The study of patient characteristics like sex and age in pathologies of this type is crucial for gaining knowledge of the disease and for avoiding biases due to the clear scarcity of data when developing representative systems. In this work, we performed an analysis of these factors in chest X-ray images to identify biases. Specifically, 11 imbalance scenarios were defined with female and male COVID-19 patients present in different proportions for the sex analysis, and 6 scenarios where only one specific age range was used for training for the age factor. In each study, 3 different approaches for automatic COVID-19 screening were used: Normal vs COVID-19, Pneumonia vs COVID-19 and Non-COVID-19 vs COVID-19. The study was validated using two public chest X-ray datasets, allowing a reliable analysis to support the clinical decision-making process.

Results

The results for the sex-related analysis indicate this factor slightly affects the system in the Normal VS COVID-19 and Pneumonia VS COVID-19 approaches, although the identified differences are not relevant enough to worsen considerably the system. Regarding the age-related analysis, this factor was observed to be influencing the system in a more consistent way than the sex factor, as it was present in all considered scenarios. However, this worsening does not represent a major factor, as it is not of great magnitude.

Conclusions

Multiple studies have been conducted in other fields in order to determine if certain patient characteristics such as sex or age influenced these deep learning systems. However, to the best of our knowledge, this study has not been done for COVID-19 despite the urgency and lack of COVID-19 chest x-ray images. The presented results evidenced that the proposed methodology and tested approaches allow a robust and reliable analysis to support the clinical decision-making process in this pandemic scenario.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-022-01578-w).

Investigating for bias in healthcare algorithms: a sex-stratified analysis of supervised machine learning models in liver disease prediction

OBJECTIVES

The Indian Liver Patient Dataset (ILPD) is used extensively to create algorithms that predict liver disease. Given the existing research describing demographic inequities in liver disease diagnosis and management, these algorithms require scrutiny for potential biases. We address this overlooked issue by investigating ILPD models for sex bias.

METHODS

Following our literature review of ILPD papers, the models reported in existing studies are recreated and then interrogated for bias. We define four experiments, training on sex-unbalanced/balanced data, with and without feature selection. We build random forests (RFs), support vector machines (SVMs), Gaussian Naïve Bayes and logistic regression (LR) classifiers, running experiments 100 times, reporting average results with SD.

RESULTS

We reproduce published models achieving accuracies of >70% (LR 71.31% (2.37 SD) - SVM 79.40% (2.50 SD)) and demonstrate a previously unobserved performance disparity. Across all classifiers females suffer from a higher false negative rate (FNR). Presently, RF and LR classifiers are reported as the most effective models, yet in our experiments they demonstrate the greatest FNR disparity (RF; -21.02%; LR; -24.07%).

DISCUSSION

We demonstrate a sex disparity that exists in published ILPD classifiers. In practice, the higher FNR for females would manifest as increased rates of missed diagnosis for female patients and a consequent lack of appropriate care. Our study demonstrates that evaluating biases in the initial stages of machine learning can provide insights into inequalities in current clinical practice, reveal pathophysiological differences between the male and females, and can mitigate the digitisation of inequalities into algorithmic systems.

CONCLUSION

Our findings are important to medical data scientists, clinicians and policy-makers involved in the implementation medical artificial intelligence systems. An awareness of the potential biases of these systems is essential in preventing the digital exacerbation of healthcare inequalities.

Annual Research Review: Translational machine learning for child and adolescent psychiatry

Children and adolescents could benefit from the use of predictive tools that facilitate personalized diagnoses, prognoses, and treatment selection. Such tools have not yet been deployed using traditional statistical methods, potentially due to the limitations of the paradigm and the need to leverage large amounts of digital data. This review will suggest that a machine learning approach could address these challenges and is designed to introduce new readers to the background, methods, and results in the field. A rationale is first introduced followed by an outline of fundamental elements of machine learning approaches. To provide an overview of the use of the techniques in child and adolescent literature, a scoping review of broad trends is then presented. Selected studies are also highlighted in order to draw attention to research areas that are closest to translation and studies that exhibit a high degree of experimental innovation. Limitations to the research, and machine learning approaches generally, are outlined in the penultimate section highlighting issues related to sample sizes, validation, clinical utility, and ethical challenges. Finally, future directions are discussed that could enhance the possibility of clinical implementation and address specific questions relevant to the child and adolescent psychiatry. The review gives a broad overview of the machine learning paradigm in order to highlight the benefits of a shift in perspective towards practically oriented statistical solutions that aim to improve clinical care of children and adolescents.

On the reproducibility of enzyme reactions and kinetic modelling

Enzyme reactions are highly dependent on reaction conditions. To ensure reproducibility of enzyme reaction parameters, experiments need to be carefully designed and kinetic modeling meticulously executed. Furthermore, to enable quality control of enzyme reaction parameters, the experimental conditions, the modeling process as well as the raw data need to be reported comprehensively. By taking these steps, enzyme reaction parameters can be open and FAIR (findable, accessible, interoperable, re-usable) as well as repeatable, replicable and reproducible. This review discusses these requirements and provides a practical guide to designing initial rate experiments for the determination of enzyme reaction parameters and gives an open, FAIR and re-editable example of the kinetic modeling of an enzyme reaction. Both the guide and example are scripted with Python in Jupyter Notebooks and are publicly available (https://fairdomhub.org/investigations/483/snapshots/1). Finally, the prerequisites of automated data analysis and machine learning algorithms are briefly discussed to provide further motivation for the comprehensive, open and FAIR reporting of enzyme reaction parameters.

The Role of Artificial Intelligence in Early Cancer Diagnosis

Improving the proportion of patients diagnosed with early-stage cancer is a key priority of the World Health Organisation. In many tumour groups, screening programmes have led to improvements in survival, but patient selection and risk stratification are key challenges. In addition, there are concerns about limited diagnostic workforces, particularly in light of the COVID-19 pandemic, placing a strain on pathology and radiology services. In this review, we discuss how artificial intelligence algorithms could assist clinicians in (1) screening asymptomatic patients at risk of cancer, (2) investigating and triaging symptomatic patients, and (3) more effectively diagnosing cancer recurrence. We provide an overview of the main artificial intelligence approaches, including historical models such as logistic regression, as well as deep learning and neural networks, and highlight their early diagnosis applications. Many data types are suitable for computational analysis, including electronic healthcare records, diagnostic images, pathology slides and peripheral blood, and we provide examples of how these data can be utilised to diagnose cancer. We also discuss the potential clinical implications for artificial intelligence algorithms, including an overview of models currently used in clinical practice. Finally, we discuss the potential limitations and pitfalls, including ethical concerns, resource demands, data security and reporting standards.

Machine learning to predict progression of non-alcoholic fatty liver to non-alcoholic steatohepatitis or fibrosis

Background

Non-alcoholic fatty liver (NAFL) can progress to the severe subtype non-alcoholic steatohepatitis (NASH) and/or fibrosis, which are associated with increased morbidity, mortality, and healthcare costs. Current machine learning studies detect NASH; however, this study is unique in predicting the progression of NAFL patients to NASH or fibrosis.

Aim

To utilize clinical information from NAFL-diagnosed patients to predict the likelihood of progression to NASH or fibrosis.

Methods

Data were collected from electronic health records of patients receiving a first-time NAFL diagnosis. A gradient boosted machine learning algorithm (XGBoost) as well as logistic regression (LR) and multi-layer perceptron (MLP) models were developed. A five-fold cross-validation grid search was utilized for hyperparameter optimization of variables, including maximum tree depth, learning rate, and number of estimators. Predictions of patients likely to progress to NASH or fibrosis within 4 years of initial NAFL diagnosis were made using demographic features, vital signs, and laboratory measurements.

Results

The XGBoost algorithm achieved area under the receiver operating characteristic (AUROC) values of 0.79 for prediction of progression to NASH and 0.87 for fibrosis on both hold-out and external validation test sets. The XGBoost algorithm outperformed the LR and MLP models for both NASH and fibrosis prediction on all metrics.

Conclusion

It is possible to accurately identify newly diagnosed NAFL patients at high risk of progression to NASH or fibrosis. Early identification of these patients may allow for increased clinical monitoring, more aggressive preventative measures to slow the progression of NAFL and fibrosis, and efficient clinical trial enrollment.

Sources of bias in artificial intelligence that perpetuate healthcare disparities-A global review

BACKGROUND

While artificial intelligence (AI) offers possibilities of advanced clinical prediction and decision-making in healthcare, models trained on relatively homogeneous datasets, and populations poorly-representative of underlying diversity, limits generalisability and risks biased AI-based decisions. Here, we describe the landscape of AI in clinical medicine to delineate population and data-source disparities.

METHODS

We performed a scoping review of clinical papers published in PubMed in 2019 using AI techniques. We assessed differences in dataset country source, clinical specialty, and author nationality, sex, and expertise. A manually tagged subsample of PubMed articles was used to train a model, leveraging transfer-learning techniques (building upon an existing BioBERT model) to predict eligibility for inclusion (original, human, clinical AI literature). Of all eligible articles, database country source and clinical specialty were manually labelled. A BioBERT-based model predicted first/last author expertise. Author nationality was determined using corresponding affiliated institution information using Entrez Direct. And first/last author sex was evaluated using the Gendarize.io API.

RESULTS

Our search yielded 30,576 articles, of which 7,314 (23.9%) were eligible for further analysis. Most databases came from the US (40.8%) and China (13.7%). Radiology was the most represented clinical specialty (40.4%), followed by pathology (9.1%). Authors were primarily from either China (24.0%) or the US (18.4%). First and last authors were predominately data experts (i.e., statisticians) (59.6% and 53.9% respectively) rather than clinicians. And the majority of first/last authors were male (74.1%).

INTERPRETATION

U.S. and Chinese datasets and authors were disproportionately overrepresented in clinical AI, and almost all of the top 10 databases and author nationalities were from high income countries (HICs). AI techniques were most commonly employed for image-rich specialties, and authors were predominantly male, with non-clinical backgrounds. Development of technological infrastructure in data-poor regions, and diligence in external validation and model re-calibration prior to clinical implementation in the short-term, are crucial in ensuring clinical AI is meaningful for broader populations, and to avoid perpetuating global health inequity.

Cardiovascular Disease Screening in Women: Leveraging Artificial Intelligence and Digital Tools

Cardiovascular disease remains the leading cause of death in women. Given accumulating evidence on sex- and gender-based differences in cardiovascular disease development and outcomes, the need for more effective approaches to screening for risk factors and phenotypes in women is ever urgent. Public health surveillance and health care delivery systems now continuously generate massive amounts of data that could be leveraged to enable both screening of cardiovascular risk and implementation of tailored preventive interventions across a woman's life span. However, health care providers, clinical guidelines committees, and health policy experts are not yet sufficiently equipped to optimize the collection of data on women, use or interpret these data, or develop approaches to targeting interventions. Therefore, we provide a broad overview of the key opportunities for cardiovascular screening in women while highlighting the potential applications of artificial intelligence along with digital technologies and tools.

Does sex matter? Analysis of sex-related differences in the diagnostic performance of a market-approved convolutional neural network for skin cancer detection

BACKGROUND

Advances in biomedical artificial intelligence may introduce or perpetuate sex and gender discriminations. Convolutional neural networks (CNN) have proven a dermatologist-level performance in image classification tasks but have not been assessed for sex and gender biases that may affect training data and diagnostic performance. In this study, we investigated sex-related imbalances in training data and diagnostic performance of a market-approved CNN for skin cancer classification (Moleanalyzer Pro®, Fotofinder Systems GmbH, Bad Birnbach, Germany).

METHODS

We screened open-access dermoscopic image repositories widely used for CNN training for distribution of sex. Moreover, the sex-related diagnostic performance of the market-approved CNN was tested in 1549 dermoscopic images stratified by sex (female n = 773; male n = 776).

RESULTS

Most open-access repositories showed a marked under-representation of images originating from female (40%) versus male (60%) patients. Despite these imbalances and well-known sex-related differences in skin anatomy or skin-directed behaviour, the tested CNN achieved a comparable sensitivity of 87.0% [80.9%-91.3%] versus 87.1% [81.1%-91.4%], specificity of 98.7% [97.4%-99.3%] versus 96.9% [95.2%-98.0%] and ROC-AUC of 0.984 [0.975-0.993] versus 0.979 [0.969-0.988] in dermoscopic images of female versus male origin, respectively. In the sample at hand, sex-related differences in ROC-AUCs were not statistically significant in the per-image analysis nor in an additional per-individual analysis (p ≥ 0.59).

CONCLUSION

Design and training of artificial intelligence algorithms for medical applications should generally acknowledge sex and gender dimensions. Despite sex-related imbalances in open-access training data, the diagnostic performance of the tested CNN showed no sex-related bias in the classification of skin lesions.

Artificial intelligence for mechanical ventilation: systematic review of design, reporting standards, and bias

BACKGROUND

Artificial intelligence (AI) has the potential to personalise mechanical ventilation strategies for patients with respiratory failure. However, current methodological deficiencies could limit clinical impact. We identified common limitations and propose potential solutions to facilitate translation of AI to mechanical ventilation of patients.

METHODS

A systematic review was conducted in MEDLINE, Embase, and PubMed Central to February 2021. Studies investigating the application of AI to patients undergoing mechanical ventilation were included. Algorithm design and adherence to reporting standards were assessed with a rubric combining published guidelines, satisfying the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis [TRIPOD] statement. Risk of bias was assessed by using the Prediction model Risk Of Bias ASsessment Tool (PROBAST), and correspondence with authors to assess data and code availability.

RESULTS

Our search identified 1,342 studies, of which 95 were included: 84 had single-centre, retrospective study design, with only one randomised controlled trial. Access to data sets and code was severely limited (unavailable in 85% and 87% of studies, respectively). On request, data and code were made available from 12 and 10 authors, respectively, from a list of 54 studies published in the last 5 yr. Ethnicity was frequently under-reported 18/95 (19%), as was model calibration 17/95 (18%). The risk of bias was high in 89% (85/95) of the studies, especially because of analysis bias.

CONCLUSIONS

Development of algorithms should involve prospective and external validation, with greater code and data availability to improve confidence in and translation of this promising approach.

TRIAL REGISTRATION NUMBER

PROSPERO - CRD42021225918.

AI in health and medicine

Artificial intelligence (AI) is poised to broadly reshape medicine, potentially improving the experiences of both clinicians and patients. We discuss key findings from a 2-year weekly effort to track and share key developments in medical AI. We cover prospective studies and advances in medical image analysis, which have reduced the gap between research and deployment. We also address several promising avenues for novel medical AI research, including non-image data sources, unconventional problem formulations and human-AI collaboration. Finally, we consider serious technical and ethical challenges in issues spanning from data scarcity to racial bias. As these challenges are addressed, AI's potential may be realized, making healthcare more accurate, efficient and accessible for patients worldwide.

Racism and Cardiology: A Global Call to Action

Racism and racial bias influence the lives and cardiovascular health of minority individuals. The fact that minority groups tend to have a higher burden of cardiovascular disease risk factors is often a result of racist policies that restrict opportunities to live in healthy neighbourhoods and have access to high-quality education and healthcare. The fact that minorities tend to have the worst outcomes when cardiovascular disease develops is often a result of institutional or individual racial bias encountered when they interact with the healthcare system. In this review, we discuss bias, discrimination, and structural racism from the viewpoints of cardiologists in Canada, the United Kingdom, and the US, and how racial bias impacts cardiovascular care. Finally, we discuss proposals to mitigate the impact of racism in our specialty.

Searching for New Technology Acceptance Model under Social Context: Analyzing the Determinants of Acceptance of Intelligent Information Technology in Digital Transformation and Implications for the Requisites of Digital Sustainability

Intelligent information technology (IIT) based on AI and intelligent network communication technology is rapidly changing the social structure and the personal lives. However, IIT acceptancefrom various perspectives still requires extensive research. The research question in this paper examines how five factors—psychological, technological, resource, risk perception, and value factors—influence IIT acceptance. Based on an analysis of survey data, it was first found that the acceptance rate of IIT itself was generally very high. Second, in terms of IIT acceptance, among twenty-five predictors, voluntariness (+), positive image of technology (+), performance expectancy (+), relative advantage (+), radical innovation (+), and experience of use (+) were found to have significant effects on the IIT acceptance. Third, in addition to technological factors, psychological factors and risk perception factors also played an important role in individuals’ decisions regarding IIT acceptance.

The Impact of Delayed Symptomatic Treatment Implementation in the Intensive Care Unit

We estimated the harm related to medication delivery delays across 12,474 medication administration instances in an intensive care unit using retrospective data in a large urban academic medical center between 2012 and 2015. We leveraged an instrumental variables (IV) approach that addresses unobserved confounds in this setting. We focused on nurse shift changes as disruptors of timely medication (vasodilators, antipyretics, and bronchodilators) delivery to estimate the impact of delay. The average delay around a nurse shift change was 60.8 min (p < 0.001) for antipyretics, 39.5 min (p < 0.001) for bronchodilators, and 57.1 min (p < 0.001) for vasodilators. This delay can increase the odds of developing a fever by 32.94%, tachypnea by 79.5%, and hypertension by 134%, respectively. Compared to estimates generated by a naïve regression approach, our IV estimates tend to be higher, suggesting the existence of a bias from providers prioritizing more critical patients.

Aligning Patient's Ideas of a Good Life with Medically Indicated Therapies in Geriatric Rehabilitation Using Smart Sensors

New technologies such as smart sensors improve rehabilitation processes and thereby increase older adults’ capabilities to participate in social life, leading to direct physical and mental health benefits. Wearable smart sensors for home use have the additional advantage of monitoring day-to-day activities and thereby identifying rehabilitation progress and needs. However, identifying and selecting rehabilitation priorities is ethically challenging because physicians, therapists, and caregivers may impose their own personal values leading to paternalism. Therefore, we develop a discussion template consisting of a series of adaptable questions for the patient–physician encounter based on the capability approach. The goal is to improve geriatric rehabilitation and thereby increase participation in social life and well-being. To achieve this goal, we first analyzed what is considered important for participation on basis of the capability approach, human rights, and ethics of care. Second, we conducted an ethical analysis of each of the four identified dimensions of participation: political, economic, socio-cultural, and care. To improve compliance with rehabilitation measures, health professionals must align rehabilitation measures in an open dialogue with the patient’s aspiration for participation in each dimension. A discussion template based on the capability approach allows for a proactive approach in patient information and stimulates a critical assessment of treatment alternatives while reducing the risk of imposing personal values.

Artificial Intelligence: A Shifting Paradigm in Cardio-Cerebrovascular Medicine

The future of healthcare is an organic blend of technology, innovation, and human connection. As artificial intelligence (AI) is gradually becoming a go-to technology in healthcare to improve efficiency and outcomes, we must understand our limitations. We should realize that our goal is not only to provide faster and more efficient care, but also to deliver an integrated solution to ensure that the care is fair and not biased to a group of sub-population. In this context, the field of cardio-cerebrovascular diseases, which encompasses a wide range of conditions-from heart failure to stroke-has made some advances to provide assistive tools to care providers. This article aimed to provide an overall thematic review of recent development focusing on various AI applications in cardio-cerebrovascular diseases to identify gaps and potential areas of improvement. If well designed, technological engines have the potential to improve healthcare access and equitability while reducing overall costs, diagnostic errors, and disparity in a system that affects patients and providers and strives for efficiency.

Regulating artificial-intelligence applications to achieve the sustainable development goals

Artificial intelligence is producing a revolution with increasing impacts on the people, planet, and prosperity. This perspective illustrates some of the AI applications that can accelerate the achievement of the United Nations Sustainable Development Goals (SDGs) and highlights some of the considerations that could hinder the efforts towards them. In this context, we strongly support the development of an 18thSDG on digital technologies. This emphasizes the importance of establishing standard AI guidelines and regulations for the beneficial applications of AI. Such regulations should focus on concrete applications of AI, rather than generally on AI technology, to facilitate both AI development and enforceability of legal implications.

A healthy debate: Exploring the views of medical doctors on the ethics of artificial intelligence

Artificial Intelligence (AI) is moving towards the health space. It is generally acknowledged that, while there is great promise in the implementation of AI technologies in healthcare, it also raises important ethical issues. In this study we surveyed medical doctors based in The Netherlands, Portugal, and the U.S. from a diverse mix of medical specializations about the ethics surrounding Health AI. Four main perspectives have emerged from the data representing different views about this matter. The first perspective (AI is a helpful tool: Let physicians do what they were trained for) highlights the efficiency associated with automation, which will allow doctors to have the time to focus on expanding their medical knowledge and skills. The second perspective (Rules & Regulations are crucial: Private companies only think about money) shows strong distrust in private tech companies and emphasizes the need for regulatory oversight. The third perspective (Ethics is enough: Private companies can be trusted) puts more trust in private tech companies and maintains that ethics is sufficient to ground these corporations. And finally the fourth perspective (Explainable AI tools: Learning is necessary and inevitable) emphasizes the importance of explainability of AI tools in order to ensure that doctors are engaged in the technological progress. Each perspective provides valuable and often contrasting insights about ethical issues that should be operationalized and accounted for in the design and development of AI Health.