Between a logic of disruption and a logic of continuation: Negotiating the legitimacy of algorithms used in automated clinical decision-making

The robot butler: How and why should we study predictive algorithms and artificial intelligence (AI) in healthcare?

Artificial intelligence (AI) and algorithms are heralded as significant solutions to the widening gap between the rising healthcare needs of ageing and multi-morbid populations and the scarcity of resources to provide such care.

Objective

This article investigates how the PMHnet algorithm - an AI prognostication tool developed in Denmark to predict the one-year all-cause mortality risk for patients hospitalized with ischemic heart disease - was presented to cardiologists working in the hospital setting, and how they responded to this novel decision-support tool.

Methods

Empirically, we draw upon ethnographic fieldwork in the Danish-led international research project, PM Heart, which since 2019 has developed the PMHnet algorithm and implemented the software into the electronic health record system in hospitals in Eastern Denmark (the Capital Region and Region Zealand).

Results

Paying careful attention to the hopes and concerns of cardiologists who will have to embrace and adapt to algorithmic tools in their everyday work of diagnosing and treating patients, we identify three analytical themes meriting attention when AI is implemented in healthcare: 1) the re-negotiation of agency and autonomy in human-algorithm relations, 2) accountability in algorithmic prognostication and 3) the complex relationship between association and causation actualized by predictive algorithms. From these analytical themes, we elicit methodological questions to guide future ethnographic explorations of how AI and advanced algorithms are put to use in the healthcare system, with what implications, and for whom.

Conclusion

We conclude that local, qualitative investigations of how algorithms are used, embraced and contested in everyday clinical practice are needed in order to understand their implications - good and bad, intended and unintended - for clinicians, patients and healthcare provision.

Stakeholder Perspectives of Clinical Artificial Intelligence Implementation: Systematic Review of Qualitative Evidence

BACKGROUND

The rhetoric surrounding clinical artificial intelligence (AI) often exaggerates its effect on real-world care. Limited understanding of the factors that influence its implementation can perpetuate this.

OBJECTIVE

In this qualitative systematic review, we aimed to identify key stakeholders, consolidate their perspectives on clinical AI implementation, and characterize the evidence gaps that future qualitative research should target.

METHODS

Ovid-MEDLINE, EBSCO-CINAHL, ACM Digital Library, Science Citation Index-Web of Science, and Scopus were searched for primary qualitative studies on individuals' perspectives on any application of clinical AI worldwide (January 2014-April 2021). The definition of clinical AI includes both rule-based and machine learning-enabled or non-rule-based decision support tools. The language of the reports was not an exclusion criterion. Two independent reviewers performed title, abstract, and full-text screening with a third arbiter of disagreement. Two reviewers assigned the Joanna Briggs Institute 10-point checklist for qualitative research scores for each study. A single reviewer extracted free-text data relevant to clinical AI implementation, noting the stakeholders contributing to each excerpt. The best-fit framework synthesis used the Nonadoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework. To validate the data and improve accessibility, coauthors representing each emergent stakeholder group codeveloped summaries of the factors most relevant to their respective groups.

RESULTS

The initial search yielded 4437 deduplicated articles, with 111 (2.5%) eligible for inclusion (median Joanna Briggs Institute 10-point checklist for qualitative research score, 8/10). Five distinct stakeholder groups emerged from the data: health care professionals (HCPs), patients, carers and other members of the public, developers, health care managers and leaders, and regulators or policy makers, contributing 1204 (70%), 196 (11.4%), 133 (7.7%), 129 (7.5%), and 59 (3.4%) of 1721 eligible excerpts, respectively. All stakeholder groups independently identified a breadth of implementation factors, with each producing data that were mapped between 17 and 24 of the 27 adapted Nonadoption, Abandonment, Scale-up, Spread, and Sustainability subdomains. Most of the factors that stakeholders found influential in the implementation of rule-based clinical AI also applied to non-rule-based clinical AI, with the exception of intellectual property, regulation, and sociocultural attitudes.

CONCLUSIONS

Clinical AI implementation is influenced by many interdependent factors, which are in turn influenced by at least 5 distinct stakeholder groups. This implies that effective research and practice of clinical AI implementation should consider multiple stakeholder perspectives. The current underrepresentation of perspectives from stakeholders other than HCPs in the literature may limit the anticipation and management of the factors that influence successful clinical AI implementation. Future research should not only widen the representation of tools and contexts in qualitative research but also specifically investigate the perspectives of all stakeholder HCPs and emerging aspects of non-rule-based clinical AI implementation.

TRIAL REGISTRATION

PROSPERO (International Prospective Register of Systematic Reviews) CRD42021256005; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=256005.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/33145.