Are residents' decisions influenced more by a decision aid or a specialist's opinion? A randomized controlled trial

Experimental evidence of effective human-AI collaboration in medical decision-making

Artificial Intelligence (AI) systems are precious support for decision-making, with many applications also in the medical domain. The interaction between MDs and AI enjoys a renewed interest following the increased possibilities of deep learning devices. However, we still have limited evidence-based knowledge of the context, design, and psychological mechanisms that craft an optimal human-AI collaboration. In this multicentric study, 21 endoscopists reviewed 504 videos of lesions prospectively acquired from real colonoscopies. They were asked to provide an optical diagnosis with and without the assistance of an AI support system. Endoscopists were influenced by AI ([Formula: see text]), but not erratically: they followed the AI advice more when it was correct ([Formula: see text]) than incorrect ([Formula: see text]). Endoscopists achieved this outcome through a weighted integration of their and the AI opinions, considering the case-by-case estimations of the two reliabilities. This Bayesian-like rational behavior allowed the human-AI hybrid team to outperform both agents taken alone. We discuss the features of the human-AI interaction that determined this favorable outcome.

Diagnostic Reboot: A Proposal to Improve Diagnostic Reasoning

Background Diagnostic errors contribute to the morbidity and mortality of patients. We created and utilized a novel diagnostic tool (Diagnostic Reboot) and assessed its practical efficacy in the inpatient setting for improving diagnostic outcomes. Design This was a prospective sequential controlled study that involved University Hospitalist Adult Teaching Service (UHATS) teams. Senior residents were instructed to use the Diagnostic Reboot (DxR) tool whenever a patient aged 19-99 years was identified who had an uncertain diagnosis 24 hours into their admission.  Results Participating residents identified a total of 32 patients as meeting the criteria of uncertain diagnosis after at least 24 hours of hospitalization during the six months of the study period. Of these, seven were during the intervention (DxR) period. The leading diagnosis was excluded in 3/7 (43%) patients in the DxR period and 13/25 (52%) in the control period. A new leading diagnosis was made in 6/7 (86%) cases in the DxR period and in 13/25 (52%) people in the control period. A new diagnostic plan was made in 100% of the patients in the DxR group and in 80% of patients in the control group. A new consultation was requested in 4/7 (57%) patients in the DxR group and in 9/25 (36%) patients in the control group. The Residents spent an average of 20 minutes on the DxR tool. Conclusions This study demonstrated that the use of DxR may help to improve analytical thinking in residents. It may also play a role in improving outcomes in medically challenging cases, but the use of the tool during the study period was not sufficient to draw concrete conclusions. The primary barrier to the use of such a diagnostic aid was identified as time pressure on a busy hospitalist service.

Developing an intervention around referral and admissions to intensive care: a mixed-methods study

Background

Intensive care treatment can be life-saving, but it is invasive and distressing for patients receiving it and it is not always successful. Deciding whether or not a patient will benefit from intensive care is a difficult clinical and ethical challenge.

Objectives

To explore the decision-making process for referral and admission to the intensive care unit and to develop and test an intervention to improve it.

Methods

A mixed-methods study comprising (1) two systematic reviews investigating the factors associated with decisions to admit patients to the intensive care unit and the experiences of clinicians, patients and families; (2) observation of decisions and interviews with intensive care unit doctors, referring doctors, and patients and families in six NHS trusts in the Midlands, UK; (3) a choice experiment survey distributed to UK intensive care unit consultants and critical care outreach nurses, eliciting their preferences for factors used in decision-making for intensive care unit admission; (4) development of a decision-support intervention informed by the previous work streams, including an ethical framework for decision-making and supporting referral and decision-support forms and patient and family information leaflets. Implementation feasibility was tested in three NHS trusts; (5) development and testing of a tool to evaluate the ethical quality of decision-making related to intensive care unit admission, based on the assessment of patient records. The tool was tested for inter-rater and intersite reliability in 120 patient records.

Results

Influences on decision-making identified in the systematic review and ethnographic study included age, presence of chronic illness, functional status, presence of a do not attempt cardiopulmonary resuscitation order, referring specialty, referrer seniority and intensive care unit bed availability. Intensive care unit doctors used a gestalt assessment of the patient when making decisions. The choice experiment showed that age was the most important factor in consultants’ and critical care outreach nurses’ preferences for admission. The ethnographic study illuminated the complexity of the decision-making process, and the importance of interprofessional relationships and good communication between teams and with patients and families. Doctors found it difficult to articulate and balance the benefits and burdens of intensive care unit treatment for a patient. There was low uptake of the decision-support intervention, although doctors who used it noted that it improved articulation of reasons for decisions and communication with patients.

Limitations

Limitations existed in each of the component studies; for example, we had difficulty recruiting patients and families in our qualitative work. However, the project benefited from a mixed-method approach that mitigated the potential limitations of the component studies.

Conclusions

Decision-making surrounding referral and admission to the intensive care unit is complex. This study has provided evidence and resources to help clinicians and organisations aiming to improve the decision-making for and, ultimately, the care of critically ill patients.

Future work

Further research is needed into decision-making practices, particularly in how best to engage with patients and families during the decision process. The development and evaluation of training for clinicians involved in these decisions should be a priority for future work.

Study registration

The systematic reviews of this study are registered as PROSPERO CRD42016039054, CRD42015019711 and CRD42015019714.

Funding

The National Institute for Health Research Health Services and Delivery Research programme. The University of Aberdeen and the Chief Scientist Office of the Scottish Government Health and Social Care Directorates fund the Health Economics Research Unit.

Comprehensive process model of clinical information interaction in primary care: results of a "best-fit" framework synthesis

Objective

To describe a new, comprehensive process model of clinical information interaction in primary care (Clinical Information Interaction Model, or CIIM) based on a systematic synthesis of published research.

Materials and Methods

We used the "best fit" framework synthesis approach. Searches were performed in PubMed, Embase, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), PsycINFO, Library and Information Science Abstracts, Library, Information Science and Technology Abstracts, and Engineering Village. Two authors reviewed articles according to inclusion and exclusion criteria. Data abstraction and content analysis of 443 published papers were used to create a model in which every element was supported by empirical research.

Results

The CIIM documents how primary care clinicians interact with information as they make point-of-care clinical decisions. The model highlights 3 major process components: (1) context, (2) activity (usual and contingent), and (3) influence. Usual activities include information processing, source-user interaction, information evaluation, selection of information, information use, clinical reasoning, and clinical decisions. Clinician characteristics, patient behaviors, and other professionals influence the process.

Discussion

The CIIM depicts the complete process of information interaction, enabling a grasp of relationships previously difficult to discern. The CIIM suggests potentially helpful functionality for clinical decision support systems (CDSSs) to support primary care, including a greater focus on information processing and use. The CIIM also documents the role of influence in clinical information interaction; influencers may affect the success of CDSS implementations.

Conclusion

The CIIM offers a new framework for achieving CDSS workflow integration and new directions for CDSS design that can support the work of diverse primary care clinicians.

A balance beam aid for instruction in clinical diagnostic reasoning

We describe a balance beam aid for instruction in diagnosis (BBAID) and demonstrate its potential use in supplementing the training of medical students to diagnose acute chest pain. We suggest the BBAID helps students understand the process of diagnosis because the impact of tokens (weights and helium balloons) attached to a beam at different distances from the fulcrum is analogous to the impact of evidence to the relative support for 2 diseases. The BBAID presents a list of potential findings and allows students to specify whether each is present, absent, or unknown. It displays the likelihood ratios corresponding to a positive (LR+) or negative (LR-) observation for each symptom, for any pair of diseases. For each specified finding, a token is placed on the beam at a location whose distance from the fulcrum is proportional to the finding's log(LR): a downward force (a weight) if the finding is present and a lifting force (a balloon) if it is absent. Combining the physical torques of multiple tokens is mathematically identical to applying Bayes' theorem to multiple independent findings, so the balance beam is a high-fidelity metaphor. Seven first-year medical students and 3 faculty members consulted the BBAID while diagnosing brief patient case vignettes. Student comments indicated the program is usable, helpful for understanding pertinent positive and negative findings' usefulness in particular situations, and welcome as a reference or self-test. All students attended the effect of the tokens on the beam, although some stated they did not use the numerical statistics. Faculty noted the BBAID might be particularly helpful in reminding students of diseases that should not be missed and identifying pertinent findings to ask for.

Why do patients derogate physicians who use a computer-based diagnostic support system?

OBJECTIVE

To better understand 1) why patients have a negative perception of the use of computerized clinical decision support systems (CDSSs) and 2) what contributes to the documented heterogeneity in the evaluations of physicians who use a CDSS.

METHODS

Three vignette-based studies examined whether negative perceptions stemmed directly from the use of a computerized decision aid or the need to seek external advice more broadly (experiment 1) and investigated the contributing role of 2 individual difference measures, attitudes toward statistics (ATS; experiment 2) and the Multidimensional Health Locus of Control Scale (MHLC; experiment 3), to these findings.

RESULTS

A physician described as making an unaided diagnosis was rated significantly more positively on a number of attributes than a physician using a computerized decision aid but not a physician who sought the advice of an expert colleague (experiment 1). ATS were unrelated to perceptions of decision aid use (experiment 2); however, greater internal locus of control was associated with more positive feelings about unaided care and more negative feelings about care when a decision aid was used (experiment 3).

CONCLUSION

Negative perceptions of computerized decision aid use may not be a product of the need to seek external advice more generally but may instead be specific to the use of a nonhuman tool and may be associated with individual differences in locus of control. Together, these 3 studies may be used to guide education efforts for patients.

Use of the isabel decision support system to improve diagnostic accuracy of pediatric nurse practitioner and family nurse practitioner students

Patient safety is a priority for healthcare today. Despite a large proportion of malpractice claims the result of diagnostic error, the use of diagnostic decision support to improve diagnostic accuracy has not been widely used among healthcare professionals. Moreover, while the use of diagnostic decision support has been studied in attending physicians, residents, medical students and advanced practice nurses, the use of decision support among Advanced Practice Nurse (APN) students has not been studied. The authors have implemented the Isabel diagnostic decision support system into the curriculum and are evaluating its impact. The goals of the evaluation study are to describe the diagnostic accuracy and self-reported confidence levels of Pediatric Nurse Practitioner (PNP) and Family Nurse Practitioner (FNP) students over the course of their programs, to examine changes in diagnostic accuracy and self-reported confidence levels over the study period, and to evaluate differences between FNP and PNP students in diagnostic accuracy and self-reported confidence levels for pediatric cases. This paper summarizes establishment of the academic/industry collaboration, case generation, integration of Isabel into the curriculum, and evaluation design.