Early diagnostic suggestions improve accuracy of GPs: a randomised controlled trial using computer-simulated patients

Adoption, acceptance, and use of a decision support tool to promote timely investigations for cancer in primary care

BACKGROUND

The complexities of diagnosing cancer in general practice has driven the development of quality improvement (QI) interventions, including clinical decision support (CDS) and auditing tools. Future Health Today (FHT) is a novel QI tool, consisting of CDS at the point-of-care, practice population-level auditing, recall, and the monitoring of QI activities.

OBJECTIVES

Explore the acceptability and usability of the FHT cancer module, which flags patients with abnormal test results that may be indicative of undiagnosed cancer.

METHODS

Interviews were conducted with general practitioners (GPs) and general practice nurses (GPNs), from practices participating in a randomized trial evaluating the appropriate follow-up of patients. Clinical Performance Feedback Intervention Theory (CP-FIT) was used to analyse and interpret the data.

RESULTS

The majority of practices reported not using the auditing and QI components of the tool, only the CDS which was delivered at the point-of-care. The tool was used primarily by GPs; GPNs did not perceive the clinical recommendations to be within their role. For the CDS, facilitators for use included a good workflow fit, ease of use, low time cost, importance, and perceived knowledge gain. Barriers for use of the CDS included accuracy, competing priorities, and the patient population.

CONCLUSIONS

The CDS aligned with the clinical workflow of GPs, was considered non-disruptive to the consultation and easy to implement into usual care. By applying the CP-FIT theory, we were able to demonstrate the key drivers for GPs using the tool, and what limited the use by GPNs.

Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial

Importance

Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves physician diagnostic reasoning.

Objective

To assess the effect of an LLM on physicians' diagnostic reasoning compared with conventional resources.

Design, Setting, and Participants

A single-blind randomized clinical trial was conducted from November 29 to December 29, 2023. Using remote video conferencing and in-person participation across multiple academic medical institutions, physicians with training in family medicine, internal medicine, or emergency medicine were recruited.

Intervention

Participants were randomized to either access the LLM in addition to conventional diagnostic resources or conventional resources only, stratified by career stage. Participants were allocated 60 minutes to review up to 6 clinical vignettes.

Main Outcomes and Measures

The primary outcome was performance on a standardized rubric of diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps, validated and graded via blinded expert consensus. Secondary outcomes included time spent per case (in seconds) and final diagnosis accuracy. All analyses followed the intention-to-treat principle. A secondary exploratory analysis evaluated the standalone performance of the LLM by comparing the primary outcomes between the LLM alone group and the conventional resource group.

Results

Fifty physicians (26 attendings, 24 residents; median years in practice, 3 [IQR, 2-8]) participated virtually as well as at 1 in-person site. The median diagnostic reasoning score per case was 76% (IQR, 66%-87%) for the LLM group and 74% (IQR, 63%-84%) for the conventional resources-only group, with an adjusted difference of 2 percentage points (95% CI, -4 to 8 percentage points; P = .60). The median time spent per case for the LLM group was 519 (IQR, 371-668) seconds, compared with 565 (IQR, 456-788) seconds for the conventional resources group, with a time difference of -82 (95% CI, -195 to 31; P = .20) seconds. The LLM alone scored 16 percentage points (95% CI, 2-30 percentage points; P = .03) higher than the conventional resources group.

Conclusions and Relevance

In this trial, the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources. The LLM alone demonstrated higher performance than both physician groups, indicating the need for technology and workforce development to realize the potential of physician-artificial intelligence collaboration in clinical practice.

Trial Registration

ClinicalTrials.gov Identifier: NCT06157944.

The influence of context on diagnostic reasoning: A narrative synthesis of experimental findings

AIMS AND OBJECTIVES

Contextual information which is implicitly available to physicians during clinical encounters has been shown to influence diagnostic reasoning. To better understand the psychological mechanisms underlying the influence of context on diagnostic accuracy, we conducted a review of experimental research on this topic.

METHOD

We searched Web of Science, PubMed, and Scopus for relevant articles and looked for additional records by reading the references and approaching experts. We limited the review to true experiments involving physicians in which the outcome variable was the accuracy of the diagnosis.

RESULTS

The 43 studies reviewed examined two categories of contextual variables: (a) case-intrinsic contextual information and (b) case-extrinsic contextual information. Case-intrinsic information includes implicit misleading diagnostic suggestions in the disease history of the patient, or emotional volatility of the patient. Case-extrinsic or situational information includes a similar (but different) case seen previously, perceived case difficulty, or external digital diagnostic support. Time pressure and interruptions are other extrinsic influences that may affect the accuracy of a diagnosis but have produced conflicting findings.

CONCLUSION

We propose two tentative hypotheses explaining the role of context in diagnostic accuracy. According to the negative-affect hypothesis, diagnostic errors emerge when the physician's attention shifts from the relevant clinical findings to the (irrelevant) source of negative affect (for instance patient aggression) raised in a clinical encounter. The early-diagnosis-primacy hypothesis attributes errors to the extraordinary influence of the initial hypothesis that comes to the physician's mind on the subsequent collecting and interpretation of case information. Future research should test these mechanisms explicitly. Possible alternative mechanisms such as premature closure or increased production of (irrelevant) rival diagnoses in response to context deserve further scrutiny. Implications for medical education and practice are discussed.

Transforming post pandemic cancer services

This paper outlines the impact of the COVID-19 pandemic on cancer services in the UK including screening, symptomatic diagnosis, treatment pathways and projections on clinical outcomes as a result of these care disruptions. A restoration of cancer services to pre-pandemic levels is not likely to mitigate this adverse impact, particularly with an ageing population and increased cancer burden. New cancer cases are projected to rise to over 500,000 per year by 2035, with over 4 million people living with and beyond cancer. This paper calls for a strategic transformation to prioritise effort on the basis of available datasets and evidence-in particular, to prioritise cancers where an earlier diagnosis is feasible and clinically useful with a focus on mortality benefit by preventing emergency presentations by harnessing data and analytics. This could be delivered by a focus on underperforming groups/areas to try and reduce inequity, linking near real-time datasets with clinical decision support systems at the primary and secondary care levels, promoting the use of novel technologies to improve patient uptake of services, screening and diagnosis, and finally, upskilling and cross-skilling healthcare workers to expand supply of diagnostic and screening services.

Collective Intelligence Increases Diagnostic Accuracy in a General Practice Setting

BACKGROUND

General practitioners (GPs) work in an ill-defined environment where diagnostic errors are prevalent. Previous research indicates that aggregating independent diagnoses can improve diagnostic accuracy in a range of settings. We examined whether aggregating independent diagnoses can also improve diagnostic accuracy for GP decision making. In addition, we investigated the potential benefit of such an approach in combination with a decision support system (DSS).

METHODS

We simulated virtual groups using data sets from 2 previously published studies. In study 1, 260 GPs independently diagnosed 9 patient cases in a vignette-based study. In study 2, 30 GPs independently diagnosed 12 patient actors in a patient-facing study. In both data sets, GPs provided diagnoses in a control condition and/or DSS condition(s). Each GP's diagnosis, confidence rating, and years of experience were entered into a computer simulation. Virtual groups of varying sizes (range: 3-9) were created, and different collective intelligence rules (plurality, confidence, and seniority) were applied to determine each group's final diagnosis. Diagnostic accuracy was used as the performance measure.

RESULTS

Aggregating independent diagnoses by weighing them equally (i.e., the plurality rule) substantially outperformed average individual accuracy, and this effect increased with increasing group size. Selecting diagnoses based on confidence only led to marginal improvements, while selecting based on seniority reduced accuracy. Combining the plurality rule with a DSS further boosted performance.

DISCUSSION

Combining independent diagnoses may substantially improve a GP's diagnostic accuracy and subsequent patient outcomes. This approach did, however, not improve accuracy in all patient cases. Therefore, future work should focus on uncovering the conditions under which collective intelligence is most beneficial in general practice.

HIGHLIGHTS

We examined whether aggregating independent diagnoses of GPs can improve diagnostic accuracy.Using data sets of 2 previously published studies, we composed virtual groups of GPs and combined their independent diagnoses using 3 collective intelligence rules (plurality, confidence, and seniority).Aggregating independent diagnoses by weighing them equally substantially outperformed average individual GP accuracy, and this effect increased with increasing group size.Combining independent diagnoses may substantially improve GP's diagnostic accuracy and subsequent patient outcomes.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study

BACKGROUND

Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution.

OBJECTIVE

The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists.

METHODS

We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input).

RESULTS

The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists).

CONCLUSIONS

Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Evaluation of a Medical Interview-Assistance System Using Artificial Intelligence for Resident Physicians Interviewing Simulated Patients: A Crossover, Randomized, Controlled Trial

Medical interviews are expected to undergo a major transformation through the use of artificial intelligence. However, artificial intelligence-based systems that support medical interviews are not yet widespread in Japan, and their usefulness is unclear. A randomized, controlled trial to determine the usefulness of a commercial medical interview support system using a question flow chart-type application based on a Bayesian model was conducted. Ten resident physicians were allocated to two groups with or without information from an artificial intelligence-based support system. The rate of correct diagnoses, amount of time to complete the interviews, and number of questions they asked were compared between the two groups. Two trials were conducted on different dates, with a total of 20 resident physicians participating. Data for 192 differential diagnoses were obtained. There was a significant difference in the rate of correct diagnosis between the two groups for two cases and for overall cases (0.561 vs. 0.393; p = 0.02). There was a significant difference in the time required between the two groups for overall cases (370 s (352-387) vs. 390 s (373-406), p = 0.04). Artificial intelligence-assisted medical interviews helped resident physicians make more accurate diagnoses and reduced consultation time. The widespread use of artificial intelligence systems in clinical settings could contribute to improving the quality of medical care.

Improving diagnostic decision support through deliberate reflection: a proposal

Digital decision support (DDS) is expected to play an important role in improving a physician's diagnostic performance and reducing the burden of diagnostic error. Studies with currently available DDS systems indicate that they lead to modest gains in diagnostic accuracy, and these systems are expected to evolve to become more effective and user-friendly in the future. In this position paper, we propose that a way towards this future is to rethink DDS systems based on deliberate reflection, a strategy by which physicians systematically review the clinical findings observed in a patient in the light of an initial diagnosis. Deliberate reflection has been demonstrated to improve diagnostic accuracy in several contexts. In this paper, we first describe the deliberate reflection strategy, including the crucial element that would make it useful in the interaction with a DDS system. We examine the nature of conventional DDS systems and their shortcomings. Finally, we propose what DDS based on deliberate reflection might look like, and consider why it would overcome downsides of conventional DDS.

Influences of early diagnostic suggestions on clinical reasoning

Previous research has highlighted the importance of physicians' early hypotheses for their subsequent diagnostic decisions. It has also been shown that diagnostic accuracy improves when physicians are presented with a list of diagnostic suggestions to consider at the start of the clinical encounter. The psychological mechanisms underlying this improvement in accuracy are hypothesised. It is possible that the provision of diagnostic suggestions disrupts physicians' intuitive thinking and reduces their certainty in their initial diagnostic hypotheses. This may encourage them to seek more information before reaching a diagnostic conclusion, evaluate this information more objectively, and be more open to changing their initial hypotheses. Three online experiments explored the effects of early diagnostic suggestions, provided by a hypothetical decision aid, on different aspects of the diagnostic reasoning process. Family physicians assessed up to two patient scenarios with and without suggestions. We measured effects on certainty about the initial diagnosis, information search and evaluation, and frequency of diagnostic changes. We did not find a clear and consistent effect of suggestions and detected mainly non-significant trends, some in the expected direction. We also detected a potential biasing effect: when the most likely diagnosis was included in the list of suggestions (vs. not included), physicians who gave that diagnosis initially, tended to request less information, evaluate it as more supportive of their diagnosis, become more certain about it, and change it less frequently when encountering new but ambiguous information; in other words, they seemed to validate rather than question their initial hypothesis. We conclude that further research using different methodologies and more realistic experimental situations is required to uncover both the beneficial and biasing effects of early diagnostic suggestions.

Clinical Profiles at the Time of Diagnosis of SARS-CoV-2 Infection in Costa Rica During the Pre-vaccination Period Using a Machine Learning Approach

The clinical manifestations of COVID-19, caused by the SARS-CoV-2, define a large spectrum of symptoms that are mainly dependent on the human host conditions. In Costa Rica, more than 169,000 cases and 2185 deaths were reported during the year 2020, the pre-vaccination period. To describe the clinical presentations at the time of diagnosis of SARS-CoV-2 infection in Costa Rica during the pre-vaccination period, we implemented a symptom-based clustering using machine learning to identify clusters or clinical profiles at the population level among 18,974 records of positive cases. Profiles were compared based on symptoms, risk factors, viral load, and genomic features of the SARS-CoV-2 sequence. A total of 18 symptoms at time of diagnosis of SARS-CoV-2 infection were reported with a frequency > 1%, and those were used to identify seven clinical profiles with a specific composition of clinical manifestations. In the comparison between clusters, a lower viral load was found for the asymptomatic group, while the risk factors and the SARS-CoV-2 genomic features were distributed among all the clusters. No other distribution patterns were found for age, sex, vital status, and hospitalization. In conclusion, during the pre-vaccination time in Costa Rica, the symptoms at the time of diagnosis of SARS-CoV-2 infection were described in clinical profiles. The host co-morbidities and the SARS-CoV-2 genotypes are not specific of a particular profile, rather they are present in all the groups, including asymptomatic cases. In addition, this information can be used for decision-making by the local healthcare institutions (first point of contact with health professionals, case definition, or infrastructure). In further analyses, these results will be compared against the profiles of cases during the vaccination period.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43657-022-00058-x.

Evaluating the Diagnostic Accuracy of a Novel Bayesian Decision-Making Algorithm for Vision Loss

The current diagnostic aids for acute vision loss are static flowcharts that do not provide dynamic, stepwise workups. We tested the diagnostic accuracy of a novel dynamic Bayesian algorithm for acute vision loss. Seventy-nine “participants” with acute vision loss in Windsor, Canada were assessed by an emergency medicine or primary care provider who completed a questionnaire about ocular symptoms/findings (without requiring fundoscopy). An ophthalmologist then attributed an independent “gold-standard diagnosis”. The algorithm employed questionnaire data to produce a differential diagnosis. The referrer diagnostic accuracy was 30.4%, while the algorithm’s accuracy was 70.9%, increasing to 86.1% with the algorithm’s top two diagnoses included and 88.6% with the top three included. In urgent cases of vision loss (n = 54), the referrer diagnostic accuracy was 38.9%, while the algorithm’s top diagnosis was correct in 72.2% of cases, increasing to 85.2% (top two included) and 87.0% (top three included). The algorithm’s sensitivity for urgent cases using the top diagnosis was 94.4% (95% CI: 85–99%), with a specificity of 76.0% (95% CI: 55–91%). This novel algorithm adjusts its workup at each step using clinical symptoms. In doing so, it successfully improves diagnostic accuracy for vision loss using clinical data collected by non-ophthalmologists.

Use of heuristics during the clinical decision process from family care physicians in real conditions

RATIONALE AIMS AND OBJECTIVES

The available evidence on the use of heuristics and their relationship with diagnostic error in primary care is very limited. The aim of the study is to identify the use of unknown thought and specifically the possible use of Representativeness, Availability and overconfidence heuristics in the clinical practice of primary care physicians in cases of dyspnoea and to analyse their possible relationship with diagnostic error.

METHODS

A total of 371 patients consulting with new episodes of dyspnoea in Primary Care centres in Spain were registered. Based on specific operational definitions, the use of unconscious thinking and the use of heuristics during the diagnostic process were assessed. Subsequently, the association between their use and diagnostic error was analysed.

RESULTS

In 49.6% of cases, the confirmatory diagnosis coincided with the first diagnostic impression, suggesting the use of the representativeness heuristic in the diagnostic decision process. In 82.3% of the cases, the confirmatory diagnosis was among the three diagnostic hypotheses that were first identified by the general physicians, suggesting a possible use of the availability heuristic. In more than 50% of the cases, the physicians were overconfident in the certainty of their own diagnosis. Finally, a diagnostic error was identified in 9.9% of the recorded cases and no statistically significant correlation was found between the use of some unconscious thinking tools (such as the use of heuristics) and the diagnostic error.

CONCLUSION

Unconscious thinking manifested through the acceptance of the first diagnostic impression and the use of heuristics is commonly used by primary care physicians in the clinical decision process in the face of new episodes of dyspnoea; however, its influence on diagnostic error is not significant. The proposed explicit and reproducible methodology may inspire further studies to confirm these results.

A Clinical Decision Support System for Sleep Staging Tasks With Explanations From Artificial Intelligence: User-Centered Design and Evaluation Study

Background

Despite the unprecedented performance of deep learning algorithms in clinical domains, full reviews of algorithmic predictions by human experts remain mandatory. Under these circumstances, artificial intelligence (AI) models are primarily designed as clinical decision support systems (CDSSs). However, from the perspective of clinical practitioners, the lack of clinical interpretability and user-centered interfaces hinders the adoption of these AI systems in practice.

Objective

This study aims to develop an AI-based CDSS for assisting polysomnographic technicians in reviewing AI-predicted sleep staging results. This study proposed and evaluated a CDSS that provides clinically sound explanations for AI predictions in a user-centered manner.

Methods

Our study is based on a user-centered design framework for developing explanations in a CDSS that identifies why explanations are needed, what information should be contained in explanations, and how explanations can be provided in the CDSS. We conducted user interviews, user observation sessions, and an iterative design process to identify three key aspects for designing explanations in the CDSS. After constructing the CDSS, the tool was evaluated to investigate how the CDSS explanations helped technicians. We measured the accuracy of sleep staging and interrater reliability with macro-F1 and Cohen κ scores to assess quantitative improvements after our tool was adopted. We assessed qualitative improvements through participant interviews that established how participants perceived and used the tool.

Results

The user study revealed that technicians desire explanations that are relevant to key electroencephalogram (EEG) patterns for sleep staging when assessing the correctness of AI predictions. Here, technicians wanted explanations that could be used to evaluate whether the AI models properly locate and use these patterns during prediction. On the basis of this, information that is closely related to sleep EEG patterns was formulated for the AI models. In the iterative design phase, we developed a different visualization strategy for each pattern based on how technicians interpreted the EEG recordings with these patterns during their workflows. Our evaluation study on 9 polysomnographic technicians quantitatively and qualitatively investigated the helpfulness of the tool. For technicians with <5 years of work experience, their quantitative sleep staging performance improved significantly from 56.75 to 60.59 with a P value of .05. Qualitatively, participants reported that the information provided effectively supported them, and they could develop notable adoption strategies for the tool.

Conclusions

Our findings indicate that formulating clinical explanations for automated predictions using the information in the AI with a user-centered design process is an effective strategy for developing a CDSS for sleep staging.

A Library of Logic Models to Explain How Interventions to Reduce Diagnostic Errors Work

OBJECTIVES

We aimed to create a library of logic models for interventions to reduce diagnostic error. This library can be used by those developing, implementing, or evaluating an intervention to improve patient care, to understand what needs to happen, and in what order, if the intervention is to be effective.

METHODS

To create the library, we modified an existing method for generating logic models. The following five ordered activities to include in each model were defined: preintervention; implementation of the intervention; postimplementation, but before the immediate outcome can occur; the immediate outcome (usually behavior change); and postimmediate outcome, but before a reduction in diagnostic errors can occur. We also included reasons for lack of progress through the model. Relevant information was extracted about existing evaluations of interventions to reduce diagnostic error, identified by updating a previous systematic review.

RESULTS

Data were synthesized to create logic models for four types of intervention, addressing five causes of diagnostic error in seven stages in the diagnostic pathway. In total, 46 interventions from 43 studies were included and 24 different logic models were generated.

CONCLUSIONS

We used a novel approach to create a freely available library of logic models. The models highlight the importance of attending to what needs to occur before and after intervention delivery if the intervention is to be effective. Our work provides a useful starting point for intervention developers, helps evaluators identify intermediate outcomes, and provides a method to enable others to generate libraries for interventions targeting other errors.

Incidence of diagnostic errors in unplanned hospitalized patients using an automated medical history-taking system with differential diagnosis generator: retrospective observational study (Preprint)

Background

Automated medical history–taking systems that generate differential diagnosis lists have been suggested to contribute to improved diagnostic accuracy. However, the effect of these systems on diagnostic errors in clinical practice remains unknown.

Objective

This study aimed to assess the incidence of diagnostic errors in an outpatient department, where an artificial intelligence (AI)–driven automated medical history–taking system that generates differential diagnosis lists was implemented in clinical practice.

Methods

We conducted a retrospective observational study using data from a community hospital in Japan. We included patients aged 20 years and older who used an AI-driven, automated medical history–taking system that generates differential diagnosis lists in the outpatient department of internal medicine for whom the index visit was between July 1, 2019, and June 30, 2020, followed by unplanned hospitalization within 14 days. The primary endpoint was the incidence of diagnostic errors, which were detected using the Revised Safer Dx Instrument by at least two independent reviewers. To evaluate the effect of differential diagnosis lists from the AI system on the incidence of diagnostic errors, we compared the incidence of these errors between a group where the AI system generated the final diagnosis in the differential diagnosis list and a group where the AI system did not generate the final diagnosis in the list; the Fisher exact test was used for comparison between these groups. For cases with confirmed diagnostic errors, further review was conducted to identify the contributing factors of these errors via discussion among three reviewers, using the Safer Dx Process Breakdown Supplement as a reference.

Results

A total of 146 patients were analyzed. A final diagnosis was confirmed for 138 patients and was observed in the differential diagnosis list from the AI system for 69 patients. Diagnostic errors occurred in 16 out of 146 patients (11.0%, 95% CI 6.4%-17.2%). Although statistically insignificant, the incidence of diagnostic errors was lower in cases where the final diagnosis was included in the differential diagnosis list from the AI system than in cases where the final diagnosis was not included in the list (7.2% vs 15.9%, P=.18).

Conclusions

The incidence of diagnostic errors among patients in the outpatient department of internal medicine who used an automated medical history–taking system that generates differential diagnosis lists seemed to be lower than the previously reported incidence of diagnostic errors. This result suggests that the implementation of an automated medical history–taking system that generates differential diagnosis lists could be beneficial for diagnostic safety in the outpatient department of internal medicine.

Should electronic differential diagnosis support be used early or late in the diagnostic process? A multicentre experimental study of Isabel

Background

Diagnostic errors unfortunately remain common. Electronic differential diagnostic support (EDS) systems may help, but it is unclear when and how they ought to be integrated into the diagnostic process.

Objective

To explore how much EDS improves diagnostic accuracy, and whether EDS should be used early or late in the diagnostic process.

Setting

6 Canadian medical schools. A volunteer sample of 67 medical students, 62 residents in internal medicine or emergency medicine, and 61 practising internists or emergency medicine physicians were recruited in May through June 2020.

Intervention

Participants were randomised to make use of EDS either early (after the chief complaint) or late (after the complete history and physical is available) in the diagnostic process while solving each of 16 written cases. For each case, we measured the number of diagnoses proposed in the differential diagnosis and how often the correct diagnosis was present within the differential.

Results

EDS increased the number of diagnostic hypotheses by 2.32 (95% CI 2.10 to 2.49) when used early in the process and 0.89 (95% CI 0.69 to 1.10) when used late in the process (both p<0.001). Both early and late use of EDS increased the likelihood of the correct diagnosis being present in the differential (7% and 8%, respectively, both p<0.001). Whereas early use increased the number of diagnostic hypotheses (most notably for students and residents), late use increased the likelihood of the correct diagnosis being present in the differential regardless of one’s experience level.

Conclusions and relevance

EDS increased the number of diagnostic hypotheses and the likelihood of the correct diagnosis appearing in the differential, and these effects persisted irrespective of whether EDS was used early or late in the diagnostic process.

Do different response formats affect how test takers approach a clinical reasoning task? An experimental study on antecedents of diagnostic accuracy using a constructed response and a selected response format

The use of response formats in assessments of medical knowledge and clinical reasoning continues to be the focus of both research and debate. In this article, we report on an experimental study in which we address the question of how much list-type selected response formats and short-essay type constructed response formats are related to differences in how test takers approach clinical reasoning tasks. The design of this study was informed by a framework developed within cognitive psychology which stresses the importance of the interplay between two components of reasoning-self-monitoring and response inhibition-while solving a task or case. The results presented support the argument that different response formats are related to different processing behavior. Importantly, the pattern of how different factors are related to a correct response in both situations seem to be well in line with contemporary accounts of reasoning. Consequently, we argue that when designing assessments of clinical reasoning, it is crucial to tap into the different facets of this complex and important medical process.

Differential diagnosis checklists reduce diagnostic error differentially: A randomised experiment

INTRODUCTION

Wrong and missed diagnoses contribute substantially to medical error. Can a prompt to generate alternative diagnoses (prompt) or a differential diagnosis checklist (DDXC) increase diagnostic accuracy? How do these interventions affect the diagnostic process and self-monitoring?

METHODS

Advanced medical students (N = 90) were randomly assigned to one of four conditions to complete six computer-based patient cases: group 1 (prompt) was instructed to write down all diagnoses they considered while acquiring diagnostic test results and to finally rank them. Groups 2 and 3 received the same instruction plus a list of 17 differential diagnoses for the chief complaint of the patient. For half of the cases, the DDXC contained the correct diagnosis (DDXC+), and for the other half, it did not (DDXC-; counterbalanced). Group 4 (control) was only instructed to indicate their final diagnosis. Mixed-effects models were used to analyse results.

RESULTS

Students using a DDXC that contained the correct diagnosis had better diagnostic accuracy, mean (standard deviation), 0.75 (0.44), compared to controls without a checklist, 0.49 (0.50), P < 0.001, but those using a DDXC that did not contain the correct diagnosis did slightly worse, 0.43 (0.50), P = 0.602. The number and relevance of diagnostic tests acquired were not affected by condition, nor was self-monitoring. However, participants spent more time on a case in the DDXC-, 4:20 min (2:36), P ≤ 0.001, and DDXC+ condition, 3:52 min (2:09), than in the control condition, 2:59 min (1:44), P ≤ 0.001.

DISCUSSION

Being provided a list of possible diagnoses improves diagnostic accuracy compared with a prompt to create a differential diagnosis list, if the provided list contains the correct diagnosis. However, being provided a diagnosis list without the correct diagnosis did not improve and might have slightly reduced diagnostic accuracy. Interventions neither affected information gathering nor self-monitoring.

Comparison of Diagnostic Recommendations from Individual Physicians versus the Collective Intelligence of Multiple Physicians in Ambulatory Cases Referred for Specialist Consultation

BACKGROUND

Studies report higher diagnostic accuracy using the collective intelligence (CI) of multiple clinicians compared with individual clinicians. However, the diagnostic process is iterative, and unexplored is the value of CI in improving clinical recommendations leading to a final diagnosis.

METHODS

To compare the appropriateness of diagnostic recommendations advised by individual physicians versus the CI of physicians, we entered actual consultation requests sent by primary care physicians to specialists onto a web-based CI platform capable of collecting diagnostic recommendations (next steps for care) from multiple physicians. We solicited responses to 35 cases (12 endocrinology, 13 gynecology, 10 neurology) from ≥3 physicians of any specialty through the CI platform, which aggregated responses into a CI output. The primary outcome was the appropriateness of individual physician recommendations versus the CI output recommendations, using recommendations agreed upon by 2 specialists in the same specialty as a gold standard. The secondary outcome was the recommendations' potential for harm.

RESULTS

A total of 177 physicians responded. Cases had a median of 7 respondents (interquartile range: 5-10). Diagnostic recommendations in the CI output achieved higher levels of appropriateness (69%) than recommendations from individual physicians (45%; χ² = 5.95, P = 0.015). Of the CI recommendations, 54% were potentially harmful, as compared with 41% of individuals' recommendations (χ² = 2.49, P = 0.11).

LIMITATIONS

Cases were from a single institution. CI was solicited using a single algorithm/platform.

CONCLUSIONS

When seeking specialist guidance, diagnostic recommendations from the CI of multiple physicians are more appropriate than recommendations from most individual physicians, measured against specialist recommendations. Although CI provides useful recommendations, some have potential for harm. Future research should explore how to use CI to improve diagnosis while limiting harm from inappropriate tests/therapies.

Can decision support combat incompleteness and bias in routine primary care data?

OBJECTIVE

Routine primary care data may be used for the derivation of clinical prediction rules and risk scores. We sought to measure the impact of a decision support system (DSS) on data completeness and freedom from bias.

MATERIALS AND METHODS

We used the clinical documentation of 34 UK general practitioners who took part in a previous study evaluating the DSS. They consulted with 12 standardized patients. In addition to suggesting diagnoses, the DSS facilitates data coding. We compared the documentation from consultations with the electronic health record (EHR) (baseline consultations) vs consultations with the EHR-integrated DSS (supported consultations). We measured the proportion of EHR data items related to the physician's final diagnosis. We expected that in baseline consultations, physicians would document only or predominantly observations related to their diagnosis, while in supported consultations, they would also document other observations as a result of exploring more diagnoses and/or ease of coding.

RESULTS

Supported documentation contained significantly more codes (incidence rate ratio [IRR] = 5.76 [4.31, 7.70] P < .001) and less free text (IRR = 0.32 [0.27, 0.40] P < .001) than baseline documentation. As expected, the proportion of diagnosis-related data was significantly lower (b = -0.08 [-0.11, -0.05] P < .001) in the supported consultations, and this was the case for both codes and free text.

CONCLUSIONS

We provide evidence that data entry in the EHR is incomplete and reflects physicians' cognitive biases. This has serious implications for epidemiological research that uses routine data. A DSS that facilitates and motivates data entry during the consultation can improve routine documentation.

General practitioner and nurse practitioner attitudes towards electronic reminders in primary care: a qualitative analysis

OBJECTIVES

Reminders in primary care administrative systems aim to help clinicians provide evidence-based care, prescribe safely and save money. However, increased use of reminders can lead to alert fatigue. Our study aimed to assess general practitioners' (GPs) and nurse practitioners' (NPs) views on electronic reminders in primary care.

DESIGN

A qualitative analysis using semistructured interviews.

SETTING AND PARTICIPANTS

Fifteen GPs and NP based in general practices located in North-West London and Yorkshire, England.

METHODS

We collected data on participants' views on: (1) perceptions of the value of information provided; (2) reminder-related behaviours and (3) how to improve reminders. We carried out a thematic analysis.

RESULTS

Participants were familiar with reminders in their clinical systems and felt many were important to support their clinical work. However, participants reported, on average, 70% of reminders were ignored. Four major themes emerged: (1) reaction to a reminder, which was mixed and varied by situation. (2) Factors influencing the decision to act on reminders, often related to experience, consultation styles and interests of participants. Time constraints, alert design, inappropriate presentation and litigation were also factors. (3) Negative consequences of using reminders were increased workload or costs and compromising GP and NPs behaviour. (4) Factors relating to improving users' engagement with reminders were prevention of unnecessary reminders through data linkage across healthcare administrative systems or the development of more intelligent algorithms. Participants felt training was vital to effectively manage reminders.

CONCLUSIONS

GPs and NPs believe reminders are useful in supporting the provision of good quality patient care. Improving GPs and NPs' engagement with reminders centres on further developing their relevance to their clinical practice, which is personalised, considers cognitive workflow and suppresses inappropriate presentation.

Barriers and facilitators to the adoption of electronic clinical decision support systems: a qualitative interview study with UK general practitioners

Background

Well-established electronic data capture in UK general practice means that algorithms, developed on patient data, can be used for automated clinical decision support systems (CDSSs). These can predict patient risk, help with prescribing safety, improve diagnosis and prompt clinicians to record extra data. However, there is persistent evidence of low uptake of CDSSs in the clinic. We interviewed UK General Practitioners (GPs) to understand what features of CDSSs, and the contexts of their use, facilitate or present barriers to their use.

Methods

We interviewed 11 practicing GPs in London and South England using a semi-structured interview schedule and discussed a hypothetical CDSS that could detect early signs of dementia. We applied thematic analysis to the anonymised interview transcripts.

Results

We identified three overarching themes: trust in individual CDSSs; usability of individual CDSSs; and usability of CDSSs in the broader practice context, to which nine subthemes contributed. Trust was affected by CDSS provenance, perceived threat to autonomy and clear management guidance. Usability was influenced by sensitivity to the patient context, CDSS flexibility, ease of control, and non-intrusiveness. CDSSs were more likely to be used by GPs if they did not contribute to alert proliferation and subsequent fatigue, or if GPs were provided with training in their use.

Conclusions

Building on these findings we make a number of recommendations for CDSS developers to consider when bringing a new CDSS into GP patient records systems. These include co-producing CDSS with GPs to improve fit within clinic workflow and wider practice systems, ensuring a high level of accuracy and a clear clinical pathway, and providing CDSS training for practice staff. These recommendations may reduce the proliferation of unhelpful alerts that can result in important decision-support being ignored.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01557-z.

Effects of a Differential Diagnosis List of Artificial Intelligence on Differential Diagnoses by Physicians: An Exploratory Analysis of Data from a Randomized Controlled Study

A diagnostic decision support system (DDSS) is expected to reduce diagnostic errors. However, its effect on physicians' diagnostic decisions remains unclear. Our study aimed to assess the prevalence of diagnoses from artificial intelligence (AI) in physicians' differential diagnoses when using AI-driven DDSS that generates a differential diagnosis from the information entered by the patient before the clinical encounter on physicians' differential diagnoses. In this randomized controlled study, an exploratory analysis was performed. Twenty-two physicians were required to generate up to three differential diagnoses per case by reading 16 clinical vignettes. The participants were divided into two groups, an intervention group, and a control group, with and without a differential diagnosis list of AI, respectively. The prevalence of physician diagnosis identical with the differential diagnosis of AI (primary outcome) was significantly higher in the intervention group than in the control group (70.2% vs. 55.1%, p < 0.001). The primary outcome was significantly >10% higher in the intervention group than in the control group, except for attending physicians, and physicians who did not trust AI. This study suggests that at least 15% of physicians' differential diagnoses were affected by the differential diagnosis list in the AI-driven DDSS.

Impact of digitally acquired peer diagnostic input on diagnostic confidence in outpatient cases: A pragmatic randomized trial

OBJECTIVE

The study sought to evaluate if peer input on outpatient cases impacted diagnostic confidence.

MATERIALS AND METHODS

This randomized trial of a peer input intervention occurred among 28 clinicians with case-level randomization. Encounters with diagnostic uncertainty were entered onto a digital platform to collect input from ≥5 clinicians. The primary outcome was diagnostic confidence. We used mixed-effects logistic regression analyses to assess for intervention impact on diagnostic confidence.

RESULTS

Among the 509 cases (255 control; 254 intervention), the intervention did not impact confidence (odds ratio [OR], 1.46; 95% confidence interval [CI], 0.999-2.12), but after adjusting for clinician and case traits, the intervention was associated with higher confidence (OR, 1.53; 95% CI, 1.01-2.32). The intervention impact was greater in cases with high uncertainty (OR, 3.23; 95% CI, 1.09- 9.52).

CONCLUSIONS

Peer input increased diagnostic confidence primarily in high-uncertainty cases, consistent with findings that clinicians desire input primarily in cases with continued uncertainty.

Assessing the utility of a differential diagnostic generator in UK general practice: a feasibility study

BACKGROUND

Despite growing positive evidence supporting the potential utility of differential diagnostic generator (DDX) tools, uptake has been limited in terms of geography and settings and calls have been made to test such tools in wider routine clinical settings. This study aims to evaluate the feasibility and utility of clinical use of Isabel, an electronic DDX tool, in a United Kingdom (UK) general practice setting.

METHODS

Mixed methods. Feasibility and utility were assessed prospectively over a 6-month period via: usage statistics, survey as well as interview data generated from clinicians before and after Isabel was available for clinical use. Normalisation process theory (NPT) was utilised as a sensitising concept in the data collection and analysis of the qualitative data.

RESULTS

Usage was extremely limited (n = 18 searches). Most potential users did not utilise the program and of those that did (n = 6), usage was restricted and did not alter subsequent patient management. Baseline interview findings indicated some prior awareness of DDX tools and ambivalent views with regards to potential utility. Post-use interviews supported analytic data and indicated low usage due to a range of endogenous (professional) and exogenous (organisational) factors.

CONCLUSIONS

In its current form, this small exploratory study suggests that Isabel is a tool that is unlikely to be utilised on a routine basis in primary care, but may have potential utility for diagnostic support in (1) education/training and (2) rare and diagnostically complex cases.

Efficacy of Artificial-Intelligence-Driven Differential-Diagnosis List on the Diagnostic Accuracy of Physicians: An Open-Label Randomized Controlled Study

Background: The efficacy of artificial intelligence (AI)-driven automated medical-history-taking systems with AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy was shown. However, considering the negative effects of AI-driven differential-diagnosis lists such as omission (physicians reject a correct diagnosis suggested by AI) and commission (physicians accept an incorrect diagnosis suggested by AI) errors, the efficacy of AI-driven automated medical-history-taking systems without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy should be evaluated. Objective: The present study was conducted to evaluate the efficacy of AI-driven automated medical-history-taking systems with or without AI-driven differential-diagnosis lists on physicians’ diagnostic accuracy. Methods: This randomized controlled study was conducted in January 2021 and included 22 physicians working at a university hospital. Participants were required to read 16 clinical vignettes in which the AI-driven medical history of real patients generated up to three differential diagnoses per case. Participants were divided into two groups: with and without an AI-driven differential-diagnosis list. Results: There was no significant difference in diagnostic accuracy between the two groups (57.4% vs. 56.3%, respectively; p = 0.91). Vignettes that included a correct diagnosis in the AI-generated list showed the greatest positive effect on physicians’ diagnostic accuracy (adjusted odds ratio 7.68; 95% CI 4.68–12.58; p < 0.001). In the group with AI-driven differential-diagnosis lists, 15.9% of diagnoses were omission errors and 14.8% were commission errors. Conclusions: Physicians’ diagnostic accuracy using AI-driven automated medical history did not differ between the groups with and without AI-driven differential-diagnosis lists.

Identification of Risk Factors and Symptoms of COVID-19: Analysis of Biomedical Literature and Social Media Data

BACKGROUND

In December 2019, the COVID-19 outbreak started in China and rapidly spread around the world. Lack of a vaccine or optimized intervention raised the importance of characterizing risk factors and symptoms for the early identification and successful treatment of patients with COVID-19.

OBJECTIVE

This study aims to investigate and analyze biomedical literature and public social media data to understand the association of risk factors and symptoms with the various outcomes observed in patients with COVID-19.

METHODS

Through semantic analysis, we collected 45 retrospective cohort studies, which evaluated 303 clinical and demographic variables across 13 different outcomes of patients with COVID-19, and 84,140 Twitter posts from 1036 COVID-19-positive users. Machine learning tools to extract biomedical information were introduced to identify mentions of uncommon or novel symptoms in tweets. We then examined and compared two data sets to expand our landscape of risk factors and symptoms related to COVID-19.

RESULTS

From the biomedical literature, approximately 90% of clinical and demographic variables showed inconsistent associations with COVID-19 outcomes. Consensus analysis identified 72 risk factors that were specifically associated with individual outcomes. From the social media data, 51 symptoms were characterized and analyzed. By comparing social media data with biomedical literature, we identified 25 novel symptoms that were specifically mentioned in tweets but have been not previously well characterized. Furthermore, there were certain combinations of symptoms that were frequently mentioned together in social media.

CONCLUSIONS

Identified outcome-specific risk factors, symptoms, and combinations of symptoms may serve as surrogate indicators to identify patients with COVID-19 and predict their clinical outcomes in order to provide appropriate treatments.

A Perspective from a Case Conference on Comparing the Diagnostic Process: Human Diagnostic Thinking vs. Artificial Intelligence (AI) Decision Support Tools

Artificial intelligence (AI) has made great contributions to the healthcare industry. However, its effect on medical diagnosis has not been well explored. Here, we examined a trial comparing the thinking process between a computer and a master in diagnosis at a clinical conference in Japan, with a focus on general diagnosis. Consequently, not only was AI unable to exhibit its thinking process, it also failed to include the final diagnosis. The following issues were highlighted: (1) input information to AI could not be weighted in order of importance for diagnosis; (2) AI could not deal with comorbidities (see Hickam’s dictum); (3) AI was unable to consider the timeline of the illness (depending on the tool); (4) AI was unable to consider patient context; (5) AI could not obtain input information by themselves. This comparison of the thinking process uncovered a future perspective on the use of diagnostic support tools.

Association between use of urgent suspected cancer referral and mortality and stage at diagnosis: a 5-year national cohort study

BACKGROUND

There is considerable variation between GP practices in England in their use of urgent referral pathways for suspected cancer.

AIM

To determine the association between practice use of urgent referral and cancer stage at diagnosis and cancer patient mortality, for all cancers and the most common types of cancer (colorectal, lung, breast, and prostate).

DESIGN AND SETTING

National cohort study of 1.4 million patients diagnosed with cancer in England between 2011 and 2015.

METHOD

The cohort was stratified according to quintiles of urgent referral metrics. Cox proportional hazards regression was used to quantify risk of death, and logistic regression to calculate odds of late-stage (III/IV) versus early-stage (I/II) cancers in relation to referral quintiles and cancer type.

RESULTS

Cancer patients from the highest referring practices had a lower hazard of death (hazard ratio [HR] = 0.96; 95% confidence interval [CI] = 0.95 to 0.97), with similar patterns for individual cancers: colorectal (HR = 0.95; CI = 0.93 to 0.97); lung (HR = 0.95; CI = 0.94 to 0.97); breast (HR = 0.96; CI = 0.93 to 0.99); and prostate (HR = 0.88; CI = 0.85 to 0.91). Similarly, for cancer patients from these practices, there were lower odds of late-stage diagnosis for individual cancer types, except for colorectal cancer.

CONCLUSION

Higher practice use of referrals for suspected cancer is associated with lower mortality for the four most common types of cancer. A significant proportion of the observed mortality reduction is likely due to earlier stage at diagnosis, except for colorectal cancer. This adds to evidence supporting the lowering of referral thresholds and consequent increased use of urgent referral for suspected cancer.

Decision support tools to improve cancer diagnostic decision making in primary care: a systematic review

BACKGROUND

The diagnosis of cancer in primary care is complex and challenging. Electronic clinical decision support tools (eCDSTs) have been proposed as an approach to improve GP decision making, but no systematic review has examined their role in cancer diagnosis.

AIM

To investigate whether eCDSTs improve diagnostic decision making for cancer in primary care and to determine which elements influence successful implementation.

DESIGN AND SETTING

A systematic review of relevant studies conducted worldwide and published in English between 1 January 1998 and 31 December 2018.

METHOD

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed. MEDLINE, EMBASE, and the Cochrane Central Register of Controlled Trials were searched, and a consultation of reference lists and citation tracking was carried out. Exclusion criteria included the absence of eCDSTs used in asymptomatic populations, and studies that did not involve support delivered to the GP. The most relevant Joanna Briggs Institute Critical Appraisal Checklists were applied according to study design of the included paper.

RESULTS

Of the nine studies included, three showed improvements in decision making for cancer diagnosis, three demonstrated positive effects on secondary clinical or health service outcomes such as prescribing, quality of referrals, or cost-effectiveness, and one study found a reduction in time to cancer diagnosis. Barriers to implementation included trust, the compatibility of eCDST recommendations with the GP's role as a gatekeeper, and impact on workflow.

CONCLUSION

eCDSTs have the capacity to improve decision making for a cancer diagnosis, but the optimal mode of delivery remains unclear. Although such tools could assist GPs in the future, further well-designed trials of all eCDSTs are needed to determine their cost-effectiveness and the most appropriate implementation methods.

Differential Diagnosis Assessment in Ambulatory Care With an Automated Medical History-Taking Device: Pilot Randomized Controlled Trial

Background

Automated medical history–taking devices (AMHTDs) are emerging tools with the potential to increase the quality of medical consultations by providing physicians with an exhaustive, high-quality, standardized anamnesis and differential diagnosis.

Objective

This study aimed to assess the effectiveness of an AMHTD to obtain an accurate differential diagnosis in an outpatient service.

Methods

We conducted a pilot randomized controlled trial involving 59 patients presenting to an emergency outpatient unit and suffering from various conditions affecting the limbs, the back, and the chest wall. Resident physicians were randomized into 2 groups, one assisted by the AMHTD and one without access to the device. For each patient, physicians were asked to establish an exhaustive differential diagnosis based on the anamnesis and clinical examination. In the intervention group, residents read the AMHTD report before performing the anamnesis. In both the groups, a senior physician had to establish a differential diagnosis, considered as the gold standard, independent of the resident’s opinion and AMHTD report.

Results

A total of 29 patients were included in the intervention group and 30 in the control group. Differential diagnosis accuracy was higher in the intervention group (mean 75%, SD 26%) than in the control group (mean 59%, SD 31%; P=.01). Subgroup analysis showed a between-group difference of 3% (83% [17/21]-80% [14/17]) for low complexity cases (1-2 differential diagnoses possible) in favor of the AMHTD (P=.76), 31% (87% [13/15]-56% [18/33]) for intermediate complexity (3 differential diagnoses; P=.02), and 24% (63% [34/54]-39% [14/35]) for high complexity (4-5 differential diagnoses; P=.08). Physicians in the intervention group (mean 4.3, SD 2) had more years of clinical practice compared with the control group (mean 5.5, SD 2; P=.03). Differential diagnosis accuracy was negatively correlated to case complexity (r=0.41; P=.001) and the residents’ years of practice (r=0.04; P=.72). The AMHTD was able to determine 73% (SD 30%) of correct differential diagnoses. Patient satisfaction was good (4.3/5), and 26 of 29 patients (90%) considered that they were able to accurately describe their symptomatology. In 8 of 29 cases (28%), residents considered that the AMHTD helped to establish the differential diagnosis.

Conclusions

The AMHTD allowed physicians to make more accurate differential diagnoses, particularly in complex cases. This could be explained not only by the ability of the AMHTD to make the right diagnoses, but also by the exhaustive anamnesis provided.

The information distortion bias: implications for medical decisions

CONTEXT

Every diagnosis involves an act of decision making, which requires proper evaluation of information. However, even seemingly objective information can require interpretation, often without our conscious awareness. In this cross-cutting edge article we describe the phenomenon of leader-driven information distortion (ID) and its implications for medical education.

INFORMATION DISTORTION

Recent research indicates that one threat to good decisions is a biased interpretation of information to favour one alternative course of action over another. Once an alternative emerges as a leader during a decision there is a strong tendency to evaluate subsequent information as supporting that option. This can occur when deciding between two competing diagnoses. It is particularly a concern if diagnostic tests provide potentially ambiguous results. This leader-driven ID is pre-decisional in nature, in that it develops during a decision and involves the interpretation of information available prior to the final decision or diagnosis, with different interpretations possible depending on whichever alternative is the leader. Studies reveal that the distortion bias is pervasive in decisions, and that awareness of the act of distortion is low in decision makers.

APPLICATION TO MEDICAL EDUCATION

Empirical research has confirmed the presence of leader-driven ID in hypothetical diagnoses made by physicians. ID creates two threats to medical decisions: First, it can make a diagnosis sticky in that it is resistant to being overturned by contradictory information. Second, it can promote unwarranted certainty in a diagnosis. The outcome may be premature closure, unnecessary testing or incorrect treatment, resulting in delayed or missed diagnoses.

METHODS

This paper summarises research related to leader-driven ID in medical and professional decisions and discusses various approaches directed towards reducing ID. A framework and language are provided for thinking about and discussing ID in medical decisions and medical education. Courses of action for mitigating the effects of ID are suggested.

Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study

BACKGROUND

Rare disease diagnosis is often delayed by years. A primary factor for this delay is a lack of knowledge and awareness regarding rare diseases. Probabilistic diagnostic decision support systems (DDSSs) have the potential to accelerate rare disease diagnosis by suggesting differential diagnoses for physicians based on case input and incorporated medical knowledge. We examine the DDSS prototype Ada DX and assess its potential to provide accurate rare disease suggestions early in the course of rare disease cases.

RESULTS

Ada DX suggested the correct disease earlier than the time of clinical diagnosis among the top five fit disease suggestions in 53.8% of cases (50 of 93), and as the top fit disease suggestion in 37.6% of cases (35 of 93). The median advantage of correct disease suggestions compared to the time of clinical diagnosis was 3 months or 50% for top five fit and 1 month or 21% for top fit. The correct diagnosis was suggested at the first documented patient visit in 33.3% (top 5 fit), and 16.1% of cases (top fit), respectively. Wilcoxon signed-rank test shows a significant difference between the time to clinical diagnosis and the time to correct disease suggestion for both top five fit and top fit (z-score -6.68, respective -5.71, α=0.05, p-value <0.001).

CONCLUSION

Ada DX provided accurate rare disease suggestions in most rare disease cases. In many cases, Ada DX provided correct rare disease suggestions early in the course of the disease, sometimes at the very beginning of a patient journey. The interpretation of these results indicates that Ada DX has the potential to suggest rare diseases to physicians early in the course of a case. Limitations of this study derive from its retrospective and unblinded design, data input by a single user, and the optimization of the knowledge base during the course of the study. Results pertaining to the system's accuracy should be interpreted cautiously. Whether the use of Ada DX reduces the time to diagnosis in rare diseases in a clinical setting should be validated in prospective studies.

Understanding implementation and usefulness of electronic clinical decision support (eCDS) for melanoma in English primary care: a qualitative investigation

Background

Timely diagnosis of the serious skin cancer melanoma can improve patient outcomes. Clinical guidelines suggest that GPs use checklists, such as the 7-point checklist (7PCL), to assess pigmented lesions. In 2016, the 7PCL was disseminated by EMIS as an electronic clinical decision support (eCDS) tool.

Aim

To understand GP and patient perspectives on the implementation and usefulness of the eCDS.

Design & setting

Semi-structured interviews with GPs and patients were undertaken. The interviews took place in four general practices in the south east of England following consultations using the eCDS for suspicious pigmented lesions.

Method

Data were collected from semi-structured face-to-face interviews with GPs and from telephone interviews with patients. They were recorded and transcribed verbatim. The Consolidated Framework for Implementation Research (CFIR) underpinned the analysis using thematic approaches.

Results

A total of 14 interviews with GPs and 14 interviews with patients were undertaken. Most GPs reported that, as the eCDS was embedded in the medical record, it was useful, easy to use, time-efficient, and could facilitate patient–GP communication. They were less clear that it could meet policy or patient needs to improve early diagnosis, and some felt that it could lead to unnecessary referrals. Few felt that it had been sufficiently implemented at practice level. More felt confident with their own management of moles, and that the eCDS could be most useful for borderline decision-making. No patients were aware that the eCDS had been used during their consultation.

Conclusion

Successful implementation of a new tool, such as eCDS for melanoma, requires GPs to perceive its value and understand how it can best be integrated into clinical practice. Disseminating a tool without such explanations is unlikely to promote its adoption into routine practice.

Is symptom-based diagnosis of lung cancer possible? A systematic review and meta-analysis of symptomatic lung cancer prior to diagnosis for comparison with real-time data from routine general practice

BACKGROUND

Lung cancer is a good example of the potential benefit of symptom-based diagnosis, as it is the commonest cancer worldwide, with the highest mortality from late diagnosis and poor symptom recognition. The diagnosis and risk assessment tools currently available have been shown to require further validation. In this study, we determine the symptoms associated with lung cancer prior to diagnosis and demonstrate that by separating prior risk based on factors such as smoking history and age, from presenting symptoms and combining them at the individual patient level, we can make greater use of this knowledge to create a practical framework for the symptomatic diagnosis of individual patients presenting in primary care.

AIM

To provide an evidence-based analysis of symptoms observed in lung cancer patients prior to diagnosis.

DESIGN AND SETTING

Systematic review and meta-analysis of primary and secondary care data.

METHOD

Seven databases were searched (MEDLINE, Embase, Cumulative Index to Nursing and Allied Health Literature, Health Management Information Consortium, Web of Science, British Nursing Index and Cochrane Library). Thirteen studies were selected based on predetermined eligibility and quality criteria for diagnostic assessment to establish the value of symptom-based diagnosis using diagnosistic odds ratio (DOR) and summary receiver operating characteristic (SROC) curve. In addition, routinely collated real-time data from primary care electronic health records (EHR), TransHis, was analysed to compare with our findings.

RESULTS

Haemoptysis was found to have the greatest diagnostic value for lung cancer, diagnostic odds ratio (DOR) 6.39 (3.32-12.28), followed by dyspnoea 2.73 (1.54-4.85) then cough 2.64 (1.24-5.64) and lastly chest pain 2.02 (0.88-4.60). The use of symptom-based diagnosis to accurately diagnose lung cancer cases from non-cases was determined using the summary receiver operating characteristic (SROC) curve, the area under the curve (AUC) was consistently above 0.6 for each of the symptoms described, indicating reasonable discriminatory power. The positive predictive value (PPV) of diagnostic symptoms depends on an individual's prior risk of lung cancer, as well as their presenting symptom pattern. For at risk individuals we calculated prior risk using validated epidemiological models for risk factors such as age and smoking history, then combined with the calculated likelihood ratios for each symptom to establish posterior risk or positive predictive value (PPV).

CONCLUSION

Our findings show that there is diagnostic value in the clinical symptoms associated with lung cancer and the potential benefit of characterising these symptoms using routine data studies to identify high-risk patients.

Requirements and validation of a prototype learning health system for clinical diagnosis

INTRODUCTION

Diagnostic error is a major threat to patient safety in the context of family practice. The patient safety implications are severe for both patient and clinician. Traditional approaches to diagnostic decision support have lacked broad acceptance for a number of well-documented reasons: poor integration with electronic health records and clinician workflow, static evidence that lacks transparency and trust, and use of proprietary technical standards hindering wider interoperability. The learning health system (LHS) provides a suitable infrastructure for development of a new breed of learning decision support tools. These tools exploit the potential for appropriate use of the growing volumes of aggregated sources of electronic health records.

METHODS

We describe the experiences of the TRANSFoRm project developing a diagnostic decision support infrastructure consistent with the wider goals of the LHS. We describe an architecture that is model driven, service oriented, constructed using open standards, and supports evidence derived from electronic sources of patient data. We describe the architecture and implementation of 2 critical aspects for a successful LHS: the model representation and translation of clinical evidence into effective practice and the generation of curated clinical evidence that can be used to populate those models, thus closing the LHS loop.

RESULTS/CONCLUSIONS

Six core design requirements for implementing a diagnostic LHS are identified and successfully implemented as part of this research work. A number of significant technical and policy challenges are identified for the LHS community to consider, and these are discussed in the context of evaluating this work: medico-legal responsibility for generated diagnostic evidence, developing trust in the LHS (particularly important from the perspective of decision support), and constraints imposed by clinical terminologies on evidence generation.

Debiasing Health-Related Judgments and Decision Making: A Systematic Review

BACKGROUND

Being confronted with uncertainty in the context of health-related judgments and decision making can give rise to the occurrence of systematic biases. These biases may detrimentally affect lay persons and health experts alike. Debiasing aims at mitigating these negative effects by eliminating or reducing the biases. However, little is known about its effectiveness. This study seeks to systematically review the research on health-related debiasing to identify new opportunities and challenges for successful debiasing strategies.

METHODS

A systematic search resulted in 2748 abstracts eligible for screening. Sixty-eight articles reporting 87 relevant studies met the predefined inclusion criteria and were categorized and analyzed with regard to content and quality. All steps were undertaken independently by 2 reviewers, and inconsistencies were resolved through discussion.

RESULTS

The majority of debiasing interventions ( n = 60) was at least partially successful. Optimistic biases ( n = 25), framing effects ( n = 14), and base rate neglects ( n = 10) were the main targets of debiasing efforts. Cognitive strategies ( n = 36) such as "consider-the-opposite" and technological interventions ( n = 33) such as visual aids were mainly tested. Thirteen studies aimed at debiasing health care professionals' judgments, while 74 interventions addressed the general population. Studies' methodological quality ranged from 26.2% to 92.9%, with an average rating of 68.7%.

DISCUSSION

In the past, the usefulness of debiasing was often debated. Yet most of the interventions reviewed here are found to be effective, pointing to the utility of debiasing in the health context. In particular, technological strategies offer a novel opportunity to pursue large-scale debiasing outside the laboratory. The need to strengthen the transfer of debiasing interventions to real-life settings and a lack of conceptual rigor are identified as the main challenges requiring further research.

The impact of a diagnostic decision support system on the consultation: perceptions of GPs and patients

Background

Clinical decision support systems (DSS) aimed at supporting diagnosis are not widely used. This is mainly due to usability issues and lack of integration into clinical work and the electronic health record (EHR). In this study we examined the usability and acceptability of a diagnostic DSS prototype integrated with the EHR and in comparison with the EHR alone.

Methods

Thirty-four General Practitioners (GPs) consulted with 6 standardised patients (SPs) using only their EHR system (baseline session); on another day, they consulted with 6 different but matched for difficulty SPs, using the EHR with the integrated DSS prototype (DSS session). GPs were interviewed twice (at the end of each session), and completed the Post-Study System Usability Questionnaire at the end of the DSS session. The SPs completed the Consultation Satisfaction Questionnaire after each consultation.

Results

The majority of GPs (74%) found the DSS useful: it helped them consider more diagnoses and ask more targeted questions. They considered three user interface features to be the most useful: (1) integration with the EHR; (2) suggested diagnoses to consider at the start of the consultation and; (3) the checklist of symptoms and signs in relation to each suggested diagnosis. There were also criticisms: half of the GPs felt that the DSS changed their consultation style, by requiring them to code symptoms and signs while interacting with the patient. SPs sometimes commented that GPs were looking at their computer more than at them; this comment was made more often in the DSS session (15%) than in the baseline session (3%). Nevertheless, SP ratings on the satisfaction questionnaire did not differ between the two sessions.

Conclusions

To use the DSS effectively, GPs would need to adapt their consultation style, so that they code more information during rather than at the end of the consultation. This presents a potential barrier to adoption. Training GPs to use the system in a patient-centred way, as well as improvement of the DSS interface itself, could facilitate coding. To enhance patient acceptability, patients should be informed about the potential of the DSS to improve diagnostic accuracy.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0477-6) contains supplementary material, which is available to authorized users.

Variations in GPs' decisions to investigate suspected lung cancer: a factorial experiment using multimedia vignettes

INTRODUCTION

Lung cancer survival is low and comparatively poor in the UK. Patients with symptoms suggestive of lung cancer commonly consult primary care, but it is unclear how general practitioners (GPs) distinguish which patients require further investigation. This study examined how patients' clinical and sociodemographic characteristics influence GPs' decisions to initiate lung cancer investigations.

METHODS

A factorial experiment was conducted among a national sample of 227 English GPs using vignettes presented as simulated consultations. A multimedia-interactive website simulated key features of consultations using actors ('patients'). GP participants made management decisions online for six 'patients', whose sociodemographic characteristics systematically varied across three levels of cancer risk. In low-risk vignettes, investigation (ie, chest X-ray ordered, computerised tomography scan or respiratory consultant referral) was not indicated; in medium-risk vignettes, investigation could be appropriate; in high-risk vignettes, investigation was definitely indicated. Each 'patient' had two lung cancer-related symptoms: one volunteered and another elicited if GPs asked. Variations in investigation likelihood were examined using multilevel logistic regression.

RESULTS

GPs decided to investigate lung cancer in 74% (1000/1348) of vignettes. Investigation likelihood did not increase with cancer risk. Investigations were more likely when GPs requested information on symptoms that 'patients' had but did not volunteer (adjusted OR (AOR)=3.18; 95% CI 2.27 to 4.70). However, GPs omitted to seek this information in 42% (570/1348) of cases. GPs were less likely to investigate older than younger 'patients' (AOR=0.52; 95% CI 0.39 to 0.7) and black 'patients' than white (AOR=0.68; 95% CI 0.48 to 0.95).

CONCLUSIONS

GPs were not more likely to investigate 'patients' with high-risk than low-risk cancer symptoms. Furthermore, they did not investigate everyone with the same symptoms equally. Insufficient data gathering could be responsible for missed opportunities in diagnosis.

The global burden of diagnostic errors in primary care

Diagnosis is one of the most important tasks performed by primary care physicians. The World Health Organization (WHO) recently prioritized patient safety areas in primary care, and included diagnostic errors as a high-priority problem. In addition, a recent report from the Institute of Medicine in the USA, 'Improving Diagnosis in Health Care', concluded that most people will likely experience a diagnostic error in their lifetime. In this narrative review, we discuss the global significance, burden and contributory factors related to diagnostic errors in primary care. We synthesize available literature to discuss the types of presenting symptoms and conditions most commonly affected. We then summarize interventions based on available data and suggest next steps to reduce the global burden of diagnostic errors. Research suggests that we are unlikely to find a 'magic bullet' and confirms the need for a multifaceted approach to understand and address the many systems and cognitive issues involved in diagnostic error. Because errors involve many common conditions and are prevalent across all countries, the WHO's leadership at a global level will be instrumental to address the problem. Based on our review, we recommend that the WHO consider bringing together primary care leaders, practicing frontline clinicians, safety experts, policymakers, the health IT community, medical education and accreditation organizations, researchers from multiple disciplines, patient advocates, and funding bodies among others, to address the many common challenges and opportunities to reduce diagnostic error. This could lead to prioritization of practice changes needed to improve primary care as well as setting research priorities for intervention development to reduce diagnostic error.

Primary care and cancer: Facing the challenge of early diagnosis and survivorship

With ageing populations and an increasing lifetime risk of cancer, primary care will continue to play an increasingly important role in early diagnosis and cancer survivorship, especially with the lowering of risk thresholds for referral and diagnostic investigations. However, primary care in many countries is in crisis with increasing workloads for primary care physicians. Potential solutions to these challenges will be outlined including development of multidisciplinary teams, diagnostic decision support, increasing access to diagnostics and cost-effective referral pathways.

How doctors diagnose diseases and prescribe treatments: an fMRI study of diagnostic salience

Understanding the brain mechanisms involved in diagnostic reasoning may contribute to the development of methods that reduce errors in medical practice. In this study we identified similar brain systems for diagnosing diseases, prescribing treatments, and naming animals and objects using written information as stimuli. Employing time resolved modeling of blood oxygen level dependent (BOLD) responses enabled time resolved (400 milliseconds epochs) analyses. With this approach it was possible to study neural processes during successive stages of decision making. Our results showed that highly diagnostic information, reducing uncertainty about the diagnosis, decreased monitoring activity in the frontoparietal attentional network and may contribute to premature diagnostic closure, an important cause of diagnostic errors. We observed an unexpected and remarkable switch of BOLD activity within a right lateralized set of brain regions related to awareness and auditory monitoring at the point of responding. We propose that this neurophysiological response is the neural substrate of awareness of one’s own (verbal) response. Our results highlight the intimate relation between attentional mechanisms, uncertainty, and decision making and may assist the advance of approaches to prevent premature diagnostic closure.

Early recognition of multiple sclerosis using natural language processing of the electronic health record

BACKGROUND

Diagnostic accuracy might be improved by algorithms that searched patients' clinical notes in the electronic health record (EHR) for signs and symptoms of diseases such as multiple sclerosis (MS). The focus this study was to determine if patients with MS could be identified from their clinical notes prior to the initial recognition by their healthcare providers.

METHODS

An MS-enriched cohort of patients with well-established MS (n = 165) and controls (n = 545), was generated from the adult outpatient clinic. A random sample cohort was generated from randomly selected patients (n = 2289) from the same adult outpatient clinic, some of whom had MS (n = 16). Patients' notes were extracted from the data warehouse and signs and symptoms mapped to UMLS terms using MedLEE. Approximately 1000 MS-related terms occurred significantly more frequently in MS patients' notes than controls'. Synonymous terms were manually clustered into 50 buckets and used as classification features. Patients were classified as MS or not using Naïve Bayes classification.

RESULTS

Classification of patients known to have MS using notes of the MS-enriched cohort entered after the initial ICD9[MS] code yielded an ROC AUC, sensitivity, and specificity of 0.90 [0.87-0.93], 0.75[0.66-0.82], and 0.91 [0.87-0.93], respectively. Similar classification accuracy was achieved using the notes from the random sample cohort. Classification of patients not yet known to have MS using notes of the MS-enriched cohort entered before the initial ICD9[MS] documentation identified 40% [23-59%] as having MS. Manual review of the EHR of 45 patients of the random sample cohort classified as having MS but lacking an ICD9[MS] code identified four who might have unrecognized MS.

CONCLUSIONS

Diagnostic accuracy might be improved by mining patients' clinical notes for signs and symptoms of specific diseases using NLP. Using this approach, we identified patients with MS early in the course of their disease which could potentially shorten the time to diagnosis. This approach could also be applied to other diseases often missed by primary care providers such as cancer. Whether implementing computerized diagnostic support ultimately shortens the time from earliest symptoms to formal recognition of the disease remains to be seen.

Diagnostic accuracy of GPs when using an early-intervention decision support system: a high-fidelity simulation

Background

Observational and experimental studies of the diagnostic task have demonstrated the importance of the first hypotheses that come to mind for accurate diagnosis. A prototype decision support system (DSS) designed to support GPs’ first impressions has been integrated with a commercial electronic health record (EHR) system.

Aim

To evaluate the prototype DSS in a high-fidelity simulation.

Design and setting

Within-participant design: 34 GPs consulted with six standardised patients (actors) using their usual EHR. On a different day, GPs used the EHR with the integrated DSS to consult with six other patients, matched for difficulty and counterbalanced.

Method

Entering the reason for encounter triggered the DSS, which provided a patient-specific list of potential diagnoses, and supported coding of symptoms during the consultation. At each consultation, GPs recorded their diagnosis and management. At the end, they completed a usability questionnaire. The actors completed a satisfaction questionnaire after each consultation.

Results

There was an 8–9% absolute improvement in diagnostic accuracy when the DSS was used. This improvement was significant (odds ratio [OR] 1.41, 95% confidence interval [CI] = 1.13 to 1.77, P<0.01). There was no associated increase of investigations ordered or consultation length. GPs coded significantly more data when using the DSS (mean 12.35 with the DSS versus 1.64 without), and were generally satisfied with its usability. Patient satisfaction ratings were the same for consultations with and without the DSS.

Conclusion

The DSS prototype was successfully employed in simulated consultations of high fidelity, with no measurable influences on patient satisfaction. The substantially increased data coding can operate as motivation for future DSS adoption.

The Role of Physicians' First Impressions in the Diagnosis of Possible Cancers without Alarm Symptoms

BACKGROUND

First impressions are thought to exert a disproportionate influence on subsequent judgments; however, their role in medical diagnosis has not been systematically studied. We aimed to elicit and measure the association between first impressions and subsequent diagnoses in common presentations with subtle indications of cancer.

METHODS

Ninety UK family physicians conducted interactive simulated consultations online, while on the phone with a researcher. They saw 6 patient cases, 3 of which could be cancers. Each cancer case included 2 consultations, whereby each patient consulted again with nonimproving and some new symptoms. After reading an introduction (patient description and presenting problem), physicians could request more information, which the researcher displayed online. In 2 of the possible cancers, physicians thought aloud. Two raters coded independently the physicians' first utterances (after reading the introduction but before requesting more information) as either acknowledging the possibility of cancer or not. We measured the association of these first impressions with the final diagnoses and management decisions.

RESULTS

The raters coded 297 verbalizations with high interrater agreement (Kappa = 0.89). When the possibility of cancer was initially verbalized, the odds of subsequently diagnosing it were on average 5 times higher (odds ratio 4.90 [95% CI 2.72 to 8.84], P < 0.001), while the odds of appropriate referral doubled (OR 1.98 [1.10 to 3.57], P = 0.002). The number of cancer-related questions physicians asked mediated the relationship between first impressions and subsequent diagnosis, explaining 29% of the total effect.

CONCLUSION

We measured a strong association between family physicians' first diagnostic impressions and subsequent diagnoses and decisions. We suggest that interventions to influence and support the diagnostic process should target its early stage of hypothesis generation.

Reducing diagnostic errors in primary care. A systematic meta-review of computerized diagnostic decision support systems by the LINNEAUS collaboration on patient safety in primary care

Background: Computerized diagnostic decision support systems (CDDSS) have the potential to support the cognitive task of diagnosis, which is one of the areas where general practitioners have greatest difficulty and which accounts for a significant proportion of adverse events recorded in the primary care setting.

Objective: To determine the extent to which CDDSS may meet the requirements of supporting the cognitive task of diagnosis, and the currently perceived barriers that prevent the integration of CDDSS with electronic health record (EHR) systems.

Methods: We conducted a meta-review of existing systematic reviews published in English, searching MEDLINE, Embase, PsycINFO and Web of Knowledge for articles on the features and effectiveness of CDDSS for medical diagnosis published since 2004. Eligibility criteria included systematic reviews where individual clinicians were primary end users. Outcomes we were interested in were the effectiveness and identification of specific features of CDDSS on diagnostic performance.

Results: We identified 1970 studies and excluded 1938 because they did not fit our inclusion criteria. A total of 45 articles were identified and 12 were found suitable for meta-review. Extraction of high-level requirements identified that a more standardized computable approach is needed to knowledge representation, one that can be readily updated as new knowledge is gained. In addition, a deep integration with the EHR is needed in order to trigger at appropriate points in cognitive workflow.

Conclusion: Developing a CDDSS that is able to utilize dynamic vocabulary tools to quickly capture and code relevant diagnostic findings, and coupling these with individualized diagnostic suggestions based on the best-available evidence has the potential to improve diagnostic accuracy, but requires evaluation.