1
|
Knitza J, Hasanaj R, Beyer J, Ganzer F, Slagman A, Bolanaki M, Napierala H, Schmieding ML, Al-Zaher N, Orlemann T, Muehlensiepen F, Greenfield J, Vuillerme N, Kuhn S, Schett G, Achenbach S, Dechant K. Comparison of Two Symptom Checkers (Ada and Symptoma) in the Emergency Department: Randomized, Crossover, Head-to-Head, Double-Blinded Study. J Med Internet Res 2024; 26:e56514. [PMID: 39163594 PMCID: PMC11372320 DOI: 10.2196/56514] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Revised: 03/19/2024] [Accepted: 06/21/2024] [Indexed: 08/22/2024] Open
Abstract
BACKGROUND Emergency departments (EDs) are frequently overcrowded and increasingly used by nonurgent patients. Symptom checkers (SCs) offer on-demand access to disease suggestions and recommended actions, potentially improving overall patient flow. Contrary to the increasing use of SCs, there is a lack of supporting evidence based on direct patient use. OBJECTIVE This study aimed to compare the diagnostic accuracy, safety, usability, and acceptance of 2 SCs, Ada and Symptoma. METHODS A randomized, crossover, head-to-head, double-blinded study including consecutive adult patients presenting to the ED at University Hospital Erlangen. Patients completed both SCs, Ada and Symptoma. The primary outcome was the diagnostic accuracy of SCs. In total, 6 blinded independent expert raters classified diagnostic concordance of SC suggestions with the final discharge diagnosis as (1) identical, (2) plausible, or (3) diagnostically different. SC suggestions per patient were additionally classified as safe or potentially life-threatening, and the concordance of Ada's and physician-based triage category was assessed. Secondary outcomes were SC usability (5-point Likert-scale: 1=very easy to use to 5=very difficult to use) and SC acceptance net promoter score (NPS). RESULTS A total of 450 patients completed the study between April and November 2021. The most common chief complaint was chest pain (160/437, 37%). The identical diagnosis was ranked first (or within the top 5 diagnoses) by Ada and Symptoma in 14% (59/437; 27%, 117/437) and 4% (16/437; 13%, 55/437) of patients, respectively. An identical or plausible diagnosis was ranked first (or within the top 5 diagnoses) by Ada and Symptoma in 58% (253/437; 75%, 329/437) and 38% (164/437; 64%, 281/437) of patients, respectively. Ada and Symptoma did not suggest potentially life-threatening diagnoses in 13% (56/437) and 14% (61/437) of patients, respectively. Ada correctly triaged, undertriaged, and overtriaged 34% (149/437), 13% (58/437), and 53% (230/437) of patients, respectively. A total of 88% (385/437) and 78% (342/437) of participants rated Ada and Symptoma as very easy or easy to use, respectively. Ada's NPS was -34 (55% [239/437] detractors; 21% [93/437] promoters) and Symptoma's NPS was -47 (63% [275/437] detractors and 16% [70/437]) promoters. CONCLUSIONS Ada demonstrated a higher diagnostic accuracy than Symptoma, and substantially more patients would recommend Ada and assessed Ada as easy to use. The high number of unrecognized potentially life-threatening diagnoses by both SCs and inappropriate triage advice by Ada was alarming. Overall, the trustworthiness of SC recommendations appears questionable. SC authorization should necessitate rigorous clinical evaluation studies to prevent misdiagnoses, fatal triage advice, and misuse of scarce medical resources. TRIAL REGISTRATION German Register of Clinical Trials DRKS00024830; https://drks.de/search/en/trial/DRKS00024830.
Collapse
Affiliation(s)
- Johannes Knitza
- Institute for Digital Medicine, University Hospital Giessen, Philipps University, Marburg, Germany
- Department of Internal Medicine 3, Friedrich-Alexander University Erlangen-Nürnberg, Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
- Université Grenoble Alpes, Grenoble, France
| | - Ragip Hasanaj
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Jonathan Beyer
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Franziska Ganzer
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Anna Slagman
- Emergency and Acute Medicine and Health Services Research in Emergency Medicine, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Myrto Bolanaki
- Emergency and Acute Medicine and Health Services Research in Emergency Medicine, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Hendrik Napierala
- Institute of General Practice and Family Medicine, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Malte L Schmieding
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Berlin, Germany
| | - Nizam Al-Zaher
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
- Department of Medicine 1, Friedrich-Alexander University Hospital Erlangen, University Erlangen-Nuremberg, Erlangen, Germany
| | - Till Orlemann
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
- Department of Medicine 1, Friedrich-Alexander University Hospital Erlangen, University Erlangen-Nuremberg, Erlangen, Germany
| | - Felix Muehlensiepen
- Université Grenoble Alpes, Grenoble, France
- Centre for Health Services Research Brandenburg, Brandenburg Medical School, Rüdersdorf, Germany
| | - Julia Greenfield
- Institute for Digital Medicine, University Hospital Giessen, Philipps University, Marburg, Germany
| | - Nicolas Vuillerme
- Université Grenoble Alpes, Grenoble, France
- Institut Universitaire de France, Paris, France
- Orange Labs & Université Grenoble Alpes, Grenoble, France
| | - Sebastian Kuhn
- Institute for Digital Medicine, University Hospital Giessen, Philipps University, Marburg, Germany
| | - Georg Schett
- Department of Internal Medicine 3, Friedrich-Alexander University Erlangen-Nürnberg, Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany
| | - Stephan Achenbach
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Katharina Dechant
- Department of Cardiology, Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany
| |
Collapse
|
2
|
Todd B, Booher M, Chen NW, Romero K, Berger D. Emergency department use of an electronic differential diagnosis generator in the evaluation of critically ill patients. Intern Emerg Med 2024; 19:797-802. [PMID: 37980319 DOI: 10.1007/s11739-023-03473-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/10/2022] [Accepted: 10/24/2023] [Indexed: 11/20/2023]
Abstract
BACKGROUND Accurate diagnosis is an essential component of managing critically ill emergency department (ED) patients. Electronic diagnosis generators (EDGs) are software tools which assist clinicians in their diagnosis generation; however, they have not been evaluated for use for critical ED patients. We aimed to evaluate the use of an EDG for this population to determine its impact on diagnosis generation and diagnostic testing. METHODS We performed an observational study on usage of an EDG in the high-acuity area of a tertiary care ED. The EDG was used by residents evaluating each patient in the area. The resident used the EDG when the case was felt to have diagnostic uncertainty and completed a data collection tool. Data were summarized by frequencies. Chi-squared or Fisher's exact tests were used to assess the association of added value of the EDG for diagnosis generation and diagnostic testing. RESULTS Over the 8-month study period, the EDG was utilized to evaluate 98 critical ED patients, of whom 60% were female, 7% were pediatric, and 46% were elderly. It was used most commonly for gastroenterological, infectious disease/immunologic, metabolic/renal, and neuropsychiatric presentations, and was least used for trauma presentations. Use of the EDG led to a diagnosis not initially considered in 47% of cases and led to additional diagnostic testing in 4% of cases. CONCLUSION EDGs have some potential to improve diagnosis in critical EM patients by expanding the differential diagnosis and, to a lesser extent, altering diagnostic testing.
Collapse
Affiliation(s)
- Brett Todd
- Department of Emergency Medicine, Oakland University William Beaumont School of Medicine, Royal Oak, MI, USA.
| | - Mathew Booher
- Department of Emergency Medicine, Oakland University William Beaumont School of Medicine, Royal Oak, MI, USA
| | - Nai-Wei Chen
- Division of Informatics and Biostatistics, Beaumont Research Institute, Royal Oak, MI, USA
| | - Kate Romero
- Oakland University William Beaumont School of Medicine, Rochester, MI, USA
| | - David Berger
- Department of Emergency Medicine, Oakland University William Beaumont School of Medicine, Royal Oak, MI, USA
| |
Collapse
|
3
|
Hirosawa T, Harada Y, Mizuta K, Sakamoto T, Tokumasu K, Shimizu T. Diagnostic performance of generative artificial intelligences for a series of complex case reports. Digit Health 2024; 10:20552076241265215. [PMID: 39229463 PMCID: PMC11369864 DOI: 10.1177/20552076241265215] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Accepted: 06/13/2024] [Indexed: 09/05/2024] Open
Abstract
Background Diagnostic performance of generative artificial intelligences (AIs) using large language models (LLMs) across comprehensive medical specialties is still unknown. Objective We aimed to evaluate the diagnostic performance of generative AIs using LLMs in complex case series across comprehensive medical fields. Methods We analyzed published case reports from the American Journal of Case Reports from January 2022 to March 2023. We excluded pediatric cases and those primarily focused on management. We utilized three generative AIs to generate the top 10 differential-diagnosis (DDx) lists from case descriptions: the fourth-generation chat generative pre-trained transformer (ChatGPT-4), Google Gemini (previously Bard), and LLM Meta AI 2 (LLaMA2) chatbot. Two independent physicians assessed the inclusion of the final diagnosis in the lists generated by the AIs. Results Out of 557 consecutive case reports, 392 were included. The inclusion rates of the final diagnosis within top 10 DDx lists were 86.7% (340/392) for ChatGPT-4, 68.6% (269/392) for Google Gemini, and 54.6% (214/392) for LLaMA2 chatbot. The top diagnoses matched the final diagnoses in 54.6% (214/392) for ChatGPT-4, 31.4% (123/392) for Google Gemini, and 23.0% (90/392) for LLaMA2 chatbot. ChatGPT-4 showed higher diagnostic accuracy than Google Gemini (P < 0.001) and LLaMA2 chatbot (P < 0.001). Additionally, Google Gemini outperformed LLaMA2 chatbot within the top 10 DDx lists (P < 0.001) and as the top diagnosis (P = 0.010). Conclusions This study demonstrated the diagnostic performance of generative AIs including ChatGPT-4, Google Gemini, and LLaMA2 chatbot. ChatGPT-4 exhibited higher diagnostic accuracy than the other platforms. These findings suggest the importance of understanding the differences in diagnostic performance among generative AIs, especially in complex case series across comprehensive medical fields, like general medicine.
Collapse
Affiliation(s)
- Takanobu Hirosawa
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| | - Yukinori Harada
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| | - Kazuya Mizuta
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| | - Tetsu Sakamoto
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| | - Kazuki Tokumasu
- Department of General Medicine, Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama University, Okayama, Japan
| | - Taro Shimizu
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| |
Collapse
|
4
|
Kafke SD, Kuhlmey A, Schuster J, Blüher S, Czimmeck C, Zoellick JC, Grosse P. Can clinical decision support systems be an asset in medical education? An experimental approach. BMC MEDICAL EDUCATION 2023; 23:570. [PMID: 37568144 PMCID: PMC10416486 DOI: 10.1186/s12909-023-04568-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 08/04/2023] [Indexed: 08/13/2023]
Abstract
BACKGROUND Diagnostic accuracy is one of the major cornerstones of appropriate and successful medical decision-making. Clinical decision support systems (CDSSs) have recently been used to facilitate physician's diagnostic considerations. However, to date, little is known about the potential assets of CDSS for medical students in an educational setting. The purpose of our study was to explore the usefulness of CDSSs for medical students assessing their diagnostic performances and the influence of such software on students' trust in their own diagnostic abilities. METHODS Based on paper cases students had to diagnose two different patients using a CDSS and conventional methods such as e.g. textbooks, respectively. Both patients had a common disease, in one setting the clinical presentation was a typical one (tonsillitis), in the other setting (pulmonary embolism), however, the patient presented atypically. We used a 2x2x2 between- and within-subjects cluster-randomised controlled trial to assess the diagnostic accuracy in medical students, also by changing the order of the used resources (CDSS first or second). RESULTS Medical students in their 4th and 5th year performed equally well using conventional methods or the CDSS across the two cases (t(164) = 1,30; p = 0.197). Diagnostic accuracy and trust in the correct diagnosis were higher in the typical presentation condition than in the atypical presentation condition (t(85) = 19.97; p < .0001 and t(150) = 7.67; p < .0001).These results refute our main hypothesis that students diagnose more accurately when using conventional methods compared to the CDSS. CONCLUSIONS Medical students in their 4th and 5th year performed equally well in diagnosing two cases of common diseases with typical or atypical clinical presentations using conventional methods or a CDSS. Students were proficient in diagnosing a common disease with a typical presentation but underestimated their own factual knowledge in this scenario. Also, students were aware of their own diagnostic limitations when presented with a challenging case with an atypical presentation for which the use of a CDSS seemingly provided no additional insights.
Collapse
Affiliation(s)
- Sean D Kafke
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
| | - Adelheid Kuhlmey
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Johanna Schuster
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Stefan Blüher
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Constanze Czimmeck
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Jan C Zoellick
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Pascal Grosse
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| |
Collapse
|
5
|
Garber A, Garabedian P, Wu L, Lam A, Malik M, Fraser H, Bersani K, Piniella N, Motta-Calderon D, Rozenblum R, Schnock K, Griffin J, Schnipper JL, Bates DW, Dalal AK. Developing, pilot testing, and refining requirements for 3 EHR-integrated interventions to improve diagnostic safety in acute care: a user-centered approach. JAMIA Open 2023; 6:ooad031. [PMID: 37181729 PMCID: PMC10172040 DOI: 10.1093/jamiaopen/ooad031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Revised: 01/04/2023] [Accepted: 04/20/2023] [Indexed: 05/16/2023] Open
Abstract
Objective To describe a user-centered approach to develop, pilot test, and refine requirements for 3 electronic health record (EHR)-integrated interventions that target key diagnostic process failures in hospitalized patients. Materials and Methods Three interventions were prioritized for development: a Diagnostic Safety Column (DSC) within an EHR-integrated dashboard to identify at-risk patients; a Diagnostic Time-Out (DTO) for clinicians to reassess the working diagnosis; and a Patient Diagnosis Questionnaire (PDQ) to gather patient concerns about the diagnostic process. Initial requirements were refined from analysis of test cases with elevated risk predicted by DSC logic compared to risk perceived by a clinician working group; DTO testing sessions with clinicians; PDQ responses from patients; and focus groups with clinicians and patient advisors using storyboarding to model the integrated interventions. Mixed methods analysis of participant responses was used to identify final requirements and potential implementation barriers. Results Final requirements from analysis of 10 test cases predicted by the DSC, 18 clinician DTO participants, and 39 PDQ responses included the following: DSC configurable parameters (variables, weights) to adjust baseline risk estimates in real-time based on new clinical data collected during hospitalization; more concise DTO wording and flexibility for clinicians to conduct the DTO with or without the patient present; and integration of PDQ responses into the DSC to ensure closed-looped communication with clinicians. Analysis of focus groups confirmed that tight integration of the interventions with the EHR would be necessary to prompt clinicians to reconsider the working diagnosis in cases with elevated diagnostic error (DE) risk or uncertainty. Potential implementation barriers included alert fatigue and distrust of the risk algorithm (DSC); time constraints, redundancies, and concerns about disclosing uncertainty to patients (DTO); and patient disagreement with the care team's diagnosis (PDQ). Discussion A user-centered approach led to evolution of requirements for 3 interventions targeting key diagnostic process failures in hospitalized patients at risk for DE. Conclusions We identify challenges and offer lessons from our user-centered design process.
Collapse
Affiliation(s)
- Alison Garber
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Pamela Garabedian
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Lindsey Wu
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Alyssa Lam
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Maria Malik
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Hannah Fraser
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Kerrin Bersani
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Nicholas Piniella
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Daniel Motta-Calderon
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Ronen Rozenblum
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | - Kumiko Schnock
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | | | - Jeffrey L Schnipper
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | - David W Bates
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
- Department of Health Policy and Management, Harvard T. H. Chan School of Public Health, Boston, Massachusetts, USA
| | - Anuj K Dalal
- Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
6
|
Yanagita Y, Shikino K, Ishizuka K, Uchida S, Li Y, Yokokawa D, Tsukamoto T, Noda K, Uehara T, Ikusaka M. Improving decision accuracy using a clinical decision support system for medical students during history-taking: a randomized clinical trial. BMC MEDICAL EDUCATION 2023; 23:383. [PMID: 37231512 PMCID: PMC10214648 DOI: 10.1186/s12909-023-04370-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/17/2022] [Accepted: 05/17/2023] [Indexed: 05/27/2023]
Abstract
BACKGROUND A clinical diagnostic support system (CDSS) can support medical students and physicians in providing evidence-based care. In this study, we investigate diagnostic accuracy based on the history of present illness between groups of medical students using a CDSS, Google, and neither (control). Further, the degree of diagnostic accuracy of medical students using a CDSS is compared with that of residents using neither a CDSS nor Google. METHODS This study is a randomized educational trial. The participants comprised 64 medical students and 13 residents who rotated in the Department of General Medicine at Chiba University Hospital from May to December 2020. The medical students were randomly divided into the CDSS group (n = 22), Google group (n = 22), and control group (n = 20). Participants were asked to provide the three most likely diagnoses for 20 cases, mainly a history of a present illness (10 common and 10 emergent diseases). Each correct diagnosis was awarded 1 point (maximum 20 points). The mean scores of the three medical student groups were compared using a one-way analysis of variance. Furthermore, the mean scores of the CDSS, Google, and residents' (without CDSS or Google) groups were compared. RESULTS The mean scores of the CDSS (12.0 ± 1.3) and Google (11.9 ± 1.1) groups were significantly higher than those of the control group (9.5 ± 1.7; p = 0.02 and p = 0.03, respectively). The residents' group's mean score (14.7 ± 1.4) was higher than the mean scores of the CDSS and Google groups (p = 0.01). Regarding common disease cases, the mean scores were 7.4 ± 0.7, 7.1 ± 0.7, and 8.2 ± 0.7 for the CDSS, Google, and residents' groups, respectively. There were no significant differences in mean scores (p = 0.1). CONCLUSIONS Medical students who used the CDSS and Google were able to list differential diagnoses more accurately than those using neither. Furthermore, they could make the same level of differential diagnoses as residents in the context of common diseases. TRIAL REGISTRATION This study was retrospectively registered with the University Hospital Medical Information Network Clinical Trials Registry on 24/12/2020 (unique trial number: UMIN000042831).
Collapse
Affiliation(s)
- Yasutaka Yanagita
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan.
| | - Kiyoshi Shikino
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Kosuke Ishizuka
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Shun Uchida
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Yu Li
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Daiki Yokokawa
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Tomoko Tsukamoto
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Kazutaka Noda
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Takanori Uehara
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| | - Masatomi Ikusaka
- Department of General Medicine, Chiba University Hospital, 1-8-1, Inohana, Chuo-Ku, Chiba-City, Chiba Pref, Japan
| |
Collapse
|
7
|
Murad DA, Tsugawa Y, Elashoff DA, Baldwin KM, Bell DS. Distinct components of alert fatigue in physicians' responses to a noninterruptive clinical decision support alert. J Am Med Inform Assoc 2022; 30:64-72. [PMID: 36264258 PMCID: PMC9748542 DOI: 10.1093/jamia/ocac191] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Accepted: 10/10/2022] [Indexed: 12/15/2022] Open
Abstract
OBJECTIVE Clinical decision support (CDS) alerts may improve health care quality but "alert fatigue" can reduce provider responsiveness. We analyzed how the introduction of competing alerts affected provider adherence to a single depression screening alert. MATERIALS AND METHODS We analyzed the audit data from all occurrences of a CDS alert at a large academic health system. For patients who screen positive for depression during ambulatory visits, a noninterruptive alert was presented, offering a number of relevant documentation actions. Alert adherence was defined as the selection of any option offered within the alert. We assessed the effect of competing clinical guidance alerts presented during the same encounter and the total of all CDS alerts that the same provider had seen in the prior 90 days, on the probability of depression screen alert adherence, adjusting for physician and patient characteristics. RESULTS The depression alert fired during 55 649 office visits involving 418 physicians and 40 474 patients over 41 months. After adjustment, physicians who had seen the most alerts in the prior 90 days were much less likely to respond (adjusted OR highest-lowest quartile, 0.38; 95% CI 0.35-0.42; P < .001). Competing alerts in the same visit further reduced the likelihood of adherence only among physicians in the middle two quartiles of alert exposure in the prior 90 days. CONCLUSIONS Adherence to a noninterruptive depression alert was strongly associated with the provider's cumulative alert exposure over the past quarter. Health systems should monitor providers' recent alert exposure as a measure of alert fatigue.
Collapse
Affiliation(s)
- Douglas A Murad
- Medical Informatics, Kaiser Permanente Southern California, San Diego, CA, USA
| | - Yusuke Tsugawa
- Division of General Internal Medicine, Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
| | - David A Elashoff
- Division of General Internal Medicine, Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
| | | | - Douglas S Bell
- Division of General Internal Medicine, Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
- UCLA Health Information Technology, Los Angeles, CA, USA
| |
Collapse
|
8
|
Chavez-Yenter D, Goodman MS, Chen Y, Chu X, Bradshaw RL, Lorenz Chambers R, Chan PA, Daly BM, Flynn M, Gammon A, Hess R, Kessler C, Kohlmann WK, Mann DM, Monahan R, Peel S, Kawamoto K, Del Fiol G, Sigireddi M, Buys SS, Ginsburg O, Kaphingst KA. Association of Disparities in Family History and Family Cancer History in the Electronic Health Record With Sex, Race, Hispanic or Latino Ethnicity, and Language Preference in 2 Large US Health Care Systems. JAMA Netw Open 2022; 5:e2234574. [PMID: 36194411 PMCID: PMC9533178 DOI: 10.1001/jamanetworkopen.2022.34574] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/01/2022] [Accepted: 08/12/2022] [Indexed: 11/14/2022] Open
Abstract
Importance Clinical decision support (CDS) algorithms are increasingly being implemented in health care systems to identify patients for specialty care. However, systematic differences in missingness of electronic health record (EHR) data may lead to disparities in identification by CDS algorithms. Objective To examine the availability and comprehensiveness of cancer family history information (FHI) in patients' EHRs by sex, race, Hispanic or Latino ethnicity, and language preference in 2 large health care systems in 2021. Design, Setting, and Participants This retrospective EHR quality improvement study used EHR data from 2 health care systems: University of Utah Health (UHealth) and NYU Langone Health (NYULH). Participants included patients aged 25 to 60 years who had a primary care appointment in the previous 3 years. Data were collected or abstracted from the EHR from December 10, 2020, to October 31, 2021, and analyzed from June 15 to October 31, 2021. Exposures Prior collection of cancer FHI in primary care settings. Main Outcomes and Measures Availability was defined as having any FHI and any cancer FHI in the EHR and was examined at the patient level. Comprehensiveness was defined as whether a cancer family history observation in the EHR specified the type of cancer diagnosed in a family member, the relationship of the family member to the patient, and the age at onset for the family member and was examined at the observation level. Results Among 144 484 patients in the UHealth system, 53.6% were women; 74.4% were non-Hispanic or non-Latino and 67.6% were White; and 83.0% had an English language preference. Among 377 621 patients in the NYULH system, 55.3% were women; 63.2% were non-Hispanic or non-Latino, and 55.3% were White; and 89.9% had an English language preference. Patients from historically medically undeserved groups-specifically, Black vs White patients (UHealth: 17.3% [95% CI, 16.1%-18.6%] vs 42.8% [95% CI, 42.5%-43.1%]; NYULH: 24.4% [95% CI, 24.0%-24.8%] vs 33.8% [95% CI, 33.6%-34.0%]), Hispanic or Latino vs non-Hispanic or non-Latino patients (UHealth: 27.2% [95% CI, 26.5%-27.8%] vs 40.2% [95% CI, 39.9%-40.5%]; NYULH: 24.4% [95% CI, 24.1%-24.7%] vs 31.6% [95% CI, 31.4%-31.8%]), Spanish-speaking vs English-speaking patients (UHealth: 18.4% [95% CI, 17.2%-19.1%] vs 40.0% [95% CI, 39.7%-40.3%]; NYULH: 15.1% [95% CI, 14.6%-15.6%] vs 31.1% [95% CI, 30.9%-31.2%), and men vs women (UHealth: 30.8% [95% CI, 30.4%-31.2%] vs 43.0% [95% CI, 42.6%-43.3%]; NYULH: 23.1% [95% CI, 22.9%-23.3%] vs 34.9% [95% CI, 34.7%-35.1%])-had significantly lower availability and comprehensiveness of cancer FHI (P < .001). Conclusions and Relevance These findings suggest that systematic differences in the availability and comprehensiveness of FHI in the EHR may introduce informative presence bias as inputs to CDS algorithms. The observed differences may also exacerbate disparities for medically underserved groups. System-, clinician-, and patient-level efforts are needed to improve the collection of FHI.
Collapse
Affiliation(s)
- Daniel Chavez-Yenter
- Huntsman Cancer Institute, University of Utah, Salt Lake City
- Department of Communication, University of Utah, Salt Lake City
| | - Melody S. Goodman
- School of Global Public Health, New York University, New York, New York
| | - Yuyu Chen
- School of Global Public Health, New York University, New York, New York
| | - Xiangying Chu
- School of Global Public Health, New York University, New York, New York
| | - Richard L. Bradshaw
- Department of Biomedical Informatics, University of Utah, Salt Lake City
- School of Medicine, University of Utah Health, Salt Lake City, Utah
| | | | | | - Brianne M. Daly
- Huntsman Cancer Institute, University of Utah, Salt Lake City
| | - Michael Flynn
- School of Medicine, University of Utah Health, Salt Lake City, Utah
| | - Amanda Gammon
- Huntsman Cancer Institute, University of Utah, Salt Lake City
| | - Rachel Hess
- Department of Population Health Sciences, University of Utah, Salt Lake City
- Department of Internal Medicine, University of Utah, Salt Lake City
| | - Cecelia Kessler
- Huntsman Cancer Institute, University of Utah, Salt Lake City
| | | | - Devin M. Mann
- Department of Population Health, New York University Grossman School of Medicine, New York University, New York, New York
| | - Rachel Monahan
- Perlmutter Cancer Center, NYU Langone Health, New York, New York
- Department of Population Health, New York University Grossman School of Medicine, New York University, New York, New York
| | - Sara Peel
- Huntsman Cancer Institute, University of Utah, Salt Lake City
| | - Kensaku Kawamoto
- Department of Biomedical Informatics, University of Utah, Salt Lake City
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah, Salt Lake City
| | | | - Saundra S. Buys
- Huntsman Cancer Institute, University of Utah, Salt Lake City
- Department of Internal Medicine, University of Utah, Salt Lake City
| | - Ophira Ginsburg
- Center for Global Health, National Cancer Institute, Rockville, Maryland
| | - Kimberly A. Kaphingst
- Huntsman Cancer Institute, University of Utah, Salt Lake City
- Department of Communication, University of Utah, Salt Lake City
| |
Collapse
|
9
|
Staal J, Hooftman J, Gunput STG, Mamede S, Frens MA, Van den Broek WW, Alsma J, Zwaan L. Effect on diagnostic accuracy of cognitive reasoning tools for the workplace setting: systematic review and meta-analysis. BMJ Qual Saf 2022; 31:899-910. [DOI: 10.1136/bmjqs-2022-014865] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 08/10/2022] [Indexed: 11/04/2022]
Abstract
BackgroundPreventable diagnostic errors are a large burden on healthcare. Cognitive reasoning tools, that is, tools that aim to improve clinical reasoning, are commonly suggested interventions. However, quantitative estimates of tool effectiveness have been aggregated over both workplace-oriented and educational-oriented tools, leaving the impact of workplace-oriented cognitive reasoning tools alone unclear. This systematic review and meta-analysis aims to estimate the effect of cognitive reasoning tools on improving diagnostic performance among medical professionals and students, and to identify factors associated with larger improvements.MethodsControlled experimental studies that assessed whether cognitive reasoning tools improved the diagnostic accuracy of individual medical students or professionals in a workplace setting were included. Embase.com, Medline ALL via Ovid, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar were searched from inception to 15 October 2021, supplemented with handsearching. Meta-analysis was performed using a random-effects model.ResultsThe literature search resulted in 4546 articles of which 29 studies with data from 2732 participants were included for meta-analysis. The pooled estimate showed considerable heterogeneity (I2=70%). This was reduced to I2=38% by removing three studies that offered training with the tool before the intervention effect was measured. After removing these studies, the pooled estimate indicated that cognitive reasoning tools led to a small improvement in diagnostic accuracy (Hedges’ g=0.20, 95% CI 0.10 to 0.29, p<0.001). There were no significant subgroup differences.ConclusionCognitive reasoning tools resulted in small but clinically important improvements in diagnostic accuracy in medical students and professionals, although no factors could be distinguished that resulted in larger improvements. Cognitive reasoning tools could be routinely implemented to improve diagnosis in practice, but going forward, more large-scale studies and evaluations of these tools in practice are needed to determine how these tools can be effectively implemented.PROSPERO registration numberCRD42020186994.
Collapse
|
10
|
Conducting a representative national randomized control trial of tailored clinical decision support for nurses remotely: Methods and implications. Contemp Clin Trials 2022; 118:106712. [PMID: 35235823 PMCID: PMC9851662 DOI: 10.1016/j.cct.2022.106712] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 01/12/2022] [Accepted: 02/16/2022] [Indexed: 01/22/2023]
Abstract
Clinical Decision Support (CDS) systems, patient specific evidence delivered to clinicians via the electronic health record (EHR) at the right time and in the right format, has the potential to improve patient outcomes. Unfortunately, outcomes of CDS research are mixed. A potential cause lies in its testing. Many CDS are implemented in practice without sufficient testing, potentially leading to patient harm. When testing is conducted, most research has focused on "what" evidence to provide with little attention to the impact of the CDS display format (e.g., textual, graphical) on the user. In an adequately powered randomized control trial with 220 hospital based registered nurses, we will compare 4 randomly assigned CDS format groups (text, text table, text graphs, tailored to subject's graph literacy score) for effects on decision time and simulated patient outcomes. We recruit using state based professional registries, which allows access to participants from multiple institutions across the nation. We use online survey software (REDCap) for efficient study workflow including screening, informed consent documentation, pre-experiment demographic data collection including a graph literacy questionnaire used in randomization. The CDS prototype is accessed via a web app and the simulation-based experiment is conducted remotely at a subject's local computer using video-conferencing software. Also included are 6 post intervention surveys to assess cognitive workload, usability, numeracy, format preference, CDS utilization rationale, and CDS interpretation. Our methods are replicable and scalable for testing of health information technologies and have the potential to improve the safety and effectiveness of these technologies across disciplines.
Collapse
|
11
|
Schmude M, Salim N, Azadzoy H, Bane M, Millen E, O'Donnell L, Bode P, Türk E, Vaidya R, Gilbert S. Investigating the Potential for Clinical Decision Support in Sub-Saharan Africa With AFYA (Artificial Intelligence-Based Assessment of Health Symptoms in Tanzania): Protocol for a Prospective, Observational Pilot Study. JMIR Res Protoc 2022; 11:e34298. [PMID: 35671073 PMCID: PMC9214611 DOI: 10.2196/34298] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2021] [Revised: 02/17/2022] [Accepted: 04/30/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Low- and middle-income countries face difficulties in providing adequate health care. One of the reasons is a shortage of qualified health workers. Diagnostic decision support systems are designed to aid clinicians in their work and have the potential to mitigate pressure on health care systems. OBJECTIVE The Artificial Intelligence-Based Assessment of Health Symptoms in Tanzania (AFYA) study will evaluate the potential of an English-language artificial intelligence-based prototype diagnostic decision support system for mid-level health care practitioners in a low- or middle-income setting. METHODS This is an observational, prospective clinical study conducted in a busy Tanzanian district hospital. In addition to usual care visits, study participants will consult a mid-level health care practitioner, who will use a prototype diagnostic decision support system, and a study physician. The accuracy and comprehensiveness of the differential diagnosis provided by the diagnostic decision support system will be evaluated against a gold-standard differential diagnosis provided by an expert panel. RESULTS Patient recruitment started in October 2021. Participants were recruited directly in the waiting room of the outpatient clinic at the hospital. Data collection will conclude in May 2022. Data analysis is planned to be finished by the end of June 2022. The results will be published in a peer-reviewed journal. CONCLUSIONS Most diagnostic decision support systems have been developed and evaluated in high-income countries, but there is great potential for these systems to improve the delivery of health care in low- and middle-income countries. The findings of this real-patient study will provide insights based on the performance and usability of a prototype diagnostic decision support system in low- or middle-income countries. TRIAL REGISTRATION ClinicalTrials.gov NCT04958577; http://clinicaltrials.gov/ct2/show/NCT04958577. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/34298.
Collapse
Affiliation(s)
| | - Nahya Salim
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | - Mustafa Bane
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | | | | | | | | | - Stephen Gilbert
- Ada Health GmbH, Berlin, Germany.,Else Kröner Fresenius Center for Digital Health, University Hospital Carl Gustav Carus Dresden, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
12
|
Scott IA. Using information technology to reduce diagnostic error: still a bridge too far? Intern Med J 2022; 52:908-911. [PMID: 35718736 DOI: 10.1111/imj.15804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Accepted: 04/28/2022] [Indexed: 11/28/2022]
Affiliation(s)
- Ian A Scott
- Internal Medicine and Clinical Epidemiology, Princess Alexandra Hospital, Brisbane, Queensland, Australia.,School of Clinical Medicine, University of Queensland, Brisbane, Queensland, Australia
| |
Collapse
|
13
|
Yung KK, Ardern CL, Serpiello FR, Robertson S. Characteristics of Complex Systems in Sports Injury Rehabilitation: Examples and Implications for Practice. SPORTS MEDICINE - OPEN 2022; 8:24. [PMID: 35192079 PMCID: PMC8864040 DOI: 10.1186/s40798-021-00405-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/08/2021] [Accepted: 12/29/2021] [Indexed: 11/22/2022]
Abstract
Complex systems are open systems consisting of many components that can interact among themselves and the environment. New forms of behaviours and patterns often emerge as a result. There is a growing recognition that most sporting environments are complex adaptive systems. This acknowledgement extends to sports injury and is reflected in the individual responses of athletes to both injury and rehabilitation protocols. Consequently, practitioners involved in return to sport decision making (RTS) are encouraged to view return to sport decisions through the complex systems lens to improve decision-making in rehabilitation. It is important to clarify the characteristics of this theoretical framework and provide concrete examples to which practitioners can easily relate. This review builds on previous literature by providing an overview of the hallmark features of complex systems and their relevance to RTS research and daily practice. An example of how characteristics of complex systems are exhibited is provided through a case of anterior cruciate ligament injury rehabilitation. Alternative forms of scientific inquiry, such as the use of computational and simulation-based techniques, are also discussed-to move the complex systems approach from the theoretical to the practical level.
Collapse
Affiliation(s)
- Kate K Yung
- Institute for Health and Sport, Victoria University, Melbourne, Australia.
| | - Clare L Ardern
- Musculoskeletal and Sports Injury Epidemiology Centre, Department of Health Promotion Science, Sophiahemmet University, Stockholm, Sweden
- Sport and Exercise Medicine Research Centre, La Trobe University, Melbourne, Australia
- Department of Family Practice, University of British Columbia, Vancouver, Canada
| | - Fabio R Serpiello
- Institute for Health and Sport, Victoria University, Melbourne, Australia
| | - Sam Robertson
- Institute for Health and Sport, Victoria University, Melbourne, Australia
| |
Collapse
|
14
|
Martínez-García M, Hernández-Lemus E. Data Integration Challenges for Machine Learning in Precision Medicine. Front Med (Lausanne) 2022; 8:784455. [PMID: 35145977 PMCID: PMC8821900 DOI: 10.3389/fmed.2021.784455] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2021] [Accepted: 12/28/2021] [Indexed: 12/19/2022] Open
Abstract
A main goal of Precision Medicine is that of incorporating and integrating the vast corpora on different databases about the molecular and environmental origins of disease, into analytic frameworks, allowing the development of individualized, context-dependent diagnostics, and therapeutic approaches. In this regard, artificial intelligence and machine learning approaches can be used to build analytical models of complex disease aimed at prediction of personalized health conditions and outcomes. Such models must handle the wide heterogeneity of individuals in both their genetic predisposition and their social and environmental determinants. Computational approaches to medicine need to be able to efficiently manage, visualize and integrate, large datasets combining structure, and unstructured formats. This needs to be done while constrained by different levels of confidentiality, ideally doing so within a unified analytical architecture. Efficient data integration and management is key to the successful application of computational intelligence approaches to medicine. A number of challenges arise in the design of successful designs to medical data analytics under currently demanding conditions of performance in personalized medicine, while also subject to time, computational power, and bioethical constraints. Here, we will review some of these constraints and discuss possible avenues to overcome current challenges.
Collapse
Affiliation(s)
- Mireya Martínez-García
- Clinical Research Division, National Institute of Cardiology ‘Ignacio Chávez’, Mexico City, Mexico
| | - Enrique Hernández-Lemus
- Computational Genomics Division, National Institute of Genomic Medicine (INMEGEN), Mexico City, Mexico
- Center for Complexity Sciences, Universidad Nacional Autnoma de Mexico, Mexico City, Mexico
| |
Collapse
|
15
|
Graber ML. Reaching 95%: decision support tools are the surest way to improve diagnosis now. BMJ Qual Saf 2021; 31:415-418. [PMID: 34642227 DOI: 10.1136/bmjqs-2021-014033] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/25/2021] [Indexed: 11/04/2022]
Affiliation(s)
- Mark L Graber
- Healthcare Quality and Outcomes, RTI International, St James, NY, USA
| |
Collapse
|
16
|
Knitza J, Tascilar K, Gruber E, Kaletta H, Hagen M, Liphardt AM, Schenker H, Krusche M, Wacker J, Kleyer A, Simon D, Vuillerme N, Schett G, Hueber AJ. Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students. Arthritis Res Ther 2021; 23:233. [PMID: 34488887 PMCID: PMC8420018 DOI: 10.1186/s13075-021-02616-6] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Accepted: 08/23/2021] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND An increasing number of diagnostic decision support systems (DDSS) exist to support patients and physicians in establishing the correct diagnosis as early as possible. However, little evidence exists that supports the effectiveness of these DDSS. The objectives were to compare the diagnostic accuracy of medical students, with and without the use of a DDSS, and the diagnostic accuracy of the DDSS system itself, regarding the typical rheumatic diseases and to analyze the user experience. METHODS A total of 102 medical students were openly recruited from a university hospital and randomized (unblinded) to a control group (CG) and an intervention group (IG) that used a DDSS (Ada - Your Health Guide) to create an ordered diagnostic hypotheses list for three rheumatic case vignettes. Diagnostic accuracy, measured as the presence of the correct diagnosis first or at all on the hypothesis list, was the main outcome measure and evaluated for CG, IG, and DDSS. RESULTS The correct diagnosis was ranked first (or was present at all) in CG, IG, and DDSS in 37% (40%), 47% (55%), and 29% (43%) for the first case; 87% (94%), 84% (100%), and 51% (98%) in the second case; and 35% (59%), 20% (51%), and 4% (51%) in the third case, respectively. No significant benefit of using the DDDS could be observed. In a substantial number of situations, the mean probabilities reported by the DDSS for incorrect diagnoses were actually higher than for correct diagnoses, and students accepted false DDSS diagnostic suggestions. DDSS symptom entry greatly varied and was often incomplete or false. No significant correlation between the number of symptoms extracted and diagnostic accuracy was seen. It took on average 7 min longer to solve a case using the DDSS. In IG, 61% of students compared to 90% in CG stated that they could imagine using the DDSS in their future clinical work life. CONCLUSIONS The diagnostic accuracy of medical students was superior to the DDSS, and its usage did not significantly improve students' diagnostic accuracy. DDSS usage was time-consuming and may be misleading due to prompting wrong diagnoses and probabilities. TRIAL REGISTRATION DRKS.de, DRKS00024433 . Retrospectively registered on February 5, 2021.
Collapse
Affiliation(s)
- Johannes Knitza
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- AGEIS, Université Grenoble Alpes, Grenoble, France
| | - Koray Tascilar
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Eva Gruber
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
| | - Hannah Kaletta
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
| | - Melanie Hagen
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Anna-Maria Liphardt
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Hannah Schenker
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Martin Krusche
- Medical Department, Division of Rheumatology and Clinical Immunology, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Jochen Wacker
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - David Simon
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Nicolas Vuillerme
- AGEIS, Université Grenoble Alpes, Grenoble, France
- Institut Universitaire de France, Paris, France
- LabCom Telecom4Health, University of Grenoble Alpes & Orange Labs, Grenoble, France
| | - Georg Schett
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Axel J Hueber
- Department of Internal Medicine 3 - Rheumatology and Immunology, Friedrich-Alexander University (FAU) Erlangen-Nürnberg and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany.
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.
| |
Collapse
|
17
|
Harada Y, Katsukura S, Kawamura R, Shimizu T. Effects of a Differential Diagnosis List of Artificial Intelligence on Differential Diagnoses by Physicians: An Exploratory Analysis of Data from a Randomized Controlled Study. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18115562. [PMID: 34070958 PMCID: PMC8196999 DOI: 10.3390/ijerph18115562] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 03/31/2021] [Revised: 05/07/2021] [Accepted: 05/21/2021] [Indexed: 11/16/2022]
Abstract
A diagnostic decision support system (DDSS) is expected to reduce diagnostic errors. However, its effect on physicians' diagnostic decisions remains unclear. Our study aimed to assess the prevalence of diagnoses from artificial intelligence (AI) in physicians' differential diagnoses when using AI-driven DDSS that generates a differential diagnosis from the information entered by the patient before the clinical encounter on physicians' differential diagnoses. In this randomized controlled study, an exploratory analysis was performed. Twenty-two physicians were required to generate up to three differential diagnoses per case by reading 16 clinical vignettes. The participants were divided into two groups, an intervention group, and a control group, with and without a differential diagnosis list of AI, respectively. The prevalence of physician diagnosis identical with the differential diagnosis of AI (primary outcome) was significantly higher in the intervention group than in the control group (70.2% vs. 55.1%, p < 0.001). The primary outcome was significantly >10% higher in the intervention group than in the control group, except for attending physicians, and physicians who did not trust AI. This study suggests that at least 15% of physicians' differential diagnoses were affected by the differential diagnosis list in the AI-driven DDSS.
Collapse
Affiliation(s)
- Yukinori Harada
- Department of General Internal Medicine, Nagano Chuo Hospital, Nagano 380-0814, Japan;
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi 321-0293, Japan; (S.K.); (R.K.)
| | - Shinichi Katsukura
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi 321-0293, Japan; (S.K.); (R.K.)
| | - Ren Kawamura
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi 321-0293, Japan; (S.K.); (R.K.)
| | - Taro Shimizu
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi 321-0293, Japan; (S.K.); (R.K.)
- Correspondence: ; Tel.: +81-282-86-1111
| |
Collapse
|
18
|
Standiford TC, Farlow JL, Brenner MJ, Conte ML, Terrell JE. Clinical Decision Support Systems in Otolaryngology-Head and Neck Surgery: A State of the Art Review. Otolaryngol Head Neck Surg 2021; 166:35-47. [PMID: 33874795 DOI: 10.1177/01945998211004529] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
Abstract
OBJECTIVE To offer practical, evidence-informed knowledge on clinical decision support systems (CDSSs) and their utility in improving care and reducing costs in otolaryngology-head and neck surgery. This primer on CDSSs introduces clinicians to both the capabilities and the limitations of this technology, reviews the literature on current state, and seeks to spur further progress in this area. DATA SOURCES PubMed/MEDLINE, Embase, and Web of Science. REVIEW METHODS Scoping review of CDSS literature applicable to otolaryngology clinical practice. Investigators identified articles that incorporated knowledge-based computerized CDSSs to aid clinicians in decision making and workflow. Data extraction included level of evidence, Osheroff classification of CDSS intervention type, otolaryngology subspecialty or domain, and impact on provider performance or patient outcomes. CONCLUSIONS Of 3191 studies retrieved, 11 articles met formal inclusion criteria. CDSS interventions included guideline or protocols support (n = 8), forms and templates (n = 5), data presentation aids (n = 2), and reactive alerts, reference information, or order sets (all n = 1); 4 studies had multiple interventions. CDSS studies demonstrated effectiveness across diverse domains, including antibiotic stewardship, cancer survivorship, guideline adherence, data capture, cost reduction, and workflow. Implementing CDSSs often involved collaboration with health information technologists. IMPLICATIONS FOR PRACTICE While the published literature on CDSSs in otolaryngology is finite, CDSS interventions are proliferating in clinical practice, with roles in preventing medical errors, streamlining workflows, and improving adherence to best practices for head and neck disorders. Clinicians may collaborate with information technologists and health systems scientists to develop, implement, and investigate the impact of CDSSs in otolaryngology.
Collapse
Affiliation(s)
| | - Janice L Farlow
- Department of Otolaryngology-Head & Neck Surgery, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Michael J Brenner
- Department of Otolaryngology-Head & Neck Surgery, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Marisa L Conte
- Department of Research and Informatics, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Jeffrey E Terrell
- Department of Otolaryngology-Head & Neck Surgery, University of Michigan Medical School, Ann Arbor, Michigan, USA
| |
Collapse
|
19
|
Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3:17. [PMID: 32047862 PMCID: PMC7005290 DOI: 10.1038/s41746-020-0221-y] [Citation(s) in RCA: 780] [Impact Index Per Article: 195.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2019] [Accepted: 12/19/2019] [Indexed: 12/16/2022] Open
Abstract
Computerized clinical decision support systems, or CDSS, represent a paradigm shift in healthcare today. CDSS are used to augment clinicians in their complex decision-making processes. Since their first use in the 1980s, CDSS have seen a rapid evolution. They are now commonly administered through electronic medical records and other computerized clinical workflows, which has been facilitated by increasing global adoption of electronic medical records with advanced capabilities. Despite these advances, there remain unknowns regarding the effect CDSS have on the providers who use them, patient outcomes, and costs. There have been numerous published examples in the past decade(s) of CDSS success stories, but notable setbacks have also shown us that CDSS are not without risks. In this paper, we provide a state-of-the-art overview on the use of clinical decision support systems in medicine, including the different types, current use cases with proven efficacy, common pitfalls, and potential harms. We conclude with evidence-based recommendations for minimizing risk in CDSS design, implementation, evaluation, and maintenance.
Collapse
Affiliation(s)
- Reed T. Sutton
- Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada
| | - David Pincock
- Chief Medical Information Office, Alberta Health Services, Edmonton, Canada
| | - Daniel C. Baumgart
- Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada
| | - Daniel C. Sadowski
- Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada
| | - Richard N. Fedorak
- Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada
| | - Karen I. Kroeker
- Department of Medicine, Division of Gastroenterology, University of Alberta, Edmonton, Canada
| |
Collapse
|
20
|
Bennett D, Mazzei MA, Collins B, Bargagli E, Pipavath S, Spina D, Valentini ML, Rinaldi C, Bettini G, Ginori A, Refini RM, Rottoli P, Raghu G. Diagnosis of idiopathic pulmonary fibrosis by virtual means using "IPFdatabase"- a new software. Respir Med 2019; 147:31-36. [PMID: 30704696 DOI: 10.1016/j.rmed.2018.12.011] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/16/2018] [Revised: 12/24/2018] [Accepted: 12/26/2018] [Indexed: 12/27/2022]
Abstract
BACKGROUND The diagnostic algorithm for idiopathic pulmonary fibrosis (IPF) guidelines has some shortcomings. The aim of the present study was to develop a novel software, "IPFdatabase", that could readily apply the diagnostic criteria per IPF guidelines and make a 'virtual' diagnosis of IPF. METHODS Software was developed as a step-by-step compilation of necessary information according to guidelines to enable a diagnosis of IPF. Software accuracy was validated primarily by comparing software diagnoses to those previously made at a Center for Interstitial Lung Diseases. RESULTS Clinical validation on 98 patients (68 male, age 61.0 ± 8.5 years), revealed high software accuracy for IPF diagnosis when compared to historical diagnoses (sensitivity 95.5%, specificity 96.2%; positive predictive value 95.5%, negative predictive value 96.2%). A general radiologist and a general pathologist reviewed relevant data with and without the new software: interobserver agreement increased when they used the IPFdatabase (kappa 0.18 to 0.64 for radiology, 0.13 to 0.59 for pathology). CONCLUSION IPFdatabase is a useful diagnostic tool for typical cases of IPF, and potentially restricts the need for MDDs to atypical and complex cases. We propose this web-designed software for instant accurate diagnosis of IPF by virtual means and for educational purposes; the software is readily accessed with mobile apps, allows incorporation of updated version of guidelines, can be utilized for gathering data useful for future studies and give physicians rapid feedback in daily practice.
Collapse
Affiliation(s)
- David Bennett
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Maria Antonietta Mazzei
- Diagnostic Imaging Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Bridget Collins
- Center for Interstitial Lung Diseases, Division of Pulmonary, Critical Care & Sleep Medicine, University of Washington (UW), Seattle, USA
| | - Elena Bargagli
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Sudhakar Pipavath
- Center for Interstitial Lung Diseases, Division of Pulmonary, Critical Care & Sleep Medicine, University of Washington (UW), Seattle, USA; Department of Radiology, University of Washington (UW), Seattle, USA
| | - Donatella Spina
- Pathology Unit, Azienda Ospedaliera Universitaria Senese (AOUS), Siena, Italy
| | - Maria Lucia Valentini
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Cesare Rinaldi
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Gloria Bettini
- Radiology Unit, Emergency Department, Azienda Ospedaliera Universitaria Pisana (AOUP), Pisa, Italy
| | - Alessandro Ginori
- Pathology Unit, Apuane Hospital, Azienda USL Toscana Nord Ovest, Carrara, Italy
| | - Rosa Metella Refini
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Paola Rottoli
- Respiratory Diseases and Lung Transplantation Unit, Azienda Ospedaliera Universitaria Senese (AOUS) and Department of Medical and Surgical Sciences & Neurosciences, University of Siena, Siena, Italy
| | - Ganesh Raghu
- Center for Interstitial Lung Diseases, Division of Pulmonary, Critical Care & Sleep Medicine, University of Washington (UW), Seattle, USA.
| |
Collapse
|
21
|
Cenek M, Hu M, York G, Dahl S. Survey of Image Processing Techniques for Brain Pathology Diagnosis: Challenges and Opportunities. Front Robot AI 2018; 5:120. [PMID: 33500999 PMCID: PMC7805910 DOI: 10.3389/frobt.2018.00120] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2018] [Accepted: 09/24/2018] [Indexed: 12/30/2022] Open
Abstract
In recent years, a number of new products introduced to the global market combine intelligent robotics, artificial intelligence and smart interfaces to provide powerful tools to support professional decision making. However, while brain disease diagnosis from the brain scan images is supported by imaging robotics, the data analysis to form a medical diagnosis is performed solely by highly trained medical professionals. Recent advances in medical imaging techniques, artificial intelligence, machine learning and computer vision present new opportunities to build intelligent decision support tools to aid the diagnostic process, increase the disease detection accuracy, reduce error, automate the monitoring of patient's recovery, and discover new knowledge about the disease cause and its treatment. This article introduces the topic of medical diagnosis of brain diseases from the MRI based images. We describe existing, multi-modal imaging techniques of the brain's soft tissue and describe in detail how are the resulting images are analyzed by a radiologist to form a diagnosis. Several comparisons between the best results of classifying natural scenes and medical image analysis illustrate the challenges of applying existing image processing techniques to the medical image analysis domain. The survey of medical image processing methods also identified several knowledge gaps, the need for automation of image processing analysis, and the identification of the brain structures in the medical images that differentiate healthy tissue from a pathology. This survey is grounded in the cases of brain tumor analysis and the traumatic brain injury diagnoses, as these two case studies illustrate the vastly different approaches needed to define, extract, and synthesize meaningful information from multiple MRI image sets for a diagnosis. Finally, the article summarizes artificial intelligence frameworks that are built as multi-stage, hybrid, hierarchical information processing work-flows and the benefits of applying these models for medical diagnosis to build intelligent physician's aids with knowledge transparency, expert knowledge embedding, and increased analytical quality.
Collapse
Affiliation(s)
- Martin Cenek
- Department of Computer Science, University of Portland, Portland, OR, United States
| | - Masa Hu
- Department of Computer Science, University of Portland, Portland, OR, United States
| | - Gerald York
- TBI Imaging and Research, Alaska Radiology Associates, Anchorage, AK, United States
| | - Spencer Dahl
- Columbia College, Columbia University, New York, NY, United States
| |
Collapse
|