51
|
Napierala H, Kopka M, Altendorf MB, Bolanaki M, Schmidt K, Piper SK, Heintze C, Möckel M, Balzer F, Slagman A, Schmieding ML. Examining the impact of a symptom assessment application on patient-physician interaction among self-referred walk-in patients in the emergency department (AKUSYM): study protocol for a multi-center, randomized controlled, parallel-group superiority trial. Trials 2022; 23:791. [PMID: 36127742 PMCID: PMC9490986 DOI: 10.1186/s13063-022-06688-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 08/24/2022] [Indexed: 11/10/2022] Open
Abstract
Background Due to the increasing use of online health information, symptom checkers have been developed to provide an individualized assessment of health complaints and provide potential diagnoses and an urgency estimation. It is assumed that they support patient empowerment and have a positive impact on patient-physician interaction and satisfaction with care. Particularly in the emergency department (ED), symptom checkers could be integrated to bridge waiting times in the ED, and patients as well as physicians could take advantage of potential positive effects. Our study therefore aims to assess the impact of symptom assessment application (SAA) usage compared to no SAA usage on the patient-physician interaction in self-referred walk-in patients in the ED population. Methods In this multi-center, 1:1 randomized, controlled, parallel-group superiority trial, 440 self-referred adult walk-in patients with a non-urgent triage category will be recruited in three EDs in Berlin. Eligible participants in the intervention group will use a SAA directly after initial triage. The control group receives standard care without using a SAA. The primary endpoint is patients’ satisfaction with the patient-physician interaction assessed by the Patient Satisfaction Questionnaire. Discussion The results of this trial could influence the implementation of SAA into acute care to improve the satisfaction with the patient-physician interaction. Trial registration German Clinical Trials Registry DRKS00028598. Registered on 25.03.2022
Collapse
Affiliation(s)
- Hendrik Napierala
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of General Practice and Family Medicine, Charitéplatz 1, 10117, Berlin, Germany
| | - Marvin Kopka
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of Medical Informatics, Charitéplatz 1, 10117, Berlin, Germany.,Cognitive Psychology and Ergonomics, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Straße des 17. Juni 135, 10623, Berlin, Germany
| | - Maria B Altendorf
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Emergency and Acute Medicine and Health Services Research in Emergency Medicine (CVK, CCM), Charitéplatz 1, 10117, Berlin, Germany
| | - Myrto Bolanaki
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Emergency and Acute Medicine and Health Services Research in Emergency Medicine (CVK, CCM), Charitéplatz 1, 10117, Berlin, Germany
| | - Konrad Schmidt
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of General Practice and Family Medicine, Charitéplatz 1, 10117, Berlin, Germany.,Jena University Hospital, Institute of General Practice and Family Medicine, Bachstr. 18, 07743, Jena, Germany
| | - Sophie K Piper
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of Medical Informatics, Charitéplatz 1, 10117, Berlin, Germany.,Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of Biometry and Clinical Epidemiology, Charitéplatz 1, 10117, Berlin, Germany.,Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Christoph Heintze
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of General Practice and Family Medicine, Charitéplatz 1, 10117, Berlin, Germany
| | - Martin Möckel
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Emergency and Acute Medicine and Health Services Research in Emergency Medicine (CVK, CCM), Charitéplatz 1, 10117, Berlin, Germany
| | - Felix Balzer
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of Medical Informatics, Charitéplatz 1, 10117, Berlin, Germany
| | - Anna Slagman
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Emergency and Acute Medicine and Health Services Research in Emergency Medicine (CVK, CCM), Charitéplatz 1, 10117, Berlin, Germany
| | - Malte L Schmieding
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Institute of Medical Informatics, Charitéplatz 1, 10117, Berlin, Germany. .,docport Services GmbH, Tußmannstr. 75, 40477, Düsseldorf, Germany.
| |
Collapse
|
52
|
Fraser HSF, Cohan G, Koehler C, Anderson J, Lawrence A, Pateña J, Bacher I, Ranney ML. Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study. JMIR Mhealth Uhealth 2022; 10:e38364. [PMID: 36121688 PMCID: PMC9531004 DOI: 10.2196/38364] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2022] [Revised: 05/31/2022] [Accepted: 06/10/2022] [Indexed: 11/26/2022] Open
Abstract
Background Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions. Objective The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED). Methods We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model. Results A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low. Conclusions This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.
Collapse
Affiliation(s)
- Hamish S F Fraser
- Brown Center for Biomedical Informatics, Warren Alpert Medical School, Brown University, Providence, RI, United States
- School of Public Health, Brown University, Providence, RI, United States
| | - Gregory Cohan
- Warren Alpert Medical School, Brown University, Providence, RI, United States
| | - Christopher Koehler
- Department of Emergency Medicine, Brown University, Providence, RI, United States
| | - Jared Anderson
- Department of Emergency Medicine, Brown University, Providence, RI, United States
| | - Alexis Lawrence
- Harvard Medical Faculty Physicians, Department of Emergency Medicine, St Luke's Hospital, New Bedford, MA, United States
| | - John Pateña
- Brown-Lifespan Center for Digital Health, Providence, RI, United States
| | - Ian Bacher
- Brown Center for Biomedical Informatics, Warren Alpert Medical School, Brown University, Providence, RI, United States
| | - Megan L Ranney
- School of Public Health, Brown University, Providence, RI, United States
- Department of Emergency Medicine, Brown University, Providence, RI, United States
- Brown-Lifespan Center for Digital Health, Providence, RI, United States
| |
Collapse
|
53
|
Patel R, Swanton AR, Gross MS. Online Symptom Checkers are Poor Tools for Diagnosing Men's Health Conditions. Urology 2022; 170:124-131. [PMID: 36115428 DOI: 10.1016/j.urology.2022.08.032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 07/24/2022] [Accepted: 08/02/2022] [Indexed: 10/14/2022]
Abstract
OBJECTIVE To analyze the accuracy of the four most commonly used online symptom checkers (OSCs) in diagnosing erectile dysfunction (ED), scrotal pain (SP), Peyronie's disease (PD), and low testosterone (LT). METHODS AND OUTCOMES One-hundred and sixty artificial vignettes were created by de-identifying recent initial outpatient consults presenting to discuss ED (40), SP (40), PD (40), and LT (40). The vignettes were entered into the 4 most frequently used OSCs (WebMD, MedicineNet, EverydayHealth, and SutterHealth) as determined by web traffic analysis tools. The top 5 conditions listed in the OSC differential diagnosis were recorded and scored. RESULTS WebMD's accuracy for ED, SP, PD, and LT vignettes was 0%, 22.5%, 0%, and 95%, respectively. EverydayHealth was only able to diagnose SP 20% of the time, and failed to diagnose ED, PD, or LT on all occasions. MedicineNet diagnosed ED, PD, SP, and LT in 100%, 98%, 27.5%, and 0% of vignettes, respectively. SutterHealth correctly diagnosed ED, SP, and LT in 100%, 20%, and 80% of patients, respectively. Cumulatively, the OSCs were most accurate in diagnosing ED and least accurate in diagnosing SP when using the Top 1 (37.5% vs. 6.9%) and Top 5 (50% vs. 24.5%) of the suggested conditions. CONCLUSIONS No OSC could accurately diagnose all the conditions tested. The OSCs, on average, were poor at suggesting precise diagnoses for ED, PD, LT, SP. Patients and practitioners should be cautioned regarding the accuracy of OSCs.
Collapse
Affiliation(s)
- Rutul Patel
- New York Institute of Technology College of Osteopathic Medicine, Old Westbury, NY, USA
| | | | | |
Collapse
|
54
|
Gräf M, Knitza J, Leipe J, Krusche M, Welcker M, Kuhn S, Mucke J, Hueber AJ, Hornig J, Klemm P, Kleinert S, Aries P, Vuillerme N, Simon D, Kleyer A, Schett G, Callhoff J. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol Int 2022; 42:2167-2176. [PMID: 36087130 PMCID: PMC9548469 DOI: 10.1007/s00296-022-05202-4] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2022] [Accepted: 08/29/2022] [Indexed: 11/29/2022]
Abstract
Symptom checkers are increasingly used to assess new symptoms and navigate the health care system. The aim of this study was to compare the accuracy of an artificial intelligence (AI)-based symptom checker (Ada) and physicians regarding the presence/absence of an inflammatory rheumatic disease (IRD). In this survey study, German-speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for 20 different real-world patient vignettes, which included only basic health and symptom-related medical history. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final rheumatologists’ diagnosis, reported on the discharge summary report. A total of 132 vignettes were completed by 33 physicians (mean rheumatology working experience 8.8 (SD 7.1) years). Ada’s diagnostic accuracy (IRD) was significantly higher compared to physicians (70 vs 54%, p = 0.002) according to top diagnosis. Ada listed the correct diagnosis more often compared to physicians (54 vs 32%, p < 0.001) as top diagnosis as well as among the top 3 diagnoses (59 vs 42%, p < 0.001). Work experience was not related to suggesting the correct diagnosis or IRD status. Confined to basic health and symptom-related medical history, the diagnostic accuracy of physicians was lower compared to an AI-based symptom checker. These results highlight the potential of using symptom checkers early during the patient journey and importance of access to complete and sufficient patient information to establish a correct diagnosis.
Collapse
Affiliation(s)
- Markus Gräf
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Johannes Knitza
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany. .,Deutsches Zentrum Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany. .,Université Grenoble Alpes, AGEIS, Grenoble, France.
| | - Jan Leipe
- Division of Rheumatology, Department of Medicine V, Medical Faculty Mannheim of the University, University Hospital Mannheim, Heidelberg, Germany
| | - Martin Krusche
- Division of Rheumatology and Systemic Inflammatory Diseases, University Hospital Hamburg-Eppendorf (UKE), Hamburg, Germany
| | - Martin Welcker
- Medizinisches Versorgungszentrum Für Rheumatologie Dr. M. Welcker GmbH, Planegg, Germany
| | - Sebastian Kuhn
- Department of Digital Medicine, Medical Faculty OWL, Bielefeld University, Bielefeld, Germany
| | - Johanna Mucke
- Policlinic and Hiller Research Unit for Rheumatology, Medical Faculty, University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Axel J Hueber
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Division of Rheumatology, Klinikum Nürnberg, Paracelsus Medical University, Nuremberg, Germany
| | | | - Philipp Klemm
- Department of Rheumatology, Immunology, Osteology and Physical Medicine, Justus Liebig University Gießen, Campus Kerckhoff, Bad Nauheim, Germany
| | - Stefan Kleinert
- Praxisgemeinschaft Rheumatologie-Nephrologie, Erlangen, Germany
| | | | - Nicolas Vuillerme
- Université Grenoble Alpes, AGEIS, Grenoble, France.,Institut Universitaire de France, Paris, France.,LabCom Telecom4Health, Orange Labs & Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP-UGA, Grenoble, France
| | - David Simon
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Georg Schett
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Johanna Callhoff
- Epidemiology Unit, German Rheumatism Research Centre, Berlin, Germany.,Institute for Social Medicine, Epidemiology and Health Economics, Charité Universitätsmedizin, Berlin, Germany
| |
Collapse
|
55
|
Wallace W, Chan C, Chidambaram S, Hanna L, Iqbal FM, Acharya A, Normahani P, Ashrafian H, Markar SR, Sounderajah V, Darzi A. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digit Med 2022; 5:118. [PMID: 35977992 PMCID: PMC9385087 DOI: 10.1038/s41746-022-00667-w] [Citation(s) in RCA: 23] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2022] [Accepted: 07/25/2022] [Indexed: 11/09/2022] Open
Abstract
Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19–37.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8–90.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.
Collapse
Affiliation(s)
- William Wallace
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK
| | - Calvin Chan
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK
| | - Swathikan Chidambaram
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK
| | - Lydia Hanna
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK
| | - Fahad Mujtaba Iqbal
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK.,Institute of Global Health Innovation, Imperial College London, South Kensington Campus, London, SW7 2AZ, UK
| | - Amish Acharya
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK.,Institute of Global Health Innovation, Imperial College London, South Kensington Campus, London, SW7 2AZ, UK
| | - Pasha Normahani
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK
| | - Hutan Ashrafian
- Institute of Global Health Innovation, Imperial College London, South Kensington Campus, London, SW7 2AZ, UK
| | - Sheraz R Markar
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK.,Department of Molecular Medicine and Surgery, Karolinska Institutet, Stockholm, Sweden.,Nuffield Department of Surgery, Churchill Hospital, University of Oxford, OX3 7LE, Oxford, UK
| | - Viknesh Sounderajah
- Institute of Global Health Innovation, Imperial College London, South Kensington Campus, London, SW7 2AZ, UK.
| | - Ara Darzi
- Department of Surgery & Cancer, Imperial College London, St. Mary's Hospital, London, W2 1NY, UK.,Institute of Global Health Innovation, Imperial College London, South Kensington Campus, London, SW7 2AZ, UK
| |
Collapse
|
56
|
Liu VDM, Kaila M, Koskela T. User initiated symptom assessment with an electronic symptom checker. Study protocol for mixed-methods validation. (Preprint). JMIR Res Protoc 2022. [PMID: 37467041 PMCID: PMC10398552 DOI: 10.2196/41423] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/30/2023] Open
Abstract
BACKGROUND The national Omaolo digital social welfare and health care service of Finland provides a symptom checker, Omaolo, which is a medical device (based on Duodecim Clinical Decision Support EBMEDS software) with a CE marking (risk class IIa), manufactured by the government-owned DigiFinland Oy. Users of this service can perform their triage by using the questions in the symptom checker. By completing the symptom checker, the user receives a recommendation for action and a service assessment with appropriate guidance regarding their health problems on the basis of a selected specific symptom in the symptom checker. This allows users to be provided with appropriate health care services, regardless of time and place. OBJECTIVE This study describes the protocol for the mixed methods validation process of the symptom checker available in Omaolo digital services. METHODS This is a mixed methods study using quantitative and qualitative methods, which will be part of the clinical validation process that takes place in primary health care centers in Finland. Each organization provides a space where the study and the nurse triage can be done in order to include an unscreened target population of users. The primary health care units provide walk-in model services, where no prior phone call or contact is required. For the validation of the Omaolo symptom checker, case vignettes will be incorporated to supplement the triage accuracy of rare and acute cases that cannot be tested extensively in real-life settings. Vignettes are produced from a variety of clinical sources, and they test the symptom checker in different triage levels by using 1 standardized patient case example. RESULTS This study plan underwent an ethics review by the regional permission, which was requested from each organization participating in the research, and an ethics committee statement was requested and granted from Pirkanmaa hospital district's ethics committee, which is in accordance with the University of Tampere's regulations. Of 964 clinical user-filled symptom checker assessments, 877 cases were fully completed with a triage result, and therefore, they met the requirements for clinical validation studies. The goal for sufficient data has been reached for most of the chief symptoms. Data collection was completed in September 2019, and the first feasibility and patient experience results were published by the end of 2020. Case vignettes have been identified and are to be completed before further testing the symptom checker. The analysis and reporting are estimated to be finalized in 2024. CONCLUSIONS The primary goals of this multimethod electronic symptom checker study are to assess safety and to provide crucial information regarding the accuracy and usability of the Omaolo electronic symptom checker. To our knowledge, this will be the first study to include real-life clinical cases along with case vignettes. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/41423.
Collapse
|
57
|
Liu AW, Odisho AY, Brown Rd W, Gonzales R, Neinstein AB, Judson T. Patient Experience and Feedback after Use of an EHR-integrated COVID-19 Symptom Checker. JMIR Hum Factors 2022; 9:e40064. [PMID: 35960593 PMCID: PMC9472505 DOI: 10.2196/40064] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2022] [Revised: 07/20/2022] [Accepted: 08/06/2022] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Symptom checkers have been widely used during the COVID-19 pandemic to alleviate strain on health systems and offer patients a 24/7 self-service triage option. Although studies suggest that users may positively perceive online symptom checkers, no studies have quantified user feedback after use of an electronic health record (EHR)-integrated COVID-19 symptom checker with self-scheduling functionality. OBJECTIVE We aimed to understand user experience, user satisfaction, and user-reported alternatives to use of a COVID-19 symptom checker with self-triage and self-scheduling functionality. METHODS We launched a patient-portal based self-triage and self-scheduling tool in March 2020 for patients with COVID-19 symptoms, exposures, or questions. We made an optional, anonymous Qualtrics survey available to patients immediately after they completed the symptom checker. RESULTS Between December 16th, 2021 and March 28th, 2022, there were 395 unique responses to the survey. Overall, respondents reported high satisfaction across all demographics, with a median rating of 8 out of 10, and 47.6% of respondents giving a rating of 9 or 10 out of 10. User satisfaction scores were not associated with any demographic factors. The most common user-reported alternatives had the online tool not been available were calling the COVID-19 telephone hotline and sending a patient-portal message to their physician for advice. The ability to schedule a test online was the most important symptom checker feature for respondents. The most common categories of user feedback were regarding other COVID-19 services (e.g. telephone hotline), policies or procedures, or requesting additional features or functionality. CONCLUSIONS This analysis suggests that COVID-19 symptom checkers with self-triage and self-scheduling functionality may have high overall user satisfaction, regardless of user demographics. By allowing users to self-triage and self-schedule tests and visits, tools like this may prevent unnecessary calls and messages to clinicians. Individual feedback suggested that the user experience for this type of tool is highly dependent on the organization's operational workflows for COVID-19 testing and care. The study provides insight for the implementation and improvement of COVID-19 symptom checkers to ensure high user satisfaction. .
Collapse
Affiliation(s)
- Andrew Wayne Liu
- Center for Digital Health Innovation, University of California, San Francisco, 1700 Owens St, Suite 541, San Francisco, US
| | - Anobel Youhana Odisho
- Center for Digital Health Innovation, University of California, San Francisco, 1700 Owens St, Suite 541, San Francisco, US.,Department of Urology, University of California, San Francisco, San Francisco, US
| | - William Brown Rd
- Center for Digital Health Innovation, University of California, San Francisco, 1700 Owens St, Suite 541, San Francisco, US.,Department of Medicine, University of California, San Francisco, 521 Parnassus Avenue, Suite U127, Box 0131, San Francisco, US.,Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, US
| | - Ralph Gonzales
- Department of Medicine, University of California, San Francisco, 521 Parnassus Avenue, Suite U127, Box 0131, San Francisco, US.,Clinical Innovation Center, University of California, San Francisco, San Francisco, US
| | - Aaron B Neinstein
- Center for Digital Health Innovation, University of California, San Francisco, 1700 Owens St, Suite 541, San Francisco, US.,Department of Medicine, University of California, San Francisco, 521 Parnassus Avenue, Suite U127, Box 0131, San Francisco, US
| | - Timothy Judson
- Center for Digital Health Innovation, University of California, San Francisco, 1700 Owens St, Suite 541, San Francisco, US.,Department of Medicine, University of California, San Francisco, 521 Parnassus Avenue, Suite U127, Box 0131, San Francisco, US.,Office of Population Health, University of California, San Francisco, San Francisco, US
| |
Collapse
|
58
|
Jones AM, Jones DR. A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History. Front Artif Intell 2022; 5:727486. [PMID: 35937138 PMCID: PMC9355422 DOI: 10.3389/frai.2022.727486] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2021] [Accepted: 06/21/2022] [Indexed: 11/25/2022] Open
Abstract
Online AI symptom checkers and diagnostic assistants (DAs) have tremendous potential to reduce misdiagnosis and cost, while increasing the quality, convenience, and availability of healthcare, but only if they can perform with high accuracy. We introduce a novel Bayesian DA designed to improve diagnostic accuracy by addressing key weaknesses of Bayesian Network implementations for clinical diagnosis. We compare the performance of our prototype DA (MidasMed) to that of physicians and six other publicly accessible DAs (Ada, Babylon, Buoy, Isabel, Symptomate, and WebMD) using a set of 30 publicly available case vignettes, and using only sparse history (no exam findings or tests). Our results demonstrate superior performance of the MidasMed DA, with the correct diagnosis being the top ranked disorder in 93% of cases, and in the top 3 in 96% of cases.
Collapse
|
59
|
Michel J, Mettler A, Starvaggi C, Travaglini N, Aebi C, Keitel K, Sauter TC. The Utility of a Pediatric COVID-19 Online Forward Triage Tool in Switzerland. Front Public Health 2022; 10:902072. [PMID: 35874988 PMCID: PMC9301458 DOI: 10.3389/fpubh.2022.902072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Accepted: 05/16/2022] [Indexed: 11/22/2022] Open
Abstract
Background To offset the burden on the health system, hospitals set up telehealth interventions, such as online forward triage tools (OFTT). The website www.coronabambini.ch was developed to specifically address the needs of children and their families in Switzerland and to facilitate the decision to test, isolate, attend school, or access the health care system. Methods Video interviews were held with key informants (n = 20) from a population of parents, teachers, guardians, as well as doctors who had used the child-specific COVID-19 OFTT and had consented to a further study. Convenience and quota sampling were done to include a variety of key informants. Interviews were recorded, transcribed verbatim, and analyzed for themes. Results Three main themes emerged: i) the usefulness of the OFTT to the users, ii) expectation management and importance of stakeholder involvement in OFTT development, and iii) OFTT limitations. Conclusion Our study highlights opportunities, limitations, and lessons to consider when developing a pediatric COVID-19 OFTT. The involvement of stakeholders, parents, teachers, and health care providers in the design, set up, implementation, and evaluation of telehealth interventions is critical as this can help with expectation management and enhance OFTT utility.
Collapse
Affiliation(s)
- Janet Michel
- Department of Emergency Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Emergency Telemedicine, University of Bern, Bern, Switzerland
- *Correspondence: Janet Michel
| | - Annette Mettler
- Department of Emergency Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Emergency Telemedicine, University of Bern, Bern, Switzerland
| | - Carl Starvaggi
- Pediatric Emergency Medicine, Department of Pediatrics, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Nicola Travaglini
- Department of Emergency Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Emergency Telemedicine, University of Bern, Bern, Switzerland
| | - Christoph Aebi
- Department of Pediatrics, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Kristina Keitel
- Pediatric Emergency Medicine, Department of Pediatrics, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Thomas C. Sauter
- Department of Emergency Medicine, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Emergency Telemedicine, University of Bern, Bern, Switzerland
| |
Collapse
|
60
|
Nguyen H, Meczner A, Burslam-Dawe K, Hayhoe B. Triage Errors in Primary and Pre-Primary Care. J Med Internet Res 2022; 24:e37209. [PMID: 35749166 PMCID: PMC9270711 DOI: 10.2196/37209] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Revised: 03/21/2022] [Accepted: 04/04/2022] [Indexed: 01/20/2023] Open
Abstract
Triage errors are a major concern in health care due to resulting harmful delays in treatments or inappropriate allocation of resources. With the increasing popularity of digital symptom checkers in pre–primary care settings, and amid claims that artificial intelligence outperforms doctors, the accuracy of triage by digital symptom checkers is ever more scrutinized. This paper examines the context and challenges of triage in primary care, pre–primary care, and emergency care, as well as reviews existing evidence on the prevalence of triage errors in all three settings. Implications for development, research, and practice are highlighted, and recommendations are made on how digital symptom checkers should be best positioned.
Collapse
Affiliation(s)
- Hai Nguyen
- Your.MD Ltd, London, United Kingdom.,Health Services and Population Research, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | | | | | - Benedict Hayhoe
- eConsult Ltd, London, United Kingdom.,Department of Primary Care, School of Public Health, Imperial College London, London, United Kingdom
| |
Collapse
|
61
|
Schmude M, Salim N, Azadzoy H, Bane M, Millen E, O'Donnell L, Bode P, Türk E, Vaidya R, Gilbert S. Investigating the Potential for Clinical Decision Support in Sub-Saharan Africa With AFYA (Artificial Intelligence-Based Assessment of Health Symptoms in Tanzania): Protocol for a Prospective, Observational Pilot Study. JMIR Res Protoc 2022; 11:e34298. [PMID: 35671073 PMCID: PMC9214611 DOI: 10.2196/34298] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2021] [Revised: 02/17/2022] [Accepted: 04/30/2022] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Low- and middle-income countries face difficulties in providing adequate health care. One of the reasons is a shortage of qualified health workers. Diagnostic decision support systems are designed to aid clinicians in their work and have the potential to mitigate pressure on health care systems. OBJECTIVE The Artificial Intelligence-Based Assessment of Health Symptoms in Tanzania (AFYA) study will evaluate the potential of an English-language artificial intelligence-based prototype diagnostic decision support system for mid-level health care practitioners in a low- or middle-income setting. METHODS This is an observational, prospective clinical study conducted in a busy Tanzanian district hospital. In addition to usual care visits, study participants will consult a mid-level health care practitioner, who will use a prototype diagnostic decision support system, and a study physician. The accuracy and comprehensiveness of the differential diagnosis provided by the diagnostic decision support system will be evaluated against a gold-standard differential diagnosis provided by an expert panel. RESULTS Patient recruitment started in October 2021. Participants were recruited directly in the waiting room of the outpatient clinic at the hospital. Data collection will conclude in May 2022. Data analysis is planned to be finished by the end of June 2022. The results will be published in a peer-reviewed journal. CONCLUSIONS Most diagnostic decision support systems have been developed and evaluated in high-income countries, but there is great potential for these systems to improve the delivery of health care in low- and middle-income countries. The findings of this real-patient study will provide insights based on the performance and usability of a prototype diagnostic decision support system in low- or middle-income countries. TRIAL REGISTRATION ClinicalTrials.gov NCT04958577; http://clinicaltrials.gov/ct2/show/NCT04958577. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/34298.
Collapse
Affiliation(s)
| | - Nahya Salim
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | - Mustafa Bane
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | | | | | | | | | - Stephen Gilbert
- Ada Health GmbH, Berlin, Germany.,Else Kröner Fresenius Center for Digital Health, University Hospital Carl Gustav Carus Dresden, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
62
|
Wetzel AJ, Koch R, Preiser C, Müller R, Klemmt M, Ranisch R, Ehni HJ, Wiesing U, Rieger MA, Henking T, Joos S. Ethical, Legal, and Social Implications of Symptom Checker Apps in Primary Health Care (CHECK.APP): Protocol for an Interdisciplinary Mixed Methods Study. JMIR Res Protoc 2022; 11:e34026. [PMID: 35576570 PMCID: PMC9152714 DOI: 10.2196/34026] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 02/18/2022] [Accepted: 03/30/2022] [Indexed: 12/19/2022] Open
Abstract
Background Symptom checker apps (SCAs) are accessible tools that provide early symptom assessment for users. The ethical, legal, and social implications of SCAs and their impact on the patient-physician relationship, the health care providers, and the health care system have sparsely been examined. This study protocol describes an approach to investigate the possible impacts and implications of SCAs on different levels of health care provision. It considers the perspectives of the users, nonusers, general practitioners (GPs), and health care experts. Objective We aim to assess a comprehensive overview of the use of SCAs and address problematic issues, if any. The primary outcomes of this study are empirically informed multi-perspective recommendations for different stakeholders on the ethical, legal, and social implications of SCAs. Methods Quantitative and qualitative methods will be used in several overlapping and interconnected study phases. In study phase 1, a comprehensive literature review will be conducted to assess the ethical, legal, social, and systemic impacts of SCAs. Study phase 2 comprises a survey that will be analyzed with a logistic regression. It aims to assess the user degree of SCAs in Germany as well as the predictors for SCA usage. Study phase 3 will investigate self-observational diaries and user interviews, which will be analyzed as integrated cases to assess user perspectives, usage pattern, and arising problems. Study phase 4 will comprise GP interviews to assess their experiences, perspectives, self-image, and concepts and will be analyzed with the basic procedure by Kruse. Moreover, interviews with health care experts will be conducted in study phase 3 and will be analyzed by using the reflexive thematical analysis approach of Braun and Clark. Results Study phase 1 will be completed in November 2021. We expect the results of study phase 2 in December 2021 and February 2022. In study phase 3, interviews are currently being conducted. The final study endpoint will be in February 2023. Conclusions The possible ethical, legal, social, and systemic impacts of a widespread use of SCAs that affect stakeholders and stakeholder groups on different levels of health care will be identified. The proposed methodological approach provides a multifaceted and diverse empirical basis for a broad discussion on these implications. Trial Registration German Clinical Trials Register (DRKS) DRKS00022465; https://tinyurl.com/yx53er67 International Registered Report Identifier (IRRID) DERR1-10.2196/34026
Collapse
Affiliation(s)
- Anna-Jasmin Wetzel
- Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany
| | - Roland Koch
- Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany
| | - Christine Preiser
- Institute of Occupational and Social Medicine and Health Services Research, University Hospital Tübingen, Tübingen, Germany
| | - Regina Müller
- Institute of Ethics and History of Medicine, University Tübingen, Tübingen, Germany
| | - Malte Klemmt
- Institute of Applied Social Science, University of Applied Science Würzburg-Schweinfurt, Würzburg, Germany
| | - Robert Ranisch
- Faculty of Health Science Brandenburg, University of Potsdam, Potsdam, Germany
| | - Hans-Jörg Ehni
- Institute of Ethics and History of Medicine, University Tübingen, Tübingen, Germany
| | - Urban Wiesing
- Institute of Ethics and History of Medicine, University Tübingen, Tübingen, Germany
| | - Monika A Rieger
- Institute of Occupational and Social Medicine and Health Services Research, University Hospital Tübingen, Tübingen, Germany
| | - Tanja Henking
- Institute of Applied Social Science, University of Applied Science Würzburg-Schweinfurt, Würzburg, Germany
| | - Stefanie Joos
- Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany
| |
Collapse
|
63
|
Schmieding ML, Kopka M, Schmidt K, Schulz-Niethammer S, Balzer F, Feufel MA. Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation. J Med Internet Res 2022; 24:e31810. [PMID: 35536633 PMCID: PMC9131144 DOI: 10.2196/31810] [Citation(s) in RCA: 29] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2021] [Revised: 11/19/2021] [Accepted: 01/30/2022] [Indexed: 12/16/2022] Open
Abstract
Background Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment. Objective This study aims to revisit the landmark index study to investigate whether and how symptom checkers’ capabilities have evolved since 2015 and how they currently compare with laypersons’ stand-alone triage appraisal. Methods In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons’ triage capability. Results We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions. Conclusions Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.
Collapse
Affiliation(s)
- Malte L Schmieding
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Marvin Kopka
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.,Cognitive Psychology and Ergonomics, Department of Psychology and Ergonomics, Technische Universität Berlin, Berlin, Germany
| | - Konrad Schmidt
- Institute of General Practice and Family Medicine, Jena University Hospital, Germany, Jena, Germany.,Institute of General Practice and Family Medicine, Charité - Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Sven Schulz-Niethammer
- Division of Ergonomics, Department of Psychology and Ergonomics, Technische Universität Berlin, Berlin, Germany
| | - Felix Balzer
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Markus A Feufel
- Division of Ergonomics, Department of Psychology and Ergonomics, Technische Universität Berlin, Berlin, Germany
| |
Collapse
|
64
|
Kujala S, Hörhammer I. Health Care Professionals' Experiences of Web-Based Symptom Checkers for Triage: Cross-sectional Survey Study. J Med Internet Res 2022; 24:e33505. [PMID: 35511254 PMCID: PMC9121216 DOI: 10.2196/33505] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2021] [Revised: 10/27/2021] [Accepted: 03/31/2022] [Indexed: 11/13/2022] Open
Abstract
Background Web-based symptom checkers are promising tools that provide help to patients seeking guidance on health problems. Many health organizations have started using them to enhance triage. Patients use the symptom checker to report their symptoms online and submit the report to the health care center through the system. Health care professionals (registered nurse, practical nurse, general physician, physiotherapist, etc) receive patient inquiries with urgency rating, decide on actions to be taken, and communicate these to the patients. The success of the adoption, however, depends on whether the tools can efficiently support health care professionals’ workflow and achieve their support. Objective This study explores the factors influencing health care professionals’ support for a web-based symptom checker for triage. Methods Data were collected through a web-based survey of 639 health care professionals using either of the two most used web-based symptom checkers in the Finnish public primary care. Linear regression models were fitted to study the associations between the study variables and health care professionals’ support for the symptom checkers. In addition, the health care professionals’ comments collected via survey were qualitatively analyzed to elicit additional insights about the benefits and challenges of the clinical use of symptom checkers. Results Results show that the perceived beneficial influence of the symptom checkers on health care professionals’ work and the perceived usability of the tools were positively associated with professionals’ support. The perceived benefits to patients and organizational support for use were positively associated, and threat to professionals’ autonomy was negatively associated with health care professionals’ support. These associations were, however, not independent of other factors included in the models. The influences on professionals’ work were both positive and negative; the tools streamlined work by providing preliminary information on patients and reduced the number of phone calls, but they also created extra work as the professionals needed to call patients and ask clarifying questions. Managing time between the use of symptom checkers and other tasks was also challenging. Meanwhile, according to health care professionals’ experience, the symptom checkers benefited patients as they received help quickly with a lower threshold for care. Conclusions The efficient use of symptom checkers for triage requires usable solutions that support health care professionals’ work. High-quality information about the patients’ conditions and an efficient way of communicating with patients are needed. Using a new eHealth tool also requires that health organizations and teams reorganize their workflows and work distributions to support clinical processes.
Collapse
Affiliation(s)
- Sari Kujala
- Department of Computer Science, Aalto University, Espoo, Finland
| | - Iiris Hörhammer
- Department of Industrial Engineering and Management, Aalto University, Espoo, Finland
| |
Collapse
|
65
|
Knitza J, Muehlensiepen F, Ignatyev Y, Fuchs F, Mohn J, Simon D, Kleyer A, Fagni F, Boeltz S, Morf H, Bergmann C, Labinsky H, Vorbrüggen W, Ramming A, Distler JHW, Bartz-Bazzanella P, Vuillerme N, Schett G, Welcker M, Hueber AJ. Patient's Perception of Digital Symptom Assessment Technologies in Rheumatology: Results From a Multicentre Study. Front Public Health 2022; 10:844669. [PMID: 35273944 PMCID: PMC8902046 DOI: 10.3389/fpubh.2022.844669] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2021] [Accepted: 01/27/2022] [Indexed: 12/14/2022] Open
Abstract
Introduction An increasing number of digital tools, including dedicated diagnostic decision support systems (DDSS) exist to better assess new symptoms and understand when and where to seek medical care. The aim of this study was to evaluate patient's previous online assessment experiences and to compare the acceptability, usability, usefulness and potential impact of artificial intelligence (AI)-based symptom checker (Ada) and an online questionnaire-based self-referral tool (Rheport). Materials and Methods Patients newly presenting to three German secondary rheumatology outpatient clinics were randomly assigned in a 1:1 ratio to complete consecutively Ada or Rheport in a prospective non-blinded multicentre controlled crossover randomized trial. DDSS completion time was recorded by local study personnel and perceptions on DDSS and previous online assessment were collected through a self-completed study questionnaire, including usability measured with the validated System Usability Scale (SUS). Results 600 patients (median age 52 years, 418 women) were included. 277/600 (46.2%) of patients used an online search engine prior to the appointment. The median time patients spent assessing symptoms was 180, 7, and 8 min, respectively using online using search engines, Ada and Rheport. 111/275 (40.4%), 266/600 (44.3%) and 395/600 (65.8%) of patients rated the respective symptom assessment as very helpful or helpful, using online search engines, Ada and Rheport, respectively. Usability of both diagnostic decision support systems (DDSS) was “good” with a significantly higher mean SUS score (SD) of Rheport 77.1/100 (16.0) compared to Ada 74.4/100 (16.8), (p < 0.0001). In male patients, usability of Rheport was rated higher than Ada (p = 0.02) and the usability rating of older (52 years ≥) patients of both DDSS was lower than in younger participants (p = 0.005). Both effects were independent of each other. 440/600 (73.3%) and 475/600 (79.2%) of the patients would recommend Ada and Rheport to friends and other patients, respectively. Conclusion In summary, patients increasingly assess their symptoms independently online, however only a minority used dedicated symptom assessment websites or DDSS. DDSS, such as Ada an Rheport are easy to use, well accepted among patients with musculoskeletal complaints and could replace online search engines for patient symptom assessment, potentially saving time and increasing helpfulness.
Collapse
Affiliation(s)
- Johannes Knitza
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Université Grenoble Alpes, AGEIS, Grenoble, France
| | - Felix Muehlensiepen
- Université Grenoble Alpes, AGEIS, Grenoble, France.,Center for Health Services Research, Faculty of Health Sciences, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany
| | - Yuriy Ignatyev
- Center for Health Services Research, Faculty of Health Sciences, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany
| | - Franziska Fuchs
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Jacob Mohn
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - David Simon
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Filippo Fagni
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Sebastian Boeltz
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Harriet Morf
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Christina Bergmann
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Hannah Labinsky
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Wolfgang Vorbrüggen
- Verein zur Förderung der Rheumatologie e.V., Würselen, Germany.,RheumaDatenRhePort (RHADAR), Planegg, Germany
| | - Andreas Ramming
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Jörg H W Distler
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Peter Bartz-Bazzanella
- RheumaDatenRhePort (RHADAR), Planegg, Germany.,Klinik für Internistische Rheumatologie, Rhein-Maas Klinikum, Würselen, Germany
| | - Nicolas Vuillerme
- Université Grenoble Alpes, AGEIS, Grenoble, France.,Institut Universitaire de France, Paris, France.,LabCom Telecom4Health, Orange Labs & Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP-UGA, Grenoble, France
| | - Georg Schett
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.,Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Martin Welcker
- RheumaDatenRhePort (RHADAR), Planegg, Germany.,MVZ für Rheumatologie Dr. Martin Welcker GmbH, Planegg, Germany
| | - Axel J Hueber
- Section Rheumatology, Sozialstiftung Bamberg, Bamberg, Germany.,Division of Rheumatology, Klinikum Nürnberg, Paracelsus Medical University, Nürnberg, Germany
| |
Collapse
|
66
|
El-Osta A, Webber I, Alaa A, Bagkeris E, Mian S, Taghavi Azar Sharabiani M, Majeed A. What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study. BMJ Open 2022; 12:e053566. [PMID: 35477872 PMCID: PMC9047920 DOI: 10.1136/bmjopen-2021-053566] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/02/2022] Open
Abstract
OBJECTIVE Assess the suitability of clinical vignettes in benchmarking the performance of online symptom checkers (OSCs). DESIGN Observational study using a publicly available free OSC. PARTICIPANTS Healthily OSC, which provided consultations in English, was used to record consultation outcomes from two lay and four expert inputters using 139 standardised patient vignettes. Each vignette included three diagnostic solutions and a triage recommendation in one of three categories of triage urgency. A panel of three independent general practitioners interpreted the vignettes to arrive at an alternative set of diagnostic and triage solutions. Both sets of diagnostic and triage solutions were consolidated to arrive at a final consolidated version for benchmarking. MAIN OUTCOME MEASURES Six inputters simulated 834 standardised patient evaluations using Healthily OSC and recorded outputs (triage solution, signposting, and whether the correct diagnostic solution appeared first or within the first three differentials). We estimated Cohen's kappa to assess how interpretations by different inputters could lead to divergent OSC output even when using the same vignette or when compared with a separate panel of physicians. RESULTS There was moderate agreement on triage recommendation (kappa=0.48), and substantial agreement on consultation outcomes between all inputters (kappa=0.73). OSC performance improved significantly from baseline when compared against the final consolidated diagnostic and triage solution (p<0.001). CONCLUSIONS Clinical vignettes are inherently limited in their utility to benchmark the diagnostic accuracy or triage safety of OSC. Real-world evidence studies involving real patients are recommended to benchmark the performance of OSC against a panel of physicians.
Collapse
Affiliation(s)
- Austen El-Osta
- Department of Primary Care and Public Health, Imperial College London, London, UK
| | - Iman Webber
- Self-Care Academic Research Unit (SCARU), Department of Primary Care and Public Health, Imperial College London Faculty of Medicine, London, UK
| | - Aos Alaa
- Self-Care Academic Research Unit (SCARU), Department of Primary Care and Public Health, Imperial College London Faculty of Medicine, London, UK
| | | | - Saba Mian
- Department of Primary Care and Public Health, Imperial College London, London, UK
| | | | - Azeem Majeed
- Department of Primary Care and Public Health, Imperial College London, London, UK
| |
Collapse
|
67
|
Knevel R, Knitza J, Hensvold A, Circiumaru A, Bruce T, Evans S, Maarseveen T, Maurits M, Beaart-van de Voorde L, Simon D, Kleyer A, Johannesson M, Schett G, Huizinga T, Svanteson S, Lindfors A, Klareskog L, Catrina A. Rheumatic?-A Digital Diagnostic Decision Support Tool for Individuals Suspecting Rheumatic Diseases: A Multicenter Pilot Validation Study. Front Med (Lausanne) 2022; 9:774945. [PMID: 35547229 PMCID: PMC9083190 DOI: 10.3389/fmed.2022.774945] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Accepted: 03/15/2022] [Indexed: 11/13/2022] Open
Abstract
Introduction Digital diagnostic decision support tools promise to accelerate diagnosis and increase health care efficiency in rheumatology. Rheumatic? is an online tool developed by specialists in rheumatology and general medicine together with patients and patient organizations. It calculates a risk score for several rheumatic diseases. We ran a pilot study retrospectively testing Rheumatic? for its ability to differentiate symptoms from existing or emerging immune-mediated rheumatic diseases from other rheumatic and musculoskeletal complaints and disorders in patients visiting rheumatology clinics. Materials and Methods The performance of Rheumatic? was tested using in three university rheumatology centers: (A) patients at Risk for RA (Karolinska Institutet, n = 50 individuals with musculoskeletal complaints and anti-citrullinated protein antibody positivity) (B) patients with early joint swelling [dataset B (Erlangen) n = 52]. (C) Patients with early arthritis where the clinician considered it likely to be of auto-immune origin [dataset C (Leiden) n = 73]. In dataset A we tested whether Rheumatic? could predict the development of arthritis. In dataset B and C we tested whether Rheumatic? could predict the development of an immune-mediated rheumatic diseases. We examined the discriminative power of the total score with the Wilcoxon rank test and the area-under-the-receiver-operating-characteristic curve (AUC-ROC). Next, we calculated the test characteristics for these patients passing the first or second expert-based Rheumatic? scoring threshold. Results The total test scores differentiated between: (A) Individuals developing arthritis or not, median 245 vs. 163, P < 0.0001, AUC-ROC = 75.3; (B) patients with an immune-mediated arthritic disease or not median 191 vs. 107, P < 0.0001, AUC-ROC = 79.0; but less patients with an immune-mediated arthritic disease or not amongst those where the clinician already considered an immune mediated disease most likely (median 262 vs. 212, P < 0.0001, AUC-ROC = 53.6). Threshold-1 (advising to visit primary care doctor) was highly specific in dataset A and B (0.72, 0.87, and 0.23, respectively) and sensitive (0.67, 0.61, and 0.67). Threshold-2 (advising to visit rheumatologic care) was very specific in all three centers but not very sensitive: specificity of 1.0, 0.96, and 0.91, sensitivity 0.05, 0.07, 0.14 in dataset A, B, and C, respectively. Conclusion Rheumatic? is a web-based patient-centered multilingual diagnostic tool capable of differentiating immune-mediated rheumatic conditions from other musculoskeletal problems. The current scoring system needs to be further optimized.
Collapse
Affiliation(s)
- Rachel Knevel
- Leiden University Medical Center, Leiden, Netherlands
- Translational and Clinical Research Institute, Newcastle University, Newcastle upon Tyne, United Kingdom
| | - Johannes Knitza
- Department of Internal Medicine 3, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- Université Grenoble Alpe, Autonomie, Gérontologie, E-santé, Imagerie et Société, Grenoble, France
| | - Aase Hensvold
- Division of Rheumatology, Department of Medicine, Karolinska University Hospital, Karolinska Institutet, Stockholm, Sweden
- Center for Rheumatology, Academic Specialist Center, Stockholm, Sweden
| | - Alexandra Circiumaru
- Division of Rheumatology, Department of Medicine, Karolinska University Hospital, Karolinska Institutet, Stockholm, Sweden
- Center for Rheumatology, Academic Specialist Center, Stockholm, Sweden
| | - Tor Bruce
- Ocean Observations AB, Design Consultancy, Stockholm, Sweden
| | | | | | - Marc Maurits
- Leiden University Medical Center, Leiden, Netherlands
| | - Liesbeth Beaart-van de Voorde
- Leiden University Medical Center, Leiden, Netherlands
- Master Advanced Nursing Practice, University of Applied Sciences Leiden, Leiden, Netherlands
| | - David Simon
- Department of Internal Medicine 3, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Martina Johannesson
- Division of Rheumatology, Department of Medicine, Karolinska University Hospital, Karolinska Institutet, Stockholm, Sweden
| | - Georg Schett
- Department of Internal Medicine 3, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie, Friedrich-Alexander-Universität Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Tom Huizinga
- Leiden University Medical Center, Leiden, Netherlands
| | | | | | - Lars Klareskog
- Division of Rheumatology, Department of Medicine, Karolinska University Hospital, Karolinska Institutet, Stockholm, Sweden
| | - Anca Catrina
- Division of Rheumatology, Department of Medicine, Karolinska University Hospital, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
68
|
Millen E, Salim N, Azadzoy H, Bane MM, O'Donnell L, Schmude M, Bode P, Tuerk E, Vaidya R, Gilbert SH. Study protocol for a pilot prospective, observational study investigating the condition suggestion and urgency advice accuracy of a symptom assessment app in sub-Saharan Africa: the AFYA-'Health' Study. BMJ Open 2022; 12:e055915. [PMID: 35410928 PMCID: PMC9003603 DOI: 10.1136/bmjopen-2021-055915] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/10/2023] Open
Abstract
INTRODUCTION Due to a global shortage of healthcare workers, there is a lack of basic healthcare for 4 billion people worldwide, particularly affecting low-income and middle-income countries. The utilisation of AI-based healthcare tools such as symptom assessment applications (SAAs) has the potential to reduce the burden on healthcare systems. The purpose of the AFYA Study (AI-based Assessment oF health sYmptoms in TAnzania) is to evaluate the accuracy of the condition suggestions and urgency advice provided by a user on a Swahili language Ada SAA. METHODS AND ANALYSIS This study is designed as an observational prospective clinical study. The setting is a waiting room of a Tanzanian district hospital. It will include patients entering the outpatient clinic with various conditions and age groups, including children and adolescents. Patients will be asked to use the SAA before proceeding to usual care. After usual care, they will have a consultation with a study-provided physician. Patients and healthcare practitioners will be blinded to the SAA's results. An expert panel will compare the Ada SAA's condition suggestions and urgency advice to usual care and study provided differential diagnoses and triage. The primary outcome measures are the accuracy and comprehensiveness of the Ada SAA evaluated against the gold standard differential diagnoses. ETHICS AND DISSEMINATION Ethical approval was received by the ethics committee (EC) of Muhimbili University of Health and Allied Sciences with an approval number MUHAS-REC-09-2019-044 and the National Institute for Medical Research, NIMR/HQ/R.8c/Vol. I/922. All amendments to the protocol are reported and adapted on the basis of the requirements of the EC. The results from this study will be submitted to peer-reviewed journals, local and international stakeholders, and will be communicated in editorials/articles by Ada Health. TRIAL REGISTRATION NUMBER NCT04958577.
Collapse
Affiliation(s)
| | - Nahya Salim
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | - Mustafa Miraji Bane
- Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania
| | | | | | | | | | | | - Stephen Henry Gilbert
- Ada Health GmbH, Berlin, Germany
- EKFZ for Digital Health, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
69
|
Lowe C, Browne M, Marsh W, Morrissey D. Usability Testing of a Digital Assessment Routing Tool for Musculoskeletal Disorders: An Iterative Convergent Mixed Methods Study (Preprint). J Med Internet Res 2022; 24:e38352. [PMID: 36040787 PMCID: PMC9472040 DOI: 10.2196/38352] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 07/19/2022] [Accepted: 07/29/2022] [Indexed: 11/23/2022] Open
Abstract
Background Musculoskeletal disorders negatively affect millions of patients worldwide, placing significant demand on health care systems. Digital technologies that improve clinical outcomes and efficiency across the care pathway are development priorities. We developed the musculoskeletal Digital Assessment Routing Tool (DART) to enable self-assessment and immediate direction to the right care. Objective We aimed to assess and resolve all serious DART usability issues to create a positive user experience and enhance system adoption before conducting randomized controlled trials for the integration of DART into musculoskeletal management pathways. Methods An iterative, convergent mixed methods design was used, with 22 adult participants assessing 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling, with quotas for age, habitual internet use, and English-language ability. Quantitative data collection was defined by the constructs within the International Organization for Standardization 9241-210-2019 standard, with user satisfaction measured by the System Usability Scale. Study end points were resolution of all grade 1 and 2 usability problems and a mean System Usability Scale score of ≥80 across a minimum of 3 user group sessions. Results All participants (mean age 48.6, SD 15.2; range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 (SD 4.44, range 1-18) minutes. Usability problems were reduced from 12 to 0, with trust and intention to act improving during the study. The relationship between eHealth literacy and age, as explored with a scatter plot and calculation of the Pearson correlation coefficient, was performed for all participants (r=–0.2; 20/22, 91%) and repeated with a potential outlier removed (r=–0.23), with no meaningful relationships observed or found for either. The mean satisfaction for daily internet users was highest (19/22, 86%; mean 86.5, SD 4.48; 90% confidence level [CL] 1.78 or –1.78), with nonnative English speakers (6/22, 27%; mean 78.1, SD 4.60; 90% CL 3.79 or –3.79) and infrequent internet users scoring the lowest (3/22, 14%; mean 70.8, SD 5.44; 90% CL 9.17 or –9.17), although the CIs overlap. The mean score across all groups was 84.3 (SD 4.67), corresponding to an excellent system, with qualitative data from all participants confirming that DART was simple to use. Conclusions All serious DART usability issues were resolved, and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials that assess safety and effectiveness against usual care comparators. The iterative, convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mobile health systems. International Registered Report Identifier (IRRID) RR2-10.2196/27205
Collapse
Affiliation(s)
- Cabella Lowe
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
| | - Mitchell Browne
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
| | - William Marsh
- Risk and Information Systems Research Group, School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom
| | - Dylan Morrissey
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
- Department of Physiotherapy, Barts Health NHS Trust, London, United Kingdom
| |
Collapse
|
70
|
Cotte F, Mueller T, Gilbert S, Blümke B, Multmeier J, Hirsch MC, Wicks P, Wolanski J, Tutschkow D, Schade Brittinger C, Timmermann L, Jerrentrup A. Safety of Triage Self-assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study. JMIR Mhealth Uhealth 2022; 10:e32340. [PMID: 35343909 PMCID: PMC9002590 DOI: 10.2196/32340] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 12/17/2021] [Accepted: 02/18/2022] [Indexed: 01/29/2023] Open
Abstract
Background Increasing use of emergency departments (EDs) by patients with low urgency, combined with limited availability of medical staff, results in extended waiting times and delayed care. Technological approaches could possibly increase efficiency by providing urgency advice and symptom assessments. Objective The purpose of this study is to evaluate the safety of urgency advice provided by a symptom assessment app, Ada, in an ED. Methods The study was conducted at the interdisciplinary ED of Marburg University Hospital, with data collection performed between August 2019 and March 2020. This study had a single-center cross-sectional prospective observational design and included 378 patients. The app’s urgency recommendation was compared with an established triage concept (Manchester Triage System [MTS]), including patients from the lower 3 MTS categories only. For all patients who were undertriaged, an expert physician panel assessed the case to detect potential avoidable hazardous situations (AHSs). Results Of 378 participants, 344 (91%) were triaged the same or more conservatively and 34 (8.9%) were undertriaged by the app. Of the 378 patients, 14 (3.7%) had received safe advice determined by the expert panel and 20 (5.3%) were considered to be potential AHS. Therefore, the assessment could be considered safe in 94.7% (358/378) of the patients when compared with the MTS assessment. From the 3 lowest MTS categories, 43.4% (164/378) of patients were not considered as emergency cases by the app, but could have been safely treated by a general practitioner or would not have required a physician consultation at all. Conclusions The app provided urgency advice after patient self-triage that has a high rate of safety, a rate of undertriage, and a rate of triage with potential to be an AHS, equivalent to telephone triage by health care professionals while still being more conservative than direct ED triage. A large proportion of patients in the ED were not considered as emergency cases, which could possibly relieve ED burden if used at home. Further research should be conducted in the at-home setting to evaluate this hypothesis. Trial Registration German Clinical Trial Registration DRKS00024909; https://www.drks.de/drks_web/navigate.do? navigationId=trial.HTML&TRIAL_ID=DRKS00024909
Collapse
Affiliation(s)
- Fabienne Cotte
- Charité Universitäsmedizin Berlin, Berlin, Germany.,Department of Emergency Medicine, University Clinic Marburg, Philipps-University, Marburg, Germany.,Ada Health GmbH, Berlin, Germany
| | - Tobias Mueller
- Center for Unknown and Rare Diseases, UKGM GmbH, University Clinic Marburg, Philipps-University, Marburg, Germany
| | - Stephen Gilbert
- Ada Health GmbH, Berlin, Germany.,Else Kröner Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
| | | | | | - Martin Christian Hirsch
- Ada Health GmbH, Berlin, Germany.,Institute of Artificial Intelligence, Philipps-University Marburg, Marburg, Germany
| | | | | | - Darja Tutschkow
- Coordinating Center for Clinical Trials, Philipps University Marburg, Marburg, Germany, Marburg, Germany
| | - Carmen Schade Brittinger
- Coordinating Center for Clinical Trials, Philipps University Marburg, Marburg, Germany, Marburg, Germany
| | - Lars Timmermann
- Department of Neurology, University Hospital of Marburg, Marburg, Germany
| | - Andreas Jerrentrup
- Department of Emergency Medicine, University Clinic Marburg, Philipps-University, Marburg, Germany
| |
Collapse
|
71
|
Ramkumar PN, Luu BC, Haeberle HS, Karnuta JM, Nwachukwu BU, Williams RJ. Sports Medicine and Artificial Intelligence: A Primer. Am J Sports Med 2022; 50:1166-1174. [PMID: 33900125 DOI: 10.1177/03635465211008648] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
Abstract
Artificial intelligence (AI) represents the fourth industrial revolution and the next frontier in medicine poised to transform the field of orthopaedics and sports medicine, though widespread understanding of the fundamental principles and adoption of applications remain nascent. Recent research efforts into implementation of AI in the field of orthopaedic surgery and sports medicine have demonstrated great promise in predicting athlete injury risk, interpreting advanced imaging, evaluating patient-reported outcomes, reporting value-based metrics, and augmenting the patient experience. Not unlike the recent emphasis thrust upon physicians to understand the business of medicine, the future practice of sports medicine specialists will require a fundamental working knowledge of the strengths, limitations, and applications of AI-based tools. With appreciation, caution, and experience applying AI to sports medicine, the potential to automate tasks and improve data-driven insights may be realized to fundamentally improve patient care. In this Current Concepts review, we discuss the definitions, strengths, limitations, and applications of AI from the current literature as it relates to orthopaedic sports medicine.
Collapse
Affiliation(s)
- Prem N Ramkumar
- Orthopaedic Machine Learning Laboratory, Cleveland Clinic, Cleveland, Ohio, USA
- Department of Orthopaedic Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA
| | - Bryan C Luu
- Orthopaedic Machine Learning Laboratory, Cleveland Clinic, Cleveland, Ohio, USA
- Department of Orthopaedic Surgery, Baylor College of Medicine, Houston, Texas, USA
| | - Heather S Haeberle
- Orthopaedic Machine Learning Laboratory, Cleveland Clinic, Cleveland, Ohio, USA
- Department of Orthopedic Surgery, Hospital for Special Surgery, New York, New York, USA
| | - Jaret M Karnuta
- Orthopaedic Machine Learning Laboratory, Cleveland Clinic, Cleveland, Ohio, USA
| | - Benedict U Nwachukwu
- Department of Orthopedic Surgery, Hospital for Special Surgery, New York, New York, USA
| | - Riley J Williams
- Department of Orthopedic Surgery, Hospital for Special Surgery, New York, New York, USA
| |
Collapse
|
72
|
Hennemann S, Kuhn S, Witthöft M, Jungmann SM. Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients. JMIR Ment Health 2022; 9:e32832. [PMID: 35099395 PMCID: PMC8844983 DOI: 10.2196/32832] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/24/2021] [Accepted: 11/09/2021] [Indexed: 12/31/2022] Open
Abstract
BACKGROUND Digital technologies have become a common starting point for health-related information-seeking. Web- or app-based symptom checkers aim to provide rapid and accurate condition suggestions and triage advice but have not yet been investigated for mental disorders in routine health care settings. OBJECTIVE This study aims to test the diagnostic performance of a widely available symptom checker in the context of formal diagnosis of mental disorders when compared with therapists' diagnoses based on structured clinical interviews. METHODS Adult patients from an outpatient psychotherapy clinic used the app-based symptom checker Ada-check your health (ADA; Ada Health GmbH) at intake. Accuracy was assessed as the agreement of the first and 1 of the first 5 condition suggestions of ADA with at least one of the interview-based therapist diagnoses. In addition, sensitivity, specificity, and interrater reliabilities (Gwet first-order agreement coefficient [AC1]) were calculated for the 3 most prevalent disorder categories. Self-reported usability (assessed using the System Usability Scale) and acceptance of ADA (assessed using an adapted feedback questionnaire) were evaluated. RESULTS A total of 49 patients (30/49, 61% women; mean age 33.41, SD 12.79 years) were included in this study. Across all patients, the interview-based diagnoses matched ADA's first condition suggestion in 51% (25/49; 95% CI 37.5-64.4) of cases and 1 of the first 5 condition suggestions in 69% (34/49; 95% CI 55.4-80.6) of cases. Within the main disorder categories, the accuracy of ADA's first condition suggestion was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders. Interrater reliabilities ranged from low (AC1=0.15 for anxiety disorders) to good (AC1=0.76 for somatoform and associated disorders). The usability of ADA was rated as high in the System Usability Scale (mean 81.51, SD 11.82, score range 0-100). Approximately 71% (35/49) of participants would have preferred a face-to-face over an app-based diagnostic. CONCLUSIONS Overall, our findings suggest that a widely available symptom checker used in the formal diagnosis of mental disorders could provide clinicians with a list of condition suggestions with moderate-to-good accuracy. However, diagnostic performance was heterogeneous between disorder categories and included low interrater reliability. Although symptom checkers have some potential to complement the diagnostic process as a screening tool, the diagnostic performance should be tested in larger samples and in comparison with further diagnostic instruments.
Collapse
Affiliation(s)
- Severin Hennemann
- Department of Clinical Psychology, Psychotherapy and Experimental Psychopathology, University of Mainz, Mainz, Germany
| | - Sebastian Kuhn
- Department of Digital Medicine, Medical Faculty OWL, Bielefeld University, Bielefeld, Germany
| | - Michael Witthöft
- Department of Clinical Psychology, Psychotherapy and Experimental Psychopathology, University of Mainz, Mainz, Germany
| | - Stefanie M Jungmann
- Department of Clinical Psychology, Psychotherapy and Experimental Psychopathology, University of Mainz, Mainz, Germany
| |
Collapse
|
73
|
Arellano Carmona K, Chittamuru D, Kravitz RL, Ramondt S, Ramírez AS. Beyond Dr. Google: Health information seeking from an intelligent online symptom checker: Cross-Sectional Questionnaire Study (Preprint). J Med Internet Res 2022; 24:e36322. [PMID: 35984690 PMCID: PMC9440406 DOI: 10.2196/36322] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2022] [Revised: 04/21/2022] [Accepted: 05/24/2022] [Indexed: 01/20/2023] Open
Abstract
Background The ever-growing amount of health information available on the web is increasing the demand for tools providing personalized and actionable health information. Such tools include symptom checkers that provide users with a potential diagnosis after responding to a set of probes about their symptoms. Although the potential for their utility is great, little is known about such tools’ actual use and effects. Objective We aimed to understand who uses a web-based artificial intelligence–powered symptom checker and its purposes, how they evaluate the experience of the web-based interview and quality of the information, what they intend to do with the recommendation, and predictors of future use. Methods Cross-sectional survey of web-based health information seekers following the completion of a symptom checker visit (N=2437). Measures of comprehensibility, confidence, usefulness, health-related anxiety, empowerment, and intention to use in the future were assessed. ANOVAs and the Wilcoxon rank sum test examined mean outcome differences in racial, ethnic, and sex groups. The relationship between perceptions of the symptom checker and intention to follow recommended actions was assessed using multilevel logistic regression. Results Buoy users were well-educated (1384/1704, 81.22% college or higher), primarily White (1227/1693, 72.47%), and female (2069/2437, 84.89%). Most had insurance (1449/1630, 88.89%), a regular health care provider (1307/1709, 76.48%), and reported good health (1000/1703, 58.72%). Three types of symptoms—pain (855/2437, 35.08%), gynecological issues (293/2437, 12.02%), and masses or lumps (204/2437, 8.37%)—accounted for almost half (1352/2437, 55.48%) of site visits. Buoy’s top three primary recommendations split across less-serious triage categories: primary care physician in 2 weeks (754/2141, 35.22%), self-treatment (452/2141, 21.11%), and primary care in 1 to 2 days (373/2141, 17.42%). Common diagnoses were musculoskeletal (303/2437, 12.43%), gynecological (304/2437, 12.47%) and skin conditions (297/2437, 12.19%), and infectious diseases (300/2437, 12.31%). Users generally reported high confidence in Buoy, found it useful and easy to understand, and said that Buoy made them feel less anxious and more empowered to seek medical help. Users for whom Buoy recommended “Waiting/Watching” or “Self-Treatment” had strongest intentions to comply, whereas those advised to seek primary care had weaker intentions. Compared with White users, Latino and Black users had significantly more confidence in Buoy (P<.05), and the former also found it significantly more useful (P<.05). Latino (odds ratio 1.96, 95% CI 1.22-3.25) and Black (odds ratio 2.37, 95% CI 1.57-3.66) users also had stronger intentions to discuss recommendations with a provider than White users. Conclusions Results demonstrate the potential utility of a web-based health information tool to empower people to seek care and reduce health-related anxiety. However, despite encouraging results suggesting the tool may fulfill unmet health information needs among women and Black and Latino adults, analyses of the user base illustrate persistent second-level digital divide effects.
Collapse
Affiliation(s)
| | - Deepti Chittamuru
- School of Social Sciences, Humanities and Arts, University of California, Merced, CA, United States
| | | | - Steven Ramondt
- Department of Donor Medicine Research, Sanquin Research, Amsterdam, Netherlands
- Department of Communication Science, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
| | - A Susana Ramírez
- School of Social Sciences, Humanities and Arts, University of California, Merced, CA, United States
| |
Collapse
|
74
|
Nesterov A, Ibragimov B, Umerenkov D, Shelmanov A, Zubkova G, Kokh V. NeuralSympCheck: A Symptom Checking and Disease Diagnostic Neural Model with Logic Regularization. Artif Intell Med 2022. [DOI: 10.1007/978-3-031-09342-5_8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
75
|
Lidströmer N, Davids J, Sood HS, Ashrafian H. AIM in Primary Healthcare. Artif Intell Med 2022. [DOI: 10.1007/978-3-030-64573-1_340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
76
|
Purich K, Zhou Y, Dodd S, Yuan Y, White J. How can we better identify patients with rectal bleeding who are at high risk of colorectal cancer? An observational study. Int J Colorectal Dis 2022; 37:573-582. [PMID: 34786597 PMCID: PMC8594862 DOI: 10.1007/s00384-021-04063-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 11/03/2021] [Indexed: 02/04/2023]
Abstract
PURPOSE Rectal bleeding is a common symptom of colorectal cancer. In this paper, we describe and evaluate the operation of a central access and triage system for patients with rectal bleeding, which uses a "high-risk"/ "low-risk" designation based on the referring doctor's subjective designation and a 10-item symptom checklist. METHODS A total of 1846 patients, referred between February 1, 2016, and December 31, 2018, were included. Exclusion criteria were the following: incorrect patient identification number, duplicate records, and pre-diagnosed gastrointestinal cancer. Data was obtained by chart review. Sensitivity, specificity, and positive and negative predictive values were calculated for each item on the symptom checklist. RESULTS Eight hundred seventy-nine (48%) patients received endoscopy, and 37 (2%) were found to have cancer. Five hundred eighty-two (32%) patients were deemed high-risk. Twenty-nine (78%) of the patients with cancer were in the high-risk group. Patients in the high-risk group had a higher incidence of cancer (5.0% vs 0.6%, p < 0.001) and shorter waits to endoscopy (201 vs 292 days). Patients designated as high-risk by the referring physician had a relative risk of 22.3 compared to those designated as low-risk. Patients deemed high-risk by the symptom checklist had a relative risk of 3.5 compared to low-risk patients. CONCLUSION Our system stratified 29/37 (78%) of the patients found to have cancer as high-risk. A total of 8/37 (22%) patients with cancer were deemed low-risk. Our research has identified two variables (weight loss and anemia) which have been added to our referral symptom checklist. This study helped us identify areas for refinement of our triage system. These findings are of interest to physicians who treat colorectal cancer.
Collapse
Affiliation(s)
- Kieran Purich
- Department of Surgery, University of Alberta Faculty of Medicine and Dentistry, c/o Dvorkin Lounge Mailroom, 2G2 Walter C. Mackenzie Health Sciences Centre, 8440 - 112 ST NW, Edmonton, AB Canada ,Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB Canada
| | - Yiling Zhou
- School of Public Health, University of Alberta, Edmonton, AB Canada
| | - Shawn Dodd
- Department of Surgery, University of Alberta Faculty of Medicine and Dentistry, c/o Dvorkin Lounge Mailroom, 2G2 Walter C. Mackenzie Health Sciences Centre, 8440 - 112 ST NW, Edmonton, AB Canada ,Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB Canada
| | - Yan Yuan
- School of Public Health, University of Alberta, Edmonton, AB Canada
| | - Jonathan White
- Department of Surgery, University of Alberta Faculty of Medicine and Dentistry, c/o Dvorkin Lounge Mailroom, 2G2 Walter C. Mackenzie Health Sciences Centre, 8440 - 112 ST NW, Edmonton, AB Canada ,Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB Canada
| |
Collapse
|
77
|
Ishimaru T, Tsuno K, Hori A, Okawara M, Yasuda Y, Fujino Y, Tabuchi T. Disrupted care during the COVID-19 state of emergency and productivity loss attributed to presenteeism in workers: a nationwide cross-sectional study. BMJ Open 2021; 11:e050068. [PMID: 34907047 PMCID: PMC8671843 DOI: 10.1136/bmjopen-2021-050068] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/03/2022] Open
Abstract
OBJECTIVES The COVID-19 pandemic has caused medical care delays and avoidance around the globe. However, little is known about the relationship between disrupted care and productivity loss attributed to presenteeism during the COVID-19 pandemic. We aimed to investigate whether disrupted care during the COVID-19 state of emergency was associated with health status and productivity loss. METHODS We used data from a nationwide, cross-sectional, Internet-based, self-administered survey. We performed multiple logistic regression analysis on data from 14 545 participants to investigate the associations among variables related to disrupted care, health status and the Work Functioning Impairment Scale, with a cut-off of 21 points. RESULTS Participants who experienced exacerbation of underlying disease (adjusted OR (aOR) 2.84; 95% CI 2.28 to 3.53) or any type of disrupted care were more likely to show low productivity at work. Experiencing disruptions in routine and non-routine clinical settings (aOR 4.64; 95% CI 3.64 to 5.92 and aOR 6.29; 95% CI 4.74 to 8.34, respectively), and running out of drugs (aOR 6.13; 95% CI 4.60 to 8.18) were strongly associated with exacerbation of underlying disease. CONCLUSIONS Workers who experienced disrupted care were much more likely to show productivity loss. Exacerbation of underlying disease is one possible pathway through which disrupted care could affect productivity loss attributed to presenteeism. Our study provides evidence of the importance of early diagnosis and continuous treatment of non-COVID-19 patients to enable them to remain healthy and continue to work during the pandemic.
Collapse
Affiliation(s)
- Tomohiro Ishimaru
- Department of Environmental Epidemiology, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan, Kitakyushu, Japan
| | - Kanami Tsuno
- School of Health Innovation, Kanagawa University of Human Services, Kawasaki, Japan
| | - Ai Hori
- Department of Global Public Health, University of Tsukuba, Tsukuba, Japan
| | - Makoto Okawara
- Department of Environmental Epidemiology, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan, Kitakyushu, Japan
| | - Yoshino Yasuda
- Department of Environmental Epidemiology, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan, Kitakyushu, Japan
| | - Yoshihisa Fujino
- Department of Environmental Epidemiology, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan, Kitakyushu, Japan
| | - Takahiro Tabuchi
- Cancer Control Center, Osaka International Cancer Institute, Osaka, Japan
| |
Collapse
|
78
|
Lowe C, Hanuman Sing H, Marsh W, Morrissey D. Validation of a Musculoskeletal Digital Assessment Routing Tool: Protocol for a Pilot Randomized Crossover Noninferiority Trial. JMIR Res Protoc 2021; 10:e31541. [PMID: 34898461 PMCID: PMC8713101 DOI: 10.2196/31541] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2021] [Revised: 10/29/2021] [Accepted: 11/04/2021] [Indexed: 01/10/2023] Open
Abstract
Background Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on millions of patients and an increasing demand for health care use. Digital technologies to improve health care outcomes and efficiency are considered a priority; however, innovations are rarely tested with sufficient rigor in clinical trials, which is the gold standard for clinical proof of safety and efficacy. We have developed a new musculoskeletal digital assessment routing tool (DART) that allows users to self-assess and be directed to the right care. DART requires validation in a real-world setting before implementation. Objective This pilot study aims to assess the feasibility of a future trial by exploring the key aspects of trial methodology, assessing the procedures, and collecting exploratory data to inform the design of a definitive randomized crossover noninferiority trial to assess DART safety and effectiveness. Methods We will collect data from 76 adults with a musculoskeletal condition presenting to general practitioners within a National Health Service (NHS) in England. Participants will complete both a DART assessment and a physiotherapist-led triage, with the order determined by randomization. The primary analysis will involve an absolute agreement intraclass correlation (A,1) estimate with 95% CI between DART and the clinician for assessment outcomes signposting to condition management pathways. Data will be collected to allow the analysis of participant recruitment and retention, randomization, allocation concealment, blinding, data collection process, and bias. In addition, the impact of trial burden and potential barriers to intervention delivery will be considered. The DART user satisfaction will be measured using the system usability scale. Results A UK NHS ethics submission was done during June 2021 and is pending approval; recruitment will commence in early 2022, with data collection anticipated to last for 3 months. The results will be reported in a follow-up paper in 2022. Conclusions This study will inform the design of a randomized controlled crossover noninferiority study that will provide evidence concerning mobile health DART system clinical signposting in an NHS setting before real-world implementation. Success should produce evidence of a safe, effective system with good usability, potentially facilitating quicker and easier patient access to appropriate care while reducing the burden on primary and secondary care musculoskeletal services. This rigorous approach to mobile health system testing could be used as a guide for other developers of similar applications. Trial Registration ClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029 International Registered Report Identifier (IRRID) PRR1-10.2196/31541
Collapse
Affiliation(s)
- Cabella Lowe
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
| | - Harry Hanuman Sing
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
| | - William Marsh
- Risk and Information Systems Research Group, School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom
| | - Dylan Morrissey
- Centre for Sports & Exercise Medicine, William Harvey Research Institute, Queen Mary University of London, London, United Kingdom.,Department of Physiotherapy, Barts Health NHS Trust, London, United Kingdom
| |
Collapse
|
79
|
Janvrin ML, Korona-Bailey J, Koehlmoos TP. Re-examining COVID-19 Self-Reported Symptom Tracking Programs in the United States: Updated Framework Synthesis. JMIR Form Res 2021; 5:e31271. [PMID: 34792469 PMCID: PMC8651180 DOI: 10.2196/31271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2021] [Revised: 09/19/2021] [Accepted: 10/06/2021] [Indexed: 11/13/2022] Open
Abstract
BACKGROUND Early in the pandemic, in 2020, Koehlmoos et al completed a framework synthesis of currently available self-reported symptom tracking programs for COVID-19. This framework described relevant programs, partners and affiliates, funding, responses, platform, and intended audience, among other considerations. OBJECTIVE This study seeks to update the existing framework with the aim of identifying developments in the landscape and highlighting how programs have adapted to changes in pandemic response. METHODS Our team developed a framework to collate information on current COVID-19 self-reported symptom tracking programs using the "best-fit" framework synthesis approach. All programs from the previous study were included to document changes. New programs were discovered using a Google search for target keywords. The time frame for the search for programs ranged from March 1, 2021, to May 6, 2021. RESULTS We screened 33 programs, of which 8 were included in our final framework synthesis. We identified multiple common data elements, including demographic information such as race, age, gender, and affiliation (all were associated with universities, medical schools, or schools of public health). Dissimilarities included questions regarding vaccination status, vaccine hesitancy, adherence to social distancing, COVID-19 testing, and mental health. CONCLUSIONS At this time, the future of self-reported symptom tracking for COVID-19 is unclear. Some sources have speculated that COVID-19 may become a yearly occurrence much like the flu, and if so, the data that these programs generate is still valuable. However, it is unclear whether the public will maintain the same level of interest in reporting their symptoms on a regular basis if the prevalence of COVID-19 becomes more common.
Collapse
Affiliation(s)
- Miranda Lynn Janvrin
- The Henry M Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD, United States.,Uniformed Services University of the Health Sciences, Bethesda, MD, United States
| | - Jessica Korona-Bailey
- The Henry M Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD, United States.,Uniformed Services University of the Health Sciences, Bethesda, MD, United States
| | | |
Collapse
|
80
|
Ellertsson S, Loftsson H, Sigurdsson EL. Artificial intelligence in the GPs office: a retrospective study on diagnostic accuracy. Scand J Prim Health Care 2021; 39:448-458. [PMID: 34585629 PMCID: PMC8725823 DOI: 10.1080/02813432.2021.1973255] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
Abstract
OBJECTIVE Machine learning (ML) is expected to play an increasing role within primary health care (PHC) in coming years. No peer-reviewed studies exist that evaluate the diagnostic accuracy of ML models compared to general practitioners (GPs). The aim of this study was to evaluate the diagnostic accuracy of an ML classifier on primary headache diagnoses in PHC, compare its performance to GPs, and examine the most impactful signs and symptoms when making a prediction. DESIGN A retrospective study on diagnostic accuracy, using electronic health records from the database of the Primary Health Care Service of the Capital Area (PHCCA) in Iceland. SETTING Fifteen primary health care centers of the PHCCA. SUBJECTS All patients that consulted a physician, from 1 January 2006 to 30 April 2020, and received one of the selected diagnoses. MAIN OUTCOME MEASURES Sensitivity, Specificity, Positive Predictive Value, Matthews Correlation Coefficient, Receiver Operating Characteristic (ROC) curve, and Area under the ROC curve (AUROC) score for primary headache diagnoses, as well as Shapley Additive Explanations (SHAP) values of the ML classifier. RESULTS The classifier outperformed the GPs on all metrics except specificity. The SHAP values indicate that the classifier uses the same signs and symptoms (features) as a physician would, when distinguishing between headache diagnoses. CONCLUSION In a retrospective comparison, the diagnostic accuracy of the ML classifier for primary headache diagnoses is superior to GPs. According to SHAP values, the ML classifier relies on the same signs and symptoms as a physician when making a diagnostic prediction.KeypointsLittle is known about the diagnostic accuracy of machine learning (ML) in the context of primary health care, despite its considerable potential to aid in clinical work. This novel research sheds light on the diagnostic accuracy of ML in a clinical context, as well as the interpretation of its predictions. If the vast potential of ML is to be utilized in primary health care, its performance, safety, and inner workings need to be understood by clinicians.
Collapse
Affiliation(s)
- Steindor Ellertsson
- Primary Health Care Service of the Capital Area, Reykjavik, Iceland
- CONTACT Steindor Ellertsson Primary Health Care Service of the Capital Area, Grenimelur 44, 107, Reykjavik, Iceland
| | - Hrafn Loftsson
- Department of Computer Science, Reykjavik University, Reykjavik, Iceland
| | - Emil L. Sigurdsson
- Primary Health Care Service of the Capital Area, Reykjavik, Iceland
- Development Centre for Primary Health Care in Iceland, Reykjavik, Iceland
- Department of Family Medicine, University of Iceland, Reykjavik, Iceland
| |
Collapse
|
81
|
Chan F, Lai S, Pieterman M, Richardson L, Singh A, Peters J, Toy A, Piccininni C, Rouault T, Wong K, Quong JK, Wakabayashi AT, Pawelec-Brzychczy A. Performance of a new symptom checker in patient triage: Canadian cohort study. PLoS One 2021; 16:e0260696. [PMID: 34852016 PMCID: PMC8635379 DOI: 10.1371/journal.pone.0260696] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2020] [Accepted: 11/15/2021] [Indexed: 11/19/2022] Open
Abstract
BACKGROUND Computerized algorithms known as symptom checkers aim to help patients decide what to do should they have a new medical concern. However, despite widespread implementation, most studies on symptom checkers have involved simulated patients. Only limited evidence currently exists about symptom checker safety or accuracy when used by real patients. We developed a new prototype symptom checker and assessed its safety and accuracy in a prospective cohort of patients presenting to primary care and emergency departments with new medical concerns. METHOD A prospective cohort study was done to assess the prototype's performance. The cohort consisted of adult patients (≥16 years old) who presented to hospital emergency departments and family physician clinics. Primary outcomes were safety and accuracy of triage recommendations to seek hospital care, seek primary care, or manage symptoms at home. RESULTS Data from 281 hospital patients and 300 clinic patients were collected and analyzed. Sensitivity to emergencies was 100% (10/10 encounters). Sensitivity to urgencies was 90% (73/81) and 97% (34/35) for hospital and primary care patients, respectively. The prototype was significantly more accurate than patients at triage (73% versus 58%, p<0.01). Compliance with triage recommendations in this cohort using this iteration of the symptom checker would have reduced hospital visits by 55% but cause potential harm in 2-3% from delay in care. INTERPRETATION The prototype symptom checker was superior to patients in deciding the most appropriate treatment setting for medical issues. This symptom checker could reduce a significant number of unnecessary hospital visits, with accuracy and safety outcomes comparable to existing data on telephone triage.
Collapse
Affiliation(s)
- Forson Chan
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Simon Lai
- University of British Columbia, Faculty of Medicine, Health Sciences Mall, Vancouver, Canada
| | - Marcus Pieterman
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Lisa Richardson
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Amanda Singh
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Jocelynn Peters
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Alex Toy
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Caroline Piccininni
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Taiysa Rouault
- University of British Columbia, Faculty of Medicine, Health Sciences Mall, Vancouver, Canada
| | - Kristie Wong
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | | | - Adrienne T. Wakabayashi
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| | - Anna Pawelec-Brzychczy
- Department of Family Medicine, Schulich School of Medicine and Dentistry, Western University, WCPHFM, London, ON, Canada
| |
Collapse
|
82
|
Kopka M, Schmieding ML, Rieger T, Roesler E, Balzer F, Feufel MA. Trust Me, I’m Not a Doctor! Determinants of Laypersons’ Trust in Medical Decision Aids: Experimental Study (Preprint). JMIR Hum Factors 2021; 9:e35219. [PMID: 35503248 PMCID: PMC9115664 DOI: 10.2196/35219] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2021] [Revised: 02/09/2022] [Accepted: 03/06/2022] [Indexed: 11/13/2022] Open
Abstract
Background Objective Methods Results Conclusions Trial Registration
Collapse
Affiliation(s)
- Marvin Kopka
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Cognitive Psychology and Ergonomics, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Berlin, Germany
| | - Malte L Schmieding
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Tobias Rieger
- Work, Engineering and Organizational Psychology, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Berlin, Germany
| | - Eileen Roesler
- Work, Engineering and Organizational Psychology, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Berlin, Germany
| | - Felix Balzer
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Markus A Feufel
- Division of Ergonomics, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Berlin, Germany
| |
Collapse
|
83
|
Meer EA, Herriman M, Lam D, Parambath A, Rosin R, Volpp KG, Chaiyachati KH, McGreevey JD. Design, Implementation, and Validation of an Automated, Algorithmic COVID-19 Triage Tool. Appl Clin Inform 2021; 12:1021-1028. [PMID: 34734403 DOI: 10.1055/s-0041-1736627] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022] Open
Abstract
OBJECTIVE We describe the design, implementation, and validation of an online, publicly available tool to algorithmically triage patients experiencing severe acute respiratory syndrome coronavirus (SARS-CoV-2)-like symptoms. METHODS We conducted a chart review of patients who completed the triage tool and subsequently contacted our institution's phone triage hotline to assess tool- and clinician-assigned triage codes, patient demographics, SARS-CoV-2 (COVID-19) test data, and health care utilization in the 30 days post-encounter. We calculated the percentage of concordance between tool- and clinician-assigned triage categories, down-triage (clinician assigning a less severe category than the triage tool), and up-triage (clinician assigning a more severe category than the triage tool) instances. RESULTS From May 4, 2020 through January 31, 2021, the triage tool was completed 30,321 times by 20,930 unique patients. Of those 30,321 triage tool completions, 51.7% were assessed by the triage tool to be asymptomatic, 15.6% low severity, 21.7% moderate severity, and 11.0% high severity. The concordance rate, where the triage tool and clinician assigned the same clinical severity, was 29.2%. The down-triage rate was 70.1%. Only six patients were up-triaged by the clinician. 72.1% received a COVID-19 test administered by our health care system within 14 days of their encounter, with a positivity rate of 14.7%. CONCLUSION The design, pilot, and validation analysis in this study show that this COVID-19 triage tool can safely triage patients when compared with clinician triage personnel. This work may signal opportunities for automated triage of patients for conditions beyond COVID-19 to improve patient experience by enabling self-service, on-demand, 24/7 triage access.
Collapse
Affiliation(s)
- Elana A Meer
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Center for Health Incentives and Behavioral Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Maguire Herriman
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Center for Health Incentives and Behavioral Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Doreen Lam
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Andrew Parambath
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - Roy Rosin
- Center for Health Incentives and Behavioral Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Penn Medicine Center for Health Care Innovation, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
| | - Kevin G Volpp
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Center for Health Incentives and Behavioral Economics, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Department of Medicine, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
| | - Krisda H Chaiyachati
- Penn Medicine Center for Health Care Innovation, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States.,Department of Medicine, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States.,Leonard Davis Institute, University of Pennsylvania, Philadelphia, Pennsylvania, United States
| | - John D McGreevey
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, United States.,Office of the Chief Medical Information Officer, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States.,Center for Applied Health Informatics, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
| |
Collapse
|
84
|
Woodcock C, Mittelstadt B, Busbridge D, Blank G. The Impact of Explanations on Layperson Trust in Artificial Intelligence-Driven Symptom Checker Apps: Experimental Study. J Med Internet Res 2021; 23:e29386. [PMID: 34730544 PMCID: PMC8600426 DOI: 10.2196/29386] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2021] [Revised: 07/11/2021] [Accepted: 07/27/2021] [Indexed: 01/22/2023] Open
Abstract
BACKGROUND Artificial intelligence (AI)-driven symptom checkers are available to millions of users globally and are advocated as a tool to deliver health care more efficiently. To achieve the promoted benefits of a symptom checker, laypeople must trust and subsequently follow its instructions. In AI, explanations are seen as a tool to communicate the rationale behind black-box decisions to encourage trust and adoption. However, the effectiveness of the types of explanations used in AI-driven symptom checkers has not yet been studied. Explanations can follow many forms, including why-explanations and how-explanations. Social theories suggest that why-explanations are better at communicating knowledge and cultivating trust among laypeople. OBJECTIVE The aim of this study is to ascertain whether explanations provided by a symptom checker affect explanatory trust among laypeople and whether this trust is impacted by their existing knowledge of disease. METHODS A cross-sectional survey of 750 healthy participants was conducted. The participants were shown a video of a chatbot simulation that resulted in the diagnosis of either a migraine or temporal arteritis, chosen for their differing levels of epidemiological prevalence. These diagnoses were accompanied by one of four types of explanations. Each explanation type was selected either because of its current use in symptom checkers or because it was informed by theories of contrastive explanation. Exploratory factor analysis of participants' responses followed by comparison-of-means tests were used to evaluate group differences in trust. RESULTS Depending on the treatment group, two or three variables were generated, reflecting the prior knowledge and subsequent mental model that the participants held. When varying explanation type by disease, migraine was found to be nonsignificant (P=.65) and temporal arteritis, marginally significant (P=.09). Varying disease by explanation type resulted in statistical significance for input influence (P=.001), social proof (P=.049), and no explanation (P=.006), with counterfactual explanation (P=.053). The results suggest that trust in explanations is significantly affected by the disease being explained. When laypeople have existing knowledge of a disease, explanations have little impact on trust. Where the need for information is greater, different explanation types engender significantly different levels of trust. These results indicate that to be successful, symptom checkers need to tailor explanations to each user's specific question and discount the diseases that they may also be aware of. CONCLUSIONS System builders developing explanations for symptom-checking apps should consider the recipient's knowledge of a disease and tailor explanations to each user's specific need. Effort should be placed on generating explanations that are personalized to each user of a symptom checker to fully discount the diseases that they may be aware of and to close their information gap.
Collapse
Affiliation(s)
- Claire Woodcock
- Oxford Internet Institute, University of Oxford, Oxford, United Kingdom
| | - Brent Mittelstadt
- Oxford Internet Institute, University of Oxford, Oxford, United Kingdom
| | | | - Grant Blank
- Oxford Internet Institute, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
85
|
Aboueid S, Meyer SB, Wallace J, Chaurasia A. Latent classes associated with the intention to use a symptom checker for self-triage. PLoS One 2021; 16:e0259547. [PMID: 34731217 PMCID: PMC8565791 DOI: 10.1371/journal.pone.0259547] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2021] [Accepted: 10/20/2021] [Indexed: 11/23/2022] Open
Abstract
It is currently unknown which attitude-based profiles are associated with symptom checker use for self-triage. We sought to identify, among university students, attitude-based latent classes (population profiles) and the association between latent classes with the future use of symptom checkers for self-triage. Informed by the Technology Acceptance Model and a larger mixed methods study, a cross-sectional survey was developed and administered to students (aged between 18 and 34 years of age) at a University in Ontario. Latent class analysis (LCA) was used to identify attitude-based profiles that exist among the sample while general linear modeling was applied to identify the association between latent classes and future symptom checker use for self-triage. Of the 1,547 students who opened the survey link, 1,365 did not use a symptom checker in the past year and were thus identified as "non-users". After removing missing data (remaining sample = n = 1,305), LCA revealed five attitude-based profiles: tech acceptors, tech rejectors, skeptics, tech seekers, and unsure acceptors. Tech acceptors and tech rejectors were the most and least prevalent classes, respectively. As compared to tech rejectors, tech seekers and unsure acceptors were the latent classes with the highest and lowest odds of future symptom checker use, respectively. After controlling for confounders, the effect of latent classes on symptom checker use remains significant (p-value < .0001) with the odds of future use in tech acceptors being 5.6 times higher than the odds of future symptom checker use in tech rejectors [CI: (3.458, 9.078); p-value < .0001]. Attitudes towards AI and symptom checker functionality result in different population profiles that have different odds of using symptom checkers for self-triage. Identifying a person's or group's membership to a population profile could help in developing and delivering tailored interventions aimed at maximizing use of validated symptom checkers.
Collapse
Affiliation(s)
- Stephanie Aboueid
- School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario, Canada
| | - Samantha B. Meyer
- School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario, Canada
| | - James Wallace
- School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario, Canada
| | - Ashok Chaurasia
- School of Public Health and Health Systems, University of Waterloo, Waterloo, Ontario, Canada
| |
Collapse
|
86
|
Drozdowski R, Sinha S, Lin G, Feng H. Accuracy of popular online symptom checkers for dermatological diagnoses. Clin Exp Dermatol 2021; 47:456-457. [PMID: 34609769 DOI: 10.1111/ced.14960] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Accepted: 09/29/2021] [Indexed: 11/27/2022]
Affiliation(s)
- R Drozdowski
- University of Connecticut School of Medicine, Farmington, CT, USA
| | - S Sinha
- Frank H. Netter MD School of Medicine at Quinnipiac University, North Haven, CT, USA
| | - G Lin
- Department of Dermatology, University of Connecticut Health Center, Farmington, CT, USA
| | - H Feng
- Department of Dermatology, University of Connecticut Health Center, Farmington, CT, USA
| |
Collapse
|
87
|
Lee JT, Hsieh CC, Lin CH, Lin YJ, Kao CY. Prediction of hospitalization using artificial intelligence for urgent patients in the emergency department. Sci Rep 2021; 11:19472. [PMID: 34593930 PMCID: PMC8484275 DOI: 10.1038/s41598-021-98961-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2021] [Accepted: 09/17/2021] [Indexed: 11/10/2022] Open
Abstract
Timely assessment to accurately prioritize patients is crucial for emergency department (ED) management. Urgent (i.e., level-3, on a 5-level emergency severity index system) patients have become a challenge since under-triage and over-triage often occur. This study was aimed to develop a computational model by artificial intelligence (AI) methodologies to accurately predict urgent patient outcomes using data that are readily available in most ED triage systems. We retrospectively collected data from the ED of a tertiary teaching hospital between January 1, 2015 and December 31, 2019. Eleven variables were used for data analysis and prediction model building, including 1 response, 2 demographic, and 8 clinical variables. A model to predict hospital admission was developed using neural networks and machine learning methodologies. A total of 282,971 samples of urgent (level-3) visits were included in the analysis. Our model achieved a validation area under the curve (AUC) of 0.8004 (95% CI 0.7963-0.8045). The optimal cutoff value identified by Youden's index for determining hospital admission was 0.5517. Using this cutoff value, the sensitivity was 0.6721 (95% CI 0.6624-0.6818), and the specificity was 0.7814 (95% CI 0.7777-0.7851), with a positive predictive value of 0.3660 (95% CI 0.3586-0.3733) and a negative predictive value of 0.9270 (95% CI 0.9244-0.9295). Subgroup analysis revealed that this model performed better in the nontraumatic adult subgroup and achieved a validation AUC of 0.8166 (95% CI 0.8199-0.8212). Our AI model accurately assessed the need for hospitalization for urgent patients, which constituted nearly 70% of ED visits. This model demonstrates the potential for streamlining ED operations using a very limited number of variables that are readily available in most ED triage systems. Subgroup analysis is an important topic for future investigation.
Collapse
Affiliation(s)
- Jung-Ting Lee
- Si-Wan College, National Sun Yat-Sen University, Kaohsiung, Taiwan
| | - Chih-Chia Hsieh
- Department of Emergency Medicine, National Cheng Kung University Hospital, College of Medicine, National Cheng Kung University, No.138, Shengli Rd., North District, Tainan, 70403, Taiwan
| | - Chih-Hao Lin
- Department of Emergency Medicine, National Cheng Kung University Hospital, College of Medicine, National Cheng Kung University, No.138, Shengli Rd., North District, Tainan, 70403, Taiwan.
| | - Yu-Jen Lin
- Department of Electrical Engineering, National Sun Yat-Sen University, Kaohsiung, Taiwan
| | - Chung-Yao Kao
- Department of Electrical Engineering, National Sun Yat-Sen University, Kaohsiung, Taiwan
| |
Collapse
|
88
|
Epstein S, Timmermans S. From Medicine to Health: The Proliferation and Diversification of Cultural Authority. JOURNAL OF HEALTH AND SOCIAL BEHAVIOR 2021; 62:240-254. [PMID: 34528483 DOI: 10.1177/00221465211010468] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
In his account of the medical profession's ascent, Paul Starr drew a distinction between the social authority of physicians and the cultural authority of medicine-between doctors' capacity to direct others' behavior and the ability of medical institutions and discourses to shape meanings of illness, health, wellness, and treatment. Subsequently, scholars have reflected on the social-structural transformations challenging physicians' social authority but neglected shifts in cultural authority. Focusing on the United States, we find a proliferation and diversification of cultural authority, reflecting a partial movement from the domain of medicine into new terrains of health. This shift is apparent in the resurgence of alternative healing, the advent of new forms of self-care and self-monitoring, the rise of health social movements, and the spread of health information online. We advance a research agenda to understand how the mechanisms and dynamics of cultural authority shape contests to speak in the name of health.
Collapse
|
89
|
Delshad S, Dontaraju VS, Chengat V. Artificial Intelligence-Based Application Provides Accurate Medical Triage Advice When Compared to Consensus Decisions of Healthcare Providers. Cureus 2021; 13:e16956. [PMID: 34405077 PMCID: PMC8352839 DOI: 10.7759/cureus.16956] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/06/2021] [Indexed: 12/23/2022] Open
Abstract
Accurate medical triage is essential for improving patient outcomes and efficient healthcare delivery. Patients increasingly rely on artificial intelligence (AI)-based applications to access healthcare information, including medical triage advice. We assessed the accuracy of triage decisions provided by an AI-based application. We presented 50 clinical vignettes to the AI-based application, seven emergency medicine providers, and five internal medicine physicians. We compared the triage decisions of the AI-based application to those of the individual providers as well as their consensus decisions. When compared to the human clinicians’ consensus triage decisions, the AI-based application performed equal or better than individual human clinicians.
Collapse
Affiliation(s)
- Sean Delshad
- Internal Medicine, David Geffen School of Medicine at University of California Los Angeles, Los Angeles, USA
| | | | | |
Collapse
|
90
|
Rosenfield D, McLean L, Clemens J, Fischer J. Opening the digital front door: digital offerings in a pediatric emergency department. CAN J EMERG MED 2021; 23:857. [PMID: 34351600 PMCID: PMC8340589 DOI: 10.1007/s43678-021-00187-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Accepted: 07/24/2021] [Indexed: 11/28/2022]
Affiliation(s)
| | - Lianne McLean
- , 555 University Avenue, 12th Floor, Toronto, ON, M5G1X8, Canada
| | - Julia Clemens
- , 555 University Avenue, 12th Floor, Toronto, ON, M5G1X8, Canada
| | - Jason Fischer
- , 555 University Avenue, 12th Floor, Toronto, ON, M5G1X8, Canada
| |
Collapse
|
91
|
Cadamuro J. Rise of the Machines: The Inevitable Evolution of Medicine and Medical Laboratories Intertwining with Artificial Intelligence-A Narrative Review. Diagnostics (Basel) 2021; 11:1399. [PMID: 34441333 PMCID: PMC8392825 DOI: 10.3390/diagnostics11081399] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2021] [Revised: 07/05/2021] [Accepted: 07/21/2021] [Indexed: 01/04/2023] Open
Abstract
Laboratory medicine has evolved from a mainly manual profession, providing few selected test results to a highly automated and standardized medical discipline, generating millions of test results per year. As the next inevitable evolutional step, artificial intelligence (AI) algorithms will need to assist us in structuring and making sense of the masses of diagnostic data collected today. Such systems will be able to connect clinical and diagnostic data and to provide valuable suggestions in diagnosis, prognosis or therapeutic options. They will merge the often so separated worlds of the laboratory and the clinics. When used correctly, it will be a tool, capable of freeing the physicians time so that he/she can refocus on the patient. In this narrative review I therefore aim to provide an overview of what AI is, what applications currently are available in healthcare and in laboratory medicine in particular. I will discuss the challenges and pitfalls of applying AI algorithms and I will elaborate on the question if healthcare workers will be replaced by such systems in the near future.
Collapse
Affiliation(s)
- Janne Cadamuro
- Department of Laboratory Medicine, Paracelsus Medical University, A-5020 Salzburg, Austria
| |
Collapse
|
92
|
Ceney A, Tolond S, Glowinski A, Marks B, Swift S, Palser T. Accuracy of online symptom checkers and the potential impact on service utilisation. PLoS One 2021; 16:e0254088. [PMID: 34265845 PMCID: PMC8282353 DOI: 10.1371/journal.pone.0254088] [Citation(s) in RCA: 33] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2020] [Accepted: 06/13/2021] [Indexed: 02/03/2023] Open
Abstract
OBJECTIVES The aims of our study are firstly to investigate the diagnostic and triage performance of symptom checkers, secondly to assess their potential impact on healthcare utilisation and thirdly to investigate for variation in performance between systems. SETTING Publicly available symptom checkers for patient use. PARTICIPANTS Publicly available symptom-checkers were identified. A standardised set of 50 clinical vignettes were developed and systematically run through each system by a non-clinical researcher. PRIMARY AND SECONDARY OUTCOME MEASURES System accuracy was assessed by measuring the percentage of times the correct diagnosis was a) listed first, b) within the top five diagnoses listed and c) listed at all. The safety of the disposition advice was assessed by comparing it with national guidelines for each vignette. RESULTS Twelve tools were identified and included. Mean diagnostic accuracy of the systems was poor, with the correct diagnosis being present in the top five diagnoses on 51.0% (Range 22.2 to 84.0%). Safety of disposition advice decreased with condition urgency (being 71.8% for emergency cases vs 87.3% for non-urgent cases). 51.0% of systems suggested additional resource utilisation above that recommended by national guidelines (range 18.0% to 61.2%). Both diagnostic accuracy and appropriate resource recommendation varied substantially between systems. CONCLUSIONS There is wide variation in performance between available symptom checkers and overall performance is significantly below what would be accepted in any other medical field, though some do achieve a good level of accuracy and safety of disposition. External validation and regulation are urgently required to ensure these public facing tools are safe.
Collapse
Affiliation(s)
- Adam Ceney
- Methods Analytics Ltd, Sheffield, United Kingdom
| | | | | | - Ben Marks
- Methods Analytics Ltd, Sheffield, United Kingdom
| | - Simon Swift
- Methods Analytics Ltd, Sheffield, United Kingdom
- University of Exeter Business School (INDEX), Exeter, United Kingdom
| | - Tom Palser
- Methods Analytics Ltd, Sheffield, United Kingdom
- Department of Surgery, University Hospitals of Leicester NHS Trust, Leicester, United Kingdom
- SAPPHIRE, Department of Health Sciences, University of Leicester, Leicester, United Kingdom
| |
Collapse
|
93
|
Duran E. Diagnostic slippage: Medical uncertainty and engaged patienthood in the case of atypical disorders. Soc Sci Med 2021; 280:114054. [PMID: 34052701 DOI: 10.1016/j.socscimed.2021.114054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Revised: 05/07/2021] [Accepted: 05/17/2021] [Indexed: 11/29/2022]
Abstract
This study explores how patients affected by depersonalization/derealization disorder confront clinical uncertainty by investing in their own diagnosis and prognosis. While depersonalization/derealization has been a legitimate medical category within psychiatric nomenclature for decades, those affected by this condition have traditionally embarked on a diagnostic odyssey. As a rare disorder, clinicians have repeatedly misdiagnosed depersonalization/derealization as a byproduct of illnesses such as anxiety disorder, depression, or schizophrenia. Drawing from in-depth interviews, as well the analysis of videos and online posts, I explore the trajectory through which patients affected by depersonalization/derealization, dissatisfied with traditional diagnostic routes, have transformed the process of diagnostic revelation from a clinical prerogative to a lay accomplishment. This study indicates that a multifaceted process, consisting of a dialectical relation between the material reality of information democratization and a heightening cultural orientation towards engaged patienthood, may shorten diagnostic odysseys for people affected by atypical disorders and undermine physicians' authority to diagnose.
Collapse
Affiliation(s)
- Eduardo Duran
- University of California, Los Angeles Department of Sociology 375 Portola Plaza, Los Angeles, CA, 90095, USA.
| |
Collapse
|
94
|
Munsch N, Martin A, Gruarin S, Nateqi J, Abdarahmane I, Weingartner-Ortner R, Knapp B. Authors' Reply to: Screening Tools: Their Intended Audiences and Purposes. Comment on "Diagnostic Accuracy of Web-Based COVID-19 Symptom Checkers: Comparison Study". J Med Internet Res 2021; 23:e26543. [PMID: 33989162 PMCID: PMC8178729 DOI: 10.2196/26543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Accepted: 05/13/2021] [Indexed: 11/24/2022] Open
Affiliation(s)
| | | | | | - Jama Nateqi
- Medical Department, Symptoma, Attersee, Austria.,Department of Internal Medicine, Paracelsus Medical University, Salzburg, Austria
| | | | | | - Bernhard Knapp
- Data Science Department, Symptoma, Vienna, Austria.,Department of Computer Science, University of Applied Sciences - Technikum Wien, Vienna, Austria
| |
Collapse
|
95
|
Menendez ME, Moverman MA, Moon AS, Ring D. State-level Google search volumes for neck and shoulder pain correlate with psychosocial and behavioral health indicators. J Natl Med Assoc 2021; 113:522-527. [PMID: 33962802 DOI: 10.1016/j.jnma.2021.04.001] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 03/22/2021] [Accepted: 04/05/2021] [Indexed: 01/08/2023]
Abstract
BACKGROUND There is growing interest in passively-generated online search data for health sciences research, but limited use to address musculoskeletal illness. QUESTIONS/PURPOSES (1) To determine geographic and seasonal patterns in Google queries for shoulder, knee, hip, and neck pain across the United States, and (2) to quantify the state-level correlation of the volume of online symptom searches with rates of various population health indicators. METHODS Using Google trends, we identified queries for shoulder, knee, hip, and neck pain in every state of the United States during 2018. We assessed the correlation of state-level Google search volumes with 9 population health indicators extracted from the 2018 Behavioral Risk Factor Surveillance System survey: tobacco use, obesity, diabetes, hyperlipidemia, depression, arthritis, cardiovascular disease, participation in physical activities, and college education. RESULTS Neck and shoulder pain were the most common queries in most Southern states, while knee pain was generally the top query elsewhere. Queries for neck and shoulder pain peaked during the late fall and winter, while searches for knee and hip pain peaked in the spring. State-level search volumes for shoulder and neck pain correlated closely with all health indicators--particularly with tobacco use, obesity, cardiovascular disease, hyperlipidemia, participation in physical activities and college education. The only strong correlation of hip pain queries was with self-reported prevalence of arthritis. Knee pain queries were not associated with any of the studied health indicators. CONCLUSION This study highlights the potential of search engine data to be utilized as population-level health indicators. The state-level correlation of psychosocial and behavioral health indicators with online search volumes for neck and shoulder pain may reflect the influence of mental and social health on the experience of pain.
Collapse
Affiliation(s)
- Mariano E Menendez
- Department of Orthopaedic Surgery, New England Baptist Hospital, Tufts University School of Medicine, 800 Washington St, TMC Box #306, Boston, MA 02111, USA.
| | - Michael A Moverman
- Department of Orthopaedic Surgery, New England Baptist Hospital, Tufts University School of Medicine, 800 Washington St, TMC Box #306, Boston, MA 02111, USA
| | - Andrew S Moon
- Department of Orthopaedic Surgery, New England Baptist Hospital, Tufts University School of Medicine, 800 Washington St, TMC Box #306, Boston, MA 02111, USA
| | - David Ring
- Department of Surgery and Perioperative Care, Dell Medical School at the University of Texas at Austin, TX, USA
| |
Collapse
|
96
|
Shimizu T. The 6C model for accurately capturing the patient's medical history. Diagnosis (Berl) 2021; 9:dx-2020-0126. [PMID: 33887130 DOI: 10.1515/dx-2020-0126] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2020] [Accepted: 02/01/2021] [Indexed: 01/08/2023]
Abstract
Diagnostic errors are an internationally recognized patient safety concern, and leading causes are faulty data gathering and faulty information processing. Obtaining a full and accurate history from the patient is the foundation for timely and accurate diagnosis. A key concept underlying ideal history acquisition is "history clarification," meaning that the history is clarified to be depicted as clearly as a video, with the chronology being accurately reproduced. A novel approach is presented to improve history-taking, involving six dimensions: Courtesy, Control, Compassion, Curiosity, Clear mind, and Concentration, the '6 C's'. We report a case that illustrates how the 6C approach can improve diagnosis, especially in relation to artificial intelligence tools that assist with differential diagnosis.
Collapse
Affiliation(s)
- Taro Shimizu
- Dokkyo Medical University Hospital, Kitakobayashi 880, Mibu, Tochigi, 321-0297, Japan
| |
Collapse
|
97
|
Knitza J, Mohn J, Bergmann C, Kampylafka E, Hagen M, Bohr D, Morf H, Araujo E, Englbrecht M, Simon D, Kleyer A, Meinderink T, Vorbrüggen W, von der Decken CB, Kleinert S, Ramming A, Distler JHW, Vuillerme N, Fricker A, Bartz-Bazzanella P, Schett G, Hueber AJ, Welcker M. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res Ther 2021; 23:112. [PMID: 33849654 PMCID: PMC8042673 DOI: 10.1186/s13075-021-02498-8] [Citation(s) in RCA: 38] [Impact Index Per Article: 12.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2021] [Accepted: 03/31/2021] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND Timely diagnosis and treatment are essential in the effective management of inflammatory rheumatic diseases (IRDs). Symptom checkers (SCs) promise to accelerate diagnosis, reduce misdiagnoses, and guide patients more effectively through the health care system. Although SCs are increasingly used, there exists little supporting evidence. OBJECTIVE To assess the diagnostic accuracy, patient-perceived usability, and acceptance of two SCs: (1) Ada and (2) Rheport. METHODS Patients newly presenting to a German secondary rheumatology outpatient clinic were randomly assigned in a 1:1 ratio to complete Ada or Rheport and consecutively the respective other SCs in a prospective non-blinded controlled randomized crossover trial. The primary outcome was the accuracy of the SCs regarding the diagnosis of an IRD compared to the physicians' diagnosis as the gold standard. The secondary outcomes were patient-perceived usability, acceptance, and time to complete the SC. RESULTS In this interim analysis, the first 164 patients who completed the study were analyzed. 32.9% (54/164) of the study subjects were diagnosed with an IRD. Rheport showed a sensitivity of 53.7% and a specificity of 51.8% for IRDs. Ada's top 1 (D1) and top 5 disease suggestions (D5) showed a sensitivity of 42.6% and 53.7% and a specificity of 63.6% and 54.5% concerning IRDs, respectively. The correct diagnosis of the IRD patients was within the Ada D1 and D5 suggestions in 16.7% (9/54) and 25.9% (14/54), respectively. The median System Usability Scale (SUS) score of Ada and Rheport was 75.0/100 and 77.5/100, respectively. The median completion time for both Ada and Rheport was 7.0 and 8.5 min, respectively. Sixty-four percent and 67.1% would recommend using Ada and Rheport to friends and other patients, respectively. CONCLUSIONS While SCs are well accepted among patients, their diagnostic accuracy is limited to date. TRIAL REGISTRATION DRKS.de, DRKS00017642 . Registered on 23 July 2019.
Collapse
Affiliation(s)
- Johannes Knitza
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany.
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany.
- Université Grenoble Alpes, AGEIS, Grenoble, France.
| | - Jacob Mohn
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Christina Bergmann
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Eleni Kampylafka
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Melanie Hagen
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Daniela Bohr
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Harriet Morf
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Elizabeth Araujo
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Matthias Englbrecht
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - David Simon
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Arnd Kleyer
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Timo Meinderink
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Wolfgang Vorbrüggen
- Verein zur Förderung der Rheumatologie e.V., Würselen, Germany
- RheumaDatenRhePort (rhadar), Planegg, Germany
| | - Cay Benedikt von der Decken
- RheumaDatenRhePort (rhadar), Planegg, Germany
- Medizinisches Versorgungszentrum Stolberg, Stolberg, Germany
- Klinik für Internistische Rheumatologie, Rhein-Maas Klinikum, Würselen, Germany
| | - Stefan Kleinert
- RheumaDatenRhePort (rhadar), Planegg, Germany
- Rheumatologische Schwerpunktpraxis, Drs. Kleinert, Rapp, Ronneberger, Schuch U. Wendler, Rheumatology, Erlangen, Germany
| | - Andreas Ramming
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Jörg H W Distler
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Nicolas Vuillerme
- Université Grenoble Alpes, AGEIS, Grenoble, France
- Institut Universitaire de France, Paris, France
- LabCom Telecom4Health, University of Grenoble Alpes & Orange Labs, Grenoble, France
| | | | - Peter Bartz-Bazzanella
- RheumaDatenRhePort (rhadar), Planegg, Germany
- Klinik für Internistische Rheumatologie, Rhein-Maas Klinikum, Würselen, Germany
| | - Georg Schett
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Deutsches Zentrum für Immuntherapie (DZI), Friedrich-Alexander-University Erlangen-Nürnberg and Universitätsklinikum Erlangen, Erlangen, Germany
| | - Axel J Hueber
- Department of Internal Medicine 3, Friedrich-Alexander-University Erlangen-Nürnberg (FAU) and Universitätsklinikum Erlangen, Ulmenweg 18, 91054, Erlangen, Germany
- Section Rheumatology, Sozialstiftung Bamberg, Bamberg, Germany
| | - Martin Welcker
- RheumaDatenRhePort (rhadar), Planegg, Germany
- MVZ für Rheumatologie Dr. Martin Welcker GmbH, Planegg, Germany
| |
Collapse
|
98
|
Ghani S, Fan KS, Fan KH, Lenti L, Raptis D. Using the Ensuring Quality Information for Patients Tool to Assess Patient Information on Appendicitis Websites: Systematic Search and Evaluation. J Med Internet Res 2021; 23:e22618. [PMID: 33729160 PMCID: PMC8035662 DOI: 10.2196/22618] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2020] [Revised: 09/07/2020] [Accepted: 03/17/2021] [Indexed: 01/22/2023] Open
Abstract
BACKGROUND Appendicitis is a common surgical problem among the young adult population, who are likely to use the internet to obtain medical information. This information may determine the health-seeking behavior of an individual and may delay medical attention. Little is known regarding the quality of patient information on appendicitis on the internet, as this has not been previously studied. OBJECTIVE The aim of our study was to identify the quality of information regarding appendicitis on websites intended for the public. METHODS We conducted a systematic review of information on appendicitis available online using the following 4 search terms in google: "appendicitis," "appendix," "appendectomy," and "appendicectomy". The top 100 websites of each search term were assessed using the validated Ensuring Quality Information for Patients (EQIP) tool (score 0-36). RESULTS A total of 119 websites met the eligibility criteria for evaluation. The overall median EQIP score for all websites was 20 (IQR 18-22). More than half the websites originated from the USA (65/119, 54.6%), and 45.4% (54/119) of all websites originated from hospitals, although 43% (23/54) of these did not mention qualitative risks from surgery. Incidence rates were only provided for complications and mortality in 12.6% (15/119) and 3.3% (4/119) of all websites, respectively. CONCLUSIONS The assessment of the quality and readability of websites concerning appendicitis by the EQIP tool indicates that most sites online were of poor credibility, with minimal information regarding complication rates and mortality. To improve education and awareness of appendicitis, there is an immediate need for more informative and patient-centered websites that are more compatible with international quality standards.
Collapse
Affiliation(s)
- Shahi Ghani
- St George's, University of London, London, United Kingdom
| | - Ka Siu Fan
- St George's, University of London, London, United Kingdom
| | - Ka Hay Fan
- Imperial College London, London, United Kingdom
| | - Lorenzo Lenti
- St George's, University of London, London, United Kingdom
| | - Dimitri Raptis
- Department of HPB Surgery and Liver Transplantation, Royal Free Hospital, London, United Kingdom
| |
Collapse
|
99
|
Schmieding ML, Mörgeli R, Schmieding MAL, Feufel MA, Balzer F. Benchmarking Triage Capability of Symptom Checkers Against That of Medical Laypersons: Survey Study. J Med Internet Res 2021; 23:e24475. [PMID: 33688845 PMCID: PMC7991983 DOI: 10.2196/24475] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2020] [Revised: 10/22/2020] [Accepted: 01/18/2021] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND Symptom checkers (SCs) are tools developed to provide clinical decision support to laypersons. Apart from suggesting probable diagnoses, they commonly advise when users should seek care (triage advice). SCs have become increasingly popular despite prior studies rating their performance as mediocre. To date, it is unclear whether SCs can triage better than those who might choose to use them. OBJECTIVE This study aims to compare triage accuracy between SCs and their potential users (ie, laypersons). METHODS On Amazon Mechanical Turk, we recruited 91 adults from the United States who had no professional medical background. In a web-based survey, the participants evaluated 45 fictitious clinical case vignettes. Data for 15 SCs that had processed the same vignettes were obtained from a previous study. As main outcome measures, we assessed the accuracy of the triage assessments made by participants and SCs for each of the three triage levels (ie, emergency care, nonemergency care, self-care) and overall, the proportion of participants outperforming each SC in terms of accuracy, and the risk aversion of participants and SCs by comparing the proportion of cases that were overtriaged. RESULTS The mean overall triage accuracy was similar for participants (60.9%, SD 6.8%; 95% CI 59.5%-62.3%) and SCs (58%, SD 12.8%). Most participants outperformed all but 5 SCs. On average, SCs more reliably detected emergencies (80.6%, SD 17.9%) than laypersons did (67.5%, SD 16.4%; 95% CI 64.1%-70.8%). Although both SCs and participants struggled with cases requiring self-care (the least urgent triage category), SCs more often wrongly classified these cases as emergencies (43/174, 24.7%) compared with laypersons (56/1365, 4.10%). CONCLUSIONS Most SCs had no greater triage capability than an average layperson, although the triage accuracy of the five best SCs was superior to the accuracy of most participants. SCs might improve early detection of emergencies but might also needlessly increase resource utilization in health care. Laypersons sometimes require support in deciding when to rely on self-care but it is in that very situation where SCs perform the worst. Further research is needed to determine how to best combine the strengths of humans and SCs.
Collapse
Affiliation(s)
- Malte L Schmieding
- Department of Anesthesiology and Operative Intensive Care, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Rudolf Mörgeli
- Department of Anesthesiology and Operative Intensive Care, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| | - Maike A L Schmieding
- Department of Biology, Chemistry, and Pharmacy, Institute of Pharmacy, Freie Universität Berlin, Berlin, Germany
| | - Markus A Feufel
- Department of Psychology and Ergonomics (IPA), Division of Ergonomics, Technische Universität Berlin, Berlin, Germany
| | - Felix Balzer
- Department of Anesthesiology and Operative Intensive Care, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
- Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany
| |
Collapse
|
100
|
Jones OT, Calanzani N, Saji S, Duffy SW, Emery J, Hamilton W, Singh H, de Wit NJ, Walter FM. Artificial Intelligence Techniques That May Be Applied to Primary Care Data to Facilitate Earlier Diagnosis of Cancer: Systematic Review. J Med Internet Res 2021; 23:e23483. [PMID: 33656443 PMCID: PMC7970165 DOI: 10.2196/23483] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2020] [Revised: 11/05/2020] [Accepted: 11/30/2020] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND More than 17 million people worldwide, including 360,000 people in the United Kingdom, were diagnosed with cancer in 2018. Cancer prognosis and disease burden are highly dependent on the disease stage at diagnosis. Most people diagnosed with cancer first present in primary care settings, where improved assessment of the (often vague) presenting symptoms of cancer could lead to earlier detection and improved outcomes for patients. There is accumulating evidence that artificial intelligence (AI) can assist clinicians in making better clinical decisions in some areas of health care. OBJECTIVE This study aimed to systematically review AI techniques that may facilitate earlier diagnosis of cancer and could be applied to primary care electronic health record (EHR) data. The quality of the evidence, the phase of development the AI techniques have reached, the gaps that exist in the evidence, and the potential for use in primary care were evaluated. METHODS We searched MEDLINE, Embase, SCOPUS, and Web of Science databases from January 01, 2000, to June 11, 2019, and included all studies providing evidence for the accuracy or effectiveness of applying AI techniques for the early detection of cancer, which may be applicable to primary care EHRs. We included all study designs in all settings and languages. These searches were extended through a scoping review of AI-based commercial technologies. The main outcomes assessed were measures of diagnostic accuracy for cancer. RESULTS We identified 10,456 studies; 16 studies met the inclusion criteria, representing the data of 3,862,910 patients. A total of 13 studies described the initial development and testing of AI algorithms, and 3 studies described the validation of an AI algorithm in independent data sets. One study was based on prospectively collected data; only 3 studies were based on primary care data. We found no data on implementation barriers or cost-effectiveness. Risk of bias assessment highlighted a wide range of study quality. The additional scoping review of commercial AI technologies identified 21 technologies, only 1 meeting our inclusion criteria. Meta-analysis was not undertaken because of the heterogeneity of AI modalities, data set characteristics, and outcome measures. CONCLUSIONS AI techniques have been applied to EHR-type data to facilitate early diagnosis of cancer, but their use in primary care settings is still at an early stage of maturity. Further evidence is needed on their performance using primary care data, implementation barriers, and cost-effectiveness before widespread adoption into routine primary care clinical practice can be recommended.
Collapse
Affiliation(s)
- Owain T Jones
- Primary Care Unit, Department of Public Health & Primary Care, University of Cambridge, Cambridge, United Kingdom
| | - Natalia Calanzani
- Primary Care Unit, Department of Public Health & Primary Care, University of Cambridge, Cambridge, United Kingdom
| | - Smiji Saji
- Primary Care Unit, Department of Public Health & Primary Care, University of Cambridge, Cambridge, United Kingdom
| | - Stephen W Duffy
- Wolfson Institute for Preventive Medicine, Queen Mary University of London, London, United Kingdom
| | - Jon Emery
- Centre for Cancer Research and Department of General Practice, University of Melbourne, Victoria, Australia
| | - Willie Hamilton
- College of Medicine and Health, University of Exeter, Exeter, United Kingdom
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, TX, United States
| | - Niek J de Wit
- Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht, Netherlands
| | - Fiona M Walter
- Primary Care Unit, Department of Public Health & Primary Care, University of Cambridge, Cambridge, United Kingdom
| |
Collapse
|