Human- Versus Machine Learning-Based Triage Using Digitalized Patient Histories in Primary Care: Comparative Study

Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review

INTRODUCTION

Patient-operated digital triage systems with AI components are becoming increasingly common. However, previous reviews have found a limited amount of research on such systems' accuracy. This systematic review of the literature aimed to identify the main challenges in determining the accuracy of patient-operated digital AI-based triage systems.

METHODS

A systematic review was designed and conducted in accordance with PRISMA guidelines in October 2021 using PubMed, Scopus and Web of Science. Articles were included if they assessed the accuracy of a patient-operated digital triage system that had an AI-component and could triage a general primary care population. Limitations and other pertinent data were extracted, synthesized and analysed. Risk of bias was not analysed as this review studied the included articles' limitations (rather than results). Results were synthesized qualitatively using a thematic analysis.

RESULTS

The search generated 76 articles and following exclusion 8 articles (6 primary articles and 2 reviews) were included in the analysis. Articles' limitations were synthesized into three groups: epistemological, ontological and methodological limitations. Limitations varied with regards to intractability and the level to which they can be addressed through methodological choices. Certain methodological limitations related to testing triage systems using vignettes can be addressed through methodological adjustments, whereas epistemological and ontological limitations require that readers of such studies appraise the studies with limitations in mind.

DISCUSSION

The reviewed literature highlights recurring limitations and challenges in studying the accuracy of patient-operated digital triage systems with AI components. Some of these challenges can be addressed through methodology whereas others are intrinsic to the area of inquiry and involve unavoidable trade-offs. Future studies should take these limitations in consideration in order to better address the current knowledge gaps in the literature.

Real-world Health Data and Precision for the Diagnosis of Acute Kidney Injury, Acute-on-Chronic Kidney Disease, and Chronic Kidney Disease: Observational Study

Background

The criteria for the diagnosis of kidney disease outlined in the Kidney Disease: Improving Global Outcomes guidelines are based on a patient’s current, historical, and baseline data. The diagnosis of acute kidney injury, chronic kidney disease, and acute-on-chronic kidney disease requires previous measurements of creatinine, back-calculation, and the interpretation of several laboratory values over a certain period. Diagnoses may be hindered by unclear definitions of the individual creatinine baseline and rough ranges of normal values that are set without adjusting for age, ethnicity, comorbidities, and treatment. The classification of correct diagnoses and sufficient staging improves coding, data quality, reimbursement, the choice of therapeutic approach, and a patient’s outcome.

Objective

In this study, we aim to apply a data-driven approach to assign diagnoses of acute, chronic, and acute-on-chronic kidney diseases with the help of a complex rule engine.

Methods

Real-time and retrospective data from the hospital’s clinical data warehouse of inpatient and outpatient cases treated between 2014 and 2019 were used. Delta serum creatinine, baseline values, and admission and discharge data were analyzed. A Kidney Disease: Improving Global Outcomes–based SQL algorithm applied specific diagnosis-based International Classification of Diseases (ICD) codes to inpatient stays. Text mining on discharge documentation was also conducted to measure the effects on diagnosis.

Results

We show that this approach yielded an increased number of diagnoses (4491 cases in 2014 vs 11,124 cases of ICD-coded kidney disease and injury in 2019) and higher precision in documentation and coding. The percentage of unspecific ICD N19-coded diagnoses of N19 codes generated dropped from 19.71% (1544/7833) in 2016 to 4.38% (416/9501) in 2019. The percentage of specific ICD N18-coded diagnoses of N19 codes generated increased from 50.1% (3924/7833) in 2016 to 62.04% (5894/9501) in 2019.

Conclusions

Our data-driven method supports the process and reliability of diagnosis and staging and improves the quality of documentation and data. Measuring patient outcomes will be the next step in this project.

Health care utilization following "digi-physical" assessment compared to physical assessment for infectious symptoms in primary care

BACKGROUND

The use of chat-based digital visits (eVisits) to assess infectious symptoms in primary care is rapidly increasing. The "digi-physical" model of care uses eVisits as the first line of assessment while assuming a certain proportion of patients will inevitably need to be further assessed through urgent physical examination within 48 h. It is unclear to what extent this approach can mitigate physical visits compared to assessing patients directly using office visits.

METHODS

This pre-COVID-19-pandemic observational study followed up "digi-physical" eVisit patients (n = 1188) compared to office visit patients (n = 599) with respiratory or urinary symptoms. Index visits occurred between March 30th 2016 and March 29th 2019. The primary outcome was subsequent physical visits to physicians within two weeks using registry data from Skåne county, Sweden (Region Skånes Vårddatabas, RSVD).

RESULTS

No significant differences in subsequent physical visits within two weeks (excluding the first 48 h) were noted following "digi-physical" care compared to office visits (179 (18.0%) vs. 102 (17.6%), P = .854). As part of the "digital-physical" concept, a significantly larger proportion of eVisit patients had a physical visit within 48 h compared to corresponding office visit patients (191 (16.1%) vs. 19 (3.2%), P < .001), with 150 (78.5%) of these eVisit patients recommended some form of follow-up by the eVisit physician.

CONCLUSIONS

Most eVisit patients (68.9%) with respiratory and urinary symptoms have no subsequent physical visits. Beyond an unavoidable portion of patients requiring urgent physical examination within 48 h, "digi-physical" management of respiratory and urinary symptoms results in comparable subsequent health care utilization compared to office visits. eVisit providers may need to optimize use of resources to minimize the proportion of patients being assessed both digitally and physically within 48 h as part of the "digi-physical" concept.

TRIAL REGISTRATION

Clinicaltrials.gov identifier: NCT03474887.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s12875-021-01618-2.

Health Equity in Artificial Intelligence and Primary Care Research: Protocol for a Scoping Review

BACKGROUND

Though artificial intelligence (AI) has the potential to augment the patient-physician relationship in primary care, bias in intelligent health care systems has the potential to differentially impact vulnerable patient populations.

OBJECTIVE

The purpose of this scoping review is to summarize the extent to which AI systems in primary care examine the inherent bias toward or against vulnerable populations and appraise how these systems have mitigated the impact of such biases during their development.

METHODS

We will conduct a search update from an existing scoping review to identify studies on AI and primary care in the following databases: Medline-OVID, Embase, CINAHL, Cochrane Library, Web of Science, Scopus, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI, and arXiv. Two screeners will independently review all abstracts, titles, and full-text articles. The team will extract data using a structured data extraction form and synthesize the results in accordance with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines.

RESULTS

This review will provide an assessment of the current state of health care equity within AI for primary care. Specifically, we will identify the degree to which vulnerable patients have been included, assess how bias is interpreted and documented, and understand the extent to which harmful biases are addressed. As of October 2020, the scoping review is in the title- and abstract-screening stage. The results are expected to be submitted for publication in fall 2021.

CONCLUSIONS

AI applications in primary care are becoming an increasingly common tool in health care delivery and in preventative care efforts for underserved populations. This scoping review would potentially show the extent to which studies on AI in primary care employ a health equity lens and take steps to mitigate bias.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/27799.

Artificial Intelligence-Based Application Provides Accurate Medical Triage Advice When Compared to Consensus Decisions of Healthcare Providers

Accurate medical triage is essential for improving patient outcomes and efficient healthcare delivery. Patients increasingly rely on artificial intelligence (AI)-based applications to access healthcare information, including medical triage advice. We assessed the accuracy of triage decisions provided by an AI-based application. We presented 50 clinical vignettes to the AI-based application, seven emergency medicine providers, and five internal medicine physicians. We compared the triage decisions of the AI-based application to those of the individual providers as well as their consensus decisions. When compared to the human clinicians’ consensus triage decisions, the AI-based application performed equal or better than individual human clinicians.

Artificial Intelligence developments in medical education: a conceptual and practical framework

This article was migrated. The article was marked as recommended. As Artificial Intelligence (AI) develops in medicine, there is a greater awareness that medical education may also benefit from AI. Many AI projects are already underway, and many more are still to come. Most medical education administrators and educators are aware of AI, but are not necessarily familiar enough with it to understand the areas of possible application in both using AI in medical education, and also in the content areas that need to be raised with their students. As such, they are at a disadvantage of not understanding the current lie of the land, and are at an even greater disadvantage of not being able to influence and guide future AI projects. This paper attempts to provide an AI conceptual and practical framework for medical education administrators and educators, so that they may have a clearer understanding of the current situation, and may be better placed to guide future AI developments to meet their needs in medical education.