Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial

Artificial intelligence in rheumatology: status quo and quo vadis-results of a national survey among German rheumatologists

Background

The development and potential of artificial intelligence (AI) is remarkable. Its application in all medical disciplines, including rheumatology, is attracting attention. To what extent AI is already used in clinical routine in rheumatology is unknown. In addition, the perceived barriers, potentials, and expectations regarding AI by rheumatologists have not yet been studied.

Objectives

To examine the current usage and perceived barriers and facilitators of AI, including large language models (LLMs), among rheumatologists.

Design

National, observational, non-interventional, and cross-sectional web-based study.

Methods

A web-based survey was developed by the Working Group Young Rheumatology (AGJR) of the German Society for Rheumatology. The survey was distributed at the Congress of the German Society for Rheumatology and via social media, QR code, and email from August 30 until November 4, 2023.

Results

Responses from 172 rheumatologists (55% female; mean age 43 years) were analyzed. The majority stated that they did not previously use AI (73%) in their daily practice. Eighty-eight percent of rheumatologists rated their AI knowledge as low to intermediate and 84% would welcome dedicated training on LLMs. The majority of rheumatologists anticipated AI implementation to improve patient care (60%) and reduce daily workload (62%). Especially for diagnosis (73%), writing medical reports (70%), and data analysis (70%), rheumatologists reported a potential positive benefit of AI. Main AI concerns addressed the responsibility for medical decisions (64%) and data security (58%).

Conclusion

Overall, the results indicate that rheumatologists currently have little AI knowledge and make very little use of AI in clinical routine. However, the majority of rheumatologists anticipate positive AI effects and would welcome increased AI implementation and dedicated training programs.

A scoping review of triage approaches for the referral of patients with suspected inflammatory arthritis, from primary to rheumatology care

We aimed to (1) identify existing triage approaches for referral of patients with suspected inflammatory arthritis (IA) from primary care physicians (PCP) to rheumatologists, (2) describe their characteristics and methodologies for clinical use, and (3) report their level of validation for use in a publicly funded healthcare system. The comprehensive search strategy of multiple databases up to October 2023 identified relevant literature and focussed on approaches applied at the PCP-Rheumatologist referral stage. Primary, quantitative studies, reported in English were included. Triage approaches were grouped into patient conditions as defined by the authors of the reports, including IA, its subtypes and combinations. 13952 records were identified, 425 full text reviewed and 55 reports of 53 unique studies were included. Heterogeneity in disease nomenclature and study sample pretest probability was found. The number of published studies rapidly increased after 2012. Studies were mostly from Europe and North America, in IA and Axial Spondyloarthritis (AxSpa). We found tools ranging the continuum of development with those best performing, indicated by the area under the receiver operating curve (AUC) >0.8), requiring only patient-reported questions. There were AUCs for some tools reported from multiple studies, these were in the outstanding to excellent range for the Early IA Questionnaire (EIAQ) (0.88 to 0.92), acceptable for the Case Finding AxSpa (CaFaSpa) (0.70 to 0.75), and poor to outstanding for the Psoriasis Epidemiology Screening Tool (PEST) (0.61 to 0.91). Given the clinical urgency to improve rheumatology referrals and considering the good.

Patient-Reported Outcomes of Depression and Fibromyalgia Symptoms Do Not Predict Non-Inflammatory versus Inflammatory Diagnoses at Initial Rheumatology Consultation

OBJECTIVE

The objective of this study was to assess the potential value of patient-reported outcomes (PROs) of depression, fibromyalgia symptoms, and pain in predicting non-inflammatory vs. inflammatory diagnoses in rheumatology patients.

METHODS

This retrospective, single-center study evaluated electronic health record (EHR) data from adults who were seen for their first rheumatology consultation and subsequently received a diagnosis of an inflammatory (e.g., rheumatoid arthritis or spondyloarthritis) or non-inflammatory (e.g., osteoarthritis or fibromyalgia) condition. The PROs evaluated included depressive symptoms (Patient Health Questionnaire-2 [PHQ-2]), fibromyalgia symptom severity (FM SS), and pain.

RESULTS

A total of 3669 patients were evaluated, including patients with (n = 984; 26.82%) and without (n = 2685; 73.18%) inflammatory rheumatologic disease, of whom 141 (3.8%) had fibromyalgia. The non-inflammatory subgroup reported higher FM SS scores, and the inflammatory subgroup had higher pain and inflammatory markers. Bivariate models based on PHQ-2 and FM SS had a very low specificity (0.3%) for predicting non-inflammatory conditions, resulting in the misclassification of >99% of inflammatory cases. Adding pain, inflammatory markers, and other relevant EHR variables increased specificity but still resulted in a high level of misclassification.

CONCLUSIONS

The PROs evaluated in this study are not suitable for predicting non-inflammatory vs. inflammatory rheumatologic disease, even when combined with other EHR variables.

Comparison of Two Symptom Checkers (Ada and Symptoma) in the Emergency Department: Randomized, Crossover, Head-to-Head, Double-Blinded Study

BACKGROUND

Emergency departments (EDs) are frequently overcrowded and increasingly used by nonurgent patients. Symptom checkers (SCs) offer on-demand access to disease suggestions and recommended actions, potentially improving overall patient flow. Contrary to the increasing use of SCs, there is a lack of supporting evidence based on direct patient use.

OBJECTIVE

This study aimed to compare the diagnostic accuracy, safety, usability, and acceptance of 2 SCs, Ada and Symptoma.

METHODS

A randomized, crossover, head-to-head, double-blinded study including consecutive adult patients presenting to the ED at University Hospital Erlangen. Patients completed both SCs, Ada and Symptoma. The primary outcome was the diagnostic accuracy of SCs. In total, 6 blinded independent expert raters classified diagnostic concordance of SC suggestions with the final discharge diagnosis as (1) identical, (2) plausible, or (3) diagnostically different. SC suggestions per patient were additionally classified as safe or potentially life-threatening, and the concordance of Ada's and physician-based triage category was assessed. Secondary outcomes were SC usability (5-point Likert-scale: 1=very easy to use to 5=very difficult to use) and SC acceptance net promoter score (NPS).

RESULTS

A total of 450 patients completed the study between April and November 2021. The most common chief complaint was chest pain (160/437, 37%). The identical diagnosis was ranked first (or within the top 5 diagnoses) by Ada and Symptoma in 14% (59/437; 27%, 117/437) and 4% (16/437; 13%, 55/437) of patients, respectively. An identical or plausible diagnosis was ranked first (or within the top 5 diagnoses) by Ada and Symptoma in 58% (253/437; 75%, 329/437) and 38% (164/437; 64%, 281/437) of patients, respectively. Ada and Symptoma did not suggest potentially life-threatening diagnoses in 13% (56/437) and 14% (61/437) of patients, respectively. Ada correctly triaged, undertriaged, and overtriaged 34% (149/437), 13% (58/437), and 53% (230/437) of patients, respectively. A total of 88% (385/437) and 78% (342/437) of participants rated Ada and Symptoma as very easy or easy to use, respectively. Ada's NPS was -34 (55% [239/437] detractors; 21% [93/437] promoters) and Symptoma's NPS was -47 (63% [275/437] detractors and 16% [70/437]) promoters.

CONCLUSIONS

Ada demonstrated a higher diagnostic accuracy than Symptoma, and substantially more patients would recommend Ada and assessed Ada as easy to use. The high number of unrecognized potentially life-threatening diagnoses by both SCs and inappropriate triage advice by Ada was alarming. Overall, the trustworthiness of SC recommendations appears questionable. SC authorization should necessitate rigorous clinical evaluation studies to prevent misdiagnoses, fatal triage advice, and misuse of scarce medical resources.

TRIAL REGISTRATION

German Register of Clinical Trials DRKS00024830; https://drks.de/search/en/trial/DRKS00024830.

[Prioritized appointment allocation in new patients, what is really decisive? : Comparative analysis of manual appointment allocation with automated and AI-assisted approaches]

BACKGROUND

The timely allocation of appointments for new patients is a daily challenge in rheumatological practice, which can be supported by digital solutions. The question is to find the simplest and most effective possible method for prioritization when allocating appointments.

METHODS

Using a registration form for new patients, standardized symptoms and laboratory results were collated. After reviewing this information by a medical specialist the allocation of appointments was carried out in three categories: a) < 6 weeks, b) 6 weeks up to 3 months and c) > 3 months. The waiting time between the time of registration and the presentation appointment was calculated and compared between patients with and without a diagnosis of an inflammatory rheumatic disease (IRD). In addition a decision tree (DT), a method taken from the field of supervised learning within artificial intelligence (AI), was established and the resulting classification was compared with respect to the accuracy and calculated saving in waiting time.

RESULTS

In this study 800 appointments between 2020 and 2023 (including 555 women, 69.4%, median age 53 years, interquartile range, IQR 39-63 years) were analyzed. An IRD could be confirmed in 409 (51.1%) cases with a waiting time of 58 vs. 93 days for non-IRD cases (-38%, p < 0.01). An AI-based stratification resulted in an accuracy of 67% for IRD and a predicted saving of 19% waiting time. The accuracy increased up to 78% with a time saving for IRD cases of up to 31%, when all basic laboratory results were known. Simplified algorithms, e.g., stratification by the use of laboratory findings alone, resulted in a lower accuracy and time savings.

CONCLUSION

Manual allocation of appointments by a medical specialist is effective and significantly reduces the waiting times for patients with IRD. An automated categorization can lead to a reduction in waiting times for appointments when taking complete laboratory results and a lower sensitivity into consideration.

Prioritising Appointments by Telephone Interview: Duration from Symptom Onset to Appointment Request Predicts Likelihood of Inflammatory Rheumatic Disease

Background: This study aims to determine the rate of inflammatory rheumatic diseases (IRDs) in a cohort of initial referrals and the efficacy of prioritising appointments to the early arthritis clinic (EAC) based on symptom duration. Methods: In the present study, we used algorithm-based telephone triage to assign routine care appointments according to the time between symptom onset and request for an appointment (cut-off criterion: 6 months). This retrospective, monocentric analysis evaluated the effectiveness of our triage in identifying patients with IRDs as a function of the assigned appointment category (elective, EAC, or emergency appointment). Results: A total of 1407 patients were included in the study (34.7% male; 65.3% female). Of the 1407 patients evaluated, 361 (25.7%) presented with IRD. There were significant differences in the frequency of inflammatory diagnoses between appointment categories (p < 0.001): elective 13.8%, EAC 32.9%, and emergency 45.9%. The sample without the emergency category included a total of 1222 patients. The classification into "inflammatory" or "non-inflammatory" in this subsample was as follows: Sensitivity was 37.7%, and specificity was 92.6%. The positive predictive value (PPV) was 59.8%, and the negative predictive value (NPV) was 83.6%. Overall, 80.2% of patients were correctly assigned using the appointment category and C-reactive protein (CRP). Conclusions: The algorithm-based triage system presented here, which focuses on the time between symptom onset and request for an appointment, allows for the prioritisation of appointments in favour of patients with IRDs and thus earlier initiation of therapy.

Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial

BACKGROUND

The diagnosis of inflammatory rheumatic diseases (IRDs) is often delayed due to unspecific symptoms and a shortage of rheumatologists. Digital diagnostic decision support systems (DDSSs) have the potential to expedite diagnosis and help patients navigate the health care system more efficiently.

OBJECTIVE

The aim of this study was to assess the diagnostic accuracy of a mobile artificial intelligence (AI)-based symptom checker (Ada) and a web-based self-referral tool (Rheport) regarding IRDs.

METHODS

A prospective, multicenter, open-label, crossover randomized controlled trial was conducted with patients newly presenting to 3 rheumatology centers. Participants were randomly assigned to complete a symptom assessment using either Ada or Rheport. The primary outcome was the correct identification of IRDs by the DDSSs, defined as the presence of any IRD in the list of suggested diagnoses by Ada or achieving a prespecified threshold score with Rheport. The gold standard was the diagnosis made by rheumatologists.

RESULTS

A total of 600 patients were included, among whom 214 (35.7%) were diagnosed with an IRD. Most frequent IRD was rheumatoid arthritis with 69 (11.5%) patients. Rheport's disease suggestion and Ada's top 1 (D1) and top 5 (D5) disease suggestions demonstrated overall diagnostic accuracies of 52%, 63%, and 58%, respectively, for IRDs. Rheport showed a sensitivity of 62% and a specificity of 47% for IRDs. Ada's D1 and D5 disease suggestions showed a sensitivity of 52% and 66%, respectively, and a specificity of 68% and 54%, respectively, concerning IRDs. Ada's diagnostic accuracy regarding individual diagnoses was heterogenous, and Ada performed considerably better in identifying rheumatoid arthritis in comparison to other diagnoses (D1: 42%; D5: 64%). The Cohen κ statistic of Rheport for agreement on any rheumatic disease diagnosis with Ada D1 was 0.15 (95% CI 0.08-0.18) and with Ada D5 was 0.08 (95% CI 0.00-0.16), indicating poor agreement for the presence of any rheumatic disease between the 2 DDSSs.

CONCLUSIONS

To our knowledge, this is the largest comparative DDSS trial with actual use of DDSSs by patients. The diagnostic accuracies of both DDSSs for IRDs were not promising in this high-prevalence patient population. DDSSs may lead to a misuse of scarce health care resources. Our results underscore the need for stringent regulation and drastic improvements to ensure the safety and efficacy of DDSSs.

TRIAL REGISTRATION

German Register of Clinical Trials DRKS00017642; https://drks.de/search/en/trial/DRKS00017642.

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

BACKGROUND

The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs' performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

OBJECTIVE

This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

METHODS

Healthily's SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

RESULTS

Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

CONCLUSIONS

We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth Services

BACKGROUND

The popularity of eHealth services has surged significantly, underscoring the importance of ensuring their usability and accessibility for users with diverse needs, characteristics, and capabilities. These services can pose cognitive demands, especially for individuals who are unwell, fatigued, or experiencing distress. Additionally, numerous potentially vulnerable groups, including older adults, are susceptible to digital exclusion and may encounter cognitive limitations related to perception, attention, memory, and language comprehension. Regrettably, many studies overlook the preferences and needs of user groups likely to encounter challenges associated with these cognitive aspects.

OBJECTIVE

This study primarily aims to gain a deeper understanding of cognitive accessibility in the practical context of eHealth services. Additionally, we aimed to identify the specific challenges that vulnerable groups encounter when using eHealth services and determine key considerations for testing these services with such groups.

METHODS

As a case study of eHealth services, we conducted qualitative usability testing on 2 online symptom checkers used in Finnish public primary care. A total of 13 participants from 3 distinct groups participated in the study: older adults, individuals with mild intellectual disabilities, and nonnative Finnish speakers. The primary research methods used were the thinking-aloud method, questionnaires, and semistructured interviews.

RESULTS

We found that potentially vulnerable groups encountered numerous issues with the tested services, with similar problems observed across all 3 groups. Specifically, clarity and the use of terminology posed significant challenges. The services overwhelmed users with excessive information and choices, while the terminology consisted of numerous complex medical terms that were difficult to understand. When conducting tests with vulnerable groups, it is crucial to carefully plan the sessions to avoid being overly lengthy, as these users often require more time to complete tasks. Additionally, testing with vulnerable groups proved to be quite efficient, with results likely to benefit a wider audience as well.

CONCLUSIONS

Based on the findings of this study, it is evident that older adults, individuals with mild intellectual disability, and nonnative speakers may encounter cognitive challenges when using eHealth services, which can impede or slow down their use and make the services more difficult to navigate. In the worst-case scenario, these challenges may lead to errors in using the services. We recommend expanding the scope of testing to include a broader range of eHealth services with vulnerable groups, incorporating users with diverse characteristics and capabilities who are likely to encounter difficulties in cognitive accessibility.

"That's just Future Medicine" - a qualitative study on users' experiences of symptom checker apps

BACKGROUND

Symptom checker apps (SCAs) are mobile or online applications for lay people that usually have two main functions: symptom analysis and recommendations. SCAs ask users questions about their symptoms via a chatbot, give a list with possible causes, and provide a recommendation, such as seeing a physician. However, it is unclear whether the actual performance of a SCA corresponds to the users' experiences. This qualitative study investigates the subjective perspectives of SCA users to close the empirical gap identified in the literature and answers the following main research question: How do individuals (healthy users and patients) experience the usage of SCA, including their attitudes, expectations, motivations, and concerns regarding their SCA use?

METHODS

A qualitative interview study was chosen to clarify the relatively unknown experience of SCA use. Semi-structured qualitative interviews with SCA users were carried out by two researchers in tandem via video call. Qualitative content analysis was selected as methodology for the data analysis.

RESULTS

Fifteen interviews with SCA users were conducted and seven main categories identified: (1) Attitudes towards findings and recommendations, (2) Communication, (3) Contact with physicians, (4) Expectations (prior to use), (5) Motivations, (6) Risks, and (7) SCA-use for others.

CONCLUSIONS

The aspects identified in the analysis emphasise the specific perspective of SCA users and, at the same time, the immense scope of different experiences. Moreover, the study reveals ethical issues, such as relational aspects, that are often overlooked in debates on mHealth. Both empirical and ethical research is more needed, as the awareness of the subjective experience of those affected is an essential component in the responsible development and implementation of health apps such as SCA.

TRIAL REGISTRATION

German Clinical Trials Register (DRKS): DRKS00022465. 07/08/2020.

Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4

Pre-clinical studies suggest that large language models (i.e., ChatGPT) could be used in the diagnostic process to distinguish inflammatory rheumatic (IRD) from other diseases. We therefore aimed to assess the diagnostic accuracy of ChatGPT-4 in comparison to rheumatologists. For the analysis, the data set of Gräf et al. (2022) was used. Previous patient assessments were analyzed using ChatGPT-4 and compared to rheumatologists' assessments. ChatGPT-4 listed the correct diagnosis comparable often to rheumatologists as the top diagnosis 35% vs 39% (p = 0.30); as well as among the top 3 diagnoses, 60% vs 55%, (p = 0.38). In IRD-positive cases, ChatGPT-4 provided the top diagnosis in 71% vs 62% in the rheumatologists' analysis. Correct diagnosis was among the top 3 in 86% (ChatGPT-4) vs 74% (rheumatologists). In non-IRD cases, ChatGPT-4 provided the correct top diagnosis in 15% vs 27% in the rheumatologists' analysis. Correct diagnosis was among the top 3 in non-IRD cases in 46% of the ChatGPT-4 group vs 45% in the rheumatologists group. If only the first suggestion for diagnosis was considered, ChatGPT-4 correctly classified 58% of cases as IRD compared to 56% of the rheumatologists (p = 0.52). ChatGPT-4 showed a slightly higher accuracy for the top 3 overall diagnoses compared to rheumatologist's assessment. ChatGPT-4 was able to provide the correct differential diagnosis in a relevant number of cases and achieved better sensitivity to detect IRDs than rheumatologist, at the cost of lower specificity. The pilot results highlight the potential of this new technology as a triage tool for the diagnosis of IRD.

Stepwise asynchronous telehealth assessment of patients with suspected axial spondyloarthritis: results from a pilot study

Patients with axial spondyloarthritis (axSpA) suffer from one of the longest diagnostic delays among all rheumatic diseases. Telemedicine (TM) may reduce this diagnostic delay by providing easy access to care. Diagnostic rheumatology telehealth studies are scarce and largely limited to traditional synchronous approaches such as resource-intensive video and telephone consultations. The aim of this study was to investigate a stepwise asynchronous telemedicine-based diagnostic approach in patients with suspected axSpA. Patients with suspected axSpA completed a fully automated digital symptom assessment using two symptom checkers (SC) (bechterew-check and Ada). Secondly, a hybrid stepwise asynchronous TM approach was investigated. Three physicians and two medical students were given sequential access to SC symptom reports, laboratory and imaging results. After each step, participants had to state if axSpA was present or not (yes/no) and had to rate their perceived decision confidence. Results were compared to the final diagnosis of the treating rheumatologist. 17 (47.2%) of 36 included patients were diagnosed with axSpA. Diagnostic accuracy of bechterew-check, Ada, TM students and TM physicians was 47.2%, 58.3%, 76.4% and 88.9% respectively. Access to imaging results significantly increased sensitivity of TM-physicians (p < 0.05). Mean diagnostic confidence of false axSpA classification was not significantly lower compared to correct axSpA classification for both students and physicians. This study underpins the potential of asynchronous physician-based telemedicine for patients with suspected axSpA. Similarly, the results highlight the need for sufficient information, especially imaging results to ensure a correct diagnosis. Further studies are needed to investigate other rheumatic diseases and telediagnostic approaches.

'Better see a doctor?' Status quo of symptom checker apps in Germany: A cross-sectional survey with a mixed-methods design (CHECK.APP)

Background

Symptom checker apps (SCAs) offer symptom classification and low-threshold self-triage for laypeople. They are already in use despite their poor accuracy and concerns that they may negatively affect primary care. This study assesses the extent to which SCAs are used by medical laypeople in Germany and which software is most popular. We examined associations between satisfaction with the general practitioner (GP) and SCA use as well as the number of GP visits and SCA use. Furthermore, we assessed the reasons for intentional non-use.

Methods

We conducted a survey comprising standardised and open-ended questions. Quantitative data were weighted, and open-ended responses were examined using thematic analysis.

Results

This study included 850 participants. The SCA usage rate was 8%, and approximately 50% of SCA non-users were uninterested in trying SCAs. The most commonly used SCAs were NetDoktor and Ada. Surprisingly, SCAs were most frequently used in the age group of 51-55 years. No significant associations were found between SCA usage and satisfaction with the GP or the number of GP visits and SCA usage. Thematic analysis revealed skepticism regarding the results and recommendations of SCAs and discrepancies between users' requirements and the features of apps.

Conclusion

SCAs are still widely unknown in the German population and have been sparsely used so far. Many participants were not interested in trying SCAs, and we found no positive or negative associations of SCAs and primary care.

An introduction to digital determinants of health

In recent years, technology has been increasingly incorporated within healthcare for the provision of safe and efficient delivery of services. Although this can be attributed to the benefits that can be harnessed, digital technology has the potential to exacerbate and reinforce preexisting health disparities. Previous work has highlighted how sociodemographic, economic, and political factors affect individuals' interactions with digital health systems and are termed social determinants of health [SDOH]. But, there is a paucity of literature addressing how the intrinsic design, implementation, and use of technology interact with SDOH to influence health outcomes. Such interactions are termed digital determinants of health [DDOH]. This paper will, for the first time, propose a definition of DDOH and provide a conceptual model characterizing its influence on healthcare outcomes. Specifically, DDOH is implicit in the design of artificial intelligence systems, mobile phone applications, telemedicine, digital health literacy [DHL], and other forms of digital technology. A better appreciation of DDOH by the various stakeholders at the individual and societal levels can be channeled towards policies that are more digitally inclusive. In tandem with ongoing work to minimize the digital divide caused by existing SDOH, further work is necessary to recognize digital determinants as an important and distinct entity.

Digital Approaches for Myositis

PURPOSE OF REVIEW

This article serves as a comprehensive review, focusing on digital approaches utilized in the diagnosis, monitoring, and treatment of patients with idiopathic inflammatory myopathies (IIM). The authors critically assess the literature published in the last three years, evaluating the advancements and progress achieved in this specific domain.

RECENT FINDINGS

Remarkable strides have been made in the realm of digital diagnostic support, particularly in image analysis and clinical prediction models, showing promise in aiding the diagnosis of IIM. The field of remote patient monitoring has also witnessed significant advancements, revolutionizing the care process by offering more convenient, data-driven, and continuous monitoring for IIM patients. Various digital tools, such as wearables, video- and voice consultations, and electronic patient-reported outcomes, have been extensively explored and implemented to enhance patient care. Survey studies consistently reveal a high acceptance of telehealth services among patients. Additionally, internet-based studies have facilitated the efficient and rapid recruitment of IIM patients for research purposes. Moreover, the integration of sensors and exoskeletons has shown great potential in significantly improving the functionality and quality of life for individuals with muscle weakness caused by IIM. The integration of digital health solutions in the care of IIM patients is steadily gaining attention and exploration. Although the existing evidence is limited, it does indicate that patients can be adequately and safely supported through digital means throughout their entire healthcare journey. The growing interest in digital health technologies holds the promise of improving the overall management and outcomes for individuals with idiopathic inflammatory myopathies.

[Digital rheumatology]

Chronic inflammatory rheumatic diseases mostly run an undulating course and with unspecific symptoms. The initial clarification and timely initiation of treatment are challenging, which is additionally exacerbated by the lack of specialized physicians. Digital approaches, including artificial intelligence (AI), should be of assistance and enable an improved, personalized and needs-based treatment; however, the evidence is currently still very limited. This article provides a compact overview of the current state of digital rheumatology.

User Experience of Symptom Checkers: A Systematic Review

This review reports the user experience of symptom checkers, aiming to characterize users studied in the existing literature, identify the aspects of user experience of symptom checkers that have been studied, and offer design suggestions. Our literature search resulted in 31 publications. We found that (1) most symptom checker users are relatively young; (2) eight relevant aspects of user experience have been explored, including motivation, trust, acceptability, satisfaction, accuracy, usability, safety/security, and functionality; (3) future symptom checkers should improve their accuracy, safety, and usability. Although many facets of user experience have been explored, methodological challenges exist and some important aspects of user experience remain understudied. Further research should be conducted to explore users' needs and the context of use. More qualitative and mixed-method studies are needed to understand actual users' experiences in the future.

Population-based user-perceived experience of Rheumatic?: a novel digital symptom-checker in rheumatology

OBJECTIVE

Digital symptom-checkers (SCs) have potential to improve rheumatology triage and reduce diagnostic delays. In addition to being accurate, SCs should be user friendly and meet patient's needs. Here, we examined usability and acceptance of Rheumatic?-a new and freely available online SC (currently with >44 000 users)-in a real-world setting.

METHODS

Study participants were recruited from an ongoing prospective study, and included people ≥18 years with musculoskeletal complaints completing Rheumatic? online. The user experience survey comprised five usability and acceptability questions (11-point rating scale), and an open-ended question regarding improvement of Rheumatic? Data were analysed in R using t-test or Wilcoxon rank test (group comparisons), or linear regression (continuous variables).

RESULTS

A total of 12 712 people completed the user experience survey. The study population had a normal age distribution, with a peak at 50-59 years, and 78% women. A majority found Rheumatic? useful (78%), thought the questionnaire gave them an opportunity to describe their complaints well (76%), and would recommend Rheumatic? to friends and other patients (74%). Main shortcoming was that 36% thought there were too many questions. Still, 39% suggested more detailed questions, and only 2% suggested a reduction of questions.

CONCLUSION

Based on real-world data from the largest user evaluation study of a digital SC in rheumatology, we conclude that Rheumatic? is well accepted by women and men with rheumatic complaints, in all investigated age groups. Wide-scale adoption of Rheumatic?, therefore, seems feasible, with promising scientific and clinical implications on the horizon.

Diagnostic delay stages and pre-diagnostic treatment in patients with suspected rheumatic diseases before special care consultation: results of a multicenter-based study

Early and effective discrimination (triage) of patients with inflammatory rheumatic diseases (IRD) and other diseases (non-IRD) is essential for successful treatment and preventing damage. The aim of this study was to investigate diagnostic delays and pre-diagnosis treatment in patients newly presenting to rheumatology outpatient clinics. A total of 600 patients newly presenting to one university hospital and two non-academic centers were included. Time from onset of symptoms to rheumatology consultation "total delay" as well as medical treatment before consultation were recorded. Median time from symptom onset to rheumatologist appointment (total delay) was 30 weeks. Median time to online search, first physician appointment request and first physician appointment was 2, 4 and 5 weeks, respectively. Total delay was significantly shorter for IRD patients compared to non-IRD patients, 26 vs 35 weeks (p = 0.007). Only 17.7% of all patients and 22.9% of IRD patients had a delay of less than 12 weeks. Total delay was significantly lower in patients seen in non-academic centers compared to the university center, 20 vs 50 weeks (p < 0.0001). 32.2% of IRD patients received medical treatment that eased their symptoms prior to the rheumatology appointment. These findings highlight the persistent diagnostic delays in rheumatology; however, they also suggest that current triage strategies effectively lead to earlier appointments for IRD patients. Improvement of triage methods and pre-diagnosis treatment could decrease overall burden of disease in IRD patients.

How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Objective

To evaluate the ability of case vignettes to assess the performance of symptom checker applications and to suggest refinements to the methodology used in case vignette-based audit studies.

Methods

We re-analyzed the publicly available data of two prominent case vignette-based symptom checker audit studies by calculating common metrics of test theory. Furthermore, we developed a new metric, the Capability Comparison Score (CCS), which compares symptom checker capability while controlling for the difficulty of the set of cases each symptom checker evaluated. We then scrutinized whether applying test theory and the CCS altered the performance ranking of the investigated symptom checkers.

Results

In both studies, most symptom checkers changed their rank order when adjusting the triage capability for item difficulty (ID) with the CCS. The previously reported triage accuracies commonly overestimated the capability of symptom checkers because they did not account for the fact that symptom checkers tend to selectively appraise easier cases (i.e., with high ID values). Also, many case vignettes in both studies showed insufficient (very low and even negative) values of item-total correlation (ITC), suggesting that individual items or the composition of item sets are of low quality.

Conclusions

A test-theoretic perspective helps identify previously undetected threats to the validity of case vignette-based symptom checker assessments and provides guidance and specific metrics to improve the quality of case vignettes, in particular by controlling for the difficulty of the vignettes an app was (not) able to evaluate correctly. Such measures might prove more meaningful than accuracy alone for the competitive assessment of symptom checkers. Our approach helps elaborate and standardize the methodology used for appraising symptom checker capability, which, ultimately, may yield more reliable results.

At-home blood self-sampling in rheumatology: a qualitative study with patients and health care professionals

BACKGROUND

The goal of the study was to investigate patients' with systemic rheumatic diseases and healthcare professionals' experiences and preferences regarding self-sampling of capillary blood in rheumatology care.

METHODS

Patients performed a supervised and consecutive unsupervised capillary blood self-collection using an upper arm based device. Subsequently, patients (n = 15) and their attending health care professionals (n = 5) participated in an explorative, qualitative study using problem-centered, telephone interviews. Interview data were analyzed using structured qualitative content analysis.

RESULTS

Interviewed patients reported easy application and high usability. Patients and health care professionals alike reported time and cost savings, increased independence and flexibility, improved monitoring and reduction of risk of infection during Covid-19 as benefits. Reported drawbacks include limited blood volume, limited usability in case of functional restrictions, and environmental concerns. Older, immobile patients with long journeys to traditional blood collection sites and young patients with little time to spare for traditional blood collection appointments could be user groups, likely to benefit from self-sampling services.

CONCLUSIONS

At-home blood self-sampling could effectively complement current rheumatology telehealth care. Appropriateness and value of this service needs to be carefully discussed with patients on an individual basis.

TRIAL REGISTRATION

WHO International Clinical Trials Registry: DRKS00024925. Registered on 15/04/2021.

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

The use of patient-facing online symptom checkers (OSCs) has expanded in recent years, but their accuracy, safety, and impact on patient behaviors and health care systems remain unclear. The lack of a standardized process of clinical evaluation has resulted in significant variation in approaches to OSC validation and evaluation. The aim of this paper is to characterize a set of congruent requirements for a standardized vignette-based clinical evaluation process of OSCs. Discrepancies in the findings of comparative studies to date suggest that different steps in OSC evaluation methodology can significantly influence outcomes. A standardized process with a clear specification for vignette-based clinical evaluation is urgently needed to guide developers and facilitate the objective comparison of OSCs. We propose 15 recommendation requirements for an OSC evaluation standard. A third-party evaluation process and protocols for prospective real-world evidence studies should also be prioritized to quality assure OSC assessment.

One year of digital health applications (DiGA) in Germany - Rheumatologists' perspectives

Background

Based on given legislation the German approach to digital health applications (DiGA) allows reimbursed prescription of approved therapeutic software products since October 2020. For the first time, we evaluated DiGA-related acceptance, usage, and level of knowledge among members of the German Society for Rheumatology (DGRh) 1 year after its legal implementation.

Materials and methods

An anonymous cross-sectional online survey, initially designed by the health innovation hub (think tank and sparring partner of the German Federal Ministry of Health) and the German Pain Society was adapted to the field of rheumatology. The survey was promoted by DGRh newsletters and Twitter-posts. Ethical approval was obtained.

Results

In total, 75 valid response-sets. 80% reported to care ≥ 70% of their working time for patients with rheumatic diseases. Most were working in outpatient clinics/offices (54%) and older than 40 years (84%). Gender distribution was balanced (50%). 70% knew the possibility to prescribe DiGA. Most were informed of this for the first time via trade press (63%), and only 8% via the scientific/professional society. 46% expect information on DiGA from the scientific societies/medical chambers (35%) but rarely from the manufacturer (10%) and the responsible ministry (4%). Respondents would like to be informed about DiGA via continuing education events (face-to-face 76%, online 84%), trade press (86%), and manufacturers' test-accounts (64%). Only 7% have already prescribed a DiGA, 46% planned to do so, and 47% did not intend DiGA prescriptions. Relevant aspects for prescription are provided. 86% believe that using DiGA/medical apps would at least partially be feasible and understandable to their patients. 83% thought that data collected by the patients using DiGA or other digital solutions could at least partially influence health care positively. 51% appreciated to get DiGA data directly into their patient documentation system/electronic health record (EHR) and 29% into patient-owned EHR.

Conclusion

Digital health applications awareness was high whereas prescription rate was low. Mostly, physician-desired aspects for DiGA prescriptions were proven efficacy and efficiency for physicians and patients, risk of adverse effects and health care costs were less important. Evaluation of patients' barriers and needs is warranted. Our results might contribute to the implementation and dissemination of DiGA.

Examining the impact of a symptom assessment application on patient-physician interaction among self-referred walk-in patients in the emergency department (AKUSYM): study protocol for a multi-center, randomized controlled, parallel-group superiority trial

Background

Due to the increasing use of online health information, symptom checkers have been developed to provide an individualized assessment of health complaints and provide potential diagnoses and an urgency estimation. It is assumed that they support patient empowerment and have a positive impact on patient-physician interaction and satisfaction with care. Particularly in the emergency department (ED), symptom checkers could be integrated to bridge waiting times in the ED, and patients as well as physicians could take advantage of potential positive effects. Our study therefore aims to assess the impact of symptom assessment application (SAA) usage compared to no SAA usage on the patient-physician interaction in self-referred walk-in patients in the ED population.

Methods

In this multi-center, 1:1 randomized, controlled, parallel-group superiority trial, 440 self-referred adult walk-in patients with a non-urgent triage category will be recruited in three EDs in Berlin. Eligible participants in the intervention group will use a SAA directly after initial triage. The control group receives standard care without using a SAA. The primary endpoint is patients’ satisfaction with the patient-physician interaction assessed by the Patient Satisfaction Questionnaire.

Discussion

The results of this trial could influence the implementation of SAA into acute care to improve the satisfaction with the patient-physician interaction.

Trial registration

German Clinical Trials Registry DRKS00028598. Registered on 25.03.2022

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Background

Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions.

Objective

The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED).

Methods

We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model.

Results

A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low.

Conclusions

This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy

Symptom checkers are increasingly used to assess new symptoms and navigate the health care system. The aim of this study was to compare the accuracy of an artificial intelligence (AI)-based symptom checker (Ada) and physicians regarding the presence/absence of an inflammatory rheumatic disease (IRD). In this survey study, German-speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for 20 different real-world patient vignettes, which included only basic health and symptom-related medical history. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final rheumatologists’ diagnosis, reported on the discharge summary report. A total of 132 vignettes were completed by 33 physicians (mean rheumatology working experience 8.8 (SD 7.1) years). Ada’s diagnostic accuracy (IRD) was significantly higher compared to physicians (70 vs 54%, p = 0.002) according to top diagnosis. Ada listed the correct diagnosis more often compared to physicians (54 vs 32%, p < 0.001) as top diagnosis as well as among the top 3 diagnoses (59 vs 42%, p < 0.001). Work experience was not related to suggesting the correct diagnosis or IRD status. Confined to basic health and symptom-related medical history, the diagnostic accuracy of physicians was lower compared to an AI-based symptom checker. These results highlight the potential of using symptom checkers early during the patient journey and importance of access to complete and sufficient patient information to establish a correct diagnosis.

Remote self-collection of capillary blood using upper arm devices for autoantibody analysis in patients with immune-mediated inflammatory rheumatic diseases

Objectives

To evaluate the feasibility, accuracy, usability and acceptability of two upper arm self-sampling devices for measurement of autoantibodies and C reactive protein (CRP) levels in patients with immune-mediated rheumatic diseases (IMRDs).

Methods

70 consecutive patients with IMRD with previously documented autoantibodies were assigned to supervised and unsupervised self-collection of capillary blood with the Tasso+ or TAP II device. Interchangeability of 17 biomarkers with standard venesection was assessed by: concordance, correlation, paired sample hypothesis testing and Bland-Altman plots. Patients completed an evaluation questionnaire, including the System Usability Scale (SUS) and Net Promoter Score (NPS).

Results

While 80.0% and 77.0% were able to safely and successfully collect capillary blood using the Tasso+ and TAP II within the first attempt, 69 of 70 (98.6%) patients were successful in collecting capillary blood within two attempts. Concordance between venous and capillary samples was high; 94.7% and 99.5% for positive and negative samples, respectively. For connective tissue disease screen, anti-Ro52 and anti-proteinase 3 autoantibody levels, no significant differences were observed. Self-sampling was less painful than standard venesection for the majority of patients (Tasso+: 71%; TAP II: 63%). Both devices were well accepted (NPS; both: +28%), usability was perceived as excellent (SUS; Tasso+: 88.6 of 100; TAP II: 86.0 of 100) and 48.6 %/62.9% of patients would prefer to use the Tasso+/TAP II, respectively, instead of a traditional venous blood collection.

Conclusions

Remote self-collection of capillary blood using upper arm-based devices for autoantibody and CRP analysis in patients with autoimmune rheumatic diseases is feasible, accurate and well accepted among patients.

Trial registration number

WHO International Clinical Trials Registry (DRKS00024925).

The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review

Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19–37.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8–90.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.

Machine learning-based improvement of an online rheumatology referral and triage system

Introduction

Rheport is an online rheumatology referral system allowing automatic appointment triaging of new rheumatology patient referrals according to the respective probability of an inflammatory rheumatic disease (IRD). Previous research reported that Rheport was well accepted among IRD patients. Its accuracy was, however, limited, currently being based on an expert-based weighted sum score. This study aimed to evaluate whether machine learning (ML) models could improve this limited accuracy.

Materials and methods

Data from a national rheumatology registry (RHADAR) was used to train and test nine different ML models to correctly classify IRD patients. Diagnostic performance was compared of ML models and the current algorithm was compared using the area under the receiver operating curve (AUROC). Feature importance was investigated using shapley additive explanation (SHAP).

Results

A complete data set of 2265 patients was used to train and test ML models. 30.5% of patients were diagnosed with an IRD, 69.3% were female. The diagnostic accuracy of the current Rheport algorithm (AUROC of 0.534) could be improved with all ML models, (AUROC ranging between 0.630 and 0.737). Targeting a sensitivity of 90%, the logistic regression model could double current specificity (17% vs. 33%). Finger joint pain, inflammatory marker levels, psoriasis, symptom duration and female sex were the five most important features of the best performing logistic regression model for IRD classification.

Conclusion

In summary, ML could improve the accuracy of a currently used rheumatology online referral system. Including further laboratory parameters and enabling individual feature importance adaption could increase accuracy and lead to broader usage.

[Digitally supported rheumatological screening consultation : How useful is a questionnaire scoring system (RhePort)?]

Regarding scarce capacities an early detection consultation (EDC) was established to discriminate patients in an outpatient setting with inflammatory from non-inflammatory rheumatic diseases. A total of 500 patients suspected of having a rheumatic disease received an appointment within 2 weeks. They were interviewed with the help of a digital questionnaire (RhePort), briefly physically examined followed by a determination of CRP. The questionnaire answers were scored using an algorithm within RhePort (from 0 = non-inflammatory to 4 = highly probably inflammatory). Likewise, after completion of the EDC, the rheumatologists scored the overall assessment. The RhePort score and EDC score were compared with the "true" diagnosis made in a detailed second examination after an average of 10 weeks. In 490 evaluable patients 133 inflammatory (27%) and 357 noninflammatory rheumatic diseases (73%) were diagnosed. A classification based solely on the RhePort questionnaire (score > 1) identified 103 out of 129 as inflammatory (sens. 80%) and 125 out of 355 as non-inflammatory (spec. 35%) resulting in an AUC of 0.62 after ROC analysis. With a score > 1, the rheumatological assessment after EDC classified 130 out of 133 patients as inflammatory (sensitivity 98%) and 261 out of 357 as non-inflammatory (specificity 73%). The combined EDC can decisively increase the sensitivity and specificity compared to an "automated" survey by means of a digital questionnaire alone. In addition to the early identification and treatment of inflammatory patients, rapid identification of patients who are not in need of rheumatological treatment can create capacities for care.

Triage Errors in Primary and Pre-Primary Care

Triage errors are a major concern in health care due to resulting harmful delays in treatments or inappropriate allocation of resources. With the increasing popularity of digital symptom checkers in pre–primary care settings, and amid claims that artificial intelligence outperforms doctors, the accuracy of triage by digital symptom checkers is ever more scrutinized. This paper examines the context and challenges of triage in primary care, pre–primary care, and emergency care, as well as reviews existing evidence on the prevalence of triage errors in all three settings. Implications for development, research, and practice are highlighted, and recommendations are made on how digital symptom checkers should be best positioned.

Accuracy and tolerability of self-sampling of capillary blood for analysis of inflammation and autoantibodies in rheumatoid arthritis patients-results from a randomized controlled trial

BACKGROUND

Rheumatoid arthritis (RA) requires early diagnosis and tight surveillance of disease activity. Remote self-collection of blood for the analysis of inflammation markers and autoantibodies could improve the monitoring of RA and facilitate the identification of individuals at-risk for RA.

OBJECTIVE

Randomized, controlled trial to evaluate the accuracy, feasibility, and acceptability of an upper arm self-sampling device (UA) and finger prick-test (FP) to measure capillary blood from RA patients for C-reactive protein (CRP) levels and the presence of IgM rheumatoid factor (RF IgM) and anti-cyclic citrullinated protein antibodies (anti-CCP IgG).

METHODS

RA patients were randomly assigned in a 1:1 ratio to self-collection of capillary blood via UA or FP. Venous blood sampling (VBS) was performed as a gold standard in both groups to assess the concordance of CRP levels as well as RF IgM and CCP IgG. General acceptability and pain during sampling were measured and compared between UA, FP, and VBS. The number of attempts for successful sampling, requests for assistance, volume, and duration of sample collection were also assessed.

RESULTS

Fifty seropositive RA patients were included. 49/50 (98%) patients were able to successfully collect capillary blood. The overall agreement between capillary and venous analyses for CRP (0.992), CCP IgG (0.984), and RF IgM (0.994) were good. In both groups, 4/25 (16%) needed a second attempt and 8/25 (32%) in the UA and 7/25 (28%) in the FP group requested assistance. Mean pain scores for capillary self-sampling (1.7/10 ± 1.1 (UA) and 1.9/10 ± 1.9 (FP)) were significantly lower on a numeric rating scale compared to venous blood collection (UA: 2.8/10 ± 1.7; FP: 2.1 ± 2.0) (p=0.003). UA patients were more likely to promote the use of capillary blood sampling (net promoter score: +28% vs. -20% for FP) and were more willing to perform blood collection at home (60% vs. 32% for FP).

CONCLUSIONS

These data show that self-sampling is accurate and feasible within one attempt by the majority of patients without assistance, allowing tight monitoring of RA disease activity as well as identifying individuals at-risk for RA. RA patients seem to prefer upper arm-based self-sampling to traditional finger pricking.

TRIAL REGISTRATION

DRKS.de Identifier: DRKS00023526 . Registered on November 6, 2020.

Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation

BACKGROUND

Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.

OBJECTIVE

This study aims to revisit the landmark index study to investigate whether and how symptom checkers' capabilities have evolved since 2015 and how they currently compare with laypersons' stand-alone triage appraisal.

METHODS

In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons' triage capability.

RESULTS

We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.

CONCLUSIONS

Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Patient's Perception of Digital Symptom Assessment Technologies in Rheumatology: Results From a Multicentre Study

Introduction

An increasing number of digital tools, including dedicated diagnostic decision support systems (DDSS) exist to better assess new symptoms and understand when and where to seek medical care. The aim of this study was to evaluate patient's previous online assessment experiences and to compare the acceptability, usability, usefulness and potential impact of artificial intelligence (AI)-based symptom checker (Ada) and an online questionnaire-based self-referral tool (Rheport).

Materials and Methods

Patients newly presenting to three German secondary rheumatology outpatient clinics were randomly assigned in a 1:1 ratio to complete consecutively Ada or Rheport in a prospective non-blinded multicentre controlled crossover randomized trial. DDSS completion time was recorded by local study personnel and perceptions on DDSS and previous online assessment were collected through a self-completed study questionnaire, including usability measured with the validated System Usability Scale (SUS).

Results

600 patients (median age 52 years, 418 women) were included. 277/600 (46.2%) of patients used an online search engine prior to the appointment. The median time patients spent assessing symptoms was 180, 7, and 8 min, respectively using online using search engines, Ada and Rheport. 111/275 (40.4%), 266/600 (44.3%) and 395/600 (65.8%) of patients rated the respective symptom assessment as very helpful or helpful, using online search engines, Ada and Rheport, respectively. Usability of both diagnostic decision support systems (DDSS) was “good” with a significantly higher mean SUS score (SD) of Rheport 77.1/100 (16.0) compared to Ada 74.4/100 (16.8), (p < 0.0001). In male patients, usability of Rheport was rated higher than Ada (p = 0.02) and the usability rating of older (52 years ≥) patients of both DDSS was lower than in younger participants (p = 0.005). Both effects were independent of each other. 440/600 (73.3%) and 475/600 (79.2%) of the patients would recommend Ada and Rheport to friends and other patients, respectively.

Conclusion

In summary, patients increasingly assess their symptoms independently online, however only a minority used dedicated symptom assessment websites or DDSS. DDSS, such as Ada an Rheport are easy to use, well accepted among patients with musculoskeletal complaints and could replace online search engines for patient symptom assessment, potentially saving time and increasing helpfulness.

Rheumatic?-A Digital Diagnostic Decision Support Tool for Individuals Suspecting Rheumatic Diseases: A Multicenter Pilot Validation Study

Introduction

Digital diagnostic decision support tools promise to accelerate diagnosis and increase health care efficiency in rheumatology. Rheumatic? is an online tool developed by specialists in rheumatology and general medicine together with patients and patient organizations. It calculates a risk score for several rheumatic diseases. We ran a pilot study retrospectively testing Rheumatic? for its ability to differentiate symptoms from existing or emerging immune-mediated rheumatic diseases from other rheumatic and musculoskeletal complaints and disorders in patients visiting rheumatology clinics.

Materials and Methods

The performance of Rheumatic? was tested using in three university rheumatology centers: (A) patients at Risk for RA (Karolinska Institutet, n = 50 individuals with musculoskeletal complaints and anti-citrullinated protein antibody positivity) (B) patients with early joint swelling [dataset B (Erlangen) n = 52]. (C) Patients with early arthritis where the clinician considered it likely to be of auto-immune origin [dataset C (Leiden) n = 73]. In dataset A we tested whether Rheumatic? could predict the development of arthritis. In dataset B and C we tested whether Rheumatic? could predict the development of an immune-mediated rheumatic diseases. We examined the discriminative power of the total score with the Wilcoxon rank test and the area-under-the-receiver-operating-characteristic curve (AUC-ROC). Next, we calculated the test characteristics for these patients passing the first or second expert-based Rheumatic? scoring threshold.

Results

The total test scores differentiated between: (A) Individuals developing arthritis or not, median 245 vs. 163, P < 0.0001, AUC-ROC = 75.3; (B) patients with an immune-mediated arthritic disease or not median 191 vs. 107, P < 0.0001, AUC-ROC = 79.0; but less patients with an immune-mediated arthritic disease or not amongst those where the clinician already considered an immune mediated disease most likely (median 262 vs. 212, P < 0.0001, AUC-ROC = 53.6). Threshold-1 (advising to visit primary care doctor) was highly specific in dataset A and B (0.72, 0.87, and 0.23, respectively) and sensitive (0.67, 0.61, and 0.67). Threshold-2 (advising to visit rheumatologic care) was very specific in all three centers but not very sensitive: specificity of 1.0, 0.96, and 0.91, sensitivity 0.05, 0.07, 0.14 in dataset A, B, and C, respectively.

Conclusion

Rheumatic? is a web-based patient-centered multilingual diagnostic tool capable of differentiating immune-mediated rheumatic conditions from other musculoskeletal problems. The current scoring system needs to be further optimized.

Future challenges in rheumatology - is telemedicine the solution?

The COVID-19 pandemic has become an unprecedented facilitator of rapid telehealth expansion within rheumatology. Due to demographic shifts and workforce shortages in the future, new models of rheumatology care will be expected to emerge, with a growing footprint of telehealth interventions. Telehealth is already being used to monitor patients with rheumatic diseases and initial studies show good results in terms of safety and disease progression. It is being used as a tool for appointment prioritization and triage, and there is good evidence for using telehealth in rehabilitation, patient education and self-management interventions. Electronic patient-reported outcomes (ePROs) offer a number of long-term benefits and opportunities, and a routine collection of ePROs also facilitates epidemiological research that can inform future healthcare delivery. Telehealth solutions should be developed in close collaboration with all stakeholders, and the option of a telehealth visit must not deprive patients of the possibility to make use of a conventional 'face-to-face' visit. Future studies should especially focus on optimal models for rheumatology healthcare delivery to patients living in remote areas who are unable to use or access computer technology, and other patient groups at risk for disparity due to technical inequity and lack of knowledge.

[Telemedical care and IT-based systems in rheumatology]

BACKGROUND

The coronavirus disease 2019 (COVID-19) pandemic and also the ever-increasing demands on the healthcare system, have led to a focus on the further development of telemedical services in rheumatology.

OBJECTIVE

What is the evidence for telemedical services in rheumatology?

MATERIAL AND METHODS

Narrative review of existing literature on telemedicine in rheumatology.

RESULTS

Electronic patient reported outcomes (ePROs) can be determined by patients from their home and sent electronically to the rheumatologist. In future, ePROs may help with the decision whether a patient needs to attend the clinic for a visit or the visit can be rescheduled due to remission and well-being. Telemedicine has already been used for well-controlled patients with rheumatic diseases with good results in terms of safety and disease activity compared to conventional face-to-face visits. Telemedicine represents an interesting tool for appointment prioritization and triaging, while automated algorithm-based applications are currently too imprecise for routine clinical use. The role of smartphone applications in the care of patients with rheumatic diseases is still unclear.

DISCUSSION

Telemedicine represents an interesting option for certain patient populations with rheumatic diseases. Apart from research on the effectiveness and safety of telemedical interventions, decision makers need to set clear rules on how telemedicine should be used to provide the best possible care for the individual patient.

Understanding the factors that affect the appropriateness of rheumatology referrals

BACKGROUND

Reducing inappropriate referrals to specialists is a challenge for the healthcare system as it seeks to transition from volume to value-based healthcare. Given the projection of a severe shortage of rheumatologists in the near future, innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. Efforts to increase appropriate utilization through reductions in capacity may have the unintended consequence of reducing appropriate care as well. This highlights the challenges in increasing the appropriate use of high cost services as the health system transitions to value based care. The objective of this study was to analyze factors affecting appropriateness of rheumatology services.

METHODS

This was a cross-sectional study of patients receiving Rheumatology services between November 2013 and October 2019. We used a proxy for "appropriateness": whether or not there was any follow-up care after the first appointment. Results from regression analysis and physicians' chart reviews were compared using an inter-rater reliability measure (kappa). Data was drawn from the EHR 2013-2019.

RESULTS

We found that inappropriate referrals increased 14.3% when a new rheumatologist was hired, which increased to 14.8% after wash-out period of 6 months; 15.7% after 12 months; 15.5% after 18 months and 16.7% after 18 months. Other factors influencing appropriateness of referrals included severity of disease, gender and insurance type, but not specialty of referring provider.

CONCLUSIONS

Given the projection of a severe shortage of rheumatologists in the near future, innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. Innovative strategies to decrease demand for rheumatology services may prove more fruitful than increasing the supply of rheumatologists. These findings may apply to other specialties as well. This study is relevant for health care systems that are implementing value-based payment models aimed at reducing inappropriate care.

[Digital diagnostic support in rheumatology]

Only the correct diagnosis enables an effective treatment of rheumatic diseases. Digitalization has already significantly accelerated and simplified our everyday life. An increasing number of digital options are available to patients and medical personnel in rheumatology to accelerate and improve the diagnosis. This work gives an overview of current developments and tools for patients and rheumatologists, regarding digital diagnostic support in rheumatology.

[Digitalization in rheumatological practice]

Digitalization in medicine is of major interest since the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic. This article tries to present the induced changes and technical solutions with respect to the different parts in the patient journey. Symptom checkers, new health applications, digital appointment management etc. are described. Apart from the technical and digital possibilities, the changes in the quality of communication additionally have to be mentioned. There is an urgent need for further technical standardization including the interfaces. In many cases further studies must confirm the equivalence of digital applications in comparison to analogue techniques.

Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students

BACKGROUND

An increasing number of diagnostic decision support systems (DDSS) exist to support patients and physicians in establishing the correct diagnosis as early as possible. However, little evidence exists that supports the effectiveness of these DDSS. The objectives were to compare the diagnostic accuracy of medical students, with and without the use of a DDSS, and the diagnostic accuracy of the DDSS system itself, regarding the typical rheumatic diseases and to analyze the user experience.

METHODS

A total of 102 medical students were openly recruited from a university hospital and randomized (unblinded) to a control group (CG) and an intervention group (IG) that used a DDSS (Ada - Your Health Guide) to create an ordered diagnostic hypotheses list for three rheumatic case vignettes. Diagnostic accuracy, measured as the presence of the correct diagnosis first or at all on the hypothesis list, was the main outcome measure and evaluated for CG, IG, and DDSS.

RESULTS

The correct diagnosis was ranked first (or was present at all) in CG, IG, and DDSS in 37% (40%), 47% (55%), and 29% (43%) for the first case; 87% (94%), 84% (100%), and 51% (98%) in the second case; and 35% (59%), 20% (51%), and 4% (51%) in the third case, respectively. No significant benefit of using the DDDS could be observed. In a substantial number of situations, the mean probabilities reported by the DDSS for incorrect diagnoses were actually higher than for correct diagnoses, and students accepted false DDSS diagnostic suggestions. DDSS symptom entry greatly varied and was often incomplete or false. No significant correlation between the number of symptoms extracted and diagnostic accuracy was seen. It took on average 7 min longer to solve a case using the DDSS. In IG, 61% of students compared to 90% in CG stated that they could imagine using the DDSS in their future clinical work life.

CONCLUSIONS

The diagnostic accuracy of medical students was superior to the DDSS, and its usage did not significantly improve students' diagnostic accuracy. DDSS usage was time-consuming and may be misleading due to prompting wrong diagnoses and probabilities.

TRIAL REGISTRATION

DRKS.de, DRKS00024433 . Retrospectively registered on February 5, 2021.

Digital Approaches for a Reliable Early Diagnosis of Psoriatic Arthritis

Psoriatic arthritis (PsA) is a chronic inflammatory disease that develops in up to 30% of patients with psoriasis. In the vast majority of cases, cutaneous symptoms precede musculoskeletal complaints. Progression from psoriasis to PsA is characterized by subclinical synovio-entheseal inflammation and often non-specific musculoskeletal symptoms that are frequently unreported or overlooked. With the development of increasingly effective therapies and a broad drug armamentarium, prevention of arthritis development through careful clinical monitoring has become priority. Identifying high-risk psoriasis patients before PsA onset would ensure early diagnosis, increased treatment efficacy, and ultimately better outcomes; ideally, PsA development could even be averted. However, the current model of care for PsA offers only limited possibilities of early intervention. This is attributable to the large pool of patients to be monitored and the limited resources of the health care system in comparison. The use of digital technologies for health (eHealth) could help close this gap in care by enabling faster, more targeted and more streamlined access to rheumatological care for patients with psoriasis. eHealth solutions particularly include telemedicine, mobile technologies, and symptom checkers. Telemedicine enables rheumatological visits and consultations at a distance while mobile technologies can improve monitoring by allowing patients to self-report symptoms and disease-related parameters continuously. Symptom checkers have the potential to direct patients to medical attention at an earlier point of their disease and therefore minimizing diagnostic delay. Overall, these interventions could lead to earlier diagnoses of arthritis, improved monitoring, and better disease control while simultaneously increasing the capacity of referral centers.