Analysing clinical reasoning characteristics using a combined methods approach

Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial

Importance

Large language models (LLMs) have shown promise in their performance on both multiple-choice and open-ended medical reasoning examinations, but it remains unknown whether the use of such tools improves physician diagnostic reasoning.

Objective

To assess the effect of an LLM on physicians' diagnostic reasoning compared with conventional resources.

Design, Setting, and Participants

A single-blind randomized clinical trial was conducted from November 29 to December 29, 2023. Using remote video conferencing and in-person participation across multiple academic medical institutions, physicians with training in family medicine, internal medicine, or emergency medicine were recruited.

Intervention

Participants were randomized to either access the LLM in addition to conventional diagnostic resources or conventional resources only, stratified by career stage. Participants were allocated 60 minutes to review up to 6 clinical vignettes.

Main Outcomes and Measures

The primary outcome was performance on a standardized rubric of diagnostic performance based on differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic evaluation steps, validated and graded via blinded expert consensus. Secondary outcomes included time spent per case (in seconds) and final diagnosis accuracy. All analyses followed the intention-to-treat principle. A secondary exploratory analysis evaluated the standalone performance of the LLM by comparing the primary outcomes between the LLM alone group and the conventional resource group.

Results

Fifty physicians (26 attendings, 24 residents; median years in practice, 3 [IQR, 2-8]) participated virtually as well as at 1 in-person site. The median diagnostic reasoning score per case was 76% (IQR, 66%-87%) for the LLM group and 74% (IQR, 63%-84%) for the conventional resources-only group, with an adjusted difference of 2 percentage points (95% CI, -4 to 8 percentage points; P = .60). The median time spent per case for the LLM group was 519 (IQR, 371-668) seconds, compared with 565 (IQR, 456-788) seconds for the conventional resources group, with a time difference of -82 (95% CI, -195 to 31; P = .20) seconds. The LLM alone scored 16 percentage points (95% CI, 2-30 percentage points; P = .03) higher than the conventional resources group.

Conclusions and Relevance

In this trial, the availability of an LLM to physicians as a diagnostic aid did not significantly improve clinical reasoning compared with conventional resources. The LLM alone demonstrated higher performance than both physician groups, indicating the need for technology and workforce development to realize the potential of physician-artificial intelligence collaboration in clinical practice.

Trial Registration

ClinicalTrials.gov Identifier: NCT06157944.

Evaluating the Clinical Reasoning of Student Health Professionals in Placement and Simulation Settings: A Systematic Review

(1) Background: Clinical reasoning is essential to the effective practice of autonomous health professionals and is, therefore, an essential capability to develop as students. This review aimed to systematically identify the tools available to health professional educators to evaluate students' attainment of clinical reasoning capabilities in clinical placement and simulation settings. (2) Methods: A systemic review of seven databases was undertaken. Peer-reviewed, English-language publications reporting studies that developed or tested relevant tools were included. Searches included multiple terms related to clinical reasoning and health disciplines. Data regarding each tool's conceptual basis and evaluated constructs were systematically extracted and analysed. (3) Results: Most of the 61 included papers evaluated students in medical and nursing disciplines, and over half reported on the Script Concordance Test or Lasater Clinical Judgement Rubric. A number of conceptual frameworks were referenced, though many papers did not reference any framework. (4) Conclusions: Overall, key outcomes highlighted an emphasis on diagnostic reasoning, as opposed to management reasoning. Tools were predominantly aligned with individual health disciplines and with limited cross-referencing within the field. Future research into clinical reasoning evaluation tools should build on and refer to existing approaches and consider contributions across professional disciplinary divides.

Using the diagnostic thinking inventory in musculoskeletal physiotherapy: a validity and reliability study

BACKGROUND

Development of clinical reasoning is an essential aspect in musculoskeletal physiotherapy practice that is linked to better outcomes. The measurement of clinical reasoning has placed an emphasis on diagnostic reasoning using different types of examinations. The Diagnostic Thinking Inventory (DTI) is a self-assessment tool developed to measure two aspects of diagnostic reasoning: flexibility in thinking (FT) and structure in memory (SM). DTI is valid and reliable that has been used extensively in medical field.

OBJECTIVE

To investigate the validity and reliability of DTI in musculoskeletal physiotherapy practice.

METHODS

Two groups of musculoskeletal physiotherapists completed DTI. Expert musculoskeletal physiotherapists assessed face and content validity. Data from the second group of musculoskeletal physiotherapists were used to assess test-retest reliability. Internal consistency was calculated using Cronbach's alpha. Construct validity was assessed by comparing both groups. Data were analyzed using the IBM SPSS statistics 25.0 version.

RESULTS

The experts agreed that DTI measures diagnostic reasoning. For test-retest reliability, average intraclass correlation coefficient was 0.91, 0.92 and 0.90 (p < 0.001) for DTI, FT and SM scores, respectively. Cronbach's alpha was 0.909, 0.919 and 0.897 (p < 0.001) for DTI, FT and SM, respectively. The independent samples t-test demonstrated that the experts group achieved higher and statistically significant score (p < 0.001).

CONCLUSION

DTI is valid and reliable in measuring diagnostic reasoning in the context of musculoskeletal physiotherapy practice. It can be used to assess the impact of continuing education on musculoskeletal physiotherapists' diagnostic reasoning.

How to develop clinical reasoning in medical students and interns based on illness script theory: An experimental study

Background: Although theory explains the development of illness script, it does not provide answers how medical students develop scripts in their learning. To fill the knowledge gap of developing illness script in medical students and interns, this study aimed to investigate the impact of educational strategies inspired by theory in the development of illness scripts.

Methods: A total of 15 medical students and 12 interns participated in an educational intervention that included theory-driven strategies. To evaluate the impact of this intervention, clinical reasoning problem (CRP) and key features (KF) tests were used for before and after the intervention. In addition to descriptive statistics, the differences in participants’ pretest and posttest variables were tested using Wilcoxon. Significance level was set at p≤0.05 for all tests.

Results: Interns significantly recognized more KF in the posttest. However, no significant difference was found between the pretest and posttest scores in total diagnostic accuracy (5.41±1.16 vs 4.91±1.44; p=0.111) and total correct discriminating score (0.41±0.66 vs 1.41±2.06; p=0.146). Medical students produced less total key features in the posttest, indicating that they became less elaborate in their case processing. However, no significant difference was observed in common KF score (0.4 [0.25-0.78] vs 0.9 [0.6-1]; p=0.791) and discriminative key features score (0.33 [0.16-0.33] vs 0.22 [0.11-0.44]; p=0.972) in the posttest compared to the pretest.

Conclusion: This study showed that theory-driven educational strategies have an impact on illness script development specifically in interns. It is recommended that this intervention would be tested on those in higher levels of expertise (ie, residents).

Combination of different clinical reasoning tests in a national exam

INTRODUCTION

Clinical reasoning as a critical and high level of clinical competency should be acquired during medical education, and medical educators should attempt to assess this ability in medical students. Nowadays, there are several ways to evaluate medical students' clinical reasoning ability in different countries worldwide. There are some well-known clinical reasoning tests such as Key Feature (KF), Clinical Reasoning Problem (CRP), Script Concordance Test (SCT), and Comprehensive Integrative Puzzle (CIP). Each of these tests has its advantages and disadvantages. In this study, we evaluated the reliability of combination of clinical reasoning tests SCT, KF, CIP, and CRP in one national exam and the correlation between the subtest scores of these tests together with the total score of the exam.

METHODS

In this cross sectional study, a total number of 339 high ranked medical students from 60 medical schools in Iran participated in a national exam named "Medical Olympiad". The ninth Medical Olympiad was held in Shahid Beheshti University of Medical Sciences, Tehran, Iran, under the direct supervision of the Ministry of Health and Medical Education in summer 2017. The expert group designed a combination of four types of clinical reasoning tests to assess both analytical and non-analytical clinical reasoning. Mean scores of SCT, CRP, KF, and CIP were measured using descriptive statistics. Reliability was calculated for each test and the combination of tests using Cronbach's alpha. Spearman's correlation coefficient was used to evaluate the correlation between the score of each subtest and the total score. SPSS version 21 was used for data analysis and the level of significance was considered <0.05.

RESULTS

The reliability of the combination of tests was 0.815. The reliability of KF was 0.81 and 0.76, 0.80, and 0.92 for SCT, CRP, and CIP, respectively. The mean total score was 169.921±41.54 from 240. All correlations between each clinical reasoning test and total score were significant (P<0.001). The highest correlation (0.887) was seen between CIP score and total score.

CONCLUSION

The study showed that combining different clinical reasoning tests can be a reliable way of measuring this ability.

Clinical Reasoning in Physical Therapy: A Concept Analysis

BACKGROUND

Physical therapy, along with most health professions, struggles to describe clinical reasoning, despite it being a vital skill in effective patient care. This lack of a unified conceptualization of clinical reasoning leads to variable and inconsistent teaching, assessment, and research.

OBJECTIVE

The objective was to conceptualize a broad description of physical therapists' clinical reasoning grounded in the published literature and to unify understanding for future work related to teaching, assessment, and research.

DESIGN/METHODS

The design included a systematic concept analysis using Rodgers' evolutionary methodology. A concept analysis is a research methodology in which a concept's characteristics and the relation between features of the concept are clarified.

RESULTS

Based on findings in the literature, clinical reasoning in physical therapy was conceptualized as integrating cognitive, psychomotor, and affective skills. It is contextual in nature and involves both therapist and client perspectives. It is adaptive, iterative, and collaborative with the intended outcome being a biopsychosocial approach to patient/client management.

LIMITATIONS

Although a comprehensive approach was intended, it is possible that the search methods or reduction of the literature were incomplete or key sources were mistakenly excluded.

CONCLUSIONS

A description of clinical reasoning in physical therapy was conceptualized, as it currently exists in representative literature. The intent is for it to contribute to the unification of an understanding of how clinical reasoning has been conceptualized to date by practitioners, academicians, and clinical educators. Substantial work remains to further develop the concept of clinical reasoning for physical therapy, including the role of movement in our reasoning in practice.

Psychometric characteristics of Clinical Reasoning Problems (CRPs) and its correlation with routine multiple choice question (MCQ) in Cardiology department

INTRODUCTION

Clinical reasoning is one of the most important skills in the process of training a medical student to become an efficient physician. Assessment of the reasoning skills in a medical school program is important to direct students' learning. One of the tests for measuring the clinical reasoning ability is Clinical Reasoning Problems (CRPs). The major aim of this study is to measure psychometric qualities of CRPs and define correlation between this test and routine MCQ in cardiology department of Shiraz medical school.

METHODS

This study was a descriptive study conducted on total cardiology residents of Shiraz Medical School. The study population consists of 40 residents in 2014. The routine CRPs and the MCQ tests was designed based on similar objectives and were carried out simultaneously. Reliability, item difficulty, item discrimination, and correlation between each item and the total score of CRPs were all measured by Excel and SPSS software for checking psycometeric CRPs test. Furthermore, we calculated the correlation between CRPs test and MCQ test. The mean differences of CRPs test score between residents' academic year [second, third and fourth year] were also evaluated by Analysis of variances test (One Way ANOVA) using SPSS software (version 20)(α=0.05).

RESULTS

The mean and standard deviation of score in CRPs was 10.19 ±3.39 out of 20; in MCQ, it was 13.15±3.81 out of 20. Item difficulty was in the range of 0.27-0.72; item discrimination was 0.30-0.75 with question No.3 being the exception (that was 0.24). The correlation between each item and the total score of CRP was 0.26-0.87; the correlation between CRPs test and MCQ test was 0.68 (p<0.001). The reliability of the CRPs was 0.72 as calculated by using Cronbach's alpha. The mean score of CRPs was different among residents based on their academic year and this difference was statistically significant (p<0.001).

CONCLUSION

The results of this present investigation revealed that CRPs could be reliable test for measuring clinical reasoning in residents. It can be included in cardiology residency assessment programs.

Script concordance test in medical schools in Brazil: possibilities and limitations

CONTEXT AND OBJECTIVE

Routine use of the script concordance test (SCT) is not common in Brazilian universities. This study aimed to analyze application of the SCT in the medical school of a Brazilian university.

DESIGN AND SETTING

Quantitative, analytical and descriptive study in the medical school of a Brazilian university.

METHODS

A total of 159/550 students participated. The test comprised ten clinical cases within internal medicine, with five items per case, rated on a five-point Likert scale. The test was scored in accordance with a marking key that had been validated by a reference panel.

RESULTS

In the pre-clinical and clinical phases, the mean scores were 51.6% and 63.4% of the maximum possible scores, respectively. Comparison of the means of the responses among all the years showed that there were significant differences in 40% of the items. The panel marked all the possible answers in five items, while in one item, all the panelists marked a single answer. Cronbach's alpha was 0.64. The results indicated that the more senior students performed better. Construction of an SCT with discriminative questions was not easy. The low reliability index may have occurred due to: a) problems with the construction of the questions; b) limitations of the reference panel; and/or c) the scoring key.

CONCLUSION

This instrument is very difficult to construct, apply and correct. These difficulties may make application of an SCT as an assessment method unfeasible in units with limited resources.

The script concordance test for clinical reasoning: re-examining its utility and potential weakness

CONTEXT

The script concordance test (SCT) assesses clinical reasoning under conditions of uncertainty. Relatively little information exists on Z-score (standard deviation [SD]) cut-offs for distinguishing more experienced from less experienced trainees, and whether scores depend on factual knowledge. Additionally, a recent review highlighted the finding that the SCT is potentially weakened by the fact that the mere avoidance of extreme responses may greatly increase test scores.

OBJECTIVES

This study was conducted in order to elucidate the best cut-off Z-scores, to correlate SCT scores with scores on a separate medical knowledge examination (MKE), and to investigate potential solutions to the weakness of the SCT.

METHODS

An analysis of scores on pulmonary and critical care medicine tests undertaken during July and August 2013 was performed. Clinical reasoning was tested using 1-hour SCTs (Question Sets 1 or 2). Medical knowledge was tested using a 3-hour, computer-adapted, multiple-choice question examination.

RESULTS

The expert panel was composed of 16 attending physicians. The SCTs were completed by 16 fellows and 10 residents. Fourteen fellows completed the MKE. Test reliability was acceptable for both Question Sets 1 and 2 (Cronbach's alphas of 0.79 and 0.89, respectively). Z-scores of - 2.91 and - 1.76 best separated the scores of residents from those of fellows, and the scores of fellows from those of attending physicians, respectively. Scores on the SCT and MKE were poorly correlated. Simply avoiding extreme answers boosted the Z-scores of the lowest 10 scorers on both Question Sets 1 and 2 by ≥ 1 SD. Increasing the proportion of questions with extreme modal answers to 50%, and using hypothetical question sets created from Question Set 1 overcame this problem, but consensus scoring did not.

CONCLUSIONS

The SCT was able to differentiate between test subjects of varying levels of competence, and results were not associated with medical knowledge. However, the test was vulnerable to responses that intentionally avoided extreme values. Increasing the proportion of questions with extreme modal answers may attenuate the effect of candidates exploiting the test weakness related to extreme responses.

Developing a viva exam to assess clinical reasoning in pre-registration osteopathy students

BACKGROUND

Clinical reasoning (CR) is a core capability for health practitioners. Assessing CR requires a suite of tools to encompass a wide scope of contexts and cognitive abilities. The aim of this project was to develop an oral examination and grading rubric for the assessment of CR in osteopathy, trial it with senior students in three accredited university programs in Australia and New Zealand, and to evaluate its content and face validity.

METHODS

Experienced osteopathic academics developed 20 cases and a grading rubric. Thirty senior students were recruited, 10 from each university. Twelve fourth year and 18 fifth year students participated. Three members of the research team were trained and examined students at an institution different from their own. Two cases were presented to each student participant in a series of vignettes. The rubric was constructed to follow a set of examiner questions that related to each attribute of CR. Data were analysed to explore differences in examiner marking, as well as relationships between cases, institutions, and different year levels. A non-examining member of the research team acted as an observer at each location.

RESULTS

No statistical difference was found between the total and single question scores, nor for the total scores between examiners. Significant differences were found between 4th and 5th students on total score and a number of single questions. The rubric was found to be internally consistent.

CONCLUSIONS

A viva examination of clinical reasoning, trialled with senior osteopathy students, showed face and content validity. Results suggested that the viva exam may also differentiate between 4th and 5th year students' capabilities in CR. Further work is required to establish the reliability of assessment, to further refine the rubric, and to train examiners before it is implemented as a high-stakes assessment in accredited osteopathy programs.