A mixed-methods exploration of cognitive dispositions to respond and clinical reasoning errors with multiple choice questions

Assessing the Capability of ChatGPT, Google Bard, and Microsoft Bing in Solving Radiology Case Vignettes

Background  The field of radiology relies on accurate interpretation of medical images for effective diagnosis and patient care. Recent advancements in artificial intelligence (AI) and natural language processing have sparked interest in exploring the potential of AI models in assisting radiologists. However, limited research has been conducted to assess the performance of AI models in radiology case interpretation, particularly in comparison to human experts. Objective  This study aimed to evaluate the performance of ChatGPT, Google Bard, and Bing in solving radiology case vignettes (Fellowship of the Royal College of Radiologists 2A [FRCR2A] examination style questions) by comparing their responses to those provided by two radiology residents. Methods  A total of 120 multiple-choice questions based on radiology case vignettes were formulated according to the pattern of FRCR2A examination. The questions were presented to ChatGPT, Google Bard, and Bing. Two residents wrote the examination with the same questions in 3 hours. The responses generated by the AI models were collected and compared to the answer keys and explanation of the answers was rated by the two radiologists. A cutoff of 60% was set as the passing score. Results  The two residents (63.33 and 57.5%) outperformed the three AI models: Bard (44.17%), Bing (53.33%), and ChatGPT (45%), but only one resident passed the examination. The response patterns among the five respondents were significantly different ( p  = 0.0117). In addition, the agreement among the generative AI models was significant (intraclass correlation coefficient [ICC] = 0.628), but there was no agreement between the residents (Kappa = -0.376). The explanation of generative AI models in support of answer was 44.72% accurate. Conclusion  Humans exhibited superior accuracy compared to the AI models, showcasing a stronger comprehension of the subject matter. All three AI models included in the study could not achieve the minimum percentage needed to pass an FRCR2A examination. However, generative AI models showed significant agreement in their answers where the residents exhibited low agreement, highlighting a lack of consistency in their responses.

The Prevalence and Associated Risk Factors for Neonatal Thrombocytopenia Among Newborns Admitted to the Neonatal Intensive Care Unit

Background Thrombocytopenia is the most prevalent hematological condition in neonates that develops in the neonatal intensive care unit (NICU). This set of illnesses is caused by either decreased platelet production due to placental insufficiency, increased platelet breakdown (consumption), or a combination of the two causes. Based on platelet count, it is defined as mild, moderate, or severe thrombocytopenia, with early and late onset. Purpose The purpose of this study is to determine the prevalence of thrombocytopenia and the factors that contribute to it in newborns hospitalized in the neonatal critical care unit at the Maternity and Children Hospital in Al Ahsa, Saudi Arabia. Methods This descriptive retrospective cross-sectional study was carried out at the NICU of the Maternity and Children Hospital in Al Ahsa, Saudi Arabia, over the span of one year (August 2022 to August 2023) among hospitalized neonates with thrombocytopenia. Thrombocytopenia is defined as a platelet count of 150,000 or less. These patients were monitored until they recovered or died. Results The inclusion criteria were met by a total of 242 newborns with thrombocytopenia. Half of the neonates (57%) were full-term, with Apgar scores greater than 5 at the first (84%) and fifth (93%) minutes, respectively. The great majority of individuals (84%) experienced early-onset thrombocytopenia of mild severity (62%) and were asymptomatic (93%). The majority of the cases resolved spontaneously, with only 21% requiring platelet transfusion. There was a significant relationship discovered between gestational age and the severity of thrombocytopenia, with very preterm infants having moderate to severe thrombocytopenia, as well as birth weight (p=0.001). Furthermore, neonates with severe thrombocytopenia had a considerably higher mortality rate (p=0.001). Conclusion The mortality and morbidity of newborns with perinatal risk for neonatal thrombocytopenia can be reduced with timely detection of the cause and development of thrombocytopenia, as well as adequate and early care.

Automated Item Generation: impact of item variants on performance and standard setting

BACKGROUND

Automated Item Generation (AIG) uses computer software to create multiple items from a single question model. There is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each.

METHODS

Pre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four 'isomorphic' 50-item MCQ tests using AIG software. Isomorphic questions use the same question template with minor alterations to test the same learning outcome. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting).

RESULTS

Two thousand two hundred eighteen students from 12 UK medical schools participated, with each school using one of the four papers. The average facility of the four papers ranged from 0.55-0.61, and the cut score ranged from 0.58-0.61. Twenty item models had a facility difference > 0.15 and 10 item models had a difference in standard setting of > 0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility.

CONCLUSIONS

Item facility varied to a greater extent than the standard set. This difference may relate to variants causing greater disruption of clinical reasoning strategies in novice learners compared to experts, but is confounded by the possibility that the performance differences may be explained at school level and therefore warrants further study.

Cognitive and Metacognitive Processes Demonstrated by Pharmacy Students When Making Therapeutic Decisions

Objective. To characterize the types of cognitive and metacognitive processes demonstrated by third-year pharmacy students during a therapeutic reasoning activity.Methods. A qualitative, descriptive study following a think-aloud protocol was used to analyze the cognitive (analytical) and metacognitive processes observed by third-year pharmacy students as they completed a 25-minute therapeutic reasoning activity. Using a deductive codebook developed from literature about reasoning, two independent coders characterized processes from students' audio-recorded, transcribed think-aloud episodes while making therapeutic decisions about simulated clinical cases.Results. A total of 40 think-aloud episodes were transcribed among the cohort. Categorization of the think-aloud transcriptions revealed a series of cognitive analytical and metacognitive processes demonstrated by students during the therapeutic decision-making activity. A total of 1792 codes were categorized as analytical processes, falling into six major themes: 69% gathering information (1232/1792), 13% processing information (227/1792), 7% making assessments (133/1792), 1% synthesizing information (19/1792), 7% articulating evidence (117/1792), and 4% making a recommendation (64/1792). In comparison to gathering information, a much lower frequency of processing and assessment was observed for students, particularly for those that were unable to resolve the case. Students' movement between major analytical processes co-occurred commonly with metacognitive processes. Of the 918 codes categorized as metacognitive processes, two major themes arose: 28% monitoring for knowledge or emotions (257/918) and 72% controlling the planning of next steps or verification of correct information (661/918). Sequencing the codes and co-occurrences of processes allowed us to propose an integrated cognitive/metacognitive model of therapeutic reasoning for students.Conclusion. This study categorizes the cognitive (analytical) and metacognitive processes engaged during pharmacy students' therapeutic reasoning process. The findings can inform current instructional practices and further research into educational activities that can strengthen pharmacy students' therapeutic reasoning skills.

Oral presentation assessment and image reading behaviour on brain computed tomography reading in novice clinical learners: an eye-tracking study

BACKGROUND

To study whether oral presentation (OP) assessment could reflect the novice learners' interpretation skills and reading behaviour on brain computed tomography (CT) reading.

METHODS

Eighty fifth-year medical students were recruited, received a 2-hour interactive workshop on how to read brain CT, and were assigned to read two brain CT images before and after instruction. We evaluated their image reading behaviour in terms of overall OP post-test rating, the lesion identification, and competency in systematic image reading after instruction. Students' reading behaviour in searching for the target lesions were recorded by the eye-tracking technique and were used to validate the accuracy of lesion reports. Statistical analyses, including lag sequential analysis (LSA), linear mixed models, and transition entropy (TE) were conducted to reveal temporal relations and spatial complexity of systematic image reading from the eye movement perspective.

RESULTS

The overall OP ratings [pre-test vs. post-test: 0 vs. 1 in case 1, 0 vs. 1 in case 2, p < 0.001] improved after instruction. Both the scores of systematic OP ratings [0 vs.1 in both cases, p < 0.001] and eye-tracking studies (Case 1: 3.42 ± 0.62 and 3.67 ± 0.37 in TE, p = 0.001; Case 2: 3.42 ± 0.76 and 3.75 ± 0.37 in TE, p = 0.002) showed that the image reading behaviour changed before and after instruction. The results of linear mixed models suggested a significant interaction between instruction and area of interests for case 1 (p < 0.001) and case 2 (p = 0.004). Visual attention to the target lesions in the case 1 assessed by dwell time were 506.50 ± 509.06 and 374.38 ± 464.68 milliseconds before and after instruction (p = 0.02). However, the dwell times in the case 2, the fixation counts and the frequencies of accurate lesion diagnoses in both cases did not change after instruction.

CONCLUSION

Our results showed OP performance may change concurrently with the medical students' reading behaviour on brain CT after a structured instruction.

Implicit bias in healthcare: clinical practice, research and decision making

Bias is the evaluation of something or someone that can be positive or negative, and implicit or unconscious bias is when the person is unaware of their evaluation. This is particularly relevant to policymaking during the coronavirus pandemic and racial inequality highlighted during the support for the Black Lives Matter movement. A literature review was performed to define bias, identify the impact of bias on clinical practice and research as well as clinical decision making (cognitive bias). Bias training could bridge the gap from the lack of awareness of bias to the ability to recognise bias in others and within ourselves. However, there are no effective debiasing strategies. Awareness of implicit bias must not deflect from wider socio-economic, political and structural barriers as well ignore explicit bias such as prejudice.

Examining the patterns of uncertainty across clinical reasoning tasks: effects of contextual factors on the clinical reasoning process

Objectives

Uncertainty is common in clinical reasoning given the dynamic processes required to come to a diagnosis. Though some uncertainty is expected during clinical encounters, it can have detrimental effects on clinical reasoning. Likewise, evidence has established the potentially detrimental effects of the presence of distracting contextual factors (i.e., factors other than case content needed to establish a diagnosis) in a clinical encounter on clinical reasoning. The purpose of this study was to examine how linguistic markers of uncertainty overlap with different clinical reasoning tasks and how distracting contextual factors might affect physicians’ clinical reasoning process.

Methods

In this descriptive exploratory study, physicians participated in a live or video recorded simulated clinical encounter depicting a patient with unstable angina with and without contextual factors. Transcribed think-aloud reflections were coded using Goldszmidt’s clinical reasoning task typology (26 tasks encompassing the domains of framing, diagnosis, management, and reflection) and then those coded categories were examined using linguistic markers of uncertainty (e.g., probably, possibly, etc.).

Results

Thirty physicians with varying levels of experience participated. Consistent with expectations, descriptive analysis revealed that physicians expressed more uncertainty in cases with distracting contextual factors compared to those without. Across the four domains of reasoning tasks, physicians expressed the most uncertainty in diagnosis and least in reflection.

Conclusions

These results highlight how linguistic markers of uncertainty can shed light on the role contextual factors might play in uncertainty which can lead to error and why it is essential to find ways of managing it.

The Linguistic Effects of Context Specificity: Exploring Affect, Cognitive Processing, and Agency in Physicians' Think-Aloud Reflections

Background The literature suggests that affect, higher-level cognitive processes (e.g. decision-making), and agency (the capacity to produce an effect) are important for reasoning; however, we do not know how these factors respond to context. Using situated cognition theory as a framework, and linguistic tools as a method, we explored the effects of context specificity [a physician seeing two patients with identical presentations (symptoms and findings), but coming to two different diagnoses], hypothesizing more linguistic markers of cognitive load in the presence of contextual factors (e.g. incorrect diagnostic suggestion). Methods In this comparative and exploratory study, 64 physicians each completed one case with contextual factors and one without. Transcribed think-aloud reflections were coded by Linguistic Inquiry and Word Count (LIWC) software for markers of affect, cognitive processes, and first-person pronouns. A repeated-measures multivariate analysis of variance was used to inferentially compare these LIWC categories between cases with and without contextual factors. This was followed by exploratory descriptive analysis of subcategories. Results As hypothesized, participants used more affective and cognitive process markers in cases with contextual factors and more I/me pronouns in cases without. These differences were statistically significant for cognitive processing words but not affective and pronominal words. Exploratory analysis revealed more negative emotions, cognitive processes of insight, and third-person pronouns in cases with contextual factors. Conclusions This study exposes linguistic differences arising from context specificity. These results demonstrate the value of a situated cognition view of patient encounters and reveal the utility of linguistic tools for examining clinical reasoning.