The influence of viewing time on visual diagnostic accuracy: Less is more

BACKGROUND

Understanding the factors that contribute to diagnostic errors is critical if we are to correct or prevent them. Some scholars influenced by the default interventionist dual-process theory of cognition (dual-process theory) emphasise a narrow focus on individual clinician's faulty reasoning as a significant contributor. In this paper, we examine the validity of claims that dual process theory is a key to error reduction.

METHODS

We examined the relationship between a clinical experience (staff and resident physicians) and viewing time on accuracy for categorising chest X-rays (CXRs) and electrocardiograms (ECGs). In two studies, participants categorised images as normal or abnormal, presented at viewing times of 175, 250, 500 and 1000 ms, to encourage System 1 processing. Study 2 extended viewing times to 1, 5, 10 and 20 s to allow time for System 2 processing and a diagnosis. Descriptives and repeated measures analysis of variance were used to analyse the proportion of true and false positive rates (TP and FP) as well as correct diagnoses.

RESULTS

In Study 1, physicians were able to detect abnormal CXRs (0.78) and ECGs (0.67) with relatively high accuracy. The effect of experience was found for ECGs only, as staff physicians (0.71, 95% CI = 0.66-0.75) had higher ECG TP than resident physicians (0.63, 95% CI = 0.58-0.68) in Study 1, and staff had lower ECG FP (0.10, 95% CI = 0.03-0.18) than resident physicians (0.27, 95% CI = 0.20-0.33) in Study 2. In other comparisons, experience was equivocal for ECG FPs and CXR TPs and FPs. In Study 2, overall diagnostic accuracy was similar for both ECGs and CXRs, (0.74). There were small interactions between experience and time for TP in ECGs and FP in CXRs, which are discussed further in the discussion and offer insights into the relationship between processing and experience.

CONCLUSION

Overall, our findings raise concerns about the practical application of models that link processing type to diagnostic error, or to specific diagnostic error reduction strategies.

Is There a Problem With Evidence in Health Professions Education?

ABSTRACT

What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume the knowledge produced as evidence might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more systemwide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations, as well as influencing decisions.

Crowdsourcing a diagnosis? Exploring the accuracy of the size and type of group diagnosis: an experimental study

BACKGROUND

The consultation process, where a clinician seeks an opinion from another clinician, is foundational in medicine. However, the effectiveness of group diagnosis has not been studied.

OBJECTIVE

To compare individual diagnosis to group diagnosis on two dimensions: group size (n=3 or 6) and group process (interactive or artificial groups).

METHODOLOGY

Thirty-six internal or emergency medicine residents participated in the study. Initially, each resident worked through four written cases on their own, providing a primary diagnosis and a differential diagnosis. Next, participants formed into groups of three. Using a videoconferencing platform, they worked through four additional cases, collectively providing a single primary diagnosis and differential diagnosis. The process was repeated using a group of six with four new cases. Cases were all counterbalanced. Retrospectively, nominal (ie, artificial) groups were formed by aggregating individual participant data into subgroups of three and six and analytically computing scores. Presence of the correct diagnosis as primary diagnosis or included in the differential diagnosis, as well as the number of diagnoses mentioned, was calculated for all conditions. Means were compared using analysis of variance.

RESULTS

For both authentic and nominal groups, the diagnostic accuracy of group diagnosis was superior to individual for both the primary diagnosis and differential diagnosis. However, there was no improvement in diagnostic accuracy when comparing a group of three to a group of six. Interactive and nominal groups were equivalent; however, this may be an artefact of the method used to combine data.

CONCLUSIONS

Group diagnosis improves diagnostic accuracy. However, a larger group is not necessarily superior to a smaller group. In this study, interactive group discussion does not result in improved diagnostic accuracy.

Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Demonstrating causality, bestowing honours, and contributing to the arms race: Threats to the sustainability of HPE research

As the field of health professions education (HPE) continues to evolve, it is necessary to occasionally pause and reflect on the potential effects and outcomes of our research practices. While future-casting does not guarantee that impending negative consequences will be evaded, the exercise can help us avoid pitfalls. In this paper, we reflect on two terms that have taken hold as powerful idols in HPE research that stand above questioning and apart from critique: patient outcomes and productivity. We argue that these terms, and the ways of thinking they uphold, threaten the sustainability of HPE research-one at the level of the community and one at the level of the scholar. First, we suggest that HPE research's history of endorsing a linear and causal association ethos has driven its quest to connect education to patient outcomes. To ensure the sustainability of HPE scholarship, we must deconstruct and disempower patient outcomes as one of HPE's god-terms, as the pinnacle goal of educational activities. To be sustained, HPE research needs to value all of its contributions equally. A second god-term is productivity; it impairs the sustainability of the careers of individual researchers. Problems of honorary authorship, research output expectations, and comparisons with other fields have constructed a space where only scholars with sufficient privilege can prevail. If productivity persists as a god-term, the field of HPE research could decay into a space where new scholars are silenced-not because they fail to make important contributions, but because access is restricted by existing research metrics. These are two of many god-terms threatening the sustainability of HPE research. By highlighting patient outcomes and productivity and by acknowledging our own participation in propagating them, we hope to encourage others to recognize how our collective choices threaten the sustainability of our field.

Does allowing access to electronic differential diagnosis support threaten the reliability of a licensing exam?

INTRODUCTION

Newer electronic differential diagnosis supports (EDSs) are efficient and effective at improving diagnostic skill. Although these supports are encouraged in practice, they are prohibited in medical licensing examinations. The purpose of this study is to determine how using an EDS impacts examinees' results when answering clinical diagnosis questions.

METHOD

The authors recruited 100 medical students from McMaster University (Hamilton, Ontario) to answer 40 clinical diagnosis questions in a simulated examination in 2021. Of these, 50 were first-year students and 50 were final-year students. Participants from each year of study were randomised into one of two groups. During the survey, half of the students had access to Isabel (an EDS) and half did not. Differences were explored using analysis of variance (ANOVA), and reliability estimates were compared for each group.

RESULTS

Test scores were higher for final-year versus first-year students (53 ± 13% versus 29 ± 10, p < 0.001) and higher with the use of EDS (44 ± 28% versus 36 ± 26%, p < 0.001). Students using the EDS took longer to complete the test (p < 0.001). Internal consistency reliability (Cronbach's alpha) increased with EDS use among final-year students but was reduced among first-year students, although the effect was not significant. A similar pattern was noted in item discrimination, which was significant.

CONCLUSION

EDS use during diagnostic licensing style questions was associated with modest improvements in performance, increased discrimination in senior students and increased testing time. Given that clinicians have access to EDS in routine clinical practice, allowing EDS use for diagnostic questions would maintain ecological validity of testing while preserving important psychometric test characteristics.

LEADS+ Developmental Model: Proposing a new model based on an integrative conceptual review

BACKGROUND

Leaders in academic health sciences centres (AHCs) must navigate multiple roles as an inherent component of their positions. Changing accountabilities, varying expectations, differing leadership capabilities required of multiple leadership roles can be exacerbated by health system disruption, such as during the COVID-19 pandemic. We need improved models that support leaders in navigating the complexity of multiple leadership roles.

METHOD

This integrative conceptual review sought to examine leadership and followership constructs and how they intersect with current leadership practices in AHCs. The goal was to develop a refined model of healthcare leadership development. The authors used iterative cycles of divergent and convergent thinking to explore and synthesise various literature and existing leadership frameworks. The authors used simulated personas and stories to test the model and, finally, the approach sought feedback from knowledge users (including healthcare leaders, medical educators and leadership developers) to offer refinements.

RESULTS

After five rounds of discussion and reformulation, the authors arrived at a refined model: the LEADS+ Developmental Model. The model describes four nested stages, organising progressive capabilities, as an individual toggles between followership and leadership. During the consultation stage, feedback from 29 out of 65 recruited knowledge users (44.6% response rate) was acquired. More than a quarter of respondents served as a senior leader in a healthcare network or national society (27.5%, n = 8). Consulted knowledge users were invited to indicate their endorsement for the refined model using a 10-point scale (10 = highest level of endorsement). There was a high level of endorsement: 7.93 (SD 1.7) out of 10.

CONCLUSION

The LEADS+ Developmental Model may help foster development of academic health centre leaders. In addition to clarifying the synergistic dynamic between leadership and followership, this model describes the paradigms adopted by leaders within health systems throughout their development journey.

Are we talking about practice? A randomized study comparing simulation-based deliberate practice and mastery learning to self-guided practice

OBJECTIVES

Simulation-based technical skills training is now ubiquitous in medicine, particularly for high acuity, low occurrence (HALO) procedures. Mastery learning and deliberate practice (ML + DP) are potentially valuable educational methods, however, they are resource intensive. We sought to compare the effect of deliberate practice and mastery learning versus self-guided practice on skill performance of the rare, life-saving procedure, a bougie-assisted cricothyroidotomy (BAC).

METHODS

We conducted a multi-center, randomized study at five North American emergency medicine (EM) residency programs. We randomly assigned 176 EM residents to either the ML + DP or self-guided practice groups. Three blinded airway experts independently evaluated BAC skill performance by video review before (pre-test), after (post-test) and 6-12 months (retention) after the training session. The primary outcome was post-test skill performance using a global rating score (GRS). Secondary outcomes included performance time and skill performance at the retention test.

RESULTS

Immediately following training, GRS scores were significantly higher as mean performance improved from pre-test, (22, 95% CI = 21-23) to post-test (27, 95% CI = 26-28), (p < 0.001) for all participants. However, there was no difference between the groups on GRS scores (p = 0.2) at the post-test or at the retention test (p = 0.2). At the retention test, participants in the ML + DP group had faster performance times (66 s, 95% CI = 57-74) compared to the self-guided group (77 s, 95% CI = 67-86), (p < 0.01).

CONCLUSIONS

There was no significant difference in skill performance between groups. Residents who received deliberate practice and mastery learning demonstrated an improvement in skill performance time.

Blueprints for Connection: A Meta-Organizational Framework for Layering Theory, Philosophy, and Praxis Within Continuing Education in the Health Professions

ABSTRACT

As a field, Continuing Professional Development (CPD) lies at the intersection of many disciplines. Tensions can occur as scholars from fields ranging from education to quality improvement seek to advance the practices and workplaces of health care professionals. Owing to the diversity of people working to affect change within the field of CPD, it remains a very challenging space to collaborate and understand the various philosophies, epistemologies, and practice of all those within the field.In this article, the authors have proposed a meta-organizational framework for how we might re-examine theory, application, and practice within the field of CPD. It is their belief that this proposal might inspire others to reflect on how we can cultivate and invite diverse scientists and scholars using a range of theories to add to the fabric of the field of CPD.

Education Theory Made Practical: Creating open educational resources via an apprenticeship model

Introduction

Clinical faculty may have limited knowledge of education theories and best practices in health professions education. Many faculty development programs focus on passive learning with limited application to practice. There is a need for more active engagement for early career educators.

Methods

We created an apprenticeship-based electronic book series focused on translating education theories into practical applications for clinician educators. Chapters were authored by teams of two to four geographically separated early career educators, who were tasked with explaining an education theory and relating it to their educational practice. The chapters underwent internal peer review, followed by open peer review as a blog post and eventual publication. Usage data were collected, and surveys were sent to authors and end-users.

Results

Six volumes (60 total chapters) have been created to date by 180 unique authors and 17 editors over a 6-year period. There have been 65,571 total blog page views and 17,180 total book downloads across the five published volumes. Authors reported an increase in their perceived knowledge (pre 2.6 ± 1.7 vs. post 7.2 ± 1.1, mean difference 4.5/9.0, 95% confidence interval [CI] 4.0-5.0, p < 0.001) after writing their chapter. Authors also reported career benefits including authorship for academic advancement/promotion and developing an area of education theory expertise. End-users also reported a mean increase in their perceived knowledge (pre 4.4 ± 2.5 vs. post 7.3 ± 1.4, mean difference 2.9/9.0, 95% CI 2.1-3.8, p < 0.001) after reading a chapter.

Conclusion

The Education Theory Made Practical electronic book series represents a proof of concept for an apprenticeship-based model to teach education theory, while also creating scholarship and open access resources for the broader community.

Implementation of Entrustable Professional Activities assessments in a Canadian obstetrics and gynecology residency program: a mixed methods study

BACKGROUND

Since the implementation of competency-based medical education (CBME) across residency training programs in Canada, there has been limited research understanding how entrustable professional activity (EPA) assessments are used by faculty supervisors and residents.

OBJECTIVE

This study examines how EPA assessments are used in an Obstetrics and Gynecology residency program and the impact of implementation on both groups.

METHODS

A mixed methods study design was used. Part one involved the aggregation of descriptive data of EPA assessment completion for postgraduate year 1 and 2 residents from July 2019 to May 2020. Part two involved a thematic analysis of semi-structured interviews of residents and faculty.

RESULTS

There was significant uptake of EPA assessments across community and teaching hospitals with widespread contribution of assessment data from faculty. However, both residents and faculty reported that the intended design of EPA assessments as low-stakes assessments to provide formative feedback is not how EPA assessments are experienced. Residents and faculty noted the increased level of administrative burden and related perceived stress amongst the resident group.

CONCLUSIONS

The implementation of EPA assessments is feasible across a variety of sites. However, previous measurement challenges remain. Neither residents nor faculty perceive the value of EPAs to improve feedback, despite their intended nature.

Intrinsic or Invisible? An Audit of CanMEDS Roles in Entrustable Professional Activities

PURPOSE

Postgraduate medical education in Canada has quickly transformed to a competency-based model featuring new entrustable professional activities (EPAs) and associated milestones. It remains unclear, however, how these milestones are distributed between the central medical expert role and 6 intrinsic roles of the larger CanMEDS competency framework. A document review was thus conducted to measure how many EPA milestones are classified under each CanMEDS role, focusing on the overall balance between representation of intrinsic roles and that of medical expert.

METHOD

Data were extracted from the EPA guides of 40 Canadian specialties in 2021 to measure the percentage of milestones formally linked to each role. Subsequent analyses explored for differences when milestones were separated by stage of postgraduate training, weighted by an EPA's minimum number of observations, or sorted by surgical and medical specialties.

RESULTS

Approximately half of all EPA milestones (mean = 48.6%; 95% confidence interval [CI] = 45.9, 51.3) were classified under intrinsic roles overall. However, representation of the health advocate role was consistently low (mean = 2.95%; 95% CI = 2.49, 3.41), and some intrinsic roles-mainly leader, scholar, and professional-were more heavily concentrated in the final stage of postgraduate training. These findings held true under all conditions examined.

CONCLUSIONS

The observed distribution of roles in EPA milestones fits with high-level descriptions of CanMEDS in that intrinsic roles are viewed as inextricably linked to medical expertise, implying both are equally important to cultivate through curricula. Yet a fine-grained analysis suggests that a low prevalence or late emphasis of some intrinsic roles may hinder how they are taught or assessed. Future work must explore whether the quantity or timing of milestones shapes the perceived value of each role, and other factors determining the optimal distribution of roles throughout training.

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals' assessment were included. We organized and synthesized the validity evidence according to Kane's framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

The scope of health professions education requires complementary and diverse approaches to knowledge synthesis

None

Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education

BACKGROUND

Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance.

OBJECTIVE

This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings.

METHODS

NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported.

RESULTS

The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%).

CONCLUSIONS

The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Electronic diagnostic support in emergency physician triage: a qualitative study (Preprint)

Background

Not thinking of a diagnosis is a leading cause of diagnostic error in the emergency department, resulting in delayed treatment, morbidity, and excess mortality. Electronic differential diagnostic support (EDS) results in small but significant reductions in diagnostic error. However, the uptake of EDS by clinicians is limited.

Objective

We sought to understand physician perceptions and barriers to the uptake of EDS within the emergency department triage process.

Methods

We conducted a qualitative study using a research associate to rapidly prototype an embedded EDS into the emergency department triage process. Physicians involved in the triage assessment of a busy emergency department were provided the output of an EDS based on the triage complaint by an embedded researcher to simulate an automated system that would draw from the electronic medical record. Physicians were interviewed immediately after their experience. Verbatim transcripts were analyzed by a team using open and axial coding, informed by direct content analysis.

Results

In all, 4 themes emerged from 14 interviews: (1) the quality of the EDS was inferred from the scope and prioritization of the diagnoses present in the EDS differential; (2) the trust of the EDS was linked to varied beliefs around the diagnostic process and potential for bias; (3) clinicians foresaw more benefit to EDS use for colleagues and trainees rather than themselves; and (4) clinicians felt strongly that EDS output should not be included in the patient record.

Conclusions

The adoption of an EDS into an emergency department triage process will require a system that provides diagnostic suggestions appropriate for the scope and context of the emergency department triage process, transparency of system design, and affordances for clinician beliefs about the diagnostic process and addresses clinician concern around including EDS output in the patient record.

The McMaster Advanced Communication Competencies Model for Psychiatry (MACC Model)

OBJECTIVE

Communication is a core competency for all physicians in training. In Canada, the importance of communication during residency is recognized through the CanMEDS framework. Although literature exists around teaching communication skills to residents, research in psychiatry residents is lacking. The purpose of this study was to explore how faculty members conceptualize the development of communication skills in psychiatry residents and develop a model reflecting this.

METHODS

The authors used a constructivist grounded theory approach. Purposive sampling was used to select 14 faculty educators who regularly supervise psychiatry residents in a single university-based residency training program. Semi-structured interviews were conducted to explore educators' perceptions of how communication skills develop during residency. Constant comparative analysis occurred concurrently with data collection until thematic theoretical sufficiency was reached and relationships between themes determined.

RESULTS

Five themes underlie the McMaster Advanced Communication Competencies model, describing the progressive development of communication abilities in residents. Three themes identify foundational sets of abilities including: refining common foundational relational abilities, developing foundational specific psychiatric communication abilities, and learning to reflect upon and manage internal reactions. These foundational pillars then allow residents to develop a personalized art of flexible psychiatric interviewing, and skillfully partner with patients in co-creating care plans.

CONCLUSIONS

This research describes a preliminary communication competency model for teaching and assessing psychiatry residents. It defines the core communication abilities required for residents to progress to independent practice. Future research could explore and test the model's validity and transferability.

Diagnostic reasoning in cardiovascular medicine

Research in cognitive psychology shows that expert clinicians make a medical diagnosis through a two step process of hypothesis generation and hypothesis testing. Experts generate a list of possible diagnoses quickly and intuitively, drawing on previous experience. Experts remember specific examples of various disease categories as exemplars, which enables rapid access to diagnostic possibilities and gives them an intuitive sense of the base rates of various diagnoses. After generating diagnostic hypotheses, clinicians then test the hypotheses and subjectively estimate the probability of each diagnostic possibility by using a heuristic called anchoring and adjusting. Although both novices and experts use this two step diagnostic process, experts distinguish themselves as better diagnosticians through their ability to mobilize experiential knowledge in a manner that is content specific. Experience is clearly the best teacher, but some educational strategies have been shown to modestly improve diagnostic accuracy. Increased knowledge about the cognitive psychology of the diagnostic process and the pitfalls inherent in the process may inform clinical teachers and help learners and clinicians to improve the accuracy of diagnostic reasoning. This article reviews the literature on the cognitive psychology of diagnostic reasoning in the context of cardiovascular disease.

Supported Independence: The Role of Supervision to Help Trainees Manage Uncertainty

PURPOSE

Safe and effective supervised practice requires a negotiated partnership between trainees and their supervisors. Substantial work has explored how supervisors make judgments about trainees' readiness to safely engage in critical professional activities, yet less is known about how trainees leverage the support of supervisors when they perceive themselves to be at the limits of their abilities. The purpose of this study is to explore how trainees use supervisory support to navigate experiences of clinical uncertainty.

METHOD

Using a constructivist grounded theory approach, the authors explored how novice emergency medicine trainees conceptualized the role of their supervisors during experiences of clinical uncertainty. They employed a critical incident technique to elicit stories from participants immediately following clinical shifts between July and September 2020, and asked participants to describe their experiences of uncertainty within the context of supervised practice. Using constant comparison, 2 investigators coded line-by-line and organized these stories into focused codes. The relationships between these codes were discussed by the research team, and this enabled them to theorize about the relationships between the emergent themes.

RESULTS

Participants reported a strong desire for supported independence, where predictable and accessible supervisory structures enabled them to work semiautonomously through challenging clinical situations. They described a process of borrowing their supervisors' comfort during moments of uncertainty and mechanisms to strategically broadcast their evolving understanding of a situation to implicitly invoke (the right level of) support from their supervisors. They also highlighted challenges they faced when they felt insufficiently supported.

CONCLUSIONS

By borrowing comfort from-or deliberately projecting their thinking to-supervisors, trainees aimed to strike the appropriate balance between independence for the purposes of learning and support to ensure safety. Understanding these strategic efforts could help educators to better support trainees in their growth toward self-regulation.

Should electronic differential diagnosis support be used early or late in the diagnostic process? A multicentre experimental study of Isabel

Background

Diagnostic errors unfortunately remain common. Electronic differential diagnostic support (EDS) systems may help, but it is unclear when and how they ought to be integrated into the diagnostic process.

Objective

To explore how much EDS improves diagnostic accuracy, and whether EDS should be used early or late in the diagnostic process.

Setting

6 Canadian medical schools. A volunteer sample of 67 medical students, 62 residents in internal medicine or emergency medicine, and 61 practising internists or emergency medicine physicians were recruited in May through June 2020.

Intervention

Participants were randomised to make use of EDS either early (after the chief complaint) or late (after the complete history and physical is available) in the diagnostic process while solving each of 16 written cases. For each case, we measured the number of diagnoses proposed in the differential diagnosis and how often the correct diagnosis was present within the differential.

Results

EDS increased the number of diagnostic hypotheses by 2.32 (95% CI 2.10 to 2.49) when used early in the process and 0.89 (95% CI 0.69 to 1.10) when used late in the process (both p<0.001). Both early and late use of EDS increased the likelihood of the correct diagnosis being present in the differential (7% and 8%, respectively, both p<0.001). Whereas early use increased the number of diagnostic hypotheses (most notably for students and residents), late use increased the likelihood of the correct diagnosis being present in the differential regardless of one’s experience level.

Conclusions and relevance

EDS increased the number of diagnostic hypotheses and the likelihood of the correct diagnosis appearing in the differential, and these effects persisted irrespective of whether EDS was used early or late in the diagnostic process.

Tensions in describing competency-based medical education: a study of Canadian key opinion leaders

The current discourse on competency-based medical education (CBME) is confounded by a lack of agreement on definitions and philosophical assumptions. This phenomenon impacts curriculum implementation, program evaluation and disrupts dialogue with the education community. The purpose of this study is to explore how Canadian key opinion leaders describe the philosophy and practice of CBME. A purposeful and snowball sample of Canadian key opinion leaders, reflecting diversity of institutions and academic roles, was recruited. A qualitative thematic analysis of semi-structured interviews was conducted using the principles of constructivist grounded theory. A modified integrated knowledge user checking process was accomplished via a national open meeting of educators, researchers, and leaders in postgraduate medical education. Research ethics board approval was received. 17 interviews were completed between September and November 2018. 43 participants attended the open meeting. There was no unified framing or definition of CBME; perspectives were heterogenous. Most participants struggled to identify a philosophy or theory that underpinned CBME. CBME was often defined by key operational practices, including an emphasis on work-based assessments and coaching relationships between learners and supervisors. CBME was articulated as addressing problems with current training models, including failure to fail, rigor in the structure of training and maintaining the social contract with the public. The unintended consequences of CBME included a reductionist framing of competence and concern for resident wellness with changes to the learning environment. This study demonstrates a heterogeneity in defining CMBE among Canadian key opinion leaders. Future work should explore the fidelity of implementation of CBME.

Maximizing the Morning Commute: A Randomized Trial Assessing the Effect of Driving on Podcast Knowledge Acquisition and Retention

STUDY OBJECTIVE

Emergency medicine residents use podcasts as part of their learning process, often listening while driving. It is unclear how driving while listening to a podcast affects knowledge acquisition and retention. This study evaluated the knowledge gained from listening to podcasts while driving compared to that gained from undistracted listening.

METHODS

This was a multicenter, randomized, crossover trial among postgraduate year (PGY) 1 to 4 emergency medicine residents at 4 institutions. Residents were randomized with stratification by site and PGY level to listen to podcasts while driving first or sitting undistracted in a room first. Within 30 minutes of listening, they completed a 20-question test. They subsequently crossed over to the alternate intervention, serving as their own controls, and listened to a different podcast before completing a second 20-question test. Each of the podcasts was professionally recorded and based on 5 emergency medicine-relevant journal articles that had not been covered in a journal club or curriculum at any of the institutions. One month later, participants completed a delayed recall test composed of 40 new questions based on both podcasts. Questions were derived and validity evidence was collected prior to use. Data were compared using a paired-sample t test and ANOVA.

RESULTS

A total of 100 residents completed the initial recall tests, and 96 residents completed the delayed recall test. There was no statistically significant difference between the driving and undistracted cohorts on the initial recall (74.2% versus 73.3%) or delayed recall (52.2% versus 52.0%).

CONCLUSION

Driving while listening to a podcast does not meaningfully affect knowledge acquisition or retention when compared with undistracted podcast listening among emergency medicine residents.

Intuitive and deliberative approaches for diagnosing 'well' versus 'unwell': evidence from eye tracking, and potential implications for training

Rapidly assessing how ill a patient is based on their immediate presentation-colloquially termed 'eyeballing' in practice-serves a vital role in acute care settings. Yet surprisingly little is known about how this diagnostic skill is learned or how it should be taught. Some authors have pointed to a dual-process model, suggesting that assessments of illness severity are driven by two distinct types of processing: an intuitive, fast, pattern recognition-like process (Type 1) that depends on many prior patient encounters and outcomes being stored in memory; and a deliberate, slow, analytic process (Type 2) characterized by additional data gathering, data scrutiny, or recollection of rules. But prior studies have supported a dual-process model for the assessment of illness severity only insofar as experienced clinicians chiefly displayed what was presumed to be Type 1 processing. Here we further explored a dual-process model by examining whether less experienced clinicians displayed both types of processing when assessing illness severity across a series of cases. Consistent with the model, a dissociation between Type 1 and Type 2 processing was observed through resident reports of deliberation, response times, and three eye tracking metrics associated with diagnostic expertise. We conclude by discussing potential implications for the training of this enigmatic diagnostic skill.

Faculty development using a virtual community of practice: Three-year outcomes of the Academic Life in Emergency Medicine Faculty Incubator program

INTRODUCTION

The Academic Life in Emergency Medicine (ALiEM) Faculty Incubator program is a longitudinal, 1-year, virtual faculty development program for early- and mid-career faculty members that crosses specialties and institutions. This study sought to evaluate the outcomes among 3 years of participants.

METHODS

This cross-sectional survey study evaluated postcourse and 1-year outcomes from three graduated classes of the ALiEM Faculty Incubator program. The program evaluation survey was designed to collect outcomes across multiple Kirkpatrick levels using pre/post surveys and tracking of abstracts, publications, speaking opportunities, new leadership positions, and new curricula.

RESULTS

Over 3 years, 89 clinician educators participated in the program. Of those, 59 (66%) completed the initial survey and 33 (37%) completed the 1-year survey. Participants reported a significant increase in knowledge (4.1/9.0 vs. 7.0/9.0). The number of abstracts, publications, and invited presentations significantly increased after course completion and continued postcourse. A total of 37 of 59 (62.7%) developed a new curriculum during the course and 19 of 33 (57.6%) developed another new curriculum after the course. A total of 29 of 59 (49.2%) began a new leadership position upon course completion with 15 of 33 (45.5%) beginning another new leadership position 1 year later.

DISCUSSION

The ALiEM Faculty Incubator program demonstrated an increase in perceived knowledge and documented academic productivity among early- and mid-career medical educators.

Skeptical self-regulation: Resident experiences of uncertainty about uncertainty

OBJECTIVES

Managing uncertainty is central to expert practice, yet how novice trainees navigate these moments is likely different than what has been described by experienced clinicians. Exploring trainees' experiences with uncertainty could therefore help explicate the unique cues that they attend to, how they appraise their comfort in these moments and how they enact responses within the affordances of their training environment.

METHODS

Informed by constructivist grounded theory, we explored how novice emergency medicine trainees experienced and managed clinical uncertainty in practice. We used a critical incident technique to prompt participants to reflect on experiences with uncertainty immediately following a clinical shift, exploring the cues they attended to and the approaches they used to navigate these moments. Two investigators coded line-by-line using constant comparison, organising the data into focused codes. The research team discussed the relationships between these codes and developed a set of themes that supported our efforts to theorise about the phenomenon.

RESULTS

We enrolled 13 trainees in their first two years of postgraduate training across two institutions. They expressed uncertainty about the root causes of the patient problems they were facing and the potential management steps to take, but also expressed a pervasive sense of uncertainty about their own abilities and their appraisals of the situation. This, in turn, led to challenges with selecting, interpreting and using the cues in their environment effectively. Participants invoked several approaches to combat this sense of uncertainty about themselves, rehearsing steps before a clinical encounter, checking their interpretations with others and implicitly calibrating their appraisals to those of more experienced team members.

CONCLUSIONS

Trainees' struggles with the legitimacy of their interpretations impact their experiences with uncertainty. Recognising these ongoing struggles may enable supervisors and other team members to provide more effective scaffolding, validation and calibration of clinical judgments and patient management.

Infographics, podcasts, and blogs: a multi-channel, asynchronous, digital faculty experience to improve clinical teaching (MAX FacDev)

Attendance at faculty development events are often limited to those with time and interest, but all clinical teachers should have access to continuing faculty development. A multi-channel, asynchronous, digital faculty experience strategy (MAX FacDev) was used to engage busy emergency medicine (EM) teachers associated with a distributed medical education network involving ten geographically distinct teaching sites. An evidence-informed education bundle on key principles for clinical teaching was developed. The education bundle included five topics, serialized via: an infographic series posted in distributed medical education EM departments, a podcast series and a blog. The target audience included 102 faculty members and 46 residents. Within 8 months of launching MAX FacDev, there were 1508 podcast listens and 7686 pageviews. An education bundle can efficiently deliver on-demand faculty development. Amplifying key messages via multiple channels increases the reach of faculty development and reinforces the messages.

"I made a mistake!": A narrative analysis of experienced physicians' stories of preventable error

RATIONALE, AIMS, AND OBJECTIVES

The complexity of healthcare systems makes errors unavoidable. To strengthen the dialogue around how physicians experience and share medical errors, the objective of this study was to understand how generalist physicians make meaning of and grow from their medical errors.

METHODS

This study used a narrative inquiry approach to conduct and analyse in-depth interviews from 26 physicians from the generalist specialties of emergency, internal, and family medicine. We gathered stories via individual interview, analysed them for key components, and rewrote a "meta-story" in a chronological sequence. We conceptualized the findings into a metaphor to draw similarities, learn from, and apply new principles from other fields of practice.

RESULTS

Through analysis we interpreted the story of a physician who is required to make numerous decisions in a short period of time in different clinical environments among the patient's family and whilst abiding by existing rules and regulations. Through sharing stories of success and failure, the clinical supervisor can help optimize the physician's emotional growth and professional development. Similarly, through sharing and learning from stories, colleagues and trainees can also contribute to the growth of the protagonist's character and the development of clinic, hospital, and healthcare system.

CONCLUSION

We draw parallels between the clinical setting and a generalist physician's experiences of a medical error with the environment and practices within professional sports. Using this comparison, we discuss the potential for meaningful coaching in medical education.

The Teacher, the Assessor, and the Patient Protector: A Conceptual Model Describing How Context Interfaces With the Supervisory Roles of Academic Emergency Physicians

OBJECTIVES

Emergency medicine is a fast-paced specialty that demands emergency physicians to respond to rapidly evolving patient presentations, while engaging in clinical supervision. Most research on supervisory roles has focused on the behaviors of attending physicians, including their individual preferences of supervision and level of entrustment of clinical tasks to trainees. However, less research has investigated how the clinical context (patient case complexity, workflow) influences clinical supervision. In this study, we examined how the context of the emergency department (ED) shapes the ways in which emergency physicians reconcile their competing roles in patient care and clinical supervision to optimize learning and ensure patient safety.

METHODS

Emergency physicians who regularly participated in clinical supervision in several academic teaching hospitals were individually interviewed using a semi-structured format. The interviews were transcribed and analyzed using a constructivist grounded theory approach.

RESULTS

Sixteen emergency physicians were asked to reflect on their clinical supervisory roles in the ED. We conceptualized a model that describes three prominent roles: teacher, assessor, and patient protector. Contextual features such as trainee competence, pace of the ED, patient complexity, and the culture of academic medicine influenced the extent to which certain roles were considered salient at any given time.

CONCLUSIONS

This conceptual model can inform researchers and medical educators about the role of context in accentuating or minimizing various roles of emergency physicians. Identifying how context interfaces with these roles may help design faculty development initiatives aimed to navigate the tension between urgent patient care and medical education for emergency physicians.

Residents' Perceptions of Effective Features of Educational Podcasts

INTRODUCTION

Educational podcasts are used by emergency medicine (EM) trainees to supplement clinical learning and to foster a sense of connection to broader physician communities. Yet residents report difficulties remembering what they learned from listening, and the features of podcasts that residents find most effective for learning remain poorly understood. Therefore, we sought to explore residents' perceptions of the design features of educational podcasts that they felt most effectively promoted learning.

METHODS

We used a qualitative approach to explore EM trainees' experiences with educational podcasts, focusing on design features that they found beneficial to their learning. We conducted 16 semi-structured interviews with residents from three institutions from March 2016-August 2017. Interview transcripts were analyzed line-by-line using constant comparison and organized into focused codes, conceptual categories, and then key themes.

RESULTS

The five canons of classical rhetoric provided a framework for thematically grouping the disparate features of podcasts that residents reported enhanced their learning. Specifically, they reported valuing the following: 1) Invention: clinically relevant material presented from multiple perspectives with explicit learning points; 2) Arrangement: efficient communication; 3) Style: narrative incorporating humor and storytelling; 4) Memory: repetition of key content; and 5) Delivery: short episodes with good production quality.

CONCLUSION

This exploratory study describes features that residents perceived as effective for learning from educational podcasts and provides foundational guidance for ongoing research into the most effective ways to structure medical education podcasts.

Using the clinical chemistry score in the emergency department to detect adverse cardiac events: a diagnostic accuracy study

BACKGROUND

The ability to rule out or in a major adverse cardiac event (MACE) in patients with suspected acute coronary syndrome at emergency department (ED) presentation would be beneficial to patient care and the health care system. The clinical chemistry score (CCS) was evaluated in this context.

METHODS

This diagnostic accuracy study evaluated 2 different ED cohorts with suspected acute coronary syndrome. For patients in cohort 1, who presented to the ED of 3 hospitals in Hamilton, Ontario, between May and August 2013, retrospective measurements were taken using the Ortho Clinical Diagnostics high-sensitivity cardiac troponin I (hs-cTnI) assay; for patients in cohort 2, who presented to the ED of the same 3 hospitals in Hamilton between November 2012 and February 2013, an ED cardiac presentation blood test panel was performed with the Abbott Diagnostics hs-cTnI assay. The sensitivity and specificity of the CCS (cut-offs of ≥ 1 and 5) and hs-cTnI alone (published cut-offs) were compared for MACE (composite of death, myocardial infarction, unstable angina, revascularization) at 30 days for both cohorts and at 90 days for cohort 2.

RESULTS

The incidence of MACE at 30 days was higher in cohort 1 (n = 1058) (19.4%, 95% confidence interval [CI] 16.8%-22.2%) than in cohort 2 (n = 5974) (14.6%, 95% CI 13.6%-15.6%). In cohort 1, a CCS of 1 or above yielded a sensitivity of 99.5% (95% CI 97.3%-99.9%). The sensitivity with an Ortho hs-cTnI cut-off of 1 ng/L or above was 91.2% (95% CI 86.5%-95.7%). The specificity of a CCS of 5 (97.8%, 95% CI 96.5%-98.7%) was higher than when the overall 99th-percentile cut-off for the Ortho hs-cTnI assay (> 11 ng/L; 90.1%, 95% CI 87.9%-92.0%) was used. A similar pattern was observed in cohort 2 at 30 days and persisted at 90 days with the Abbott hs-cTnI assay.

INTERPRETATION

The CCS derived with 2 different hs-cTnI assays and ED populations yielded higher sensitivity and specificity estimates for MACE than hs-cTnI alone. An intervention study is needed to evaluate the impact of the CCS at both the patient and hospital levels.

TRIAL REGISTRATION

ClinicalTrials.gov, no. NCT01994577.

Risk Stratification for Patients with Chest Pain Discharged Home from the Emergency Department

For patients with chest pain who are deemed clinically to be low risk and discharged home from the emergency department (ED), it is unclear whether further laboratory tests can improve risk stratification. Here, we investigated the utility of a clinical chemistry score (CCS), which comprises plasma glucose, the estimated glomerular filtration rate, and high-sensitivity cardiac troponin (I or T) to generate a common score for risk stratification. In a cohort of 14,676 chest pain patients in the province of Ontario, Canada and who were discharged home from the ED (November 2012-February 2013 and April 2013-September 2015) we evaluated the CCS as a risk stratification tool for all-cause mortality, plus hospitalization for myocardial infarction or unstable angina (primary outcome) at 30, 90, and 365 days post-discharge using Cox proportional hazard models. At 30 days the primary outcome occurred in 0.3% of patients with a CCS < 2 (n = 6404), 0.9% of patients with a CCS = 2 (n = 4336), and 2.3% of patients with a CCS > 2 (n = 3936) (p < 0.001). At 90 days, patients with CCS < 2 (median age = 52y (IQR = 46-60), 59.4% female) had an adjusted HR = 0.51 (95% confidence interval (CI) = 0.32-0.82) for the composite outcome and patients with a CCS > 2 (median age = 74y (IQR = 64-82), 48.0% female) had an adjusted HR = 2.80 (95%CI = 1.98-3.97). At 365 days, 1.3%, 3.4%, and 11.1% of patients with a CCS < 2, 2, or >2 respectively, had the composite outcome (p < 0.001). In conclusion, the CCS can risk stratify chest pain patients discharged home from the ED and identifies both low- and high-risk patients who may warrant different medical care.

The effect of prior experience on diagnostic reasoning: exploration of availability bias

Objectives Diagnostic reasoning has been shown to be influenced by a prior similar patient case. However, it is unclear whether this process influences diagnostic error rates or whether clinicians at all experience levels are equally susceptible. The present study measured the influence of specific prior exposure and experience level on diagnostic accuracy. Methods To create the experience of prior exposure, participants (pre-clerkship medical students, emergency medicine residents, and faculty) first verified diagnoses of clinical vignettes. The influence of prior exposures was measured using equiprobable clinical vignettes; indicating two diagnoses. Participants diagnosed equiprobable cases that were: 1) matched to exposure cases (in one of three conditions: a) similar patient features, similar clinical features; b) dissimilar patient features, similar clinical features; c) similar patient features, dissimilar clinical features), or 2) not matched to any prior case (d) no exposure). A diagnosis consistent with a matched exposure case was scored correct. Cases with no prior exposure had no matched cases, hence validated the equiprobable design. Results Diagnosis A represented 47% of responses in condition d, but there was no influence of specific similarity of patient characteristics for Diagnosis A, F(3,712)=7.28, p=0.28 or Diagnosis B, F(3,712)=4.87, p=0.19. When re-scored based on matching both equiprobable diagnoses, accuracy was high, but favored faculty (n=40) 98%, and residents (n=39) 98% over medical students (n=32) 85%, F(2,712)=35.6, p<0.0001. Accuracy for medical students was 84, 87, 94, and 73% for conditions a-d, respectively, interaction F(2,712)=3.55, p<0.002. Conclusions The differential diagnosis of pre-clerkship medical students improved with prior exposure, but this was unrelated to specific case or patient features. The accuracy of medical residents and staff was not influenced by prior exposure.

Making Decisions in the Era of the Clinical Decision Rule: How Emergency Physicians Use Clinical Decision Rules

PURPOSE

Physicians are often asked to integrate clinical decision rules (CDRs) with their own cognitive processes to reach a diagnosis. Clinicians, researchers, and educators must understand these cognitive processes to evaluate and improve the diagnostic process. The authors sought to explore emergency physicians' diagnostic processes and to examine how they integrated CDRs into their reasoning using simulated cases (with chest pain or leg pain).

METHOD

From August 2015 to July 2016, 16 practicing emergency physicians from 3 teaching hospitals associated with McMaster University, Ontario, Canada, were interviewed via a novel "teach aloud" protocol. Six videos of simulated patients with chest pain, breathlessness, or leg discomfort were used as prompts for the physicians to demonstrate their diagnostic thinking. Using a constructivist grounded theory analysis, 3 investigators independently reviewed the interview transcripts, meeting regularly to discuss identified themes and subthemes until sufficiency was reached.

RESULTS

A model to describe how clinicians integrate their own decision making with CDRs was developed, showing that physicians engage in an iterative diagnostic process that repeatedly refines the differential diagnosis list. The steps in the diagnostic process were: refinement of the differential diagnosis, ordering a hierarchy of risk, the decision to test, choosing the tests, and interpreting test results. Physicians applied CDRs when they had already decided to test.

CONCLUSIONS

To date, CDRs assume a static, linear model of clinical decision making. Findings demonstrate that participants engaged in iterative and dynamic decision-making processes that changed throughout their patient encounter, contingent on multiple contextual features. Understanding these processes could inform future development of CDRs and educational strategies around these decision aids.

Comparison of two biomarker only algorithms for early risk stratification in patients with suspected acute coronary syndrome

BACKGROUND

We developed a biomarker algorithm encompassing the clinical chemistry score (CCS; which includes the combination of a random glucose concentration, an estimated glomerular filtration rate and high-sensitivity cardiac troponin; hs-cTn) with the Ortho Clinical Diagnostics hs-cTnI assay (CCS-serial) and compared it to the cutoffs derived from Ortho Clinical Diagnostics 0/1 h (h) algorithm for 7-day myocardial infarction (MI) or cardiovascular (CV)-death.

METHODS

The study cohort was an emergency department (ED) population (n = 906) with symptoms suggestive of acute coronary syndrome (ACS) who had two Ortho hs-cTnI results approximately 3 h apart. Diagnostic parameters (sensitivity/specificity/negative predictive value; NPV/positive predictive value; PPV) were derived for the CCS-serial and the 0/1 h algorithm for 7-day MI/CV-death. A safety analysis was performed for patients in the rule-out arms of the algorithms for 30-day MI/death.

RESULTS

The CCS-serial algorithm yielded 100% sensitivity/NPV (32% low-risk) and 95.7% specificity/65% PPV (11% high-risk). The 0/1 h algorithm-cutoffs yielded sensitivity/NPV/specificity/PPV of 97.8%/99.4%/91.3%/50%, which classified 38% of patients as low-risk and 16% of patients as high-risk. Four patients (1.2%) in the 0/1 h algorithm-cutoff rule-out arm had a 30-day MI/death outcome as compared to zero patients in the CCS-serial rule-out arm (p = 0.06).

CONCLUSION

Both the CCS-serial and 0/1 h algorithm cutoffs yield high NPVs with a similar proportion of patients identified as low-risk. These data may be useful for sites who are unable to collect samples at 0/1 h in the emergency department.

High-Sensitivity Cardiac Troponin I vs a Clinical Chemistry Score for Predicting All-Cause Mortality in an Emergency Department Population

BACKGROUND

For patients investigated for suspected acute coronary syndrome, there is uncertainty if a single measurement of high-sensitivity cardiac troponin I (hs-cTnI) at emergency department (ED) presentation can identify patients at both low and high risk for mortality.

METHODS

We included consecutive adult patients in the ED who had a Clinical Chemistry Score (CCS) taken at presentation (ie, combination of glucose, creatinine for estimated glomerular filtration rate determination, and hs-cTnI assay) in a Canadian city between 2012 and 2013. Outcomes were 3-month, 1-year, and 5-year all-cause mortality using the provincial death registry. Mortality rates and test performance (eg, sensitivity and specificity) with 95% confidence intervals (CIs) were obtained for the CCS or hs-cTnI assay alone using established cutoffs for these tests.

RESULTS

Our cohort included 5974 patients with a 1-year mortality rate of 17.2% (95% CI, 16.2-18.3). A CCS ≥ 1 yielded a sensitivity of 99.2% (95% CI, 98.4-99.6) compared with the hs-cTnI ≥ 5 ng/L cutoff sensitivity of 88.4% (95% CI, 86.3-90.3), with the mortality rate being significantly lower for patients with CCS < 1 (2.0%; 95% CI, 0.9-4.0) vs patients with hs-cTnI < 5 ng/L (5.0%; 95% CI, 4.2-6.0) at 1 year (P = 0.01). A CCS of 5 also yielded a higher specificity (88.5%; 95% CI, 87.5-89.3) compared with hs-cTnI > 26 ng/L (83.9%; 95% CI, 82.9-84.9), with no difference in mortality rates (37.4% vs 36.3%; P = 0.66). This trend was consistent at 3-month and 5-year mortality.

CONCLUSION

For patients in the ED with a potential cardiac issue, using the CCS cutoffs can better identify patients at low and high risk for mortality than using published cutoffs for hs-cTnI alone.

A Signal Through the Noise: Do Professionalism Concerns Impact the Decision Making of Competence Committees?

PURPOSE

To characterize how professionalism concerns influence individual reviewers' decisions about resident progression using simulated competence committee (CC) reviews.

METHOD

In April 2017, the authors conducted a survey of 25 Royal College of Physicians and Surgeons of Canada emergency medicine residency program directors and senior faculty who were likely to function as members of a CC (or equivalent) at their institution. Participants took a survey with 12 resident portfolios, each containing hypothetical formative and summative assessments. Six portfolios represented residents progressing as expected (PAE) and 6 represented residents not progressing as expected (NPAE). A professionalism variable (PV) was developed for each portfolio. Two counterbalanced surveys were developed in which 6 portfolios contained a PV and 6 portfolios did not (for each PV condition, 3 portfolios represented residents PAE and 3 represented residents NPAE). Participants were asked to make progression decisions based on each portfolio.

RESULTS

Without PVs, the consistency of participants giving scores of 1 or 2 (i.e., little or no need for educational intervention) to residents PAE and to those NPAE was 92% and 10%, respectively. When a PV was added, the consistency decreased by 34% for residents PAE and increased by 4% for those NPAE (P = .01).

CONCLUSIONS

When reviewing a simulated resident portfolio, individual reviewer scores for residents PAE were responsive to the addition of professionalism concerns. Considering this, educators using a CC should have a system to report, collect, and document professionalism issues.

It's a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

PURPOSE

Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change.

METHOD

Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen's University, Canada, using iterative cycles of Rapid Evaluation in 2017-2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders' experiences.

RESULTS

In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in "missing the forest for the trees," prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents' needs were slow to follow.

CONCLUSIONS

Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

Clinical evaluation of Ortho Clinical Diagnostics high-sensitivity cardiac Troponin I assay in patients with symptoms suggestive of acute coronary syndrome

BACKGROUND

As more companies obtain regulatory approval for high-sensitivity cardiac troponin (hs-cTn) assays there is an urgent need for independent analytical and clinical evaluations. To this end, we have evaluated Ortho Clinical Diagnostics' hs-cTnI assay and compared it to their contemporary cTnI-ES assay in emergency department (ED) patients with suspected acute coronary syndrome (ACS).

METHODS

The study cohort consisted of ED patients (n = 906) with symptoms suggestive of ACS who had Ortho hs-cTnI and cTnI-ES results at presentation and 3 h (with calculated delta (0-3 h) defined as the absolute concentration difference between paired results). The primary composite outcome was 7-day myocardial infarction (MI) or cardiovascular death, with secondary analyses performed for 7-day MI and index-MI. Analytical imprecision testing (i.e., coefficient of variation; CV), receiver-operating characteristic (ROC) curve analyses with area under the curve (AUC), and diagnostic parameters (sensitivity/specificity/predictive values) were calculated.

RESULTS

The hs-cTnI assay had superior precision compared to the cTnI-ES assay below 5 ng/L in EDTA plasma (hs-cTnI CV ≤ 15% versus cTnI-ES CV ≥ 85%). The AUCs were higher for hs-cTnI as compared to cTnI-ES at 0 h (0.88 vs. 0.85), 3 h (0.94 vs. 0.92), and the delta (0-3 h) value (0.91 vs. 0.85) for the primary composite outcome (p < 0.05). At 3 h, the sensitivity/specificity for index-MI was ≥97%/≥82%, for 7-day MI was ≥89%/≥84%, and for the primary composite outcome was ≥90%/≥85% using the manufacturer's sex-specific 99th-percentile cutoffs.

CONCLUSION

The Ortho hs-cTnI assay has superior analytical and clinical performance over their contemporary cTnI-ES assay in evaluating ED patients with symptoms suggestive of ACS.

It's the destination: diagnostic accuracy and reasoning

While multiple theories exist to explain the diagnostic process, there are few available assessments that reliably determine diagnostic competence in trainees. Most methods focus on aspects of the process of diagnostic reasoning, such as the relation between case features and diagnostic hypotheses. Inevitably, detailed elucidation of aspects of the process requires substantial time per case and limits the number of cases that can be examined given a limited testing time. Shifting assessment to the outcome of diagnostic reasoning, accuracy of the diagnosis, may serve as a reliable measure of diagnostic competence and would allow increased sampling across cases. The present study is a retrospective analysis of 7 large studies, conducted by 3 research teams, that all used a series of brief written cases to examine the outcome of diagnostic reasoning-the diagnosis. The studies involved over 600 clinicians ranging from final year medical students to practicing emergency physicians. For 4 studies with usable reliability data, reliability for a 2 h test ranged from .63 to .94. On average speeded tests were more reliable (.85 vs. .73).To achieve a reliability of .75 required an average test time of 1.11 h for speeded tests and 1.99 for unspeeded tests. The measure was shown to be positively correlated with both written knowledge tests and measures of problem solving derived from OSCE performance tests. This retrospective analysis provides evidence to support the implementation of outcome-based assessments of clinical reasoning.

Critical thinking, biases and dual processing: The enduring myth of generalisable skills

CONTEXT

The myth of generalisable thinking skills in medical education is gaining popularity once again. The implications are significant as medical educators decide on how best to use limited resources to prepare trainees for safe medical practice. This myth-busting critical review cautions against the proliferation of curricular interventions based on the acquisition of generalisable skills.

STRUCTURE

This paper begins by examining the recent history of general thinking skills, as defined by research in cognitive psychology and medical education. We describe three distinct epochs: (a) the Renaissance, which marked the beginning of cognitive psychology as a discipline in the 1960s and 1970s and was paralleled by educational reforms in medical education focused on problem solving and problem-based learning; (b) the Enlightenment, when an accumulation of evidence in psychology and in medical education cast doubt on the assumption of general reasoning or problem-solving skill and shifted the focus to consideration of the role of knowledge in expert clinical performance; and (c) the Counter-Enlightenment, in the current time, when the notion of general thinking skills has reappeared under different guises, but the fundamental problems related to lack of generality of skills and centrality of knowledge remain.

CONCLUSIONS

The myth of general thinking skills persists, despite the lack of evidence. Progress in medical education is more likely to arise from devising strategies to improve the breadth and depth of experiential knowledge.

Independent and Interwoven: A Qualitative Exploration of Residents' Experiences With Educational Podcasts

PURPOSE

Educational podcasts are an increasingly popular platform for teaching and learning in health professions education. Yet it remains unclear why residents are drawn to podcasts for educational purposes, how they integrate podcasts into their broader learning experiences, and what challenges they face when using podcasts to learn.

METHOD

The authors used a constructivist grounded theory approach to explore residents' motivations and listening behaviors. They conducted 16 semistructured interviews with residents from 2 U.S. and 1 Canadian institution from March 2016 to August 2017. Interviews were recorded and transcribed. The transcripts were analyzed using constant comparison, and themes were identified iteratively, working toward an explanatory framework that illuminated relationships among themes.

RESULTS

Participants described podcasts as easy to use and engaging, enabling both broad exposure to content and targeted learning. They reported often listening to podcasts while doing other activities, being motivated by an ever-present desire to use their time productively; this practice led to challenges retaining and applying the content they learned from the podcasts to their clinical work. Listening to podcasts also fostered participants' sense of connection to their peers, supervisors, and the larger professional community, yet it created tensions in their local relationships.

CONCLUSIONS

Despite the challenges of distracted, contextually constrained listening and difficulties translating their learning into clinical practice, residents found podcasts to be an accessible and engaging learning platform that offered them broad exposure to core content and personalized learning, concurrently fostering their sense of connection to local and national professional communities.

Effect of Teaching Bayesian Methods Using Learning by Concept vs Learning by Example on Medical Students' Ability to Estimate Probability of a Diagnosis: A Randomized Clinical Trial

Importance

Clinicians use probability estimates to make a diagnosis. Teaching students to make more accurate probability estimates could improve the diagnostic process and, ultimately, the quality of medical care.

Objective

To test whether novice clinicians can be taught to make more accurate bayesian revisions of diagnostic probabilities using teaching methods that apply either explicit conceptual instruction or repeated examples.

Design, Setting, and Participants

A randomized clinical trial of 2 methods for teaching bayesian updating and diagnostic reasoning was performed. A web-based platform was used for consent, randomization, intervention, and testing of the effect of the intervention. Participants included 61 medical students at McMaster University and Eastern Virginia Medical School recruited from May 1 to September 30, 2018.

Interventions

Students were randomized to (1) receive explicit conceptual instruction regarding diagnostic testing and bayesian revision (concept group), (2) exposure to repeated examples of cases with feedback regarding posttest probability (experience group), or (3) a control condition with no conceptual instruction or repeated examples.

Main Outcomes and Measures

Students in all 3 groups were tested on their ability to update the probability of a diagnosis based on either negative or positive test results. Their probability revisions were compared with posttest probability revisions that were calculated using the Bayes rule and known test sensitivity and specificity.

Results

Of the 61 participants, 22 were assigned to the concept group, 20 to the experience group, and 19 to the control group. Approximate age was 25 years. Two participants were first-year; 37, second-year; 12, third-year; and 10, fourth-year students. Mean (SE) probability estimates of students in the concept group were statistically significantly closer to calculated bayesian probability than the other 2 groups (concept, 0.4%; [0.7%]; experience, 3.5% [0.7%]; control, 4.3% [0.7%]; P < .001). Although statistically significant, the differences between groups were relatively modest, and students in all groups performed better than expected, based on prior reports in the literature.

Conclusions and Relevance

The study showed a modest advantage for students who received theoretical instruction on bayesian concepts. All participants' probability estimates were, on average, close to the bayesian calculation. These findings have implications for how to teach diagnostic reasoning to novice clinicians.

Trial Registration

ClinicalTrials.gov identifier: NCT04130607.