Pediatric resident knowledge of transfusion medicine: Results from the BEST-TEST3 international education needs assessment

Current state of undergraduate medical school training in transfusion medicine and its impact on postgraduate trainee knowledge

BACKGROUND

Studies have described poor transfusion medicine (TM) knowledge in postgraduate trainees. The impact of undergraduate medical TM education on postgraduate knowledge is unclear.

METHODS

Canadian medical schools were surveyed on the number of hours dedicated to TM teaching and topics covered by curricula during 2016-2020. Postgraduate trainees attending Transfusion Camp in 2021 completed a pretest of 20 multiple-choice questions. The survey results and pretest scores were compared to evaluate the association between undergraduate medical TM education and pretest scores.

RESULTS

The survey was completed by 16 of 17 Canadian medical schools. The number of hours (h) of TM teaching were <2 h (25%), 3-4 h (25%), and >4 h (50%). Twelve of 19 Transfusion Camp topics were covered in ≥50% of schools. Eleven medical schools provided ethics approvals/waivers to include trainee pretest scores in the analysis (N = 200). The median pretest scores by medical school ranged from 48% to 70%. No association was found between number of TM teaching hours and average pretest scores (p = .60). There was an association between higher postgraduate year level and individual pretest score (p < .0001). The analysis by topic demonstrated questions where trainees from different schools performed uniformly well or poorly; other topics showed considerable variation.

CONCLUSION

Variation in quantity and content of undergraduate TM teaching exists across Canadian medical schools. In this limited assessment, the number of TM teaching hours was not associated with performance on the pretest. This study raises the opportunity to re-evaluate the delivery (content, timing, consistency) of TM education in undergraduate medical schools.

Classification of posttransfusion adverse events using a publicly available artificial intelligence system

BACKGROUND

Correct classification of transfusion reactions is important not only for effective patient care and donor management but also for accurate tracking of events in hemovigilance systems. We compared the ability of a generative artificial intelligence (AI) system to correctly diagnose hypothetical clinical situations as transfusion reactions in comparison to previous studies reporting the accuracy of transfusion medicine (TM) specialists in assessing these cases.

METHODS

An AI system was requested to assess 36 case scenarios to provide a diagnosis, severity, and imputability of the transfusion reactions using the CDC National Healthcare Safety Network (NHSN) criteria. Responses were compared to an expert panel's classifications and to the published responses of a panel of TM specialists. Additionally, the AI's responses were compared to the TM specialists' prior attempts to use the TrDDx web-based algorithm for the five most challenging cases.

RESULTS

The AI's classification accuracy varied widely depending on the NHSN category. The AI accurately classified all transfusion-associated circulatory overload and transfusion-related acute lung injury cases, exceeding TM specialists' assessments. Conversely, it did not correctly identify any cases in select NHSN categories such as DSTR. Overall accuracy among all diagnostic categories was 48.7% for AI responses versus 72.1% for prior TM specialist responses (p = 0.005). AI-generated responses included non-standard terminology, limited severity assessments, and no imputability determinations.

DISCUSSION

A generative AI system may have a role in helping healthcare providers to consider transfusion reaction categories that might be missed, but caution is advised in applying the AI's output to transfusion reaction classification at present.

Anesthesiology Resident Knowledge of Transfusion Medicine: Results From the Anesthesiology Transfusion Education Study

BACKGROUND

Transfusion of blood products is a common practice in anesthesiology. Inadequate transfusion medicine knowledge may lead to inappropriate transfusion practices and patient risk. Using a validated assessment tool modified for anesthesiology, we conducted a survey of anesthesiology residents in the United States to assess transfusion medicine knowledge.

METHODS

A validated transfusion medicine examination and accompanying survey were forwarded by program directors to residents for anonymous completion on May 5 and closed on June 30, 2021. The outcome of interest was the mean examination score. Secondary areas of interest were performance by year of training and previous educational experience in transfusion reported by the trainees. Rasch analysis was performed on the examination quality and individual question performance. Kruskal-Wallis H tests were used to identify differences between mean scores. Post hoc comparisons were used to assess specific pairwise differences between mean test scores by survey variable.

RESULTS

Four hundred twenty-three anesthesiology residents in 37 programs completed the examination. The mean score was 45.5% ± 12.6%. There was a significant difference in mean cumulative examination scores between different resident training levels ( P < 0.001). There was a significant difference in scores between clinical anesthesia (CA)-1 and CA-2 residents ( P = 0.011) and CA-1 and CA-3 residents ( P = 0.012). No significant difference in examination scores was observed between CA-2 and CA-3 residents ( P = 0.95). All these subgroups scored below 50% on the examination. Significant differences between the residency training programs and cumulative scores were identified ( P < 0.001).

CONCLUSIONS

This examination highlights gaps in transfusion medicine knowledge within US anesthesiology residents. Targeted education may improve knowledge in this area and patient care.

Would doctors dream of electric blood bankers? Large language model-based artificial intelligence performs well in many aspects of transfusion medicine

BACKGROUND

Large language models (LLMs) excel at answering knowledge-based questions. Many aspects of blood banking and transfusion medicine involve no direct patient care and require only knowledge and judgment. We hypothesized that public LLMs could perform such tasks with accuracy and precision.

STUDY DESIGN AND METHODS

We presented three sets of tasks to three publicly-available LLMs (Bard, GPT-3.5, and GPT-4). The first was to review short case presentations and then decide if a red blood cell transfusion was indicated. The second task was to answer a set of consultation questions common in clinical transfusion practice. The third task was to take a multiple-choice test experimentally validated to assess internal medicine postgraduate knowledge of transfusion practice (the BEST-TEST).

RESULTS

In the first task, the area under the receiver operating characteristic curve for correct transfusion decisions was 0.65, 0.90, and 0.92, respectively for Bard, GPT-3.5 and GPT-4. All three models had a modest rate of acceptable responses to the consultation questions. Average scores on the BEST-TEST were 55%, 40%, and 87%, respectively.

CONCLUSION

When presented with transfusion medicine tasks in natural language, publicly available LLMs demonstrated a range of ability, but GPT-4 consistently scored very well in all tasks. Research is needed to assess the utility of LLMs in transfusion medicine practice. Transfusion Medicine physicians should consider their role alongside such technologies, and how they might be used for the benefit and safety of patients.

Paediatric clinical decision support: Evaluation of a best practice alert for red blood cell transfusion

BACKGROUND AND OBJECTIVES

Providing red blood cell (RBC) transfusion to paediatric patients with a haemoglobin (Hb) level of <7 g/dL is the current best practice, but it is often difficult to ensure appropriateness of RBC transfusion on a health system level. Electronic health record (EHR) clinical decision support systems have been shown to be effective in encouraging providers to transfuse at appropriate Hb thresholds. We present our experience with an interruptive best practice alert (BPA) at a paediatric healthcare system.

MATERIALS AND METHODS

An interruptive BPA requiring physician response was implemented in our EHR (Epic Systems Corp., Verona, WI, USA) in 2018 based on Hb thresholds for inpatients. The threshold was initially <8 g/dL and later changed to <7 g/dL in 2019. We assessed total activations, number of RBC transfusions and hospital metrics through 2022 compared to the 2 years prior to implementation.

RESULTS

The BPA activated 6956 times over 4 years, slightly less than 5/day, and the success rate, with no RBC transfusions within 24 h of order attempt, was 14.5% (1012/6956). There was a downward trend in the number of total RBC transfusions and RBC transfusions per admission after implementation, non-significant (p = 0.41 and p = >0.99). The annual case mix index was similar over the years evaluated. The estimated cost savings based on acquisition costs for RBC units were 213,822 USD or about $51,891 per year.

CONCLUSION

BPA implementation led to sustained change in RBC transfusion towards best practice, and there were long-term savings in RBC expenditure.

Transfusion medicine curricular content for general pediatricians and pediatric subspecialists: A national multi-specialty Delphi consensus study

BACKGROUND

Although pediatric residents frequently order blood products, transfusion medicine (TM) education is both limited and unstandardized during postgraduate training. Using Delphi methodology, this study aimed to identify and prioritize which pediatric TM curricular topics are most important to inform postgraduate training in TM for general pediatricians and pediatric subspecialists.

METHODS

A national panel of experts iteratively rated potential curricular topics, on a 5-point scale, to determine their priority for inclusion within a TM curriculum. After each round, responses were analyzed. Topics receiving a mean rating <3/5 were removed from subsequent rounds and remaining topics were resent to the panel for further ratings until consensus was achieved, defined as Cronbach α ≥ 0.95. At conclusion of the Delphi process, topics rated ≥4/5 were considered core curricular topics, while topics rated ≥3 to <4 were considered extended topics.

RESULTS

Forty-five TM experts from 17 Canadian institutions and 12 subspecialties completed the first Delphi round and 31 completed the second. Fifty-seven potential curricular topics were generated from a systematic literature review and Delphi panelists. Two survey rounds were completed before consensus was achieved. Seventy-three topics in six domains reached consensus: 31 core curricular topics and 42 extended topics. There were no significant differences in ratings between TM and non-TM specialists.

DISCUSSION

A multispecialty Delphi panel reached consensus in identification of curricular topics for pediatric resident physicians. These results set the stage to develop a pediatric TM curriculum that will be foundational for pediatric trainees to enhance learning and improve transfusion safety.

Optimizing Informed Consent Discussions: Developing a Narrative for Transfusion Consent

Ensuring patient informed consent is a key tenet of modern medicine. Although transfusion of blood products is among the most common medical procedures performed in hospitalized patients, there is evidence that informed consent for transfusion is at times incomplete, poorly understood, hurried, and/or inaccurate. This study aimed to develop a narrative that can be used as a framework for practicing physicians and for educational purposes to optimize the process for obtaining informed consent for blood transfusion. The narrative was developed using a modified Delphi approach with 5 Rounds that included feedback from transfusion medicine (TM) experts, transfusion-provider physicians, and lay people. The surveys collected qualitative and quantitative data analyzed using thematic content analysis and descriptive statistics, respectively. Results from Rounds 1 and 2 generated a draft narrative and Rounds 3 to 5 informed further modifications. Round 1 included draft narrative scripts from 28 TM experts; thematic coding generated 97 topics. In round 2, 22/28 of the initial experts rated items identified from Round 1. Those with a content validity index (CVI) ≥ 0.8 were used by the authors to develop a narrative. In Round 3, 20/24 participants from Round 2 reviewed the narrative with 100% agreeing on the items included and 90% agreeing the flow was logical. In Round 4, 23 transfusion prescribers (non-TM physicians) reviewed the narrative for flow, manner, length, and usability; there was 83% agreement with the nonexclusion of important topics; 91% felt it would be effective for teaching trainees. Round 5 included 24 nonmedical laypeople of different demographics. Most participants (92%) thought that the script was appropriate in length and there were opportunities to ask questions. Participants could also identify the adverse transfusion reactions and understand that they could refuse the transfusion. A narrative for obtaining informed consent for blood transfusion was created through multiple rigorous iterations of review and feedback with both transfusion providers and the lay public. The narrative, developed for a specific clinical scenario, was well-received by medical and nonmedical participants and can be used, and modified, to help ensure patients understand the risks and benefits of blood transfusion.

Does training make a difference? Proficiency training in transfusion guidelines and its effect on red blood cell administration

Background

Packed red blood cell (RBC) transfusion is a very common and frequently lifesaving therapeutic intervention, but a liberal transfusion policy may be associated with inferior patient outcomes. Various guidelines have been proposed to reduce the rate of unnecessary RBC transfusions. However, physicians' proficiency in such guidelines and the effect of training on RBC administration remain unknown.

Methods

We performed a questionnaire‐based assessment of physicians' knowledge of the guidelines in a tertiary hospital in Israel, followed by an analysis of RBC administration six months before and six months after training was delivered.

Results

The level of proficiency was higher among Israeli university graduates (Odds Ratio [OR] 2.59, p‐value = 0.02), internists (OR 2.8, p‐value = 0.02), and physicians beyond the step‐one residency exam (OR 3.08, p‐value = 0.02). There was no significant effect of training on the rates of RBC administration (incidence rate ratio [IRR] = 0.96 [CI 95% 0.81–1.14], p‐value = 0.655).

Conclusion

Educational intervention alone is an ineffective means of reducing the rates of RBC administration. A more complex approach is required to prevent unnecessary RBC transfusions.

Targeted Training for Subspecialist Care in Children With Medical Complexity

BACKGROUND

Children with medical complexity (CMC) are prone to medical errors and longer hospital stays, while residents do not feel prepared to provide adequate medical care for this vulnerable population. No educational guidance for the training of future pediatric tertiary care specialists outside their field of expertise involving the multidisciplinary care of CMC exists. We investigated pediatric residents past educational needs and challenges to identify key learning content for future training involving care for CMC.

METHODS

This was a prospective mixed-methods study at a single pediatric tertiary care center. Qualitative semi-structured interviews with residents were conducted, submitted to thematic content analysis, linked to the American Board of Pediatrics (ABP) general pediatrics content outline, and analyzed with importance performance analysis (IPA). Quantitative validation was focused on key themes of pediatric nephrology within the scope of an online survey among pediatric residents and specialists.

RESULTS

A total of 16 interviews, median duration 69 min [interquartile range IQR 35], were conducted. The 280 listed themes of the ABP general pediatrics content outline were reduced to 165 themes, with 86% (theoretical) knowledge, 12% practical skills, and 2% soft skills. IPA identified 23 knowledge themes to be of high importance where improvement is necessary and deemed fruitful. Quantitative validation among 84 residents and specialists (response rate 55%) of key themes in nephrology yielded high agreement among specialists in pediatric nephrology but low interrater agreement among trainees and "trained" non-nephrologists. The occurrence of themes in the qualitative interviews and their calculated importance in the quantitative survey were highly correlated (tau = 0.57, p = 0.001). Two clusters of high importance for other pediatric specialties emerged together with a contextual cluster of frequent encounters in both in- and outpatient care.

CONCLUSION

Regarding patient safety, this study revealed the heterogeneous aspects and the importance of training future pediatric tertiary care specialists outside their field of expertise involving the multidisciplinary care of CMC. Our results may lay the groundwork for future detailed analysis and development of training boot camps that might be able to aid the improvement of patient safety by decreasing preventable harm by medical errors, especially for vulnerable patient groups, such as CMC in tertiary care pediatrics.