Evaluation of Trainee Competency with Point-of-Care Ultrasonography (POCUS): a Conceptual Framework and Review of Existing Assessments

Accuracy of focused cardiac ultrasound interpretation among emergency and critical care medicine residents in Ethiopia: A multi-center cross-sectional study

Introduction

Focused cardiac ultrasound (FoCUS) has emerged as a valuable tool in emergency and critical care medicine, allowing for rapid assessment of cardiac function and structure at the bedside. This rapid diagnostic technique holds particular promise in resource-limited settings like Ethiopia, where access to standard echocardiography may be limited and delayed. However, the accuracy of FoCUS interpretation is highly dependent on the operator's skills and expertise. To inform the design of effective interventions, the study aimed to assess the accuracy of FoCUS interpretation and associated factors among senior Emergency Medicine residents at two large referral teaching hospitals in Ethiopia.

Methods

A cross-sectional study was conducted from October to December 2023 among 80 residents at Tikur Anbessa Specialized Hospital and St. Paul's Hospital Millennium Medical College. To assess diagnostic accuracy, 15 pre-selected cardiac ultrasound videos (normal and pathological cases) were selected from American College of Emergency Physicians website and the PoCUS Atlas, and accurate interpretation was defined as correctly answering at least 12 out of 15 readings. A binary logistic regression model was fitted to identify significant factors at the 5% level of significance, where significant results were interpreted using adjusted odds ratio (AOR) with 95% confidence interval (CI).

Result

The overall accuracy in interpreting FoCUS findings was 47.5% (95% CI: 38.8-60.0%), with highest for collapsing Inferior Vena Cava (91.3%) and standstill (90.0%), and lowest for Regional Wall Motion Abnormality of Left Ventricle (46.3%). Residents who received training (AOR=4.14, 95%CI:1.32-13.04, p = 0.015), perceived themselves as skilled (AOR=4.81, 95%CI=1.06-21.82, p = 0.042), and felt confident in acquiring and interpretation (AOR=3.16, 95%CI=1.01-9.82, p = 0.047) demonstrated significantly higher accuracy.

Conclusion

The study identified a low overall accuracy in FoCUS interpretation, with accuracy improving with training and better perceived skill and confidence. Training programs with simulation, continuous education, and mentorship are crucial to enhance these critical skills.

Critical Advances for Democratizing Ultrasound Diagnostics in Human and Veterinary Medicine

The democratization of ultrasound imaging refers to the process of making ultrasound technology more accessible. Traditionally, ultrasound imaging has been predominately used in specialized medical facilities by trained professionals. Advancements in technology and changes in the health-care landscape have inspired efforts to broaden the availability of ultrasound imaging to various settings such as remote and resource-limited areas. In this review, we highlight several key factors that have contributed to the ongoing democratization of ultrasound imaging, including portable and handheld devices, recent advancements in technology, and training and education. Examples of diagnostic point-of-care ultrasound (POCUS) imaging used in emergency and critical care, gastroenterology, musculoskeletal applications, and other practices are provided for both human and veterinary medicine. Open challenges and the future of POCUS imaging are presented, including the emerging role of artificial intelligence in technology development.

The Role of POCUS to Face COVID-19: A Narrative Review

COVID-19 has been a challenging outbreak to face, with millions of deaths among the globe. Acute respiratory failure due to interstitial pneumonia was the leading cause of death other than prothrombotic activation and complications. Lung ultrasound (LUS) and point-of-care ultrasound (POCUS) are widely used not only to triage, to identify, and to monitor lungs involvement but also to assess hemodynamic status and thrombotic and hemorrhagic complications, mainly in critically ill patients. POCUS has gained growing consideration due to its bedside utilization, reliability, and reproducibility even in emergency settings especially in unstable patients. In this narrative review, we aim to describe LUS and POCUS utilization in COVID-19 infection based on the literature found on this topic. We reported the LUS patterns of COVID-19 pulmonary infection, the diagnostic accuracy with respect to CT lung scan, its prognostic value, the variety of scores and protocols proposed, and the utilization of POCUS to investigate the extra-lung complications.

Training, Competency, and Interdisciplinary Collaboration in Point-of-Care Ultrasound

Point-of-care ultrasound can provide critical diagnostic information and add to procedural safety. As with any clinical skill, however, it must be applied by an adequately trained provider, with systems to ensure safety. Training can include a mix of hands-on training with traditional didactics, online coursework, and simulation, but each carries its own costs and benefits worth of review. Following training it is essential to think about assessment of competency in point-of-care ultrasound to reflect the combination of cognitive and procedural skills that makes up the practice. Within the frameworks described, expansion can be expected across specialty and professional boundaries.

Change in economy of ultrasound probe motion among general medicine trainees

OBJECTIVES

To observe change in economy of 9 ultrasound probe movement metrics among internal medicine trainees during a 5-day training course in cardiac point of care ultrasound (POCUS).

METHODS

We used a novel probe tracking device to record nine features of ultrasound probe movement, while trainees and experts optimized ultrasound clips on the same volunteer patients. These features included translational movements, gyroscopic movements (titling, rocking, and rotation), smoothness, total path length, and scanning time. We determined the adjusted difference between each trainee's movements and the mean value of the experts' movements for each patient. We then used a mixed effects model to trend average the adjusted differences between trainees and experts throughout the 5 days of the course.

RESULTS

Fifteen trainees were enrolled. Three echocardiographer technicians and the course director served as experts. Across 16 unique patients, 294 ultrasound clips were acquired. For all 9 movements, the adjusted difference between trainees and experts narrowed day-to-day (p value < 0.05), suggesting ongoing improvement during training. By the last day of the course, there were no statistically significant differences between trainees and experts in translational movement, gyroscopic movement, smoothness, or total path length; yet on average trainees took 28 s (95% CI [14.7-40.3] seconds) more to acquire a clip.

CONCLUSIONS

We detected improved ultrasound probe motion economy among internal medicine trainees during a 5-day training course in cardiac POCUS using an inexpensive probe tracking device. Objectively quantifying probe motion economy may help assess a trainee's level of proficiency in this skill and individualize their POCUS training.

The Role of Pocus in Acute Respiratory Failure: A Narrative Review on Airway and Breathing Assessment

Acute respiratory failure (ARF) is a challenging condition that clinicians, especially in emergency settings, have to face frequently. Especially in emergency settings, many underlying diseases can lead to ARF and life-threatening conditions have to be promptly assessed and correctly treated to avoid unfavorable outcomes. In recent years, point-of-care ultrasound (POCUS) gained growing consideration due to its bedside utilization, reliability and reproducibility even in emergency settings especially in unstable patients. Research on POCUS application to assess ARF has been largely reported mainly with observational studies showing heterogeneous results from many different applications. This narrative review describes the wide potentiality of POCUS to face airways and breathing life-threatening conditions such as upper airway management, pulmonary and pleural pathologies and diaphragm impairment. We conducted extensive research of the literature to report from major studies to case reports deemed useful in practical clinical utilization of POCUS in ARF. Due to the huge amount of the literature found, we focused on airways and breathing assessment trying to systematize the evidence according to clinical care of ARF in emergency settings. Further studies, possibly trials, should determine how POCUS is crucial in clinical practice in terms of standard of care improvements, patient safety and cost-benefit analysis.

Development of a hybrid point-of-care ultrasound curriculum for first year medical students in a rural medical education program: a pilot study

BACKGROUND

The field of point-of-care ultrasound (POCUS) has advanced in recent decades due to the benefits it holds for medical providers. However, aspiring POCUS practitioners require adequate training. Unfortunately, there remains a paucity of resources to deliver this training, particularly in rural and underserved areas. Despite these barriers, calls for POCUS training in undergraduate medical education are growing, and many medical schools now deliver some form of POCUS education. Our program lacked POCUS training; therefore, we developed and implemented a POCUS curriculum for our first-year medical students.

METHODS

We developed a POCUS curriculum for first year medical students in a rural medically underserved region of the United States. To evaluate our course, we measured learning outcomes, self-reported confidence in a variety of POCUS domains, and gathered feedback on the course with a multi-modal approach: an original written pre- and post-test, survey tool, and semi-structured interview protocol, respectively.

RESULTS

Student (n=24) knowledge of POCUS significantly increased (pre-test average score = 55%, post-test average score = 79%, P<0.0001), and the course was well received based on student survey and interview feedback. In addition, students reported increased confidence toward a variety of knowledge and proficiency domains in POCUS use and their future clinical education and practice.

CONCLUSIONS

Despite a lack of consensus in POCUS education, existing literature describes many curricular designs across institutions. We leveraged a combination of student initiatives, online resources, remote collaborations, local volunteers, and faculty development to bring POCUS to our institution in a rural and medically underserved region. Moreover, we demonstrate positive learning and experiential outcomes that may translate to improved outcomes in students' clinical education and practice. Further research is needed to evaluate the psychomotor skills, broader learning outcomes, and clinical performance of students who take part in our POCUS course.

FAST skill assessment from kinematics data using convolutional neural networks

PURPOSE

FAST is a point of care ultrasound study that evaluates for the presence of free fluid, typically hemoperitoneum in trauma patients. FAST is an essential skill for Emergency Physicians. Thus, it requires objective evaluation tools that can reduce the necessity of direct observation for proficiency assessment. In this work, we use deep neural networks to automatically assess operators' FAST skills.

METHODS

We propose a deep convolutional neural network for FAST proficiency assessment based on motion data. Prior work has shown that operators demonstrate different domain-specific dexterity metrics that can distinguish novices, intermediates, and experts. Therefore, we augment our dataset with this domain knowledge and employ fine-tuning to improve the model's classification capabilities. Our model, however, does not require specific points of interest (POIs) to be defined for scanning.

RESULTS

The results show that the proposed deep convolutional neural network can classify FAST proficiency with 87.5% accuracy and 0.884, 0.886, 0.247 sensitivity for novices, intermediates, and experts, respectively. It demonstrates the potential of using kinematics data as an input in FAST skill assessment tasks. We also show that the proposed domain-specific features and region fine-tuning increase the model's classification accuracy and sensitivity.

CONCLUSIONS

Variations in probe motion at different learning stages can be derived from kinematics data. These variations can be used for automatic and objective skill assessment without prior identification of clinical POIs. The proposed approach can improve the quality and objectivity of FAST proficiency evaluation. Furthermore, skill assessment combining ultrasound images and kinematics data can provide a more rigorous and diversified evaluation than using ultrasound images alone.

A Program of Assessment Model for Point-of-Care Ultrasound Training for Pediatric Critical Care Providers: A Comprehensive Approach to Enhance Competency-Based Point-of-Care Ultrasound Training

Point-of-care ultrasound (POCUS) is increasingly accepted in pediatric critical care medicine as a tool for guiding the evaluation and treatment of patients. POCUS is a complex skill that requires user competency to ensure accuracy, reliability, and patient safety. A robust competency-based medical education (CBME) program ensures user competency and mitigates patient safety concerns. A programmatic assessment model provides a longitudinal, holistic, and multimodal approach to teaching, assessing, and evaluating learners. The authors propose a fit-for-purpose and modifiable CBME model that is adaptable for different institutions' resources and needs for any intended competency level. This educational model drives and supports learning, ensures competency attainment, and creates a clear pathway for POCUS education while enhancing patient care and safety.

Development of a novel observed structured clinical exam to assess clinical ultrasound proficiency in undergraduate medical education

OBJECTIVES

A pilot study was performed to develop and test an observed structured clinical exam (OSCE) for clinical ultrasound in second-year medical students. The goal was to assess a longitudinal clinical ultrasound curriculum for medical students and to help determine readiness to perform ultrasound during clinical clerkships.

METHODS

The OSCE contained 40 tasks over 30 min in a one-to-one examiner to examinee environment using standardized patients covering cardiac, pulmonary, and inferior vena cava (IVC) ultrasound exams along with 6 critical diagnoses. Examinees were assessed using a binary checklist approach. A two-way ANOVA analysis was performed to determine if there were differences between the day and session the OSCE was administered. Results are presented as mean ± standard deviation.

RESULTS

One hundred fifty-two students were tested with an overall mean score of 64.9 ± 17.6%. Scores between the cardiac, IVC, and lung sections varied-67.8% ± 18.8%, 62.4% ± 26.2%, and 57.1% ± 20.6%, respectively. One hundred twenty-six (82.9%) answered at least one critical diagnosis incorrectly. Students in the late session performed better than the early session (1: 60% vs 2: 69%, p = .001).

CONCLUSIONS

Students performed better in later sessions. Additionally, the number of questions left blank at the end of the exam suggests that the length of the OSCE should be evaluated. Incorporating critical diagnoses was challenging for examinees. The proposed OSCE is a valuable assessment tool that could be adapted to assess student's readiness to use clinical ultrasound prior to clerkships.

Evaluation of an emergency medicine point-of-care ultrasound curriculum adapted for a resource-limited setting in Guyana

BACKGROUND

This is a cohort pilot study of senior emergency medicine residents and residency-trained emergency medicine registrars practicing in an urban tertiary academic hospital in Guyana in South America. The primary aim was to assess the effectiveness of the current adapted residency ultrasound training curriculum and guide future ultrasound-specific continuing medical education. Ultrasound image acquisition, interpretation, and integration of ultrasound findings into clinical medical decision-making competency were assessed in a multimodal fashion: a written assessment and a practical assessment.

METHODS

This was a cross-sectional cohort pilot study of senior emergency medicine residents and registrars (residency graduates) practicing in Guyana, a low-middle-income country. Ultrasound image acquisition and interpretation competency were assessed in a multimodal fashion: a written assessment and a practical assessment. The results will be reported as simple percentages. Participants will be deemed competent if the combined score is greater than 80% on the assessment.

RESULTS

All senior residents and graduates of the residency program were competent in the core point-of-care ultrasound applications. The senior residents averaged 89% proficiency, and the residency graduates subdivided based on years since graduation averaged 87-100% proficiency. The more experienced providers performed better on the practical portion of the evaluation. Evaluating the composite analysis of all the participants revealed the Extended Focused Assessment with Sonography in Trauma (EFAST) exam (96%) and cardiac exam (93%) were done with the most proficiency.

CONCLUSION

The assessment results of this pilot study suggest that the current residency ultrasound training curriculum adapted to the resources available is associated with sustained competency after graduation. There was minor attrition of knowledge amongst the senior physicians, but all senior residents and all participating residency graduates were competent in the core ultrasound applications.

Point-of-Care-ultrasound in undergraduate medical education: a scoping review of assessment methods

BACKGROUND

Point-of-Care-Ultrasound (POCUS) curricula have rapidly expanded in undergraduate medical education (UME). However, the assessments used in UME remain variable without national standards. This scoping review characterizes and categorizes current assessment methods using Miller's pyramid for skills, performance, and competence of POCUS in UME. A structured protocol was developed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR). A literature search of MEDLINE was performed from January 1, 2010, to June 15, 2021. Two independent reviewers screened all titles and abstracts for articles that met inclusion criteria. The authors included all POCUS UME publications in which POCUS-related knowledge, skills, or competence were taught and objectively assessed. Articles were excluded if there were no assessment methods used, if they exclusively used self-assessment of learned skills, were duplicate articles, or were summaries of other literature. Full text analysis and data extraction of included articles were performed by two independent reviewers. A consensus-based approach was used to categorize data and a thematic analysis was performed.

RESULTS

A total of 643 articles were retrieved and 157 articles met inclusion criteria for full review. Most articles (n = 132; 84%) used technical skill assessments including objective structured clinical examinations (n = 27; 17%), and/or other technical skill-based formats including image acquisition (n = 107; 68%). Retention was assessed in n = 98 (62%) studies. One or more levels of Miller's pyramid were included in 72 (46%) articles. A total of four articles (2.5%) assessed for students' integration of the skill into medical decision making and daily practice.

CONCLUSIONS

Our findings demonstrate a lack of clinical assessment in UME POCUS that focus on integration of skills in daily clinical practice of medical students corresponding to the highest level of Miller's Pyramid. There exists opportunities to develop and integrate assessment that evaluate higher level competencies of POCUS skills of medical students. A mixture of assessment methods that correspond to multiple levels of Miller's pyramid should be used to best assess POCUS competence in UME.

External validation of the ultrasound competency assessment tool

Objective

Point-of-care ultrasound (POCUS) is a core component of emergency medicine (EM) residency training. No standardized competency-based tool has gained widespread acceptance. The ultrasound competency assessment tool (UCAT) was recently derived and validated. We sought to externally validate the UCAT in a 3-year EM residency program.

Methods

This was a convenience sample of PGY-1 to -3 residents. Utilizing the UCAT and an entrustment scale, as described in the original study, six different evaluators split into two groups graded residents in a simulated scenario involving a patient with blunt trauma and hypotension. Residents were asked to perform and interpret a focused assessment with sonography in trauma (FAST) examination and apply the findings to the simulated scenario. Demographics, prior POCUS experience, and self-assessed competency were collected. Each resident was evaluated simultaneously by three different evaluators with advanced ultrasound training utilizing the UCAT and entrustment scales. Intraclass correlation coefficient (ICC) between evaluators was calculated for each assessment domain; analysis of variance was used to compare UCAT performance and PGY level and prior POCUS experience.

Results

Thirty-two residents (14 PGY-1, nine PGY-2, and nine PGY-3) completed the study. Overall, ICC was 0.9 for preparation, 0.57 for image acquisition, 0.3 for image optimization, and 0.46 for clinical integration. There was moderate correlation between number of FAST examinations performed and entrustment and UCAT composite scores. There was poor correlation between self-reported confidence and entrustment and UCAT composite scores.

Conclusions

We had mixed results in our attempt to externally validate the UCAT with poor correlation between faculty and moderate to good correlation with faculty to diagnostic sonographer. More work is needed to validate the UCAT before adoption.

The use of simulation in medical ultrasound: Current perspectives on applications and practical implementation (WFUMB state-of-the-art paper)

Simulation has been shown to improve clinical learning outcomes, speed up the learning process, and improve trainee confidence, while taking the pressure off initial face-to-face patient clinical areas. The second part of The World Federation for Ultrasound in Medicine and Biology state-of-the-art paper on the use of simulators provides a general approach on the practical implementation. The importance of needs assessment before developing a simulation-based training program is outlined. We describe the current practical implementation and critically analyze how simulators can be integrated into complex task scenarios to train small or large groups. A wide range of simulation equipment is available especially for those seeking interventional ultrasound training, ranging from animal tissue models, simple synthetic phantoms, to sophisticated high-fidelity simulation platforms using virtual reality. Virtual reality simulators provide feedback and thereby allow trainees to not only to practice their motor skills and hand eye coordination but also to interact with the simulator. Future developments will integrate more elements of automated assessment and artificial intelligence, thereby enabling enhanced realistic training experience and improving skill transfer into clinical practice.

Unravelling the skillset of point-of-care ultrasound: a systematic review

BACKGROUND

The increasing number of physicians that are trained in point-of-care ultrasound (POCUS) warrants critical evaluation and improvement of current training methods. Performing POCUS is a complex task and it is unknown which (neuro)cognitive mechanisms are most important in competence development of this skill. This systematic review was conducted to identify determinants of POCUS competence development that can be used to optimize POCUS training.

METHODS

PubMed, Web of Science, Cochrane Library, Emcare, PsycINFO and ERIC databases were searched for studies measuring ultrasound (US) skills and aptitude. The papers were divided into three categories: "Relevant knowledge", "Psychomotor ability" and 'Visuospatial ability'. The 'Relevant knowledge' category was further subdivided in 'image interpretation', 'technical aspects' and 'general cognitive abilities'. Visuospatial ability was subdivided in visuospatial subcategories based on the Cattell-Horn-Carroll (CHC) Model of Intelligence v2.2, which includes visuospatial manipulation and visuospatial perception. Post-hoc, a meta-analysis was performed to calculate pooled correlations.

RESULTS

26 papers were selected for inclusion in the review. 15 reported on relevant knowledge with a pooled coefficient of determination of 0.26. Four papers reported on psychomotor abilities, one reported a significant relationship with POCUS competence. 13 papers reported on visuospatial abilities, the pooled coefficient of determination was 0.16.

CONCLUSION

There was a lot of heterogeneity in methods to assess possible determinants of POCUS competence and POCUS competence acquisition. This makes it difficult to draw strong conclusions on which determinants should be part of a framework to improve POCUS education. However, we identified two determinants of POCUS competence development: relevant knowledge and visuospatial ability. The content of relevant knowledge could not be retrieved in more depth. For visuospatial ability we used the CHC model as theoretical framework to analyze this skill. We could not point out psychomotor ability as a determinant of POCUS competence.

Development and Validation of a Point-of-Care-Ultrasound Image Quality Assessment Tool: The POCUS IQ Scale

OBJECTIVES

We aimed to develop a standardized scoring tool to measure point-of-care ultrasound (POCUS) image quality and to determine validity evidence for its use to assess lung ultrasound image quality.

METHODS

The POCUS Image Quality (POCUS IQ) scale was developed by POCUS-trained physicians to assess sonographers' image acquisition skills by evaluating image quality for any POCUS application. The scale was piloted using lung images of healthy standardized patients acquired by three expert sonographers compared to three novices before and after training. All images (experts, novices pre-training, novices post-training) were scored on the POCUS IQ scale by three blinded POCUS-trained physicians. Reliability was assessed with fully-crossed generalizability and decision studies. Validity was assessed using Messick's framework.

RESULTS

Content validity was supported by the tool's development process of literature review, expert consensus, and pilot testing. Response process was supported by reviewer training and the blinded scoring process. Relation to other variables was supported by scores relating to sonographer experience: median expert score = 10.5/14 (IQR: 4), median novice pre-training score = 6/14 (IQR: 2.25), and novices' improvement after training (median post-training score = 12/14, IQR: 3.25). Internal structure was supported by internal consistency data (coefficient alpha = 0.84, omega coefficient = 0.91) and the generalizability study showing the main contributor to score variability was the sonographer (51%). The G-coefficient was 0.89, suggesting very good internal structure, however, Gwet's AC₂  was 0.5, indicating moderate interrater reliability. The D study projected a minimum of 1 reviewer and 2 patients are needed for good psychometric reliability.

CONCLUSIONS

The POCUS scale has good preliminary validity evidence as an assessment tool for lung POCUS image acquisition skills. Further studies are needed to demonstrate its utility for other POCUS applications and as a feedback tool for POCUS learners.

How much POCUS for Czech internists?

The introduction of point-of-care ultrasonography into practice in internal medicine and subsequently into the educational program for our specialty brought the need to define a curriculum for training in this method. We solve the question of "what to teach" - i.e. what core ultrasound competencies internists should learn for their practice. It is also necessary to define the procedure "how to teach" - the form and content of the education program, what the basic course should contain, and above all how the subsequent training should take place in practice. The third major problem to be solved is "who should teach", i.e. the definition of the requirements for trainers who will lead the training.

Evaluation of antenatal point-of-care ultrasound training workshops for rural/remote healthcare clinicians: a prospective single cohort study

BACKGROUND

There is limited access to life-saving antenatal ultrasound in low-resource rural and remote settings worldwide, including Australia, mainly due to shortages in skilled staff. Point-of-care ultrasound (PoCUS) offers a viable solution to this service deficit, however, rural clinicians face many barriers accessing training and professional development critical to advancing their clinical practice. Standards for PoCUS training and competency assessment are unclear. Regulation is lacking globally, allowing untrained and inexperienced clinicians to practice PoCUS clinically.

METHODS

This prospective single cohort study aimed to evaluate antenatal PoCUS training workshops for General Practitioners (GPs) and Midwives/Nurses (M/Ns) from rural/remote Australia, assessing the impact of the training on trainees' knowledge, confidence and translation of PoCUS into clinical practice. Two-day antenatal ultrasound workshops were delivered at the University of South Australia (UniSA) in 2018 and 2019 to 41 rural/remote clinicians . The training was designed and evaluated using the New world Kirkpatrick Evaluation Framework. Sixteen GPs and 25 M/Ns with mixed prior ultrasound experience were funded to attend. The course consisted of lectures interspaced with hands-on training sessions using high-fidelity simulators and live pregnant models. Pre- and post-knowledge assessments were performed. Post-workshop evaluation and follow-up surveys (3- and 6-month post-training) assessed the workshops and changes to trainees' clinical practice. A 2-day follow-up training session was conducted 12 months after the workshops for 9 trainees.

RESULTS

Pre/post knowledge testing demonstrated a 22% mean score improvement (95% CI 17.1 to 27.8, P < 0.0001). At 6 months, 62% of trainees were performing PoCUS that had assisted in patient management and clinical diagnosis, and 46% reported earlier diagnosis and changes to patient management. 74% of trainees had increased scanning frequency and 93% reported improved scanning confidence.

CONCLUSION

This study demonstrated intensive 2-day workshops can equip clinicians with valuable antenatal PoCUS skills, offering a viable solution to assist in the assessment and management of pregnant women in the rural/resource-poor setting where access to ultrasound services is limited or non-existent. Geographical isolation and lack of onsite specialist supervision poses an ongoing challenge to the continuing professional development of remote trainees and the implementation of PoCUS.

Evaluation of antenatal Point-of-Care Ultrasound (PoCUS) training: a systematic review

INTRODUCTION

There is limited access to life-saving antenatal ultrasound in rural and low-resource settings largely due to shortages in skilled staff. Studies have shown healthcare practitioners can be upskilled in PoCUS through focused training, offering a viable solution to this deficit. However, standards for training and competency assessment are unclear and regulation surrounding practice is lacking. We aimed to review published literature examining antenatal PoCUS training programs, comparing teaching approaches and study methodologies.

METHODS

A search of electronic databases EMBASE, MEDLINE and Google Scholar was conducted. Original research articles evaluating antenatal PoCUS training of healthcare professionals worldwide were identified for analysis. Articles with limited detail on the PoCUS training intervention and those describing comprehensive diagnostic training programs were excluded. Evaluations were compared against the Kirkpatrick Evaluation Framework (KEF).

RESULTS

Twenty-seven studies were included from an initial search result of 484 articles. There was considerable heterogeneity between the PoCUS training programs described. Course duration ranged from 3 hours to 2 years, with 11 of the 27 studies delivering obstetric-exclusive content. 44% trained multidisciplinary groups of health professionals. Long-term follow-up training and skills assessments were lacking in over half of the reviewed studies. Study quality and reporting detail varied, but overall beneficial outcomes were reported with 3/4s of the studies reaching upper KEF levels 3 and 4.

CONCLUSION

PoCUS performed by upskilled healthcare professionals offers an attractive solution to the problem of inequitable access to antenatal ultrasound. A review of available literature highlighted a paucity of comparable high-quality studies needed to establish a stronger evidence base for antenatal PoCUS, and a need to standardise training and competency assessment. This review may inform educators, researchers and policy-makers on existing training formats and methodologies to assist in establishing best practice antenatal PoCUS training methods for safe service delivery by remote healthcare professionals.

Neonatal POCUS: Embracing our modern day "stethoscope"

The dramatic technologic advancements seen in ultrasound have accelerated the growth of point-of-care ultrasound (POCUS) in medicine. Neonatology has lagged behind other pediatric and adult specialties in incorporating POCUS into clinical practice despite there being numerous applications in cardiac and non-cardiac arenas. Widely available training programs are aiding in improving this situation but significantly more structure and orchestration for neonatal POCUS dissemination will be needed to fully actualize the potential for POCUS to augment its widespread clinical application.

Competence of anesthesiology residents following a longitudinal point-of-care ultrasound curriculum

PURPOSE

Point-of-care ultrasound (POCUS) facilitates diagnostic, procedural, and resuscitative applications in anesthesiology. Structured POCUS curricula improve learner satisfaction, test scores, and clinical management, but the learning curve towards competency and retention of skills over time remain unknown.

METHODS

We conducted a prospective observational study to determine when anesthesiology trainees enrolled in a POCUS curriculum achieve competency in POCUS skills. We also investigated the learning curve of trainees' competency using a POCUS-specific competency-based medical education assessment. The structured, longitudinal POCUS curriculum included online lectures, journal articles, live model scanning sessions, video review of cases, and a portfolio of supervised scans. Point-of-care ultrasound scanning sessions on standardized patients were conducted in the simulation lab for 2.5 hr a week and each resident completed eight sessions (20 hr) per academic year. At each scanning session, timed image acquisition scores were collected and POCUS skills entrustment scale evaluations were conducted. The primary outcome was the number of supervised scans and sessions required to achieve a mean entrustment score of 4 ("may use independently"). Secondary outcomes included image acquisition scores and retention of skills after six months.

RESULTS

The mean (standard deviation) number of supervised scans required for trainees (n = 29) to reach a mean entrustment score of ≥ 4 was 36 (10) scans over nine sessions for rescue echo. A mean entrustment score of ≥ 4 was observed for lung ultrasound after a mean (SD) of 8 (3) scans over two sessions.

CONCLUSIONS

Our study shows that anesthesiology residents can achieve competence in rescue echo and lung ultrasound through participation in a structured, longitudinal POCUS curriculum, and outlines the learning curve for progression towards competency.

The P2Network-Advancing Pediatric Emergency Care With Point-of-Care Ultrasound

ABSTRACT

Over the last 2 decades, the use of point-of-care ultrasound (POCUS) in pediatric emergency medicine (PEM) has grown exponentially. In 2014, a group of PEM POCUS leaders met and formed the P2Network. The P2Network provides a platform to build collaborative relationships and share expertise among members from various countries and practice settings. It works with educators and researchers within and outside of the field to advance POCUS practice in PEM. As an organization, the P2Network promotes the evidence-based application of POCUS to facilitate and improve care in the PEM setting and addresses issues related to integration of the PEM POCUS practitioner in this nascent field. The P2Network is building and augmenting its infrastructure for PEM POCUS research and education and has already made some progress in the areas, with published manuscripts and ongoing clinical research studies under its sponsorship. Future goals include developing a PEM POCUS research agenda, formalizing teaching and assessment of PEM POCUS skills, and implementing multicenter research studies on potentially high impact applications.

Teaching point-of-care transfontanellar ultrasound for pediatricians and medical students

OBJECTIVES

The authors aim to evaluate the "point-of-care" transfontanellar ultrasound (TU) as an extension to pediatric physical examination and suggest a TU teaching protocol.

METHODS

The students were randomly divided into two groups, group A (12 participants) and group B (15 participants). The first group only received theoretical training, while the second group received theoretical and practical training. A third group, group C, included 15 pediatricians and interns who also received theoretical and practical training. All the participants underwent multiple-choice testing before and after a four-hour short course on TU. Six months later, another evaluation was performed to analyze the retained knowledge. Furthermore, a questionnaire based on the Likert scale was administered to evaluate satisfaction.

RESULTS

The cognitive evaluation (maximum score=10 points) before and after training increased in group A from 4,0±1,04 to 7,5±1,2 (p<0.001) and, 6 months later, to 6,5±1,16 (p<0.003); in group B from 3,8±1,24 to 8,8±1,01 (p<0.001) and, 6 months later, to 8,46±0,91 (p<0.001); and in group C from 6,0±0,75 to 9,0±0,75 (p<0.001) and, 6 months later, to 8,8±0,77 (p<0.001). The average satisfaction estimated by the Likert scale was over 80% for all questions.

CONCLUSION

Cognitive assessment before and after classes and training reveals progress in learning, with knowledge retention in 6 months. Theoretical-practical courses are well accepted.

Skills Classification in Cardiac Ultrasound with Temporal Convolution and Domain Knowledge Using a Low-Cost Probe Tracker

As point-of-care ultrasound (POCUS) becomes more integrated into clinical practice, it is essential to address all aspects of ultrasound operator proficiency. Ultrasound proficiency requires the ability to acquire, interpret and integrate bedside ultrasound images. The difference in image acquisition psychomotor skills between novice (trainee) and expert (instructor) ultrasonographer has not been described. We created an inexpensive system, called Probe Watch, to record probe motion and assess image acquisition in cardiac POCUS using an inertial measurement device and software for data recording based on open-source components. We designed a temporal convolutional network for skills classification from probe motion that integrates clinical domain knowledge. We further designed data augmentation methods to improve its generalization. Subsequently, we validated the setup and assessment method on a set of novice and expert sonographers performing cardiac ultrasound in a simulation-based training environment. The proposed methods classified participants as novice or expert with areas under the receiver operating characteristic curve of 0.931 and 0.761 for snippets and trials, respectively. Integrating domain knowledge into the neural network had added value. Furthermore, we identified the most discriminative features for assessment. Probe Watch quantifies motion during cardiac ultrasound and provides insight into probe motion behavior. It may be deployed during cardiac ultrasound training to monitor learning curves objectively and automatically.

Pediatric emergency medicine fellowship point-of-care ultrasound training in 2020

OBJECTIVES

The primary objective was to survey pediatric emergency medicine (PEM) leaders and fellows regarding point-of-care ultrasound (POCUS) training in PEM fellowship programs, including teaching methods, training requirements, and applications taught. Secondary objectives were to compare fellows' and program leaders' perceptions of fellow POCUS competency and training barriers.

METHODS

This was a cross-sectional survey of U.S. PEM fellows and fellowship program leaders of the 78 fellowship programs using two online group-specific surveys exploring five domains: program demographics; training strategies and requirements; perceived competency; barriers, strengths, and weaknesses of POCUS training; and POCUS satisfaction.

RESULTS

Eighty-three percent (65/78) of programs and 53% (298/558) of fellows responded. All participating PEM fellowship programs included POCUS training in their curriculum. Among the 65 programs, 97% of programs and 92% of programs utilized didactics and supervised scanning shifts as educational techniques, respectively. Sixty percent of programs integrated numerical benchmarks and 49% of programs incorporated real-time, hands-on demonstration as training requirements. Of the 19 POCUS applications deemed in the literature as core requirements for fellows, at least 75% of the 298 fellows reported training in 13 of those applications. Although less than half of fellows endorsed competency for identifying intussusception, ultrasound-guided pericardiocentesis, and transvaginal pregnancy evaluation, a higher proportion of leaders reported fellows as competent for these applications (40% vs. 68%, p ≤ 0.001; 21% vs. 39%, p = 0.003; and 21% vs. 43%, p ≤ 0.001). Forty-six percent of fellows endorsed a lack of PEM POCUS evidence as a training barrier compared to 31% of leaders (p = 0.02), and 39% of leaders endorsed a lack of local financial support as a training barrier compared to 23% of fellows (p = 0.01).

CONCLUSIONS

Although most PEM fellowship programs provide POCUS training, there is variation in content and requirements. Training does conform to many of the expert recommended guidelines; however, there are some discrepancies and perceived barriers to POCUS training remain.

Use of Hand-motion Analysis to Assess Competence and Skill Decay for Cardiac and Lung Point-of-care Ultrasound

OBJECTIVES

Assessment of competence in technical skills, including point-of-care ultrasound (POCUS), is required before a novice can safely perform the skill independently. Ongoing assessment of competence is also required because technical skills degrade over time, especially when they are infrequently performed or complex. Hand-motion analysis (HMA) is an objective assessment tool that has been used to evaluate competency in many technical skills. The purpose of this study was to demonstrate the feasibility and validity of HMA as an assessment tool for competence in both simple and complex technical skills as well as skill degradation over time.

METHODS

This prospective cohort study included 36 paramedics with no POCUS experience and six physicians who were fellowship trained in POCUS. The novices completed a 4-hour didactic and hands-on training program for cardiac and lung POCUS. HMA measurements, objective structured clinical examinations (OSCE), and written examinations were collected for novices immediately before and after training as well as 2 and 4 months after training. Expert HMA metrics were also recorded.

RESULTS

Expert HMA metrics for cardiac and lung POCUS were significantly better than those of novices. After completion of the training program, the novices improved significantly in all HMA metrics, knowledge test scores, and OSCE scores. Novices showed skill degradation in cardiac POCUS based on HMA metrics and OSCE scores while lung POCUS image acquisition skills were preserved. Novices deemed competent by OSCE score performed significantly better in HMA metrics than those not deemed competent.

CONCLUSION

We have demonstrated that HMA is a feasible and valid tool for assessment of competence in technical skills and can also evaluate skill degradation over time. Skill degradation appears more apparent in complex skills, such as cardiac POCUS. HMA may provide a more efficient and reliable assessment of technical skills, including POCUS, when compared to traditional assessment tools.

Evaluating a Web-based Point-of-care Ultrasound Curriculum for the Diagnosis of Intussusception

OBJECTIVES

Intussusception is a pediatric medical emergency that can be difficult to diagnose. Radiology-performed ultrasound is the diagnostic study of choice but may lead to delays due to lack of availability. Point-of-care ultrasound for intussusception (POCUS-I) studies have shown excellent accuracy and reduced lengths of stay, but there are limited POCUS-I training materials for pediatric emergency medicine (PEM) providers.

METHODS

We performed a prospective cohort study assessing PEM physicians undergoing a primarily Web-based POCUS-I curriculum. We developed the POCUS-I curriculum using Kern's six-step model. The curriculum included a Web-based module and a brief, hands-on practice that was developed with a board-certified pediatric radiologist. POCUS-I technical skill, knowledge, and confidence were determined by a direct observation checklist, multiple-choice test, and a self-reported Likert-scale survey, respectively. We assessed participants immediately pre- and postcourse as well as 3 months later to assess for retention of skill, knowledge, and confidence.

RESULTS

A total of 17 of 17 eligible PEM physicians at a single institution participated in the study. For the direct observation skills test, participants scored well after the course with a median (interquartile range [IQR]) score of 20 of 22 (20-21) and maintained high scores even after 3 months (20 [20-21]). On the written knowledge test, there was significant improvement from 57.4% (95% CI = 49.8 to 65.2) to 75.3% (95% CI = 68.1 to 81.6; p < 0.001) and this improvement was maintained at 3 months at 81.2% (95% CI = 74.5 to 86.8). Physicians also demonstrated improved confidence with POCUS-I after exposure to the curriculum, with 5.9% reporting somewhat or very confident prior to the course to 76.5% both after the course and after 3 months (p < 0.001).

CONCLUSION

After a primarily Web-based curriculum for POCUS-I, PEM physicians performed well in technical skill in POCUS-I and showed improvement in knowledge and confidence, all of which were maintained over 3 months.

Building and Maintaining an Ultrasound Program: It Takes a Village

Building and maintaining a successful point-of-care ultrasound program is a complex process that involves establishing an ecosystem between three unique but overlapping domains: ultrasound equipment, ultrasound users, and the health care system. By highlighting the different areas of focus and each of the key stakeholders and components, a group can ensure adequate attention is paid to all aspects of point-of-care ultrasound program development in nephrology.

Handheld Ultrasound Device Usage and Image Acquisition Ability Among Internal Medicine Trainees: A Randomized Trial

BACKGROUND

There is insufficient knowledge about how personal access to handheld ultrasound devices (HUDs) improves trainee learning with point-of-care ultrasound (POCUS).

OBJECTIVE

To assess whether HUDs, alongside a yearlong lecture series, improved trainee POCUS usage and ability to acquire images.

METHODS

Internal medicine intern physicians (n = 47) at a single institution from 2017 to 2018 were randomized 1:1 to receive personal HUDs (n = 24) for patient care/self-directed learning vs no-HUDs (n = 23). All interns received a repeated lecture series on cardiac, thoracic, and abdominal POCUS. Main outcome measures included self-reported HUD usage rates and post-intervention assessment scores using the Rapid Assessment of Competency in Echocardiography (RACE) scale between HUD and no-HUD groups.

RESULTS

HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in non-HUD arm (P = .66). There was no relationship between the number of self-reported examinations per week and a trainee's post-intervention RACE score (rho = 0.022, P = .95). HUD interns did not have significantly higher post-intervention RACE scores (median HUD score 17.0 vs no-HUD score 17.8; P = .72). Trainee confidence with cardiac POCUS did not correlate with RACE scores.

CONCLUSIONS

Personal HUDs without direct supervision did not increase the amount of POCUS usage or improve interns' acquisition abilities. Interns who reported performing more examinations per week did not have higher RACE scores. Improved HUD access and lectures without additional feedback may not improve POCUS mastery.

Skill retention with ultrasound curricula

Background

Implementation of a point of care ultrasound curricula is valuable, but optimal integration for internal medicine residency is unclear. The purpose of this study was to evaluate if a structured ultrasound curriculum vs. structured ultrasound curriculum plus supervised thoracic ultrasounds would improve internal medicine residents’ skill and retention 6 and 12 months from baseline.

Methods

We conducted a randomized controlled study evaluating internal medical residents’ skill retention of thoracic ultrasound using a structured curriculum (control, n = 14) vs. structured curriculum plus 20 supervised bedside thoracic ultrasounds (intervention, n = 14). We used a stratified randomization based on program year. All subjects attended a half-day course that included 5 lectures and hands-on sessions at baseline. Assessments included written and practical exams at baseline, immediately post-course and at 6 and 12 months. Scores are reported as a percentage for the number of correct responses/number of questions (range 0–100%). The Mann Whitney U and the Friedman tests were used for analyses.

Results

Twenty-eight residents were enrolled. Two subjects withdrew prior to the 6-month exams. Written exam scores for all subjects improved, baseline median (IQR) 60 (46.47 to 66.67) post-course 80 (65 to 86.67), 6-month 80 (66.67 to 86.67) and 12-month 86.67 (80 to 88.34), p = <0.001. All subjects practical exam scores median (IQR) significantly improved, baseline 18.18 (7.95 to 32.95), post-course 59.09 (45.45 to 70.45), 6 month 71.74 (60.87 to 82.61) and 12-month 76.09 (65.22 to 88.05), p = <0.001. Comparing the control group to the intervention group, there were statistically significant higher scores, median (IQR), in the intervention group on the practical exam at 6 months 63.05 (48.92 to 69.57) vs. 82.61(72.83 to89.13), p = <0.001.

Conclusion

In this cohort, internal medicine residents participating in a structured thoracic ultrasound course plus 20-supervised ultrasounds achieved higher practical exam scores long-term compared to controls.

Teaching vs learning: Impact of deliberate practice and formative feedback on developing point of care ultrasound skills

PURPOSE

The study investigators hypothesized that Point of Care Ultrasound (POCUS) training through bolus didactic and workshop experiences may be sufficient for trainees to learn the cognitive aspects, while an extended period of exposure with formative feedback is responsible for developing the psychomotor skills critical for POCUS.

METHODS

The investigators studied trainees over the course of an academic year. They compared trainees' performance on written (cognitive) and observed image acquisition (psychomotor) exams at baseline and at each subsequent quarter, using a stepped-wedge design. They performed linear regression analysis to determine which variables contributed to knowledge and psychomotor skill development.

RESULTS

Twenty-six trainees met the study requirements and participated in the POCUS curriculum. Participating in a POCUS rotation was consistently associated with an increase in psychomotor scores. There was no consistent variable to predict an increase in trainee's score on written knowledge assessments.

CONCLUSIONS

Extended exposure to POCUS over a 4-week rotation with direct and indirect formative feedback can explain difference in scores on psychomotor skills assessments. Trainees scored similarly on the written assessment with or without a POCUS rotation. Training through didactic and workshop experiences may be sufficient to learn the cognitive aspects, but not psychomotor skills required for POCUS.

A comparison between the effectiveness of a gamified approach with the conventional approach in point-of-care ultrasonographic training

BACKGROUND

Although gamification increases user engagement, its effectiveness in point-of-care ultrasonographic training has yet to be fully established. This study was conducted with the primary outcome of evaluating its effectiveness in point-of-care ultrasonographic training as compared to conventional approach.

METHODS

Participants consisting of junior doctors were randomized into either the (1) gamified or the (2) conventional educational approach for ultrasonographic training.

RESULTS

A total of 31 junior doctors participated in this study (16 participants in gamified arm, 15 in the conventional arm after one participant from the conventional arm dropped out due to work commitment). Two-way mixed ANOVA test showed that there was no statistically significant interaction between the types of educational approach and time of testing (pre-test, post-test, 2 months post-training) for both theoretical knowledge score and practical skills score, with F(2, 58) = 39.6, p < 0.001, partial η² = 0.4 and F(2, 58) = 3.06, p = 0.06, partial η² = 0.095, respectively. For theoretical knowledge score, pairwise comparisons showed that the mean 2 months post-training scores (20.28 +/- 0.70, 95% CI 18.87-21.69) and mean post-test scores (20.27 +/- 0.65, 95% CI 18.94-21.60) were better than the pre-test scores (12.99 +/- 0.50, 95% CI 11.97-14.00) with p-values < 0.001 for both comparisons respectively. Similarly, for practical skill score, pairwise comparisons showed that the mean 2 months post-training scores (20.28 +/- 0.70, 95% CI 18.87-21.69) and mean post-test scores (20.27 +/- 0.65, 95% CI 18.94-21.60) were also better than the pre-test scores (12.99 +/- 0.50, 95% CI 11.97-14.00) with p-values < 0.001 for both comparisons respectively. Participants in the gamification arm generally perceived the various game elements and game mechanics as useful in contributing and motivating them to learn ultrasonography.

CONCLUSIONS

Gamification approach could be an effective alternative to conventional approach in point-of-care ultrasonographic training.

Portable Ultrasound Device Usage and Learning Outcomes Among Internal Medicine Trainees: A Parallel-Group Randomized Trial

BACKGROUND

Little is known about how to effectively train residents with point-of-care ultrasonography (POCUS) despite increasing usage.

OBJECTIVE

This study aimed to assess whether handheld ultrasound devices (HUDs), alongside a year-long lecture series, improved trainee image interpretation skills with POCUS.

METHODS

Internal medicine intern physicians (N = 149) at a single academic institution from 2016 to 2018 participated in the study. The 2017 interns (n = 47) were randomized 1:1 to receive personal HUDs (n = 24) for patient care vs no-HUDs (n = 23). All 2017 interns received a repeated lecture series regarding cardiac, thoracic, and abdominal POCUS. Interns were assessed on their ability to interpret POCUS images of normal/abnormal findings. The primary outcome was the difference in end-of-the-year assessment scores between interns randomized to receive HUDs vs not. Secondary outcomes included trainee scores after repeating lectures and confidence with POCUS. Intern scores were also compared with historical (2016, N = 50) and contemporaneous (2018, N = 52) controls who received no lectures.

RESULTS

Interns randomized to HUDs did not have significantly higher image interpretation scores (median HUD score: 0.84 vs no-HUD score: 0.84; P = .86). However, HUD interns felt more confident in their abilities. The 2017 cohort had higher scores (median 0.84), compared with the 2016 historical control (median 0.71; P = .001) and 2018 contemporaneous control (median 0.48; P < .001). Assessment scores improved after first-time exposure to the lecture series, while repeated lectures did not improve scores.

CONCLUSIONS

Despite feeling more confident, personalized HUDs did not improve interns' POCUS-related knowledge or interpretive ability. Repeated lecture exposure without further opportunities for deliberate practice may not be beneficial for mastering POCUS.