Reliable and valid assessment of ultrasound operator competence in obstetrics and gynecology

Probe motion during mid-trimester fetal anomaly scan in the clinical setting: A prospective observational study

INTRODUCTION

This study aims to investigate probe motion during full mid-trimester anomaly scans.

METHODS

We undertook a prospective, observational study of obstetric sonographers at a UK University Teaching Hospital. We collected prospectively full-length video recordings of routine second-trimester anomaly scans synchronized with probe trajectory tracking data during the scan. Videos were reviewed and trajectories analyzed using duration, path metrics (path length, velocity, acceleration, jerk, and volume) and angular metrics (spectral arc, angular area, angular velocity, angular acceleration, and angular jerk). These trajectories were then compared according to the participant level of expertise, fetal presentation, and patient BMI.

RESULTS

A total of 17 anomaly scans were recorded. The average velocity of the probe was 12.9 ± 3.4 mm/s for the consultants versus 24.6 ± 5.7 mm/s for the fellows (p = 0.02), the average acceleration 170.4 ± 26.3 mm/s2 versus 328.9 ± 62.7 mm/s2 (p = 0.02), and the average jerk 7491.7 ± 1056.1 mm/s3 versus 14944.1 ± 3146.3 mm/s3 (p = 0.02), the working volume 9.106 ± 4.106 mm3 versus 29.106 ± 11.106 mm3 (p = 0.03), respectively. The angular metrics were not significantly different according to the participant level of expertise, the fetal presentation, or to patients BMI.

CONCLUSION

Some differences in the probe path metrics (velocity, acceleration, jerk and working volume) were noticed according to operator's level.

Emergency medicine residents' learning curve in diagnosing deep vein thrombosis with 3-point venous point-of-care ultrasound

BACKGROUND

Many cases of deep vein thrombosis (DVT) are diagnosed in the emergency department, and abbreviated lower extremity venous point-of-care ultrasound (POCUS) has already shown an accuracy comparable to that of specialists. This study aimed to identify the learning curve necessary for emergency medicine (EM) residents to achieve expertise-level accuracy in diagnosing DVT through a 3-point lower extremity venous POCUS.

METHODS

This prospective study was conducted at an emergency department between May 2021 and October 2022. Four EM residents underwent a one-hour POCUS training session and performed DVT assessments in participants with DVT symptoms or confirmed pulmonary embolism. POCUS was performed at three proximal lower extremity sites to evaluate the thrombi presence and vein compressibility, with results validated by specialized radiology ultrasound. Cumulative sum (CUSUM) and the Bush and Mosteller models were used to analyze the learning curve, while generalized estimating equations were used to identify factors affecting diagnostic accuracy.

RESULTS

91 POCUS scans were conducted in 49 patients, resulting in 22% DVT confirmed by specialized venous ultrasound. In the CUSUM analysis, all four EM residents attained a 90% success rate at the common femoral vein, whereas only half achieved this rate when all three sites were considered. According to Bush and Mosteller models, 13-18 cases are required to attain 90-95% diagnostic accuracy. After 10-16 cases, the examination time for each resident decreased, and a 20% increase in examiner confidence was linked to a 2.506-fold increase in the DVT diagnosis accuracy.

CONCLUSION

EM residents generally required 13-18 cases for 90-95% DVT diagnostic accuracy, but proficiency varied among individuals, particularly requiring more cases for regions outside the common femoral vein.

Development of a 3D-printed nuchal translucency model: a pilot study for prenatal ultrasound training

BACKGROUND

We used two 3D ultrasound volumes of fetal heads at 13 weeks to create live-size 3D-printed phantoms with a view to training or assessment of diagnostic abilities for normal and abnormal nuchal translucency measurements. The phantoms are suitable for use in a water bath, imitating a real-life exam. They were then used to study measurement accuracy and reproducibility in examiners of different skill levels.

METHODS

Ultrasound scans of a 13 + 0-week fetus were processed using 3D Slicer software, producing a stereolithography file for 3D printing. The model, crafted in Autodesk Fusion360™, adhered to FMF guidelines for NT dimensions (NT 2.3 mm). Additionally, a model with pathologic NT was designed (NT 4.2 mm). Printing was performed via Formlabs Form 3® printer using High Temp Resin V2. The externally identical looking 3D models were embedded in water-filled condoms for ultrasound examination. Eight specialists of varying expertise levels conducted five NT measurements for each model, classifying them in physiological and abnormal models.

RESULTS

Classification of the models in physiological or abnormal NT resulted in a detection rate of 100%. Average measurements for the normal NT model and the increased NT model were 2.27 mm (SD ± 0.38) and 4.165 mm (SD ± 0.51), respectively. The interrater reliability was calculated via the intraclass correlation coefficient (ICC) which yielded a result of 0.883, indicating robust agreement between the raters. Cost-effectiveness analysis demonstrated the economical nature of the 3D printing process.

DISCUSSION

This study underscores the potential of 3D printed fetal models for enhancing ultrasound training through high inter-rater reliability, consistency across different expert levels, and cost-effectiveness. Limitations, including population variability and direct translation to clinical outcomes, warrant further exploration. The study contributes to ongoing discussions on integrating innovative technologies into medical education, offering a practical and economical method to acquire, refine and revise diagnostic skills in prenatal ultrasound. Future research should explore broader applications and long-term economic implications, paving the way for transformative advancements in medical training and practice.

Validated, Consensus-Based Second- and Third-Trimester Sonographic Learning Objectives for the Obstetrics Clerkship

OBJECTIVE

Currently, there are no established guidelines for obstetric ultrasound training at the core medical student clerkship level. Our objective was to develop practical, consensus-based learning objectives in second- and third-trimester ultrasound and assess the feasibility of implementing these objectives into a clinical curriculum.

STUDY DESIGN

Using the modified Delphi approach, a panel of expert stakeholders (n = 23) was assembled to reach consensus on the proposed learning objectives. A model curriculum was created consisting of a video tutorial, in-person training, and an optional tracker and piloted at two academic medical centers. Participant perception of acquired ultrasound skills and the program were assessed using a modified, previously validated survey. Pre- and postprogram survey responses were compared using the Chi-squared test of association. A p-value of <0.05 was considered statistically significant.

RESULTS

The expert panel reached consensus on five learning objectives to include determination of fetal number and presentation, assessment of fetal heart rate, determination of placental location, assessment of amniotic fluid volume, and a general understanding of the biophysical profile and its indications. A total of 26 students participated in the piloted curriculum. On completion, a majority of participants reported improved confidence in understanding and performing prenatal ultrasound in the second and third trimesters (p < 0.05). The majority of participants valued a hands-on ultrasound curriculum.

CONCLUSION

We propose practical, consensus-based learning objectives for second- and third-trimester sonography during the core clerkship with demonstrated feasibility in a two-center pilot program.

KEY POINTS

· Ultrasound is a valuable skill for medical students.. · Specific learning objectives in obstetric ultrasound are currently lacking.. · We propose consensus-based objectives and a model curriculum.. · A two-center pilot study demonstrated efficacy..

Validating the virtual: a deep dive into ultrasound simulator metrics in otorhinolaryngology

PURPOSE

This study aimed to assess the validity of simulation-based assessment of ultrasound skills for thyroid ultrasound.

METHODS

The study collected validity evidence for simulation-based ultrasound assessment of thyroid ultrasound skills. Experts (n = 8) and novices (n = 21) completed a test containing two tasks and four cases on a virtual reality ultrasound simulator (U/S Mentor's Neck Ultrasound Module). Validity evidence was collected and structured according to Messick's validity framework. The assessments being evaluated included built-in simulator metrics and expert-based evaluations using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale.

RESULTS

Out of 64 built-in simulator metrics, 9 (14.1%) exhibited validity evidence. The internal consistency of these metrics was strong (Cronbach's α = 0.805) with high test-retest reliability (intraclass correlation coefficient = 0.911). Novices achieved an average score of 41.9% (SD = 24.3) of the maximum, contrasting with experts at 81.9% (SD = 16.7). Time comparisons indicated minor differences between experts (median: 359 s) and novices (median: 376.5 s). All OSAUS items differed significantly between the two groups. The correlation between correctly entered clinical findings and the OSAUS scores was 0.748 (p < 0.001). The correlation between correctly entered clinical findings and the metric scores was 0.801 (p < 0.001).

CONCLUSION

While simulation-based training is promising, only 14% of built-in simulator metrics could discriminate between novices and ultrasound experts. Already-established competency frameworks such as OSAUS provided strong validity evidence for the assessment of otorhinolaryngology ultrasound competence.

Validity evidence for simulator-based obstetric ultrasound competency assessment tool: a multi-center study

PURPOSE

To collect validity evidence for a simulator-based obstetric ultrasound competency assessment tool (OUCAT).

METHODS

89 sonographers from three centers (XY, MC, DT), including novices (n=21), experienced trainees (n=44), and experts (n=24), participated in the competency assessment. Validity evidence of OUCAT was collected according to Standards for Educational and Psychological Testing. Content validity was ensured by reviewing guidelines and reaching expert consensus. The response process was ensured via training raters. Internal structure was explored through internal consistency, inter-rater reliability, and test-retest reliability. Relations to other variables were explored by comparing OUCAT scores of sonographers with different experience. Evidence for consequences was collected by determining the pass/fail level.

RESULTS

OUCAT included 123 items, of which 117 items could effectively distinguish novices from experts (P<0.05). The internal consistency was represented by the Cronbach's α coefficient (0.978). The inter-rater reliability was high, with XY being 0.868, MC being 0.877, and DT being 0.937 (P<0.001). Test-retest reliability was 0.732 (P=0.001). The performance of experts was significantly better than experienced trainees, and the performance of experienced trainees was significantly better than novices (70.3±10.7 vs. 39.8±15.0 vs. 20.5±10.6, P<0.001). The pass/fail level determined by contrast group method was 45 points. The passing rate of novices, experienced trainees and experts was 0% (0/21), 31.8% (14/44), and 100% (24/24), respectively.

CONCLUSION

Simulator-based OUCAT exhibits good reliability and validity in assessing obstetric ultrasound skills.

Solid validity evidence for two tools assessing competences in musculoskeletal ultrasound: a validity study

OBJECTIVES

Musculoskeletal ultrasound (MSUS) is increasingly used by rheumatologists in daily clinical practice. However, MSUS is only valuable in trained hands, and assessment of trainee competences is therefore essential before independent practice. Thus, this study aimed to establish validity evidence for the EULAR and the Objective Structured Assessment of Ultrasound Skills (OSAUS) tools used for assessing MSUS competences.

METHODS

Thirty physicians with different levels of MSUS experience (novices, intermediates, and experienced) performed four MSUS examinations of different joint areas on the same rheumatoid arthritis patient. All examinations were video recorded (n = 120), anonymized, and subsequently assessed in random order by two blinded raters using first the OSAUS assessment tool followed by the EULAR tool 1 month after.

RESULTS

The inter-rater reliability between the two raters was high for both the OSAUS and EULAR tools, with a Pearson correlation coefficient (PCC) of 0.807 and 0.848, respectively. Both tools demonstrated excellent inter-case reliability, with a Cronbach's alpha of 0.970 for OSAUS and 0.964 for EULAR. Furthermore, there was a strong linear correlation between the OSAUS and the EULAR performance scores and the participants' experience levels (R2 = 0.897 and R2 = 0.868, respectively) and a significant discrimination between different MSUS experience levels (P < 0.001 for both).

CONCLUSIONS

MSUS operator competences can be assessed reliably and valid using either the OSAUS or the EULAR assessment tool, thereby allowing a uniform competency-based MSUS education in the future. Although both tools demonstrated high inter-rater reliability, the EULAR tool was superior to OSAUS.

TRIAL REGISTRATION

ClinicalTrials.gov, http://clinicaltrials.gov, NCT05256355.

Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters

BACKGROUND

Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively.

METHODS

The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score.

RESULTS

A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument.

CONCLUSION

There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research.

Tailored photoacoustic apertures with superimposed optical holograms

A new method of generating potentially arbitrary photoacoustic wavefronts with optical holograms is presented. This method uses nanosecond laser pulses at 1064 nm that are split into four time-delayed components by means of a configurable multipass optical delay apparatus, which serves to map the pulses onto phase-delayed regions of a given acoustic wavefront. A single spatial light modulator generates separate holograms for each component, which are imaged onto a photoacoustic transducer comprised of a thermoelastic polymer. As a proof of concept of the broader arbitrary wavefront construction technique, the spatially- and temporally-modulated holograms in this study produce a phased array effect that enables beam steering of the resulting acoustic pulse. For a first experimental demonstration of the method, as verified by simulation, the acoustic beam is steered in four directions by around 5 degrees.

E-learning and practical performance in musculoskeletal ultrasound: a multicentre randomized study

OBJECTIVES

To examine the effect of pre-course e-learning on residents' practical performance in musculoskeletal ultrasound (MSUS).

METHODS

This was a multicentre, randomized controlled study following the Consolidated Standards of Reporting Trials (CONSORT) statement. Residents with no or little MSUS experience were randomized to either an e-learning group or a traditional group. One week before a 2-day face-to-face MSUS course, the e-learning group received access to an interactive platform consisting of online lectures, assignments, and practical instruction videos aligned with the content of the course. The traditional group only received standard pre-course information (program, venue, and time). All participants performed a pre- and post-course practical MSUS examination and were assessed by two individual raters, blinded to the group allocation, using the validated Objective Structured Assessment of Ultrasound Skills (OSAUS) tool.

RESULTS

Twenty-eight participants completed the study. There were no statistically significant differences in the pre- or post-course practical MSUS performance between the e-learning group and the traditional group; the mean pre-course OSAUS score (s.d.) in the -learning group was 5.4 (3.7) compared with 5.2 (2.4) in the traditional group (P = 0.8), whereas the post-course OSAUS score in the e-learning group was 11.1 (2.8) compared with 10.9 (2.4) in the traditional group (P = 0.8). There was a significant difference between the mean pre- and post-course scores (5.74 points, P < 0.001). The OSAUS assessment tool demonstrated good inter-rater reliability (intra-class correlation = 0.84).

CONCLUSION

We found no significant impact of pre-course e-learning on novices' acquisition of practical MSUS skills. Hands-on training is of the utmost importance and improves MSUS performance significantly. The OSAUS assessment tool is an applicable tool with high interrater reliability.

TRIAL REGISTRATION

https://clinicaltrials.gov/ NCT04959162.

Objective structured assessment ultrasound skill scale for hyomental distance competence - psychometric study

BACKGROUND

Ultrasound assessment of the airway recently integrates the point-of-care approach to patient evaluation since ultrasound measurements can predict a difficult laryngoscopy and tracheal intubation. Because ultrasonography is performer-dependent, a proper training and assessment tool is needed to increase diagnostic accuracy. An objective, structured assessment ultrasound skill (OSAUS) scale was recently developed to guide training and assess competence. This work aims to study the psychometric properties of OSAUS Scale when used to evaluate competence in ultrasound hyomental distance (HMD) measurement.

METHODS

Prospective and experimental study. Volunteers were recruited and enrolled in groups with different expertise. Each participant performed three ultrasonographic HMD evaluation. The performance was videorecorded and anonymized. Five assessors blindly rated participants' performance using OSAUS scale and a Global Rating Scale (GRS). A psychometric study of OSAUS scale as assessment tool for ultrasound HMD competence was done.

RESULTS

Fifteen voluntaries participated on the study. Psychometric analysis of OSAUS showed strong internal consistency (Cronbach's alpha 0.916) and inter-rater reliability (ICC 0.720; p < 0.001). The novice group scored 15.4±0.18 (mean±SD), the intermediate 14.3±0.75 and expert 13.6±0.1.25, with a significant difference between novice and expert groups (p = 0.036). The time in seconds to complete the task was evaluated: novice (90±34) (mean±SD), intermediate (84±23) and experts (83±15), with no significant differences between groups. A strong correlation was observed between OSAUS and global rating scale (r = 0.970, p < 0.001).

CONCLUSION

The study demonstrated evidence of validity and reliability. Further studies are needed to implement OSAUS scale in the clinical setting for training and assessment of airway ultrasound competence.

Incorporating Evidence-based Ultrasound-guided Vascular Access (USGVA) Standards Into the Nurse Anesthetist Armamentarium: A Quality Improvement Project

PURPOSE

This quality improvement (QI) project developed and implemented a hybrid training program, that included online modules and hands-on training for experienced certified registered nurse anesthetists (CRNAs) to increase confidence, knowledge, and competency with ultrasound-guided vascular access (USGVA).

DESIGN

This QI project used a pre-post design. Seventeen volunteer CRNAs participated in USGVA training and education, and a 90-day follow-up assessment was performed.

METHODS

The Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) guidelines were used as a framework to assess confidence, knowledge, and hands-on competency of 17 CRNAs who regularly place vascular access devices using USGVA in patients with difficult vascular access at a single facility. These assessments were measured at: (1) baseline, (2) immediately after attending a hybrid training educational program, and (3) 90 days after implementation into clinical practice. Additionally, the number of vascular access attempts required for successful placement of peripheral intravenous (PIV) catheters, arterial catheters, and central venous catheters (CVC) with ultrasound assistance over a 90-day period was reviewed.

FINDINGS

Certified registered nurse anesthetists' median confidence score increased significantly from pre- to post-education (P = .009). The confidence reported from post- to 90 days post-education improved, however it did not reach statistical significance (P = .812). The knowledge scores from pre- to post-education indicated significant improvement (P <. 001), as well as from pre- to 90 days post-education (P = .03). However, knowledge scores from post- to 90 days post-education revealed a statistically significant decline (P = .004). The overall median score for hands-on USGVA competency declined from post- to 90 days post-education (P = .109). The number of successful USGVA placements increased from 50% to 80% within a 90-day period.

CONCLUSIONS

The implementation of a USGVA hybrid training and education program improved overall provider confidence, knowledge, and competency. While confidence remained high in the 90-day follow-up, knowledge retention declined. Despite a decline in knowledge retention over time, results showed a significant improvement when compared to baseline scores. Although a decline in hands-on USGVA competency was seen at 90 days posteducation, it was not statistically significant. The percentage of overall successful USGVA placements in clinical practice increased following implementation.

Does group size matter during collaborative skills learning? A randomised study

BACKGROUND

Collaborative skills learning in the form of dyad learning compared with individual learning has been shown to lead to non-inferior skills retention and transfer. However, we have limited knowledge on which learning activities improve collaborative skills training and how the number of collaborators may impact skills transfer. We explored the effects of skills training individually, in dyads, triads or tetrads on learning activities during training and on subsequent skills transfer.

METHODS

In a randomised, controlled study, participants completed a pre-post-transfer-test set-up in groups of one to four. Participants completed 2 hours of obstetric ultrasound training. In the dyad, triad and tetrad group participants took turns actively handling the ultrasound probe. All performances were rated by two blinded experts using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale and a Global Rating Scale (GRS). All training was video recorded, and learning activities were analysed using the Interactive-Constructive-Active-Passive (ICAP) framework.

RESULTS

One hundred one participants completed the simulation-based training, and ninety-seven completed the transfer test. Performance scores improved significantly from pre- to post-test for all groups (p < 0.001, ηp²  = 0.55). However, group size did not affect transfer test performance on OSAUS scores (p = 0.13, ηp²  = 0.06) or GRS scores (p = 0.23, ηp²  = 0.05). ICAP analyses of training activities showed that time spent on non-learning and passive learning activities increased with group size (p < 0.001, ηp²  = 0.31), whereas time spent on constructive and interactive learning activities was constant between groups compared with singles (p < 0.001, ηp²  = 0.72).

CONCLUSION

Collaborative skills learning in groups of up to four did not impair skills transfer despite less hands-on time. This may be explained by a compensatory shift towards constructive and interactive learning activities that outweigh the effect of shorter hands-on time.

Development and implementation of a comprehensive postgraduate ultrasound curriculum for residents in obstetrics and gynecology: a feasibility study

Background

In obstetric and gynecological practice, ultrasound is the essential diagnostic tool. Nevertheless, few clinics have standardized and structured training curricula for young obstetricians in the field of obstetric and gynecological ultrasound. Since ultrasound is best learned hands-on in small supervised groups, we developed and implemented a comprehensive ultrasound curriculum for all postgraduate residents of our department using a peer-teaching concept.

Methods

We used Kern‘s six-step model of curricular development comprising (1) problem identification and general needs assessment, (2) needs assessment of the targeted learners, (3) goals and objectives, (4) educational strategies, (5) implementation, and (6) evaluation and feedback.

Results

Assistant physicians in the 1st and 2nd year of training received a theoretical and practical ultrasound basic course (six modules) in addition to their obligatory clinic rotations. The six main topics were prioritized according to service relevance and included the main features according to DEGUM, EBCOG and ISUOG. The units focused on a three-level training based on the AMEE levels: theoretical knowledge, well-founded theoretical knowledge and basic practical skills under guidance and self-employment of practical skills.

Conclusion

Structured and standardized sonographic training allows young gynecology and obstetrics residents to conceptually grasp and practically implement topic-related themes. Furthermore, the course concept demonstrates the high inter-rater agreement among DEGUM-certified examiners. More research is needed to analyze the learning outcomes for residents and the improvement of the patient's outcome by establishing such an ultrasound curriculum.

The use of the myometrial-cervical ratio in the ultrasound diagnosis of adenomyosis - A validation study

BACKGROUND

Adenomyosis is a benign disorder defined by ectopic endometrial glands within the uterine myometrium. A study by Mooney et al reported the myometrial-cervical ratio (MCR), a novel ultrasound measurement that was found to improve the preoperative diagnosis of adenomyosis.

AIMS

To validate the association between sonographic MCR and adenomyosis confirmed on histopathology in an independent patient group.

MATERIALS AND METHODS

Single-centre retrospective cohort study including women who underwent hysterectomy between 1 January 2016 and 31 December 2018 for a benign, non-obstetric indication with an ultrasound at the study centre prior to surgery. Clinical details and histopathology were extracted. Ultrasound images were reviewed by a gynaecology ultrasound subspecialist blinded to histological findings.

RESULTS

Eight hundred eighty-seven patients underwent hysterectomy in the study period for eligible indications; 317 had an ultrasound at the study centre and were included. There was no statistically significant association between the MCR and adenomyosis on histology when all patients were included; however, increased MCR was associated with adenomyosis when those with fibroids on ultrasound were excluded. The area under the receiver operating characteristic for this model was 0.614 (95% CI: 0.53 to 0.69). The optimal MCR cut-point in this subgroup was 1.79, which achieved 55.6% sensitivity and 62.8% specificity, with 58.5% correctly classified. There was no significant difference in MCR compared to traditional ultrasound markers of adenomyosis.

CONCLUSIONS

In a population undergoing hysterectomy for benign and non-obstetric indications, the MCR applied to preoperative ultrasound was only weakly associated with a histological diagnosis of adenomyosis.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

An Augmented Reality Device for Remote Supervision of Ultrasound Examinations in International Exercise Science Projects: Usability Study

Background

Support for long-distance research and clinical collaborations is in high demand and has increased owing to COVID-19–related restrictions on travel and social contact. New digital approaches are required for remote scientific exchange.

Objective

This study aims to analyze the options of using an augmented reality device for remote supervision of exercise science examinations.

Methods

A mobile ultrasound examination of the diameter and intima-media thickness of the femoral and carotid arteries was remotely supervised using a head-mounted augmented reality device. All participants were provided with a link to a YouTube video of the technique in advance. In part 1, 8 international experts from the fields of engineering and sports science were remotely connected to the study setting. Internet connection speed was noted, and a structured interview was conducted. In part 2, 2 remote supervisors evaluated 8 physicians performing an examination on a healthy human subject. The results were recorded, and an evaluation was conducted using a 25-item questionnaire.

Results

In part 1, the remote experts were connected over a mean distance of 1587 km to the examination site. Overall transmission quality was good (mean upload speed: 28.7 Mbps, mean download speed: 97.3 Mbps, mean ping: 21.6 milliseconds). In the interview, participants indicated that the main potential benefits would be to the fields of education, movement analysis, and supervision. Challenges regarding internet connection stability and previous training with the devices used were reported. In part 2, physicians’ examinations showed good interrater correlation (interclass correlation coefficient: 0.84). Participants valued the experienced setting as highly positive.

Conclusions

The study showed the good feasibility of the chosen design and a highly positive attitude of all participants toward this digital approach. Head-mounted augmented reality devices are generally recommended for collaborative research projects with physical examination–based research questions.

Point-of-care ultrasound in obstetrics and gynecology

BACKGROUND

The rapid technical development and portability of ultrasound systems over recent years has had a profound impact on the area of point-of-care-ultrasound (POCUS), both in general medicine and in obstetrics and gynecology. The use of POCUS enables the clinician to perform the ultrasound scan either at the medical office or the patient's bedside and used as an extension of the physical examination. Real-time images can immediately be correlated with the patient's symptoms, and any changes in a (critical) patient's condition can be more rapidly detected.

POCUS IN OBGYN

POCUS is also suitable for time-critical scenarios, and depending on the situation and its dynamics, the course and results of any therapy may be observed in real time. POCUS should be considered to be a routine extension of practice for most OB/GYN clinicians as it can give immediate answers to what could be life-threatening situations for the mother and/or baby. With its proven usefulness, the applications and use of POCUS should be incorporated in teaching programs for medical students, OBGYN residents and emergency physicians.

Fetal biometry in ultrasound: A new approach to assess the long-term impact of simulation on learning patterns

CONTEXT

Simulation-based education (SBE) has demonstrated its acceptability and effectiveness in improving ultrasound training. Because of the high cost of its implementation (investment in equipment and supervision), a pragmatic assessment of the transfer of skills learned in SBE to clinical practice and the identification of its optimal scheduling conditions have been requested to optimize its input.

OBJECTIVES

To quantify the long-term impact of simulation-based education (SBE) on the adequate performance of ultrasound fetal biometry measurements (I). The secondary objective was to identify the temporal patterns that enhanced SBE input in learning (II).

METHODS

Trainees were arbitrarily assigned to a 6-month course in obstetric ultrasound with or without an SBE workshop. In the SBE group, the workshop was implemented 'before' or at an 'early' or a 'late-stage' of the course. Those who did not receive SBE were the control group. The ultrasound skills of all trainees were prospectively collected, evaluated by calculating the delta between OSAUS (Objective Structured Assessment of Ultrasound Skills) scores before and after the course (I). Concomitantly, the accuracy of trainees' measurements was assessed throughout the course by verifying their correlation with the corresponding measurements by their supervisors. The percentage of trainees able to perform five consecutive sets of correct measurements in the control group and in each SBE subgroup were compared (II).

RESULTS

The study included 61 trainees (39 SBE and 22 controls). Comparisons between groups showed no significant difference in the quantitative assessment of skill enhancement (difference in the pre- and post-internship OSAUS score: 1.09 ± 0.87 in the SBE group and 0.72 ± 0.98 in the control group) (I). Conversely, the predefined acceptable skill level was reached by a significantly higher proportion of trainees in the 'early' SBE subgroup (74%, compared with 30% in the control group, P<0.01)(II).

CONCLUSIONS

The quantitative assessment does not support the existence of long-term benefits from SBE training, although the qualitative assessment confirmed SBE helped to raise the minimal level within a group when embedded in an 'early' stage of a practical course.

[Skills assessment for the practice of fetal ultrasound biometry: Prospective validation of the OSAUS METHOD score]

OBJECTIVES

To develop and validate a customized variant for fetal biometry of the generic OSAUS score (Objective Structured Assessment of Ultrasound Skills) METHODS: The 5-points OSAUS METHOD grid was elaborated by defining five target skills specific to fetal biometry for each thematic item of the generic score. The level of skills of 43 trainees was prospectively assessed during an ultrasound examination by using this grid. The results of the "novice" level group (experience<10 ultrasounds) were compared to those of the "intermediate" level group (experience≥10 ultrasounds) (I). The reached/non-reached skills ratio within the different items composing the score allowed the identification of priority areas of improvement (II). Previously published distribution and actual distribution of ratings according to the generic pass/fail score were compared (III).

RESULTS

Median scores of "novices" (n=29) and "intermediates" (n=14) groups were statistically different, 1.87 (±0.75) and 3.31 (±0.83) (P=1.85-5), respectively and corresponded to the pre-existing experience (I). A lower ratio of reached skill allowed the identification of "documentation of the examination" item as a priority area of improvement for both groups (II). The relevance of the pass/fail score is consolidated, even if an overlap was observed between novices and intermediates groups (III).

CONCLUSIONS

The relevancy and feasibility of using OSAUS scoring method for fetal biometry are supported. In addition, the possibility of comparisons with generic OSAUS remains.

Implementing the transvaginal ultrasound simulation training (TRUSST) programme for obstetric registrars

BACKGROUND

Transvaginal ultrasound (TVUS) training opportunities are limited due to its intimate nature; however, TVUS is an important component of early pregnancy assessment. Simulation can bridge this learning gap.

AIM

To describe and measure the effect of a transvaginal ultrasound simulation programme for obstetric registrars.

MATERIALS AND METHODS

The transvaginal ultrasound simulation training (TRUSST) curriculum consisted of supported practice using virtual reality transvaginal simulators (ScanTrainer, Medaphor) and communication skills training to assist obstetric registrars in obtaining required competencies to accurately and holistically care for women with early pregnancy complications. Trainee experience of live transvaginal scanning was evaluated with a questionnaire. Programme evaluation was by pre-post self-reported confidence level and objective pre-post training assessment using Objective Structured Assessment of Ultrasound Skills (OSAUS) and modified Royal Australian and New Zealand College of Obstetrics and Gynaecology assessment scores. Quantitative data was compared using paired t tests.

RESULTS

Fifteen obstetric registrars completed the programme. Numbers of performed live transvaginal ultrasound by trainees were low. Participants reported an increase in confidence level in performing a TVUS following training: mean pre score 1.6/5, mean post score 3/5. Objective assessments improved significantly across both OSAUS and RANZCOG scores following training; mean improvement scores 7.6 points (95% CI 6.2-8.9, p < 0.05) and 32.5 (95% CI 26.4-38.6, p < 0.05) respectively. It was noted that scores for a systematic approach and documentation were most improved: 1.9 (95% CI 1.4-2.5, p < 0.05) and 2.1 (95% CI 1.5-2.7, p < 0.05) respectively.

CONCLUSION

The implementation of a simulation-based training curriculum resulted in improved confidence and ability in TVUS scanning, especially with regard to a systematic approach and documentation.

Metrics used to evaluate obstetric ultrasound skills on simulators: A systematic review

Obstetric ultrasound simulators are now used for training and evaluating OB/GYN students but there is a lack of literature about evaluation metrics in this setting. In this literature review, we searched MEDLINE and the COCHRANE database using the keywords: (Obstetric OR Fetal) AND (Sonography OR Ultrasound) AND Simulation. Of a total of 263 studies screened, we selected nine articles from the title and the abstract in PubMed, in the past 5 years. Two more article were added from bibliographies. A total of 11 articles were therefore included. from which nine articles were selected from the title and the abstract in PubMed. Two more articles were added from the bibliographies For each study, data about the type of simulation, and the metrics (qualitative or quantitative) used for assessment were collected. The selection of studies shows that evaluation criteria for ultrasound training were qualitative metrics (binary success/fail exercise ; dexterity quoted by an external observer ; Objective Structured Assessment of Ultrasound Skills (OSAUS) Score ; quality of images according to Salomon's score) or quantitative criteria (Accuracy of Biometry - Simulator generated metrics). Most studies used a combination of both. To date, simulator metrics used to discriminate ultrasound skills are performance score quoted by external observers and image quality scoring. Whether probe trajectory metrics can be used to discriminate skills is unknown.

Evaluation of competence in ultrasound-guided procedures-a generic assessment tool developed through the Delphi method

OBJECTIVES

To develop a generic and objective tool for assessing competence in percutaneous ultrasound-guided procedures.

METHODS

Interventional ultrasound experts from the Nordic countries were invited to participate in a three-round Delphi process. A steering committee was formed to manage the process. In round 1, the experts were asked to suggest all aspects to consider when assessing competence in US-guided procedures. Suggestions were analyzed and condensed into assessment items. In round 2, the expert panel rated these items on a 1-5 scale and suggested new items. Items with a mean rating of ≤ 3.5 were excluded. In round 3, the expert panel rated the list items and suggested changes to the items.

RESULTS

Twenty-five experts were invited, and response rates in the three rounds were 68% (17 out of 25), 100% (17 out of 17), and 100% (17 out of 17). The three-round Delphi process resulted in a 12-item assessment tool, using a five-point rating scale. The final assessment tool evaluates pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork.

CONCLUSIONS

Expert consensus was achieved on a generic tool for assessment of competence in percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE). This is the initial step in ensuring a valid and reliable method for assessment of interventional US skill.

KEY POINTS

• Through a Delphi process, expert consensus was achieved on the content of an assessment tool for percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE) tool. • The IUSE tool is comprehensive and covers pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork. • This is an important step in ensuring valid and reliable assessment of interventional US skills.

Effects of simulation for gynaecological ultrasound scan training: a systematic review

Introduction

Simulation is a potentially promising solution to some of the challenges in delivering ultrasound training, and in the last decade, research in simulation-based education has rapidly expanded. There have been no prior focused systematic reviews on the effects of simulation on gynaecological ultrasound scan training. The aim of this review was to summarise and critically appraise the available evidence.

Methods

Medline, EMBASE, CINHAL, BNI, PubMed, ERIC, Cochrane Library and ISRCTN Register of Clinical Trials were searched using selected terminology. No language restrictions were applied. The selection criteria were studies of all types that investigated the use of a simulator to teach gynaecological ultrasound. Data evaluating study outcomes, along with methodological details, were extracted in duplicate. The outcomes were classified according to the Kirkpatrick hierarchy and included: patient-related outcomes, such as discomfort; organisational efficiency outcomes, such as examination times; and trainee-related outcomes, such as competence, skill and confidence.

Results

Fourteen studies with 573 participants were included: seven randomised controlled trials, three controlled observational studies and four uncontrolled observational studies. The use of simulation practice in gynaecological ultrasound in addition to standard approaches for teaching is associated with significant improvements in competence, skill and confidence of trainees and reduced patient discomfort, increased perceived patient safety and enhanced confidence in the trainee.

Conclusion

Our findings suggest that simulation in the early stages of training in gynaecological ultrasound is beneficial for the trainee, patients and has organisational benefits through reduced examinations times and need for supervision. Future research needs to confirm these findings in the different settings.

Instructing Ultrasound-guided Examination Techniques Using a Social Media Smartphone App

Social media applications on smartphones allow for new avenues of instruction in sports medicine and exercise sciences. This study tested the feasibility of instructing health care personnel through videos of ultrasound vascular measurements distributed by a social media messenger application. After two training sessions with an ultrasound device, voluntary physicians (n=10) and nurses (n=10) received a video for the performance of an ultrasound-guided determination of intima-media-thickness and diameter of the femoral arteries via a social media messenger application. All participants examined the same healthy human subject. There was no significant difference between the groups regarding overall time of performance, measurements of the femoral arteries, or a specifically designed "assessment of mobile imparted arterial ultrasound determination" score. The physicians group achieved significantly higher scores in the established "objective structured assessment of ultrasound skills" score (p=0.019). Approval of the setting was high in both groups. Transmission of videos via social media applications can be used for instructions on the performance of ultrasound-guided vascular examinations in sports medicine, even if investigators' performances differ depending on their grade of ultrasound experience. In the future, the chosen approach should be tested in practical scientific examination settings.

Reaudit of transvaginal ultrasound practice in a general gynecology clinic

PURPOSE

A recent pilot audit found that the quality of transvaginal ultrasound practice in a teaching hospital did not reflect current guidelines. This was a concern given the frequency and importance of ultrasound examinations in approaching a diagnosis. Interventional measures involved ad hoc training sessions coincident with installation of updated equipment. A reaudit was performed to assess any changes in the standard of ultrasound practice.

METHODS

Data were collected by random direct observation. To minimize the Hawthorne effect, staffs were not made aware of when active data collection took place. Observations of both patients and end user were compared with current guidelines. Data collected maintained the anonymity of users throughout. Practice was graded as: compliant, partially-compliant, or noncompliant. Descriptive statistics were used to demonstrate any change post-intervention.

RESULTS

Observations (n = 48) of completed gynecology ultrasound practice were recorded, and results described as percentage frequencies (%). Image optimization noncompliant rates declined in the reaudit from 65.1% to 41.67% (P = .07) and noncompliant rates of global examination of gynecology decreased from 60.47% to 14.58% (P < .05). Substantial improvements were seen in terms of image annotations (initial audit, 41.9% vs reaudit, 66.67%), and end user examination of bladder, vagina and cervix when indicated (initial audit, 25.6% vs reaudit, 77.08%).

CONCLUSION

Regular audit, ad hoc ultrasound training sessions and updated ultrasound equipment resulted in considerably improved compliance of transvaginal ultrasound practice in gynecology.

[Training in fetal ultrasound biometry: Prospective assesment of Objective Structured Assessment of Ultrasound Skills (OSAUS) efficiency]

BACKGROUND

Fetal biometry quality directly influences obstetrical care relevance. However, obstetrician proficiencies are heterogeneous in particular during initial training.

OBJECTIVES

To assess the predictive value of OSAUS scale to identify operators with enough command to perform a valid estimation of fetal weight (EFW) (I). This study also assesses OSAUS intra-operator inter-exams variability (II) and pass/fail score relevancy (III).

METHODS

Lecturers in Nancy University Hospital assessed trainees' proficiency for EWF systematically and prospectively through OSAUS scale. The trainee assessment was performed right after the one of the senior operator (reference EFW) on three consecutive patients during standard care ultrasounds. To ensure variability in proficiency within the sample, previous practice was taken into account during enrollment ("novices" and "intermediates" for<20 and 20 past exams, respectively). Correlation between mean OSAUS and validity of EFW (a valid EFW was defined by a difference with the reference EWF<0.8 Z-score) and variability between consecutive assessments were assessed.

RESULTS

The study population was constituted of 8 "novice" and 8 "intermediate" trainees. Association between OSAUS and EFW validity was significant (P<0.03) (I). Intra-operator inter-exams variability was majored in the "novice" group (coefficients of variation were 25% vs. 10% in "novice" and "intermediate" group respectively) (II). Within the sample, specificity and positive predictive value of a pass/fail score OSAUS>3.5 to predict EFW validity were 77% and 71%, respectively (III).

CONCLUSION

A 3.5 OSAUS pass/fail score could provide a relevant threshold to estimate operator proficiency in assessing fetal biometry in an autonomous and secure way.

Multicenter randomized trial exploring effects of simulation-based ultrasound training on obstetricians' diagnostic accuracy: value for experienced operators

OBJECTIVE

To explore the effects of simulation-based ultrasound training on the accuracy of fetal weight estimation in the third trimester among obstetricians with different levels of clinical experience.

METHODS

This was a multicenter, randomized pre-post-test practical trial conducted between March 2016 and January 2018. Obstetricians with different levels of clinical experience were randomized to either simulation-based ultrasound training focusing on fetal weight scans or no intervention. Participants completed two scans in pregnant women at term to establish baseline accuracy of fetal weight estimation. Another two scans were performed at follow-up. Accuracy was defined by the percentage difference between estimated fetal weight and actual birth weight. Ultrasound image quality was rated by two expert raters.

RESULTS

Seventy participants with different levels of clinical experience completed the study. Adjusting for clinical experience, the intervention group demonstrated an improvement in measurement accuracy of 31.9% (95% CI, 6.9-50.1%) (P = 0.02), whereas the control group did not improve (relative difference, 13.1% (95% CI, -17.9 to 55.9%); P = 0.45). The change in accuracy was significantly different between the groups (P = 0.02) and independent of clinical experience (P = 0.54). Image-quality scores improved by a mean of 1.2 (95% CI, 0.4-2.1) (P < 0.01) in the intervention group, with no change in the control group (mean difference, 0.1 (95% CI, -0.8 to 1.0); P = 0.78). There was a strong negative correlation between time spent using the simulator and clinical experience (r = -0.70, P = 0.0001).

CONCLUSION

Simulation-based ultrasound training improved accuracy and image quality when performing fetal weight estimation in women at term, independent of obstetricians' clinical experience. Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.

A pilot study to evaluate the practice of transvaginal gynecological sonography in a general gynecology clinic

PURPOSE

Ultrasound is the imaging method of choice in gynecology. The quality and diagnostic accuracy of ultrasound depends on the skills of the individual performing the scan. Evaluation of ultrasound practice has received limited attention.

METHODS

A video recording device was connected to an ultrasound machine in gynecology clinics in a teaching hospital. To minimize the observer effect, all staff were notified through email in advance. Data were collected over a 3-week period. Anonymous recordings of both patients and user were compared with current guidelines, and practice was categorized as: compliant, partially compliant, or non-compliant.

RESULTS

Observations (n = 43) were categorized and the results described as percentage frequencies (%). Image optimization was compliant in 23.3% of recorded observations, 11.6% was partially compliant, and 65.1% was noncompliant. The study also found that global examination on gynecology was 20.9% compliant, 18.6% partially compliant, and 60.47% noncompliant. Images were annotated in appropriately 41.9% of instances, and 25.6% of end-users examined bladder, vagina, and cervix when indicated.

CONCLUSION

The pilot evaluation showed that ultrasound practice among end-users did not reflect current guidelines, suggesting a need for improvement. Accuracy and performance of ultrasound examination remains highly operator-dependent although video evaluation can be an effective tool for assessing such skills.

Introduction of Ultrasound Simulation in Medical Education: Exploratory Study

BACKGROUND

Ultrasound is ubiquitous across all disciplines of medicine; it is one of the most commonly used noninvasive, painless diagnostic tools. However, not many are educated and trained well enough in its use. Ultrasound requires not only theoretical knowledge but also extensive practical experience. The simulated setting offers the safest environment for health care professionals to learn and practice using ultrasound.

OBJECTIVE

This study aimed to (1) assess health care professionals' need for and enthusiasm toward practicing using ultrasound via simulation and (2) gauge their perception and acceptance of simulation as an integral element of ultrasound education in medical curricula.

METHODS

A day-long intervention was organized at the American University of Beirut Medical Center (AUBMC) to provide a free-of-charge interactive ultrasound simulation workshop-using CAE Vimedix high-fidelity simulator-for health care providers, including physicians, nurses, ultrasound technicians, residents, and medical students. Following the intervention, attendees completed an evaluation, which included 4 demographic questions and 16 close-ended questions based on a Likert scale agree-neutral-disagree. The results presented are based on this evaluation form.

RESULTS

A total of 41 participants attended the workshop (46% [19/41] physicians, 30% [12/41] residents, 19% [8/41] sonographers, and 5% [2/41] medical students), mostly from AUBMC (88%, 36/41), with an average experience of 2.27 (SD 3.45) years and 30 (SD 46) scans per attendee. Moreover, 15 out of 41 (36%) participants were from obstetrics and gynecology, 11 (27%) from internal medicine, 4 (10%) from pediatrics, 4 (10%) from emergency medicine, 2 (5%) from surgery and family medicine, and 5 (12%) were technicians. The majority of participants agreed that ultrasound provided a realistic setting (98%, 40/41) and that it allowed for training and identification of pathologies (88%, 36/41). Furthermore, 100% (41/41) of the participants agreed that it should be part of the curriculum either in medical school or residency, and most of the participants approved it for training (98%, 40/41) and teaching (98%, 40/41).

CONCLUSIONS

All attendees were satisfied with the intervention. There was a positive perception toward the use of simulation for training and teaching medical students and residents in using ultrasound, and there was a definite need and enthusiasm for its integration into curricula. Simulation offers an avenue not only for teaching but also for practicing the ultrasound technology by both medical students and health care providers.

[Impact of sonography simulation in the training of midwifery students]

OBJECTIVE

During their studies, French midwives need to acquire basic ultrasound skills to measure the amniotic fluid index, fetal biometry and identify fetal lie and placental positions. They have a clinical ultrasound training during their curriculum to acquire these skills. Assess the contribution that a training on an obstetrical ultrasound simulator (OUS) could make to the competence of midwives and to evaluate the best time to integrate it into their curriculum.

METHODS

Forty students in their 4th and final year in midwifery were randomized into 2 groups. Group A learned with the traditional academic course, with a clinical ultrasound training which was followed by an evaluation on an OUS (E1A). Group B was evaluated on a simulator before the clinical ultrasound training (E1B). After these initial assessments, both groups were provided with learning and training sessions on OUS, and re-evaluated in the same exercise (E2A and E2B). Group B then completed its clinical ultrasound training and was evaluated one month later on the simulator (E3B). The evaluation consisted of scoring the biometry images according to previously published quality criteria, and of assessing their competence based on a published objective and structured assessment score of ultrasound skills (OSAUS).

RESULTS

Evaluation 1 found comparable OSAUS scores (2.375/5 vs. 2.24/5 P=0.52) between the two groups (E1A vs. E1B) but a significantly higher image quality score for Group A (P=0.02). In Evaluation 2, an improvement was noted for both groups with comparable OSAUS averages between the two groups (P=0.76). After the clinical training of Group B, their OSAUS score (E3B) was significantly better than that of Group A (E1A) after their clinical training (3.45/5 vs. 2.375/5 [P=0.00017]). The same was true for the quality of their images (12.67/16 vs. 9.95/16 [P=0.003]).

CONCLUSION

Ultrasound simulation-based training, as an adjunct to ultrasound clinical training, significantly improves obstetrical ultrasound skills. The best time to train on simulators seems to be prior to clinical training on real patients.

Transvaginal ultrasound training for the obstetrics and gynecology resident: A multisite randomized controlled trial of educational DVD

PURPOSE

Residents in obstetrics and gynecology are deemed to be proficient in transvaginal ultrasound (TVUS) upon graduation, although TVUS education in residency is not standardized. The objective of this study is to assess for improvement in TVUS knowledge among residents after viewing an educational DVD.

METHODS

This is a multisite prospective randomized controlled trial using an educational DVD ("Gynecology: Beginners Only"), compared to routine education. All participants completed a pretest on TVUS images and principles. The intervention group repeated the test after DVD viewing. During the trial, performing and logging TVUS examinations were encouraged. All enrolled residents repeated the test 6-10 months later.

RESULTS

Fifty-seven residents completed the study with a mean pretest score of 9.7 (1st year resident 9.8, 2nd year resident 9.6, 3rd year resident 10.1, 4th year resident 9.4, P = .763), with a mean of 31.5 TVUS examinations logged prior to intervention. The mean score in the intervention group (n = 34) improved significantly after viewing (11.2, P < 0.003). This improvement did not persist 6-10 months later on a follow-up quiz (mean 10.7, P = .894). At completion of the study, participants logged an average of 56.7 TVUS examinations.

CONCLUSION

An educational DVD is easily implemented and demonstrates short-term benefit. Exploration of different teaching modalities in development of a comprehensive training program may improve long-term retention.

Developing and Gathering Validity Evidence for a Simulation-Based Test of Competencies in Lung Ultrasound

BACKGROUND

Clinical lung ultrasound (LUS) is a fast bedside diagnostic tool which can assist clinicians in decisions regarding the treatment and monitoring of patients with respiratory symptoms. LUS training and education differ widely, and is often done in a clinical setting, with potential risks for patients if decisions are made based on the wrong interpretations. No clear guidelines or recommendations for objective and standardized assessment of LUS skills exist, and those that do are often based on a fixed time-frame or an arbitrary number of examinations performed; this does not ensure adequate competencies.

OBJECTIVES

The study aimed to develop and gather validity evidence for a practical, simulation-based test in LUS.

METHODS

Nine cases were developed in collaboration with 3D Systems Healthcare, Littleton, CO, USA, representing the most common diagnosis and sonographic findings in patients with respiratory symptoms. Thirty-six participants with different levels of competence in LUS, completed the test. The participants were divided into groups, i.e., novices, intermediates, and experienced, according to their experience with LUS, the number of examinations they had performed, and any research they had conducted. Their answers were used for item analyses.

RESULTS

The intraclass correlation coefficient, Cronbachs' α, was 0.69 summarized, and there was a statistically significant difference (p < 0.001) between the novices and the trained participants (intermediates and experienced). A pass/fail score of 16 points was calculated according to the contrasting-groups method.

CONCLUSION

We developed a test for the assessment of clinical competencies in LUS. The test proved solid validity evidence, and a pass/fail standard without any false-negatives, and only 2 explained false-positives.

Up or down? A randomized trial comparing image orientations during transvaginal ultrasound training

INTRODUCTION

There is no international consensus on the orientation of transvaginal ultrasound images and no evidence exists to support the superiority of one image orientation over the other. The aim of this study was to compare learning curves and skills transfer in a group of novices randomized to top-down or bottom-up image orientation, and to determine whether individual preferences for image orientation affect learning and skills transfer.

MATERIAL AND METHODS

60 senior medical students, with no prior ultrasound experience, were randomized to orient the image top-down or bottom-up during training on an ultrasound simulator until attaining expert levels of performance. Participants then completed a transfer test involving a systematic ultrasound examination on a physical mannequin using real ultrasound equipment. Performance was assessed during the transfer test by two independent raters using the objective structured assessment of ultrasound skills (OSAUS) score and a global rating score.

RESULTS

The bottom-up group reached the expert level with significantly fewer attempts than did the top-down group [median ± interquartile range: 4 ± 2 vs 5 ± 3] (U = 285.5, P = 0.014). The bottom-up group used less time to achieve the expert level (median ± interquartile range: 3 h 2 m ± 1 h 14 m vs 3 h 28 m ± 2 h 21 m) (U = 301.5, P = 0.029). The two groups performed similarly during the transfer test with respect to their OSAUS scores (top-down 56.7% vs bottom-up 53.2%, P = 0.13). The global rating scores were higher in the top-down group (top-down 57.1% vs bottom-up 50.0%, P = 0.02).

CONCLUSIONS

Orientation of the images bottom-up rather than top-down, led to a steeper learning curve, but had little or no impact on the subsequent transfer of skills.

A multiple-perspective approach for the assessment and learning of ultrasound skills

Ultrasound has become a core skill in many specialties. We evaluated the learning and assessment of ultrasound skills in Obstetrics-Gynaecology in a series of eight studies. In the clinical setting, we found that trainees as well as experienced clinicians struggle with technical aspects of performance such as image optimization. We examined how to improve these aspects of performance in the simulated setting by determining mastery learning levels and exploring learning curves for novices. We then examined how to improve the efficiency of training as well as transfer of learning through the use of dyad practice as compared with single practice. We found that the use of simulation-based training focusing on technical aspects of performance in addition to clinical training led to sustained improvements in performance after two months of clinical training in all aspects of performance. In addition, we found an interaction effect between initial simulation-based training and subsequent clinical training on trainees' need for supervision. These findings suggest that simulation-based training can work as preparation for future learning rather than merely as added learning. Finally, we found that the use of simulation-based initial training led to a large decrease in patients' discomfort, improvements in their perceived safety and confidence in their ultrasound operator. However, simulation-based training comes at a cost and in the final study we developed a model for conducting cost-effectiveness studies and provided data from an example study on how to link training costs with quality of care.

Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations

Background

The contrasting groups’ standard setting method is commonly used for consequences analysis in validity studies for performance in medicine and surgery. The method identifies a pass/fail cut-off score, from which it is possible to determine false positives and false negatives based on observed numbers in each group. Since groups in validity studies are often small, e.g., due to a limited number of experts, these analyses are sensitive to outliers on the normal distribution curve.

Methods

We propose that these shortcomings can be addressed in a simple manner using the cumulative distribution function.

Results

We demonstrate considerable absolute differences between the observed false positives/negatives and the theoretical false positives/negatives. In addition, several important examples are given.

Conclusions

We propose that a better reporting strategy is to report theoretical false positives and false negatives together with the observed false positives and negatives, and we have developed an Excel sheet to facilitate such calculations.

Trial registration

Not relevant.

Electronic supplementary material

The online version of this article (10.1186/s41077-018-0064-7) contains supplementary material, which is available to authorized users.

Credentialing of Hospitalists in Ultrasound-Guided Bedside Procedures: A Position Statement of the Society of Hospital Medicine

Ultrasound guidance is used increasingly to perform the following 6 bedside procedures that are core competencies of hospitalists: abdominal paracentesis, arterial catheter placement, arthrocentesis, central venous catheter placement, lumbar puncture, and thoracentesis. Yet most hospitalists have not been certified to perform these procedures, whether using ultrasound guidance or not, by specialty boards or other institutions extramural to their own hospitals. Instead, hospital privileging committees often ask hospitalist group leaders to make ad hoc intramural certification assessments as part of credentialing. Given variation in training and experience, such assessments are not straightforward "sign offs." We thus convened a panel of experts to conduct a systematic review to provide recommendations for credentialing hospitalist physicians in ultrasound guidance of these 6 bedside procedures. Pathways for initial and ongoing credentialing are proposed. A guiding principle of both is that certification assessments for basic competence are best made through direct observation of performance on actual patients.

Point-of-care ultrasound (POCUS) of the upper airway

Airway management is a critical skill in the practice of several medical specialities including anesthesia, emergency medicine, and critical care. Over the years mounting evidence has showed an increasing role of ultrasound (US) in airway management. The objective of this narrative review is to provide an overview of the indications for point-of-care ultrasound (POCUS) of the upper airway. The use of US to guide and assist clinical airway management has potential benefits for both provider and patient. Ultrasound can be utilized to determine airway size and predict the appropriate diameter of single-lumen endotracheal tubes (ETTs), double-lumen ETTs, and tracheostomy tubes. Ultrasonography can differentiate tracheal, esophageal, and endobronchial intubation. Ultrasonography of the neck can accurately localize the cricothyroid membrane for emergency airway access and similarly identify tracheal rings for US-guided tracheostomy. In addition, US can identify vocal cord dysfunction and pathology before induction of anesthesia. A rapidly growing body of evidence showing ultrasonography used in conjunction with hands-on management of the airway may benefit patient care. Increasing awareness and use of POCUS for many indications have resulted in technologic advancements and increased accessibility and portability. Upper airway POCUS has the potential to become the first-line non-invasive adjunct assessment tool in airway management.

Improving the quality of transvaginal ultrasound scan by simulation training for general practice residents

Background

Ultrasonography (US) is an essential tool for the diagnosis of acute gynecological conditions. General practice (GP) residents are involved in the first-line management of gynecologic emergencies. They are not familiar with US equipment. Initial training on simulators was conducted.

The aim of this study was to evaluate the impact of simulation-based training on the quality of the sonographic images achieved by GP residents 2 months after the simulation training versus clinical training alone.

Methods

Young GP residents assigned to emergency gynecology departments were invited to a one-day simulation-based US training session. A prospective controlled trial aiming to assess the impact of such training on TVS (transvaginal ultrasound scan) image quality was conducted. The first group included GP residents who attended the simulation training course. The second group included GP residents who did not attend the course. Written consent to participate was obtained from all participants. Images achieved 2 months after the training were scored using standardized quality criteria and compared in both groups. The stress generated by this examination was also assessed with a simple numeric scale.

Results

A total of 137 residents attended the simulation training, 26 consented to participate in the controlled trial. Sonographic image quality was significantly better in the simulation group for the sagittal view of the uterus (3.6 vs 2.7, p = 0.01), for the longitudinal view of the right ovary (2.8 vs 1.4, p = 0.027), and for the Morrison space (1.7 vs 0.4, p = 0.034), but the difference was not significant for the left ovary (2.9 vs 1.7, p = 0.189). The stress generated by TVS after 2 months was not different between the groups (6.0 vs 4.8, p = 0.4).

Conclusion

Simulation-based training improved the quality of pelvic US images in GP residents assessed after 2 months of experience in gynecology compared to clinical training alone.

The Effects of Simulation-based Transvaginal Ultrasound Training on Quality and Efficiency of Care: A Multicenter Single-blind Randomized Trial

OBJECTIVE

To explore the effect of adding simulation-based transvaginal ultrasound training to trainees' clinical training compared with only clinical training on quality of and efficiency of care.

BACKGROUND

Simulation-based ultrasound training may be an effective adjunct to clinical training, but no studies have examined its effects on quality and efficiency of care.

METHODS

Trainees from 4 University Hospitals in East Denmark were included (N = 54). Participants were randomized to either simulation-based ultrasound training and clinical training (intervention group, n = 28), or to clinical training only (control group, n = 26).The primary outcome was patient-reported discomfort during transvaginal ultrasound examinations performed by study participants. Secondary outcomes included patient-reported perceived safety and confidence in ultrasound provider. Finally, the need for trainee supervision or repeated patient examinations was recorded.

RESULTS

In total, 1150 patient ratings were collected. The intervention was associated with a reduction of patient discomfort by 18.5% [95% confidence interval (CI), 10.7-25.5; P < 0.001), and with a 7.9% (95% CI, 0.5-14.7; P = 0.04) increase in perceived safety. The intervention group participants received 11.1% (95% CI, 2.5-18.9) higher scores on patients' confidence compared with control group participants (P = 0.01). When the number of days of clinical training was doubled, the odds for trainee supervision or repeated patient examination was reduced by 45.3% (95% CI, 33.5-55.1) and 19.8% (95% CI, 4.1-32.9) in the intervention and control group, respectively (P = 0.005).

CONCLUSIONS

Simulation-based ultrasound training improved quality of care and reduced the need for repeated patient examination and trainee supervision.

An Audit of Second-Trimester Fetal Anomaly Scans Based on a Novel Image-Scoring Method in the Southwest Region of the Netherlands

OBJECTIVES

Since 2007 the second-trimester fetal anomaly scan is offered to all pregnant women as part of the national prenatal screening program in the Netherlands. Dutch population-based screening programs generally have a well-described system to achieve quality assurance. Because of the absence of a uniform system to monitor the actual performance of the fetal anomaly scan in 2012, we developed a standardized image-scoring method. The aim of this study was to evaluate the scanning performance of all sonographers in the southwestern region of the Netherlands using this image-scoring method.

METHODS

Each sonographer was requested to set up a digital portfolio. A portfolio consists of five logbooks from five different pregnant women, each containing 25 fetal anatomical structures and six biometric measures of randomly selected fetal anomaly scans.

RESULTS

During the study period, 425 logbooks of 85 sonographers were assessed as part of the audit process. Seventy-three out of 85 sonographers (86%) met the criteria in the primary audit, and 12 sonographers required individual hands-on training. A successful assessment was achieved for 11 sonographers in the re-audit and one sonographer ceased her contract. Moreover, 2.1% of the required images were not digitally stored and therefore could not be reviewed.

CONCLUSIONS

Quality assessment using the image-scoring method demonstrated that most of the sonographers met the expectations of the audit process, but those who had subpar performance met the expectations after retraining.