Evaluation of competence in ultrasound-guided procedures-a generic assessment tool developed through the Delphi method

Translating Guidelines into Practical Practice: Point-of-Care Ultrasound for Pediatric Critical Care Clinicians

Point-of-care ultrasound (POCUS) is now transitioning from an emerging technology to a standard of care for critically ill children. POCUS can provide immediate answers to clinical questions impacting management and outcomes within this fragile population. Recently published international guidelines specific to POCUS use in neonatal and pediatric critical care populations now complement previous Society of Critical Care Medicine guidelines. The authors review consensus statements within guidelines, identify important limitations to statements, and provide considerations for the successful implementation of POCUS in the pediatric critical care setting.

Objective structured clinical examination in basic thoracic ultrasound: a European study of validity evidence

BACKGROUND

Basic thoracic ultrasound is being used more frequently by clinicians in several settings due to its high diagnostic accuracy for many common causes of respiratory failure and dyspnoea. However, ultrasound examinations are operator-dependent, and sufficient competences are needed to obtain high sensitivity and specificity of examinations. Additionally, it is crucial for ultrasound operators to perceive the competence to interpret the images and integrate them into the patient history and other examinations. This study aims to explore and gather validity evidence for an objective structured clinical examination test of basic thoracic ultrasound competences and establish a pass/fail score.

METHODS

An expert panel created the test which included two theoretical and five practical stations representing cases with different diagnoses that cause respiratory symptoms and which are possible to establish by basic thoracic ultrasound. Twenty-five participants with different levels of experience in basic thoracic ultrasound completed the test. Data of the test scores were used for item analysis, and exploring validity evidence was done according to Messick's framework which is recommended. The contrasting groups' standard setting method was used to establish a pass/fail score.

RESULTS

The summarised internal consistency reliability was high with a Cronbach's alpha of 0.87. The novice group (n = 4) had a mean test score of 42 ± 10.1 points, the intermediate group (n = 8) scored 79.1 ± 8.1 points, and the experienced group (n = 13) 89.0 ± 6.2 points (one-way ANOVA, p < 0.001). A pass/fail score of 71 points was thus derived (maximum test score = 105 points).

CONCLUSION

We developed a test for the assessment of clinical competences in basic thoracic ultrasound with solid validity evidence, and a pass/fail standard with no false positives or false negatives.

Head and Neck Ultrasound - EFSUMB Training Recommendations for the Practice of Medical Ultrasound in Europe

Different surgical and medical specialists increasingly use head and neck ultrasound and ultrasound-guided interventions as part of their clinical practice. We need to ensure high quality and standardized practice across specialties, and this position paper of the European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) describes the training requirements for head and neck ultrasound. Traditionally, a minimum number of ultrasound examinations indicates competence, but this is unreliable, and a general shift towards competence-based training is ongoing. For each EFSUMB level, we will outline the theoretical knowledge and skills needed for clinical practice. The recommendations follow the three EFSUMB competency levels for medical ultrasound practice. Level 1 describes the skills required to perform essential head and neck ultrasound examinations independently, level 2 includes ultrasound-guided interventions, while level 3 involves the practice of high-level neck ultrasound and use of advanced technologies. Our goal is to ensure high quality and standardized head and neck ultrasound practice performed by different clinical specialists with these recommendations.

Development of a multisystem point of care ultrasound skills assessment checklist

Background

Many institutions are training clinicians in point-of-care ultrasound (POCUS), but few POCUS skills checklists have been developed and validated. We developed a consensus-based multispecialty POCUS skills checklist with anchoring references for basic cardiac, lung, abdominal, and vascular ultrasound, and peripheral intravenous line (PIV) insertion.

Methods

A POCUS expert panel of 14 physicians specializing in emergency, critical care, and internal/hospital medicine participated in a modified-Delphi approach to develop a basic POCUS skills checklist by group consensus. Three rounds of voting were conducted, and consensus was defined by ≥ 80% agreement. Items achieving < 80% consensus were discussed and considered for up to two additional rounds of voting.

Results

Thirteen POCUS experts (93%) completed all three rounds of voting. Cardiac, lung, abdominal, and vascular ultrasound checklists included probe location and control, basic machine setup, image quality and optimization, and identification of anatomical structures. PIV insertion included additional items for needle tip tracking. During the first round of voting, 136 (82%) items achieved consensus, and after revision and revoting, an additional 21 items achieved consensus. A total of 153 (92%) items were included in the final checklist.

Conclusions

We have developed a consensus-based, multispecialty POCUS checklist to evaluate skills in image acquisition and anatomy identification for basic cardiac, lung, abdominal, and vascular ultrasound, and PIV insertion.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13089-022-00268-4.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.