Competency‐based assessment in surgeon‐performed head and neck ultrasonography: A validity study

Validating the virtual: a deep dive into ultrasound simulator metrics in otorhinolaryngology

PURPOSE

This study aimed to assess the validity of simulation-based assessment of ultrasound skills for thyroid ultrasound.

METHODS

The study collected validity evidence for simulation-based ultrasound assessment of thyroid ultrasound skills. Experts (n = 8) and novices (n = 21) completed a test containing two tasks and four cases on a virtual reality ultrasound simulator (U/S Mentor's Neck Ultrasound Module). Validity evidence was collected and structured according to Messick's validity framework. The assessments being evaluated included built-in simulator metrics and expert-based evaluations using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale.

RESULTS

Out of 64 built-in simulator metrics, 9 (14.1%) exhibited validity evidence. The internal consistency of these metrics was strong (Cronbach's α = 0.805) with high test-retest reliability (intraclass correlation coefficient = 0.911). Novices achieved an average score of 41.9% (SD = 24.3) of the maximum, contrasting with experts at 81.9% (SD = 16.7). Time comparisons indicated minor differences between experts (median: 359 s) and novices (median: 376.5 s). All OSAUS items differed significantly between the two groups. The correlation between correctly entered clinical findings and the OSAUS scores was 0.748 (p < 0.001). The correlation between correctly entered clinical findings and the metric scores was 0.801 (p < 0.001).

CONCLUSION

While simulation-based training is promising, only 14% of built-in simulator metrics could discriminate between novices and ultrasound experts. Already-established competency frameworks such as OSAUS provided strong validity evidence for the assessment of otorhinolaryngology ultrasound competence.

Solid validity evidence for two tools assessing competences in musculoskeletal ultrasound: a validity study

OBJECTIVES

Musculoskeletal ultrasound (MSUS) is increasingly used by rheumatologists in daily clinical practice. However, MSUS is only valuable in trained hands, and assessment of trainee competences is therefore essential before independent practice. Thus, this study aimed to establish validity evidence for the EULAR and the Objective Structured Assessment of Ultrasound Skills (OSAUS) tools used for assessing MSUS competences.

METHODS

Thirty physicians with different levels of MSUS experience (novices, intermediates, and experienced) performed four MSUS examinations of different joint areas on the same rheumatoid arthritis patient. All examinations were video recorded (n = 120), anonymized, and subsequently assessed in random order by two blinded raters using first the OSAUS assessment tool followed by the EULAR tool 1 month after.

RESULTS

The inter-rater reliability between the two raters was high for both the OSAUS and EULAR tools, with a Pearson correlation coefficient (PCC) of 0.807 and 0.848, respectively. Both tools demonstrated excellent inter-case reliability, with a Cronbach's alpha of 0.970 for OSAUS and 0.964 for EULAR. Furthermore, there was a strong linear correlation between the OSAUS and the EULAR performance scores and the participants' experience levels (R2 = 0.897 and R2 = 0.868, respectively) and a significant discrimination between different MSUS experience levels (P < 0.001 for both).

CONCLUSIONS

MSUS operator competences can be assessed reliably and valid using either the OSAUS or the EULAR assessment tool, thereby allowing a uniform competency-based MSUS education in the future. Although both tools demonstrated high inter-rater reliability, the EULAR tool was superior to OSAUS.

TRIAL REGISTRATION

ClinicalTrials.gov, http://clinicaltrials.gov, NCT05256355.

Head and neck ultrasound training improves the diagnostic performance of otolaryngology residents

Objective

Surgeon-performed head and neck ultrasound (US) is increasingly used among otolaryngologists in office-based and surgical settings. However, it is unknown how formal US training affects otolaryngology residents' diagnostic workup of patients with cervical pathology. This study examined how a formal US course for residents affected their outpatient clinic US performance and diagnostic accuracy.

Methods

We conducted a randomized cross-over trial, where 13 otolaryngology residents participated in a 6-h formal US course. Participants were randomized to perform head and neck US on four patient cases before and after completing the course. Eight patients with and without neck pathology were invited to participate as test cases. The ultrasound examinations were video recorded and anonymized before two consultants rated the US performance using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale. Otolaryngology residents wrote an ultrasound report with a diagnosis based on their US examination, which was used to calculate the specificity and sensitivity.

Results

We found a statistically significant difference in the OSAUS score before compared to after the hands-on training (p = .035). The diagnostic accuracy also increased from 62% before the course to 75% after the course (p = .02). Specificity increased from 54% prior to the course to 62% following the course, and sensitivity increased from 64% prior to the course to 79% following the course. The intraclass correlation coefficient with "absolute agreement" was 0.63.

Conclusion

This study demonstrates that short, formal ultrasound training can improve otolaryngology residents' ultrasound skills and diagnostic accuracy in an outpatient clinic setting.

Lay summary

This study looks at the change of otolaryngology residents' diagnostic workup of patients after they take a formal ultrasound course and shows that they get better at using ultrasound and make more accurate diagnoses if they take a formal course.

Level of Evidence

Level 2.

Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters

BACKGROUND

Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively.

METHODS

The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score.

RESULTS

A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument.

CONCLUSION

There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research.

E-learning and practical performance in musculoskeletal ultrasound: a multicentre randomized study

OBJECTIVES

To examine the effect of pre-course e-learning on residents' practical performance in musculoskeletal ultrasound (MSUS).

METHODS

This was a multicentre, randomized controlled study following the Consolidated Standards of Reporting Trials (CONSORT) statement. Residents with no or little MSUS experience were randomized to either an e-learning group or a traditional group. One week before a 2-day face-to-face MSUS course, the e-learning group received access to an interactive platform consisting of online lectures, assignments, and practical instruction videos aligned with the content of the course. The traditional group only received standard pre-course information (program, venue, and time). All participants performed a pre- and post-course practical MSUS examination and were assessed by two individual raters, blinded to the group allocation, using the validated Objective Structured Assessment of Ultrasound Skills (OSAUS) tool.

RESULTS

Twenty-eight participants completed the study. There were no statistically significant differences in the pre- or post-course practical MSUS performance between the e-learning group and the traditional group; the mean pre-course OSAUS score (s.d.) in the -learning group was 5.4 (3.7) compared with 5.2 (2.4) in the traditional group (P = 0.8), whereas the post-course OSAUS score in the e-learning group was 11.1 (2.8) compared with 10.9 (2.4) in the traditional group (P = 0.8). There was a significant difference between the mean pre- and post-course scores (5.74 points, P < 0.001). The OSAUS assessment tool demonstrated good inter-rater reliability (intra-class correlation = 0.84).

CONCLUSION

We found no significant impact of pre-course e-learning on novices' acquisition of practical MSUS skills. Hands-on training is of the utmost importance and improves MSUS performance significantly. The OSAUS assessment tool is an applicable tool with high interrater reliability.

TRIAL REGISTRATION

https://clinicaltrials.gov/ NCT04959162.

Objective structured assessment ultrasound skill scale for hyomental distance competence - psychometric study

BACKGROUND

Ultrasound assessment of the airway recently integrates the point-of-care approach to patient evaluation since ultrasound measurements can predict a difficult laryngoscopy and tracheal intubation. Because ultrasonography is performer-dependent, a proper training and assessment tool is needed to increase diagnostic accuracy. An objective, structured assessment ultrasound skill (OSAUS) scale was recently developed to guide training and assess competence. This work aims to study the psychometric properties of OSAUS Scale when used to evaluate competence in ultrasound hyomental distance (HMD) measurement.

METHODS

Prospective and experimental study. Volunteers were recruited and enrolled in groups with different expertise. Each participant performed three ultrasonographic HMD evaluation. The performance was videorecorded and anonymized. Five assessors blindly rated participants' performance using OSAUS scale and a Global Rating Scale (GRS). A psychometric study of OSAUS scale as assessment tool for ultrasound HMD competence was done.

RESULTS

Fifteen voluntaries participated on the study. Psychometric analysis of OSAUS showed strong internal consistency (Cronbach's alpha 0.916) and inter-rater reliability (ICC 0.720; p < 0.001). The novice group scored 15.4±0.18 (mean±SD), the intermediate 14.3±0.75 and expert 13.6±0.1.25, with a significant difference between novice and expert groups (p = 0.036). The time in seconds to complete the task was evaluated: novice (90±34) (mean±SD), intermediate (84±23) and experts (83±15), with no significant differences between groups. A strong correlation was observed between OSAUS and global rating scale (r = 0.970, p < 0.001).

CONCLUSION

The study demonstrated evidence of validity and reliability. Further studies are needed to implement OSAUS scale in the clinical setting for training and assessment of airway ultrasound competence.

Development and Integration of DOPS as Formative Tests in Head and Neck Ultrasound Education: Proof of Concept Study for Exploration of Perceptions

In Germany, progress assessments in head and neck ultrasonography training have been carried out mainly theoretically and lack standardisation. Thus, quality assurance and comparisons between certified courses from various course providers are difficult. This study aimed to develop and integrate a direct observation of procedural skills (DOPS) in head and neck ultrasound education and explore the perceptions of both participants and examiners. Five DOPS tests oriented towards assessing basic skills were developed for certified head and neck ultrasound courses on national standards. DOPS tests were completed by 76 participants from basic and advanced ultrasound courses (n = 168 documented DOPS tests) and evaluated using a 7-point Likert scale. Ten examiners performed and evaluated the DOPS after detailed training. The variables of "general aspects" (6.0 Scale Points (SP) vs. 5.9 SP; p = 0.71), "test atmosphere" (6.3 SP vs. 6.4 SP; p = 0.92), and "test task setting" (6.2 SP vs. 5.9 SP; p = 0.12) were positively evaluated by all participants and examiners. There were no significant differences between a basic and advanced course in relation to the overall results of DOPS tests (p = 0.81). Regardless of the courses, there were significant differences in the total number of points achieved between individual DOPS tests. DOPS tests are accepted by participants and examiners as an assessment tool in head and neck ultrasound education. In view of the trend toward "competence-based" teaching, this type of test format should be applied and validated in the future.

Head and Neck Ultrasound - EFSUMB Training Recommendations for the Practice of Medical Ultrasound in Europe

Different surgical and medical specialists increasingly use head and neck ultrasound and ultrasound-guided interventions as part of their clinical practice. We need to ensure high quality and standardized practice across specialties, and this position paper of the European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB) describes the training requirements for head and neck ultrasound. Traditionally, a minimum number of ultrasound examinations indicates competence, but this is unreliable, and a general shift towards competence-based training is ongoing. For each EFSUMB level, we will outline the theoretical knowledge and skills needed for clinical practice. The recommendations follow the three EFSUMB competency levels for medical ultrasound practice. Level 1 describes the skills required to perform essential head and neck ultrasound examinations independently, level 2 includes ultrasound-guided interventions, while level 3 involves the practice of high-level neck ultrasound and use of advanced technologies. Our goal is to ensure high quality and standardized head and neck ultrasound practice performed by different clinical specialists with these recommendations.

“A Head Start or a Pain in the Neck?”—Establishment and Evaluation of a Video-Based “Hands-On” Head and Neck Ultrasound Course

The COVID-19 pandemic has strongly highlighted the need for more digitalization in healthcare. Teaching ultrasound skills in online courses is a key challenge in this context. The aim of this study was to establish an online video-based head and neck ultrasound course with an evaluation of the quality, effectiveness, and feasibility of this teaching method compared to in-person teaching. Twenty-two medical students were taught head and neck ultrasound in two groups: one group in an in-person course and the other one in a video-based course. Learning success was analyzed using self-evaluation forms and external assessment by an experienced ultrasonographer. Comparing pre- and post-training self-evaluation, all participants showed statistically significant learning progress. In the external assessment, the overall scores in both groups did not differ significantly. The courses themselves were positively evaluated by all participants. Herein, we present the first feasibility study of a web-based head and neck-ultrasound course for medical students. The methodology provides the potential for future changes in telemedical education and sustainable improvements in telemedical teaching and global intra-clinical and interdisciplinary patient care.

A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation

CONSTRUCT

For assessing the skill of visual diagnosis such as radiograph interpretation, competency standards are often developed in an ad hoc method, with a poorly delineated connection to the target clinical population.

BACKGROUND

Commonly used methods to assess for competency in radiograph interpretation are subjective and potentially biased due to a small sample size of cases, subjective evaluations, or include an expert-generated case-mix versus a representative sample from the clinical field. Further, while digital platforms are available to assess radiograph interpretation skill against an objective standard, they have not adopted a data-driven competency standard which informs educators and the public that a physician has achieved adequate mastery to enter practice where they will be making high-stakes clinical decisions.

APPROACH

Operating on a purposeful sample of radiographs drawn from the clinical domain, we adapted the Ebel Method, an established standard setting method, to ascertain a defensible, clinically relevant mastery learning competency standard for the skill of radiograph interpretation as a model for deriving competency thresholds in visual diagnosis. Using a previously established digital platform, emergency physicians interpreted pediatric musculoskeletal extremity radiographs. Using one-parameter item response theory, these data were used to categorize radiographs by interpretation difficulty terciles (i.e. easy, intermediate, hard). A panel of emergency physicians, orthopedic surgeons, and plastic surgeons rated each radiograph with respect to clinical significance (low, medium, high). These data were then used to create a three-by-three matrix where radiographic diagnoses were categorized by interpretation difficulty and significance. Subsequently, a multidisciplinary panel that included medical and parent stakeholders determined acceptable accuracy for each of the nine cells. An overall competency standard was derived from the weighted sum. Finally, to examine consequences of implementing this standard, we reported on the types of diagnostic errors that may occur by adhering to the derived competency standard.

FINDINGS

To determine radiograph interpretation difficulty scores, 244 emergency physicians interpreted 1,835 pediatric musculoskeletal extremity radiographs. Analyses of these data demonstrated that the median interpretation difficulty rating of the radiographs was -1.8 logits (IQR -4.1, 3.2), with a significant difference of difficulty across body regions (p < 0.0001). Physician review classified the radiographs as 1,055 (57.8%) as low, 424 (23.1%) medium or 356 (19.1%) high clinical significance. The multidisciplinary panel suggested a range of acceptable scores between cells in the three-by-three table of 76% to 95% and the sum of equal-weighted scores resulted in an overall performance-based competency score of 85.5% accuracy. Of the 14.5% diagnostic interpretation errors that may occur at the bedside if this competency standard were implemented, 9.8% would be in radiographs of low-clinical significance, while 2.5% and 2.3% would be in radiographs of medium or high clinical significance, respectively.

CONCLUSION(S)

This study's novel integration of radiograph selection and a standard setting method could be used to empirically drive evidence-based competency standard for radiograph interpretation and can serve as a model for deriving competency thresholds for clinical tasks emphasizing visual diagnosis.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Ultrasound-guided fine-needle aspiration biopsy of thyroid nodules

Fine needle aspiration biopsy (FNAB) remains crucial in the evaluation of thyroid nodules with suspicious clinical findings or ultrasound (US) features suggestive of malignancy. The use of US-guidance for FNAB allows real-time visualization of the needle, but is also highly operator-dependent. Physicians from many specialties (endocrinologists, otolaryngologists/endocrine surgeons, nuclear medicine physicians, radiologists, and pathologists) are involved in the diagnostic workup of thyroid nodules and a standardized and meticulous technique for US-FNAB is essential for lowering the yield of nondiagnostic specimens and false-negative results. This video, therefore, demonstrates a well-proved technique and technical tips to increase the diagnostic results from US-FNAB.

Ultrasound Training for Head and Neck Surgeons in Rural Kenya: A Feasibility Study

OBJECTIVE

To determine the feasibility of training otolaryngologists to perform head and neck ultrasound in low- and middle-income countries over a short timeframe.

DESIGN

Prospective cohort evaluating an ultrasound training course for otolaryngologists in low- and middle-income countries.

SETTING

Surgical training camp for otolaryngologists in semirural Kenya.

METHODS

Participants were prospectively recruited to participate in an ultrasound training course during a 2-week surgical training camp. They performed baseline ultrasound exams (9-item test, 5-point scale) with no assistance, followed by didactic sessions teaching ultrasound techniques. Participating surgeons then performed head and neck ultrasound exams on patients in clinic or in the operating room with direct supervision. Postcourse ultrasounds were performed, and the results of these tests were compared to baseline exams.

RESULTS

Eight surgeons were enrolled out of a total of 13 (62%). Three were attending surgeons (37.5%), 1 enrollee was a head and neck surgery fellow, and 4 were senior residents (50%). Six of 8 surgeons were from Kenya. The mean precourse test score was 25 (range 21-30) compared to a mean postcourse test score of 40 (range 37-45, p < 0.005). All participants significantly improved their scores, with a mean improvement of 16 points (range 10-23).

DISCUSSION

Otolaryngologists are uniquely equipped to learn head and neck ultrasound given their familiarity with the anatomy and pathology in this region. Training physicians without prior experience can be done even over short timeframes. Early training should focus on ultrasound, with later stages of training focusing on pathology.

Point-of-care ultrasound for general practitioners: a systematic needs assessment

Objective: The aim of the study was to achieve consensus among a group of ultrasound proficient general practitioners (GPs) from Denmark, Norway, Sweden and Finland on which ultrasound scanning modalities and ultrasound-guided procedures are essential to GPs in their daily work for the purpose of including them in a basic ultrasound curriculum.Design: The Delphi methodology was used to obtain consensus.Subjects: Sixty Scandinavian GPs with more than two years of point-of-care ultrasound (POCUS) experience were invited to join the Delphi expert panel.Main outcome measures: In the first Delphi round each member of the panel was asked to produce a list of scanning modalities and procedures which they found relevant to include in a basic ultrasound curriculum. In Delphi round two, these suggestions were presented to the entire panel who assessed whether they found them essential in their daily work. Items not reaching consensus in round two, were presented to the panel in a third and final round. Items reaching more than 67% agreement were included.Results: Forty-five GPs were included in the study and 41 GPs completed all rounds. Agreement was obtained on 30 scanning modalities and procedures primarily within the musculoskeletal (8), abdominal (5), obstetric (5) and soft tissue (3) diagnostic areas. Four ultrasound-guided procedures were also agreed upon.Conclusion: A prioritized list of 30 scanning modalities and procedures was agreed upon by a group of ultrasound proficient GPs. This list could serve as a guideline when planning future POCUS educational activities for GPs.Key pointsPoint-of-care ultrasound (POCUS) is increasingly being used by general practitioners (GPs), but little is known about which ultrasound applications are most used.We performed a systematic needs assessment among a group of ultrasound proficient GPs using the Delphi methodology for the purpose of establishing a basic POCUS curriculum.The process resulted in a prioritized list of 30 scanning modalities and ultrasound guided procedures.Our study provides the basis for an evidence-based basic POCUS curriculum for GPs.

Training and assessment of competence in resuscitative endovascular balloon occlusion of the aorta (REBOA) - a systematic review

BACKGROUND

Resuscitative endovascular balloon occlusion of the aorta (REBOA) is a potentially life-saving but high-risk emergency procedure in patients with haemorrhagic shock. Lack of physicians with competence in the procedure is a barrier to implementation of REBOA. It is currently unclear how training and assessment of competence should be done.

OBJECTIVES

To report and evaluate research in training and assessment of competence in REBOA and femoral arterial access with the aim to investigate the effect of simulation-based training in the procedure and to provide suggestions for the future design of training programs and assessment tools.

METHODS

Following PRISMA guidelines, PubMed, Embase, and Cochrane Library databases were searched for studies on training or assessment of competence in REBOA and femoral arterial access. Bias assessment was done using the Medical Education Research Study Quality Instrument. Evidence level was assessed using GRADE.

RESULTS

Sixteen studies were included, six of them published as abstracts. Full-text studies included 189 trainees ranging in experience level from military medics to surgical specialists. Outcome measures were heterogenous; the most used were rater checklists, knowledge testing, and procedure time. All studies confirmed an effect of training of REBOA on procedural competence in a simulation setting but had a high degree of bias. No study developed or used an assessment tool supported by validity evidence and no study investigated mid and long-term outcomes.

CONCLUSION

Simulation-based training of REBOA improves skills, however, the evidence level is very low and data cannot answer important questions on effect size, skill transfer and retention, and optimal course design. To advance research and training programmes, an assessment tool supported by validity evidence with broad applicability is needed.

A Novel Technique for Intraoral Ultrasound-Guided Aspiration of Peritonsillar Abscess

Peritonsillar abscess (PTA) is a common complication to acute tonsillitis. The treatment is drainage of the abscess, but many needle aspirations are unsuccessful due to a low diagnostic accuracy based on oral examination only. In this article, we describe how intraoral ultrasound can be added to improve the diagnostic work-up of PTA and present a novel technique for ultrasound-guided aspiration of PTA, using a small pencil-shaped transducer. We present our first clinical experiences with this technique and describe how it could be integrated in a clinical setting to guide safe and successful needle aspirations of PTA.