Introduction of Directly Observed Procedural Skills (DOPS) as a Part of Competency-Based Medical Education in Otorhinolaryngology

Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review

BACKGROUND

Studies that have methodically compiled the body of research on the competency-based medical education (CBME) assessment procedure and pinpointed knowledge gaps about the structure of the assessment process are few. Thus, the goals of the study were to create a model assessment framework for competency-based medical education that would be applicable in the Indian setting as well as to thoroughly examine the competency-based medical education assessment framework.

METHODS

PubMed, MEDLINE (Ovid), EMBASE (Ovid), Scopus, Web of Science, and Google Scholar were the databases that were searched. The search parameters were restricted to English language publications about competency-based education and assessment methods, which were published between January 2006 and December 2020. A descriptive overview of the included research (in tabular form) served as the foundation for the data synthesis.

RESULTS

Databases provided 732 records; out of which 36 fulfilled the inclusion and exclusion criteria. Thirty-six studies comprised a mix of randomized controlled trials, focus group interviews, and questionnaire studies, including cross-sectional studies, qualitative studies (03), mixed-method studies, etc. The papers were published in 10 different journals. The greatest number was published in BMC Medical Education (18). The average quality score for included studies was 62.53% (range: 35.71-83.33%). Most authors are from the UK (07), followed by the USA (05). The included studies were grouped into seven categories based on their dominant focus: moving away from a behavioristic approach to a constructive approach of assessment (01 studies), formative assessment (FA) and feedback (10 studies), the hurdles in the implementation of feedback (04 studies), utilization of computer or online based formative test with automated feedback (05 studies), video feedback (02 studies), e-learning platforms for formative assessment (04 studies), studies related to workplace-based assessment (WBA)/mini-clinical evaluation exercise (mini-CEX)/direct observation of procedural skills (DOPS) (10 studies).

CONCLUSIONS

Various constructivist techniques, such as concept maps, portfolios, and rubrics, can be used for assessments. Self-regulated learning, peer feedback, online formative assessment, an online computer-based formative test with automated feedback, the use of a computerized web-based objective structured clinical examination (OSCE) evaluation system, and the use of narrative feedback instead of numerical scores in mini-CEX are all ways to increase student involvement in the design and implementation of the formative assessment.

Development and Integration of DOPS as Formative Tests in Head and Neck Ultrasound Education: Proof of Concept Study for Exploration of Perceptions

In Germany, progress assessments in head and neck ultrasonography training have been carried out mainly theoretically and lack standardisation. Thus, quality assurance and comparisons between certified courses from various course providers are difficult. This study aimed to develop and integrate a direct observation of procedural skills (DOPS) in head and neck ultrasound education and explore the perceptions of both participants and examiners. Five DOPS tests oriented towards assessing basic skills were developed for certified head and neck ultrasound courses on national standards. DOPS tests were completed by 76 participants from basic and advanced ultrasound courses (n = 168 documented DOPS tests) and evaluated using a 7-point Likert scale. Ten examiners performed and evaluated the DOPS after detailed training. The variables of "general aspects" (6.0 Scale Points (SP) vs. 5.9 SP; p = 0.71), "test atmosphere" (6.3 SP vs. 6.4 SP; p = 0.92), and "test task setting" (6.2 SP vs. 5.9 SP; p = 0.12) were positively evaluated by all participants and examiners. There were no significant differences between a basic and advanced course in relation to the overall results of DOPS tests (p = 0.81). Regardless of the courses, there were significant differences in the total number of points achieved between individual DOPS tests. DOPS tests are accepted by participants and examiners as an assessment tool in head and neck ultrasound education. In view of the trend toward "competence-based" teaching, this type of test format should be applied and validated in the future.

Introduction of direct observation of procedural skills (DOPS) as a formative assessment tool during postgraduate training in anaesthesiology: Exploration of perceptions

Background and Aims:

Formative assessment of procedural skills of Anaesthesiology postgraduate (PG) students is not conducted conventionally. Direct observation of procedural skills (DOPS) helps to identify gaps in performance and provides structured feedback. The present study was taken to explore perceptions of PG students and faculty about DOPS.

Methods:

This mixed design interventional study was conducted on 12 PGs and 10 faculty members in Department of Anaesthesiology. After conducting DOPS, a pre-validated questionnaire was given to them about perceptions using 3-point Likert's scale along with open ended questions. Statistical analysis was done using descriptive statistics of perception to calculate percentages and themes were identified for qualitative data.

Results:

Responses of students were positive about skill improvement (83%), time provided (75%), feedback (100%), interaction (83%), motivation (83%), satisfaction (83%), effectiveness (83%) and opportunity creation (92%). Faculty responded positively regarding change in attitude (100%), effectiveness (100%), scope (90%), feasible application (90%), ease (90%), opportunity creation (80%), gap identification (100%), satisfaction (80%). However, 60% felt training was required, 50% thought more time and commitment was required. Themes identified were DOPS is comprehensive, interactive, student-friendly, good teaching-learning tool, identifies gaps, provides focus for learning, provides systematic constructive feedback, improves skills, prepares for future practice, requires planning, may not reflect competence, has assessor variability and can be included in PG curriculum.

Conclusion:

DOPS was perceived as an effective assessment and teaching-learning tool by PG students as well as faculty.

The Effect of Repeated Direct Observation of Procedural Skills (R-DOPS) Assessment Method on the Clinical Skills of Anesthesiology Residents

Background

The ultimate result of patient care is one of the most important outcomes in medical education. Several methods, including the direct observation of procedural skills (DOPS), have been proposed to assess professional competencies in clinical practice.

Objectives

This study aimed to assess the effects of the Repeated DOPS (R-DOPS) method on the performance of procedural skills in anesthesiology residents.

Methods

The procedural skill performance of anesthesiology residents was assessed using a standard DOPS protocol from May to October 2019. Their scores were then objectively recorded, and the satisfaction rates regarding the 2 DOPS exams were assessed.

Results

We found a considerable improvement in anesthesiology residents’ procedural skill performance, especially in the anesthesiology residency curriculum’s basic items. Besides, anesthesiology residents’ satisfaction was significantly improved after the 2nd DOPS.

Conclusions

R-DOPS leads to improved training outcomes, including assessing the procedural skills, time to feedback to trainees, and trainee satisfaction.