Development and validation of a multiple-choice questionnaire-based theoretical test in direct ophthalmoscopy

Theoretical Frameworks in Medical Education: Using a Systematic Review of Ophthalmology Education Research to Create a Theory of Change Model

BACKGROUND

Theoretical frameworks provide a lens to examine questions and interpret results; however, they are underutilized in medical education.

OBJECTIVE

To systematically evaluate the use of theoretical frameworks in ophthalmic medical education and present a theory of change model to guide educational initiatives.

METHODS

Six electronic databases were searched for peer-reviewed, English-language studies published between 2016 and 2021 on ophthalmic educational initiatives employing a theoretical framework. Quality of studies was assessed using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach; risk of bias was evaluated using the Medical Education Research Study Quality Instrument (MERSQI) and the Accreditation Council for Graduate Medical Education (ACGME) guidelines for evaluation of assessment methods. Abstracted components of the included studies were used to develop a theory of change model.

RESULTS

The literature search yielded 1661 studies: 666 were duplicates, 834 studies were excluded after abstract review, and 132 after full-text review; 29 studies (19.2%) employing a theoretical framework were included. The theories used most frequently were the Dreyfus model of skill acquisition and Messick's contemporary validity framework. GRADE ratings were predominantly "low," the average MERSQI score was 10.04, and the ACGME recommendation for all assessment development studies was the lowest recommendation. The theory of change model outlined how educators can select, apply, and evaluate theory-based interventions.

CONCLUSIONS

Few ophthalmic medical education studies employed a theoretical framework; their overall rigor was low as assessed by GRADE, MERSQI, and ACGME guidelines. A theory of change model can guide integration of theoretical frameworks into educational initiatives.

Modern Educational Simulation-Based Tools Among Residents of Ophthalmology: A Narrative Review

INTRODUCTION

With no specific overview on simulation-based training for educational purposes among residents in ophthalmology, this narrative review attempts to highlight the current literature on modern educational simulation-based tools used to educate residents in ophthalmology.

METHODS

We searched databases Web of Science and PubMed between March 15 and July 21, 2022. Relevant and accessible articles and abstracts published after 2006 and in English only were included.

RESULTS

Simulation-based cataract surgery is associated with better outcomes in the operating room and faster surgeries. Construct validity has been established across different procedures and levels in simulation-based cataract surgery and simulation-based vitreoretinal surgery. Other simulation-based procedures indicate promising results but in general lack evidence-based validity.

DISCUSSION

This narrative review highlights and evaluates the current and relevant literature of modern educational simulation-based tools to train ophthalmology residents in different fundamental skills like simulation-based ophthalmoscopy and complex surgical procedures like simulation-based cataract surgery and vitreoretinal surgery. Some studies attempt to develop simulators for the use in education of ophthalmology residents. Other studies strive to establish validity of the respective procedures or modern education tools and some studies investigate the effect of simulation-based training. The most validated modern educational simulation-based tool is the Eyesi Surgical Simulator (VRmagic, Germany). However, other modern educational simulation-based tools have also been evaluated, including the HelpMeSee Eye Surgery Simulator (HelpMeSee Inc., New York, USA) and the MicroVisTouch Surgical Simulator (ImmersiveTouch, USA).

CONCLUSION

Simulation-based training has already been established for residents in ophthalmology to benefit the most from skill-demanding procedures resulting in better learning and better patient handling. Future studies should aim to validate more simulation-based procedures for the teaching of ophthalmology residents so that the evidence is kept at a high standard.

Development of and Gathering Validity Evidence for a Theoretical Test in Contrast-Enhanced Ultrasound

Contrast-enhanced ultrasound (CEUS) is an imaging modality applied in a broad field of medical specialties for diagnostic uses, guidance during biopsy procedures and ablation therapies and sonoporation therapy. Appropriate training and assessment of theoretical and practical competencies are recommended before practicing CEUS, but no validated assessment tools exist. This study was aimed at developing a theoretical multiple-choice question-based test for core CEUS competencies and gathering validity evidence for the test. An expert team developed the test via a Delphi process. The test was administered to medical doctors with varying CEUS experience, and the results were used to evaluate test items, internal-consistency reliability, ability to distinguish between different proficiency levels and to establish a pass/fail score. Validity evidence was gathered according to Messick's framework. The final test with 47 test items could distinguish between operators with and without CEUS experience with acceptable reliability. The pass/fail score led to considerable risk of false positives and negatives. The test may be used as an entry test before learning practical CEUS competencies but is not recommended for certification purposes because of the risk of false positives and negatives.

Reliable test of clinicians' mastery in skin cancer diagnostics

Differentiating between benign and malignant skin lesions can be very difficult and should only be done by sufficiently trained and skilled clinicians. To our knowledge there are no validated tests for reliable assessments of clinicians' ability to perform skin cancer diagnostics. To develop and gather validity evidence for a test in skin cancer diagnostics, a multiple-choice questionnaire (MCQ) was developed based on informal interviews with seven content experts from five skin cancer centers in Denmark. Validity evidence for the test was gathered from May until July 2019 using Messick's validity framework (content, response process, internal structure, relationship to other variables and consequences). Item content was revised through a Delphi-like review process and then piloted on 36 medical students and 136 doctors using a standardized response process. Results enabled an analysis of the internal structure and relationship to other variables of the test. Finally, the contrasting groups method was used to investigate the test's consequences (pass-fail standard). The initial 90-item MCQ was reduced to 40 items during the Delphi-like review process. Item analysis revealed that 25 of the 40 selected items were level I-III quality items with a high internal consistency (Cronbach's α = 0.83) and highly significant (P ≤ 0.0001) differences in test scores between participants with different occupations or levels of experience. A pass-fail standard of 12 (48%) correct answers was established using the contrasting groups' method. The skin cancer diagnostics MCQ developed in this study can be used for reliable assessments of clinicians' competencies.