Current Status of Handheld Otoscopy Training: A Systematic Review

Gathering Validity Evidence for a Simulation-Based Test of Otoscopy Skills

OBJECTIVE

Otoscopy is a key clinical examination used by multiple healthcare providers but training and testing of otoscopy skills remain largely uninvestigated. Simulator-based assessment of otoscopy skills exists, but evidence on its validity is scarce. In this study, we explored automated assessment and performance metrics of an otoscopy simulator through collection of validity evidence according to Messick's framework.

METHODS

Novices and experienced otoscopists completed a test program on the Earsi otoscopy simulator. Automated assessment of diagnostic ability and performance were compared with manual ratings of technical skills. Reliability of assessment was evaluated using Generalizability theory. Linear mixed models and correlation analysis were used to compare automated and manual assessments. Finally, we used the contrasting groups method to define a pass/fail level for the automated score.

RESULTS

A total of 12 novices and 12 experienced otoscopists completed the study. We found an overall G-coefficient of .69 for automated assessment. The experienced otoscopists achieved a significantly higher mean automated score than the novices (59.9% (95% CI [57.3%-62.6%]) vs. 44.6% (95% CI [41.9%-47.2%]), P < .001). For the manual assessment of technical skills, there was no significant difference, nor did the automated score correlate with the manually rated score (Pearson's r = .20, P = .601). We established a pass/fail standard for the simulator's automated score of 49.3%.

CONCLUSION

We explored validity evidence supporting an otoscopy simulator's automated score, demonstrating that this score mainly reflects cognitive skills. Manual assessment therefore still seems necessary at this point and external video-recording is necessary for valid assessment. To improve the reliability, the test course should include more cases to achieve a higher G-coefficient and a higher pass/fail standard should be used.

Smartphone-Enabled versus Conventional Otoscopy in Detecting Middle Ear Disease: A Meta-Analysis

Traditional otoscopy has some limitations, including poor visualization and inadequate time for evaluation in suboptimal environments. Smartphone-enabled otoscopy may improve examination quality and serve as a potential diagnostic tool for middle ear diseases using a telemedicine approach. The main objectives are to compare the correctness of smartphone-enabled otoscopy and traditional otoscopy and to evaluate the diagnostic confidence of the examiner via meta-analysis. From inception through 20 January 2022, the Cochrane Library, PubMed, EMBASE, Web of Science, and Scopus databases were searched. Studies comparing smartphone-enabled otoscopy with traditional otoscopy regarding the outcome of interest were eligible. The relative risk (RR) for the rate of correctness in diagnosing ear conditions and the standardized mean difference (SMD) in diagnostic confidence were extracted. Sensitivity analysis and trial sequential analyses (TSAs) were conducted to further examine the pooled results. Study quality was evaluated by using the revised Cochrane risk of bias tool 2. Consequently, a total of 1840 examinees were divided into the smartphone-enabled otoscopy group and the traditional otoscopy group. Overall, the pooled result showed that smartphone-enabled otoscopy was associated with higher correctness than traditional otoscopy (RR, 1.26; 95% CI, 1.06 to 1.51; p = 0.01; I² = 70.0%). Consistently significant associations were also observed in the analysis after excluding the simulation study (RR, 1.10; 95% CI, 1.00 to 1.21; p = 0.04; I² = 0%) and normal ear conditions (RR, 1.18; 95% CI, 1.01 to 1.40; p = 0.04; I² = 65.0%). For the confidence of examiners using both otoscopy methods, the pooled result was nonsignificant between the smartphone-enabled otoscopy and traditional otoscopy groups (SMD, 0.08; 95% CI, -0.24 to 0.40; p = 0.61; I² = 16.3%). In conclusion, smartphone-enabled otoscopy was associated with a higher rate of correctness in the detection of middle ear diseases, and in patients with otologic complaints, the use of smartphone-enabled otoscopy may be considered. More large-scale studies should be performed to consolidate the results.