Deliberate practice as an educational method for learning to interpret the prepubescent female genital examination

Deliberate Practice as an Effective Method for Reducing Diagnostic Error in Identifying Burn and Bruise Injuries Suspicious for an Abusive Injury

OBJECTIVES

To examine the effectiveness of an education intervention for reducing physician diagnostic error in identifying pediatric burn and bruise injuries suspicious for abuse, and to determine case-specific variables associated with an increased risk of diagnostic error.

STUDY DESIGN

This was a multicenter, prospective, cross-sectional study. A convenience sample of pediatricians and other front-line physicians who treat acutely injured children in the United States and Canada were eligible for participation. Using a web-based education and assessment platform, physicians deliberately practiced with a spectrum of 300 pediatric burn and bruise injury image-based cases. Participants were asked if there was a suspicion for abuse present or absent, were given corrective feedback after every case, and received summative diagnostic performance overall (accuracy), suspicion for abuse present (sensitivity), and absent (specificity).

RESULT

Of the 93/137 (67.9%) physicians who completed all 300 cases, there was a significant reduction in diagnostic error (initial 16.7%, final 1.6%; delta -15.1%; 95% CI 13.5, 16.7), sensitivity error (initial 11.9%, final 0.7%; delta -11.2%; 95% CI 9.8, 12.5), and specificity error (initial 23.3%, final 6.6%; delta -16.7%; 95% CI 14.8, 18.6). Based on 35,627 case interpretations, variables associated with diagnostic error included patient age, sex, skin color, mechanism of injury, and size and pattern of injury.

CONCLUSION

The education intervention substantially reduced diagnostic error in differentiating the presence versus absence of a suspicion for abuse in children with burn and bruise injuries. Several case-based variables were associated with diagnostic error, and these data can be used to close specific skill gaps in this clinical domain.

Interactive computer-assisted learning as an educational method for learning pediatric interproximal dental caries identification

OBJECTIVE

We developed a web-based tool to measure the amount and rate of skill acquisition in pediatric interproximal caries diagnosis among pre- and postdoctoral dental students and identified variables predictive for greater image interpretation difficulty.

STUDY DESIGN

In this multicenter prospective cohort study, a convenience sample of pre- and postdoctoral dental students participated in computer-assisted learning in the interpretation of bitewing radiographs of 193 children. Participants were asked to identify the presence or absence of interproximal caries and, where applicable, locate the lesions. After every case, participants received specific visual and text feedback on their diagnostic performance. They were requested to complete the 193-case set but could complete enough cases to achieve a competency performance standard of 75% accuracy, sensitivity, and specificity.

RESULTS

Of 130 participants, 62 (47.7%) completed all cases. The mean change from initial to maximal diagnostic accuracy was +15.3% (95% CI, 13.0-17.7), sensitivity was +10.8% (95% CI, 9.0-12.7), and specificity was +15.5% (95% CI, 12.9-18.1). The median number of cases completed to achieve competency was 173 (interquartile range, 82-363). Of these 62 participants, 45 (72.6%) showed overall improvement in diagnostic accuracy. Greater numbers of interproximal lesions (P < .001) and the presence of noninterproximal caries (P < .001) predicted greater interpretation difficulty.

CONCLUSIONS

Computer-assisted learning led to improved diagnosis of interproximal caries on bitewing radiographs among pre- and postdoctoral dental students.

Competency Standard Derivation for Point-of-Care Ultrasound Image Interpretation for Emergency Physicians

STUDY OBJECTIVE

Because number-based standards are increasingly controversial, the objective of this study was to derive a performance-based competency standard for the image interpretation task of point-of-care ultrasound (POCUS).

METHODS

This was a prospective study. Operating on a clinically-relevant sample of POCUS images, we adapted the Ebel standard-setting method to derive a performance benchmark in 4 diverse pediatric POCUS applications: soft tissue, lung, cardiac and focused assessment with sonography in trauma (FAST). In Phase I (difficulty calibration), cases were categorized into interpretation difficulty terciles (easy, intermediate, hard) using emergency physician-derived data. In Phase II (significance), a 4-person expert panel categorized cases as low, medium, or high clinical significance. In Phase III (standard setting), a 3x3 matrix was created, categorizing cases by difficulty and significance, and a 6-member panel determined acceptable accuracy for each of the 9 cells. An overall competency standard was derived from the weighted sum.

RESULTS

We obtained data from 379 emergency physicians resulting in 67,093 interpretations and a median of 184 (interquartile range, 154, 190) interpretations per case. There were 78 (19.5%) easy, 272 (68.0%) medium, and 50 (12.5%) hard-to-interpret cases, and 237 (59.3%) low, 65 (16.3%) medium, and 98 (24.5%) cases of high clinical significance across the 4 POCUS applications. The panel determined an overall performance-based competency score of 85.0% for lung, 89.5% for cardiac, 90.5% for soft tissue, and 92.7% for FAST.

CONCLUSION

This research provides a transparent chain of evidence that derived clinically relevant competency standards for POCUS image interpretation.

Multi-level longitudinal learning curve regression models integrated with item difficulty metrics for deliberate practice of visual diagnosis: groundwork for adaptive learning

Visual diagnosis of radiographs, histology and electrocardiograms lends itself to deliberate practice, facilitated by large online banks of cases. Which cases to supply to which learners in which order is still to be worked out, with there being considerable potential for adapting the learning. Advances in statistical modeling, based on an accumulating learning curve, offer methods for more effectively pairing learners with cases of known calibrations. Using demonstration radiograph and electrocardiogram datasets, the advantages of moving from traditional regression to multilevel methods for modeling growth in ability or performance are demonstrated, with a final step of integrating case-level item-response information based on diagnostic grouping. This produces more precise individual-level estimates that can eventually support learner adaptive case selection. The progressive increase in model sophistication is not simply statistical but rather brings the models into alignment with core learning principles including the importance of taking into account individual differences in baseline skill and learning rate as well as the differential interaction with cases of varying diagnosis and difficulty. The developed approach can thus give researchers and educators a better basis on which to anticipate learners' pathways and individually adapt their future learning.

Prepubescent Female Genital Examination Images: Evidence-Informed Learning Opportunities

OBJECTIVES

To determine diagnoses and image features that are associated with difficult prepubescent female genital image interpretations.

DESIGN AND SETTING

This was a mixed-methods study conducted at a tertiary care pediatric center using images from a previously developed education platform.

PARTICIPANTS

Participants comprised 107 medical students, residents, fellows, and attendings who interpreted 158 cases to derive case difficulty estimates.

INTERVENTIONS

This was a planned secondary analysis of participant performance data obtained from a prospective multi-center cross-sectional study. An expert panel also performed a descriptive review of images with the highest frequency of diagnostic error.

MAIN OUTCOME MEASURES

We derived the proportion of participants who interpreted an image correctly, and features that were common in images with the most frequent diagnostic errors.

RESULTS

We obtained 16,906 image interpretations. The mean proportion correct scores for each diagnosis were as follows: normal/normal variants 0.84 (95% confidence interval [CI] 0.82, 0.87); infectious/dermatology pathology 0.59 (95% CI 0.45, 0.73); anatomic pathology 0.61 (95% CI 0.41, 0.81); and, traumatic pathology 0.64 (95% CI 0.49, 0.79). The mean proportion correct scores varied by diagnosis (P < .001). The descriptive review demonstrated that poor image quality, infant genitalia, normal variant anatomy, external material (eg, diaper cream) in the genital area, and nonspecific erythema were common features in images with lower accuracy scores.

CONCLUSIONS

A quantitative and qualitative examination of prepubescent female genital examination image interpretations provided insight into diagnostic challenges for this complex examination. These data can be used to inform the design of teaching interventions to improve skill in this area.

Child Abuse Recognition Training for Prehospital Providers Using Deliberate Practice

BACKGROUND

In most states, prehospital professionals (PHPs) are mandated reporters of suspected abuse but cite a lack of training as a challenge to recognizing and reporting physical abuse. We developed a learning platform for the visual diagnosis of pediatric abusive versus non-abusive burn and bruise injuries and examined the amount and rate of skill acquisition.

METHODS

This was a prospective cross-sectional study of PHPs participating in an online educational intervention containing 114 case vignettes. PHPs indicated whether they believed a case was concerning for abuse and would report a case to child protection services. Participants received feedback after submitting a response, permitting deliberate practice of the cases. We describe learning curves, overall accuracy, sensitivity (diagnosis of abusive injuries) and specificity (diagnosis of non-abusive injuries) to determine the amount of learning. We performed multivariable regression analysis to identify specific demographic and case variables associated with a correct case interpretation. After completing the educational intervention, PHPs completed a self-efficacy survey on perceived gains in their ability to recognize cutaneous signs of abuse and report to social services.

RESULTS

We enrolled 253 PHPs who completed all the cases; 158 (63.6%) emergency medical technicians (EMT), 95 (36.4%) advanced EMT and paramedics. Learning curves demonstrated that, with one exception, there was an increase in learning for participants throughout the educational intervention. Mean diagnostic accuracy increased by 4.9% (95% CI 3.2, 6.7), and the mean final diagnostic accuracy, sensitivity, and specificity were 82.1%, 75.4%, and 85.2%, respectively. There was an increased odds of getting a case correct for bruise versus burn cases (OR = 1.4; 95% CI 1.3, 1.5); if the PHP was an Advanced EMT/Paramedic (OR = 1.3; 95% CI 1.1, 1.4) ; and, if the learner indicated prior training in child abuse (OR = 1.2; 95% CI 1.0, 1.3). Learners indicated increased comfort in knowing which cases should be reported and interpreting exams in children with cutaneous injuries with a median Likert score of 5 out of 6 (IQR 5, 6).

CONCLUSION

An online module utilizing deliberate practice led to measurable skill improvement among PHPs for differentiating abusive from non-abusive burn and bruise injuries.