Adverse Events from Emergency Physician Pediatric Extremity Radiograph Interpretations: A Prospective Cohort Study

Black and white: how good are clinicians at diagnosing elbow injuries from paediatric elbow radiographs alone?

OBJECTIVES

Paediatric trauma elbow radiographs are difficult to interpret and there is a potential for harm if misdiagnosed. The primary goal of this study was to assess the ability of healthcare professionals internationally to interpret paediatric trauma elbow radiographs from the radiograph alone by formulating the correct diagnosis.

METHODS

This prospective international study was conducted online via the Free Open Access Medical Education platform, Don't Forget the Bubbles (DFTB, ISSN 2754-5407). Participants were recruited via the DFTB social media accounts between 17 August and 14 September 2021. Submissions that were incomplete or from participants who do not interpret paediatric elbow radiographs in their clinical practice were excluded. Participants completed an online survey of demographic data followed by interpreting 10 trauma-indicated elbow radiographs, by selecting multiple-choice options. The primary outcome was correct diagnosis.

RESULTS

Participant responses from 18 countries were analysed, with most responses from the UK, Australia and Ireland. Participants had backgrounds in emergency medicine (EM), paediatric emergency medicine (PEM), general practice (GP) and paediatrics, with over 70% having 6+ years of postgraduate experience. 3180 radiographs were interpreted by 318 healthcare professionals. Only nine (2.8%) participants correctly diagnosed all 10. The mean number of radiographs correctly interpreted was 5.44 (SD 2.3). The mean number for those with 6+ years of experience was 6.02 (SD 2.2). On reviewing the normal radiograph, 158 (49.7%) overcalled injuries. Participants with EM or PEM background were equally likely to have more correct answers than those from paediatric or GP backgrounds.

CONCLUSION

Globally, healthcare professional's success in correctly diagnosing paediatric elbow injuries from radiographs was suboptimal in this non-clinical exercise, despite capturing quite an experienced cohort of clinicians. This study has provided us with detailed baseline data to accurately assess the impact of interventions aimed at improving clinicians' interpretation of paediatric elbow radiographs in future studies.

The diagnostic accuracy of community spine radiology for adolescent idiopathic scoliosis brace candidates

PURPOSE

The study aims to establish the diagnostic accuracy of community spine x-rays for brace candidates.

METHODS

A review of adolescent idiopathic scoliosis patients seen for initial visit at a tertiary care pediatric hospital was conducted (n = 170). The index test was the pre-referral community spine x-ray interpreted by a community radiologist. Measures of diagnostic accuracy for the index test were determined against the reference standard if images were obtained within 90 days (n = 111). The reference standard was the 3-foot standing EOS spine x-ray evaluated by spine specialists. Diagnostic criterion for a brace candidate was dichotomized by Cobb angle range (25-40°) according to Scoliosis Research Society criteria. Risser stage was not included given significant missing data in index reports. To mitigate the uncertainty around true progression, sensitivity analyses were conducted on a sub-sample of data when index test was within 60 days of the reference standard (n = 67).

RESULTS

Accuracy of the community spine x-ray to detect a brace candidate was 65.8% (95% CI 56.2-74.5). Sensitivity of the index test was 65.4% with a false negative rate of 34.6%. Specificity was 66.1% with a false positive rate of 33.9%. Positive and negative predictive values were 63.0% and 68.4%, respectively. Of the total number of brace candidates (n = 52), 32.7% were missed because of underestimation in Cobb angle (95% CI 21.5-46.2). The proportion of missed brace candidates because of underestimation was unchanged with 60-day data (p = 0.37).

CONCLUSIONS

Inaccuracies in community spine radiology may lead to missed opportunities for non-operative treatment.

"How I would like AI used for my imaging": children and young persons' perspectives

OBJECTIVES

Artificial intelligence (AI) tools are becoming more available in modern healthcare, particularly in radiology, although less attention has been paid to applications for children and young people. In the development of these, it is critical their views are heard.

MATERIALS AND METHODS

A national, online survey was publicised to UK schools, universities and charity partners encouraging any child or young adult to participate. The survey was "live" for one year (June 2022 to 2023). Questions about views of AI in general, and in specific circumstances (e.g. bone fractures) were asked.

RESULTS

One hundred and seventy-one eligible responses were received, with a mean age of 19 years (6-23 years) with representation across all 4 UK nations. Most respondents agreed or strongly agreed they wanted to know the accuracy of an AI tool that was being used (122/171, 71.3%), that accuracy was more important than speed (113/171, 66.1%), and that AI should be used with human oversight (110/171, 64.3%). Many respondents (73/171, 42.7%) felt AI would be more accurate at finding problems on bone X-rays than humans, with almost all respondents who had sustained a missed fracture strongly agreeing with that sentiment (12/14, 85.7%).

CONCLUSIONS

Children and young people in our survey had positive views regarding AI, and felt it should be integrated into modern healthcare, but expressed a preference for a "medical professional in the loop" and accuracy of findings over speed. Key themes regarding information on AI performance and governance were raised and should be considered prior to future AI implementation for paediatric healthcare.

CLINICAL RELEVANCE STATEMENT

Artificial intelligence (AI) integration into clinical practice must consider all stakeholders, especially paediatric patients who have largely been ignored. Children and young people favour AI involvement with human oversight, seek assurances for safety, accuracy, and clear accountability in case of failures.

KEY POINTS

Paediatric patient's needs and voices are often overlooked in AI tool design and deployment. Children and young people approved of AI, if paired with human oversight and reliability. Children and young people are stakeholders for developing and deploying AI tools in paediatrics.

Diagnostic Accuracy of Point-of-Care Ultrasound Versus Radiographic Imaging for Pediatric Distal Forearm Fractures: A Randomized Controlled Trial

STUDY OBJECTIVE

In patients aged 5 to 15 years with a clinically nondeformed distal forearm injury presenting to the emergency department (ED), we examined whether point-of-care ultrasound or radiographic imaging had better diagnostic accuracy, with the reference diagnosis determined by an expert panel review.

METHODS

This multicenter, open-label, diagnostic randomized controlled trial was conducted in South East Queensland, Australia. Eligible patients were randomized to receive initial imaging through point-of-care ultrasound performed by an ED clinician or radiograph. Images were defined as "no," "buckle," or "other" fracture by the treating clinician. The primary outcome was the diagnostic accuracy of the treating clinician's interpretation compared against the reference standard diagnosis, which was determined retrospectively by an expert panel consisting of an emergency physician, pediatric radiologist, and pediatric orthopedic surgeon, who reviewed all imaging and follow-up.

RESULTS

Two-hundred and seventy participants were enrolled, with 135 randomized to each initial imaging modality. There were 132 (97.8%) and 112 (83.0%) correctly diagnosed participants by ED clinicians in the point-of-care ultrasound and radiograph groups, respectively (absolute difference [AD]=14.8%; 95% confidence interval [CI] 8.0% to 21.6%; P<.001). Point-of-care ultrasound had better accuracy for participants with "buckle" fractures (AD=18.5%; 95% CI 7.1% to 29.8%) and "other" fractures (AD=17.1%; 95% CI 2.7% to 31.6%). No clinically important fractures were missed in either group.

CONCLUSION

In children and adolescents presenting to the ED with a clinically nondeformed distal forearm injury, clinician-performed (acquired and interpreted) point-of-care ultrasound more accurately identified the correct diagnosis than clinician-interpreted radiographic imaging.

External validation of an artificial intelligence solution for the detection of elbow fractures and joint effusions in children

PURPOSE

The purpose of this study was to conduct an external validation of an artificial intelligence (AI) solution for the detection of elbow fractures and joint effusions using radiographs from a real-life cohort of children.

MATERIALS AND METHODS

This single-center retrospective study was conducted on 758 radiographic sets (1637 images) obtained from consecutive emergency room visits of 712 children (mean age, 7.27 ± 3.97 [standard deviation] years; age range, 7 months and 10 days to 15 years and 10 months), referred for a trauma of the elbow. For each set, fracture and/or effusion detection by eleven senior radiologists (reference standard) and AI solution was recorded. Diagnostic performance of the AI solution was measured via four different approaches: fracture detection (presence/absence of fracture as binary variable), fracture enumeration, fracture localization and lesion detection (fracture and/or a joint effusion used as constructed binary variable).

RESULTS

The sensitivity of the AI solution for each of the four approaches was >89%. Greatest sensitivity of the AI solution was obtained for lesion detection (95.0%; 95% confidence interval: 92.1-96.9). The specificity of the AI solution ranged between 63% (for lesion detection) and 77% (for fracture detection). For all four approaches, the negative predictive values were >92% and the positive predictive values ranged between 54% (for fracture enumeration and localization) and 73% (for lesion detection). Specificity was lower for plastered children for all approaches (P < 0.001).

CONCLUSION

The AI solution demonstrates high performances for detecting elbow's fracture and/or joint effusion in children. However, in our context of use, 8% of the radiographic sets ruled-out by the algorithm concerned children with a genuine traumatic elbow lesion.

Identification of fractures on pediatric foot radiographs: do localization cues improve diagnostic accuracy and reduce interpretation time?

OBJECTIVE

To investigate the diagnostic accuracy and time in the detection of fractures on pediatric foot radiographs marked without and with localization cues.

METHOD

One-hundred randomly selected foot radiographic examinations that were performed on children (<18 years old) after injury and with at least 4 weeks of follow-up were included. Blinded to history and diagnosis, 4 readers (one each: medical student, pediatrician, pediatric orthopedic surgeon, and pediatric musculoskeletal radiologist) retrospectively and independently reviewed each examination twice (without and with cue, at least 1 month apart, and after randomization). Each reader recorded the presence or absence of a fracture, fracture location, diagnostic confidence, and the total (interpretation) time spent on each study. Diagnostic accuracy, reader confidence, and interpretation time were compared between examinations without and with cues.

RESULTS

Our study included 59 examinations without and 41 with fractures (21 phalangeal, 18 metatarsal, and 2 tarsal fractures). Localization cues improved inter-reader agreement (κ=0.36 to 0.64), overall sensitivity (68 to 72%), specificity (66 to 73%), and diagnostic accuracy (67 to 73%); thus, overcalled and missed rates also improved from 34 to 27% and 32 to 28%, respectively. Reader confidence improved with cue (49 to 61%, p<0.01) with higher incremental improvement with younger children (30% for 1-6 years; 14% for 7-11 years; and 10% for 12-17 years). Interpretation time decreased by 40% per examination (40±22 s without to 24±13 s with cues, p<0.001).

CONCLUSION

Localization cues improved diagnostic accuracy and reader confidence, reducing interpretation time in the detection of pediatric foot fractures.

Common and Consequential Fractures That Should Not Be Missed in Children

Missed or delayed diagnosis of fractures in children is not uncommon owing to their immature skeletons, unique fracture patterns, and distinctive radiologic findings. The term occult is used to describe radiographically subtle fractures. Some of these fractures can be associated with excellent outcomes despite the pitfalls of delayed diagnosis. However, a subset of these injuries have more guarded prognoses when missed, despite their harmless radiographic appearance. A high index of suspicion should be maintained when treating pediatric extremity injuries with clinical findings disproportionate to a benign-appearing radiograph. Moreover, overreliance on radiology reports can perpetuate diagnostic error. In cases of discrepancy, timely follow-up for repeat examination or immediate advanced imaging can help avoid missed diagnoses. Most critically, the one diagnosis not to miss is nonaccidental trauma, as continued exposure to abuse puts the child at risk of further injury and death. [Pediatr Ann. 2022;51(9):e357-e363.].

Artificial intelligence for radiological paediatric fracture assessment: a systematic review

BACKGROUND

Majority of research and commercial efforts have focussed on use of artificial intelligence (AI) for fracture detection in adults, despite the greater long-term clinical and medicolegal implications of missed fractures in children. The objective of this study was to assess the available literature regarding diagnostic performance of AI tools for paediatric fracture assessment on imaging, and where available, how this compares with the performance of human readers.

MATERIALS AND METHODS

MEDLINE, Embase and Cochrane Library databases were queried for studies published between 1 January 2011 and 2021 using terms related to 'fracture', 'artificial intelligence', 'imaging' and 'children'. Risk of bias was assessed using a modified QUADAS-2 tool. Descriptive statistics for diagnostic accuracies were collated.

RESULTS

Nine eligible articles from 362 publications were included, with most (8/9) evaluating fracture detection on radiographs, with the elbow being the most common body part. Nearly all articles used data derived from a single institution, and used deep learning methodology with only a few (2/9) performing external validation. Accuracy rates generated by AI ranged from 88.8 to 97.9%. In two of the three articles where AI performance was compared to human readers, sensitivity rates for AI were marginally higher, but this was not statistically significant.

CONCLUSIONS

Wide heterogeneity in the literature with limited information on algorithm performance on external datasets makes it difficult to understand how such tools may generalise to a wider paediatric population. Further research using a multicentric dataset with real-world evaluation would help to better understand the impact of these tools.

Pediatric Musculoskeletal Radiographs: Anatomy and Fractures Prone to Diagnostic Error Among Emergency Physicians

BACKGROUND

Pediatric musculoskeletal (pMSK) radiograph interpretations are common, but the specific radiograph features at risk of incorrect diagnosis are relatively unknown.

OBJECTIVE

We determined the radiograph factors that resulted in diagnostic interpretation challenges for emergency physicians (EPs) reviewing pMSK radiographs.

METHODS

EPs interpreted 1850 pMSK radiographs via a web-based platform and we derived interpretation difficulty scores for each radiograph in 13 body regions using one-parameter item response theory. We compared the difficulty scores by presence or absence of a fracture and, where applicable, by fracture location and morphology; significance was adjusted for multiple comparisons. An expert panel reviewed the 65 most commonly misdiagnosed fracture-negative radiographs to identify imaging features mistaken for fractures.

RESULTS

We included data from 244 EPs, which resulted in 185,653 unique interpretations. For elbow, forearm, wrist, femur, knee, and tibia-fibula radiographs, those without a fracture had higher interpretation difficulty scores relative to those with a fracture; the opposite was true for the hand, pelvis, foot, and ankle radiographs (p < 0.004 for all comparisons). The descriptive review demonstrated that specific normal anatomy, overlapping bones, and external artefact from muscle or skin folds were often mistaken for fractures. There was a significant difference in difficulty score by anatomic locations of the fracture in the elbow, pelvis, and ankle (p < 0.004 for all comparisons). Ankle and elbow growth plate, fibular avulsion, and humerus condylar fractures were more difficult to diagnose than other fracture patterns (p < 0.004 for all comparisons).

CONCLUSIONS

We identified actionable learning opportunities in pMSK radiograph interpretation for EPs.

A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation

CONSTRUCT

For assessing the skill of visual diagnosis such as radiograph interpretation, competency standards are often developed in an ad hoc method, with a poorly delineated connection to the target clinical population.

BACKGROUND

Commonly used methods to assess for competency in radiograph interpretation are subjective and potentially biased due to a small sample size of cases, subjective evaluations, or include an expert-generated case-mix versus a representative sample from the clinical field. Further, while digital platforms are available to assess radiograph interpretation skill against an objective standard, they have not adopted a data-driven competency standard which informs educators and the public that a physician has achieved adequate mastery to enter practice where they will be making high-stakes clinical decisions.

APPROACH

Operating on a purposeful sample of radiographs drawn from the clinical domain, we adapted the Ebel Method, an established standard setting method, to ascertain a defensible, clinically relevant mastery learning competency standard for the skill of radiograph interpretation as a model for deriving competency thresholds in visual diagnosis. Using a previously established digital platform, emergency physicians interpreted pediatric musculoskeletal extremity radiographs. Using one-parameter item response theory, these data were used to categorize radiographs by interpretation difficulty terciles (i.e. easy, intermediate, hard). A panel of emergency physicians, orthopedic surgeons, and plastic surgeons rated each radiograph with respect to clinical significance (low, medium, high). These data were then used to create a three-by-three matrix where radiographic diagnoses were categorized by interpretation difficulty and significance. Subsequently, a multidisciplinary panel that included medical and parent stakeholders determined acceptable accuracy for each of the nine cells. An overall competency standard was derived from the weighted sum. Finally, to examine consequences of implementing this standard, we reported on the types of diagnostic errors that may occur by adhering to the derived competency standard.

FINDINGS

To determine radiograph interpretation difficulty scores, 244 emergency physicians interpreted 1,835 pediatric musculoskeletal extremity radiographs. Analyses of these data demonstrated that the median interpretation difficulty rating of the radiographs was -1.8 logits (IQR -4.1, 3.2), with a significant difference of difficulty across body regions (p < 0.0001). Physician review classified the radiographs as 1,055 (57.8%) as low, 424 (23.1%) medium or 356 (19.1%) high clinical significance. The multidisciplinary panel suggested a range of acceptable scores between cells in the three-by-three table of 76% to 95% and the sum of equal-weighted scores resulted in an overall performance-based competency score of 85.5% accuracy. Of the 14.5% diagnostic interpretation errors that may occur at the bedside if this competency standard were implemented, 9.8% would be in radiographs of low-clinical significance, while 2.5% and 2.3% would be in radiographs of medium or high clinical significance, respectively.

CONCLUSION(S)

This study's novel integration of radiograph selection and a standard setting method could be used to empirically drive evidence-based competency standard for radiograph interpretation and can serve as a model for deriving competency thresholds for clinical tasks emphasizing visual diagnosis.