Reporting Discrepancy Resolved by Findings and Time in 2947 Emergency Department Ankle X-rays

Evaluating the impact of artificial intelligence-assisted image analysis on the diagnostic accuracy of front-line clinicians in detecting fractures on plain X-rays (FRACT-AI): protocol for a prospective observational study

INTRODUCTION

Missed fractures are the most frequent diagnostic error attributed to clinicians in UK emergency departments and a significant cause of patient morbidity. Recently, advances in computer vision have led to artificial intelligence (AI)-enhanced model developments, which can support clinicians in the detection of fractures. Previous research has shown these models to have promising effects on diagnostic performance, but their impact on the diagnostic accuracy of clinicians in the National Health Service (NHS) setting has not yet been fully evaluated.

METHODS AND ANALYSIS

A dataset of 500 plain radiographs derived from Oxford University Hospitals (OUH) NHS Foundation Trust will be collated to include all bones except the skull, facial bones and cervical spine. The dataset will be split evenly between radiographs showing one or more fractures and those without. The reference ground truth for each image will be established through independent review by two senior musculoskeletal radiologists. A third senior radiologist will resolve disagreements between two primary radiologists. The dataset will be analysed by a commercially available AI tool, BoneView (Gleamer, Paris, France), and its accuracy for detecting fractures will be determined with reference to the ground truth diagnosis. We will undertake a multiple case multiple reader study in which clinicians interpret all images without AI support, then repeat the process with access to AI algorithm output following a 4-week washout. 18 clinicians will be recruited as readers from four hospitals in England, from six distinct clinical groups, each with three levels of seniority (early-stage, mid-stage and later-stage career). Changes in the accuracy, confidence and speed of reporting will be compared with and without AI support. Readers will use a secure web-based DICOM (Digital Imaging and Communications in Medicine) viewer (www.raiqc.com), allowing radiograph viewing and abnormality identification. Pooled analyses will be reported for overall reader performance as well as for subgroups including clinical role, level of seniority, pathological finding and difficulty of image.

ETHICS AND DISSEMINATION

The study has been approved by the UK Healthcare Research Authority (IRAS 310995, approved on 13 December 2022). The use of anonymised retrospective radiographs has been authorised by OUH NHS Foundation Trust. The results will be presented at relevant conferences and published in a peer-reviewed journal.

TRIAL REGISTRATION NUMBERS

This study is registered with ISRCTN (ISRCTN19562541) and ClinicalTrials.gov (NCT06130397). The paper reports the results of a substudy of STEDI2 (Simulation Training for Emergency Department Imaging Phase 2).

Influence of residency discipline and seniority on traumatic musculoskeletal radiographs interpretation accuracy: a multicenter study

OBJECTIVES

Imaging studies are a significant and integral part of the initial assessment of patients admitted to the emergency department. Developing imaging diagnostic abilities early in residency is of paramount importance. The purpose of this study was to evaluate and compare diagnosis accuracy of common musculoskeletal X-rays (XR) between residency disciplines and seniority.

METHODS

A multicenter study which evaluated orthopedic surgery, emergency medicine (EM), and radiology residents, through a test set of common MSK XR. Residents were classified as "beginner" or "advanced" according to postgraduate year per residency. Residents were asked to answer whether the radiograph shows normal or pathological findings (success rate) and what is the diagnosis ("diagnosis accuracy"). Residents' answers were analyzed and assessed compared to experts' consensus.

RESULTS

A total of 100 residents (62% beginners) participated in this study. Fifty-four were orthopedic surgeons, 29 were EM residents and 17 were radiologists. The entire cohort's overall success rate was 88.5%. The overall mean success rates for orthopedic, EM, and radiology residents were 93.2%, 82.8%, and 83.3%, respectively, and were significantly different (p < 0.0001). Orthopedic residents had significantly higher diagnostic accuracy rates compared with both radiology and EM residents (p < 0.001). Advanced orthopedic and EM residents demonstrated higher diagnostic accuracy rates compared to beginner residents (p = 0.001 and p = 0.03, respectively).

CONCLUSION

Orthopedic residents presented higher diagnosis accuracy of MSK imaging compared to EM and radiology residents. Seniority had a positive effect on diagnosis accuracy. The development of an educational program on MSK XR is necessary to enhance the competency of physicians in their daily practice.

Evaluation of ensemble strategy on the development of multiple view ankle fracture detection algorithm

OBJECTIVE

To identify the feasibility and efficiency of deep convolutional neural networks (DCNNs) in the detection of ankle fractures and to explore ensemble strategies that applied multiple projections of radiographs.Ankle radiographs (AXRs) are the primary tool used to diagnose ankle fractures. Applying DCNN algorithms on AXRs can potentially improve the diagnostic accuracy and efficiency of detecting ankle fractures.

METHODS

A DCNN was trained using a trauma image registry, including 3102 AXRs. We separately trained the DCNN on anteroposterior (AP) and lateral (Lat) AXRs. Different ensemble methods, such as "sum-up," "severance-OR," and "severance-Both," were evaluated to incorporate the results of the model using different projections of view.

RESULTS

The AP/Lat model's individual sensitivity, specificity, positive-predictive value, accuracy, and F1 score were 79%/84%, 90%/86%, 88%/86%, 83%/85%, and 0.816/0.850, respectively. Furthermore, the area under the receiver operating characteristic curve (AUROC) of the AP/Lat model was 0.890/0.894 (95% CI: 0.826-0.954/0.831-0.953). The sum-up method generated balanced results by applying both models and obtained an AUROC of 0.917 (95% CI: 0.863-0.972) with 87% accuracy. The severance-OR method resulted in a better sensitivity of 90%, and the severance-Both method obtained a high specificity of 94%.

CONCLUSION

Ankle fracture in the AXR could be identified by the trained DCNN algorithm. The selection of ensemble methods can depend on the clinical situation which might help clinicians detect ankle fractures efficiently without interrupting the current clinical pathway.

ADVANCES IN KNOWLEDGE

This study demonstrated different ensemble strategies of AI algorithms on multiple view AXRs to optimize the performance in various clinical needs.

Reporting errors in plain radiographs for lower limb trauma-a systematic review and meta-analysis

INTRODUCTION

Plain radiographs are a globally ubiquitous means of investigation for injuries to the musculoskeletal system. Despite this, initial interpretation remains a challenge and inaccuracies give rise to adverse sequelae for patients and healthcare providers alike. This study sought to address the limited, existing meta-analytic research on the initial reporting of radiographs for skeletal trauma, with specific regard to diagnostic accuracy of the most commonly injured region of the appendicular skeleton, the lower limb.

METHOD

A prospectively registered, systematic review and meta-analysis was performed using published research from the major clinical-science databases. Studies identified as appropriate for inclusion underwent methodological quality and risk of bias analysis. Meta-analysis was then performed to establish summary rates for specificity and sensitivity of diagnostic accuracy, including covariates by anatomical site, using HSROC and bivariate models.

RESULTS

A total of 3887 articles were screened, with 10 identified as suitable for analysis based on the eligibility criteria. Sensitivity and specificity across the studies were 93.5% and 89.7% respectively. Compared with other anatomical subdivisions, interpretation of ankle radiographs yielded the highest sensitivity and specificity, with values of 98.1% and 94.6% respectively, and a diagnostic odds ratio of 929.97.

CONCLUSION

Interpretation of lower limb skeletal radiographs operates at a reasonably high degree of sensitivity and specificity. However, one in twenty true positives is missed on initial radiographic interpretation and safety netting systems need to be established to address this. Virtual fracture clinic reviews and teleradiology services in conjunction with novel technology will likely be crucial in these circumstances.

Clinician and computer: a study on patient perceptions of artificial intelligence in skeletal radiography

BACKGROUND

Up to half of all musculoskeletal injuries are investigated with plain radiographs. However, high rates of image interpretation error mean that novel solutions such as artificial intelligence (AI) are being explored.

OBJECTIVES

To determine patient confidence in clinician-led radiograph interpretation, the perception of AI-assisted interpretation and management, and to identify factors which might influence these views.

METHODS

A novel questionnaire was distributed to patients attending fracture clinic in a large inner-city teaching hospital. Categorical and Likert scale questions were used to assess participant demographics, daily electronics use, pain score and perceptions towards AI used to assist in interpretation of their radiographs, and guide management.

RESULTS

216 questionnaires were included (M=126, F=90). Significantly higher confidence in clinician rather than AI-assisted interpretation was observed (clinician=9.20, SD=1.27 vs AI=7.06, SD=2.13), 95.4% reported favouring clinician over AI-performed interpretation in the event of disagreement.Small positive correlations were observed between younger age/educational achievement and confidence in AI-assistance. Students demonstrated similarly increased confidence (8.43, SD 1.80), and were over-represented in the minority who indicated a preference for AI-assessment over their clinicians (50%).

CONCLUSIONS

Participant's held the clinician's assessment in the highest regard and expressed a clear preference for it over the hypothetical AI assessment. However, robust confidence scores for the role of AI-assistance in interpreting skeletal imaging suggest patients view the technology favourably.Findings indicate that younger, more educated patients are potentially more comfortable with a role for AI-assistance however further research is needed to overcome the small number of responses on which these observations are based.

Radiologists Make More Errors Interpreting Off-Hours Body CT Studies during Overnight Assignments as Compared with Daytime Assignments

Background There is increasing research attention on the impact of overnight work on radiologist performance. Prior studies on overnight imaging interpretive errors have focused on radiology residents, not on the relative performance of board-eligible or board-certified radiologists at night compared with during the day. Purpose To analyze the rate of clinically important interpretation errors on CT examinations of the abdomen, pelvis, or both ("body CT studies") committed by radiology fellows working off-hours based on day or night assignment. Materials and Methods Between July 2014 and June 2018, attending physicians at one tertiary care institution reviewed all body CT studies independently interpreted off-hours by radiologists in an academic fellowship within 10 hours of initial interpretation. Discrepancies affecting acute or follow-up clinical care were classified as errors. In this retrospective study, the error rate for studies interpreted during the day (between 7:00 am and 5:59 pm) was compared with that of studies interpreted at night (between 6:00 pm and 6:59 am). Error rate in the first half of day and night assignments was compared with error rate in the latter half. Statistical analyses used χ² tests and general estimating equations; significance was defined as P < .05. Results There were 10 090 body CT studies interpreted by 32 radiologists. Forty-four of 2195 daytime studies (2.0%) had errors compared with 240 of 7895 nighttime studies (3.0%; P = .02). Twenty-two of 32 (69%) radiologists had higher error rates for night cases (P = .03). There were more errors in the last half of a night assignment (125 of 3358, 3.7%; P = .002) compared with the first half (115 of 4537, 2.5%). Conclusion On the basis of a subspecialty review, clinically important off-hours body CT interpretation errors occurred more frequently overnight and more frequently in the latter half of assignments, with more radiologists having worse error rates at night compared with the day. © RSNA, 2020 See also the editorial by Bruno in this issue.