Effects of time of day on radiological interpretation

Implementation of an AI Algorithm in Clinical Practice to Reduce Missed Incidental Pulmonary Embolisms on Chest CT and Its Impact on Short-Term Survival

OBJECTIVES

A substantial number of incidental pulmonary embolisms (iPEs) in computed tomography scans are missed by radiologists in their daily routine. This study analyzes the radiological reports of iPE cases before and after implementation of an artificial intelligence (AI) algorithm for iPE detection. Furthermore, we investigate the anatomic distribution patterns within missed iPE cases and mortality within a 90-day follow-up in patients before and after AI use.

MATERIALS AND METHODS

This institutional review board-approved observational single-center study included 5298 chest computed tomography scans performed for reasons other than suspected pulmonary embolism (PE). We compared 2 cohorts: cohort 1, consisting of 1964 patients whose original radiology reports were generated before the implementation of an AI algorithm, and cohort 2, consisting of 3334 patients whose scans were analyzed after the implementation of an Food and Drug Administration-approved and CE-certified AI algorithm for iPE detection (Aidoc Medical, Tel Aviv, Israel). For both cohorts, any discrepancies between the original radiology reports and the AI results were reviewed by 2 thoracic imaging subspecialized radiologists. In the original radiology report and in case of discrepancies with the AI algorithm, the expert review served as reference standard. Sensitivity, specificity, prevalence, negative predictive value (NPV), and positive predictive value (PPV) were calculated. The rates of missed iPEs in both cohorts were compared statistically using STATA (Version 17.1). Kaplan-Meier curves and Cox proportional hazards models were used for survival analysis.

RESULTS

In cohort 1 (mean age 70.6 years, 48% female [n = 944], 52% male [n = 1020]), the prevalence of confirmed iPE was 2.2% (n = 42), and the AI detected 61 suspicious iPEs, resulting in a sensitivity of 95%, a specificity of 99%, a PPV of 69%, and an NPV of 99%. Radiologists missed 50% of iPE cases in cohort 1. In cohort 2 (mean age 69 years, 47% female [n = 1567], 53% male [n = 1767]), the prevalence of confirmed iPEs was 1.7% (56/3334), with AI detecting 59 suspicious cases (sensitivity 90%, specificity 99%, PPV 95%, NPV 99%). The rate of missed iPEs by radiologists dropped to 7.1% after AI implementation, showing a significant improvement (P < 0.001). Most overlooked iPEs (61%) were in the right lower lobe. The survival analysis showed no significantly decreased 90-day mortality rate, with a hazards ratio of 0.95 (95% confidence interval, 0.45-1.96; P = 0.88).

CONCLUSIONS

The implementation of an AI algorithm significantly reduced the rate of missed iPEs from 50% to 7.1%, thereby enhancing diagnostic accuracy. Despite this improvement, the 90-day mortality rate remained unchanged. These findings highlight the AI tool's potential to assist radiologists in accurately identifying iPEs, although its implementation does not significantly affect short-term survival. Notably, most missed iPEs were located in the right lower lobe, suggesting that radiologists should pay particular attention to this area during evaluations.

The "Hungry Judge" effect on prostate MRI reporting: Chronobiological trends from 35'004 radiologist interpretations

AIM

To investigate the associations between the hour of the day and Prostate Imaging-Reporting and Data System (PI-RADS) scores assigned by radiologists in prostate MRI reports.

MATERIALS AND METHODS

Retrospective single-center collection of prostate MRI reports over an 8-year period. Mean PI-RADS scores assigned between 0800 and 1800 h were examined with a regression model.

RESULTS

A total of 35'004 prostate MRI interpretations by 26 radiologists were included. A significant association between the hour of day and mean PI-RADS score was identified (β2 = 0.005, p < 0.001), with malignant scores more frequently assigned later in the day.

CONCLUSION

These findings suggest chronobiological factors may contribute to variability in radiological assessments. Though the magnitude of the effect is small, this may potentially add variability and impact diagnostic accuracy.

The Mixed Role of Sleep and Time of Day in Working Memory Performance of Older Adults with Mild Cognitive Impairment

The importance of night sleep for maintaining good physical and cognitive health is well documented as well as its negative changes during aging. Since Mild Cognitive Impairment (MCI) patients bear additional disturbances in their sleep, this study aimed at examining whether there are potential mixed effects of sleep and afternoon time of day (ToD) on the storage, processing, and updating components of working memory (WM) capacity in older adults with MCI. In particular, the study compared patients' performance in the three working memory components, in two-time conditions: "early in the morning and after night sleep", and "in the afternoon and after many hours since night sleep". The Working Memory Capacity & Updating Task from the R4Alz battery was administered twice to 50 older adults diagnosed with MCI. The repeated measures analysis showed statistically significant higher performance in the morning condition for the working memory updating component (p < 0.001). Based on the findings, it seems that the afternoon ToD condition negatively affects tasks with high cognitive demands such as the WM updating task in MCI patients. These findings could determine the optimal timing for cognitive rehabilitation programs for MCI patients and the necessary sleep duration when they are engaged in cognitively demanding daily activities.

Discrepancies between Radiology Specialists and Residents in Fracture Detection from Musculoskeletal Radiographs

(1) Background: The aim of this study was to compare the competence in appendicular trauma radiograph image interpretation between radiology specialists and residents. (2) Methods: In this multicenter retrospective cohort study, we collected radiology reports from radiology specialists (N = 506) and residents (N = 500) during 2018-2021. As a reference standard, we used the consensus of two subspecialty-level musculoskeletal (MSK) radiologists, who reviewed all original reports. (3) Results: A total of 1006 radiograph reports were reviewed by the two subspecialty-level MSK radiologists. Out of the 1006 radiographs, 41% were abnormal. In total, 67 radiographic findings were missed (6.7%) and 31 findings were overcalled (3.1%) in the original reports. Sensitivity, specificity, positive predictive value, and negative predictive value were 0.86, 0.92, 0.91 and 0.88 respectively. There were no statistically significant differences between radiology specialists' and residents' competence in interpretation (p = 0.44). However, radiology specialists reported more subtle cases than residents did (p = 0.04). There were no statistically significant differences between errors made in the morning, evening, or night shifts (p = 0.57). (4) Conclusions: This study found a lack of major discrepancies between radiology specialists and residents in radiograph interpretation, although there were differences between MSK regions and in subtle or obvious radiographic findings. In addition, missed findings found in this study often affected patient treatment. Finally, there are MSK regions where the sensitivity or specificity is below 90%, and these should raise concerns and highlight the need for double reading and should be taken into consideration in radiology education.

AI-Based Automated Lipomatous Tumor Segmentation in MR Images: Ensemble Solution to Heterogeneous Data

Deep learning (DL) has been proposed to automate image segmentation and provide accuracy, consistency, and efficiency. Accurate segmentation of lipomatous tumors (LTs) is critical for correct tumor radiomics analysis and localization. The major challenge of this task is data heterogeneity, including tumor morphological characteristics and multicenter scanning protocols. To mitigate the issue, we aimed to develop a DL-based Super Learner (SL) ensemble framework with different data correction and normalization methods. Pathologically proven LTs on pre-operative T1-weighted/proton-density MR images of 185 patients were manually segmented. The LTs were categorized by tumor locations as distal upper limb (DUL), distal lower limb (DLL), proximal upper limb (PUL), proximal lower limb (PLL), or Trunk (T) and grouped by 80%/9%/11% for training, validation and testing. Six configurations of correction/normalization were applied to data for fivefold-cross-validation trainings, resulting in 30 base learners (BLs). A SL was obtained from the BLs by optimizing SL weights. The performance was evaluated by dice-similarity-coefficient (DSC), sensitivity, specificity, and Hausdorff distance (HD95). For predictions of the BLs, the average DSC, sensitivity, and specificity from the testing data were 0.72 [Formula: see text] 0.16, 0.73 [Formula: see text] 0.168, and 0.99 [Formula: see text] 0.012, respectively, while for SL predictions were 0.80 [Formula: see text] 0.184, 0.78 [Formula: see text] 0.193, and 1.00 [Formula: see text] 0.010. The average HD95 of the BLs were 11.5 (DUL), 23.2 (DLL), 25.9 (PUL), 32.1 (PLL), and 47.9 (T) mm, whereas of SL were 1.7, 8.4, 15.9, 2.2, and 36.6 mm, respectively. The proposed method could improve the segmentation accuracy and mitigate the performance instability and data heterogeneity aiding the differential diagnosis of LTs in real clinical situations.

Identification of factors associated with diagnostic performance variation in reporting of mammograms: A review

OBJECTIVES

This narrative review aims to identify what factors are linked to diagnostic performance variation for those who interpret mammograms. Identification of influential factors has potential to contribute to the optimisation of breast cancer diagnosis. PubMed, ScienceDirect and Google Scholar databases were searched using the following terms: 'Radiology', 'Radiologist', 'Radiographer', 'Radiography', 'Mammography', 'Interpret', 'read', 'observe' 'report', 'screen', 'image', 'performance' and 'characteristics.' Exclusion criteria included articles published prior to 2000 as digital mammography was introduced at this time. Non-English articles language were also excluded. 38 of 2542 studies identified were analysed.

KEY FINDINGS

Influencing factors included, new technology, volume of reads, experience and training, availability of prior images, social networking, fatigue and time-of-day of interpretation. Advancements in breast imaging such as digital breast tomosynthesis and volume of mammograms are primary factors that affect performance as well as tiredness, time-of-day when images are interpreted, stages of training and years of experience. Recent studies emphasised the importance of social networking and knowledge sharing if breast cancer diagnosis is to be optimised.

CONCLUSION

It was demonstrated that data on radiologist performance variability is widely available but there is a paucity of data on radiographers who interpret mammographic images.

IMPLICATIONS FOR PRACTICE

This scarcity of research needs to be addressed in order to optimise radiography-led reporting and set baseline values for diagnostic efficacy.

Assessment of the Role of Artificial Intelligence in the Association Between Time of Day and Colonoscopy Quality

IMPORTANCE

Time of day was associated with a decline in adenoma detection during colonoscopy. Artificial intelligence (AI) systems are effective in improving the adenoma detection rate (ADR), but the performance of AI during different times of the day remains unknown.

OBJECTIVE

To validate whether the assistance of an AI system could overcome the time-related decline in ADR during colonoscopy.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study is a secondary analysis of 2 prospective randomized controlled trials (RCT) from Renmin Hospital of Wuhan University. Consecutive patients undergoing colonoscopy were randomly assigned to either the AI-assisted group or unassisted group from June 18, 2019, to September 6, 2019, and July 1, 2020, to October 15, 2020. The ADR of early and late colonoscopy sessions per half day were compared before and after the intervention of the AI system. Data were analyzed from March to June 2022.

EXPOSURE

Conventional colonoscopy or AI-assisted colonoscopy.

MAIN OUTCOMES AND MEASURES

Adenoma detection rate.

RESULTS

A total of 1780 patients (mean [SD] age, 48.61 [13.35] years, 837 [47.02%] women) were enrolled. A total of 1041 procedures (58.48%) were performed in early sessions, with 357 randomized into the unassisted group (34.29%) and 684 into the AI group (65.71%). A total of 739 procedures (41.52%) were performed in late sessions, with 263 randomized into the unassisted group (35.59%) and 476 into the AI group (64.41%). In the unassisted group, the ADR in early sessions was significantly higher compared with that of late sessions (13.73% vs 5.70%; P = .005; OR, 2.42; 95% CI, 1.31-4.47). After the intervention of the AI system, as expected, no statistically significant difference was found (22.95% vs 22.06%, P = .78; OR, 0.96; 95% CI; 0.71-1.29). Furthermore, the AI systems showed better assistance ability on ADR in late sessions compared with early sessions (odds ratio, 3.81; 95% CI, 2.10-6.91 vs 1.60; 95% CI, 1.10-2.34).

CONCLUSIONS AND RELEVANCE

In this cohort study, AI systems showed higher assistance ability in late sessions per half day, which suggests the potential to maintain high quality and homogeneity of colonoscopies and further improve endoscopist performance in large screening programs and centers with high workloads.

Persistent inter-observer variability of breast density assessment using BI-RADS® 5th edition guidelines

OBJECTIVES

Due to most states' legislation, mammographic density categorization has potentially far-reaching implications, but remains subjective based on BIRADS® guidelines. We aimed to determine 1) effect of BI-RADS® 5th edition (5th-ed) vs 4th-edition (4th-ed) guidelines on reader agreement regarding density assessment; 2) 5th-ed vs 4th-ed density distribution, and visual vs quantitative assessment agreement; 3) agreement between experienced vs less experienced readers.

METHODS

In a retrospective review, six breast imaging radiologists (BIR) (23-30 years' experience) visually assessed density of 200 screening mammograms performed September 2012-January 2013 using 5th-ed guidelines. Results were compared to 2016 data of the same readers evaluating the same mammograms using 4th-ed guidelines after a training module. 5th-ed density categorization by seven junior BIR (1-5 years' experience) was compared to eight experienced BIR. Nelson et al.'s kappas (κ_m, κ_w), Fleiss' κ_F, and Cohen's κ were calculated. Quantitative density using Volpara was compared with reader assessments.

RESULTS

Inter-reader weighted agreement using 5th-ed is moderately strong, 0.73 (κ_w, s.e. = 0.01), similar to 4th-ed, 0.71 (κ_w, s.e. = 0.03). Intra-reader Cohen's κ is 0.23-0.34, similar to 4th-ed. Binary not-dense vs dense categorization, using 5th-ed results in higher dense categorization vs 4th-ed (p < 0.001). 5th-ed density distribution results in higher numbers in categories B/C vs 4th-ed (p < 0.001). Distribution for 5th-ed does not differ based on reader experience (p = 0.09). Reader vs quantitative weighted agreement is similar (5th-ed, Cohen's κ = 0.76-0.85; 4th-ed, Cohen's κ = 0.68-0.83).

CONCLUSION

There is persistent subjectivity of visually assessed mammographic density using 5th-ed guidelines; experience does not correlate with better inter-reader agreement.

Digital Breast Tomosynthesis and Digital Mammography Recall and False-Positive Rates by Time of Day and Reader Experience

Background Digital breast tomosynthesis (DBT) image interpretation might be more cognitively demanding than interpretation of digital mammography (DM) images. The time of day of interpretation might affect recall and false-positive (FP) rates, especially for DBT. Purpose To determine whether recall and FP rates vary by time of day of interpretation for screening mammography for breast cancer performed with DM and DBT. Materials and Methods This is a retrospective study examining 97 671 screening mammograms interpreted by 18 radiologists between January 2018 and December 2019 at one of 12 community radiology sites. The association between the time of day of interpretation, the type of image interpreted (DM vs DBT), and radiologist experience (≤5 posttraining years vs >5 posttraining years) and the likelihood of a patient being recalled from screening mammography and the likelihood of whether the interpretation was FP or true positive were analyzed. Analyses were conducted using generalized linear mixed modeling with a binary distribution and sandwich estimation where observations were nested by radiologist. Results Screening mammograms interpreted by 18 radiologists were reviewed (40 220 DBTs, 57 451 DMs). Nine radiologists had 5 or fewer posttraining years of experience, and nine had more than 5 posttraining years of experience. The overall recall rates for DM (10.2%) and DBT (9.0%) were different (P = .006); FP rate also differed (9.8% DM, 8.6% DBT; P = .004). For radiologists with 5 or fewer posttraining years of experience, odds of recall increased 11.5% (odds ratio [OR] = 1.12, P = .01) with every hour when using DBT, but this was not found for DM (OR = 1.09, P = .06); DBT and DM were different (OR = 1.12 vs 1.09, P = .02). For radiologists with more than 5 posttraining years of experience, no evidence of increase in recall was observed for DBT (OR = 1.02, P = .27) or DM (OR = 1.0, P = .80), and there was no evidence that these were different (OR = 1.02 vs 1.0, P = .13). Conclusion Patients were more likely to be recalled when their screening digital breast tomosynthesis images were interpreted later in the day by less-experienced radiologists. © RSNA, 2022 Online supplemental material is available for this article.

Examining the effects of time of day and sleep on generalization

Extracting shared structure across our experiences allows us to generalize our knowledge to novel contexts. How do different brain states influence this ability to generalize? Using a novel category learning paradigm, we assess the effect of both sleep and time of day on generalization that depends on the flexible integration of recent information. Counter to our expectations, we found no evidence that this form of generalization is better after a night of sleep relative to a day awake. Instead, we observed an effect of time of day, with better generalization in the morning than the evening. This effect also manifested as increased false memory for generalized information. In a nap experiment, we found that generalization did not benefit from having slept recently, suggesting a role for time of day apart from sleep. In follow-up experiments, we were unable to replicate the time of day effect for reasons that may relate to changes in category structure and task engagement. Despite this lack of consistency, we found a morning benefit for generalization when analyzing all the data from experiments with matched protocols (n = 136). We suggest that a state of lowered inhibition in the morning may facilitate spreading activation between otherwise separate memories, promoting this form of generalization.

Diurnal variation of major error rates in the interpretation of abdominal/pelvic CT studies

RATIONALE AND OBJECTIVE

Variation of visual selective attention through the day has been demonstrated in several arenas of human performance, including radiology. It is uncertain whether this variation translates to an identifiable diurnal pattern of error rates for radiology interpretation. The purpose of this study was to attempt to identify particular days of the week and times of the day when radiologists might be most prone to error.

MATERIALS AND METHODS

Abdomen/pelvis CT studies containing at least one major error were collected from a 10-year period from the quality assurance (QA) database at our institution. A major error was defined as a missed finding that had altered management in a way potentially detrimental to the patient. The identified studies were categorized by the day of the week and hour of the day that the study was interpreted. Study volume data over this same period was also obtained by day of the week and time of day, so to normalize the data based on case volume. Standard errors of the volume-adjusted error rates were obtained based on the binomial distribution. The null hypothesis of constant error rates over time was tested using a weighted logistic regression model with linear time as predictor.

RESULTS

A total of 252 major errors were identified. More errors were made on Monday than on any other day of the week (n = 58). Major error rates increased through the mid to late morning (9 am to 12 pm), and then decreased progressively through the afternoon until 4 pm, when a rise in the error rate was seen. This pattern persisted when error rates were normalized by study volume within each hour. Overall tests of time-constancy of error rates by day and hour were statistically significant (both p-values < 0.001).

CONCLUSION

Our study shows that error rates in abdominal CT do seem to vary with time of day and day of the week. During the workweek, error rates were highest in the late morning and at the close of the workday, and greater on Mondays than other days.

Impact of time of day on radiology image interpretations

AIM

To examine the impact of the time of day on radiologists' mammography reading performance.

MATERIALS AND METHODS

Retrospective mammographic reading assessment data were collected from the BreastScreen Reader Assessment Strategy database and included timestamps of the readings and reader-specific demographic data of 197 radiologists. The radiologists performed the readings in a workshop setting with test case sets enriched with malignancies (one-third of cases were malignant). The collected data were evaluated with an analysis of covariance to determine whether time of day influenced radiologists' specificity, lesion sensitivity or the jackknife alternative free-response receiver operating characteristic (JAFROC).

RESULTS

After adjusting for radiologist experience and fellowship, specificity varied significantly by time of day (p=0.027), but there was no evidence of any significant impact on lesion sensitivity (p=0.441) or JAFROC (p=0.120). The collected data demonstrated that specificity during the late morning (10.00-12.00) was 71.7%; this was significantly lower than in the early morning (08.00-10.00) and mid-afternoon (14.00-16.00), which were 82.74% (p=0.003) and 81.39% (p=0.031), respectively. Specificity during the late afternoon (16.00-18.00) was 73.95%; this was significantly lower than in the early morning (08.00-10.00) and mid-afternoon (14.00-16.00), which were 82.74% (p=0.003) and 81.39% (p=0.031), respectively.

CONCLUSION

The results indicated that the time of day may influence radiologists' performance, specifically their ability to identify normal images correctly.