Radiologists' reading times using PACS and using films: one practice's experience

Added value of an artificial intelligence solution for fracture detection in the radiologist's daily trauma emergencies workflow

PURPOSE

The main objective of this study was to compare radiologists' performance without and with artificial intelligence (AI) assistance for the detection of bone fractures from trauma emergencies.

MATERIALS AND METHODS

Five hundred consecutive patients (232 women, 268 men) with a mean age of 37 ± 28 (SD) years (age range: 0.25-99 years) were retrospectively included. Three radiologists independently interpreted radiographs without then with AI assistance after a 1-month minimum washout period. The ground truth was determined by consensus reading between musculoskeletal radiologists and AI results. Patient-wise sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for fracture detection and reading time were compared between unassisted and AI-assisted readings of radiologists. Their performances were also assessed by receiver operating characteristic (ROC) curves.

RESULTS

AI improved the patient-wise sensitivity of radiologists for fracture detection by 20% (95% confidence interval [CI]: 14-26), P< 0.001) and their specificity by 0.6% (95% CI: -0.9-1.5; P = 0.47). It increased the PPV by 2.9% (95% CI: 0.4-5.4; P = 0.08) and the NPV by 10% (95% CI: 6.8-13.3; P < 0.001). Thanks to AI, the area under the ROC curve for fracture detection of readers increased respectively by 10.6%, 10.2% and 9.9%. Their mean reading time per patient decreased by respectively 10, 16 and 12 s (P < 0.001).

CONCLUSIONS

AI-assisted radiologists work better and faster compared to unassisted radiologists. AI is of great aid to radiologists in daily trauma emergencies, and could reduce the cost of missed fractures.

Effect of Implementation of Picture Archiving and Communication System on Radiologist Reporting Time and Utilization of Radiology Services: A Case Study in Iran

We aimed to determine the effect of PACS implementation on the radiologist reporting time and utilization of radiology services in a hospital in Iran. This study was conducted in three 6-month periods (before PACS implementation, immediately after the implementation, and 1 year after implementation). Data related to reporting time of CT examinations were collected from 5074 patients with 6613 examinations and compared using the Friedman test. The utilization rate of radiology services was conducted about CT scans and radiographies. Therefore, 17,862 patients with 36,321 radiographies and 7155 patients with 10,571 CT scans were evaluated, and the ratio of the number of examinations to the number of patients and patient days was compared in three periods. The mean of reporting time on CT scan examinations in the period of immediately after PACS was changed compared to the period of before PACS from 13.05 to 24.18 days and compared to 1 year after PACS implementation, to 4.14 days (P value < 0.05). The utilization rate in CT scans, 1 year after PACS increased at least 10% to 25% compared to the immediately after PACS and before implementing PACS. The utilization rate in radiographies, 1 year after PACS, increased at least 16% to 78% compared to the immediately after PACS and before PACS implementation. In conclusion, the mean of the radiologist reporting time for CT scans is significantly decreased by PACS in the long-term. Additionally, the utilization rate of radiology services is increased in the short- and long-term after PACS implementation in most examinations of CT scan and radiography examinations.

Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment

Chest radiography is the most common radiographic examination performed in daily clinical practice for the detection of various heart and lung abnormalities. The large amount of data to be read and reported, with more than 100 studies per day for a single radiologist, poses a challenge in consistently maintaining high interpretation accuracy. The introduction of large-scale public datasets has led to a series of novel systems for automated abnormality classification. However, the labels of these datasets were obtained using natural language processed medical reports, yielding a large degree of label noise that can impact the performance. In this study, we propose novel training strategies that handle label noise from such suboptimal data. Prior label probabilities were measured on a subset of training data re-read by 4 board-certified radiologists and were used during training to increase the robustness of the training model to the label noise. Furthermore, we exploit the high comorbidity of abnormalities observed in chest radiography and incorporate this information to further reduce the impact of label noise. Additionally, anatomical knowledge is incorporated by training the system to predict lung and heart segmentation, as well as spatial knowledge labels. To deal with multiple datasets and images derived from various scanners that apply different post-processing techniques, we introduce a novel image normalization strategy. Experiments were performed on an extensive collection of 297,541 chest radiographs from 86,876 patients, leading to a state-of-the-art performance level for 17 abnormalities from 2 datasets. With an average AUC score of 0.880 across all abnormalities, our proposed training strategies can be used to significantly improve performance scores.

Triple attention learning for classification of 14 thoracic diseases using chest radiography

Chest X-ray is the most common radiology examinations for the diagnosis of thoracic diseases. However, due to the complexity of pathological abnormalities and lack of detailed annotation of those abnormalities, computer-aided diagnosis (CAD) of thoracic diseases remains challenging. In this paper, we propose the triple-attention learning (A ³ Net) model for this CAD task. This model uses the pre-trained DenseNet-121 as the backbone network for feature extraction, and integrates three attention modules in a unified framework for channel-wise, element-wise, and scale-wise attention learning. Specifically, the channel-wise attention prompts the deep model to emphasize the discriminative channels of feature maps; the element-wise attention enables the deep model to focus on the regions of pathological abnormalities; the scale-wise attention facilitates the deep model to recalibrate the feature maps at different scales. The proposed model has been evaluated on 112,120images in the ChestX-ray14 dataset with the official patient-level data split. Compared to state-of-the-art deep learning models, our model achieves the highest per-class AUC in classifying 13 out of 14 thoracic diseases and the highest average per-class AUC of 0.826 over 14 thoracic diseases.

Refining dataset curation methods for deep learning-based automated tuberculosis screening

Background

The study objective was to determine whether unlabeled datasets can be used to further train and improve the accuracy of a deep learning system (DLS) for the detection of tuberculosis (TB) on chest radiographs (CXRs) using a two-stage semi-supervised approach.

Methods

A total of 111,622 CXRs from the National Institute of Health ChestX-ray14 database were collected. A cardiothoracic radiologist reviewed a subset of 11,000 CXRs and dichotomously labeled each for the presence or absence of potential TB findings; these interpretations were used to train a deep convolutional neural network (DCNN) to identify CXRs with possible TB (Phase I). The best performing algorithm was then used to label the remaining database consisting of 100,622 radiographs; subsequently, these newly-labeled images were used to train a second DCNN (phase II). The best-performing algorithm from phase II (TBNet) was then tested against CXRs obtained from 3 separate sites (2 from the USA, 1 from China) with clinically confirmed cases of TB. Receiver operating characteristic (ROC) curves were generated with area under the curve (AUC) calculated.

Results

The phase I algorithm trained using 11,000 expert-labelled radiographs achieved an AUC of 0.88. The phase II algorithm trained on images labeled by the phase I algorithm achieved an AUC of 0.91 testing against a TB dataset obtained from Shenzhen, China and Montgomery County, USA. The algorithm generalized well to radiographs obtained from a tertiary care hospital, achieving an AUC of 0.87; TBNet’s sensitivity, specificity, positive predictive value, and negative predictive value were 85%, 76%, 0.64, and 0.9, respectively. When TBNet was used to arbitrate discrepancies between 2 radiologists, the overall sensitivity reached 94% and negative predictive value reached 0.96, demonstrating a synergistic effect between the algorithm’s output and radiologists’ interpretations.

Conclusions

Using semi-supervised learning, we trained a deep learning algorithm that detected TB at a high accuracy and demonstrated value as a CAD tool by identifying relevant CXR findings, especially in cases that were misinterpreted by radiologists. When dataset labels are noisy or absent, the described methods can significantly reduce the required amount of curated data to build clinically-relevant deep learning models, which will play an important role in the era of precision medicine.

Comprehensive review of the limitations of relative value unit measurements of productivity in academic departments

Measuring radiologist, especially sonologist, productivity has never been of greater interest than now, as radiology has moved from its historical status as a cost center to the largest revenue generator in most institutions. With more local institutional and federal oversight and regulation into the reimbursement and valuation of imaging procedures, including emphasis on outsourcing, overvalued procedures, and bundling, the ability to measure productivity and to draw meaningful conclusions from the data becomes increasingly imperative if we as radiologists and sonologists expect to contribute meaningfully to the process. This article presents a history of the "relative value unit" and discusses the valuation of radiologist/sonologist productivity in this era of ubiquitous high-technology implementation.

PNist: interactive volumetric measurements of plexiform neurofibromas in MRI scans

PURPOSE

Volumetric measurements of plexiform neurofibromas (PNs) are time consuming and error prone, as they require the delineation of the PN boundaries, which is mostly impractical in the daily clinical setup. Accurate volumetric measurements are seldom performed for these tumors mainly due to their great dispersion, size and multiple locations. This paper presents a semiautomatic method for segmentation of PN from STIR MRI scans.

METHODS

Plexiform neurofibroma interactive segmentation tool (PNist) is a new tool to segment PNs in STIR MRI scans. The method is based on histogram tumor models computed from a training set.

RESULTS

Experimental results from 28 datasets show an average absolute volume difference of 6.8 % with an average user time of approximately 7 min versus more than 13 min with manual delineation. In complex cases, the PNist user time is less than half in compared to state-of-the-art tools.

CONCLUSIONS

PNist is a new method for the semiautomatic segmentation of PN lesions. Its simplicity and reliability make it unique among other state-of-the-art methods. It has the potential to become a clinical tool that allows the reliable evaluation of PN burden and progression.

Emerging trends in the volume and format of outside examinations submitted for secondary interpretation

OBJECTIVE

The purpose of this article is to describe the trends of secondary interpretations, including the total volume and format of cases.

MATERIALS AND METHODS

This retrospective study involved all outside neuroradiology examinations submitted for secondary interpretation from November 2006 through December 2010. This practice utilizes consistent criteria and includes all images that cover the brain, neck, and spine. For each month, the total number of outside examinations and their format (i.e., hard-copy film, DICOM CD-ROM, or non-DICOM CD-ROM) were recorded.

RESULTS

There was no significant change in the volume of cases (1043 ± 131 cases/month; p = 0.46, two-sided Student t test). There was a significant decrease in the volume of hard-copy films submitted, with the mean number of examinations submitted per month on hard-copy film declining from 297 in 2007 to 57 in 2010 (p < 0.0001, Student t test). This decrease was mirrored by an increase in the mean number of cases submitted on CD-ROM (753 cases/month in 2007 and 1036 cases/month in 2010; p < 0.0001). Although most were submitted in DICOM format, there was almost a doubling of the volume of cases submitted on non-DICOM CD-ROM (mean number of non-DICOM CD-ROMs, nine cases/month in 2007 and 17 cases/month in 2010; p < 0.001).

CONCLUSION

There has been a significant decrease in the number of hard-copy films submitted for secondary interpretation. There has been almost a doubling of the volume of cases submitted in non-DICOM formats, which is unfortunate, given the many advantages of the internationally derived DICOM standard, including ease of archiving, standardized display, efficient review, improved interpretation, and quality of patient care.

Experience with importation of electronic images into the medical record from physical media

The purpose of this article is to describe a system we developed for importing images on compact discs (CDs) from external imaging departments into our clinical image viewing system, and to report on key metrics regarding veracity of information seen on the CDs. We recommend careful attention to the process of CD importation because of the error rate we have seen. We developed a system and process for importing images on CD into our EMR. The importation system scans the CD for digital imaging and communications in medicine (DICOM) images, and collects all patient information seen. That information is presented to the patient for verification. Once validated, the image data is copied into our clinical viewing system. The importation system includes facilities for collecting instances of incorrect data. About 90% of images are now exchanged between our healthcare enterprise and other entities via CD. Data for the wrong patient (e.g., the wrong CD) is seen in about 0.1% of cases, and a similar number of CDs have data for more than one patient on the CD(s) the patient bring to our facility. Most data are now exchanged via DICOM files. DICOM images burned onto CD media are now commonly used for image exchange. However, applications to import DICOM images are not enough. One must implement a process to assure high confidence that the data imported belongs to the patient you are importing.

Optimal presentation modes for detecting brain tumor progression

BACKGROUND AND PURPOSE

A common task in radiology interpretation is visual comparison of images. The purpose of this study was to compare traditional side-by-side and in-place (flicker) image presentation modes with advanced methods for detecting primary brain tumors on MR imaging.

MATERIALS AND METHODS

We identified 66 patients with gliomas and 3 consecutive brain MR imaging examinations (a "triplet"). A display application that presented images in side-by-side mode with or without flicker display as well as display of image subtraction or automated change detection information (also with and without flicker display) was used by 3 board-certified neuroradiologists. They identified regions of brain tumor progression by using this display application. Each case was reviewed using all modes (side-by-side presentation with and without flicker, subtraction with and without flicker, and change detection with and without flicker), with results compared via a panel rating.

RESULTS

Automated change detection with or without flicker (P < .0027) as well as subtraction with or without flicker (P < .0027) were more sensitive to tumor progression than side-by-side presentation in cases where all 3 raters agreed. Change detection afforded the highest interrater agreement, followed by subtraction. Clinically determined time to progression was longer for cases rated as nonprogressing by using subtraction images and change-detection images both with and without flicker display mode compared with side-by-side presentation.

CONCLUSIONS

Automated change detection and image subtraction, with and without flicker display mode, are superior to side-by-side image comparison.

Virtual organization of hospital medical imaging: a user satisfaction survey

A virtual medical imaging department is an innovative and demanding organizational model, to the extent that the underlying goal is to achieve a continuous and advanced organizational integration of human and physical resources, clinical data, and clienteles. To better understand the kind of benefits offered, we conducted a survey of three groups of users--radiologists, radiological technologists, and medical specialists--working in a five-site virtual organization. We received 127 valid questionnaires, for an overall response rate of 66%. The assessments vary according to the use made of the system. The scores for system quality and the quality of the data produced were markedly higher for intra-hospital use (respectively 7.9 and 8.7 out of 10) than for inter-hospital use (5.4 and 7.0). Despite the negative assessments they made of inter-hospital use, users maintained a positive attitude toward some type of virtual organization of medical imaging. Indeed, the score for Overall satisfaction with the system was very high, 8.9 out of 10. Moreover, the scores for Intended future use of the system were very high for both intra-hospital use (8.9) and inter-hospital use (8.7). We also found significant differences in perceptions among user groups.

Digital imaging data on CD-R: a time trap for orthopaedic surgeons in outpatient clinics

PURPOSE

X-rays are presented on CD-Rs in a digital format with increasing frequency. This technique is potentially more time-consuming for the clinician compared to conventional pictures.

METHODS

Ten sets of six X-rays for each case were prepared in both the conventional way and in digital format on CD-R. The order was randomised. Time in seconds was measured for six experienced orthopaedic residents to present the most recent a.p. view out of a given set. As a precondition, the computer was turned on and the same viewing software was used for all digital sets. The results were compared using a non-linked Student's t-test (significance level P = 0.05).

RESULTS

The presentation of conventional X-rays required 21 s (+/-7.5 s) and of digital X-rays 90 s (+/-27 s), respectively (P < 0.001).

CONCLUSION

In spite of ideal conditions, digital X-rays on CD-R need significantly more time in the orthopaedic clinic. In major centres, patients present with different software and software in other languages, which increase the required time even further. This latter problem will be assessed in another study. This increase of preparation time required by a highly qualified staff member has implications on the economics and logistics and has a negative impact on daily clinical practice. One of the solutions might be that the CD-Rs be given to the registration desk and the data being already prepared when the patient sees the doctor. This can also include it being implemented in the digital system used in the inpatient part of the hospital and, therefore, making any other consultations or second opinions easily accessible.

Picture archiving and communication systems lead to sustained improvements in reporting times and productivity: results of a 5-year audit

AIM

To evaluate the impact of picture archiving and communications systems (PACS) on reporting times and productivity in a large teaching hospital.

MATERIALS AND METHODS

Reporting time, defined as the time taken from patient registration to report availability, and productivity, defined as the number of reports issued per whole time equivalent (WTE) radiologist per month, were studied for 2 years pre- and 3 years post-PACS installation. Mean reporting time was calculated for plain radiographs and specialist radiology techniques [computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, and nuclear medicine]. Productivity, total department workload, and unreported film rates were also assessed. Pre- and post-PACS findings were compared.

RESULTS

Between 2002-2006 the number of radiological patient episodes increased by 30% from 11,531/month to 15,057/month. This was accompanied by a smaller increase in WTE reporting radiologists, from 32 to 37 (15%). Mean reporting times have improved substantially post-PACS, plain radiograph reporting time decreased by 26% (from 6.8 to 5 days; p=0.002) and specialty modalities by 24% (4.1 to 3.1 days; p<0.001). Radiologist productivity has increased by 18% (337 films to 407 films/WTE radiologist/month). Unreported films have decreased from 5 to 4% for plain radiographs and are steady for specialty modalities (< 1%). In most areas improvements have been sustained over a 3-year period.

CONCLUSION

Since the introduction of PACS, reporting times have decreased by 25% and the productivity improved by 18%. Sustained improvements are felt to reflect the efficiencies and cultural change that accompanied the introduction of PACS and digital dictation.