A comparison of conventional film, CR hard copy and PACS soft copy images of the chest: analyses of ROC curves and inter-observer agreement

Effect of picture archiving and communication system on diagnosis accuracy in CT and radiography examinations in emergency departments

INTRODUCTION

Picture archiving and communication system (PACS) affects the radiologists' and physicians' performance. We aimed to evaluate the effect of implementing PACS on the emergency department (ED) physicians' accuracy compared to a radiologist's diagnosis in Iran.

METHODS

We retrospectively collected data for three six-month periods before and after the implementation of PACS on CT scan and radiography examinations. We compared ED physicians' diagnoses of CT scan and radiography images with a radiologist's interpretations for the same images. We compared 374 CT scans and 346 radiography examinations before implementing PACS (July 2015 to December 2015); 507 CT scans and 480 radiography examinations immediately after PACS (July 2016 to December 2016); and 870 CT scans and 1137 radiography examinations one year after PACS (July 2017 to December 2017).

RESULTS

We found that diagnosis accuracy of ED physicians on CT scans increased from 75.9 % before implementing PACS to 84.4 % immediately after PACS and 94.9 % one year after PACS (p-value < 0.0001). Diagnosis accuracy for radiography images increased from 63.0 % before implementing PACS to 80.2 % immediately after PACS and 93.1 % one year after PACS (p-value < 0.0001).

CONCLUSION

Implementation of PACS technology increases ED physicians' diagnosis accuracy.

Evaluation of radiographers' knowledge and attitudes of image quality optimisation in paediatric digital radiography in Saudi Arabia and Australia: a survey-based study

INTRODUCTION

Digital radiography (DR) systems enable radiographers to reduce the radiation dose to patients while maintaining optimised image quality. However, concerns still exist about paediatric patients who may be exposed to an increased level of radiation dose which is not needed for clinical practice. The purpose of this study was to evaluate the knowledge, awareness and attitudes, in terms of image quality optimisation of radiographers undertaking paediatric DR in Australia and Saudi Arabia.

METHODS

A survey-based study was devised and distributed to radiographers from Australia and Saudi Arabia. Questions focused on Australian and Saudi Arabian radiographers' knowledge and attitude of paediatric DR examinations.

RESULTS

There were 376 participants who responded to the survey from both countries. A major finding showed that most participants lack knowledge in the area of paediatric DR examinations. Most participants from Australia had received no formal training in paediatric digital radiography (79%), whereas nearly half of the participants from Saudi Arabia received no training (45%). Approximately three out of four radiographers from both countries believed that when using DR they did not need to change the way they collimate the beam as DR images can be cropped using post-processing methods.

CONCLUSION

The finding of this study demonstrates that radiographers from both countries should improve their understanding and clinical use of DR in paediatric imaging. More education and training for both students and clinicians is needed to enhance radiographer performance in digital radiography and improve their clinical practices.

Repeatability and intra- and inter-observer agreement of cervical vertebral sagittal diameter ratios in horses with neurological disease

BACKGROUND

Sagittal ratio values (SRVs) of cervical vertebrae are used for ante-mortem diagnosis of cervical vertebral stenotic myelopathy, but intraobserver and interobserver variability in measurement may influence radiographic interpretation of vertebral stenosis in horses with neurological disease.

OBJECTIVES

To determine intraobserver repeatability in SRVs, intra- and interobserver agreement in SRVs and whether or not agreement was influenced by animal age.

ANIMALS

Forty-two horses (>1 year old) with neurological disease from which laterolateral computed radiographic images of C2-C7 were obtained.

METHODS

Four observers made measurements from C2 to C7 for each horse and interobserver agreement for intra- and intervertebral SRVs was determined using Bland-Altman analysis (acceptable agreement: limits of agreement [LOA] ≤ 0.05) on all horses and those ≤3 (n = 25) and >3 (n = 17) years old. Each observer also made repeated measurements for 10 horses and intraobserver repeatability and agreement were determined.

RESULTS

Adequate intraobserver repeatability was achieved for 6 sites. Within observers, paired measurements had a median difference ≤5.7%, but a large range in differences often occurred, most frequently at intervertebral sites. For C5, C6, C7, and C3-4, LOA ≤ 0.05 were achieved by at least 1 observer. With the exception of C5 for 1 pair, LOA were >0.05 for interobserver agreement, regardless of animal age. LOA were largest at intervertebral sites.

CONCLUSIONS AND CLINICAL IMPORTANCE

Within and between observers, measurement error may limit the diagnostic accuracy of SRVs and result in discrepancies of diagnosis and treatment and warrants consideration when used clinically in horses with neurological disease.

Anatomical landmark position--can we trust what we see? Results from an online reliability and validity study of osteopaths

BACKGROUND

Practitioners traditionally use observation to classify the position of patients' anatomical landmarks. This information may contribute to diagnosis and patient management.

OBJECTIVES

To calculate a) Inter-rater reliability of categorising the sagittal plane position of four anatomical landmarks (lateral femoral epicondyle, greater trochanter, mastoid process and acromion) on side-view photographs (with landmarks highlighted and not-highlighted) of anonymised subjects; b) Intra-rater reliability; c) Individual landmark inter-rater reliability; d) Validity against a 'gold standard' photograph.

DESIGN

Online inter- and intra-rater reliability study.

SUBJECTS

Photographed subjects: convenience sample of asymptomatic students; raters: randomly selected UK registered osteopaths.

METHODS

40 photographs of 30 subjects were used, a priori clinically acceptable reliability was ≥0.4. Inter-rater arm: 20 photographs without landmark highlights plus 10 with highlights; Intra-rater arm: 10 duplicate photographs (non-highlighted landmarks). Validity arm: highlighted landmark scores versus 'gold standard' photographs with vertical line. Research ethics approval obtained.

RATERS

Osteopaths (n = 48) categorised landmark position relative to imagined vertical-line; Gwet's Agreement Coefficient 1 (AC1) calculated and chance-corrected coefficient benchmarked against Landis and Koch's scale; Validity calculation used Kendall's tau-B.

RESULTS

Inter-rater reliability was 'fair' (AC1 = 0.342; 95% confidence interval (CI) = 0.279-0.404) for non-highlighted landmarks and 'moderate' (AC1 = 0.700; 95% CI = 0.596-0.805) for highlighted landmarks. Intra-rater reliability was 'fair' (AC1 = 0.522); range was 'poor' (AC1 = 0.160) to 'substantial' (AC1 = 0.896). No differences were found between individual landmarks. Validity was 'low' (TB = 0.327; p = 0.104).

CONCLUSION

Both inter- and intra-rater reliability was 'fair' but below clinically acceptable levels, validity was 'low'. Together these results challenge the clinical practice of using observation to categorise anterio-posterior landmark position.

The role of key image notes in CT imaging study interpretation

The objective of the study was to investigate the clinical effects of CT key image notes (KIN) in the interpretation of a CT image study. All experiments were approved by the ethics committee of the local district. Six experienced radiologists were equally divided into routine reporting (RR) group and KIN reporting (KIN) group. CT scans of each 100 consecutive cases before and after using KIN technique were randomly selected, and the reports were made by group RR and KIN, respectively. All the reports were again reviewed 3 months later by both groups. All the results with using or not using KIN were interpreted and reinterpreted after 3 months by six clinicians, who were experienced in picture archiving and communication system (PACS) applications and were equally divided into the clinical routine report group and the clinical KIN report group, respectively. The results were statistically analyzed; the time used in making a report, the re-reading time 3 months later, and the consistency of imaging interpretation were determined and compared between groups. After using KIN technique, the time used in making a report was significantly increased (8.77 ± 5.27 vs. 10.53 ± 5.71 min, P < 0.05), the re-reading time was decreased (5.23 ± 2.54 vs. 4.99 ± 1.70 min, P < 0.05), the clinical interpretation and reinterpretation time after 3 months were decreased, and the consistency of the interpretation, reinterpretation between different doctors in different time was markedly improved (P < 0.01). CT report with KIN technique in PACS can significantly improve the consistency of the interpretation and efficiency in routine clinical work.

Effect of LCD monitor type and observer experience on diagnostic performance in soft-copy interpretations of the maxillary sinus on panoramic radiographs

Purpose

The aim of this study was to evaluate the effect of liquid crystal display (LCD) monitor type and observer experience on the diagnostic performance in soft-copy interpretations of maxillary sinus inflammatory lesions on panoramic radiographs.

Materials and Methods

Ninety maxillary sinuses on panoramic images were grouped into negative and positive groups according to the presence of inflammatory lesions, using CT for confirmation. Monochrome and color LCDs were used. Six observers participated and ROC analysis was performed to evaluate the diagnostic performance. The reading time, fatigue score, and inter-/intra-observer agreements were assessed.

Results

The interpretation of maxillary sinus inflammatory lesions was affected by the LCD monitor type used and by the experience of the observer. The reading time was not significantly different, however the fatigue score was significantly different between two LCD monitors. Inter-observer agreement was relatively good in experienced observers, while the intra-observer agreement for all observers was good with monochrome LCD but not with color LCD.

Conclusion

The less experienced observers showed lowered diagnostic ability with a general color LCD.

Clinic application of Epub of medical imaging film in CT workflow

OBJECTIVE

To investigate the clinical effects of the Epub of medical imaging film (EMIF) in CT workflow.

METHODS

All experiments were approved by the ethics committee of the local district. An EMIF application is fully implemented in PACS according to the DICOM 3.0 standard. For a statistical comparison in the effects of the EMIF in CT workflow, consecutive 50,000 CT hardcopy film prints have been randomly selected before and after the EMIF application in PACS, respectively. The role of the EMIF in the application of hardcopy films, the clinic practice, the integrality of the quality control data archived in PACS/RIS and so on is investigated.

RESULTS

For consecutive 50,000 CT hardcopy films before vs. after the EMIF application respectively, the counts of wasted films are 21 vs. 14 sheets due to the hardware problem, 28 vs. 0 sheets owing to operation errors, and 24 vs. 0 sheets caused by serious defects in image layout. Reissuing films for lost films and other special needs is 60 vs. 55 sheets. The average reissuing film time for each case is 16.97±6.14 vs. 4.36±1.26 min (t=5.94, p<0.01). The EMIF application has offered more convenience for medical viewing and reviewing, and has enhanced the integrality of the quality control data archived in PACS/RIS (χ2=320.835, P<0.01).

CONCLUSION

EMIF application in clinical healthcare can help enhance the function of PACS/RIS, and improve the quality and efficiency in CT routine work.

Comparison of PACS and hard-copy 51-inch radiographs for measuring leg length and deformity

BACKGROUND

The picture archiving and communication system (PACS) eventually will replace the use of standard hard-copy radiographs. It is unknown whether measurements of limb length discrepancy (LLD) and deformity on PACS compare in accuracy and reproducibility with those from hard-copy radiographs.

QUESTIONS/PURPOSES

We compared the reproducibility and reliability of LLD and deformity measurements for each of these two media.

METHODS

We retrospectively reviewed 51-inch standing lower extremity images obtained for LLD or deformity analysis from 40 patients to compare the measurements and their reliability on hard-copy film with those performed on soft-copy PACS. Two observers independently performed measurements twice using each system at 1-week intervals to minimize interobserver or intraobserver bias. Intraclass correlation coefficients (ICCs) were determined to test intraobserver and interrater reliability of Rater 1 and Rater 2.

RESULTS

Interrater reliability of measurements made on hard copy ranged from 0.69 to 0.99 and PACS-derived measurements ranged from 0.66 to 0.98. Intraobserver reliability for Rater 1 for measurements made on hard copy ranged from 0.853 to 0.999 and PACS-derived measurements ranged from 0.80 to 0.996. Intraobserver reliability for Rater 2 for measurements made on hard copy ranged from 0.931 to 0.999 and PACS-derived measurements ranged from 0.962 to 0.999.

CONCLUSIONS

Each system yielded comparable reliability for measurements, therefore, transition to PACS to perform measurements in patients with LLD or deformity can be made with confidence.

LEVEL OF EVIDENCE

Level III, diagnostic study. See the Guidelines for Authors for a complete description of levels of evidence.

Does PACS improve diagnostic accuracy in chest radiograph interpretations in clinical practice?

OBJECTIVES

To assess the impact of a Picture Archiving and Communication System (PACS) on the diagnostic accuracy of the interpretation of chest radiology examinations in a "real life" radiology setting.

MATERIALS AND METHODS

During a period before PACS was introduced to radiologists, when images were still interpreted on film and reported on paper, images and reports were also digitally stored in an image database. The same database was used after the PACS introduction. This provided a unique opportunity to conduct a blinded retrospective study, comparing sensitivity (the main outcome parameter) in the pre and post-PACS periods. We selected 56 digitally stored chest radiograph examinations that were originally read and reported on film, and 66 examinations that were read and reported on screen 2 years after the PACS introduction. Each examination was assigned a random number, and both reports and images were scored independently for pathological findings. The blinded retrospective score for the original reports were then compared with the score for the images (the gold standard).

RESULTS

Sensitivity was improved after the PACS introduction. When both certain and uncertain findings were included, this improvement was statistically significant. There were no other statistically significant changes.

CONCLUSION

The result is consistent with prospective studies concluding that diagnostic accuracy is at least not reduced after PACS introduction. The sensitivity may even be improved.

Comparison of flat-panel digital to conventional film-screen radiography in detection of experimentally created lesions of the equine third metacarpal bone

Radiographic diagnosis of equine bone disease using digital radiography is prevalent in veterinary practice. However, the diagnostic quality of digital vs. conventional radiography has not been compared systematically. We hypothesized that digital radiography would be superior to film-screen radiography for detection of subtle lesions of the equine third metacarpal bone. Twenty-four third metacarpal bones were collected from horses euthanized for reasons other than orthopedic disease. Bones were dissected free of soft tissue and computed tomography was performed to ensure that no osseous abnormalities were present. Subtle osseous lesions were produced in the dorsal cortex of the third metacarpal bones, and the bones were radiographed in a soft tissue phantom using indirect digital and conventional radiography at standard exposures. Digital radiographs were printed onto film. Three Diplomates of the American College of Veterinary Radiology evaluated the radiographs for the presence or absence of a lesion. Receiver operator characteristic curves were constructed, and the area under these curves were compared to assess the ability of the digital and film-screen radiographic systems to detect lesions. The area under the ROC curves for film-screen and digital radiography were 0.87 and 0.90, respectively (P = 0.59). We concluded that the digital radiographic system was comparable to the film-screen system for detection of subtle lesions of the equine third metacarpal bone.

Clinical magnification error in lateral spinal digital radiographs

STUDY DESIGN

Observational.

OBJECTIVE

The objectives of this study were to determine the range of clinical magnification error in lateral spinal digital radiographs, and to determine the effect of body mass index (BMI) on this error.

SUMMARY OF BACKGROUND DATA

The magnification error in plain radiographs is often estimated at 15% to 30%. The variability of this error in digital spinal radiographs has not been assessed.

METHODS

An analysis of 250 patients with digital radiographs and computed tomography (CT)/magnetic resonance images (MRIs) was performed. Digital imaging software was used to measure the anteroposterior vertebral body dimensions at C2, C5, L1, and L4. Magnification values were determined in comparison to CT/MRI. CT measurements were also compared with MRI. BMI for each patient was obtained by chart review.

RESULTS

The mean magnification at the cervical spine (C2 and C5 combined) was 1.22 +/- 0.01, with a range of 1.06 to 1.57 (n = 198, STDEV = 0.08); at the lumbar spine (L1 and L4 combined) it was 1.31 +/- 0.01, with a range of 1.09 to 1.63 (n = 300, STDEV = 0.08). The difference between the mean anteroposterior vertebral body dimensions as measured on CT and MRI was < 0.1 mm (n = 135, P < 0.2514, paired t test). There was a significant positive correlation between BMI and magnification at both the cervical and lumbar spines by linear regression (Cervical: n = 99; P = 0.0019; Lumbar: n = 150; P < 0.0001). There was a significant difference in magnification between nonobese and obese patients at both the cervical and lumbar levels. Cervical: 1.19 +/- 0.01 magnification for nonobese (n = 144), versus 1.26 +/- 0.01 for obese (n = 39) (P < 0.0001). Lumbar: 1.28 +/- 0.01 (n = 208), versus 1.38 +/- 0.01 (n = 78) (P < 0.0001), respectively.

CONCLUSION

Linear clinical measurements obtained on digital radiographs are subject to significant magnification errors at both the cervical and lumbar spines. This error correlates to the patient's BMI. Consequently, clinical decision-making that is based on linear measurements obtained from radiographs that do not account for this error is invalid. In a scenario where this measurement is crucial (e.g., dynamic radiographs), this error can be corrected by comparison to morphometric data from a CT/MRI.

The radiology task: Bayesian theory and perception

The use of a Bayesian framework to understand how radiologists search images for pathology is important as it formalizes, mathematically, how visual and cognitive processes control eye movements by modelling the ideal searcher against which human performance can be compared. It is important that the interpretation of medical images is understood so that new developments in the ways images are presented and the use of image processing software are matched to human abilities and limitations.

Computer-aided nodule detection on digital chest radiography: validation test on consecutive T1 cases of resectable lung cancer

PURPOSE

To evaluate the usefulness of a commercially available computer-assisted diagnosis (CAD) system on operable T1 cases of lung cancer by use of digital chest radiography equipment.

MATERIALS AND METHODS

Fifty consecutive patients underwent surgery for primary lung cancer, and 50 normal cases were selected. All cancer cases were histopathologically confirmed T1 cases. All normal individuals were selected on the basis of chest computed tomography (CT) confirmation and were matched with cancer cases in terms of age and gender distributions. All chest radiographs were obtained with one computed radiography or two flat-panel detector systems. Eight radiologists (four chest radiologists and four residents) participated in observer tests and interpreted soft copy images by using an exclusive display system without and with CAD output. When radiologists diagnosed cases as positives, the locations of lesions were recorded on hard copies. The observers' performance was evaluated by receiver operating characteristic analysis.

RESULTS

The overall detectability of lung cancer cases with CAD system was 74% (37/50), and the false-positive rate was 2.28 (114/50) false positives per case for normal cases. The mean A(z) value increased significantly from 0.896 without CAD output to 0.923 with CAD output (P = 0.018). The main cause of the improvement in performance is attributable to changes from false negatives without CAD to true positives with CAD (19/31, 61%). Moreover, improvement in the location of the tumor was observed in 1.5 cases, on average, for radiology residents.

CONCLUSION

This CAD system for digital chest radiographs is useful in assisting radiologists in the detection of early resectable lung cancer.

Evaluation of the image quality of temporal subtraction images produced automatically in a PACS environment

The aim of the study is to evaluate the reliable production of temporal subtraction images in a picture archiving and communication system environment and to establish objective criteria for the evaluation of image quality. A total of 117 temporal subtraction chest images (55 in the upright position, 62 in the supine position) were obtained in five consecutive days. In all of these, we confirmed that there were no interval changes on the original images, and cases with diffuse lung disease were excluded. The temporal subtraction images were classified by three chest radiologists into five levels: 5, excellent; 4, good; 3, acceptable; 2, poor; and 1, very poor. The following were examined: (1) the yield of adequate quality of the temporal subtraction images; (2) whether the temporal subtraction images were obtained in the warping or nonwarping mode; and (3) the correlation of the overall subjective image quality with the relative shift angles, relative shift distances, and the standard deviation of gray levels in the temporal subtraction images. The percentages of acceptable temporal subtraction images were 100% and 66% in the upright and supine positions, respectively. Sixteen (26%) of the 62 supine-position images were made in nonwarping mode, whereas all upright images were made in warping mode. Significant correlations were obtained in the relative shift angle (P < 0.05), relative horizontal shift distance (P < 0.05), and standard deviation of gray levels (P < 0.0001). Temporal subtraction images with acceptable image quality were obtained in the upright position. The objective criteria may be useful for the evaluation of image quality.

Information loss in visual assessments of medical images

We applied information theory to quantify information losses in assessing contrast-detail (C-D) analysis. Images of a C-D phantom were acquired with a flat panel detector (FPD) and a computed radiography (CR) by changing surface entrance doses. Six phantom radiographs (FPD: five images; CR: one image) were prepared for visual evaluations. Thirteen radiographers and two radiologists participated in the observation test. Detectability was defined as the shortest length of the cylinders of which border the observers could recognize from the background, and was recorded using row number. Information content was defined as the entropy summation operatorp(i)log(1/p(i)) with detection probabilities p(i), which were calculated from distribution of detection rate of the ith column. Information loss, in unit of bits, was calculated as the difference between information obtained and information content when all the columns were detected. The information losses decreased with the increase in cylinder diameters and with the increase in surface entrance dose. Because the information loss varies depending on distribution of detection rate, this method of using the information theory was expected to be more sensitive in evaluating the C-D image quality than using the averaged values of detectability.

Blind placements of peripherally inserted antecubital central catheters: initial catheter tip position in relation to carina

BACKGROUND

We investigated how often blind placement of peripherally inserted central catheters (PICCs) through the antecubital veins results in a correct tip location in relation to carina and evaluated the inter-observer agreement in locating the tip of PICCs in plain radiography with digital imaging.

METHODS

In this study, 202 suitable chest radiographs with PICCs out of 803 patients were identified. An initial audit on the tip of these catheters in relation to carina was done by a consultant anaesthetist and was recorded as the first observer. The same sets of CXRs were examined by a consultant radiologist and the tips were identified and recorded as the second observer. Inter-observer agreement was assessed.

RESULTS

In 75 of 202 (37%), PICCs had a central tip location in relation to the carina. Fifty-five of 131 (42%) right-sided catheters had a central location compared with 20 of 71 (28%) of the left-sided catheters. The tip position for right-sided catheters was most frequently centrally located whereas the tip for left-sided catheters was most commonly positioned in the ipsilateral innominate vein. There was excellent agreement between the observers in reporting the tip of PICCs at all positions (kappa=0.87) including central locations (kappa=0.83).

CONCLUSIONS

Right antecubital PICCs are more likely to be placed in the central location in relation to the carina. PICCs inserted through the left antecubital veins need to be pushed further down to aim for a central location. Inter-observer variability in identifying the tip of PICCs is least with the introduction of digital imaging.

Does variation in interpretation of ultrasonograms account for the variation in incidence of germinal matrix/intraventricular haemorrhage between newborn intensive care units in New Zealand?

BACKGROUND

The incidence of germinal matrix/intraventricular haemorrhage (GM/IVH) reported to the Australian and New Zealand Neonatal Network (ANZNN) varies between neonatal intensive care units (NICUs).

HYPOTHESIS

Differences in the capture, storage, and interpretation of the cerebral ultrasound scans may account for some of this variation.

METHODS

A total of 255 infants with birth weight <1500 g and gestation <32 weeks born between 1997 and 2002 were randomly selected from the ANZNN database, 44 from each of the six NICUs in New Zealand. Twenty two infants from each NICU had cerebral ultrasound scans previously reported to ANZNN as normal; another 22 had scans reported as abnormal. The original scans were copied using digital photography and anonymised and independently read by a panel of three experts using a standardised method of reviewing and reporting.

RESULTS

There was considerable variation between NICUs in methods of image capture and quality and completeness of the scans. However, there was little variation in the reporting of scans between the reviewers and the reports to ANZNN (weighted kappa 0.75-0.91). Grade 1 GM/IVH was generally over-reported and grade 4 under-reported to the ANZNN.

CONCLUSION

For all NICUs, a high level of agreement was found between the reviewers' reports and the reports to the ANZNN. Thus the variation between NICUs in the incidence of GM/IVH reported to the ANZNN is unlikely to be due to differences in capture, storage, and interpretation of the cerebral ultrasound scans. Further investigation is warranted into the reasons for the variation in incidence of GM/IVH between NICUs.