Comparing film and digital radiographs for reliability of pneumoconiosis classifications: a modeling approach

Potential of digital chest radiography-based deep learning in screening and diagnosing pneumoconiosis: An observational study

The diagnosis of pneumoconiosis is complex and subjective, leading to inevitable variability in readings. This is especially true for inexperienced doctors. To improve accuracy, a computer-assisted diagnosis system is used for more effective pneumoconiosis diagnoses. Three models (Resnet50, Resnet101, and DenseNet) were used for pneumoconiosis classification based on 1250 chest X-ray images. Three experienced and highly qualified physicians read the collected digital radiography images and classified them from category 0 to category III in a double-blinded manner. The results of the 3 physicians in agreement were considered the relative gold standards. Subsequently, 3 models were used to train and test these images and their performance was evaluated using multi-class classification metrics. We used kappa values and accuracy to evaluate the consistency and reliability of the optimal model with clinical typing. The results showed that ResNet101 was the optimal model among the 3 convolutional neural networks. The AUC of ResNet101 was 1.0, 0.9, 0.89, and 0.94 for detecting pneumoconiosis categories 0, I, II, and III, respectively. The micro-average and macro-average mean AUC values were 0.93 and 0.94, respectively. The accuracy and Kappa values of ResNet101 were 0.72 and 0.7111 for quadruple classification and 0.98 and 0.955 for dichotomous classification, respectively, compared with the relative standard classification of the clinic. This study develops a deep learning based model for screening and staging of pneumoconiosis is using chest radiographs. The ResNet101 model performed relatively better in classifying pneumoconiosis than radiologists. The dichotomous classification displayed outstanding performance, thereby indicating the feasibility of deep learning techniques in pneumoconiosis screening.

Bayesian inference of dependent kappa for binary ratings

In medical and social science research, reliability of testing methods measured through inter- and intraobserver agreement is critical in disease diagnosis. Often comparison of agreement across multiple testing methods is sought in situations where testing is carried out on the same experimental units rendering the outcomes to be correlated. In this article, we first developed a Bayesian method for comparing dependent agreement measures under a grouped data setting. Simulation studies showed that the proposed methodology outperforms the competing methods in terms of power, while maintaining a decent type I error rate. We further developed a Bayesian joint model for comparing dependent agreement measures adjusting for subject and rater-level heterogeneity. Simulation studies indicate that our model outperforms a competing method that is used in this context. The developed methodology was implemented on a key measure on a dichotomous rating scale from a study with six raters evaluating three classification methods for chest radiographs for pneumoconiosis developed by the International Labor Office.

Association Between Financial Conflicts of Interest and ILO Classifications for Black Lung Disease

RATIONALE

The US Department of Labor administers the Federal Black Lung Program (FBLP), an administrative system charged with managing claims by coal miners for workers' compensation for totally disabling coal mine dust lung disease. Specific case reports have raised concern that financial conflicts of interest (COI) may systematically bias physicians when they are classifying chest x-rays (CXRs) for the absence, presence and severity of pneumoconiosis.

OBJECTIVE

To evaluate the direction and magnitude of association between financial COI of physicians participating in the FBLP and international standards for the classification of radiographs of pneumoconiosis.

METHODS

An epidemiologic assessment of black lung claims filed to the FBLP, 2000-2013, was conducted to determine physician classifications of radiographs. FBLP court decisions for years 2002-2019 (n=7,656) were used to evaluate financial COI of each physician. The main outcome measures used were classifications of radiographs for the absence of pneumoconiosis (small opacity classifications of 0/0 or 0/1), simple pneumoconiosis (small opacity classifications of 1/0 through 3/+) and Progressive Massive Fibrosis (PMF; large opacities with classifications of A, B or C).

RESULTS

Out of 63,780 radiograph classifications made by 264 physicians, 31.4% were read positive for simple pneumoconiosis and 3.6% were read as having progressive massive fibrosis (PMF). There were 52 physicians who classified CXRs as having no evidence of pneumoconiosis in 99%+ of their readings, and 18 physicians who classified CXRs as positive for simple pneumoconiosis in 99%+ of their readings. The adjusted odds of a negative classification of pneumoconiosis was 1.46 (CI95%: 1.44, 1.47) per 10% increase in the proportion of court records demonstrating that a physician was hired by the employer. Per 10% increase in court records indicating a physician was hired by the miner-claimant, the adjusted odds ratio for classifying simple pneumoconiosis was 1.51 (CI95%: 1.49, 1.52) and for finding PMF was 1.28 (CI95%: 1.26, 1.30).

CONCLUSIONS

There was a strong association between source of payment and radiograph classification suggesting the importance of eliminating financial COI in what should be an objective determination of eligibility for Black Lung workers' compensation benefits.

International Classification of the Pneumoconiosis Radiograph Reader Training in Turkey

OBJECTIVES

We aimed to determine the characteristics of physicians who had attended the Readers Training of the International Labour Organization International Classification of Radiographs of Pneumoconioses (ILO ICRP) in Turkey.

MATERIALS AND METHODS

This study included 601 physicians attending the Reader Training of the ILO ICRP. Data were collected using an electronic questionnaire, and the inclusiveness of the study was 29.8% (n=179).

RESULTS

In this study, 70.6% of the physicians were men, and the mean age was 48.6±9.6 years; 46.6% of the participants had at least one medical specialty or side branch specialty, and 51.8% were pulmonologists. Furthermore, 52.6% of the physicians worked in the private sector, and 86.6% had an occupational physician certificate. Moreover, 55.3% of the participants evaluated the radiographs using the authority gained by the certification, and 68.3% of those who did not evaluate the films stated that the reason for not evaluating the films was a lack of demand. Participants who evaluated radiographs had received a demand for films most frequently from 1 to 2 different jobs (33.4%) and from 1 to 3 different workplaces (30.1%). Most films came from the mining (47.5%) and quarrying (50.5%) sectors. Some participants (64.3%) stated that the quality of the radiographs was insufficient, 59.2% experienced difficulties because the radiographs were not obtained using proper techniques, 23.4% stated that the fees per film evaluated were low, and 81.5% believed that update training is necessary.

CONCLUSION

The demand for these services will increase in line with the training and surveillance as stipulated by the legislation.

Potential of deep learning in assessing pneumoconiosis depicted on digital chest radiography

Objectives

To investigate the potential of deep learning in assessing pneumoconiosis depicted on digital chest radiographs and to compare its performance with certified radiologists.

Methods

We retrospectively collected a dataset consisting of 1881 chest X-ray images in the form of digital radiography. These images were acquired in a screening setting on subjects who had a history of working in an environment that exposed them to harmful dust. Among these subjects, 923 were diagnosed with pneumoconiosis, and 958 were normal. To identify the subjects with pneumoconiosis, we applied a classical deep convolutional neural network (CNN) called Inception-V3 to these image sets and validated the classification performance of the trained models using the area under the receiver operating characteristic curve (AUC). In addition, we asked two certified radiologists to independently interpret the images in the testing dataset and compared their performance with the computerised scheme.

Results

The Inception-V3 CNN architecture, which was trained on the combination of the three image sets, achieved an AUC of 0.878 (95% CI 0.811 to 0.946). The performance of the two radiologists in terms of AUC was 0.668 (95% CI 0.555 to 0.782) and 0.772 (95% CI 0.677 to 0.866), respectively. The agreement between the two readers was moderate (kappa: 0.423, p<0.001).

Conclusion

Our experimental results demonstrated that the deep leaning solution could achieve a relatively better performance in classification as compared with other models and the certified radiologists, suggesting the feasibility of deep learning techniques in screening pneumoconiosis.

ACR Appropriateness Criteria® Occupational Lung Diseases

Ordering the appropriate diagnostic imaging for occupational lung disease requires a firm understanding of the relationship between occupational exposure and expected lower respiratory track manifestation. Where particular inorganic dust exposures typically lead to nodular and interstitial lung disease, other occupational exposures may lead to isolated small airway obstruction. Certain workplace exposures, like asbestos, increase the risk of malignancy, but also produce pulmonary findings that mimic malignancy. This publication aims to delineate the common and special considerations associated with occupational lung disease to assist the ordering physician in selecting the most appropriate imaging study, while still stressing the importance of a multidisciplinary approach. The American College of Radiology Appropriateness Criteria are evidence-based guidelines for specific clinical conditions that are reviewed annually by a multidisciplinary expert panel. The guideline development and revision include an extensive analysis of current medical literature from peer reviewed journals and the application of well-established methodologies (RAND/UCLA Appropriateness Method and Grading of Recommendations Assessment, Development, and Evaluation or GRADE) to rate the appropriateness of imaging and treatment procedures for specific clinical scenarios. In those instances where evidence is lacking or equivocal, expert opinion may supplement the available evidence to recommend imaging or treatment.

Comparison of digital and film chest radiography for detection and medical surveillance of silicosis in a setting with a high burden of tuberculosis

BACKGROUND

Continuing use of analog film and digital chest radiography for screening and surveillance for pneumoconiosis and tuberculosis in lower and middle income countries raises questions of equivalence of disease detection. This study compared analog to digital images for intra-rater agreement across formats and prevalence of changes related to silicosis and tuberculosis among South African gold miners using the International Labour Organization classification system.

METHODS

Miners with diverse radiological presentations of silicosis and tuberculosis were recruited. Digital and film chest images on each subject were classified by four expert readers.

RESULTS

Readings of film and soft copy digital images showed no significant differences in prevalence of tuberculosis or silicosis, and intra-rater agreement across formats was fair to good. Hard copy images yielded higher prevalences.

CONCLUSION

Film and digital soft copy images show consistent prevalence of findings, and generally fair to good intra-rater agreement for findings related to silicosis and tuberculosis.

Validation of the international labour office digitized standard images for recognition and classification of radiographs of pneumoconiosis

RATIONALE AND OBJECTIVES

Chest radiographs are recommended for prevention and detection of pneumoconiosis. In 2011, the International Labour Office (ILO) released a revision of the International Classification of Radiographs of Pneumoconioses that included a digitized standard images set. The present study compared results of classifications of digital chest images performed using the new ILO 2011 digitized standard images to classification approaches used in the past.

MATERIALS AND METHODS

Underground coal miners (N = 172) were examined using both digital and film-screen radiography (FSR) on the same day. Seven National Institute for Occupational Safety and Health-certified B Readers independently classified all 172 digital radiographs, once using the ILO 2011 digitized standard images (DRILO2011-D) and once using digitized standard images used in the previous research (DRRES). The same seven B Readers classified all the miners' chest films using the ILO film-based standards.

RESULTS

Agreement between classifications of FSR and digital radiography was identical, using a standard image set (either DRILO2011-D or DRRES). The overall weighted κ value was 0.58. Some specific differences in the results were seen and noted. However, intrareader variability in this study was similar to the published values and did not appear to be affected by the use of the new ILO 2011 digitized standard images.

CONCLUSIONS

These findings validate the use of the ILO digitized standard images for classification of small pneumoconiotic opacities. When digital chest radiographs are obtained and displayed appropriately, results of pneumoconiosis classifications using the 2011 ILO digitized standards are comparable to film-based ILO classifications and to classifications using earlier research standards.

Reliability and validity of soft copy images based on flat-panel detector in pneumoconiosis classification: comparison with the analog radiographs

RATIONALE AND OBJECTIVES

The aim of this study was to evaluate the reliability and validity of soft copy images based on flat-panel detector of digital radiography (DR-FPD soft copy images) compared to analog radiographs (ARs) in pneumoconiosis classification and diagnosis.

MATERIALS AND METHODS

DR-FPD soft copy images and ARs from 349 subjects were independently read by four-experienced readers according to the International Labor Organization 2000 guidelines. DR-FPD soft copy images were used to obtain consensus reading (CR) by all readers as the gold standard. Reliability and validity were evaluated by a κ and receiver operating characteristic analysis, respectively.

RESULTS

In small opacity, overall interreader agreement of DR-FPD soft copy images was significantly higher than that of ARs, but it was not significantly different in large opacity and costophrenic angle obliteration. In small opacity, agreement of DR-FPD soft copy images with CR was significantly higher than that of ARs with CR. It was also higher than that of ARs with CR in pleural plaque and thickening. Receiver operating characteristic areas were not different significantly between DR-FPD soft copy images and ARs.

CONCLUSIONS

DR-FPD soft copy images showed accurate and reliable results in pneumoconiosis classification and diagnosis compared to ARs.

The classic pneumoconioses: new epidemiological and laboratory observations

The purpose of this article is to provide an update on selected issues of current interest and recent developments related to 3 types of inorganic mineral dust exposures causing classic forms of pneumoconiosis: coal mine dust, crystalline silica, and asbestos. Common themes include new imaging modalities, emerging exposures, and evolving appreciation of additional adverse health effects associated with exposure to these inorganic mineral dusts.

Utility of digital radiography for the screening of pneumoconiosis as compared to analog radiography: radiation dose, image quality, and pneumoconiosis classification

The purpose of this study was to compare digital radiography (DR) and analog radiography (AR) for the screening of pneumoconiosis with respect to radiation dose, image quality, and pneumoconiosis classification. DR was performed on 50 subjects who were enrolled for an examination of pneumoconiosis (Digital Diagnost™, Philips, Netherlands), and AR (MXO-15B, Toshiba, Japan) was performed the same day after the study was approved by the Institutional Review Board and written informed consent was obtained from all subjects. Entrance surface doses (ESDs) of DR and AR were measured using a glass dosimeter attached to a Rando human phantom (Alderson Co., U.S.) under exposure conditions commonly used in clinical practice in Korea. Visibilities on all images were evaluated using a 5-point scale by four chest radiologists using a modified form of the European Chest Guidelines (EUR 16260). All the images were classified using the ILO's guidelines by referencing standard analog radiographs. ESDs of DR were significantly lower than those of AR (0.15 mGy vs. 0.21 mGy, p < 0.05). All anatomic structures were significantly more visible by DR images (p < 0.0001), especially the left main bronchus, ribs, and thoracic spine. Body mass index did not correlate with anatomic structure visibility by DR (r = -0.029, p = 0.842) or AR images (r = -0.076, p = 0.602). Overall intra- and inter-reader agreements for DR images were significantly higher than for AR images. DR offers improved image quality with a significant reduction of up to 23.6% in radiation dose and more accurate pneumoconiosis classification than AR.

Comparison of digital with film radiographs for the classification of pneumoconiotic pleural abnormalities

RATIONALE AND OBJECTIVES

Analog film radiographs are typically used to classify pneumoconiosis to allow comparison with standard film radiographs. The aim of this study was to determine if digital radiography is comparable to film for the purpose of classifying pneumoconiotic pleural abnormalities.

MATERIALS AND METHODS

Subjects were 200 asbestos-exposed patients, from whom digital and film chest radiographs were obtained along with chest high-resolution computed tomographic scans. Using a crossover design, radiographs were independently read on two occasions by seven readers, using conventional International Labour Organization standards for film and digitized standards for digital. High-resolution computed tomographic scans were read independently by three readers. Areas under the receiver-operating characteristic curves were calculated using high-resolution computed tomographic ratings as the gold standard for disease status. Mixed linear models were fit to estimate the effects of order of presentation, occasion, and modality, treating the seven readers as a random effect. Comparing digital and film radiography for each reader and occasion, crude agreement and agreement beyond chance (κ) were also calculated.

RESULTS

The linear models showed no statistically significant sequence effect for order of presentation (P = .73) or occasion (P = .28). Most important, the difference between modalities was not statistically significant (digital vs film, P = .54). The mean area under the curve for film was 0.736 and increased slightly to 0.741 for digital. Mean crude agreement for the presence of pleural abnormalities consistent with pneumoconiosis across all readers and occasions was 78.3%, while the mean κ value was 0.49.

CONCLUSIONS

These results indicate that digital radiography is not statistically different from analog film for the purpose of classifying pneumoconiotic pleural abnormalities, when appropriate standards are used.

Development of standard digital images for pneumoconiosis

We developed the standard digital images (SDIs) to be used in the classification and recognition of pneumoconiosis. From July 3, 2006 through August 31, 2007, 531 retired male workers exposed to inorganic dust were examined by digital (DR) and analog radiography (AR) on the same day, after being approved by our institutional review board and obtaining informed consent from all participants. All images were twice classified according to the International Labour Office (ILO) 2000 guidelines with reference to ILO standard analog radiographs (SARs) by four chest radiologists. After consensus reading on 349 digital images matched with the first selected analog images, 120 digital images were selected as the SDIs that considered the distribution of pneumoconiosis findings. Images with profusion category 0/1, 1, 2, and 3 were 12, 50, 40, and 15, respectively, and a large opacity were in 43 images (A = 20, B = 22, C = 1). Among pleural abnormality, costophrenic angle obliteration, pleural plaque and thickening were in 11 (9.2%), 31 (25.8%), and 9 (7.5%) images, respectively. Twenty-one of 29 symbols were present except cp, ef, ho, id, me, pa, ra, and rp. A set of 120 SDIs had more various pneumoconiosis findings than ILO SARs that were developed from adequate methods. It can be used as digital reference images for the recognition and classification of pneumoconiosis.

Comparison of digital direct readout radiography with conventional film-screen radiography for the recognition of pneumoconiosis in dust-exposed Chinese workers

OBJECTIVES

Pneumoconiosis in China remains a disease with substantial public health significance. Diagnostic standards for the pneumoconioses are based on traditional film-screen radiography (FSR). However, FSR is increasingly being replaced with digital radiographic imaging, which has become the predominant technology available in Chinese clinical practice. To evaluate the applicability of digital direct readout radiography (DR) images in the recognition of pneumoconioses, we compared the profusion of small opacities and large opacities between FSR and DR radiographs.

METHODS

We enrolled 161 pneumoconioses patients and 31 dust-exposed workers during the course of the study, with FSR and DR images obtained from all participants. Each chest film was interpreted by five readers using the Chinese Diagnostic Criteria classification of radiographs of pneumoconiosis, as were DR images displayed on medical-grade computer monitors.

RESULTS

No statistically significant differences were observed when the data were analyzed by small opacity profusion subcategory except for 1/1. The overall intermodality agreement of small opacities was good, with a weighted kappa (κ) of 0.77.

CONCLUSIONS

DR images with soft copy display are equivalent with respect to image quality and the recognition and classification of small parenchymal lung opacities. Additionally, we observed likeness between modalities with respect to the classification of large opacities. Overall, our study findings demonstrate that in a population of Chinese workers with pneumoconiosis, direct readout digital systems are equivalent to traditional film-screen radiography in the recognition and classification of small pneumoconiotic opacities.

Intramodality and intermodality comparisons of storage phosphor computed radiography and conventional film-screen radiography in the recognition of small pneumoconiotic opacities

BACKGROUND

Digital radiography systems are replacing traditional film for chest radiographic monitoring in the recognition of pneumoconiosis.

METHODS

To further investigate previous findings regarding the equivalence of film-screen radiographs (FSRs) and storage phosphor computed radiographs (CRs), FSRs and CRs from 172 underground coal miners were classified independently by seven National Institute for Occupational Safety and Health-approved B readers, using the International Labor Office (ILO) classification of radiographs of pneumoconiosis.

RESULTS

More CRs were classified as "good" quality compared with FSRs (prevalence ratio [PR], 1.5; 95% CI, 1.4-1.6; P , .001). B readers showed good overall agreement on scoring small opacity profusion using CRs vs FSRs (weighted κ, 0.58; 95% CI, 0.54-0.62). Significantly more irregular opacities (compared with rounded) were classified using CR images compared with FSR (PR, 1.3; 95% CI, 1.1-1.6; P = .01). Similarly, the smallest sized opacities (width < 1.5 mm, p and s type) were reported more frequently using CR vs FSR images (PR, 1.3; 95% CI, 1.1-1.5; P < .001). Interreader and intrareader agreement was lower with respect to the classification of shape and size than for small opacity profusion. Overall, interreader and intrareader variability did not differ significantly using CR vs FSR.

CONCLUSIONS

Under optimal conditions, using standardized methods and equipment, reader visualization of small pneumoconiotic opacities does not appear to differ meaningfully, whether using CR or FSR. Variability in ILO classifications between imaging modalities appears to be considerably lower than variability among readers. The well-documented challenge of reader variability does not appear to be resolved through the use of digital imaging alone, and additional approaches must be evaluated.