1
|
Hsieh SS, Inoue A, Yalon M, Cook DA, Gong H, Sudhir Pillai P, Johnson MP, Fidler JL, Leng S, Yu L, Carter RE, Holmes DR, McCollough CH, Fletcher JG. Targeted Training Reduces Search Errors but Not Classification Errors for Hepatic Metastasis Detection at Contrast-Enhanced CT. Acad Radiol 2024; 31:448-456. [PMID: 37567818 PMCID: PMC10853479 DOI: 10.1016/j.acra.2023.06.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Revised: 06/15/2023] [Accepted: 06/20/2023] [Indexed: 08/13/2023]
Abstract
RATIONALE AND OBJECTIVES Methods are needed to improve the detection of hepatic metastases. Errors occur in both lesion detection (search) and decisions of benign versus malignant (classification). Our purpose was to evaluate a training program to reduce search errors and classification errors in the detection of hepatic metastases in contrast-enhanced abdominal computed tomography (CT). MATERIALS AND METHODS After Institutional Review Board approval, we conducted a single-group prospective pretest-posttest study. Pretest and posttest were identical and consisted of interpreting 40 contrast-enhanced abdominal CT exams containing 91 liver metastases under eye tracking. Between pretest and posttest, readers completed search training with eye-tracker feedback and coaching to increase interpretation time, use liver windows, and use coronal reformations. They also completed classification training with part-task practice, rating lesions as benign or malignant. The primary outcome was metastases missed due to search errors (<2 seconds gaze under eye tracker) and classification errors (>2 seconds). Jackknife free-response receiver operator characteristic (JAFROC) analysis was also conducted. RESULTS A total of 31 radiologist readers (8 abdominal subspecialists, 8 nonabdominal subspecialists, 15 senior residents/fellows) participated. Search errors were reduced (pretest 11%, posttest 8%, difference 3% [95% confidence interval, 0.3%-5.1%], P = .01), but there was no difference in classification errors (difference 0%, P = .97) or in JAFROC figure of merit (difference -0.01, P = .36). In subgroup analysis, abdominal subspecialists demonstrated no evidence of change. CONCLUSION Targeted training reduced search errors but not classification errors for the detection of hepatic metastases at contrast-enhanced abdominal CT. Improvements were not seen in all subgroups.
Collapse
Affiliation(s)
- Scott S Hsieh
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.); Department of General Internal Medicine, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H.).
| | - Akitoshi Inoue
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Mariana Yalon
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - David A Cook
- Quantitative Health Services - Clinical Trials and Biostatistics, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (D.A.C.)
| | - Hao Gong
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Parvathy Sudhir Pillai
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Matthew P Johnson
- Department of Physiology Biomedical Engineering, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (M.P.J., R.E.C.)
| | - Jeff L Fidler
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Shuai Leng
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Lifeng Yu
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Rickey E Carter
- Department of Physiology Biomedical Engineering, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (M.P.J., R.E.C.)
| | - David R Holmes
- Quantitative Health Services - Clinical Trials and Biostatistics, Mayo Clinic, 4500 San Pablo Road, Jacksonville, FL 32224 (D.R.H. III)
| | - Cynthia H McCollough
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| | - Joel G Fletcher
- Department of Radiology, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 (S.S.H., A.I., M.Y., H.G., P.S.P., J.L.F., S.L., L.Y., C.H.McC., J.G.F.)
| |
Collapse
|
2
|
Yapp KE, Suleiman M, Brennan P, Ekpo E. Periapical Radiography versus Cone Beam Computed Tomography in Endodontic Disease Detection: A Free-response, Factorial Study. J Endod 2023; 49:419-429. [PMID: 36773745 DOI: 10.1016/j.joen.2023.02.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Revised: 01/17/2023] [Accepted: 02/01/2023] [Indexed: 02/11/2023]
Abstract
AIM To assess and compare reader performance in interpreting digital periapical (PA) radiography and cone beam computed tomography (CBCT) in endodontic disease detection, using a free-response, factorial model. MATERIALS AND METHODS A reader performance study of 2 image test sets was undertaken using a factorial, free-response design, accounting for the independent variables: case type, case severity, reader type, and imaging modality. Twenty-two readers interpreted 60 PA and 60 CBCT images divided into 5 categories: diseased-subtle, diseased-moderate, diseased-obvious, nondiseased-subtle, and nondiseased-obvious. Lesion localization fraction, specificity, false positive (FP) marks, and the weighted alternative free-response receiver operating characteristic figure of merit were calculated. RESULTS CBCT had greater specificity than PA in the obvious nondiseased cases (P = .01) and no significant difference in the subtle nondiseased category. Weighted alternative free-response receiver operating characteristic values were higher for PA than CBCT in the subtle diseased (P = .02) and moderate diseased (P = .01) groups with no significant difference between in the obvious diseased groups. CBCT had higher mean FPs than PA (P < .05) in subtle diseased cases. Mean lesion localization fraction in the moderate diseased group was higher in PA than CBCT (P = .003). No relationships were found between clinical experience and all diagnostic performance measures, except for in the obvious diseased CBCT group, where increasing experience was associated mean FP marks (P = .04). CONCLUSIONS Reader performance in the detection of endodontic disease is better with PA radiography than CBCT. Clinical experience does not impact upon the accuracy of interpretation of both PA radiography and CBCT.
Collapse
Affiliation(s)
- Kehn E Yapp
- Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Imaging Science, Faculty of Medicine and Health, School of Health Sciences, The University of Sydney, Camperdown, New South Wales, Australia.
| | - Mo'ayyad Suleiman
- Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Imaging Science, Faculty of Medicine and Health, School of Health Sciences, The University of Sydney, Camperdown, New South Wales, Australia
| | - Patrick Brennan
- Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Imaging Science, Faculty of Medicine and Health, School of Health Sciences, The University of Sydney, Camperdown, New South Wales, Australia
| | - Ernest Ekpo
- Medical Image Optimisation and Perception Group (MIOPeG), Discipline of Medical Imaging Science, Faculty of Medicine and Health, School of Health Sciences, The University of Sydney, Camperdown, New South Wales, Australia
| |
Collapse
|
3
|
The effect of clinical history on diagnostic performance of endodontic cone-beam CT interpretation. Clin Radiol 2023; 78:e433-e441. [PMID: 36702710 DOI: 10.1016/j.crad.2022.12.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2022] [Revised: 11/21/2022] [Accepted: 12/09/2022] [Indexed: 01/12/2023]
Abstract
AIM To assess the effect of clinical history on the interpretation of endodontic disease in dental cone-beam computed tomography (CBCT). MATERIALS AND METHODS A reader performance study of an image test-set was undertaken using a factorial, free-response, crossover design, accounting for the independent variables: case type, case severity, reader type, and reading modality. Twenty-three readers interpreted 60 CBCT images twice over two reading sessions using a balanced design, once with access to clinical history and once without, where 30 in each session included history. Lesion localisations, specificity, false-positive marks and the weighted alternative free-response receiver operating characteristic (wAFROC1) figure of merit were calculated. RESULTS Clinical history had no significant effect on specificity and false-positive rates in non-diseased cases (p>0.05), but improved lesion localisation in subtle and obvious diseased cases (p<0.01). wAFROC1 values were higher with clinical history for subtle (0.58 versus 0.48; p<0.001) and obvious (0.77 versus 0.71; p=0.006) diseased categories. No associations were observed between clinical history and both readers' years of experience and reading volume in the non-diseased categories. Readers with fewer (p=0.03) and moderate (p=0.008) years of experience and low (p=0.002) CBCT reading volume demonstrated better lesion localisation in subtle diseased cases when clinical history was available. CONCLUSIONS Clinical history improved the interpretation of CBCT images with disease without affecting the interpretation of images without disease. Less and moderately experienced readers and low-volume readers benefitted more from availability of clinical history.
Collapse
|
5
|
Schultheiss M, Schmette P, Bodden J, Aichele J, Müller-Leisse C, Gassert FG, Gassert FT, Gawlitza JF, Hofmann FC, Sasse D, von Schacky CE, Ziegelmayer S, De Marco F, Renger B, Makowski MR, Pfeiffer F, Pfeiffer D. Lung nodule detection in chest X-rays using synthetic ground-truth data comparing CNN-based diagnosis to human performance. Sci Rep 2021; 11:15857. [PMID: 34349135 PMCID: PMC8339004 DOI: 10.1038/s41598-021-94750-z] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2020] [Accepted: 07/15/2021] [Indexed: 12/24/2022] Open
Abstract
We present a method to generate synthetic thorax radiographs with realistic nodules from CT scans, and a perfect ground truth knowledge. We evaluated the detection performance of nine radiologists and two convolutional neural networks in a reader study. Nodules were artificially inserted into the lung of a CT volume and synthetic radiographs were obtained by forward-projecting the volume. Hence, our framework allowed for a detailed evaluation of CAD systems' and radiologists' performance due to the availability of accurate ground-truth labels for nodules from synthetic data. Radiographs for network training (U-Net and RetinaNet) were generated from 855 CT scans of a public dataset. For the reader study, 201 radiographs were generated from 21 nodule-free CT scans with altering nodule positions, sizes and nodule counts of inserted nodules. Average true positive detections by nine radiologists were 248.8 nodules, 51.7 false positive predicted nodules and 121.2 false negative predicted nodules. The best performing CAD system achieved 268 true positives, 66 false positives and 102 false negatives. Corresponding weighted alternative free response operating characteristic figure-of-merits (wAFROC FOM) for the radiologists range from 0.54 to 0.87 compared to a value of 0.81 (CI 0.75-0.87) for the best performing CNN. The CNN did not perform significantly better against the combined average of the 9 readers (p = 0.49). Paramediastinal nodules accounted for most false positive and false negative detections by readers, which can be explained by the presence of more tissue in this area.
Collapse
Affiliation(s)
- Manuel Schultheiss
- Chair of Biomedical Physics, Department of Physics and Munich School of BioEngineering, Technical University of Munich, 85748, Garching, Germany.
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany.
| | - Philipp Schmette
- Chair of Biomedical Physics, Department of Physics and Munich School of BioEngineering, Technical University of Munich, 85748, Garching, Germany
| | - Jannis Bodden
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Juliane Aichele
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Christina Müller-Leisse
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Felix G Gassert
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Florian T Gassert
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Joshua F Gawlitza
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Felix C Hofmann
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Daniel Sasse
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Claudio E von Schacky
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Sebastian Ziegelmayer
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Fabio De Marco
- Chair of Biomedical Physics, Department of Physics and Munich School of BioEngineering, Technical University of Munich, 85748, Garching, Germany
| | - Bernhard Renger
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Marcus R Makowski
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Franz Pfeiffer
- Chair of Biomedical Physics, Department of Physics and Munich School of BioEngineering, Technical University of Munich, 85748, Garching, Germany
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| | - Daniela Pfeiffer
- Department of Diagnostic and Interventional Radiology, School of Medicine and Klinikum rechts der Isar, Technical University of Munich, 81675, Munich, Germany
| |
Collapse
|