6
|
Lee JM, Ichikawa L, Valencia E, Miglioretti DL, Wernli K, Buist DSM, Kerlikowske K, Henderson LM, Sprague BL, Onega T, Rauscher GH, Lehman CD. Performance Benchmarks for Screening Breast MR Imaging in Community Practice. Radiology 2017; 285:44-52. [PMID: 28582633 PMCID: PMC5621720 DOI: 10.1148/radiol.2017162033] [Citation(s) in RCA: 64] [Impact Index Per Article: 9.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Purpose To compare screening magnetic resonance (MR) imaging performance in the Breast Cancer Surveillance Consortium (BCSC) with Breast Imaging Reporting and Data System (BI-RADS) benchmarks. Materials and Methods This study was approved by the institutional review board and compliant with HIPAA and included BCSC screening MR examinations collected between 2005 and 2013 from 5343 women (8387 MR examinations) linked to regional Surveillance, Epidemiology, and End Results program registries, state tumor registries, and pathologic information databases that identified breast cancer cases and tumor characteristics. Clinical, demographic, and imaging characteristics were assessed. Performance measures were calculated according to BI-RADS fifth edition and included cancer detection rate (CDR), positive predictive value of biopsy recommendation (PPV2), sensitivity, and specificity. Results The median patient age was 52 years; 52% of MR examinations were performed in women with a first-degree family history of breast cancer, 46% in women with a personal history of breast cancer, and 15% in women with both risk factors. Screening MR imaging depicted 146 cancers, and 35 interval cancers were identified (181 total-54 in situ, 125 invasive, and two status unknown). The CDR was 17 per 1000 screening examinations (95% confidence interval [CI]: 15, 20 per 1000 screening examinations; BI-RADS benchmark, 20-30 per 1000 screening examinations). PPV2 was 19% (95% CI: 16%, 22%; benchmark, 15%). Sensitivity was 81% (95% CI: 75%, 86%; benchmark, >80%), and specificity was 83% (95% CI: 82%, 84%; benchmark, 85%-90%). The median tumor size of invasive cancers was 10 mm; 88% were node negative. Conclusion The interpretative performance of screening MR imaging in the BCSC meets most BI-RADS benchmarks and approaches benchmark levels for remaining measures. Clinical practice performance data can inform ongoing benchmark development and help identify areas for quality improvement. © RSNA, 2017.
Collapse
Affiliation(s)
- Janie M. Lee
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Laura Ichikawa
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Elizabeth Valencia
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Diana L. Miglioretti
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Karen Wernli
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Diana S. M. Buist
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Karla Kerlikowske
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Louise M. Henderson
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Brian L. Sprague
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Tracy Onega
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Garth H. Rauscher
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| | - Constance D. Lehman
- From the Department of Radiology, University of Washington, Seattle, Wash (J.M.L., E.V.); Kaiser Permanente Washington Health Research Institute, Seattle, Wash (L.I., D.L.M., K.W., D.S.M.B.); Department of Public Health Sciences, School of Medicine, University of California, Davis School of Medicine, Davis, Calif (D.L.M.); Department of Medicine, Epidemiology and Biostatistics, University of California, San Francisco, Calif (K.K.); Department of Radiology, University of North Carolina School of Medicine, Chapel Hill, NC (L.M.H.); Department of Surgery, University of Vermont, Burlington, Vt (B.L.S.); Norris Cotton Cancer Center and Geisel School of Medicine at Dartmouth, Lebanon, NH (T.O.); Institute for Health Research and Policy, University of Illinois at Chicago, Chicago, Ill (G.H.R.); and Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (C.D.L.)
| |
Collapse
|
9
|
Lacson R, Raja AS, Osterbur D, Ip I, Schneider L, Bain P, Mita C, Whelan J, Silveira P, Dement D, Khorasani R. Assessing Strength of Evidence of Appropriate Use Criteria for Diagnostic Imaging Examinations. J Am Med Inform Assoc 2016; 23:649-53. [PMID: 26911819 DOI: 10.1093/jamia/ocv194] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2015] [Accepted: 11/09/2015] [Indexed: 12/15/2022] Open
Abstract
OBJECTIVE For health information technology tools to fully inform evidence-based decisions, recommendations must be reliably assessed for quality and strength of evidence. We aimed to create an annotation framework for grading recommendations regarding appropriate use of diagnostic imaging examinations. METHODS The annotation framework was created by an expert panel (clinicians in three medical specialties, medical librarians, and biomedical scientists) who developed a process for achieving consensus in assessing recommendations, and evaluated by measuring agreement in grading the strength of evidence for 120 empirically selected recommendations using the Oxford Levels of Evidence. RESULTS Eighty-two percent of recommendations were assigned to Level 5 (expert opinion). Inter-annotator agreement was 0.70 on initial grading (κ = 0.35, 95% CI, 0.23-0.48). After systematic discussion utilizing the annotation framework, agreement increased significantly to 0.97 (κ = 0.88, 95% CI, 0.77-0.99). CONCLUSIONS A novel annotation framework was effective for grading the strength of evidence supporting appropriate use criteria for diagnostic imaging exams.
Collapse
Affiliation(s)
- Ronilda Lacson
- Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA
| | - Ali S Raja
- Harvard Medical School, Boston, MA 02115, USA Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA 02114, USA
| | - David Osterbur
- Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA
| | - Ivan Ip
- Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Louise Schneider
- Harvard Medical School, Boston, MA 02115, USA Department of Medicine, Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Paul Bain
- Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA
| | - Carol Mita
- Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA
| | - Julia Whelan
- Harvard Medical School, Boston, MA 02115, USA Countway Library of Medicine, Boston, MA 02115, USA
| | - Patricia Silveira
- Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA
| | - David Dement
- Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Ramin Khorasani
- Center for Evidence Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA 02115, USA Harvard Medical School, Boston, MA 02115, USA
| |
Collapse
|