1
|
Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example. J Am Coll Radiol 2022; 19:1162-1169. [PMID: 35981636 DOI: 10.1016/j.jacr.2022.05.030] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 05/19/2022] [Accepted: 05/20/2022] [Indexed: 11/23/2022]
Abstract
OBJECTIVE Address model drift in a machine learning (ML) model for predicting diagnostic imaging follow-up using data augmentation with more recent data versus retraining new predictive models. METHODS This institutional review board-approved retrospective study was conducted January 1, 2016, to December 31, 2020, at a large academic institution. A previously trained ML model was trained on 1,000 radiology reports from 2016 (old data). An additional 1,385 randomly selected reports from 2019 to 2020 (new data) were annotated for follow-up recommendations and randomly divided into two sets: training (n = 900) and testing (n = 485). Support vector machine and random forest (RF) algorithms were constructed and trained using 900 new data reports plus old data (augmented data, new models) and using only new data (new data, new models). The 2016 baseline model was used as comparator as is and trained with augmented data. Recall was compared with baseline using McNemar's test. RESULTS Follow-up recommendations were contained in 11.3% of reports (157 or 1,385). The baseline model retrained with new data had precision = 0.83 and recall = 0.54; none significantly different from baseline. A new RF model trained with augmented data had significantly better recall versus the baseline model (0.80 versus 0.66, P = .04) and comparable precision (0.90 versus 0.86). DISCUSSION ML methods for monitoring follow-up recommendations in radiology reports suffer model drift over time. A newly developed RF model achieved better recall with comparable precision versus simply retraining a previously trained original model with augmented data. Thus, regularly assessing and updating these models is necessary using more recent historical data.
Collapse
|
2
|
Lacson R, Eskian M, Cochon L, Gujrathi I, Licaros A, Zhao A, Vetrano N, Schneider L, Raja A, Khorasani R. Representing narrative evidence as clinical evidence logic statements. JAMIA Open 2022; 5:ooac024. [PMID: 35474718 PMCID: PMC9030217 DOI: 10.1093/jamiaopen/ooac024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Revised: 03/05/2022] [Accepted: 03/25/2022] [Indexed: 11/12/2022] Open
Abstract
Abstract
Objective
Clinical evidence logic statements (CELS) are shareable knowledge artifacts in a semistructured “If-Then” format that can be used for clinical decision support systems. This project aimed to assess factors facilitating CELS representation.
Materials and Methods
We described CELS representation of clinical evidence. We assessed factors that facilitate representation, including authoring instruction, evidence structure, and educational level of CELS authors. Five researchers were tasked with representing CELS from published evidence. Represented CELS were compared with the formal representation. After an authoring instruction intervention, the same researchers were asked to represent the same CELS and accuracy was compared with that preintervention using McNemar’s test. Moreover, CELS representation accuracy was compared between evidence that is structured versus semistructured, and between CELS authored by specialty-trained versus nonspecialty-trained researchers, using χ2 analysis.
Results
261 CELS were represented from 10 different pieces of published evidence by the researchers pre- and postintervention. CELS representation accuracy significantly increased post-intervention, from 20/261 (8%) to 63/261 (24%, P value < .00001). More CELS were assigned for representation with 379 total CELS subsequently included in the analysis (278 structured and 101 semistructured) postintervention. Representing CELS from structured evidence was associated with significantly higher CELS representation accuracy (P = .002), as well as CELS representation by specialty-trained authors (P = .0004).
Discussion
CELS represented from structured evidence had a higher representation accuracy compared with semistructured evidence. Similarly, specialty-trained authors had higher accuracy when representing structured evidence.
Conclusion
Authoring instructions significantly improved CELS representation with a 3-fold increase in accuracy. However, CELS representation remains a challenging task.
Collapse
Affiliation(s)
- Ronilda Lacson
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | - Mahsa Eskian
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Laila Cochon
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Isha Gujrathi
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Andro Licaros
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Anna Zhao
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | - Nicole Vetrano
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
| | - Louise Schneider
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| | - Ali Raja
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - Ramin Khorasani
- Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women’s Hospital, Boston, Massachusetts, USA
- Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
3
|
Rousseau JF, Ip IK, Raja AS, Schuur JD, Khorasani R. Can emergency department provider notes help to achieve more dynamic clinical decision support? J Am Coll Emerg Physicians Open 2020; 1:1269-1277. [PMID: 33392531 PMCID: PMC7771753 DOI: 10.1002/emp2.12232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Revised: 07/31/2020] [Accepted: 08/05/2020] [Indexed: 11/19/2022] Open
Abstract
OBJECTIVE Assess whether clinical data were present in emergency department (ED) provider notes at time of order entry for cervical spine (c-spine) imaging that could be used to augment or pre-populate clinical decision support (CDS) attributes. METHODS This Institutional Review Board-approved retrospective study, performed in a quaternary hospital, included all encounters for adult ED patients seen April 1, 2013-September 30, 2014 for a chief complaint of trauma who received c-spine computed tomography (CT) or x-ray. We assessed proportion of ED encounters with at least 1 c-spine-specific CDS rule attribute in clinical notes available at the time of imaging order and agreement between attributes in clinical notes and data entered into CDS. RESULTS A portion of the clinical note was submitted before imaging order in 42% (184/438) of encounters reviewed; 59.2% (109/184) of encounters with note portions submitted before imaging order had at least 1 positive CDS attribute identified supporting imaging study appropriateness; 34.8% (64/184) identified exclusion criteria where CDS appropriateness recommendations would not be applicable. 65.8% (121/184) of encounters had either a positive CDS attribute or an exclusion criterion. Concordance of c-spine CDS attributes when present in both notes and CDS was 68.4% (κ = 0.35 95% CI: 0.15-0.56; McNemar P = 0.23). CONCLUSIONS Clinical notes are an underutilized source of clinical attributes needed for CDS, available in a substantial percentage of encounters at the time of imaging order. Automated pre-population of imaging order requisitions with relevant clinical information extracted from electronic health record provider notes may: (1) improve ordering efficiency by reducing redundant data entry, (2) help improve clinical relevance of CDS alerts, and (3) potentially reduce provider burnout from extraneous alerts.
Collapse
Affiliation(s)
- Justin F. Rousseau
- Center for Evidence‐Based ImagingBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of RadiologyBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of Population HealthDell Medical School, The University of Texas at AustinAustinTexasUSA
- Department of NeurologyDell Medical School, The University of Texas at AustinAustinTexasUSA
| | - Ivan K. Ip
- Center for Evidence‐Based ImagingBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of RadiologyBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of MedicineBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
| | - Ali S. Raja
- Center for Evidence‐Based ImagingBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of RadiologyBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
| | - Jeremiah D. Schuur
- Department of Emergency MedicineBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
| | - Ramin Khorasani
- Center for Evidence‐Based ImagingBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
- Department of RadiologyBrigham and Women's HospitalHarvard Medical SchoolBostonMassachusettsUSA
| |
Collapse
|
4
|
Physician Agreement With Recommendations Contained in a National Guideline for the Management of Incidental Pulmonary Nodules: A Case Study. J Am Coll Radiol 2020; 17:1437-1442. [PMID: 32783898 PMCID: PMC7655688 DOI: 10.1016/j.jacr.2020.07.020] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Accepted: 07/15/2020] [Indexed: 11/22/2022]
|
5
|
Diekhoff T, Kainberger F, Oleaga L, Dewey M, Zimmermann E. Effectiveness of the clinical decision support tool ESR eGUIDE for teaching medical students the appropriate selection of imaging tests: randomized cross-over evaluation. Eur Radiol 2020; 30:5684-5689. [PMID: 32435929 PMCID: PMC7476994 DOI: 10.1007/s00330-020-06942-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2019] [Revised: 03/17/2020] [Accepted: 05/07/2020] [Indexed: 11/25/2022]
Abstract
OBJECTIVES To evaluate ESR eGUIDE-the European Society of Radiology (ESR) e-Learning tool for appropriate use of diagnostic imaging modalities-for learning purposes in different clinical scenarios. METHODS This anonymized evaluation was performed after approval of ESR Education on Demand leadership. Forty clinical scenarios were developed in which at least one imaging modality was clinically most appropriate, and the scenarios were divided into sets 1 and 2. These sets were provided to medical students randomly assigned to group A or B to select the most appropriate imaging test for each scenario. Statistical comparisons were made within and across groups. RESULTS Overall, 40 medical students participated, and 31 medical students (78%) answered both sets. The number of correctly chosen imaging methods per set in these 31 paired samples was significantly higher when answered with versus without use of ESR eGUIDE (13.7 ± 2.6 questions vs. 12.1 ± 3.2, p = 0.012). Among the students in group A, who first answered set 1 without ESR eGUIDE (11.1 ± 3.2), there was significant improvement when set 2 was answered with ESR eGUIDE (14.3 ± 2.5, p = 0.013). The number of correct answers in group B did not drop when set 2 was answered without ESR eGUIDE (12.4 ± 2.6) after having answered set 1 first with ESR eGUIDE (13.0 ± 2.7, p = 0.66). CONCLUSION The clinical decision support tool ESR eGUIDE is suitable for training medical students in choosing the best radiological imaging modality in typical scenarios, and its use in teaching radiology can thus be recommended. KEY POINTS • ESR eGUIDE improved the number of appropriately selected imaging modalities among medical students. • This improvement was also seen in the group of students which first selected imaging tests without ESR eGUIDE. • In the student group which used ESR eGUIDE first, appropriate selection remained stable even without the teaching tool.
Collapse
Affiliation(s)
- Torsten Diekhoff
- Department of Radiology, Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universitat Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Franz Kainberger
- Department of Radiology, Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universitat Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Laura Oleaga
- Department of Radiology, Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universitat Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Marc Dewey
- Department of Radiology, Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universitat Berlin, Charitéplatz 1, 10117, Berlin, Germany.
| | - Elke Zimmermann
- Department of Radiology, Charité - Universitätsmedizin Berlin, Humboldt-Universität zu Berlin, Freie Universitat Berlin, Charitéplatz 1, 10117, Berlin, Germany
| |
Collapse
|
6
|
Varada S, Lacson R, Raja AS, Ip IK, Schneider L, Osterbur D, Bain P, Vetrano N, Cellini J, Mita C, Coletti M, Whelan J, Khorasani R. Characteristics of knowledge content in a curated online evidence library. J Am Med Inform Assoc 2019; 25:507-514. [PMID: 29092054 DOI: 10.1093/jamia/ocx092] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2017] [Accepted: 08/09/2017] [Indexed: 11/12/2022] Open
Abstract
Objective To describe types of recommendations represented in a curated online evidence library, report on the quality of evidence-based recommendations pertaining to diagnostic imaging exams, and assess underlying knowledge representation. Materials and Methods The evidence library is populated with clinical decision rules, professional society guidelines, and locally developed best practice guidelines. Individual recommendations were graded based on a standard methodology and compared using chi-square test. Strength of evidence ranged from grade 1 (systematic review) through grade 5 (recommendations based on expert opinion). Finally, variations in the underlying representation of these recommendations were identified. Results The library contains 546 individual imaging-related recommendations. Only 15% (16/106) of recommendations from clinical decision rules were grade 5 vs 83% (526/636) from professional society practice guidelines and local best practice guidelines that cited grade 5 studies (P < .0001). Minor head trauma, pulmonary embolism, and appendicitis were topic areas supported by the highest quality of evidence. Three main variations in underlying representations of recommendations were "single-decision," "branching," and "score-based." Discussion Most recommendations were grade 5, largely because studies to test and validate many recommendations were absent. Recommendation types vary in amount and complexity and, accordingly, the structure and syntax of statements they generate. However, they can be represented in single-decision, branching, and score-based representations. Conclusion In a curated evidence library with graded imaging-based recommendations, evidence quality varied widely, with decision rules providing the highest-quality recommendations. The library may be helpful in highlighting evidence gaps, comparing recommendations from varied sources on similar clinical topics, and prioritizing imaging recommendations to inform clinical decision support implementation.
Collapse
Affiliation(s)
- Sowmya Varada
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA
| | - Ronilda Lacson
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| | - Ali S Raja
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA.,Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, USA
| | - Ivan K Ip
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| | - Louise Schneider
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| | - David Osterbur
- Harvard Medical School, Boston, MA, USA.,Countway Library of Medicine, Boston, MA, USA
| | - Paul Bain
- Harvard Medical School, Boston, MA, USA.,Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, USA
| | - Nicole Vetrano
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA
| | - Jacqueline Cellini
- Harvard Medical School, Boston, MA, USA.,Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, USA
| | - Carol Mita
- Harvard Medical School, Boston, MA, USA.,Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA, USA
| | - Margaret Coletti
- Agoos Medical Library/Knowledge Services, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Julia Whelan
- Agoos Medical Library/Knowledge Services, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Ramin Khorasani
- Center for Evidence-Based Imaging, Department of Radiology, Brigham and Women's Hospital, Boston, MA, USA.,Harvard Medical School, Boston, MA, USA
| |
Collapse
|
7
|
Powell AC, Rogstad TL, Winchester DE, Shanser JD, Long JW, Deshmukh UU, Rao VM. Discordance in Clinical Recommendations Regarding the Use of Imaging. Am J Med Qual 2019; 35:117-124. [PMID: 31113208 DOI: 10.1177/1062860619851561] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
Abstract
As physicians strive to provide evidence-based care, challenges arise if different entities disseminate divergent Appropriate Use Criteria (AUC) or clinical guidelines on the same topic. To characterize these challenges in one field, this study reviews the literature on comparisons of clinical recommendations regarding medical imaging. The PubMed database was searched for the years 2013-2018 for studies describing discordance among clinical recommendations regarding the performance of imaging. Of the 406 articles identified, 15 met the selection criteria: 8 qualitative and 7 quantitative. Reasons for discordance varied, with lack of evidence often cited. Quantitative studies often found that different decisions would be reached depending on the clinical recommendation followed. Nonetheless, quantitative studies also tended not to consider one set of recommendations superior to another. The findings of this review might help clinicians seek guidance more thoughtfully and could inform use of guidelines and AUC for quality improvement and clinical decision support.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Vijay M Rao
- Thomas Jefferson University, Philadelphia, PA
| |
Collapse
|
8
|
Odigie E, Lacson R, Raja A, Osterbur D, Ip I, Schneider L, Khorasani R. Fast Healthcare Interoperability Resources, Clinical Quality Language, and Systematized Nomenclature of Medicine-Clinical Terms in Representing Clinical Evidence Logic Statements for the Use of Imaging Procedures: Descriptive Study. JMIR Med Inform 2019; 7:e13590. [PMID: 31094359 PMCID: PMC6535979 DOI: 10.2196/13590] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2019] [Revised: 03/21/2019] [Accepted: 03/24/2019] [Indexed: 12/21/2022] Open
Abstract
Background Evidence-based guidelines and recommendations can be transformed into “If-Then” Clinical Evidence Logic Statements (CELS). Imaging-related CELS were represented in standardized formats in the Harvard Medical School Library of Evidence (HLE). Objective We aimed to (1) describe the representation of CELS using established Systematized Nomenclature of Medicine—Clinical Terms (SNOMED CT), Clinical Quality Language (CQL), and Fast Healthcare Interoperability Resources (FHIR) standards and (2) assess the limitations of using these standards to represent imaging-related CELS. Methods This study was exempt from review by the Institutional Review Board as it involved no human subjects. Imaging-related clinical recommendations were extracted from evidence sources and translated into CELS. The clinical terminologies of CELS were represented using SNOMED CT and the condition-action logic was represented in CQL and FHIR. Numbers of fully and partially represented CELS were tallied. Results A total of 765 CELS were represented in the HLE as of December 2018. We were able to fully represent 137 of 765 (17.9%) CELS using SNOMED CT, CQL, and FHIR. We were able to represent terms using SNOMED CT in the temporal component for action (“Then”) statements in CQL and FHIR in 755 of 765 (98.7%) CELS. Conclusions CELS were represented as shareable clinical decision support (CDS) knowledge artifacts using existing standards—SNOMED CT, FHIR, and CQL—to promote and accelerate adoption of evidence-based practice. Limitations to standardization persist, which could be minimized with an add-on set of standard terms and value sets and by adding time frames to the CQL framework.
Collapse
Affiliation(s)
- Eseosa Odigie
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States
| | - Ronilda Lacson
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Harvard Medical School, Boston, MA, United States
| | - Ali Raja
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Harvard Medical School, Boston, MA, United States
| | - David Osterbur
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Countway Medical Library, Harvard Medical School, Boston, MA, United States
| | - Ivan Ip
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Harvard Medical School, Boston, MA, United States
| | - Louise Schneider
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Harvard Medical School, Boston, MA, United States
| | - Ramin Khorasani
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Brookline, MA, United States.,Harvard Medical School, Boston, MA, United States
| |
Collapse
|
9
|
Raja AS, Ip IK, Cochon L, Pourjabbar S, Yun BJ, Schuur JD, Khorasani R. Will publishing evidence-based guidelines for low back pain imaging decrease imaging use? Am J Emerg Med 2019; 37:545-546. [DOI: 10.1016/j.ajem.2018.07.039] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2018] [Accepted: 07/18/2018] [Indexed: 11/17/2022] Open
|
10
|
Raja AS, Pourjabbar S, Ip IK, Baugh CW, Sodickson AD, O'Leary M, Khorasani R. Impact of a Health Information Technology–Enabled Appropriate Use Criterion on Utilization of Emergency Department CT for Renal Colic. AJR Am J Roentgenol 2019; 212:142-145. [DOI: 10.2214/ajr.18.19966] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/30/2023]
Affiliation(s)
- Ali S. Raja
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115
- Department of Radiology, Brigham and Women's Hospital, Boston, MA
- Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA
- Harvard Medical School, Boston, MA
| | - Sarvenaz Pourjabbar
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115
| | - Ivan K. Ip
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115
- Department of Radiology, Brigham and Women's Hospital, Boston, MA
- Harvard Medical School, Boston, MA
- Department of Medicine, Brigham and Women's Hospital, Boston, MA
| | - Christopher W. Baugh
- Harvard Medical School, Boston, MA
- Department of Emergency Medicine, Brigham and Women's Hospital, Boston, MA
| | - Aaron D. Sodickson
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115
- Department of Radiology, Brigham and Women's Hospital, Boston, MA
- Harvard Medical School, Boston, MA
| | - Michael O'Leary
- Harvard Medical School, Boston, MA
- Department of Urology, Brigham and Women's Hospital, Boston, MA
| | - Ramin Khorasani
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, 75 Francis St, Boston, MA 02115
- Department of Radiology, Brigham and Women's Hospital, Boston, MA
- Harvard Medical School, Boston, MA
| |
Collapse
|
11
|
Medicare Imaging Demonstration: Assessing Attributes of Appropriate Use Criteria and Their Influence on Ordering Behavior. AJR Am J Roentgenol 2017; 208:1051-1057. [PMID: 28267371 DOI: 10.2214/ajr.16.17169] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
OBJECTIVE Persistent concern exists about the variable and possibly inappropriate utilization of high-cost imaging tests. The purpose of this study is to assess the influence of appropriate use criteria attributes on altering ambulatory imaging orders deemed inappropriate. MATERIALS AND METHODS This secondary analysis included Medicare Imaging Demonstration data collected from three health care systems in 2011-2013 via the use of clinical decision support (CDS) during ambulatory imaging order entry. The CDS system captured whether orders were inappropriate per the appropriate use criteria of professional societies and provided advice during the intervention period. For orders deemed inappropriate, we assessed the impact of the availability of alternative test recommendations, conflicts with local best practices, and the strength of evidence for appropriate use criteria on the primary outcome of cancellation or modification of inappropriate orders. Expert review determined conflicts with local best practices for 250 recommendations for abdominal and thoracic CT orders. Strength of evidence was assessed for the 15 most commonly triggered recommendations that were deemed inappropriate. A chi-square test was used for univariate analysis. RESULTS A total of 1691 of 63,222 imaging test orders (2.7%) were deemed inappropriate during the intervention period; this amount decreased from 364 of 11,675 test orders (3.1%) in the baseline period (p < 0.00001). Of 270 inappropriate recommendations with alternative test recommendations, 28 (10.4%) were modified, compared with four of 1024 inappropriate recommendations without alternatives (0.4%) (p < 0.0001). Seventy-eight of 250 recommendations (31%) conflicted with local best practices, but only six of 69 inappropriate recommendations (9%) conflicted (p < 0.001). No inappropriate recommendations that conflicted with local best practices were modified. All 15 commonly triggered recommendations had an Oxford Centre for Evidence-Based Medicine level of evidence of 5 (i.e., expert opinion). CONCLUSION Orders for imaging tests that were deemed inappropriate were modified infrequently, more often with alternative recommendations present and only for appropriate use criteria consistent with local best practices.
Collapse
|
12
|
Yan Z, Lacson R, Ip I, Valtchinov V, Raja A, Osterbur D, Khorasani R. Evaluating Terminologies to Enable Imaging-Related Decision Rule Sharing. AMIA ... ANNUAL SYMPOSIUM PROCEEDINGS. AMIA SYMPOSIUM 2017; 2016:2082-2089. [PMID: 28269968 PMCID: PMC5333322] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
Purpose: Clinical decision support tools provide recommendations based on decision rules. A fundamental challenge regarding decision rule-sharing involves inadequate expression using standard terminology. We aimed to evaluate the coverage of three standard terminologies for mapping imaging-related decision rules. Methods: 50 decision rules, randomly selected from an existing library, were mapped to Systemized Nomenclature of Medicine (SNOMED CT), Radiology Lexicon (RadLex) and International Classification of Disease (ICD-10-CM). Decision rule attributes and values were mapped to unique concepts, obtaining the best possible coverage with the fewest concepts. Manual and automated mapping using Clinical Text Analysis and Knowledge Extraction System (cTAKES) were performed. Results: Using manual mapping, SNOMED CT provided the greatest concept coverage (83%), compared to RadLex (36%) and ICD-10-CM (8%) (p<0.0001). Combined mapping had 86% concept coverage. Automated mapping achieved 85% mapping coverage vs. 94% with manual mapping (p<0.001). Conclusion: Although some gaps remain, standard terminologies provide ample coverage for mapping imaging- related evidence.
Collapse
Affiliation(s)
- Zihao Yan
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Harvard Medical School, Boston, MA
| | - Ronilda Lacson
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Harvard Medical School, Boston, MA
| | - Ivan Ip
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Department of Medicine, Brigham and Women's Hospital, MA; Harvard Medical School, Boston, MA
| | - Vladimir Valtchinov
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Harvard Medical School, Boston, MA
| | - Ali Raja
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Department of Emergency Medicine, Massachusetts General Hospital, Boston, MA; Harvard Medical School, Boston, MA
| | - David Osterbur
- Countway Medical Library, Boston, MA; Harvard Medical School, Boston, MA
| | - Ramin Khorasani
- Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, MA; Department of Radiology, Brigham and Women's Hospital, MA; Harvard Medical School, Boston, MA
| |
Collapse
|