|
1
|
Stead WW, Miller RA, Ohno-Machado L, Bakken S. JAMIA at 30: looking back and forward. J Am Med Inform Assoc 2023; 31:1-9. [PMID: 38134400 PMCID: PMC10746314 DOI: 10.1093/jamia/ocad215] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 10/30/2023] [Indexed: 12/24/2023] Open
Affiliation(s)
- William W Stead
- Department of Biomedical Informatics, Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States
| | - Randolph A Miller
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Alexandria, VA 37232, United States
| | - Lucila Ohno-Machado
- Section of Biomedical Informatics & Data Science, Yale University School of Medicine, New Haven, CT 06520, United States
| | - Suzanne Bakken
- School of Nursing, Department of Biomedical Informatics, Data Science Institute, Columbia University, New York, NY 10032, United States
| |
Collapse
|
|
2
|
Kukhareva PV, Weir C, Fiol GD, Aarons GA, Taft TY, Schlechter CR, Reese TJ, Curran RL, Nanjo C, Borbolla D, Staes CJ, Morgan KL, Kramer HS, Stipelman CH, Shakib JH, Flynn MC, Kawamoto K. Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation. J Biomed Inform 2022; 127:104014. [PMID: 35167977 PMCID: PMC8959015 DOI: 10.1016/j.jbi.2022.104014] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2021] [Revised: 11/16/2021] [Accepted: 02/02/2022] [Indexed: 11/28/2022]
Abstract
OBJECTIVE Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.
Collapse
Affiliation(s)
- Polina V. Kukhareva
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Charlene Weir
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Gregory A. Aarons
- Department of Psychiatry, UC San Diego ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA
| | - Teresa Y. Taft
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Chelsey R. Schlechter
- Department of Population Health Sciences, Center for Health Outcomes and Population Equity, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA
| | - Thomas J. Reese
- Department of Biomedical Informatics, Vanderbilt University, Nashville, TN, USA
| | - Rebecca L. Curran
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Claude Nanjo
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.
| | - Damian Borbolla
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.
| | | | - Keaton L. Morgan
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - Heidi S. Kramer
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | | | - Julie H. Shakib
- Department of Pediatrics, University of Utah, Salt Lake City, UT, USA
| | - Michael C. Flynn
- Department of Family & Preventive Medicine, University of Utah, Salt Lake City, UT, USA
| | - Kensaku Kawamoto
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.
| |
Collapse
|