1
|
Van Woensel W, Tu SW, Michalowski W, Sibte Raza Abidi S, Abidi S, Alonso JR, Bottrighi A, Carrier M, Edry R, Hochberg I, Rao M, Kingwell S, Kogan A, Marcos M, Martínez Salvador B, Michalowski M, Piovesan L, Riaño D, Terenziani P, Wilk S, Peleg M. A Community-of-Practice-based Evaluation Methodology for Knowledge Intensive Computational Methods and its Application to Multimorbidity Decision Support. J Biomed Inform 2023; 142:104395. [PMID: 37201618 DOI: 10.1016/j.jbi.2023.104395] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Revised: 04/25/2023] [Accepted: 05/15/2023] [Indexed: 05/20/2023]
Abstract
OBJECTIVE The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support.Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).
Collapse
Affiliation(s)
| | - Samson W Tu
- Center for BioMedical Informatics Research, Stanford University, Stanford, CA, 94305, USA
| | | | | | - Samina Abidi
- Faculty of Computer Science, Dalhousie University, Halifax, Canada
| | | | | | | | - Ruth Edry
- Bruce Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa, Israel; Rambam Medical Center, Haifa, Israel
| | - Irit Hochberg
- Bruce Rappaport Faculty of Medicine, Technion - Israel Institute of Technology, Haifa, Israel; Rambam Medical Center, Haifa, Israel
| | - Malvika Rao
- Telfer School of Management, University of Ottawa, Ottawa, ON, Canada
| | | | - Alexandra Kogan
- Department of Information Systems, University of Haifa, Haifa, Israel, 3498838
| | - Mar Marcos
- Universitat Jaume I, Castelló de la Plana, Spain
| | | | | | - Luca Piovesan
- DISIT, Università del Piemonte Orientale, Alessandria, Italy
| | - David Riaño
- Universitat Rovira i Virgili, Tarragona, Spain; Institut d'Investigació Sanitària Pere Virgili, Tarragona, Spain
| | | | - Szymon Wilk
- Institute of Computing Science, Poznan University of Technology, Poznan, Poland
| | - Mor Peleg
- Department of Information Systems, University of Haifa, Haifa, Israel, 3498838
| |
Collapse
|
2
|
López B, Raya O, Baykova E, Saez M, Rigau D, Cunill R, Mayoral S, Carrion C, Serrano D, Castells X. APPRAISE-RS: Automated, updated, participatory, and personalized treatment recommender systems based on GRADE methodology. Heliyon 2023; 9:e13074. [PMID: 36798764 PMCID: PMC9925880 DOI: 10.1016/j.heliyon.2023.e13074] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2022] [Revised: 01/04/2023] [Accepted: 01/16/2023] [Indexed: 01/26/2023] Open
Abstract
Purpose Clinical practice guidelines (CPGs) have become fundamental tools for evidence-based medicine (EBM). However, CPG suffer from several limitations, including obsolescence, lack of applicability to many patients, and limited patient participation. This paper presents APPRAISE-RS, which is a methodology that we developed to overcome these limitations by automating, extending, and iterating the methodology that is most commonly used for building CPGs: the GRADE methodology. Method APPRAISE-RS relies on updated information from clinical studies and adapts and automates the GRADE methodology to generate treatment recommendations. APPRAISE-RS provides personalized recommendations because they are based on the patient's individual characteristics. Moreover, both patients and clinicians express their personal preferences for treatment outcomes which are considered when making the recommendation (participatory). Rule-based system approaches are used to manage heuristic knowledge. Results APPRAISE-RS has been implemented for attention deficit hyperactivity disorder (ADHD) and tested experimentally on 28 simulated patients. The resulting recommender system (APPRAISE-RS/TDApp) shows a higher degree of treatment personalization and patient participation than CPGs, while recommending the most frequent interventions in the largest body of evidence in the literature (EBM). Moreover, a comparison of the results with four blinded psychiatrist prescriptions supports the validation of the proposal. Conclusions APPRAISE-RS is a valid methodology to build recommender systems that manage updated, personalized and participatory recommendations, which, in the case of ADHD includes at least one intervention that is identical or very similar to other drugs prescribed by psychiatrists.
Collapse
Affiliation(s)
- Beatriz López
- Control Engineering and Intelligent Systems (eXiT), University of Girona, Spain,Corresponding author.
| | - Oscar Raya
- Control Engineering and Intelligent Systems (eXiT), University of Girona, Spain
| | | | - Marc Saez
- Research Group on Statistics, Econometrics and Health, University of Girona, Spain,CIBER of Epidemiology and Public Health (CIBERESP), Madrid, Spain
| | | | - Ruth Cunill
- Sant Joan de Deu-Numancia Health Park, Barcelona, Spain
| | | | - Carme Carrion
- Health Lab Research Group, Universitat Oberta de Catalunya, Spain
| | | | - Xavier Castells
- TransLab Research Group, Dept. of Medical Sciences, University of Girona, Spain
| |
Collapse
|
3
|
Bottrighi A, Piovesan L, Terenziani P. Supporting physicians in the coordination of distributed execution of CIGs to treat comorbid patients. Artif Intell Med 2023; 135:102472. [PMID: 36628779 DOI: 10.1016/j.artmed.2022.102472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2022] [Revised: 11/25/2022] [Accepted: 11/30/2022] [Indexed: 12/12/2022]
Abstract
Clinical Practice Guidelines (CPGs) encode the "best" medical practices to treat patients affected by a specific disease and are widely used in the medical practice. Starting from the '90s', several Computer-Interpretable Guideline (CIG) systems have been devised to provide physicians with CPG-based decision support. CPGs (and CIGs) are devoted to provide evidence-based recommendations for one specific disease. In order to support the treatment of patients affected by multiple diseases (i.e., comorbid patients), challenging additional tasks have to be addressed, such as (i) the detection of the interactions between CIG actions, (ii) their management, and, finally, (iii) the "merge" or conciliation of the CIGs. Several CIG approaches have been recently extended in order to face (at least one of) such challenging problems, and one of them is GLARE. However, besides the solutions to tasks (i)-(iii) above, the "run-time" support to physicians treating a comorbid patient requires additional capabilities, to support the distribution of the management of interactions and of the execution of CIGs among different physicians. In this paper, we propose a general framework, based on GLARE and GLARE-SSCPM, to provide such additional capabilities.
Collapse
Affiliation(s)
- Alessio Bottrighi
- DISIT, Università del Piemonte Orientale, Viale Teresa Michel 11, Alessandria, Italy; AI@UPO, Università del Piemonte Orientale, Vercelli, Italy.
| | - Luca Piovesan
- DISIT, Università del Piemonte Orientale, Viale Teresa Michel 11, Alessandria, Italy; AI@UPO, Università del Piemonte Orientale, Vercelli, Italy.
| | - Paolo Terenziani
- DISIT, Università del Piemonte Orientale, Viale Teresa Michel 11, Alessandria, Italy; AI@UPO, Università del Piemonte Orientale, Vercelli, Italy.
| |
Collapse
|