1
|
Zhang Y, Callaghan-Koru JA, Koru G. The challenges and opportunities of continuous data quality improvement for healthcare administration data. JAMIA Open 2024; 7:ooae058. [PMID: 39091510 PMCID: PMC11293638 DOI: 10.1093/jamiaopen/ooae058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 05/12/2024] [Accepted: 06/18/2024] [Indexed: 08/04/2024] Open
Abstract
Background Various data quality issues have prevented healthcare administration data from being fully utilized when dealing with problems ranging from COVID-19 contact tracing to controlling healthcare costs. Objectives (i) Describe the currently adopted approaches and practices for understanding and improving the quality of healthcare administration data. (ii) Explore the challenges and opportunities to achieve continuous quality improvement for such data. Materials and Methods We used a qualitative approach to obtain rich contextual data through semi-structured interviews conducted at a state health agency regarding Medicaid claims and reimbursement data. We interviewed all data stewards knowledgeable about the data quality issues experienced at the agency. The qualitative data were analyzed using the Framework method. Results Sixteen themes emerged from our analysis, collected under 4 categories: (i) Defect characteristics: Data defects showed variability, frequently remained obscure, and led to negative outcomes. Detecting and resolving them was often difficult, and the work required often exceeded the organizational boundaries. (ii) Current process and people issues: The agency adopted primarily ad-hoc, manual approaches to resolving data quality problems leading to work frustration. (iii) Challenges: Communication and lack of knowledge about legacy software systems and the data maintained in them constituted challenges, followed by different standards used by various organizations and vendors, and data verification difficulties. (iv) Opportunities: Training, tool support, and standardization of data definitions emerged as immediate opportunities to improve data quality. Conclusions Our results can be useful to similar agencies on their journey toward becoming learning health organizations leveraging data assets effectively and efficiently.
Collapse
Affiliation(s)
- Yili Zhang
- Innovation Center for Biomedical Informatics, Georgetown University, Washington, DC 20007, United States
| | - Jennifer A Callaghan-Koru
- Department of Internal Medicine, University of Arkansas for Medical Sciences, Fayetteville, AR 72703, United States
| | - Güneş Koru
- Departments of Health Policy and Management & Biomedical Informatics, University of Arkansas for Medical Sciences, Fayetteville, AR 72703, United States
| |
Collapse
|
2
|
Sáez C, Ferri P, García-Gómez JM. Resilient Artificial Intelligence in Health: Synthesis and Research Agenda Toward Next-Generation Trustworthy Clinical Decision Support. J Med Internet Res 2024; 26:e50295. [PMID: 38941134 PMCID: PMC11245653 DOI: 10.2196/50295] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Revised: 04/16/2024] [Accepted: 05/18/2024] [Indexed: 06/29/2024] Open
Abstract
Artificial intelligence (AI)-based clinical decision support systems are gaining momentum by relying on a greater volume and variety of secondary use data. However, the uncertainty, variability, and biases in real-world data environments still pose significant challenges to the development of health AI, its routine clinical use, and its regulatory frameworks. Health AI should be resilient against real-world environments throughout its lifecycle, including the training and prediction phases and maintenance during production, and health AI regulations should evolve accordingly. Data quality issues, variability over time or across sites, information uncertainty, human-computer interaction, and fundamental rights assurance are among the most relevant challenges. If health AI is not designed resiliently with regard to these real-world data effects, potentially biased data-driven medical decisions can risk the safety and fundamental rights of millions of people. In this viewpoint, we review the challenges, requirements, and methods for resilient AI in health and provide a research framework to improve the trustworthiness of next-generation AI-based clinical decision support.
Collapse
Affiliation(s)
- Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| | - Pablo Ferri
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| | - Juan M García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
3
|
Ferri P, Romero-Garcia N, Badenes R, Lora-Pablos D, Morales TG, Gómez de la Cámara A, García-Gómez JM, Sáez C. Extremely missing numerical data in Electronic Health Records for machine learning can be managed through simple imputation methods considering informative missingness: A comparative of solutions in a COVID-19 mortality case study. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 242:107803. [PMID: 37703700 DOI: 10.1016/j.cmpb.2023.107803] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 08/28/2023] [Accepted: 09/05/2023] [Indexed: 09/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Reusing Electronic Health Records (EHRs) for Machine Learning (ML) leads on many occasions to extremely incomplete and sparse tabular datasets, which can hinder the model development processes and limit their performance and generalization. In this study, we aimed to characterize the most effective data imputation techniques and ML models for dealing with highly missing numerical data in EHRs, in the case where only a very limited number of data are complete, as opposed to the usual case of having a reduced number of missing values. METHODS We used a case study including full blood count laboratory data, demographic and survival data in the context of COVID-19 hospital admissions and evaluated 30 processing pipelines combining imputation methods with ML classifiers. The imputation methods included missing mask, translation and encoding, mean imputation, k-nearest neighbors' imputation, Bayesian ridge regression imputation and generative adversarial imputation networks. The classifiers included k-nearest neighbors, logistic regression, random forest, gradient boosting and deep multilayer perceptron. RESULTS Our results suggest that in the presence of highly missing data, combining translation and encoding imputation-which considers informative missingness-with tree ensemble classifiers-random forest and gradient boosting-is a sensible choice when aiming to maximize performance, in terms of area under curve. CONCLUSIONS Based on our findings, we recommend the consideration of this imputer-classifier configuration when constructing models in the presence of extremely incomplete numerical data in EHR.
Collapse
Affiliation(s)
- Pablo Ferri
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, Spain.
| | | | - Rafael Badenes
- Departament de Cirugia, Universitat de València, Spain; Instituto INCLIVA, Hospital Clínico Universitario de Valencia, Spain; Department Anesthesiology, Surgical-Trauma Intensive Care and Pain Clinic, Hospital Clínic Universitari, Valencia, Spain
| | - David Lora-Pablos
- Instituto de Investigación imas12, Hospital 12 de Octubre, Madrid, Spain; Facultad de Estudios Estadísticos, Universidad Complutense de Madrid, Spain
| | | | | | - Juan M García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, Spain
| | - Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, Spain
| |
Collapse
|
4
|
Souza J, Caballero I, Vasco Santos J, Fernandes Lobo M, Pinto A, Viana J, Sáez C, Lopes F, Freitas A. Multisource and temporal variability in Portuguese hospital administrative datasets: data quality implications. J Biomed Inform 2022; 136:104242. [DOI: 10.1016/j.jbi.2022.104242] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2021] [Revised: 08/18/2022] [Accepted: 11/06/2022] [Indexed: 11/13/2022]
|
5
|
Zhou L, Romero N, Martínez-Miranda J, Conejero JA, García-Gómez JM, Sáez C. Subphenotyping of COVID-19 patients at pre-admission towards anticipated severity stratification: an analysis of 778 692 Mexican patients through an age-sex unbiased meta-clustering technique. JMIR Public Health Surveill 2022; 8:e30032. [PMID: 35144239 PMCID: PMC9098229 DOI: 10.2196/30032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2021] [Revised: 01/29/2022] [Accepted: 02/08/2022] [Indexed: 11/13/2022] Open
Abstract
Background The COVID-19 pandemic has led to an unprecedented global health care challenge for both medical institutions and researchers. Recognizing different COVID-19 subphenotypes—the division of populations of patients into more meaningful subgroups driven by clinical features—and their severity characterization may assist clinicians during the clinical course, the vaccination process, research efforts, the surveillance system, and the allocation of limited resources. Objective We aimed to discover age-sex unbiased COVID-19 patient subphenotypes based on easily available phenotypical data before admission, such as pre-existing comorbidities, lifestyle habits, and demographic features, to study the potential early severity stratification capabilities of the discovered subgroups through characterizing their severity patterns, including prognostic, intensive care unit (ICU), and morbimortality outcomes. Methods We used the Mexican Government COVID-19 open data, including 778,692 SARS-CoV-2 population-based patient-level data as of September 2020. We applied a meta-clustering technique that consists of a 2-stage clustering approach combining dimensionality reduction (ie, principal components analysis and multiple correspondence analysis) and hierarchical clustering using the Ward minimum variance method with Euclidean squared distance. Results In the independent age-sex clustering analyses, 56 clusters supported 11 clinically distinguishable meta-clusters (MCs). MCs 1-3 showed high recovery rates (90.27%-95.22%), including healthy patients of all ages, children with comorbidities and priority in receiving medical resources (ie, higher rates of hospitalization, intubation, and ICU admission) compared with other adult subgroups that have similar conditions, and young obese smokers. MCs 4-5 showed moderate recovery rates (81.30%-82.81%), including patients with hypertension or diabetes of all ages and obese patients with pneumonia, hypertension, and diabetes. MCs 6-11 showed low recovery rates (53.96%-66.94%), including immunosuppressed patients with high comorbidity rates, patients with chronic kidney disease with a poor survival length and probability of recovery, older smokers with chronic obstructive pulmonary disease, older adults with severe diabetes and hypertension, and the oldest obese smokers with chronic obstructive pulmonary disease and mild cardiovascular disease. Group outcomes conformed to the recent literature on dedicated age-sex groups. Mexican states and several types of clinical institutions showed relevant heterogeneity regarding severity, potentially linked to socioeconomic or health inequalities. Conclusions The proposed 2-stage cluster analysis methodology produced a discriminative characterization of the sample and explainability over age and sex. These results can potentially help in understanding the clinical patient and their stratification for automated early triage before further tests and laboratory results are available and even in locations where additional tests are not available or to help decide resource allocation among vulnerable subgroups such as to prioritize vaccination or treatments.
Collapse
Affiliation(s)
- Lexin Zhou
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Valencia, ES
| | - Nekane Romero
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Valencia, ES
| | - Juan Martínez-Miranda
- CONACyT - Centro de Investigación Científica y de Educación Superior de Ensenada - CICESE-UT3, Ensenada, MX
| | - J Alberto Conejero
- Instituto Universitario de Matemática Pura y Aplicada (IUMPA), Universitat Politècnica de València, Valencia, ES
| | - Juan M García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Valencia, ES
| | - Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Camino de Vera s/n, Valencia 46022, España, Valencia, ES
| |
Collapse
|
6
|
Tute E, Ganapathy N, Wulff A. A data driven learning approach for the assessment of data quality. BMC Med Inform Decis Mak 2021; 21:302. [PMID: 34724930 PMCID: PMC8561935 DOI: 10.1186/s12911-021-01656-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Accepted: 10/14/2021] [Indexed: 11/16/2022] Open
Abstract
Background Data quality assessment is important but complex and task dependent. Identifying suitable measurement methods and reference ranges for assessing their results is challenging. Manually inspecting the measurement results and current data driven approaches for learning which results indicate data quality issues have considerable limitations, e.g. to identify task dependent thresholds for measurement results that indicate data quality issues. Objectives To explore the applicability and potential benefits of a data driven approach to learn task dependent knowledge about suitable measurement methods and assessment of their results. Such knowledge could be useful for others to determine whether a local data stock is suitable for a given task. Methods We started by creating artificial data with previously defined data quality issues and applied a set of generic measurement methods on this data (e.g. a method to count the number of values in a certain variable or the mean value of the values). We trained decision trees on exported measurement methods’ results and corresponding outcome data (data that indicated the data’s suitability for a use case). For evaluation, we derived rules for potential measurement methods and reference values from the decision trees and compared these regarding their coverage of the true data quality issues artificially created in the dataset. Three researchers independently derived these rules. One with knowledge about present data quality issues and two without. Results Our self-trained decision trees were able to indicate rules for 12 of 19 previously defined data quality issues. Learned knowledge about measurement methods and their assessment was complementary to manual interpretation of measurement methods’ results. Conclusions Our data driven approach derives sensible knowledge for task dependent data quality assessment and complements other current approaches. Based on labeled measurement methods’ results as training data, our approach successfully suggested applicable rules for checking data quality characteristics that determine whether a dataset is suitable for a given task. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-021-01656-x.
Collapse
Affiliation(s)
- Erik Tute
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Carl-Neuberg-Str. 1, 30625, Hannover, Germany.
| | - Nagarajan Ganapathy
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Carl-Neuberg-Str. 1, 30625, Hannover, Germany
| | - Antje Wulff
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Carl-Neuberg-Str. 1, 30625, Hannover, Germany
| |
Collapse
|
7
|
Aerts H, Kalra D, Sáez C, Ramírez-Anguita JM, Mayer MA, Garcia-Gomez JM, Durà-Hernández M, Thienpont G, Coorevits P. Quality of Hospital Electronic Health Record (EHR) Data Based on the International Consortium for Health Outcomes Measurement (ICHOM) in Heart Failure: Pilot Data Quality Assessment Study. JMIR Med Inform 2021; 9:e27842. [PMID: 34346902 PMCID: PMC8374665 DOI: 10.2196/27842] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2021] [Revised: 05/30/2021] [Accepted: 06/05/2021] [Indexed: 12/21/2022] Open
Abstract
Background There is increasing recognition that health care providers need to focus attention, and be judged against, the impact they have on the health outcomes experienced by patients. The measurement of health outcomes as a routine part of clinical documentation is probably the only scalable way of collecting outcomes evidence, since secondary data collection is expensive and error-prone. However, there is uncertainty about whether routinely collected clinical data within electronic health record (EHR) systems includes the data most relevant to measuring and comparing outcomes and if those items are collected to a good enough data quality to be relied upon for outcomes assessment, since several studies have pointed out significant issues regarding EHR data availability and quality. Objective In this paper, we first describe a practical approach to data quality assessment of health outcomes, based on a literature review of existing frameworks for quality assessment of health data and multistakeholder consultation. Adopting this approach, we performed a pilot study on a subset of 21 International Consortium for Health Outcomes Measurement (ICHOM) outcomes data items from patients with congestive heart failure. Methods All available registries compatible with the diagnosis of heart failure within an EHR data repository of a general hospital (142,345 visits and 12,503 patients) were extracted and mapped to the ICHOM format. We focused our pilot assessment on 5 commonly used data quality dimensions: completeness, correctness, consistency, uniqueness, and temporal stability. Results We found high scores (>95%) for the consistency, completeness, and uniqueness dimensions. Temporal stability analyses showed some changes over time in the reported use of medication to treat heart failure, as well as in the recording of past medical conditions. Finally, the investigation of data correctness suggested several issues concerning the characterization of missing data values. Many of these issues appear to be introduced while mapping the IMASIS-2 relational database contents to the ICHOM format, as the latter requires a level of detail that is not explicitly available in the coded data of an EHR. Conclusions Overall, results of this pilot study revealed good data quality for the subset of heart failure outcomes collected at the Hospital del Mar. Nevertheless, some important data errors were identified that were caused by fundamentally different data collection practices in routine clinical care versus research, for which the ICHOM standard set was originally developed. To truly examine to what extent hospitals today are able to routinely collect the evidence of their success in achieving good health outcomes, future research would benefit from performing more extensive data quality assessments, including all data items from the ICHOM standards set and across multiple hospitals.
Collapse
Affiliation(s)
- Hannelore Aerts
- Medical Informatics and Statistics Unit, Department of Public Health and Primary Care, Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium.,The European Institute for Innovation through Health Data (i~HD), Ghent, Belgium
| | - Dipak Kalra
- Medical Informatics and Statistics Unit, Department of Public Health and Primary Care, Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium.,The European Institute for Innovation through Health Data (i~HD), Ghent, Belgium
| | - Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| | - Juan Manuel Ramírez-Anguita
- Research Programme on Biomedical Informatics, Hospital del Mar Medical Research Institute and Universitat Pompeu Fabra, Barcelona, Spain
| | - Miguel-Angel Mayer
- Research Programme on Biomedical Informatics, Hospital del Mar Medical Research Institute and Universitat Pompeu Fabra, Barcelona, Spain
| | - Juan M Garcia-Gomez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| | - Marta Durà-Hernández
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Valencia, Spain
| | - Geert Thienpont
- The European Institute for Innovation through Health Data (i~HD), Ghent, Belgium.,Research in Advanced Medical Informatics and Telematics (RAMIT), Ghent, Belgium
| | - Pascal Coorevits
- Medical Informatics and Statistics Unit, Department of Public Health and Primary Care, Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium
| |
Collapse
|
8
|
Derakhshan P, Azadmanjir Z, Naghdi K, Habibi Arejan R, Safdarian M, Zarei MR, Jazayeri SB, Sharif-Alhoseini M, Arab Kheradmand J, Amirjamshidi A, Ghodsi Z, Faghih Jooybari M, Mohammadzadeh M, Khazaeipour Z, Abdollah Zadegan S, Abedi A, Oreilly G, Noonan V, Benzel EC, Vaccaro AR, Sadeghian F, Rahimi-Movaghar V. The impact of data quality assurance and control solutions on the completeness, accuracy, and consistency of data in a national spinal cord injury registry of Iran (NSCIR-IR). Spinal Cord Ser Cases 2021; 7:51. [PMID: 34112766 DOI: 10.1038/s41394-020-00358-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2020] [Revised: 09/29/2020] [Accepted: 10/27/2020] [Indexed: 11/09/2022] Open
Abstract
STUDY DESIGN Descriptive study. OBJECTIVE This study aimed to develop and evaluate a systematic arrangement for improvement and monitoring of data quality of the National Spinal Cord (and Column) Injury Registry of Iran (NSCIR-IR)-a multicenter hospital-based registry. SETTING SCI community in Iran. METHODS Quality assurance and quality control were the primary objectives in improving overall quality of data that were considered in designing a paper-based and computerized case report. To prevent incorrect data entry, we implemented several validation algorithms, including 70 semantic rules, 18 syntactic rules, seven temporal rules, and 13 rules for acceptable value range. Qualified and trained staff members were also employed to review and identify any defect, inaccuracy, or inconsistency in the data to improve data quality. A set of functions were implemented in the software to cross-validate, and feedback on data was provided by reviewers and registrars. RESULTS Socio-demographic data items were 100% complete, except for national ID and education level, which were 97% and 92.3% complete, respectively. Completeness of admission data and emergency medical services data were 100% except for arrival and transfer time (99.4%) and oxygen saturation (48.9%). Evaluation of data received from two centers located in Tehran proved to be 100% accurate following validation by quality reviewers. All data was also found to be 100% consistent. CONCLUSIONS This approach to quality assurance and consistency validation proved to be effective. Our solutions resulted in a significant decrease in the number of missing data.
Collapse
Affiliation(s)
- Pegah Derakhshan
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Zahra Azadmanjir
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran.,Department of Health Information Management, School of Allied Medical Sciences, Tehran University of Medical Sciences, Tehran, Iran
| | - Khatereh Naghdi
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Roya Habibi Arejan
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Mahdi Safdarian
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Mohammad Reza Zarei
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Seyed Behzad Jazayeri
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Mahdi Sharif-Alhoseini
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | | | - Abbas Amirjamshidi
- Department of Neurosurgery, Sina Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Zahra Ghodsi
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Morteza Faghih Jooybari
- Department of Neurosurgery, Shariati Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | | | - Zahra Khazaeipour
- Brain and Spinal Injuries Research Center (BASIR), Imam Khomeini Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Shayan Abdollah Zadegan
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Aidin Abedi
- Department of Orthopaedic Surgery, Keck School of Medicine, University of Southern California, Los Angeles, CA, 90033, USA
| | - Gerard Oreilly
- Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, 3800, Australia
| | - Vanessa Noonan
- Rick Hansen Institute, University of British Columbia, Vancouver, BC, V6T 1Z4, Canada
| | - Edward C Benzel
- Department of Neurosurgery, Cleveland Clinic Foundation, Cleveland, OH, 44195, USA
| | - Alexander R Vaccaro
- Department of Orthopaedic Surgery, The Rothman Institute, Thomas Jefferson University, Philadelphia, PA, 19107, USA
| | - Farideh Sadeghian
- Center for Health Related Social and Behavioral Sciences Research, Shahroud University of Medical Sciences, Shahroud, Iran
| | - Vafa Rahimi-Movaghar
- Sina Trauma and Surgery Research Center, Tehran University of Medical Sciences, Tehran, Iran. .,Brain and Spinal Injuries Research Center (BASIR), Imam Khomeini Hospital, Tehran University of Medical Sciences, Tehran, Iran.
| |
Collapse
|
9
|
Deep ensemble multitask classification of emergency medical call incidents combining multimodal data improves emergency medical dispatch. Artif Intell Med 2021; 117:102088. [PMID: 34127234 DOI: 10.1016/j.artmed.2021.102088] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2020] [Revised: 04/19/2021] [Accepted: 05/03/2021] [Indexed: 11/20/2022]
Abstract
The objective of this work was to develop a predictive model to aid non-clinical dispatchers to classify emergency medical call incidents by their life-threatening level (yes/no), admissible response delay (undelayable, minutes, hours, days) and emergency system jurisdiction (emergency system/primary care) in real time. We used a total of 1 244 624 independent incidents from the Valencian emergency medical dispatch service in Spain, compiled in retrospective from 2009 to 2012, including clinical features, demographics, circumstantial factors and free text dispatcher observations. Based on them, we designed and developed DeepEMC2, a deep ensemble multitask model integrating four subnetworks: three specialized to context, clinical and text data, respectively, and another to ensemble the former. The four subnetworks are composed in turn by multi-layer perceptron modules, bidirectional long short-term memory units and a bidirectional encoding representations from transformers module. DeepEMC2 showed a macro F1-score of 0.759 in life-threatening classification, 0.576 in admissible response delay and 0.757 in emergency system jurisdiction. These results show a substantial performance increase of 12.5 %, 17.5 % and 5.1 %, respectively, with respect to the current in-house triage protocol of the Valencian emergency medical dispatch service. Besides, DeepEMC2 significantly outperformed a set of baseline machine learning models, including naive bayes, logistic regression, random forest and gradient boosting (α = 0.05). Hence, DeepEMC2 is able to: 1) capture information present in emergency medical calls not considered by the existing triage protocol, and 2) model complex data dependencies not feasible by the tested baseline models. Likewise, our results suggest that most of this unconsidered information is present in the free text dispatcher observations. To our knowledge, this study describes the first deep learning model undertaking emergency medical call incidents classification. Its adoption in medical dispatch centers would potentially improve emergency dispatch processes, resulting in a positive impact in patient wellbeing and health services sustainability.
Collapse
|
10
|
Catalá ODT, Igual IS, Pérez-Benito FJ, Escrivá DM, Castelló VO, Llobet R, Peréz-Cortés JC. Bias Analysis on Public X-Ray Image Datasets of Pneumonia and COVID-19 Patients. IEEE ACCESS : PRACTICAL INNOVATIONS, OPEN SOLUTIONS 2021; 9:42370-42383. [PMID: 34812384 PMCID: PMC8545228 DOI: 10.1109/access.2021.3065456] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Accepted: 03/07/2021] [Indexed: 05/03/2023]
Abstract
Chest X-ray images are useful for early COVID-19 diagnosis with the advantage that X-ray devices are already available in health centers and images are obtained immediately. Some datasets containing X-ray images with cases (pneumonia or COVID-19) and controls have been made available to develop machine-learning-based methods to aid in diagnosing the disease. However, these datasets are mainly composed of different sources coming from pre-COVID-19 datasets and COVID-19 datasets. Particularly, we have detected a significant bias in some of the released datasets used to train and test diagnostic systems, which might imply that the results published are optimistic and may overestimate the actual predictive capacity of the techniques proposed. In this article, we analyze the existing bias in some commonly used datasets and propose a series of preliminary steps to carry out before the classic machine learning pipeline in order to detect possible biases, to avoid them if possible and to report results that are more representative of the actual predictive power of the methods under analysis.
Collapse
Affiliation(s)
- Omar Del Tejo Catalá
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
| | - Ismael Salvador Igual
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
| | | | - David Millán Escrivá
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
| | - Vicent Ortiz Castelló
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
| | - Rafael Llobet
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
- Department of Computer Systems and Computation (DSIC)Universitat Politècnica de València46022ValenciaSpain
| | - Juan-Carlos Peréz-Cortés
- Instituto Tecnológico de Informática (ITI), Universitat Politècnica de València46022ValenciaSpain
- Department of Computing Engineering (DISCA)Universitat Politècnica de València46022ValenciaSpain
| |
Collapse
|
11
|
Sáez C, Romero N, Conejero JA, García-Gómez JM. Potential limitations in COVID-19 machine learning due to data source variability: A case study in the nCov2019 dataset. J Am Med Inform Assoc 2021; 28:360-364. [PMID: 33027509 PMCID: PMC7797735 DOI: 10.1093/jamia/ocaa258] [Citation(s) in RCA: 27] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2020] [Revised: 09/07/2020] [Accepted: 09/28/2020] [Indexed: 02/02/2023] Open
Abstract
OBJECTIVE The lack of representative coronavirus disease 2019 (COVID-19) data is a bottleneck for reliable and generalizable machine learning. Data sharing is insufficient without data quality, in which source variability plays an important role. We showcase and discuss potential biases from data source variability for COVID-19 machine learning. MATERIALS AND METHODS We used the publicly available nCov2019 dataset, including patient-level data from several countries. We aimed to the discovery and classification of severity subgroups using symptoms and comorbidities. RESULTS Cases from the 2 countries with the highest prevalence were divided into separate subgroups with distinct severity manifestations. This variability can reduce the representativeness of training data with respect the model target populations and increase model complexity at risk of overfitting. CONCLUSIONS Data source variability is a potential contributor to bias in distributed research networks. We call for systematic assessment and reporting of data source variability and data quality in COVID-19 data sharing, as key information for reliable and generalizable machine learning.
Collapse
Affiliation(s)
- Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, España
| | - Nekane Romero
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, España
| | - J Alberto Conejero
- Instituto Universitario de Matemática Pura y Aplicada, Universitat Politécnica de València, Valencia, Spain
| | - Juan M García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, España
| |
Collapse
|
12
|
Sáez C, Gutiérrez-Sacristán A, Kohane I, García-Gómez JM, Avillach P. EHRtemporalVariability: delineating temporal data-set shifts in electronic health records. Gigascience 2020; 9:giaa079. [PMID: 32729900 PMCID: PMC7391413 DOI: 10.1093/gigascience/giaa079] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2019] [Revised: 05/28/2020] [Accepted: 07/03/2020] [Indexed: 11/18/2022] Open
Abstract
BACKGROUND Temporal variability in health-care processes or protocols is intrinsic to medicine. Such variability can potentially introduce dataset shifts, a data quality issue when reusing electronic health records (EHRs) for secondary purposes. Temporal data-set shifts can present as trends, as well as abrupt or seasonal changes in the statistical distributions of data over time. The latter are particularly complicated to address in multimodal and highly coded data. These changes, if not delineated, can harm population and data-driven research, such as machine learning. Given that biomedical research repositories are increasingly being populated with large sets of historical data from EHRs, there is a need for specific software methods to help delineate temporal data-set shifts to ensure reliable data reuse. RESULTS EHRtemporalVariability is an open-source R package and Shiny app designed to explore and identify temporal data-set shifts. EHRtemporalVariability estimates the statistical distributions of coded and numerical data over time; projects their temporal evolution through non-parametric information geometric temporal plots; and enables the exploration of changes in variables through data temporal heat maps. We demonstrate the capability of EHRtemporalVariability to delineate data-set shifts in three impact case studies, one of which is available for reproducibility. CONCLUSIONS EHRtemporalVariability enables the exploration and identification of data-set shifts, contributing to the broad examination and repurposing of large, longitudinal data sets. Our goal is to help ensure reliable data reuse for a wide range of biomedical data users. EHRtemporalVariability is designed for technical users who are programmatically utilizing the R package, as well as users who are not familiar with programming via the Shiny user interface.Availability: https://github.com/hms-dbmi/EHRtemporalVariability/Reproducible vignette: https://cran.r-project.org/web/packages/EHRtemporalVariability/vignettes/EHRtemporalVariability.htmlOnline demo: http://ehrtemporalvariability.upv.es/.
Collapse
Affiliation(s)
- Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, España
- Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
| | | | - Isaac Kohane
- Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
| | - Juan M García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de València, Camino de Vera s/n, Valencia 46022, España
| | - Paul Avillach
- Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts, USA
- Computational Health Informatics Program, Boston Children’s Hospital, Boston, Massachusetts, USA
| |
Collapse
|
13
|
Zhang Y, Koru G. Understanding and detecting defects in healthcare administration data: Toward higher data quality to better support healthcare operations and decisions. J Am Med Inform Assoc 2020; 27:386-395. [PMID: 31841149 DOI: 10.1093/jamia/ocz201] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2019] [Revised: 10/06/2019] [Accepted: 10/26/2019] [Indexed: 01/12/2023] Open
Abstract
OBJECTIVE Development of systematic approaches for understanding and assessing data quality is becoming increasingly important as the volume and utilization of health data steadily increases. In this study, a taxonomy of data defects was developed and utilized when automatically detecting defects to assess Medicaid data quality maintained by one of the states in the United States. MATERIALS AND METHODS There were more than 2.23 million rows and 32 million cells in the Medicaid data examined. The taxonomy was developed through document review, descriptive data analysis, and literature review. A software program was created to automatically detect defects by using a set of constraints whose development was facilitated by the taxonomy. RESULTS Five major categories and seventeen subcategories of defects were identified. The major categories are missingness, incorrectness, syntax violation, semantic violation, and duplicity. More than 3 million defects were detected indicating substantial problems with data quality. Defect density exceeded 10% in five tables. The majority of the data defects belonged to format mismatch, invalid code, dependency-contract violation, and implausible value types. Such contextual knowledge can support prioritized quality improvement initiatives for the Medicaid data studied. CONCLUSIONS This research took the initial steps to understand the types of data defects and detect defects in large healthcare datasets. The results generally suggest that healthcare organizations can potentially benefit from focusing on data quality improvement. For those purposes, the taxonomy developed and the approach followed in this study can be adopted.
Collapse
Affiliation(s)
- Yili Zhang
- Department of Information Systems, University of Maryland, Baltimore County, Baltimore, Maryland, USA.,Postdoctoral Fellow the Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA
| | - Güneş Koru
- Department of Information Systems, University of Maryland, Baltimore County, Baltimore, Maryland, USA
| |
Collapse
|
14
|
Rockenschaub P, Nguyen V, Aldridge RW, Acosta D, García-Gómez JM, Sáez C. Data-driven discovery of changes in clinical code usage over time: a case-study on changes in cardiovascular disease recording in two English electronic health records databases (2001-2015). BMJ Open 2020; 10:e034396. [PMID: 32060159 PMCID: PMC7045100 DOI: 10.1136/bmjopen-2019-034396] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/17/2022] Open
Abstract
OBJECTIVES To demonstrate how data-driven variability methods can be used to identify changes in disease recording in two English electronic health records databases between 2001 and 2015. DESIGN Repeated cross-sectional analysis that applied data-driven temporal variability methods to assess month-by-month changes in routinely collected medical data. A measure of difference between months was calculated based on joint distributions of age, gender, socioeconomic status and recorded cardiovascular diseases. Distances between months were used to identify temporal trends in data recording. SETTING 400 English primary care practices from the Clinical Practice Research Datalink (CPRD GOLD) and 451 hospital providers from the Hospital Episode Statistics (HES). MAIN OUTCOMES The proportion of patients (CPRD GOLD) and hospital admissions (HES) with a recorded cardiovascular disease (CPRD GOLD: coronary heart disease, heart failure, peripheral arterial disease, stroke; HES: International Classification of Disease codes I20-I69/G45). RESULTS Both databases showed gradual changes in cardiovascular disease recording between 2001 and 2008. The recorded prevalence of included cardiovascular diseases in CPRD GOLD increased by 47%-62%, which partially reversed after 2008. For hospital records in HES, there was a relative decrease in angina pectoris (-34.4%) and unspecified stroke (-42.3%) over the same time period, with a concomitant increase in chronic coronary heart disease (+14.3%). Multiple abrupt changes in the use of myocardial infarction codes in hospital were found in March/April 2010, 2012 and 2014, possibly linked to updates of clinical coding guidelines. CONCLUSIONS Identified temporal variability could be related to potentially non-medical causes such as updated coding guidelines. These artificial changes may introduce temporal correlation among diagnoses inferred from routine data, violating the assumptions of frequently used statistical methods. Temporal variability measures provide an objective and robust technique to identify, and subsequently account for, those changes in electronic health records studies without any prior knowledge of the data collection process.
Collapse
Affiliation(s)
- Patrick Rockenschaub
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | - Vincent Nguyen
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | - Robert W Aldridge
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | - Dionisio Acosta
- Institute of Health Informatics, University College London, London, UK
- Health Data Research UK, London, UK
| | - Juan Miguel García-Gómez
- Instituto de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Valencia, Spain
| | - Carlos Sáez
- Instituto de Aplicaciones de las Tecnologías de la Información y de las Comunicaciones Avanzadas (ITACA), Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
15
|
Pérez-Benito FJ, Conejero JA, Sáez C, García-Gómez JM, Navarro-Pardo E, Florencio LL, Fernández-de-Las-Peñas C. Subgrouping Factors Influencing Migraine Intensity in Women: A Semi-automatic Methodology Based on Machine Learning and Information Geometry. Pain Pract 2019; 20:297-309. [PMID: 31677218 DOI: 10.1111/papr.12854] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2019] [Revised: 10/27/2019] [Accepted: 10/31/2019] [Indexed: 12/21/2022]
Abstract
BACKGROUND Migraine is a heterogeneous condition with multiple clinical manifestations. Machine learning algorithms permit the identification of population groups, providing analytical advantages over other modeling techniques. OBJECTIVE The aim of this study was to analyze critical features that permit the differentiation of subgroups of patients with migraine according to the intensity and frequency of attacks by using machine learning algorithms. METHODS Sixty-seven women with migraine participated. Clinical features of migraine, related disability (Migraine Disability Assessment Scale), anxiety/depressive levels (Hospital Anxiety and Depression Scale), anxiety state/trait levels (State-Trait Anxiety Inventory), and pressure pain thresholds (PPTs) over the temporalis, neck, second metacarpal, and tibialis anterior were collected. Physical examination included the flexion-rotation test, cervical range of cervical motion, forward head position while sitting and standing, passive accessory intervertebral movements (PAIVMs) with headache reproduction, and joint positioning sense error. Subgrouping was based on machine learning algorithms by using the nearest neighbors algorithm, multisource variability assessment, and random forest model. RESULTS For migraine intensity, group 2 (women with a regular migraine headache intensity score of 7 on an 11-point Numeric Pain Rating Scale [where 0 = no pain and 10 = maximum pain]) were younger and had lower joint positioning sense error in cervical rotation, greater cervical mobility in rotation and flexion, lower flexion-rotation test scores, positive PAIVMs reproducing migraine, normal PPTs over the tibialis anterior, shorter migraine history, and lower cranio-vertebral angles while standing than the remaining migraine intensity subgroups. The most discriminative variable was the flexion-rotation test score of the symptomatic side. For migraine frequency, no model was able to identify differences between groups (ie, patients with episodic or chronic migraine). CONCLUSIONS A subgroup of women with migraine who had common migraine intensity was identified with machine learning algorithms.
Collapse
Affiliation(s)
- Francisco J Pérez-Benito
- Biomedical Data Science Lab (BDSLab), Instituto Universitario de las Tecnologías de la Información y Comunicaciones (ITACA), Univeritat Politècnica de València, Valencia, Spain
| | - J Alberto Conejero
- Instituto Universitario de Matemática Pura y Aplicada (IUMPA), Universitat Politécnica de València, Valencia, Spain
| | - Carlos Sáez
- Biomedical Data Science Lab (BDSLab), Instituto Universitario de las Tecnologías de la Información y Comunicaciones (ITACA), Univeritat Politècnica de València, Valencia, Spain
| | - Juan M García-Gómez
- Biomedical Data Science Lab (BDSLab), Instituto Universitario de las Tecnologías de la Información y Comunicaciones (ITACA), Univeritat Politècnica de València, Valencia, Spain
| | - Esperanza Navarro-Pardo
- Departamento de Psicología Evolutiva y de la Educación, Universitat de València, València, Spain
| | - Lidiane L Florencio
- Department of Physical Therapy, Occupational Therapy, Rehabilitation and Physical Medicine, Universidad Rey Juan Carlos, Madrid, Spain
| | - César Fernández-de-Las-Peñas
- Department of Physical Therapy, Occupational Therapy, Rehabilitation and Physical Medicine, Universidad Rey Juan Carlos, Madrid, Spain
| |
Collapse
|
16
|
Li Y, Sperrin M, Martin GP, Ashcroft DM, van Staa TP. Examining the impact of data quality and completeness of electronic health records on predictions of patients' risks of cardiovascular disease. Int J Med Inform 2019; 133:104033. [PMID: 31785526 DOI: 10.1016/j.ijmedinf.2019.104033] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2019] [Revised: 10/16/2019] [Accepted: 11/08/2019] [Indexed: 10/25/2022]
Abstract
OBJECTIVE To assess the extent of variation of data quality and completeness of electronic health records and impact on the robustness of risk predictions of incident cardiovascular disease (CVD) using a risk prediction tool that is based on routinely collected data (QRISK3). DESIGN Longitudinal cohort study. SETTINGS 392 general practices (including 3.6 million patients) linked to hospital admission data. METHODS Variation in data quality was assessed using Sáez's stability metrics quantifying outlyingness of each practice. Statistical frailty models evaluated whether accuracy of QRISK3 predictions on individual predictions and effects of overall risk factors (linear predictor) varied between practices. RESULTS There was substantial heterogeneity between practices in CVD incidence unaccounted for by QRISK3. In the lowest quintile of statistical frailty, a QRISK3 predicted risk of 10 % for female was in a range between 7.1 % and 9.0 % when incorporating practice variability into the statistical frailty models; for the highest quintile, this was 10.9%-16.4%. Data quality (using Saez metrics) and completeness were comparable across different levels of statistical frailty. For example, recording of missing information on ethnicity was 55.7 %, 62.7 %, 57.8 %, 64.8 % and 62.1 % for practices from lowest to highest quintiles of statistical frailty respectively. The effects of risk factors did not vary between practices with little statistical variation of beta coefficients. CONCLUSIONS The considerable unmeasured heterogeneity in CVD incidence between practices was not explained by variations in data quality or effects of risk factors. QRISK3 risk prediction should be supplemented with clinical judgement and evidence of additional risk factors.
Collapse
Affiliation(s)
- Yan Li
- Health e-Research Centre, Farr Institute, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Sciences Centre (MAHSC), Oxford Road, Manchester, M13 9PL, UK
| | - Matthew Sperrin
- Health e-Research Centre, Farr Institute, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Sciences Centre (MAHSC), Oxford Road, Manchester, M13 9PL, UK
| | - Glen P Martin
- Health e-Research Centre, Farr Institute, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Sciences Centre (MAHSC), Oxford Road, Manchester, M13 9PL, UK
| | - Darren M Ashcroft
- Centre for Pharmacoepidemiology and Drug Safety, School of Health Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Oxford Road, Manchester, M13 9PL, UK; NIHR Greater Manchester Patient Safety Translational Research Centre, School of Health Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Oxford Road, Manchester, M13 9PL, UK
| | - Tjeerd Pieter van Staa
- Health e-Research Centre, Farr Institute, School of Health Sciences, Faculty of Biology, Medicine and Health, The University of Manchester, Manchester Academic Health Sciences Centre (MAHSC), Oxford Road, Manchester, M13 9PL, UK; Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands; Alan Turing Institute, Headquartered at the British Library, London, UK.
| |
Collapse
|
17
|
Sáez C, Liaw ST, Kimura E, Coorevits P, Garcia-Gomez JM. Guest editorial: Special issue in biomedical data quality assessment methods. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2019; 181:104954. [PMID: 31242965 DOI: 10.1016/j.cmpb.2019.06.013] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Affiliation(s)
- Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Camino de Vera s/n, Valencia 46022, Spain.
| | - Siaw-Teng Liaw
- WHO Collaborating Centre on eHealth, School of Public Health & Community Medicine, UNSW Sydney, Australia.
| | | | | | - Juan M Garcia-Gomez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones (ITACA), Universitat Politècnica de València (UPV), Camino de Vera s/n, Valencia 46022, Spain.
| |
Collapse
|
18
|
Looten V, Kong Win Chang L, Neuraz A, Landau-Loriot MA, Vedie B, Paul JL, Mauge L, Rivet N, Bonifati A, Chatellier G, Burgun A, Rance B. What can millions of laboratory test results tell us about the temporal aspect of data quality? Study of data spanning 17 years in a clinical data warehouse. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2019; 181:104825. [PMID: 30612785 DOI: 10.1016/j.cmpb.2018.12.030] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2018] [Revised: 12/24/2018] [Accepted: 12/28/2018] [Indexed: 06/09/2023]
Abstract
OBJECTIVE To identify common temporal evolution profiles in biological data and propose a semi-automated method to these patterns in a clinical data warehouse (CDW). MATERIALS AND METHODS We leveraged the CDW of the European Hospital Georges Pompidou and tracked the evolution of 192 biological parameters over a period of 17 years (for 445,000 + patients, and 131 million laboratory test results). RESULTS We identified three common profiles of evolution: discretization, breakpoints, and trends. We developed computational and statistical methods to identify these profiles in the CDW. Overall, of the 192 observed biological parameters (87,814,136 values), 135 presented at least one evolution. We identified breakpoints in 30 distinct parameters, discretizations in 32, and trends in 79. DISCUSSION AND CONCLUSION our method allowed the identification of several temporal events in the data. Considering the distribution over time of these events, we identified probable causes for the observed profiles: instruments or software upgrades and changes in computation formulas. We evaluated the potential impact for data reuse. Finally, we formulated recommendations to enable safe use and sharing of biological data collection to limit the impact of data evolution in retrospective and federated studies (e.g. the annotation of laboratory parameters presenting breakpoints or trends).
Collapse
Affiliation(s)
- Vincent Looten
- INSERM, Centre de Recherche des Cordeliers, UMRS 1138, Université Paris Descartes, Sorbonne Paris Cité, Paris, France; Hôpital Européen Georges Pompidou, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, 20 rue Leblanc, 75015 Paris, France
| | | | - Antoine Neuraz
- INSERM, Centre de Recherche des Cordeliers, UMRS 1138, Université Paris Descartes, Sorbonne Paris Cité, Paris, France; Hôpital Necker - Enfants Malades, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Marie-Anne Landau-Loriot
- Hôpital Européen Georges Pompidou, Department of Biochimistry, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Benoit Vedie
- Hôpital Européen Georges Pompidou, Department of Biochimistry, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Jean-Louis Paul
- Hôpital Européen Georges Pompidou, Department of Biochimistry, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Laëtitia Mauge
- Hôpital Européen Georges Pompidou, Department of Hematology, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Nadia Rivet
- Hôpital Européen Georges Pompidou, Department of Hematology, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Angela Bonifati
- LIRIS UMR CNRS 5205, Université Claude Bernard Lyon 1, Villeurbanne, France
| | - Gilles Chatellier
- Hôpital Européen Georges Pompidou, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, 20 rue Leblanc, 75015 Paris, France
| | - Anita Burgun
- INSERM, Centre de Recherche des Cordeliers, UMRS 1138, Université Paris Descartes, Sorbonne Paris Cité, Paris, France; Hôpital Européen Georges Pompidou, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, 20 rue Leblanc, 75015 Paris, France; Hôpital Necker - Enfants Malades, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, France
| | - Bastien Rance
- INSERM, Centre de Recherche des Cordeliers, UMRS 1138, Université Paris Descartes, Sorbonne Paris Cité, Paris, France; Hôpital Européen Georges Pompidou, Department of Medical Informatics, Assistance Publique - Hôpitaux de Paris (AP-HP), Université Paris Descartes, 20 rue Leblanc, 75015 Paris, France.
| |
Collapse
|
19
|
Pérez-Benito FJ, Sáez C, Conejero JA, Tortajada S, Valdivieso B, García-Gómez JM. Temporal variability analysis reveals biases in electronic health records due to hospital process reengineering interventions over seven years. PLoS One 2019; 14:e0220369. [PMID: 31390350 PMCID: PMC6685618 DOI: 10.1371/journal.pone.0220369] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2019] [Accepted: 07/15/2019] [Indexed: 12/28/2022] Open
Abstract
OBJECTIVE To evaluate the effects of Process-Reengineering interventions on the Electronic Health Records (EHR) of a hospital over 7 years. MATERIALS AND METHODS Temporal Variability Assessment (TVA) based on probabilistic data quality assessment was applied to the historic monthly-batched admission data of Hospital La Fe Valencia, Spain from 2010 to 2016. Routine healthcare data with a complete EHR was expanded by processed variables such as the Charlson Comorbidity Index. RESULTS Four Process-Reengineering interventions were detected by quantifiable effects on the EHR: (1) the hospital relocation in 2011 involved progressive reduction of admissions during the next four months, (2) the hospital services re-configuration incremented the number of inter-services transfers, (3) the care-services re-distribution led to transfers between facilities (4) the assignment to the hospital of a new area with 80,000 patients in 2015 inspired the discharge to home for follow up and the update of the pre-surgery planned admissions protocol that produced a significant decrease of the patient length of stay. DISCUSSION TVA provides an indicator of the effect of process re-engineering interventions on healthcare practice. Evaluating the effect of facilities' relocation and increment of citizens (findings 1, 3-4), the impact of strategies (findings 2-3), and gradual changes in protocols (finding 4) may help on the hospital management by optimizing interventions based on their effect on EHRs or on data reuse. CONCLUSIONS The effects on hospitals EHR due to process re-engineering interventions can be evaluated using the TVA methodology. Being aware of conditioned variations in EHR is of the utmost importance for the reliable reuse of routine hospitalization data.
Collapse
Affiliation(s)
- Francisco Javier Pérez-Benito
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de Información y Comunicaciones Avanzadas (ITACA), Univeritat Politécnica de València, València, Spain
- Instituto Universitario de Matemática Pura y Aplicada, Universitat Politécnica de València, València, Spain
| | - Carlos Sáez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de Información y Comunicaciones Avanzadas (ITACA), Univeritat Politécnica de València, València, Spain
| | - J. Alberto Conejero
- Instituto Universitario de Matemática Pura y Aplicada, Universitat Politécnica de València, València, Spain
- * E-mail:
| | - Salvador Tortajada
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de Información y Comunicaciones Avanzadas (ITACA), Univeritat Politécnica de València, València, Spain
- Unidad conjunta de investigación en reingeniería de procesos socio-sanitarios, Instituto de Investigación Sanitaria La Fe, Hospital Universitario La Fe, València, Spain
- Red de Investigación en Servicios de Salud en Enfermedades Crónicas (REDISSEC), València, Spain
| | - Bernardo Valdivieso
- Unidad conjunta de investigación en reingeniería de procesos socio-sanitarios, Instituto de Investigación Sanitaria La Fe, Hospital Universitario La Fe, València, Spain
| | - Juan M. García-Gómez
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de Información y Comunicaciones Avanzadas (ITACA), Univeritat Politécnica de València, València, Spain
- Unidad conjunta de investigación en reingeniería de procesos socio-sanitarios, Instituto de Investigación Sanitaria La Fe, Hospital Universitario La Fe, València, Spain
- Red de Investigación en Servicios de Salud en Enfermedades Crónicas (REDISSEC), València, Spain
| |
Collapse
|
20
|
Sáez C, García-Gómez JM. Kinematics of Big Biomedical Data to characterize temporal variability and seasonality of data repositories: Functional Data Analysis of data temporal evolution over non-parametric statistical manifolds. Int J Med Inform 2018; 119:109-124. [DOI: 10.1016/j.ijmedinf.2018.09.015] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2018] [Revised: 09/05/2018] [Accepted: 09/13/2018] [Indexed: 01/26/2023]
|
21
|
Abstract
Veracity, one of the five V's used to describe big data, has received attention when it comes to using electronic medical record data for research purposes. In this perspective article, we discuss the idea of data veracity and associated concepts as it relates to the use of electronic medical record data and administrative data in research. We discuss the idea that electronic medical record data are "good enough" for clinical practice and, as such, are "good enough" for certain applications. We then propose three primary issues to attend to when establishing data veracity: data provenance, cross validation, and context.
Collapse
|
22
|
Fernandez-Luque L, Imran M. Humanitarian health computing using artificial intelligence and social media: A narrative literature review. Int J Med Inform 2018; 114:136-142. [PMID: 29395987 DOI: 10.1016/j.ijmedinf.2018.01.015] [Citation(s) in RCA: 36] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2017] [Revised: 01/14/2018] [Accepted: 01/19/2018] [Indexed: 01/22/2023]
Abstract
INTRODUCTION According to the World Health Organization (WHO), over 130 million people are in constant need of humanitarian assistance due to natural disasters, disease outbreaks, and conflicts, among other factors. These health crises can compromise the resilience of healthcare systems, which are essential for achieving the health objectives of the sustainable development goals (SDGs) of the United Nations (UN). During a humanitarian health crisis, rapid and informed decision making is required. This is often challenging due to information scarcity, limited resources, and strict time constraints. Moreover, the traditional approach to digital health development, which involves a substantial requirement analysis, a feasibility study, and deployment of technology, is ill-suited for many crisis contexts. The emergence of Web 2.0 technologies and social media platforms in the past decade, such as Twitter, has created a new paradigm of massive information and misinformation, in which new technologies need to be developed to aid rapid decision making during humanitarian health crises. OBJECTIVE Humanitarian health crises increasingly require the analysis of massive amounts of information produced by different sources, such as social media content, and, hence, they are a prime case for the use of artificial intelligence (AI) techniques to help identify relevant information and make it actionable. To identify challenges and opportunities for using AI in humanitarian health crises, we reviewed the literature on the use of AI techniques to process social media. METHODOLOGY We performed a narrative literature review aimed at identifying examples of the use of AI in humanitarian health crises. Our search strategy was designed to get a broad overview of the different applications of AI in a humanitarian health crisis and their challenges. A total of 1459 articles were screened, and 24 articles were included in the final analysis. RESULTS Successful case studies of AI applications in a humanitarian health crisis have been reported, such as for outbreak detection. A commonly shared concern in the reviewed literature is the technical challenge of analyzing large amounts of data in real time. Data interoperability, which is essential to data sharing, is also a barrier with regard to the integration of online and traditional data sources. Human and organizational aspects that might be key factors for the adoption of AI and social media remain understudied. There is also a publication bias toward high-income countries, as we identified few examples in low-income countries. Further, we did not identify any examples of certain types of major crisis, such armed conflicts, in which misinformation might be more common. CONCLUSIONS The feasibility of using AI to extract valuable information during a humanitarian health crisis is proven in many cases. There is a lack of research on how to integrate the use of AI into the work-flow and large-scale deployments of humanitarian aid during a health crisis.
Collapse
|
23
|
Meidani Z, Mahdian M, Ayan A, Mohammadzade M, Nickfarjam A, Moosavi GA. Registry Data Coordinator (RDC): a Proper Accessible Strategy for Improving Road Traffic Injury (RTI) Hospital Based Trauma Registry Systems in Developing Countries and Low Income Countries. Acta Inform Med 2018; 26:35-41. [PMID: 29719311 PMCID: PMC5869281 DOI: 10.5455/aim.2018.26.35-41] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2017] [Accepted: 02/28/2018] [Indexed: 11/21/2022] Open
Abstract
INTRODUCTION Evidence suggested that a significant level of trauma mortality can be prevented using registry system. AIM This study aimed to improve Kashan Hospital Based Trauma Registry System (KHBTRS) for Road Traffic Injury (RTI). MATERIAL AND METHODS After conducting focus group discussion absence of minimum data set (MDS) and poor data collection process (DCP) were identified as main problems for KHBTRS- RTI. Proposed MDS were surveyed by 20 experts of trauma research center of throughout the Iran. Then approved MDS applied for trauma registry system data base in form of SQL. DCP were reform from prospective data collection (review of medical record) to concurrent (through the interview) approach. RESULTS Most of participants for MDS approval belonged to clinical group 13(65%). 146 MDS in eighteen main categories were proposed for RTI. The maximum score for each MDS main categories were attributed to body parts injured 220 (100%) and patient vital signs 139 (99.29%) respectively. Pilot testing of KHBTRS- RTI database of 50 (50%) riders indicated fully completeness 50 (100%) for concurrent approach. It was concluded that based on experts' viewpoints MDS relating to injury nature and place of occurrence have more priority in comparisons to MDS relating to causes of injury. It may attribute to health care providers focus on clinical care and treatment. CONCLUSION It was concluded that based on experts' viewpoints MDS relating to injury nature and place of occurrence have more priority in comparisons to MDS relating to RTI prevention; it may attribute to health care providers focus on clinical care and treatment. To develop injury interventions based on given data, recruitment of professionals as registry data coordinator with specific job description to collect and advocacy of injury external causes data seems imperative.
Collapse
Affiliation(s)
- Zahra Meidani
- Health Information Management Research Center, Department of Health Information Management and Technology, School of Allied Health Professions, Kashan University of Medical Sciences, Kashan, Iran
| | - Mehrdad Mahdian
- Trauma Research Center, Kashan University of Medical Sciences, Kashan, Iran
| | - Atefe Ayan
- Health Information Management Research Center, Department of Health Information Management and Technology, School of Allied Health Professions, Kashan University of Medical Sciences, Kashan, Iran
| | - Mahdi Mohammadzade
- Trauma Research Center, Kashan University of Medical Sciences, Kashan, Iran
| | | | - Gholam Abbas Moosavi
- Department of Vital Statistics and Epidemiology, School of Health, Kashan University of Medical Sciences, Kashan, Iran
| |
Collapse
|
24
|
Schlegel DR, Ficheur G. Secondary Use of Patient Data: Review of the Literature Published in 2016. Yearb Med Inform 2017; 26:68-71. [PMID: 29063536 DOI: 10.15265/iy-2017-032] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Objectives: To summarize recent research and emerging trends in the area of secondary use of healthcare data, and to present the best papers published in this field, selected to appear in the 2017 edition of the IMIA Yearbook. Methods: A literature review of articles published in 2016 and related to secondary use of healthcare data was performed using two bibliographic databases. From this search, 941 papers were identified. The section editors independently reviewed the papers for relevancy and impact, resulting in a consensus list of 14 candidate best papers. External reviewers examined each of the candidate best papers and the final selection was made by the editorial board of the Yearbook. Results: From the 941 retrieved papers, the selection process resulted in four best papers. These papers discuss data quality concerns, issues in preserving privacy of patients in shared datasets, and methods of decision support when consuming large amounts of raw electronic health record (EHR) data. Conclusion: In 2016, a significant effort was put into the development of new systems which aim to avoid significant human understanding and pre-processing of healthcare data, though this is still only an emerging area of research. The value of temporal relationships between data received significant study, as did effective information sharing while preserving patient privacy.
Collapse
|