101
|
Terry AL, Stewart M, Cejic S, Marshall JN, de Lusignan S, Chesworth BM, Chevendra V, Maddocks H, Shadd J, Burge F, Thind A. A basic model for assessing primary health care electronic medical record data quality. BMC Med Inform Decis Mak 2019; 19:30. [PMID: 30755205 PMCID: PMC6373085 DOI: 10.1186/s12911-019-0740-0] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2018] [Accepted: 01/02/2019] [Indexed: 11/29/2022] Open
Abstract
Background The increased use of electronic medical records (EMRs) in Canadian primary health care practice has resulted in an expansion of the availability of EMR data. Potential users of these data need to understand their quality in relation to the uses to which they are applied. Herein, we propose a basic model for assessing primary health care EMR data quality, comprising a set of data quality measures within four domains. We describe the process of developing and testing this set of measures, share the results of applying these measures in three EMR-derived datasets, and discuss what this reveals about the measures and EMR data quality. The model is offered as a starting point from which data users can refine their own approach, based on their own needs. Methods Using an iterative process, measures of EMR data quality were created within four domains: comparability; completeness; correctness; and currency. We used a series of process steps to develop the measures. The measures were then operationalized, and tested within three datasets created from different EMR software products. Results A set of eleven final measures were created. We were not able to calculate results for several measures in one dataset because of the way the data were collected in that specific EMR. Overall, we found variability in the results of testing the measures (e.g. sensitivity values were highest for diabetes, and lowest for obesity), among datasets (e.g. recording of height), and by patient age and sex (e.g. recording of blood pressure, height and weight). Conclusions This paper proposes a basic model for assessing primary health care EMR data quality. We developed and tested multiple measures of data quality, within four domains, in three different EMR-derived primary health care datasets. The results of testing these measures indicated that not all measures could be utilized in all datasets, and illustrated variability in data quality. This is one step forward in creating a standard set of measures of data quality. Nonetheless, each project has unique challenges, and therefore requires its own data quality assessment before proceeding. Electronic supplementary material The online version of this article (10.1186/s12911-019-0740-0) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Amanda L Terry
- Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich Interfaculty Program in Public Health, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada.
| | - Moira Stewart
- Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| | - Sonny Cejic
- Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| | - J Neil Marshall
- Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| | - Simon de Lusignan
- Department of Clinical and Experimental Medicine, University of Surrey, Guildford, Surrey, GU2 7XH, UK
| | - Bert M Chesworth
- School of Physical Therapy, Faculty of Health Sciences, Department of Epidemiology & Biostatistics, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| | - Vijaya Chevendra
- Science and Software Educator and Consultant, 58 Moraine Walk, London, Ontario, N6G 4Y8, Canada
| | - Heather Maddocks
- Department of Family Medicine, Schulich School of Medicine & Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| | - Joshua Shadd
- Department of Family Medicine, McMaster University, 100 Main Street West, 6th Floor, Hamilton, Ontario, L8P 1H6, Canada
| | - Fred Burge
- Department of Family Medicine, Dalhousie University, 5909 Veterans Memorial Lane, Abbie J Lane Building, Room 8101B, Halifax, Nova Scotia, B3H 2E2, Canada
| | - Amardeep Thind
- Department of Family Medicine, Department of Epidemiology & Biostatistics, Schulich Interfaculty Program in Public Health, Schulich School of Medicine and Dentistry, The University of Western Ontario, 1151 Richmond Street, London, Ontario, N6A 3K7, Canada
| |
Collapse
|
102
|
Cohen DJ, Dorr DA, Knierim K, DuBard CA, Hemler JR, Hall JD, Marino M, Solberg LI, McConnell KJ, Nichols LM, Nease DE, Edwards ST, Wu WY, Pham-Singer H, Kho AN, Phillips RL, Rasmussen LV, Duffy FD, Balasubramanian BA. Primary Care Practices' Abilities And Challenges In Using Electronic Health Record Data For Quality Improvement. Health Aff (Millwood) 2019; 37:635-643. [PMID: 29608365 DOI: 10.1377/hlthaff.2017.1254] [Citation(s) in RCA: 55] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
Federal value-based payment programs require primary care practices to conduct quality improvement activities, informed by the electronic reports on clinical quality measures that their electronic health records (EHRs) generate. To determine whether EHRs produce reports adequate to the task, we examined survey responses from 1,492 practices across twelve states, supplemented with qualitative data. Meaningful-use participation, which requires the use of a federally certified EHR, was associated with the ability to generate reports-but the reports did not necessarily support quality improvement initiatives. Practices reported numerous challenges in generating adequate reports, such as difficulty manipulating and aligning measurement time frames with quality improvement needs, lack of functionality for generating reports on electronic clinical quality measures at different levels, discordance between clinical guidelines and measures available in reports, questionable data quality, and vendors that were unreceptive to changing EHR configuration beyond federal requirements. The current state of EHR measurement functionality may be insufficient to support federal initiatives that tie payment to clinical quality measures.
Collapse
Affiliation(s)
- Deborah J Cohen
- Deborah J. Cohen ( ) is a professor of family medicine and vice chair of research in the Department of Family Medicine at Oregon Health & Science University, in Portland
| | - David A Dorr
- David A. Dorr is a professor and vice chair of medical informatics and clinical epidemiology, both at Oregon Health & Science University
| | - Kyle Knierim
- Kyle Knierim is an assistant research professor of family medicine and associate director of the Practice Innovation Program, both at the University of Colorado School of Medicine, in Aurora
| | - C Annette DuBard
- C. Annette DuBard is vice president of Clinical Strategy at Aledade, Inc., in Bethesda, Maryland
| | - Jennifer R Hemler
- Jennifer R. Hemler is a research associate in the Department of Family Medicine and Community Health, Research Division, Rutgers Robert Wood Johnson Medical School, in New Brunswick, New Jersey
| | - Jennifer D Hall
- Jennifer D. Hall is a research associate in family medicine at Oregon Health & Science University
| | - Miguel Marino
- Miguel Marino is an assistant professor of family medicine at Oregon Health & Science University
| | - Leif I Solberg
- Leif I. Solberg is a senior adviser and director for care improvement research at HealthPartners Institute, in Minneapolis, Minnesota
| | - K John McConnell
- K. John McConnell is a professor of emergency medicine and director of the Center for Health Systems Effectiveness, both at Oregon Health & Science University
| | - Len M Nichols
- Len M. Nichols is director of the Center for Health Policy Research and Ethics and a professor of health policy at George Mason University, in Fairfax, Virginia
| | - Donald E Nease
- Donald E. Nease Jr is an associate professor of family medicine at the University of Colorado School of Medicine, in Aurora
| | - Samuel T Edwards
- Samuel T. Edwards is an assistant research professor of family medicine and an assistant professor of medicine at Oregon Health & Science University and a staff physician in the Section of General Internal Medicine, Veterans Affairs Portland Health Care System
| | - Winfred Y Wu
- Winfred Y. Wu is clinical and scientific director in the Primary Care Information Project at the New York City Department of Health and Mental Hygiene, in Long Island City, New York
| | - Hang Pham-Singer
- Hang Pham-Singer is senior director of quality improvement in the Primary Care Information Project at the New York City Department of Health and Mental Hygiene
| | - Abel N Kho
- Abel N. Kho is an associate professor and director of the Center for Health Information Partnerships, Northwestern University, in Chicago, Illinois
| | - Robert L Phillips
- Robert L. Phillips Jr is vice president for research and policy at the American Board of Family Medicine, in Washington, D.C
| | - Luke V Rasmussen
- Luke V. Rasmussen is a clinical research associate in the Department of Preventive Medicine, Northwestern University
| | - F Daniel Duffy
- F. Daniel Duffy is professor of medical informatics and internal medicine at the University of Oklahoma School of Community Medicine-Tulsa
| | - Bijal A Balasubramanian
- Bijal A. Balasubramanian is an associate professor in the Department of Epidemiology, Human Genetics, and Environmental Sciences, and regional dean of UTHealth School of Public Health, in Dallas, Texas
| |
Collapse
|
103
|
Hagiwara MA, Magnusson C, Herlitz J, Seffel E, Axelsson C, Munters M, Strömsöe A, Nilsson L. Adverse events in prehospital emergency care: a trigger tool study. BMC Emerg Med 2019; 19:14. [PMID: 30678636 PMCID: PMC6345067 DOI: 10.1186/s12873-019-0228-3] [Citation(s) in RCA: 35] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2018] [Accepted: 01/15/2019] [Indexed: 11/30/2022] Open
Abstract
Background Prehospital emergency care has developed rapidly during the past decades. The care is given in a complex context which makes prehospital care a potential high-risk activity when it comes to patient safety. Patient safety in the prehospital setting has been only sparsely investigated. The aims of the present study were 1) To investigate the incidence of adverse events (AEs) in prehospital care and 2) To investigate the factors contributing to AEs in prehospital care. Methods We used a retrospective study design where 30 randomly selected prehospital medical records were screened for AEs each month in three prehospital organizations in Sweden during a period of one year. A total of 1080 prehospital medical records were included. The record review was based on the use of 11 screening criteria. Results The reviewers identified 46 AEs in 46 of 1080 (4.3%) prehospital medical records. Of the 46 AEs, 43 were classified as potential for harm (AE1) (4.0, 95% CI = 2.9–5.4) and three as harm identified (AE2) (0.3, 95% CI = 0.1–0.9). However, among patients with a life-threatening condition (priority 1), the risk of AE was higher (16.5%). The most common factors contributing to AEs were deviations from standard of care and missing, incomplete, or unclear documentation. The most common cause of AEs was the result of action(s) or inaction(s) by the emergency medical service (EMS) crew. Conclusions There were 4.3 AEs per 100 ambulance missions in Swedish prehospital care. The majority of AEs originated from deviations from standard of care and incomplete documentation. There was an increase in the risk of AE among patients who the EMS team assessed as having a life-threatening condition. Most AEs were possible to avoid. Electronic supplementary material The online version of this article (10.1186/s12873-019-0228-3) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Magnus Andersson Hagiwara
- Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden.
| | - Carl Magnusson
- Department of Molecular and Clinical Medicine, University of Gothenburg and Sahlgrenska University Hospital, SE-405 30, Gothenburg, Sweden
| | - Johan Herlitz
- Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Elin Seffel
- Department of Ambulance Care, Södra Älvsborg Hospital (SÄS), SE-501 82, Borås, Sweden
| | - Christer Axelsson
- Faculty of Caring Science, Work Life and Social Welfare, University of Borås, SE-501 90, Borås, Sweden
| | - Monica Munters
- Department of Ambulance Care, Region of Dalarna, SE-791 29, Falun, Sweden
| | - Anneli Strömsöe
- School of Health, Care and Social Welfare, Mälardalens högskola, SE-721 23, Västerås, Sweden
| | - Lena Nilsson
- Department of Anaesthesiology and Intensive Care and Department of Medical and Health Sciences, Linköping University, SE-581 85, Linköping, Sweden
| |
Collapse
|
104
|
Lettvin RJ, Wayal A, McNutt A, Miller RS, Hauser R. Assessment and Stratification of High-Impact Data Elements in Electronic Clinical Quality Measures: A Joint Data Quality Initiative Between CancerLinQ® and Cancer Treatment Centers of America. JCO Clin Cancer Inform 2019; 2:1-10. [PMID: 30652592 DOI: 10.1200/cci.17.00139] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
PURPOSE A joint data quality initiative between the Cancer Treatment Centers of America and the ASCO big data health technology platform CancerLinQ® was initiated to document and codify the steps taken to evaluate, stratify, and determine the potential effect of data elements used for electronic clinical quality measures as captured within structured fields in electronic health records. METHODS The processes involved the identification of clinical concepts required in measure population criteria and then to map these to the corresponding components of the CancerLinQ data model. A quantitative assessment of mappings between electronic clinical quality measure clinical concepts and attributes from the CancerLinQ clinical database was performed. In parallel, a qualitative analysis of high-impact data elements from the Cancer Treatment Centers of America clinical measures was made using local, expert consensus. RESULTS An impact assessment was derived using a count of the data elements across measures and the specific population criteria affected. CONCLUSION A list of putative high-impact data elements can provide guidance for clinicians to facilitate specific data element capture related to quality metrics in an electronic environment.
Collapse
Affiliation(s)
- Rory J Lettvin
- Rory J. Lettvin, Alpna Wayal, Amy McNutt, and Robert S. Miller, American Society of Clinical Oncology, Alexandria, VA; and Robert Hauser, Cancer Treatment Centers of America, Boca Raton, FL
| | - Alpna Wayal
- Rory J. Lettvin, Alpna Wayal, Amy McNutt, and Robert S. Miller, American Society of Clinical Oncology, Alexandria, VA; and Robert Hauser, Cancer Treatment Centers of America, Boca Raton, FL
| | - Amy McNutt
- Rory J. Lettvin, Alpna Wayal, Amy McNutt, and Robert S. Miller, American Society of Clinical Oncology, Alexandria, VA; and Robert Hauser, Cancer Treatment Centers of America, Boca Raton, FL
| | - Robert S Miller
- Rory J. Lettvin, Alpna Wayal, Amy McNutt, and Robert S. Miller, American Society of Clinical Oncology, Alexandria, VA; and Robert Hauser, Cancer Treatment Centers of America, Boca Raton, FL
| | - Robert Hauser
- Rory J. Lettvin, Alpna Wayal, Amy McNutt, and Robert S. Miller, American Society of Clinical Oncology, Alexandria, VA; and Robert Hauser, Cancer Treatment Centers of America, Boca Raton, FL
| |
Collapse
|
105
|
Alwhaibi M, Balkhi B, Alshammari TM, AlQahtani N, Mahmoud MA, Almetwazi M, Ata S, Basyoni M, Alhawassi T. Measuring the quality and completeness of medication-related information derived from hospital electronic health records database. Saudi Pharm J 2019; 27:502-506. [PMID: 31061618 PMCID: PMC6488812 DOI: 10.1016/j.jsps.2019.01.013] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2018] [Accepted: 01/14/2019] [Indexed: 11/28/2022] Open
Abstract
Objective Electronic Health Records (EHRs) database is a great source for pharmacoepidemiological research as thousands of patients' clinical and medication information is stored in the database. However, the use of EHRs database for research purposes depends greatly on the accuracy and completeness of the data being used. This study mainly aimed to assess the completeness of EHRs patients' medication-related information. Design A retrospective cross-sectional study using data extracted from the EHRs database was conducted. Setting The EHRs data was obtained from a single tertiary hospital in Saudi Arabia. Main outcome measures The completeness of data was measured considering if a patients' record contains all desired types of data (i.e., patients' demographics, clinical diagnosis, and medication-related information). Results A total of 23,411 unique individuals were identified after extracting the data from the EHRs. The study found that 89.9% of the patients had a complete data (i.e., age, gender, marital status, nationality, encounter type, and clinical diagnosis). Further, 83.1% of the patients had complete medication-related information. Subgroup analysis by the encounter type indicated that the data was 91.0% complete for outpatient encounter and 93.2% complete for inpatient encounter. Conclusion The study findings indicate that the completeness of the data varies by the desired types of data. EHRs can be a potentially great resource to conduct research to assess medication use. Further studies focusing on the content and completeness of EHRs for a specific patient population and evaluate other dimensions of EHRs data quality are needed.
Collapse
Affiliation(s)
- Monira Alwhaibi
- Department of Clinical Pharmacy, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | - Bander Balkhi
- Department of Clinical Pharmacy, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | - Thamir M Alshammari
- Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Saudi Food and Drug Authority, Riyadh, Saudi Arabia.,Department of Clinical Pharmacy, College of Pharmacy, University of Hail, Hail, Saudi Arabia
| | | | - Mansour A Mahmoud
- Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | - Mansour Almetwazi
- Department of Clinical Pharmacy, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | - Sondus Ata
- Pharmacy Care Services, King Saud University Medical City, Riyadh, Saudi Arabia
| | - Mada Basyoni
- Saudi Patient Safety Centre, Riyadh, Saudi Arabia
| | - Tariq Alhawassi
- Department of Clinical Pharmacy, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Medication Safety Research Chair, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia.,Pharmacy Care Services, King Saud University Medical City, Riyadh, Saudi Arabia
| |
Collapse
|
106
|
Chong JL, Low LL, Chan DYL, Shen Y, Thin TN, Ong MEH, Matchar DB. Can we understand population healthcare needs using electronic medical records? Singapore Med J 2019; 60:446-453. [PMID: 30644525 DOI: 10.11622/smedj.2019012] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
INTRODUCTION The identification of population-level healthcare needs using hospital electronic medical records (EMRs) is a promising approach for the evaluation and development of tailored healthcare services. Population segmentation based on healthcare needs may be possible using information on health and social service needs from EMRs. However, it is currently unknown if EMRs from restructured hospitals in Singapore provide information of sufficient quality for this purpose. We compared the inter-rater reliability between a population segment that was assigned prospectively and one that was assigned retrospectively based on EMR review. METHODS 200 non-critical patients aged ≥ 55 years were prospectively evaluated by clinicians for their healthcare needs in the emergency department at Singapore General Hospital, Singapore. Trained clinician raters with no prior knowledge of these patients subsequently accessed the EMR up to the prospective rating date. A similar healthcare needs evaluation was conducted using the EMR. The inter-rater reliability between the two rating sets was evaluated using Cohen's Kappa and the incidence of missing information was tabulated. RESULTS The inter-rater reliability for the medical 'global impression' rating was 0.37 for doctors and 0.35 for nurses. The inter-rater reliability for the same variable, retrospectively rated by two doctors, was 0.75. Variables with a higher incidence of missing EMR information such as 'social support in case of need' and 'patient activation' had poorer inter-rater reliability. CONCLUSION Pre-existing EMR systems may not capture sufficient information for reliable determination of healthcare needs. Thus, we should consider integrating policy-relevant healthcare need variables into EMRs.
Collapse
Affiliation(s)
- Jia Loon Chong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore
| | - Lian Leng Low
- SingHealth Duke-NUS Family Medicine Academic Clinical Programme, Singapore.,Department of Family Medicine and Continuing Care, Singapore General Hospital, Singapore
| | - Darren Yak Leong Chan
- Department of Family Medicine and Continuing Care, Singapore General Hospital, Singapore
| | - Yuzeng Shen
- Department of Emergency Medicine, Singapore General Hospital, Singapore
| | - Thiri Naing Thin
- Health Services Research Unit, Singapore General Hospital, Singapore
| | - Marcus Eng Hock Ong
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore.,Department of Emergency Medicine, Singapore General Hospital, Singapore
| | - David Bruce Matchar
- Programme in Health Services and Systems Research, Duke-NUS Medical School, Singapore.,Department of Medicine (General Internal Medicine), Duke University, Durham, NC, USA
| |
Collapse
|
107
|
Dharod A, Bellinger C, Foley K, Case LD, Miller D. The Reach and Feasibility of an Interactive Lung Cancer Screening Decision Aid Delivered by Patient Portal. Appl Clin Inform 2019; 10:19-27. [PMID: 30625501 DOI: 10.1055/s-0038-1676807] [Citation(s) in RCA: 26] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022] Open
Abstract
OBJECTIVE Health systems could adopt population-level approaches to screening by identifying potential screening candidates from the electronic health record and reaching out to them via the patient portal. However, whether patients would read or act on sent information is unknown. We examined the feasibility of this digital health outreach strategy. METHODS We conducted a single-arm pragmatic trial in a large academic health system. An electronic health record algorithm identified primary care patients who were potentially eligible for lung cancer screening (LCS). Identified patients were sent a patient portal invitation to visit a LCS interactive Web site which assessed screening eligibility and included a decision aid. The primary outcome was screening completion. Secondary outcomes included the proportion of patients who read the invitation, visited the interactive Web site, and completed the interactive Web site. RESULTS We sent portal invitations to 1,000 patients. Almost all patients (86%, 862/1,000) read the invitation, 404 (40%) patients visited the interactive Web site, and 349 patients (35%) completed it. Of the 99 patients who were confirmed screening eligible by the Web site, 81 made a screening decision (30% wanted screening, 44% unsure, 26% declined screening), and 22 patients had a chest computed tomography completed. CONCLUSION The digital outreach strategy reached the majority of patient portal users. While the study focused on LCS, this digital outreach approach could be generalized to other health needs. Given the broad reach and potential low cost of this digital strategy, future research should investigate best practices for implementing the system.
Collapse
Affiliation(s)
- Ajay Dharod
- Department of Internal Medicine, Section on General Internal Medicine, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States.,Department of Implementation Science, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States
| | - Christina Bellinger
- Department of Internal Medicine, Section on Pulmonary, Critical Care, Allergy and Immunology, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States
| | - Kristie Foley
- Department of Implementation Science, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States
| | - L Doug Case
- Department of Biostatistical Sciences, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States
| | - David Miller
- Department of Internal Medicine, Section on General Internal Medicine, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States.,Department of Implementation Science, Wake Forest School of Medicine, Wake Forest University, Winston-Salem, North Carolina, United States
| |
Collapse
|
108
|
Sharafoddini A, Dubin JA, Maslove DM, Lee J. A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study. JMIR Med Inform 2019; 7:e11605. [PMID: 30622091 PMCID: PMC6329436 DOI: 10.2196/11605] [Citation(s) in RCA: 30] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2018] [Revised: 10/30/2018] [Accepted: 10/30/2018] [Indexed: 01/08/2023] Open
Abstract
Background The data missing from patient profiles in intensive care units (ICUs) are substantial and unavoidable. However, this incompleteness is not always random or because of imperfections in the data collection process. Objective This study aimed to investigate the potential hidden information in data missing from electronic health records (EHRs) in an ICU and examine whether the presence or missingness of a variable itself can convey information about the patient health status. Methods Daily retrieval of laboratory test (LT) measurements from the Medical Information Mart for Intensive Care III database was set as our reference for defining complete patient profiles. Missingness indicators were introduced as a way of representing presence or absence of the LTs in a patient profile. Thereafter, various feature selection methods (filter and embedded feature selection methods) were used to examine the predictive power of missingness indicators. Finally, a set of well-known prediction models (logistic regression [LR], decision tree, and random forest) were used to evaluate whether the absence status itself of a variable recording can provide predictive power. We also examined the utility of missingness indicators in improving predictive performance when used with observed laboratory measurements as model input. The outcome of interest was in-hospital mortality and mortality at 30 days after ICU discharge. Results Regardless of mortality type or ICU day, more than 40% of the predictors selected by feature selection methods were missingness indicators. Notably, employing missingness indicators as the only predictors achieved reasonable mortality prediction on all days and for all mortality types (for instance, in 30-day mortality prediction with LR, we achieved area under the curve of the receiver operating characteristic [AUROC] of 0.6836±0.012). Including indicators with observed measurements in the prediction models also improved the AUROC; the maximum improvement was 0.0426. Indicators also improved the AUROC for Simplified Acute Physiology Score II model—a well-known ICU severity of illness score—confirming the additive information of the indicators (AUROC of 0.8045±0.0109 for 30-day mortality prediction for LR). Conclusions Our study demonstrated that the presence or absence of LT measurements is informative and can be considered a potential predictor of in-hospital and 30-day mortality. The comparative analysis of prediction models also showed statistically significant prediction improvement when indicators were included. Moreover, missing data might reflect the opinions of examining clinicians. Therefore, the absence of measurements can be informative in ICUs and has predictive power beyond the measured data themselves. This initial case study shows promise for more in-depth analysis of missing data and its informativeness in ICUs. Future studies are needed to generalize these results.
Collapse
Affiliation(s)
- Anis Sharafoddini
- Health Data Science Lab, School of Public Health and Health Systems, University of Waterloo, Waterloo, ON, Canada
| | - Joel A Dubin
- Health Data Science Lab, School of Public Health and Health Systems, University of Waterloo, Waterloo, ON, Canada.,Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, ON, Canada
| | - David M Maslove
- Department of Critical Care Medicine, Queen's University, Kingston, ON, Canada
| | - Joon Lee
- Health Data Science Lab, School of Public Health and Health Systems, University of Waterloo, Waterloo, ON, Canada.,Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.,Department of Cardiac Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| |
Collapse
|
109
|
Estiri H, Stephens KA, Klann JG, Murphy SN. Exploring completeness in clinical data research networks with DQe-c. J Am Med Inform Assoc 2019; 25:17-24. [PMID: 29069394 PMCID: PMC6481389 DOI: 10.1093/jamia/ocx109] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2017] [Accepted: 09/15/2017] [Indexed: 12/30/2022] Open
Abstract
Objective To provide an open source, interoperable, and scalable data quality assessment tool for evaluation and visualization of completeness and conformance in electronic health record (EHR) data repositories. Materials and Methods This article describes the tool’s design and architecture and gives an overview of its outputs using a sample dataset of 200 000 randomly selected patient records with an encounter since January 1, 2010, extracted from the Research Patient Data Registry (RPDR) at Partners HealthCare. All the code and instructions to run the tool and interpret its results are provided in the Supplementary Appendix. Results DQe-c produces a web-based report that summarizes data completeness and conformance in a given EHR data repository through descriptive graphics and tables. Results from running the tool on the sample RPDR data are organized into 4 sections: load and test details, completeness test, data model conformance test, and test of missingness in key clinical indicators. Discussion Open science, interoperability across major clinical informatics platforms, and scalability to large databases are key design considerations for DQe-c. Iterative implementation of the tool across different institutions directed us to improve the scalability and interoperability of the tool and find ways to facilitate local setup. Conclusion EHR data quality assessment has been hampered by implementation of ad hoc processes. The architecture and implementation of DQe-c offer valuable insights for developing reproducible and scalable data science tools to assess, manage, and process data in clinical data repositories.
Collapse
Affiliation(s)
- Hossein Estiri
- Harvard Medical School.,Massachusetts General Hospital.,Partners HealthCare, Boston, MA, USA
| | - Kari A Stephens
- Department of Biomedical Informatics and Medical Education.,Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Jeffrey G Klann
- Harvard Medical School.,Massachusetts General Hospital.,Partners HealthCare, Boston, MA, USA
| | - Shawn N Murphy
- Harvard Medical School.,Massachusetts General Hospital.,Partners HealthCare, Boston, MA, USA
| |
Collapse
|
110
|
Khoong EC, Cherian R, Rivadeneira NA, Gourley G, Yazdany J, Amarnath A, Schillinger D, Sarkar U. Accurate Measurement In California's Safety-Net Health Systems Has Gaps And Barriers. Health Aff (Millwood) 2018; 37:1760-1769. [PMID: 30395496 DOI: 10.1377/hlthaff.2018.0709] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
Patient safety in ambulatory care has not been routinely measured. California implemented a pay-for-performance program in safety-net hospitals that incentivized measurement and improvement in key areas of ambulatory safety: referral completion, medication safety, and test follow-up. We present two years of program data (collected during July 2015-June 2017) and show both suboptimal performance in aspects of ambulatory safety and questionable reliability in data reporting. Performance was better in areas that required limited coordination or patient engagement-for example, annual medication monitoring versus follow-up after high-risk mammograms. Health care systems that lack seamlessly integrated electronic health records and patient registries encountered barriers to reporting reliable ambulatory safety data, particularly for measures that integrated multiple data elements. These data challenges precluded accurate performance measurement in many areas. Policy makers and safety advocates need to support the development of information systems and measures that facilitate the accurate ascertainment of the health systems, patients, and clinical tasks at greatest risk for ambulatory safety failures.
Collapse
Affiliation(s)
- Elaine C Khoong
- Elaine C. Khoong ( ) is a primary care research fellow in the Division of General Internal Medicine, University of California San Francisco (UCSF), and the Zuckerberg San Francisco General Hospital and Trauma Center
| | - Roy Cherian
- Roy Cherian is a research data analyst at the Center for Vulnerable Populations, UCSF, and the Zuckerberg San Francisco General Hospital and Trauma Center
| | - Natalie A Rivadeneira
- Natalie A. Rivadeneira is a research data analyst at the Center for Vulnerable Populations, UCSF, and the Zuckerberg San Francisco General Hospital and Trauma Center
| | - Gato Gourley
- Gato Gourley is a project coordinator at the Center for Vulnerable Populations, UCSF, and the Zuckerberg San Francisco General Hospital and Trauma Center
| | - Jinoos Yazdany
- Jinoos Yazdany is an associate professor of medicine in the Division of Rheumatology, UCSF
| | - Ashrith Amarnath
- Ashrith Amarnath is a patient safety officer at the Sutter Medical Foundation and a former patient safety officer in the Office of the Medical Director, Department of Health Care Services, both in Sacramento, California
| | - Dean Schillinger
- Dean Schillinger is a professor of medicine at UCSF and chief of the Division of General Internal Medicine at Zuckerberg San Francisco General Hospital and Trauma Center
| | - Urmimala Sarkar
- Urmimala Sarkar is an associate professor of medicine in the Division of General Internal Medicine, UCSF, and a primary care physician at Zuckerberg San Francisco General Hospital's Richard H. Fine People's Clinic
| |
Collapse
|
111
|
Kruse CS, Stein A, Thomas H, Kaur H. The use of Electronic Health Records to Support Population Health: A Systematic Review of the Literature. J Med Syst 2018; 42:214. [PMID: 30269237 PMCID: PMC6182727 DOI: 10.1007/s10916-018-1075-6] [Citation(s) in RCA: 144] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2018] [Accepted: 09/19/2018] [Indexed: 12/16/2022]
Abstract
Electronic health records (EHRs) have emerged among health information technology as "meaningful use" to improve the quality and efficiency of healthcare, and health disparities in population health. In other instances, they have also shown lack of interoperability, functionality and many medical errors. With proper implementation and training, are electronic health records a viable source in managing population health? The primary objective of this systematic review is to assess the relationship of electronic health records' use on population health through the identification and analysis of facilitators and barriers to its adoption for this purpose. Authors searched Cumulative Index of Nursing and Allied Health Literature (CINAHL) and MEDLINE (PubMed), 10/02/2012-10/02/2017, core clinical/academic journals, MEDLINE full text, English only, human species and evaluated the articles that were germane to our research objective. Each article was analyzed by multiple reviewers. Group members recognized common facilitators and barriers associated with EHRs effect on population health. A final list of articles was selected by the group after three consensus meetings (n = 55). Among a total of 26 factors identified, 63% (147/232) of those were facilitators and 37% (85/232) barriers. About 70% of the facilitators consisted of productivity/efficiency in EHRs occurring 33 times, increased quality and data management each occurring 19 times, surveillance occurring 17 times, and preventative care occurring 15 times. About 70% of the barriers consisted of missing data occurring 24 times, no standards (interoperability) occurring 13 times, productivity loss occurring 12 times, and technology too complex occurring 10 times. The analysis identified more facilitators than barriers to the use of the EHR to support public health. Wider adoption of the EHR and more comprehensive standards for interoperability will only enhance the ability for the EHR to support this important area of surveillance and disease prevention. This review identifies more facilitators than barriers to using the EHR to support public health, which implies a certain level of usability and acceptance to use the EHR in this manner. The public-health industry should combine their efforts with the interoperability projects to make the EHR both fully adopted and fully interoperable. This will greatly increase the availability, accuracy, and comprehensiveness of data across the country, which will enhance benchmarking and disease surveillance/prevention capabilities.
Collapse
Affiliation(s)
- Clemens Scott Kruse
- Texas State University, 601 University Dr, Encino 250, San Marcos, TX, 78666, USA.
| | - Anna Stein
- Texas State University, 601 University Dr, Encino 250, San Marcos, TX, 78666, USA
| | - Heather Thomas
- Texas State University, 601 University Dr, Encino 250, San Marcos, TX, 78666, USA
| | - Harmander Kaur
- Texas State University, 601 University Dr, Encino 250, San Marcos, TX, 78666, USA
| |
Collapse
|
112
|
Koziatek CA, Simon E, Horwitz LI, Makarov DV, Smith SW, Jones S, Gyftopoulos S, Swartz JL. Automated Pulmonary Embolism Risk Classification and Guideline Adherence for Computed Tomography Pulmonary Angiography Ordering. Acad Emerg Med 2018; 25:1053-1061. [PMID: 29710413 DOI: 10.1111/acem.13442] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2018] [Revised: 04/18/2018] [Accepted: 04/23/2018] [Indexed: 11/29/2022]
Abstract
BACKGROUND The assessment of clinical guideline adherence for the evaluation of pulmonary embolism (PE) via computed tomography pulmonary angiography (CTPA) currently requires either labor-intensive, retrospective chart review or prospective collection of PE risk scores at the time of CTPA order. The recording of clinical data in a structured manner in the electronic health record (EHR) may make it possible to automate the calculation of a patient's PE risk classification and determine whether the CTPA order was guideline concordant. OBJECTIVES The objective of this study was to measure the performance of automated, structured data-only versions of the Wells and revised Geneva risk scores in emergency department (ED) encounters during which a CTPA was ordered. The hypothesis was that such an automated method would classify a patient's PE risk with high accuracy compared to manual chart review. METHODS We developed automated, structured data-only versions of the Wells and revised Geneva risk scores to classify 212 ED encounters during which a CTPA was performed as "PE likely" or "PE unlikely." We then combined these classifications with D-dimer ordering data to assess each encounter as guideline concordant or discordant. The accuracy of these automated classifications and assessments of guideline concordance were determined by comparing them to classifications and concordance based on the complete Wells and revised Geneva scores derived via abstractor manual chart review. RESULTS The automatically derived Wells and revised Geneva risk classifications were 91.5 and 92% accurate compared to the manually determined classifications, respectively. There was no statistically significant difference between guideline adherence calculated by the automated scores compared to manual chart review (Wells, 70.8% vs. 75%, p = 0.33; revised Geneva, 65.6% vs. 66%, p = 0.92). CONCLUSION The Wells and revised Geneva score risk classifications can be approximated with high accuracy using automated extraction of structured EHR data elements in patients who received a CTPA. Combining these automated scores with D-dimer ordering data allows for the automated assessment of clinical guideline adherence for CTPA ordering in the ED, without the burden of manual chart review.
Collapse
Affiliation(s)
- Christian A. Koziatek
- Ronald O. Perelman Department of Emergency Medicine NYU School of Medicine New York NY
| | - Emma Simon
- Department of Population Health NYU School of Medicine New York NY
- Center for Healthcare Innovation and Delivery Science NYU School of Medicine New York NY
| | - Leora I. Horwitz
- Department of Population Health NYU School of Medicine New York NY
- Center for Healthcare Innovation and Delivery Science NYU School of Medicine New York NY
- Department of Medicine NYU School of Medicine New York NY
| | - Danil V. Makarov
- Department of Population Health NYU School of Medicine New York NY
- Department of Urology NYU School of Medicine New York NY
| | - Silas W. Smith
- Ronald O. Perelman Department of Emergency Medicine NYU School of Medicine New York NY
| | - Simon Jones
- Department of Population Health NYU School of Medicine New York NY
| | | | - Jordan L. Swartz
- Ronald O. Perelman Department of Emergency Medicine NYU School of Medicine New York NY
| |
Collapse
|
113
|
Finazzi S, Mandelli G, Garbero E, Mondini M, Trussardi G, Giardino M, Tavola M, Bertolini G. Data collection and research with MargheritaTre. Physiol Meas 2018; 39:084004. [PMID: 29972378 DOI: 10.1088/1361-6579/aad10f] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE MargheritaTre is an electronic health record developed by the Italian Group for the Evaluation of Interventions in Intensive Care Medicine designed to support clinical practice in intensive care units (ICUs) and ensure high-quality data for research purposes. APPROACH MargheritaTre was developed in collaboration with clinical experts, researchers, and IT specialists. It is currently installed in 40 ICUs and its database contains complete records of more than 65,000 patients. To facilitate data analysis, information is mostly stored in structured or partially structured form. MAIN RESULTS Data collected with MargheritaTre allow one to conduct research studies on complex clinical problems from manifold perspectives and with different levels of detail, such as epidemiological studies, analyses of the process of care and physiopathological investigations, at both single-organ and organism level. In this paper we describe some of the first projects based on this electronic health record to illustrate its potential for research. SIGNIFICANCE The MargheritaTre database is a huge and rapidly growing mine of data that will be exploited by our laboratory and shared with other groups to address complex and innovative research and clinical questions. The ultimate aim of these projects is the improvement of the quality of care and patient outcomes, through the development of expert systems integrated in the electronic health record to support clinical practice.
Collapse
Affiliation(s)
- Stefano Finazzi
- Mario Negri Institute for Pharmacological Research IRCCS, Villa Camozzi, Via G.B. Camozzi, Ranica (BG), Italy
| | | | | | | | | | | | | | | |
Collapse
|
114
|
Ding J, Johnson CE, Cook A. How We Should Assess the Delivery of End-Of-Life Care in General Practice? A Systematic Review. J Palliat Med 2018; 21:1790-1805. [PMID: 30129811 DOI: 10.1089/jpm.2018.0194] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/16/2023] Open
Abstract
BACKGROUND The majority of end-of-life (EOL) care occurs in general practice. However, we still have little knowledge about how this care is delivered or how it can be assessed and supported. AIM (i) To review the existing evaluation tools used for assessment of the delivery of EOL care from the perspective of general practice; (ii) To describe how EOL care is provided in general practice; (iii) To identify major areas of concern in providing EOL care in this context. DESIGN A systematic review. DATA SOURCES Systematic searches of major electronic databases (Medline, EMBASE, PsycINFO, and CINAHL) from inception to 2017 were used to identify evaluation tools focusing on organizational structures/systems and process of end-of-life care from a general practice perspective. RESULTS A total of 43 studies representing nine evaluation tools were included. A relatively restricted focus and lack of validation were common limitations. Key general practitioner (GP) activities assessed by the evaluation tools were summarized and the main issues in current GP EOL care practice were identified. CONCLUSIONS The review of evaluation tools revealed that GPs are highly involved in management of patients at the EOL, but there are a range of issues relating to the delivery of care. An EOL care registration system integrated with electronic health records could provide an optimal approach to address the concerns about recall bias and time demands in retrospective analyses. Such a system should ideally capture the core GP activities and any major issues in care provision on a case-by-case basis.
Collapse
Affiliation(s)
- Jinfeng Ding
- 1 School of Population and Global Health, University of Western Australia , Perth, Western Australia, Australia
| | - Claire E Johnson
- 2 Cancer and Palliative Care Research and Evaluation Unit (CaPCREU), Medical School, University of Western Australia , Perth, Western Australia, Australia
- 3 School of Nursing and Midwifery, Monash University , Melbourne, Victoria, Australia
| | - Angus Cook
- 1 School of Population and Global Health, University of Western Australia , Perth, Western Australia, Australia
| |
Collapse
|
115
|
Burm S, Boese K, Faden L, DeLuca S, Huda N, Hibbert K, Goldszmidt M. Recognising the importance of informal communication events in improving collaborative care. BMJ Qual Saf 2018; 28:289-295. [DOI: 10.1136/bmjqs-2017-007441] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2017] [Revised: 07/23/2018] [Accepted: 07/26/2018] [Indexed: 11/03/2022]
Abstract
BackgroundWhile the concept of collaboration is highly touted in the literature, most descriptions of effective collaboration highlight formal collaborative events; largely ignored are the informal collaborative events and none focusing on the frequent, ‘seemingly’ by chance communication events that arise and their role in supporting patient safety and quality care.ObjectiveTo identify the types of informal communication events that exist in the inpatient setting and better understand the barriers contributing to their necessity.MethodsWe undertook a constructivist grounded theory study in an inpatient internal medicine teaching unit in Ontario, Canada. Interview and observational data were collected across two phases; in total, 56 participants were consented for the study. Data collection and analysis occurred iteratively; themes were identified using constant comparison methods.ResultsSeveral types of informal communication events were identified and appeared valuable in three ways: (1) providing a better sense of a patient’s baseline function in comparison to their current function; (2) gaining a more holistic understanding of the patient’s needs; and (3) generating better insight into a patient’s wishes and goals of care. Participants identified a number of organisational and communication challenges leading to the need for informal communication events. These included: scheduling, competing demands and the spatial and temporal organisation of the ward. As a result, nursing staff, allied health professionals and caregivers had to develop strategies for interacting with the physician team.ConclusionWe highlight the importance of informal communication in supporting patient care and the gaps in the system contributing to their necessity. Changes at the system level are needed to ensure we are not leaving important collaborative opportunities to chance alone.
Collapse
|
116
|
DeMellow J, Kim TY. Technology-enabled performance monitoring in intensive care: An integrative literature review. Intensive Crit Care Nurs 2018; 48:42-51. [PMID: 30054118 DOI: 10.1016/j.iccn.2018.07.003] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2017] [Revised: 05/20/2018] [Accepted: 07/07/2018] [Indexed: 10/28/2022]
Abstract
BACKGROUND Implementation of evidence-based bundles in intensive care units is integral to improving quality of care and patient outcomes. However, it increases the burden of data collection and analysis required for performance monitoring and feedback of an inter-disciplinary care team. Health information technology including electronic health records and data analytic tools could automate this process and provide real-time feedback to the team. AIM This integrative literature review aimed to examine the extent to which technology-enabled performance monitoring and feedback contributed to improving quality of care and patient outcomes when implementing evidence-based bundles. METHODS A literature search of scientific databases was conducted using PubMed, Embase, Scopus, CINHAL and Ovid Medline. RESULTS Of nine studies included in this review, all reported improved compliance of the team with evidence-based bundles, ranging from 3% to 60% post implementation of technology-enabled performance monitoring and feedback. Significant reductions (p < .05) in hospital acquired infections were also reported in five studies. CONCLUSIONS Overall, the addition of documentation fields to electronic health records was essential in providing real-time feedback to teams and improving their compliance with evidence-based bundles. Further research is needed to assess the effectiveness of technology-enabled performance monitoring and feedback in improving patient outcomes on a larger scale, especially in resource-limited settings such as community hospitals.
Collapse
Affiliation(s)
- Jacqueline DeMellow
- University of California, Davis, Betty Irene Moore School of Nursing, 2450 48th Street, Suite 2600, Sacramento, CA 95817, United States.
| | - Tae Youn Kim
- University of California, Davis, Betty Irene Moore School of Nursing, 2450 48th Street, Suite 2600, Sacramento, CA 95817, United States
| |
Collapse
|
117
|
Godfrey EM, West II, Holmes J, Keppel GA, Baldwin LM. Use of an electronic health record data sharing system for identifying current contraceptive use within the WWAMI region Practice and Research Network. Contraception 2018; 98:476-481. [PMID: 29936151 DOI: 10.1016/j.contraception.2018.06.005] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2018] [Revised: 06/08/2018] [Accepted: 06/14/2018] [Indexed: 12/23/2022]
Abstract
OBJECTIVE To evaluate the ability of electronic health record (EHR) data extracted into a data-sharing system to accurately identify contraceptive use. STUDY DESIGN We compared rates of contraceptive use from electronic extraction of EHR data via a data-sharing system and manual abstraction of the EHR among 142 female patients ages 15-49 years from a family medicine clinic within a primary care practice-based research network (PBRN). Cohen's kappa coefficient measured agreement between electronic extraction and manual abstraction. RESULTS Manual abstraction identified 62% of women as contraceptive users, whereas electronic extraction identified only 27%. Long acting reversible (LARC) methods had 96% agreement (Cohen's kappa 0.78; confidence interval, 0.57-0.99) between electronic extraction and manual abstraction. EHR data extracted via a data-sharing system was unable to identify barrier or over-the-counter contraceptives. CONCLUSIONS Electronic extraction found substantially lower overall rates of contraceptive method use, but produced more comparable LARC method use rates when compared to manual abstraction among women in this study's primary care clinic. IMPLICATIONS Quality metrics related to contraceptive use that rely on EHR data in this study's data-sharing system likely under-estimated true contraceptive use.
Collapse
Affiliation(s)
- Emily M Godfrey
- Department of Family Medicine, University of Washington, Box 354982, Seattle, WA 98105, USA; Department of Obstetrics and Gynecology, University of Washington, Box 356460, Seattle, WA 98195, USA.
| | - Imara I West
- Department of Family Medicine, University of Washington, Box 354982, Seattle, WA 98105, USA
| | - John Holmes
- Departments of Pharmacy Practice and Family Medicine, Idaho State University, 465 Memorial Drive, Pocatello, ID 83201, USA
| | - Gina A Keppel
- Department of Family Medicine, University of Washington, Box 354982, Seattle, WA 98105, USA; Institute of Translational Health Sciences, Box 357184, Seattle, WA 98195, USA
| | - Laura-Mae Baldwin
- Department of Family Medicine, University of Washington, Box 354982, Seattle, WA 98105, USA; Institute of Translational Health Sciences, Box 357184, Seattle, WA 98195, USA
| |
Collapse
|
118
|
D'Amore JD, Li C, McCrary L, Niloff JM, Sittig DF, McCoy AB, Wright A. Using Clinical Data Standards to Measure Quality: A New Approach. Appl Clin Inform 2018; 9:422-431. [PMID: 29898468 PMCID: PMC5999523 DOI: 10.1055/s-0038-1656548] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/08/2023] Open
Abstract
Background
Value-based payment for care requires the consistent, objective calculation of care quality. Previous initiatives to calculate ambulatory quality measures have relied on billing data or individual electronic health records (EHRs) to calculate and report performance. New methods for quality measure calculation promoted by federal regulations allow qualified clinical data registries to report quality outcomes based on data aggregated across facilities and EHRs using interoperability standards.
Objective
This research evaluates the use of clinical document interchange standards as the basis for quality measurement.
Methods
Using data on 1,100 patients from 11 ambulatory care facilities and 5 different EHRs, challenges to quality measurement are identified and addressed for 17 certified quality measures.
Results
Iterative solutions were identified for 14 measures that improved patient inclusion and measure calculation accuracy. Findings validate this approach to improving measure accuracy while maintaining measure certification.
Conclusion
Organizations that report care quality should be aware of how identified issues affect quality measure selection and calculation. Quality measure authors should consider increasing real-world validation and the consistency of measure logic in respect to issues identified in this research.
Collapse
Affiliation(s)
- John D D'Amore
- Diameter Health, Inc., Farmington, Connecticut, United States.,Boston University Metropolitan College, Boston University, Boston, Massachusetts, United States
| | - Chun Li
- Diameter Health, Inc., Farmington, Connecticut, United States
| | - Laura McCrary
- Kansas Health Information Network, Topeka, Kansas, United States
| | | | - Dean F Sittig
- School of Biomedical Informatics, University of Texas-Memorial Hermann Center for Healthcare Quality and Safety, University of Texas Health Science Center, Houston, Texas, United States
| | - Allison B McCoy
- Department of Global Biostatistics and Data Science, Tulane University School of Public Health and Tropical Medicine, New Orleans, Louisiana, United States
| | - Adam Wright
- Division of General Internal Medicine, Brigham and Women's Hospital, Boston, Massachusetts, United States
| |
Collapse
|
119
|
Verheij RA, Curcin V, Delaney BC, McGilchrist MM. Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse. J Med Internet Res 2018; 20:e185. [PMID: 29844010 PMCID: PMC5997930 DOI: 10.2196/jmir.9134] [Citation(s) in RCA: 150] [Impact Index Per Article: 25.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2017] [Revised: 02/11/2018] [Accepted: 03/01/2018] [Indexed: 01/26/2023] Open
Abstract
BACKGROUND Enormous amounts of data are recorded routinely in health care as part of the care process, primarily for managing individual patient care. There are significant opportunities to use these data for other purposes, many of which would contribute to establishing a learning health system. This is particularly true for data recorded in primary care settings, as in many countries, these are the first place patients turn to for most health problems. OBJECTIVE In this paper, we discuss whether data that are recorded routinely as part of the health care process in primary care are actually fit to use for other purposes such as research and quality of health care indicators, how the original purpose may affect the extent to which the data are fit for another purpose, and the mechanisms behind these effects. In doing so, we want to identify possible sources of bias that are relevant for the use and reuse of these type of data. METHODS This paper is based on the authors' experience as users of electronic health records data, as general practitioners, health informatics experts, and health services researchers. It is a product of the discussions they had during the Translational Research and Patient Safety in Europe (TRANSFoRm) project, which was funded by the European Commission and sought to develop, pilot, and evaluate a core information architecture for the learning health system in Europe, based on primary care electronic health records. RESULTS We first describe the different stages in the processing of electronic health record data, as well as the different purposes for which these data are used. Given the different data processing steps and purposes, we then discuss the possible mechanisms for each individual data processing step that can generate biased outcomes. We identified 13 possible sources of bias. Four of them are related to the organization of a health care system, whereas some are of a more technical nature. CONCLUSIONS There are a substantial number of possible sources of bias; very little is known about the size and direction of their impact. However, anyone that uses or reuses data that were recorded as part of the health care process (such as researchers and clinicians) should be aware of the associated data collection process and environmental influences that can affect the quality of the data. Our stepwise, actor- and purpose-oriented approach may help to identify these possible sources of bias. Unless data quality issues are better understood and unless adequate controls are embedded throughout the data lifecycle, data-driven health care will not live up to its expectations. We need a data quality research agenda to devise the appropriate instruments needed to assess the magnitude of each of the possible sources of bias, and then start measuring their impact. The possible sources of bias described in this paper serve as a starting point for this research agenda.
Collapse
Affiliation(s)
- Robert A Verheij
- Netherlands Institute for Health Services Research, Utrecht, Netherlands
| | - Vasa Curcin
- King's College London, London, United Kingdom
| | - Brendan C Delaney
- Imperial College London, Imperial College Business School, London, United Kingdom
| | - Mark M McGilchrist
- University of Dundee, Department of Public Health Sciences, Dundee, United Kingdom
| |
Collapse
|
120
|
Dufour É, Duhoux A, Contandriopoulos D. Reliability of a Canadian database for primary care nursing services' clinical and administrative data. Int J Med Inform 2018; 117:1-5. [PMID: 30032957 DOI: 10.1016/j.ijmedinf.2018.05.007] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2018] [Revised: 05/04/2018] [Accepted: 05/20/2018] [Indexed: 11/19/2022]
Abstract
BACKGROUND The use of electronic clinical and administrative data can be an advantageous source of information for assessing nursing performance in primary care. In Québec (Canada), the I-CLSC electronic database could be used to measure performance indicators. However, little is known about the reliability of the data contained in this database. The objective of this study was to assess the reliability of the clinical and administrative data contained in the I-CLSC electronic database based on the data entered in medical records. METHODS We used a longitudinal design for this study. A sample of 100 patients who had experienced 107 episodes of wound care were randomly selected from all patients who had two or more consultations during the year 2015. The paper records were used as reference. We collected data regarding eight nursing sensitive indicators from both sources. We assessed the concordance between the electronic data and the paper records by measuring inter-rater agreement. RESULTS Six of the eight indicators showed a percentage agreement ≥ 85%, and kappa scores between 0.7 and 1.00 (p < 0.001), indicating high to perfect levels of agreement between the two data sources. Two indicators presented fair kappa scores. CONCLUSION This database provides reliable data relating to the organization of care but shows lower reliability for specific acts performed by nurses in primary care. This existing database can be used to assess, manage and improve certain dimensions of nursing performance in primary care.
Collapse
Affiliation(s)
- Émilie Dufour
- Faculty of Nursing, Université de Montréal, Marguerite-d'Youville Campus, Montréal, QC, H3C 3J7, Canada.
| | - Arnaud Duhoux
- Faculty of Nursing, Université de Montréal, Marguerite-d'Youville Campus, Montréal, QC, H3C 3J7, Canada; CR-CSIS (Centre de recherche Charles-Le Moyne - Saguenay-Lac-Saint-Jean sur les innovations en santé), Université de Sherbrooke, Longueuil Campus, Longueuil, QC, J4K 0A8, Canada
| | | |
Collapse
|
121
|
Lesselroth BJ, Adams K, Church VL, Tallett S, Russ Y, Wiedrick J, Forsberg C, Dorr DA. Evaluation of Multimedia Medication Reconciliation Software: A Randomized Controlled, Single-Blind Trial to Measure Diagnostic Accuracy for Discrepancy Detection. Appl Clin Inform 2018; 9:285-301. [PMID: 29719884 DOI: 10.1055/s-0038-1645889] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023] Open
Abstract
BACKGROUND The Veterans Affairs Portland Healthcare System developed a medication history collection software that displays prescription names and medication images. OBJECTIVE This article measures the frequency of medication discrepancy reporting using the medication history collection software and compares with the frequency of reporting using a paper-based process. This article also determines the accuracy of each method by comparing both strategies to a best possible medication history. STUDY DESIGN Randomized, controlled, single-blind trial. SETTING Three community-based primary care clinics associated with the Veterans Affairs Portland Healthcare System: a 300-bed teaching facility and ambulatory care network serving Veteran soldiers in the Pacific Northwest United States. PARTICIPANTS Of 212 patients with primary care appointments, 209 patients fulfilled the study requirements. INTERVENTION Patients randomized to a software-directed medication history or a paper-based medication history. Randomization and allocation to treatment groups were performed using a computer-based random number generator. Assignments were placed in a sealed envelope and opened after participant consent. The research coordinator did not know or have access to the treatment assignment until the time of presentation. MAIN OUTCOME MEASURES The primary analysis compared the discrepancy detection rates between groups with respect to the health record and a best possible medication history. RESULTS Of 3,500 medications reviewed, we detected 1,435 discrepancies. Forty-six percent of those discrepancies were potentially high risk for causing an adverse drug event. There was no difference in detection rates between treatment arms. Software sensitivity was 83% and specificity was 91%; paper sensitivity was 81% and specificity was 94%. No participants were lost to follow-up. CONCLUSION The medication history collection software is an efficient and scalable method for gathering a medication history and detecting high-risk discrepancies. Although it included medication images, the technology did not improve accuracy over a paper list when compared with a best possible medication history. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT02135731.
Collapse
Affiliation(s)
- Blake J Lesselroth
- NorthWest Innovation Center, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States.,Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon, United States
| | - Kathleen Adams
- NorthWest Innovation Center, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States
| | - Victoria L Church
- NorthWest Innovation Center, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States
| | - Stephanie Tallett
- NorthWest Innovation Center, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States
| | - Yelizaveta Russ
- Division of Primary Care, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States
| | - Jack Wiedrick
- Oregon Clinical and Translational Research Institute, Oregon Health and Science University, Portland, Oregon, United States
| | - Christopher Forsberg
- Center of Innovation, Veterans' Affairs Portland Healthcare System, Portland, Oregon, United States
| | - David A Dorr
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon, United States
| |
Collapse
|
122
|
Tung EL, Wroblewski KE, Boyd K, Makelarski JA, Peek ME, Lindau ST. Police-Recorded Crime and Disparities in Obesity and Blood Pressure Status in Chicago. J Am Heart Assoc 2018; 7:e008030. [PMID: 29574461 PMCID: PMC5907588 DOI: 10.1161/jaha.117.008030] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/14/2017] [Accepted: 02/22/2018] [Indexed: 01/12/2023]
Abstract
BACKGROUND The purpose of this study was to examine associations between several types of police-recorded crime (violent, nonviolent, and homicide) and cardiometabolic health (obesity and elevated blood pressure [BP]), and to determine if associations were modified by age and sex. METHODS AND RESULTS We analyzed cross-sectional data (N=14 799 patients) from 3 primary care clinics at an academic medical center in Chicago, IL. Patient-level health data were obtained from the electronic health record (June 1, 2014-May 31, 2015), including body mass index and BP, and linked to the City of Chicago Police Data Portal. Geocoded crime counts were aggregated to census tract and calculated as the annual crime rate per 1000 population. Generalized linear mixed models were used to assess obesity and BP status as a function of crime rate quartile, controlling for patient, clinic, and neighborhood characteristics. Median violent crime rates in each quartile ranged from 15 to 84 per 1000 population. Median age was 56 years (interquartile range, 38-72 years); 42% of patients were obese and 33% had elevated BP. Compared with patients living in the lowest quartile, patients living in the highest quartile for violent crime had 53% higher adjusted odds of obesity (95% confidence interval, 1.15-2.03) and 25% higher adjusted odds of elevated BP (95% confidence interval, 1.01-1.56). In subanalyses examining homicide, a relatively rare event, exposure was not associated with obesity and was inconsistently associated with elevated BP. CONCLUSIONS In a densely populated, high-poverty region in Chicago, recurrent exposure to high rates of violent crime was consistently associated with obesity and elevated BP, but rare exposure to homicide was not.
Collapse
Affiliation(s)
- Elizabeth L Tung
- Section of General Internal Medicine and Chicago Center for Diabetes Translation Research, University of Chicago, IL
| | | | - Kelly Boyd
- Department of Obstetrics and Gynecology, University of Chicago, IL
| | | | - Monica E Peek
- Section of General Internal Medicine and Chicago Center for Diabetes Translation Research, University of Chicago, IL
- MacLean Center for Clinical Medical Ethics, University of Chicago, IL
| | - Stacy Tessler Lindau
- Department of Obstetrics and Gynecology, University of Chicago, IL
- Department of Medicine-Geriatrics, MacLean Center on Clinical Medical Ethics and Comprehensive Cancer Center, University of Chicago, IL
| |
Collapse
|
123
|
Jackson R, Patel R, Velupillai S, Gkotsis G, Hoyle D, Stewart R. Knowledge discovery for Deep Phenotyping serious mental illness from Electronic Mental Health records. F1000Res 2018; 7:210. [PMID: 29899974 PMCID: PMC5968362 DOI: 10.12688/f1000research.13830.2] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 04/30/2018] [Indexed: 11/23/2022] Open
Abstract
Background: Deep Phenotyping is the precise and comprehensive analysis of phenotypic features in which the individual components of the phenotype are observed and described. In UK mental health clinical practice, most clinically relevant information is recorded as free text in the Electronic Health Record, and offers a granularity of information beyond what is expressed in most medical knowledge bases. The SNOMED CT nomenclature potentially offers the means to model such information at scale, yet given a sufficiently large body of clinical text collected over many years, it is difficult to identify the language that clinicians favour to express concepts. Methods: By utilising a large corpus of healthcare data, we sought to make use of semantic modelling and clustering techniques to represent the relationship between the clinical vocabulary of internationally recognised SMI symptoms and the preferred language used by clinicians within a care setting. We explore how such models can be used for discovering novel vocabulary relevant to the task of phenotyping Serious Mental Illness (SMI) with only a small amount of prior knowledge. Results: 20 403 terms were derived and curated via a two stage methodology. The list was reduced to 557 putative concepts based on eliminating redundant information content. These were then organised into 9 distinct categories pertaining to different aspects of psychiatric assessment. 235 concepts were found to be expressions of putative clinical significance. Of these, 53 were identified having novel synonymy with existing SNOMED CT concepts. 106 had no mapping to SNOMED CT. Conclusions: We demonstrate a scalable approach to discovering new concepts of SMI symptomatology based on real-world clinical observation. Such approaches may offer the opportunity to consider broader manifestations of SMI symptomatology than is typically assessed via current diagnostic frameworks, and create the potential for enhancing nomenclatures such as SNOMED CT based on real-world expressions.
Collapse
Affiliation(s)
- Richard Jackson
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, SE5 8AF, UK
| | - Rashmi Patel
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, SE5 8AF, UK.,South London and Maudsley NHS Foundation Trust, London, SE5 8AZ, UK
| | - Sumithra Velupillai
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, SE5 8AF, UK.,School of Computer Science and Communication, TH Royal Institute of Technology, Stockholm, SE-100 44, Sweden
| | - George Gkotsis
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, SE5 8AF, UK
| | | | - Robert Stewart
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, SE5 8AF, UK.,South London and Maudsley NHS Foundation Trust, London, SE5 8AZ, UK
| |
Collapse
|
124
|
Hübner U, Liebe JD. Developing and Trialling an Independent, Scalable and Repeatable IT-benchmarking Procedure for Healthcare Organisations. Methods Inf Med 2018; 52:360-9. [DOI: 10.3414/me12-02-0016] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2012] [Accepted: 06/27/2013] [Indexed: 11/09/2022]
Abstract
SummaryBackground: Continuous improvements of IT-performance in healthcare organisations require actionable performance indicators, regularly conducted, independent measurements and meaningful and scalable reference groups. Existing IT-benchmarking initiatives have focussed on the development of reliable and valid indicators, but less on the questions about how to implement an environment for conducting easily repeatable and scalable IT-benchmarks.Objectives: This study aims at developing and trialling a procedure that meets the afore-mentioned requirements.Methods: We chose a well established, regularly conducted (inter-) national IT-survey of healthcare organisations (IT-Report Health-care) as the environment and offered the participants of the 2011 survey (CIOs of hospitals) to enter a benchmark. The 61 structural and functional performance indicators cov -ered among others the implementation status and integration of IT-systems and functions, global user satisfaction and the resources of the IT-department. Healthcare organisations were grouped by size and ownership. The benchmark results were made available electronically and feedback on the use of these results was requested after several months.Results: Fifty-nine hospitals participated in the benchmarking. Reference groups consisted of up to 141 members depending on the number of beds (size) and the ownership (public vs. private). A total of 122 charts showing single indicator frequency views were sent to each participant. The evaluation showed that 94.1% of the CIOs who participated in the evaluation considered this benchmarking beneficial and reported that they would enter again. Based on the feedback of the participants we developed two additional views that provide a more consolidated picture.Conclusion: The results demonstrate that establishing an independent, easily repeatable and scalable IT-benchmarking procedure is possible and was deemed desirable. Based on these encouraging results a new benchmarking round which includes process indicators is currently conducted.
Collapse
|
125
|
Cole AM, Pflugeisen B, Schwartz MR, Miller SC. Cross sectional study to assess the accuracy of electronic health record data to identify patients in need of lung cancer screening. BMC Res Notes 2018; 11:14. [PMID: 29321038 PMCID: PMC5763525 DOI: 10.1186/s13104-018-3124-0] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2016] [Accepted: 01/03/2018] [Indexed: 11/10/2022] Open
Abstract
Objective Lung cancer is the leading cause of cancer death in the United States [Siegel et al. in CA Cancer J Clin 66:7–30, 1]. However, evidence from clinical trials indicates that annual low-dose computed tomography screening reduces lung cancer mortality [Humphrey et al. in Ann Intern Med 159:411–420, 2]. The objective of this study is to report results of a study designed to assess the sensitivity, specificity, and positive and negative predictive value of an electronic health record (EHR) query in comparison to patient self-report, to identify patients who may benefit from lung cancer screening. Cross sectional study comparing patient self report to EHR derived assessment of tobacco status and need for lung cancer screening. We invited 200 current or former smokers, ages 55–80 to complete a brief paper survey. 26 responded and 24 were included in the analysis. Results For 30% of respondents, there was not adequate EHR data to make a lung cancer screening determination. Compared to patient self-report, EHR derived data has a 67% sensitivity and 82% specificity for identifying patients that meet criteria for lung cancer screening. While the degree of accuracy may be insufficient to make a final lung cancer screening determination, EHR data may be useful in prompting clinicians to initiate conversations with patients in regards to lung cancer screening.
Collapse
Affiliation(s)
- Allison M Cole
- Department of Family Medicine, University of Washington, Box 354696, Seattle, WA, 98195-4696, USA.
| | - Bethann Pflugeisen
- MultiCare Institute for Research and Innovation, 314 Martin Luther King Jr. Way Suite 402, Tacoma, WA, 98405, USA
| | - Malaika R Schwartz
- Department of Family Medicine, University of Washington, Box 354696, Seattle, WA, 98195-4696, USA
| | - Sophie Cain Miller
- University of Washington School of Medicine, Box 354696, Seattle, WA, 98195-4696, USA
| |
Collapse
|
126
|
Abstract
Context: The High Value Healthcare Collaborative (HVHC) sepsis project was a two-year multi-site project where Member health care delivery systems worked on improving sepsis care using a dissemination & implementation framework designed by HVHC. As part of the project evaluation, participating Members provided 5 data submissions over the project period. Members created data files using a uniform specification, but the data sources and methods used to create the data sets differed. Extensive data cleaning was necessary to get a data set usable for the evaluation analysis. Case Description: HVHC was the coordinating center for the project and received and cleaned all data submissions. Submissions received 3 sequentially more detailed levels of checking by HVHC. The most detailed level evaluated validity by comparing values within-Member over time and between Member. For a subset of episodes Member-submitted data were compared to matched Medicare claims data. Findings: Inconsistencies in data submissions, particularly for length-of-stay variables were common in early submissions and decreased with subsequent submissions. Multiple resubmissions were sometimes required to get clean data. Data checking also uncovered a systematic difference in the way Medicare and some members defined intensive care unit stay. Conclusions: Data checking is a critical for ensuring valid analytic results for projects using electronic health record data. It is important to budget sufficient resources for data checking. Interim data submissions and checks help find anomalies early. Data resubmissions should be checked as fixes can introduce new errors. Communicating with those responsible for creating the data set provides critical information.
Collapse
|
127
|
Johnson SG, Speedie S, Simon G, Kumar V, Westra BL. Quantifying the Effect of Data Quality on the Validity of an eMeasure. Appl Clin Inform 2017; 8:1012-1021. [PMID: 29241241 DOI: 10.4338/aci-2017-03-ra-0042] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023] Open
Abstract
Objective The objective of this study was to demonstrate the utility of a healthcare data quality framework by using it to measure the impact of synthetic data quality issues on the validity of an eMeasure (CMS178—urinary catheter removal after surgery).
Methods Data quality issues were artificially created by systematically degrading the underlying quality of EHR data using two methods: independent and correlated degradation. A linear model that describes the change in the events included in the eMeasure quantifies the impact of each data quality issue.
Results Catheter duration had the most impact on the CMS178 eMeasure with every 1% reduction in data quality causing a 1.21% increase in the number of missing events. For birth date and admission type, every 1% reduction in data quality resulted in a 1% increase in missing events.
Conclusion This research demonstrated that the impact of data quality issues can be quantified using a generalized process and that the CMS178 eMeasure, as currently defined, may not measure how well an organization is meeting the intended best practice goal. Secondary use of EHR data is warranted only if the data are of sufficient quality. The assessment approach described in this study demonstrates how the impact of data quality issues on an eMeasure can be quantified and the approach can be generalized for other data analysis tasks. Healthcare organizations can prioritize data quality improvement efforts to focus on the areas that will have the most impact on validity and assess whether the values that are reported should be trusted.
Collapse
Affiliation(s)
- Steven G Johnson
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, United States
| | - Stuart Speedie
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, United States
| | - Gyorgy Simon
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, United States
| | - Vipin Kumar
- Department of Computer Science, University of Minnesota, Minneapolis, Minnesota, United States
| | - Bonnie L Westra
- Institute for Health Informatics, University of Minnesota, Minneapolis, Minnesota, United States.,School of Nursing, University of Minnesota, Minneapolis, Minnesota, United States
| |
Collapse
|
128
|
Conesa González A, Pastor Duran X, Lozano-Rubí R. [Effectiveness of an assessment of computerised medical records in a university hospital]. ACTA ACUST UNITED AC 2017; 32:328-334. [PMID: 29169963 DOI: 10.1016/j.cali.2017.09.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2017] [Revised: 09/14/2017] [Accepted: 09/26/2017] [Indexed: 10/18/2022]
Abstract
OBJECTIVE The objective of this study was to evaluate a sample of electronic medical records (EMR) that was sufficiently representative of the different areas of care in a university hospital, as well as to verify the effectiveness of an initial intervention through a second evaluation. METHODS Medical records audits were performed in 2012 and 2013 by a blind peer review of random samples of care episodes, proportional to the activity of each clinical department, and with the same evaluation method being applied to all of them. RESULTS More than 1,000 episodes of care were reviewed in the 2audits. A significant improvement was found in hospital admissions (P=.000) in all the sections of the EMR analysed (P=.002), and was especially significant for the reason for consultation, for which its completion increased by 8.5% (p<.05), and also in the sections of the current process record (7.1%), physical examination (4.7%), allergies (3.9%), and clinical course (3.6%). The assessment of the discharge report, as a whole, showed an improvement (P=.001). In outpatient follow-up visits, a significant positive improvement was observed in the 4sections evaluated (P<.05), and also overall (P=.000). CONCLUSIONS According to study conditions, the dissemination of the results was effective in improving the quality of the EMR. The results have made it possible to implement actions to review the work processes in certain departments, and also the partial redesign of the interface on being a reproducible methodology accepted by the organisation.
Collapse
Affiliation(s)
- A Conesa González
- Unidad de Informática Médica, Hospital Clínic de Barcelona, Universidad de Barcelona, Barcelona, España.
| | - X Pastor Duran
- Unidad de Informática Médica, Hospital Clínic de Barcelona, Universidad de Barcelona, Barcelona, España
| | - R Lozano-Rubí
- Unidad de Informática Médica, Hospital Clínic de Barcelona, Universidad de Barcelona, Barcelona, España
| |
Collapse
|
129
|
Abstract
Introduction: We describe the formulation, development, and initial expert review of 3x3 Data Quality Assessment (DQA), a dynamic, evidence-based guideline to enable electronic health record (EHR) data quality assessment and reporting for clinical research. Methods: 3x3 DQA was developed through the triangulation results from three studies: a review of the literature on EHR data quality assessment, a quantitative study of EHR data completeness, and a set of interviews with clinical researchers. Following initial development, the guideline was reviewed by a panel of EHR data quality experts. Results: The guideline embraces the task-dependent nature of data quality and data quality assessment. The core framework includes three constructs of data quality: complete, correct, and current data. These constructs are operationalized according to the three primary dimensions of EHR data: patients, variables, and time. Each of the nine operationalized constructs maps to a methodological recommendation for EHR data quality assessment. The initial expert response to the framework was positive, but improvements are required. Discussion: The initial version of 3x3 DQA promises to enable explicit guideline-based best practices for EHR data quality assessment and reporting. Future work will focus on increasing clarity on how and when 3x3 DQA should be used during the research process, improving the feasibility and ease-of-use of recommendation execution, and clarifying the process for users to determine which operationalized constructs and recommendations are relevant for a given dataset and study.
Collapse
|
130
|
Reporting Data Quality Assessment Results: Identifying Individual and Organizational Barriers and Solutions. EGEMS 2017; 5:16. [PMID: 29881736 PMCID: PMC5982990 DOI: 10.5334/egems.214] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/31/2022]
Abstract
Introduction: Electronic health record (EHR) data are known to have significant data quality issues, yet the practice and frequency of assessing EHR data is unknown. We sought to understand current practices and attitudes towards reporting data quality assessment (DQA) results by data professionals. Methods: The project was conducted in four Phases: (1) examined current DQA practices among informatics/CER stakeholders via engagement meeting (07/2014); (2) characterized organizations conducting DQA by interviewing key personnel and data management professionals (07-08/2014); (3) developed and administered an anonymous survey to data professionals (03-06/2015); and (4) validated survey results during a follow-up informatics/CER stakeholder engagement meeting (06/2016). Results: The first engagement meeting identified the theme of unintended consequences as a primary barrier to DQA. Interviewees were predominantly medical groups serving distributed networks with formalized DQAs. Consistent with the interviews, most survey (N=111) respondents utilized DQA processes/programs. A lack of resources and clear definitions of how to judge the quality of a dataset were the most commonly cited individual barriers. Vague quality action plans/expectations and data owners not trained in problem identification and problem-solving skills were the most commonly cited organizational barriers. Solutions included allocating resources for DQA, establishing standards and guidelines, and changing organizational culture. Discussion: Several barriers affecting DQA and reporting were identified. Community alignment towards systematic DQA and reporting is needed to overcome these barriers. Conclusion: Understanding barriers and solutions to DQA reporting is vital for establishing trust in the secondary use of EHR data for quality improvement and the pursuit of personalized medicine.
Collapse
|
131
|
Kern DM, Barron JJ, Wu B, Ganetsky A, Willey VJ, Quimbo RA, Fisch MJ, Singer J, Nguyen A, Mamtani R. A validation of clinical data captured from a novel Cancer Care Quality Program directly integrated with administrative claims data. Pragmat Obs Res 2017; 8:149-155. [PMID: 28894396 PMCID: PMC5584892 DOI: 10.2147/por.s140579] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/23/2022] Open
Abstract
Background Data from a Cancer Care Quality Program are directly integrated with administrative claims data to provide a level of clinical detail not available in claims-based studies, and referred to as the HealthCore Integrated Research Environment (HIRE)-Oncology data. This study evaluated the validity of the HIRE-Oncology data compared with medical records of breast, lung, and colorectal cancer patients. Methods Data elements included cancer type, stage, histology (lung only), and biomarkers. A sample of 300 breast, 200 lung, and 200 colorectal cancer patients within the HIRE-Oncology data were identified for medical record review. Statistical measures of validity (agreement, positive predictive value [PPV], negative predictive value [NPV], sensitivity, specificity) were used to compare clinical information between data sources, with medical record data considered the gold standard. Results All 300 breast cancer records reviewed were confirmed breast cancer, while 197 lung and 197 colorectal records were confirmed (PPV =0.99 for each). The agreement of disease stage was 85% for breast, 90% for lung, and 94% for colorectal cancer. The agreement of lung cancer histology (small cell vs non-small cell) was 97%. Agreement of progesterone receptor, estrogen receptor, and human epidermal growth factor receptor 2 status biomarkers in breast cancer was 92%, 97%, and 92%, respectively; epidermal growth factor receptor and anaplastic lymphoma kinase agreement in lung was 97% and 92%, respectively; and agreement of KRAS status in colorectal cancer was 95%. Measures of PPV, NPV, sensitivity, and specificity showed similarly strong evidence of validity. Conclusion Good agreement between the HIRE-Oncology data and medical records supports the validity of these data for research.
Collapse
Affiliation(s)
- David M Kern
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | - John J Barron
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | - Bingcao Wu
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | - Alex Ganetsky
- Department of Pharmacy, Hospital of the University of Pennsylvania, Philadelphia, PA
| | - Vincent J Willey
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | - Ralph A Quimbo
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | | | - Joseph Singer
- Health Economics and Outcomes Research, HealthCore, Inc, Wilmington, DE
| | - Ann Nguyen
- Oncology Solutions, Anthem, Inc, Indianapolis, IN
| | - Ronac Mamtani
- Division of Hematology/Oncology, Abramson Cancer Center, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
132
|
Comparing Population-based Risk-stratification Model Performance Using Demographic, Diagnosis and Medication Data Extracted From Outpatient Electronic Health Records Versus Administrative Claims. Med Care 2017; 55:789-796. [PMID: 28598890 DOI: 10.1097/mlr.0000000000000754] [Citation(s) in RCA: 67] [Impact Index Per Article: 9.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
BACKGROUND There is an increasing demand for electronic health record (EHR)-based risk stratification and predictive modeling tools at the population level. This trend is partly due to increased value-based payment policies and the increasing availability of EHRs at the provider level. Risk stratification models, however, have been traditionally derived from claims or encounter systems. This study evaluates the challenges and opportunities of using EHR data instead of or in addition to administrative claims for risk stratification. METHODS This study used the structured EHR records and administrative claims of 85,581 patients receiving outpatient care at a large integrated provider system. Common data elements for risk stratification (ie, age, sex, diagnosis, and medication) were extracted from outpatient EHR records and administrative claims. The performance of a validated risk-stratification model was assessed using data extracted from claims alone, EHR alone, and claims and EHR combined. RESULTS EHR-derived metrics overlapped considerably with administrative claims (eg, number of chronic conditions). The accuracy of the model, when using EHR data alone, was acceptable with an area under the curve of ∼0.81 for hospitalization and ∼0.85 for identifying top 1% utilizers using the concurrent model. However, when using EHR data alone, the predictive model explained a lower amount of variation in utilization-based outcomes compared with administrative claims. DISCUSSION The results show a promising performance of models predicting cost and hospitalization using outpatient EHR's diagnosis and medication data. More research is needed to evaluate the benefits of other EHR data types (eg, lab values and vital signs) for risk stratification.
Collapse
|
133
|
Bayesian analysis of genetic association across tree-structured routine healthcare data in the UK Biobank. Nat Genet 2017; 49:1311-1318. [PMID: 28759005 PMCID: PMC5580804 DOI: 10.1038/ng.3926] [Citation(s) in RCA: 48] [Impact Index Per Article: 6.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2017] [Accepted: 07/06/2017] [Indexed: 12/15/2022]
Abstract
Genetic discovery from the multitude of phenotypes extractable from routine healthcare data can transform our understanding of the human phenome and accelerate progress towards precision medicine. However, a critical question when analysing high-dimensional and heterogeneous data is how to best interrogate increasingly specific subphenotypes whilst retaining statistical power to detect genetic associations. Here we develop and employ a novel Bayesian analysis framework that exploits the hierarchical structure of diagnosis classifications to analyse genetic variants against UK Biobank disease phenotypes derived from self-reporting and hospital episode statistics. Our method displays a more than 20% increase in power to detect genetic effects over other approaches and identifies novel associations between classical human leukocyte antigen (HLA) alleles and common immune-mediated diseases (IMDs). By applying the approach to genetic risk scores (GRSs) we reveal the extent of genetic sharing between IMDs and expose differences in disease perception or diagnosis with potential clinical implications.
Collapse
|
134
|
Hoeven LRV, Bruijne MCD, Kemper PF, Koopman MMW, Rondeel JMM, Leyte A, Koffijberg H, Janssen MP, Roes KCB. Validation of multisource electronic health record data: an application to blood transfusion data. BMC Med Inform Decis Mak 2017; 17:107. [PMID: 28709453 PMCID: PMC5512751 DOI: 10.1186/s12911-017-0504-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2017] [Accepted: 07/10/2017] [Indexed: 11/10/2022] Open
Abstract
Background Although data from electronic health records (EHR) are often used for research purposes, systematic validation of these data prior to their use is not standard practice. Existing validation frameworks discuss validity concepts without translating these into practical implementation steps or addressing the potential influence of linking multiple sources. Therefore we developed a practical approach for validating routinely collected data from multiple sources and to apply it to a blood transfusion data warehouse to evaluate the usability in practice. Methods The approach consists of identifying existing validation frameworks for EHR data or linked data, selecting validity concepts from these frameworks and establishing quantifiable validity outcomes for each concept. The approach distinguishes external validation concepts (e.g. concordance with external reports, previous literature and expert feedback) and internal consistency concepts which use expected associations within the dataset itself (e.g. completeness, uniformity and plausibility). In an example case, the selected concepts were applied to a transfusion dataset and specified in more detail. Results Application of the approach to a transfusion dataset resulted in a structured overview of data validity aspects. This allowed improvement of these aspects through further processing of the data and in some cases adjustment of the data extraction. For example, the proportion of transfused products that could not be linked to the corresponding issued products initially was 2.2% but could be improved by adjusting data extraction criteria to 0.17%. Conclusions This stepwise approach for validating linked multisource data provides a basis for evaluating data quality and enhancing interpretation. When the process of data validation is adopted more broadly, this contributes to increased transparency and greater reliability of research based on routinely collected electronic health records. Electronic supplementary material The online version of this article (doi:10.1186/s12911-017-0504-7) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Loan R van Hoeven
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Universiteitsweg 100, 3508, GA, Utrecht, The Netherlands. .,Transfusion Technology Assessment Department, Sanquin Research, Plesmanlaan 125, 1066, CX, Amsterdam, The Netherlands.
| | - Martine C de Bruijne
- Department of Public and Occupational Health, EMGO Institute, VU University Medical Center, Van der Boechorststraat 7, 1081, BT, Amsterdam, The Netherlands
| | - Peter F Kemper
- Transfusion Technology Assessment Department, Sanquin Research, Plesmanlaan 125, 1066, CX, Amsterdam, The Netherlands
| | - Maria M W Koopman
- Department of Transfusion Medicine, Sanquin Blood bank, Plesmanlaan 125, 1066, CX, Amsterdam, The Netherlands
| | | | - Anja Leyte
- OLVG, Oosterpark 9, 1091, AC, Amsterdam, The Netherlands
| | - Hendrik Koffijberg
- Department of Health Technology & Services Research, MIRA Institute for biomedical technology and technical medicine, University of Twente, Drienerlolaan 5, 7522, NB, Enschede, The Netherlands
| | - Mart P Janssen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Universiteitsweg 100, 3508, GA, Utrecht, The Netherlands.,Transfusion Technology Assessment Department, Sanquin Research, Plesmanlaan 125, 1066, CX, Amsterdam, The Netherlands
| | - Kit C B Roes
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Universiteitsweg 100, 3508, GA, Utrecht, The Netherlands
| |
Collapse
|
135
|
Abstract
Objective: To compare rule-based data quality (DQ) assessment approaches across multiple national clinical data sharing organizations. Methods: Six organizations with established data quality assessment (DQA) programs provided documentation or source code describing current DQ checks. DQ checks were mapped to the categories within the data verification context of the harmonized DQA terminology. To ensure all DQ checks were consistently mapped, conventions were developed and four iterations of mapping performed. Difficult-to-map DQ checks were discussed with research team members until consensus was achieved. Results: Participating organizations provided 11,026 DQ checks, of which 99.97 percent were successfully mapped to a DQA category. Of the mapped DQ checks (N=11,023), 214 (1.94 percent) mapped to multiple DQA categories. The majority of DQ checks mapped to Atemporal Plausibility (49.60 percent), Value Conformance (17.84 percent), and Atemporal Completeness (12.98 percent) categories. Discussion: Using the common DQA terminology, near-complete (99.97 percent) coverage across a wide range of DQA programs and specifications was reached. Comparing the distributions of mapped DQ checks revealed important differences between participating organizations. This variation may be related to the organization’s stakeholder requirements, primary analytical focus, or maturity of their DQA program. Not within scope, mapping checks within the data validation context of the terminology may provide additional insights into DQA practice differences. Conclusion: A common DQA terminology provides a means to help organizations and researchers understand the coverage of their current DQA efforts as well as highlight potential areas for additional DQA development. Sharing DQ checks between organizations could help expand the scope of DQA across clinical data networks.
Collapse
|
136
|
Estiri H, Stephens K. DQ e-v: A Database-Agnostic Framework for Exploring Variability in Electronic Health Record Data Across Time and Site Location. EGEMS 2017; 5:3. [PMID: 29930954 PMCID: PMC5994933 DOI: 10.13063/2327-9214.1277] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/26/2022]
Abstract
Data variability is a commonly observed phenomenon in Electronic Health Records (EHR) data networks. A common question asked in scientific investigations of EHR data is whether the cross-site and -time variability reflects an underlying data quality error at one or more contributing sites versus actual differences driven by various idiosyncrasies in the healthcare settings. Although research analysts and data scientists have commonly used various statistical methods to detect and account for variability in analytic datasets, self service tools to facilitate exploring cross-organizational variability in EHR data warehouses are lacking and could benefit from meaningful data visualizations. DQe-v, an interactive, database-agnostic tool for visually exploring variability in EHR data provides such a solution. DQe-v is built on an open source platform, R statistical software, with annotated scripts and a readme document that makes it fully reproducible. To illustrate and describe functionality of DQe-v, we describe the DQe-v’s readme document which includes a complete guide to installation, running the program, and interpretation of the outputs. We also provide annotated R scripts and an example dataset as supplemental materials. DQe-v offers a self service tool to visually explore data variability within EHR datasets irrespective of the data model. GitHub and CIELO offer hosting and distribution of the tool and can facilitate collaboration across any interested community of users as we target improving usability, efficiency, and interoperability.
Collapse
|
137
|
Abstract
The National Committee for Quality Assurance (NCQA) piloted a new approach to quality measurement meant to reduce avoidable cardiac events and improve overall population health. In this pilot, we investigated whether a standardized technical specification could sufficiently define a process to reliably generate predicted outcome scores from heterogeneous electronic clinical data systems (ECDS) [1]. Patient data were electronically extracted from four health care organizations and processed by the Archimedes, Inc. Global Outcomes calculator, generating scores indicating future cardiovascular event probability for each provider's patient population. These Global Cardiovascular Risk (GCVR) scores represent the gap between current level of care achieved and optimal care for each clinician's patients, with a greater score indicating better performance. As GCVR requires more patient data than do traditional quality measures, we addressed feasibility and data completeness questions in order to understand the prospects of a wholly new quality concept. This pilot successfully produced GCVR scores for 2,251 clinicians, representing approximately 60 percent of the total patient population under study. To our knowledge, this is the first time predictive models have been proposed for quality measurement, and our pilot successfully demonstrated that a predicted outcome measure is feasible using electronic patient data. However, new specification standards are required before this approach is fully scalable to a national quality reporting program.
Collapse
|
138
|
Piccinni C, Antonazzo IC, Simonetti M, Mennuni MG, Parretti D, Cricelli C, Colombo D, Nica M, Cricelli I, Lapi F. The Burden of Chronic Heart Failure in Primary Care in Italy. High Blood Press Cardiovasc Prev 2017; 24:171-178. [DOI: 10.1007/s40292-017-0193-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2017] [Accepted: 03/15/2017] [Indexed: 12/12/2022] Open
|
139
|
Strategies for handling missing clinical data for automated surgical site infection detection from the electronic health record. J Biomed Inform 2017; 68:112-120. [PMID: 28323112 DOI: 10.1016/j.jbi.2017.03.009] [Citation(s) in RCA: 43] [Impact Index Per Article: 6.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2016] [Revised: 03/02/2017] [Accepted: 03/12/2017] [Indexed: 11/23/2022]
Abstract
Proper handling of missing data is important for many secondary uses of electronic health record (EHR) data. Data imputation methods can be used to handle missing data, but their use for analyzing EHR data is limited and specific efficacy for postoperative complication detection is unclear. Several data imputation methods were used to develop data models for automated detection of three types (i.e., superficial, deep, and organ space) of surgical site infection (SSI) and overall SSI using American College of Surgeons National Surgical Quality Improvement Project (NSQIP) Registry 30-day SSI occurrence data as a reference standard. Overall, models with missing data imputation almost always outperformed reference models without imputation that included only cases with complete data for detection of SSI overall achieving very good average area under the curve values. Missing data imputation appears to be an effective means for improving postoperative SSI detection using EHR clinical data.
Collapse
|
140
|
Feder SL. Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res 2017; 40:753-766. [PMID: 28322657 DOI: 10.1177/0193945916689084] [Citation(s) in RCA: 65] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
The proliferation of the electronic health record (EHR) has led to increasing interest and opportunities for nurse scientists to use EHR data in a variety of research designs. However, methodological problems pertaining to data quality may arise when EHR data are used for nonclinical purposes. Therefore, this article describes common domains of data quality and approaches for quality appraisal in EHR research. Common data quality domains include data accuracy, completeness, consistency, credibility, and timeliness. Approaches for quality appraisal include data validation with data rules, evaluation and verification of data abstraction methods with statistical measures, data comparisons with manual chart review, management of missing data using statistical methods, and data triangulation between multiple EHR databases. Quality data enhance the validity and reliability of research findings, form the basis for conclusions derived from the data, and are, thus, an integral component in EHR-based study design and implementation.
Collapse
|
141
|
Gupta P, Patel P, Horne R, Buchanan H, Williams B, Tomaszewski M. How to Screen for Non-Adherence to Antihypertensive Therapy. Curr Hypertens Rep 2016; 18:89. [PMID: 27889904 PMCID: PMC5124437 DOI: 10.1007/s11906-016-0697-7] [Citation(s) in RCA: 57] [Impact Index Per Article: 7.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
Abstract
The quality of assessment of non-adherence to treatment in hypertensive is poor. Within this review, we discuss the different methods used to assess adherence to blood-pressure-lowering medications in hypertension patients. Subjective reports such as physicians' perceptions are inaccurate, and questionnaires completed by patients tend to overreport adherence and show a low diagnostic specificity. Indirect objective methods such as pharmacy database records can be useful, but they are limited by the robustness of the recorded data. Electronic medication monitoring devices are accurate but usually track adherence to only a single medication and can be expensive. Overall, the fundamental issue with indirect objective measures is that they do not fully confirm ingestion of antihypertensive medications. Detection of antihypertensive medications in body fluids using liquid chromatography-tandem mass spectrometry is currently, in our view, the most robust and clinically useful method to assess non-adherence to blood-pressure-lowering treatment. It is particularly helpful in patients presenting with resistant, refractory or uncontrolled hypertension despite the optimal therapy. We recommend using this diagnostic strategy to detect non-adherence alongside a no-blame approach tailoring support to address the perceptions (e.g. beliefs about the illness and treatment) and practicalities (e.g. capability and resources) influencing motivation and ability to adhere.
Collapse
Affiliation(s)
- Pankaj Gupta
- Department of Metabolic Medicine and Chemical Pathology, University Hospitals of Leicester NHS Trust, Leicester, UK
- National Institute of Health Research Leicester Cardiovascular Biomedical Research Unit, Glenfield Hospital, Leicester, UK
- British Heart Foundation Cardiovascular Research Centre, Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
| | - Prashanth Patel
- Department of Metabolic Medicine and Chemical Pathology, University Hospitals of Leicester NHS Trust, Leicester, UK
- National Institute of Health Research Leicester Cardiovascular Biomedical Research Unit, Glenfield Hospital, Leicester, UK
- British Heart Foundation Cardiovascular Research Centre, Department of Cardiovascular Sciences, University of Leicester, Leicester, UK
| | - Robert Horne
- Department of Behavioral Medicine, School of Pharmacy, University College London, London, UK
| | - Heather Buchanan
- Division of Rehabilitation and Ageing, School of Medicine, University of Nottingham, Nottingham, UK
| | - Bryan Williams
- Institute of Cardiovascular Science, University College London, London, UK
- National Institute for Health Research University College London Hospitals Biomedical Research Centre, London, UK
| | - Maciej Tomaszewski
- Division of Cardiovascular Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK.
- Division of Medicine, Central Manchester NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, UK.
| |
Collapse
|
142
|
Vickery AW, Ryan J, Pang J, Garton-Smith J, Watts GF. Increasing the Detection of Familial Hypercholesterolaemia Using General Practice Electronic Databases. Heart Lung Circ 2016; 26:450-454. [PMID: 27889294 DOI: 10.1016/j.hlc.2016.09.012] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2015] [Revised: 02/29/2016] [Accepted: 09/24/2016] [Indexed: 10/20/2022]
Abstract
BACKGROUND Familial hypercholesterolaemia (FH) is a common autosomal co-dominant condition that causes premature cardiovascular disease. Awareness of FH is poor and only 10-15% of the affected population is identified. Electronic health records provide an opportunity to increase detection and awareness in general practice OBJECTIVE: To determine whether a simple electronic extraction tool can increase detection of FH in general practice. METHOD An extraction tool applied to general practice electronic health records (EHR) to screen for FH, total cholesterol and low density lipoprotein cholesterol (LDL-c) levels in association with entered diagnostic criteria and demographic data in five general practices. RESULTS Of 157,290 active patients examined, 0.7% (n=1081) had an LDL-c>5.0 mmol/L representing 1 in 146 of active patients. An additional 0.8% (n=1276) patients were at possible risk of FH. Of those with an LDL-c>5.0 mmol/L 43.7% of patients had no record of being prescribed statins. Twenty patients (0.013%) had a clinical diagnosis of FH entered in the EHR. CONCLUSIONS Patients at high risk of FH can be identified by a simple electronic screening method in general practice. Clinical data entry is variable in general practice. Targeted screening enables clinical assessment of patients at risk of cardiovascular disease and using the DLCNS will enable primary care to increase identification of FH. Approximately one in five patients extracted using this method, are likely to have phenotypically probable FH, making it a useful screening tool.
Collapse
Affiliation(s)
- Alistair W Vickery
- School of Primary, Aboriginal and Rural Health Care, University of Western Australia, Perth, WA, Australia.
| | - Jackie Ryan
- Lipid Disorders Clinic, Cardiovascular Medicine, Royal Perth Hospital, School of Medicine and Pharmacology, University of Western Australia, Perth, WA, Australia
| | - Jing Pang
- Lipid Disorders Clinic, Cardiovascular Medicine, Royal Perth Hospital, School of Medicine and Pharmacology, University of Western Australia, Perth, WA, Australia
| | | | - Gerald F Watts
- Lipid Disorders Clinic, Cardiovascular Medicine, Royal Perth Hospital, School of Medicine and Pharmacology, University of Western Australia, Perth, WA, Australia
| |
Collapse
|
143
|
Weng C, Kahn MG. Clinical Research Informatics for Big Data and Precision Medicine. Yearb Med Inform 2016:211-218. [PMID: 27830253 DOI: 10.15265/iy-2016-019] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/17/2022] Open
Abstract
OBJECTIVES To reflect on the notable events and significant developments in Clinical Research Informatics (CRI) in the year of 2015 and discuss near-term trends impacting CRI. METHODS We selected key publications that highlight not only important recent advances in CRI but also notable events likely to have significant impact on CRI activities over the next few years or longer, and consulted the discussions in relevant scientific communities and an online living textbook for modern clinical trials. We also related the new concepts with old problems to improve the continuity of CRI research. RESULTS The highlights in CRI in 2015 include the growing adoption of electronic health records (EHR), the rapid development of regional, national, and global clinical data research networks for using EHR data to integrate scalable clinical research with clinical care and generate robust medical evidence. Data quality, integration, and fusion, data access by researchers, study transparency, results reproducibility, and infrastructure sustainability are persistent challenges. CONCLUSION The advances in Big Data Analytics and Internet technologies together with the engagement of citizens in sciences are shaping the global clinical research enterprise, which is getting more open and increasingly stakeholder-centered, where stakeholders include patients, clinicians, researchers, and sponsors.
Collapse
Affiliation(s)
- C Weng
- Chunhua Weng, PhD, FACMI, Department of Biomedical Informatics, Columbia University, 622 W 168 Street, PH-20, New York, NY 10032, USA, E-mail:
| | | |
Collapse
|
144
|
Scholte M, van Dulmen SA, Neeleman-Van der Steen CWM, van der Wees PJ, Nijhuis-van der Sanden MWG, Braspenning J. Data extraction from electronic health records (EHRs) for quality measurement of the physical therapy process: comparison between EHR data and survey data. BMC Med Inform Decis Mak 2016; 16:141. [PMID: 27825333 PMCID: PMC5101697 DOI: 10.1186/s12911-016-0382-4] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2016] [Accepted: 11/02/2016] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND With the emergence of the electronic health records (EHRs) as a pervasive healthcare information technology, new opportunities and challenges for use of clinical data for quality measurements arise with respect to data quality, data availability and comparability. The objective of this study is to test whether data extracted from electronic health records (EHRs) was of comparable quality as survey data for the calculation of quality indicators. METHODS Data from surveys describing patient cases and filled out by physiotherapists in 2009-2010 were used to calculate scores on eight quality indicators (QIs) to measure the quality of physiotherapy care. In 2011, data was extracted directly from EHRs. The data collection methods were evaluated for comparability. EHR data was compared to survey data on completeness and correctness. RESULTS Five of the eight QIs could be extracted from the EHRs. Three were omitted from the indicator set, as they proved too difficult to be extracted from the EHRs. Another QI proved incomparable due to errors in the extraction software of some of the EHRs. Three out of four comparable QIs performed better (p < 0.001) in EHR data on completeness. EHR data also proved to be correct; the relative change in indicator scores between EHR and survey data were small (<5 %) in three out of four QIs. CONCLUSION Data quality of EHRs was sufficient to be used for the calculation of QIs, although comparability to survey data was problematic. Standardization is needed, not only to be able to compare different data collection methods properly, but also to compare between practices with different EHRs. EHRs have the option to administrate narrative data, but natural language processing tools are needed to quantify these text boxes. Such development, can narrow the comparability gap between scoring QIs based on EHR data and based on survey data. EHRs have the potential to provide real time feedback to professionals and quality measurements for research, but more effort is needed to create unambiguous and uniform information and to unlock written text in a standardized manner.
Collapse
Affiliation(s)
- Marijn Scholte
- Radboud Institute for Health Sciences, Scientific Institute for Quality of Healthcare, Radboud University Medical Center, Geert Grooteplein 21, 6525 EZ, Nijmegen, The Netherlands. .,Present address: Faculty of Social Sciences, Department of Sociology, Radboud University, Thomas van Aquinostraat 6, 6525 GD, Nijmegen, The Netherlands.
| | - Simone A van Dulmen
- Research Institute for Health Sciences, Scientific Institute for Quality of Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Philip J van der Wees
- Research Institute for Health Sciences, Scientific Institute for Quality of Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Maria W G Nijhuis-van der Sanden
- Research Institute for Health Sciences, Scientific Institute for Quality of Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Jozé Braspenning
- Research Institute for Health Sciences, Scientific Institute for Quality of Healthcare, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
145
|
Bailey SR, Heintzman JD, Marino M, Hoopes MJ, Hatch BA, Gold R, Cowburn SC, Nelson CA, Angier HE, DeVoe JE. Measuring Preventive Care Delivery: Comparing Rates Across Three Data Sources. Am J Prev Med 2016; 51:752-761. [PMID: 27522472 PMCID: PMC5067199 DOI: 10.1016/j.amepre.2016.07.004] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/26/2016] [Revised: 06/08/2016] [Accepted: 07/07/2016] [Indexed: 11/26/2022]
Abstract
INTRODUCTION Preventive care delivery is an important quality outcome, and electronic data reports are being used increasingly to track these services. It is highly informative when electronic data sources are compared to information manually extracted from medical charts to assess validity and completeness. METHODS This cross-sectional study used a random sample of Medicaid-insured patients seen at 43 community health centers in 2011 to calculate standard measures of correspondence between manual chart review and two automated sources (electronic health records [EHRs] and Medicaid claims), comparing documentation of orders for and receipt of ten preventive services (n=150 patients/service). Data were analyzed in 2015. RESULTS Using manual chart review as the gold standard, automated EHR extraction showed near-perfect to perfect agreement (κ=0.96-1.0) for services received within the primary care setting (e.g., BMI, blood pressure). Receipt of breast and colorectal cancer screenings, services commonly referred out, showed moderate (κ=0.42) to substantial (κ=0.62) agreement, respectively. Automated EHR extraction showed near-perfect agreement (κ=0.83-0.97) for documentation of ordered services. Medicaid claims showed near-perfect agreement (κ=0.87) for hyperlipidemia and diabetes screening, and substantial agreement (κ=0.67-0.80) for receipt of breast, cervical, and colorectal cancer screenings, and influenza vaccination. Claims showed moderate agreement (κ=0.59) for chlamydia screening receipt. Medicaid claims did not capture ordered or unbilled services. CONCLUSIONS Findings suggest that automated EHR and claims data provide valid sources for measuring receipt of most preventive services; however, ordered and unbilled services were primarily captured via EHR data and completed referrals were more often documented in claims data.
Collapse
Affiliation(s)
- Steffani R Bailey
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon.
| | - John D Heintzman
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon
| | - Miguel Marino
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon; Department of Public Health and Preventive Medicine, Division of Biostatistics, Oregon Health & Science University, Portland, Oregon
| | | | - Brigit A Hatch
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon; OCHIN, Inc., Portland, Oregon
| | - Rachel Gold
- OCHIN, Inc., Portland, Oregon; Kaiser Permanente Northwest Center for Health Research, Portland, Oregon
| | | | | | - Heather E Angier
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon
| | - Jennifer E DeVoe
- Department of Family Medicine, Oregon Health & Science University, Portland, Oregon; OCHIN, Inc., Portland, Oregon
| |
Collapse
|
146
|
Shaha SH, Sayeed Z, Anoushiravani AA, El-Othmani MM, Saleh KJ. Big Data, Big Problems: Incorporating Mission, Values, and Culture in Provider Affiliations. Orthop Clin North Am 2016; 47:725-32. [PMID: 27637659 DOI: 10.1016/j.ocl.2016.05.009] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
This article explores how integration of data from clinical registries and electronic health records produces a quality impact within orthopedic practices. Data are differentiated from information, and several types of data that are collected and used in orthopedic outcome measurement are defined. Furthermore, the concept of comparative effectiveness and its impact on orthopedic clinical research are assessed. This article places emphasis on how the concept of big data produces health care challenges balanced with benefits that may be faced by patients and orthopedic surgeons. Finally, essential characteristics of an electronic health record that interlinks musculoskeletal care and big data initiatives are reviewed.
Collapse
Affiliation(s)
- Steven H Shaha
- Center for Public Policy and Administration, 260 South Central Campus Drive, Room 214, University of Utah, Salt Lake City, UT 84112, USA
| | - Zain Sayeed
- Division of Orthopaedics and Rehabilitation, Southern Illinois University School of Medicine, 701 North First Street, Springfield, IL 62781, USA
| | - Afshin A Anoushiravani
- Division of Orthopaedics and Rehabilitation, Southern Illinois University School of Medicine, 701 North First Street, Springfield, IL 62781, USA
| | - Mouhanad M El-Othmani
- Division of Orthopaedics and Rehabilitation, Southern Illinois University School of Medicine, 701 North First Street, Springfield, IL 62781, USA
| | - Khaled J Saleh
- Department of Orthopaedic and Sports Medicine, Detroit Medical Center, 311 Mack Avenue, 5th Floor, Detroit, MI 48201, USA.
| |
Collapse
|
147
|
Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, Estiri H, Goerg C, Holve E, Johnson SG, Liaw ST, Hamilton-Lopez M, Meeker D, Ong TC, Ryan P, Shang N, Weiskopf NG, Weng C, Zozus MN, Schilling L. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. ACTA ACUST UNITED AC 2016; 4:1244. [PMID: 27713905 PMCID: PMC5051581 DOI: 10.13063/2327-9214.1244] [Citation(s) in RCA: 190] [Impact Index Per Article: 23.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Abstract
Objective: Harmonized data quality (DQ) assessment terms, methods, and reporting practices can establish a common understanding of the strengths and limitations of electronic health record (EHR) data for operational analytics, quality improvement, and research. Existing published DQ terms were harmonized to a comprehensive unified terminology with definitions and examples and organized into a conceptual framework to support a common approach to defining whether EHR data is ‘fit’ for specific uses. Materials and Methods: DQ publications, informatics and analytics experts, managers of established DQ programs, and operational manuals from several mature EHR-based research networks were reviewed to identify potential DQ terms and categories. Two face-to-face stakeholder meetings were used to vet an initial set of DQ terms and definitions that were grouped into an overall conceptual framework. Feedback received from data producers and users was used to construct a draft set of harmonized DQ terms and categories. Multiple rounds of iterative refinement resulted in a set of terms and organizing framework consisting of DQ categories, subcategories, terms, definitions, and examples. The harmonized terminology and logical framework’s inclusiveness was evaluated against ten published DQ terminologies. Results: Existing DQ terms were harmonized and organized into a framework by defining three DQ categories: (1) Conformance (2) Completeness and (3) Plausibility and two DQ assessment contexts: (1) Verification and (2) Validation. Conformance and Plausibility categories were further divided into subcategories. Each category and subcategory was defined with respect to whether the data may be verified with organizational data, or validated against an accepted gold standard, depending on proposed context and uses. The coverage of the harmonized DQ terminology was validated by successfully aligning to multiple published DQ terminologies. Discussion: Existing DQ concepts, community input, and expert review informed the development of a distinct set of terms, organized into categories and subcategories. The resulting DQ terms successfully encompassed a wide range of disparate DQ terminologies. Operational definitions were developed to provide guidance for implementing DQ assessment procedures. The resulting structure is an inclusive DQ framework for standardizing DQ assessment and reporting. While our analysis focused on the DQ issues often found in EHR data, the new terminology may be applicable to a wide range of electronic health data such as administrative, research, and patient-reported data. Conclusion: A consistent, common DQ terminology, organized into a logical framework, is an initial step in enabling data owners and users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary. Future work will leverage the framework and terminology to develop reusable data quality assessment and reporting methods.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Hossein Estiri
- University of Washington, Institute of Translational Health Sciences
| | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
148
|
Learning About Missing Data Mechanisms in Electronic Health Records-based Research: A Survey-based Approach. Epidemiology 2016; 27:82-90. [PMID: 26484425 DOI: 10.1097/ede.0000000000000393] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
BACKGROUND Bias due to missing data is a major concern in electronic health record (EHR)-based research. As part of an ongoing EHR-based study of weight change among patients treated for depression, we conducted a survey to investigate determinants of missingness in the available weight information and to evaluate the missing-at-random assumption. METHODS We identified 8,345 individuals enrolled in a large EHR-based health care system who had monotherapy treatment for depression from April 2008 to March 2010. A stratified sample of 1,153 individuals completed a detailed survey. Logistic regression was used to investigate determinants of whether a patient (1) had an opportunity to be weighed at treatment initiation (baseline), and (2) had a weight measurement recorded. Parallel analyses were conducted to investigate missingness during follow-up. Throughout, inverse-probability weighting was used to adjust for the design and survey nonresponse. Analyses were also conducted to investigate potential recall bias. RESULTS Missingness at baseline and during follow-up was associated with numerous factors not routinely collected in the EHR including whether or not the patient had ever chosen not to be weighed, external weight control activities, and self-reported baseline weight. Patient attitudes about their weight and perceptions regarding the potential impact of their depression treatment on weight were not related to missingness. CONCLUSION Adopting a comprehensive strategy to investigate missingness early in the research process gives researchers information necessary to evaluate key assumptions. While the survey presented focuses on outcome data, the overarching strategy can be applied to any and all data elements subject to missingness.
Collapse
|
149
|
Payne TH. The electronic health record as a catalyst for quality improvement in patient care. Heart 2016; 102:1782-1787. [DOI: 10.1136/heartjnl-2015-308724] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/04/2016] [Revised: 07/06/2016] [Accepted: 07/07/2016] [Indexed: 11/03/2022] Open
|
150
|
Huang Y, Voorham J, Haaijer-Ruskamp FM. Using primary care electronic health record data for comparative effectiveness research: experience of data quality assessment and preprocessing in The Netherlands. J Comp Eff Res 2016; 5:345-54. [PMID: 27346480 DOI: 10.2217/cer-2015-0022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022] Open
Abstract
AIM Details of data quality and how quality issues were solved have not been reported in published comparative effectiveness studies using electronic health record data. METHODS We developed a conceptual framework of data quality assessment and preprocessing and apply it to a study comparing angiotensin-converting enzyme inhibitors with angiotensin receptor blockerss on renal function decline in diabetes patients. RESULTS The framework establishes a line of thought to identify and act on data issues. The core concept is to evaluate whether data are fit-for-use for research tasks. Possible quality problems are listed through specific signal detections, and verified whether they are true problems. Optimal solutions are selected for the identified problems. CONCLUSION This framework can be used in observational studies to improve validity of results.
Collapse
Affiliation(s)
- Yunyu Huang
- Department of Clinical Pharmacy & Pharmacology, University Medical Center Groningen, Antonius Deusinglaan 1, 9713AV Groningen, The Netherlands.,School of Public Health, Fudan University, 130 Dong An Road, 200032 Shanghai, China
| | - Jaco Voorham
- Department of Clinical Pharmacy & Pharmacology, University Medical Center Groningen, Antonius Deusinglaan 1, 9713AV Groningen, The Netherlands
| | - Flora M Haaijer-Ruskamp
- Department of Clinical Pharmacy & Pharmacology, University Medical Center Groningen, Antonius Deusinglaan 1, 9713AV Groningen, The Netherlands
| |
Collapse
|