Evaluation of data completeness in the electronic health record for the purpose of patient recruitment into clinical trials: a retrospective analysis of element presence

Advancing birth cohort studies using administrative and other research-independent data repositories: Opportunities and challenges

The birth cohort study design is an essential epidemiologic tool for investigating the developmental origins of health and disease. Birth cohorts have greatly improved the etiologic understanding of asthma and allergic diseases, setting the stage for advancements in translational interventions. Increasingly, investigators leverage data repositories that have been compiled and maintained independently of research investigations (administrative data) to establish large birth cohorts or to augment data generated through active participant interaction. In many cases, administrative data can greatly enhance the capacity of birth cohorts to achieve their scientific goals. However, investigators must be wary of common pitfalls and carefully consider whether administrative data are well suited to the scientific inquiry. This article reviews the strengths and challenges of using administrative data and the pragmatic solutions that have been developed to optimize their use in birth cohorts. As birth cohorts continue to play an important role in understanding the etiology of early-life disease, unleashing the power of administrative data will greatly assist in this scientific process.

Reusing data from HL7 CDA-based shared EHR systems for clinical trial conduct: a method for analyzing feasibility

BACKGROUND

Electronic health record (EHR) systems have been shown to represent a valuable source of data reuse in the design and conduct of clinical trials. Earlier work has mostly focused on institutional EHR systems. Shared EHR systems have been neglected so far, even though they are highly prevalent today and their characteristics (integrated data across a patient's care providers, standardized information model) make them attractive for the task. However, as they typically focus on a limited data set for the most common care situations, it remains unclear, whether shared EHR systems actually cover the data elements required for clinical trial conduct. In this paper we present a method, which allows shared EHR systems to be analyzed in this regard.

METHODS

We focus on shared EHR systems using HL7 CDA as this is currently the most-widely used content standard. For the data elements that are commonly used in clinical trials we refer to the EHR4CR reference list. The latter is semiautomatically mapped to the EHR system's information model using the open source tool ART-DECOR. For the final automatic analysis of the mappings, another open source tool is provided.

RESULTS

A stepwise approach was developed to analyze HL7 CDA-based shared EHR systems for their coverage of data elements that are relevant for clinical trials. All tools used in this work as well as all mappings are publicly accessible to make the method reusable and the results reproducible. We applied our approach to the Austrian nation-wide EHR system ELGA and showed that the latter allows the recording of 88% of all EHR4CR data elements, 77% in structured format.

CONCLUSIONS

Our method allows HL7 CDA-based shared EHR systems to be easily analyzed to what extent their content could be reused in the context of clinical trials. The results for ELGA indicate that it has a substantial corresponding potential.

CLINICAL TRIAL NUMBER

Not applicable.

Developing a Coordinated Registry Network for devices used for acute ischemic stroke intervention: basilar artery occlusion quality assessment pilot

BACKGROUND

Real-world data can be helpful in evaluating endovascular therapy (EVT) in ischemic stroke care. We conducted a pilot study to aggregate data on basilar artery occlusion (BAO) EVT from existing registries in the USA. We evaluated the availability, completeness, quality, and consistency of common data elements (CDEs) across data sources.

METHODS

We harmonized patient-level data from five registry data sources and assessed the availability, completeness (defined by the presence in at least four data sources), and consistency of CDEs. We assessed data quality based on seven pre-defined critical domains for BAO EVT investigation: baseline patient and disease characteristics; time metrics; description of intervention; adjunctive devices, revascularization scores, complications; post-intervention National Institutes of Health Stroke Scale scores; discharge disposition; 30-day and 90-day mortality and modified Rankin Scale (mRS) scores.

RESULTS

The aggregated dataset of five registries included 493 BAO procedures between January 2013 and January 2020. In total, 88 CDEs were screened and 35 (40%) elements were considered prevalent. Of these 35 CDEs, the majority were collected for >80% of cases when aggregated. All seven pre-defined domains for BAO device investigation could be fulfilled with harmonized data elements. Most data elements were collected with consistent or compatible definitions across registries. The main challenge was the collection of 90-day outcomes.

CONCLUSIONS

This pilot shows the feasibility of aggregating and harmonizing critical CDEs across registries to create a Coordinated Registry Network (CRN). The CRN with partnerships between multiple registries and stakeholders could help improve the breadth and/or depth of real-world data to help answer relevant questions and support clinical and regulatory decisions.

Discrepancies in Aggregate Patient Data between Two Sources with Data Originating from the Same Electronic Health Record: A Case Study

BACKGROUND

 Data exploration in modern electronic health records (EHRs) is often aided by user-friendly graphical interfaces providing "self-service" tools for end users to extract data for quality improvement, patient safety, and research without prerequisite training in database querying. Other resources within the same institution, such as Honest Brokers, may extract data sourced from the same EHR but obtain different results leading to questions of data completeness and correctness.

OBJECTIVES

 Our objectives were to (1) examine the differences in aggregate output generated by a "self-service" graphical interface data extraction tool and our institution's clinical data warehouse (CDW), sourced from the same database, and (2) examine the causative factors that may have contributed to these differences.

METHODS

 Aggregate demographic data of patients who received influenza vaccines at three static clinics and three drive-through clinics in similar locations between August 2020 and December 2020 was extracted separately from our institution's EHR data exploration tool and our CDW by our organization's Honest Brokers System. We reviewed the aggregate outputs, sliced by demographics and vaccination sites, to determine potential differences between the two outputs. We examined the underlying data model, identifying the source of each database.

RESULTS

 We observed discrepancies in patient volumes between the two sources, with variations in demographic information, such as age, race, ethnicity, and primary language. These variations could potentially influence research outcomes and interpretations.

CONCLUSION

 This case study underscores the need for a thorough examination of data quality and the implementation of comprehensive user education to ensure accurate data extraction and interpretation. Enhancing data standardization and validation processes is crucial for supporting reliable research and informed decision-making, particularly if demographic data may be used to support targeted efforts for a specific population in research or quality improvement initiatives.

Targeted electronic health record-based recruitment strategy to enhance COVID-19 vaccine response clinical research study enrollment

Efficient recruitment of eligible participants is a significant challenge for clinical research studies. This challenge was exacerbated during the COVID-19 pandemic when in-person recruitment was not an option. In 2020, the University of Minnesota was tasked, as part of the National Cancer Institute's Serological Sciences Network for COVID-19 (SeroNet), to recruit participants for a longitudinal serosurveillance clinical research study with a goal of characterizing the COVID-19 vaccine-elicited immune response among immunocompromised individuals, which necessitated reliance on non-traditional strategies for participant recruitment. To meet our enrollment target of 300 transplant patients, 300 cancer patients, 100 persons living with HIV, and 200 immunocompetent individuals, we utilized targeted electronic health record (EHR)-based recruitment in addition to traditional recruitment tools, which was an effective combination of recruitment strategies. A significant advantage of patient portal messaging or other digital recruitment strategies such as email communication is timing. We reached 85 % (769 out of 900) of our enrollment target within one year with a 14.3 % response rate to invitations to participate in our study. This achievement is perhaps more salient given the COVID-19 pandemic-related constraints within which we were operating. We demonstrated that the EHR can be leveraged to quickly identify potentially eligible study participants either via EHR communication or mail. We also illustrate how the online portal MyChart can be used to efficiently send targeted recruitment messages.

Evaluation of Eligibility Criteria Relevance for the Purpose of IT-Supported Trial Recruitment: Descriptive Quantitative Analysis

BACKGROUND

Clinical trials (CTs) are crucial for medical research; however, they frequently fall short of the requisite number of participants who meet all eligibility criteria (EC). A clinical trial recruitment support system (CTRSS) is developed to help identify potential participants by performing a search on a specific data pool. The accuracy of the search results is directly related to the quality of the data used for comparison. Data accessibility can present challenges, making it crucial to identify the necessary data for a CTRSS to query. Prior research has examined the data elements frequently used in CT EC but has not evaluated which criteria are actually used to search for participants. Although all EC must be met to enroll a person in a CT, not all criteria have the same importance when searching for potential participants in an existing data pool, such as an electronic health record, because some of the criteria are only relevant at the time of enrollment.

OBJECTIVE

In this study, we investigated which groups of data elements are relevant in practice for finding suitable participants and whether there are typical elements that are not relevant and can therefore be omitted.

METHODS

We asked trial experts and CTRSS developers to first categorize the EC of their CTs according to data element groups and then to classify them into 1 of 3 categories: necessary, complementary, and irrelevant. In addition, the experts assessed whether a criterion was documented (on paper or digitally) or whether it was information known only to the treating physicians or patients.

RESULTS

We reviewed 82 CTs with 1132 unique EC. Of these 1132 EC, 350 (30.9%) were considered necessary, 224 (19.8%) complementary, and 341 (30.1%) total irrelevant. To identify the most relevant data elements, we introduced the data element relevance index (DERI). This describes the percentage of studies in which the corresponding data element occurs and is also classified as necessary or supplementary. We found that the query of "diagnosis" was relevant for finding participants in 79 (96.3%) of the CTs. This group was followed by "date of birth/age" with a DERI of 85.4% (n=70) and "procedure" with a DERI of 35.4% (n=29).

CONCLUSIONS

The distribution of data element groups in CTs has been heterogeneously described in previous works. Therefore, we recommend identifying the percentage of CTs in which data element groups can be found as a more reliable way to determine the relevance of EC. Only necessary and complementary criteria should be included in this DERI.

Use of comprehensive recruitment strategies in the glycemia reduction approaches in diabetes: A comparative effectiveness study (GRADE) multi-center clinical trial

BACKGROUND/AIMS

We present and describe recruitment strategies implemented from 2013 to 2017 across 45 clinical sites in the United States, participating in the Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness Study, an unmasked, randomized controlled trial evaluating four glucose-lowering medications added to metformin in individuals with type 2 diabetes mellitus (duration of diabetes <10 years). We examined the yield of participants recruited through Electronic Health Records systems compared to traditional recruitment methods to leverage access to type 2 diabetes patients in primary care.

METHODS

Site selection criteria included availability of the study population, geographic representation, the ability to recruit and retain a diverse pool of participants including traditionally underrepresented groups, and prior site research experience in diabetes clinical trials. Recruitment initiatives were employed to support and monitor recruitment, such as creation of a Recruitment and Retention Committee, development of criteria for Electronic Health Record systems queries, conduct of remote site visits, development of a public screening website, and other central and local initiatives. Notably, the study supported a dedicated recruitment coordinator at each site to manage local recruitment and facilitate screening of potential participants identified by Electronic Health Record systems.

RESULTS

The study achieved the enrollment goal of 5000 participants, meeting its target with Black/African American (20%), Hispanic/Latino (18%), and age ≧60 years (42%) subgroups but not with women (36%). Recruitment required 1 year more than the 3 years originally planned. Sites included academic hospitals, integrated health systems, and Veterans Affairs Medical Centers. Participants were enrolled through Electronic Health Record queries (68%), physician referral (13%), traditional mail outreach (7%), TV, radio, flyers, and Internet (7%), and other strategies (5%). Early implementation of targeted Electronic Health Record queries yielded a greater number of eligible participants compared to other recruitment methods. Efforts over time increasingly emphasized engagement with primary care networks.

CONCLUSION

Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness successfully recruited a diverse study population with relatively new onset of type 2 diabetes mellitus, relying to a large extent on the use of Electronic Health Record to screen potential participants. A comprehensive approach to recruitment with frequent monitoring was critical to meet the recruitment goal.

How to Improve Cancer Patients ENrollment in Clinical Trials From rEal-Life Databases Using the Observational Medical Outcomes Partnership Oncology Extension: Results of the PENELOPE Initiative in Urologic Cancers

PURPOSE

To compare the computability of Observational Medical Outcomes Partnership (OMOP)-based queries related to prescreening of patients using two versions of the OMOP common data model (CDM; v5.3 and v5.4) and to assess the performance of the Greater Paris University Hospital (APHP) prescreening tool.

MATERIALS AND METHODS

We identified the prescreening information items being relevant for prescreening of patients with cancer. We randomly selected 15 academic and industry-sponsored urology phase I-IV clinical trials (CTs) launched at APHP between 2016 and 2021. The computability of the related prescreening criteria (PC) was defined by their translation rate in OMOP-compliant queries and by their execution rate on the APHP clinical data warehouse (CDW) containing data of 205,977 patients with cancer. The overall performance of the prescreening tool was assessed by the rate of true- and false-positive cases of three randomly selected CTs.

RESULTS

We defined a list of 15 minimal information items being relevant for patients' prescreening. We identified 83 PC of the 534 eligibility criteria from the 15 CTs. We translated 33 and 62 PC in queries on the basis of OMOP CDM v5.3 and v5.4, respectively (translation rates of 40% and 75%, respectively). Of the 33 PC translated in the v5.3 of the OMOP CDM, 19 could be executed on the APHP CDW (execution rate of 58%). Of 83 PC, the computability rate on the APHP CDW reached 23%. On the basis of three CTs, we identified 17, 32, and 63 patients as being potentially eligible for inclusion in those CTs, resulting in positive predictive values of 53%, 41%, and 21%, respectively.

CONCLUSION

We showed that PC could be formalized according to the OMOP CDM and that the oncology extension increased their translation rate through better representation of cancer natural history.

The quality of vital signs measurements and value preferences in electronic medical records varies by hospital, specialty, and patient demographics

We aimed to assess the frequency of value preferences in recording of vital signs in electronic healthcare records (EHRs) and associated patient and hospital factors. We used EHR data from Oxford University Hospitals, UK, between 01-January-2016 and 30-June-2019 and a maximum likelihood estimator to determine the prevalence of value preferences in measurements of systolic and diastolic blood pressure (SBP/DBP), heart rate (HR) (readings ending in zero), respiratory rate (multiples of 2 or 4), and temperature (readings of 36.0 °C). We used multivariable logistic regression to investigate associations between value preferences and patient age, sex, ethnicity, deprivation, comorbidities, calendar time, hour of day, days into admission, hospital, day of week and speciality. In 4,375,654 records from 135,173 patients, there was an excess of temperature readings of 36.0 °C above that expected from the underlying distribution that affected 11.3% (95% CI 10.6-12.1%) of measurements, i.e. these observations were likely inappropriately recorded as 36.0 °C instead of the true value. SBP, DBP and HR were rounded to the nearest 10 in 2.2% (1.4-2.8%) and 2.0% (1.3-5.1%) and 2.4% (1.7-3.1%) of measurements. RR was also more commonly recorded as multiples of 2. BP digit preference and an excess of temperature recordings of 36.0 °C were more common in older and male patients, as length of stay increased, following a previous normal set of vital signs and typically more common in medical vs. surgical specialities. Differences were seen between hospitals, however, digit preference reduced over calendar time. Vital signs may not always be accurately documented, and this may vary by patient groups and hospital settings. Allowances and adjustments may be needed in delivering care to patients and in observational analyses and predictive tools using these factors as outcomes or exposures.

Body mass index prediction rule for mid-upper arm circumference: the atherosclerosis risk in communities study

OBJECTIVES

Electronic health records (EHR) are a convenient data source for clinical trial recruitment and allow for inexpensive participant screening. However, EHR may lack pertinent screening variables. One strategy is to identify surrogate EHR variables which can predict the screening variable of interest. In this article, we use BMI to develop a prediction rule for arm circumference using data from the Atherosclerosis Risk in Communities (ARIC) Study. This work applies to EHR patient screening for clinical trials of hypertension.

METHODS

We included 11 585 participants aged 52-75 years with BMI and arm circumference measured at ARIC follow-up visit 4 (1996-1998). We selected the following arm circumference cutpoints based on the American Heart Association recommendations for blood pressure (BP) cuffs: small adult (≤26 cm), adult (≤34 cm) and large adult (≤44 cm). We calculated the sensitivity and specificity of BMI values for predicting arm circumference using receiver operating characteristic curves. We report the BMI threshold that maximized Youden's Index for each arm circumference upper limit of a BP cuff.

RESULTS

Participants' mean BMI and arm circumference were 28.8 ± 5.6 kg/m2 and 33.4 ± 4.3 cm, respectively. The BMI-arm circumference Pearson's correlation coefficient was 0.86. The BMI threshold for arm circumference≤26 cm was 23.0 kg/m2, arm circumference≤34 cm was 29.2 kg/m2 and arm circumference≤44 cm was 37.4 kg/m2. Only the BMI threshold for arm circumference≤34 cm varied significantly by sex.

CONCLUSIONS

BMI predicts arm circumference with high sensitivity and specificity and can be an accurate surrogate variable for arm circumference. These findings are useful for participant screening for hypertension trials. Providers can use this information to counsel patients on appropriate cuff size for BP self-monitoring.

Clinical trial recruiters' experiences working with trial eligibility criteria: results of an exploratory, cross-sectional, online survey in the UK

BACKGROUND

Eligibility criteria are a fundamental element of clinical trial design, defining who can and who should not participate in a trial. Problems with the design or application of criteria are known to occur and pose risks to participants' safety and trial integrity, sometimes also negatively impacting on trial recruitment and generalisability. We conducted a short, exploratory survey to gather evidence on UK recruiters' experiences interpreting and applying eligibility criteria and their views on how criteria are communicated and developed.

METHODS

Our survey included topics informed by a wider programme of work at the Clinical Trials Research Unit, University of Leeds, on assuring eligibility criteria quality. Respondents were asked to answer based on all their trial experience, not only on experiences with our trials. The survey was disseminated to recruiters collaborating on trials run at our trials unit, and via other mailing lists and social media. The quantitative responses were descriptively analysed, with inductive analysis of free-text responses to identify themes.

RESULTS

A total of 823 eligible respondents participated. In total, 79% of respondents reported finding problems with eligibility criteria in some trials, and 9% in most trials. The main themes in the types of problems experienced were criteria clarity (67% of comments), feasibility (34%), and suitability (14%). In total, 27% of those reporting some level of problem said these problems had led to patients being incorrectly included in trials; 40% said they had led to incorrect exclusions. Most respondents (56%) reported accessing eligibility criteria mainly in the trial protocol. Most respondents (74%) supported the idea of recruiter review of eligibility criteria earlier in the protocol development process.

CONCLUSIONS

Our survey corroborates other evidence about the existence of suboptimal trial eligibility criteria. Problems with clarity were the most often reported, but the number of comments on feasibility and suitability suggest some recruiters feel eligibility criteria and associated assessments can hinder recruitment to trials. Our proposal for more recruiter involvement in protocol development has strong support and some potential benefits, but questions remain about how best to implement this. We invite other trialists to consider our other suggestions for how to assure quality in trial eligibility criteria.

Application of Information Technology to Clinical Trial Evaluation and Enrollment: A Review

Importance

As cancer treatment has become more individualized, oncologic clinical trials have become more complex. Increasingly numerous and stringent eligibility criteria frequently include tumor molecular or genomic characteristics that may not be readily identified in medical records, rendering it difficult to best match clinical trials with clinical sites and to identify potentially eligible patients once a clinical trial has been selected and activated. Partly because of these factors, enrollment rates for cancer clinical trials remain low, creating delays and increased costs for drug development. Information technology (IT) platforms have been applied to the implementation and conduct of clinical trials to improve efficiencies in several medical fields, and these platforms have recently been introduced to oncologic studies.

Observations

This review summarizes cancer and noncancer studies that used IT platforms for assistance with clinical trial site selection, patient recruitment, and patient screening. The review does not address the use of IT in other aspects of clinical research, such as wearable physical activity monitors or telehealth visits. A large number of IT platforms (which may be patient facing, site or investigator facing, or sponsor facing) are now commercially available. These applications use artificial intelligence and/or natural language processing to identify and summarize protocol eligibility criteria, institutional patient populations, and individual electronic health records. Although there is an expanding body of literature examining the role of this technology, relatively few studies to date have been performed in oncologic settings.

Conclusions and Relevance

This review found that an increasing number and variety of IT platforms were available to assist in the planning and conduct of clinical trials. Because oncologic clinical care and clinical trial protocols are particularly complex, nuanced, and individualized, published experience with this technology in other fields may not be fully applicable to cancer settings. The extent to which these services will overcome ongoing and increasing challenges in cancer clinical research remains unclear.

Analysis of Not Structurable Oncological Study Eligibility Criteria for Improved Patient-Trial Matching

Background  Higher enrolment rates of cancer patients into clinical trials are necessary to increase cancer survival. As a prerequisite, an improved semiautomated matching of patient characteristics with clinical trial eligibility criteria is needed. This is based on the computer interpretability, i.e., structurability of eligibility criteria texts. To increase structurability, the common content, phrasing, and structuring problems of oncological eligibility criteria need to be better understood.

Objectives  We aimed to identify oncological eligibility criteria that were not possible to be structured by our manual approach and categorize them by the underlying structuring problem. Our results shall contribute to improved criteria phrasing in the future as a prerequisite for increased structurability.

Methods  The inclusion and exclusion criteria of 159 oncological studies from the Clinical Trial Information System of the National Center for Tumor Diseases Heidelberg were manually structured and grouped into content-related subcategories. Criteria identified as not structurable were analyzed further and manually categorized by the underlying structuring problem.

Results  The structuring of criteria resulted in 4,742 smallest meaningful components (SMCs) distributed across seven main categories (Diagnosis, Therapy, Laboratory, Study, Findings, Demographics, and Lifestyle, Others). A proportion of 645 SMCs (13.60%) was not possible to be structured due to content- and structure-related issues. Of these, a subset of 415 SMCs (64.34%) was considered not remediable, as supplementary medical knowledge would have been needed or the linkage among the sentence components was too complex. The main category “Diagnosis and Study” contained these two subcategories to the largest parts and thus were the least structurable. In the inclusion criteria, reasons for lacking structurability varied, while missing supplementary medical knowledge was the largest factor within the exclusion criteria.

Conclusion  Our results suggest that further improvement of eligibility criterion phrasing only marginally contributes to increased structurability. Instead, physician-based confirmation of the matching results and the exclusion of factors harming the patient or biasing the study is needed.

Identifying Patient Phenotype Cohorts Using Prehospital Electronic Health Record Data

Objective: Emergency medical services (EMS) provide critical interventions for patients with acute illness and injury and are important in implementing prehospital emergency care research. Retrospective, manual patient record review, the current reference-standard for identifying patient cohorts, requires significant time and financial investment. We developed automated classification models to identify eligible patients for prehospital clinical trials using EMS clinical notes and compared model performance to manual review.Methods: With eligibility criteria for an ongoing prehospital study of chest pain patients, we used EMS clinical notes (n = 1208) to manually classify patients as eligible, ineligible, and indeterminate. We randomly split these same records into training and test sets to develop and evaluate machine-learning (ML) algorithms using natural language processing (NLP) for feature (variable) selection. We compared models to the manual classification to calculate sensitivity, specificity, accuracy, positive predictive value, and F1 measure. We measured clinical expert time to perform review for manual and automated methods.Results: ML models' sensitivity, specificity, accuracy, positive predictive value, and F1 measure ranged from 0.93 to 0.98. Compared to manual classification (N = 363 records), the automated method excluded 90.9% of records as ineligible and leaving only 33 records for manual review.Conclusions: Our ML derived approach demonstrates the feasibility of developing a high-performing, automated classification system using EMS clinical notes to streamline the identification of a specific cardiac patient cohort. This efficient approach can be leveraged to facilitate prehospital patient-trial matching, patient phenotyping (i.e. influenza-like illness), and create prehospital patient registries.

Leveraging Real-World Data for the Selection of Relevant Eligibility Criteria for the Implementation of Electronic Recruitment Support in Clinical Trials

BACKGROUND

Even though clinical trials are indispensable for medical research, they are frequently impaired by delayed or incomplete patient recruitment, resulting in cost overruns or aborted studies. Study protocols based on real-world data with precisely expressed eligibility criteria and realistic cohort estimations are crucial for successful study execution. The increasing availability of routine clinical data in electronic health records (EHRs) provides the opportunity to also support patient recruitment during the prescreening phase. While solutions for electronic recruitment support have been published, to our knowledge, no method for the prioritization of eligibility criteria in this context has been explored.

METHODS

In the context of the Electronic Health Records for Clinical Research (EHR4CR) project, we examined the eligibility criteria of the KATHERINE trial. Criteria were extracted from the study protocol, deduplicated, and decomposed. A paper chart review and data warehouse query were executed to retrieve clinical data for the resulting set of simplified criteria separately from both sources. Criteria were scored according to disease specificity, data availability, and discriminatory power based on their content and the clinical dataset.

RESULTS

The study protocol contained 35 eligibility criteria, which after simplification yielded 70 atomic criteria. For a cohort of 106 patients with breast cancer and neoadjuvant treatment, 47.9% of data elements were captured through paper chart review, with the data warehouse query yielding 26.9% of data elements. Score application resulted in a prioritized subset of 17 criteria, which yielded a sensitivity of 1.00 and specificity 0.57 on EHR data (paper charts, 1.00 and 0.80) compared with actual recruitment in the trial.

CONCLUSION

It is possible to prioritize clinical trial eligibility criteria based on real-world data to optimize prescreening of patients on a selected subset of relevant and available criteria and reduce implementation efforts for recruitment support. The performance could be further improved by increasing EHR data coverage.

Predicting Missing Values in Medical Data via XGBoost Regression

Purpose

The data in a patient's laboratory test result is a notable resource to support clinical investigation and enhance medical research. However, for a variety of reasons, this type of data often contains a non-trivial number of missing values. For example, physicians may neglect to order tests or document the results. Such a phenomenon reduces the degree to which this data can be utilized to learn efficient and effective predictive models. To address this problem, various approaches have been developed to impute missing laboratory values; however, their performance has been limited. This is due, in part, to the fact no approaches effectively leverage the contextual information 1) in individual or 2) between laboratory test variables.

Method

We introduce an approach to combine an unsupervised prefilling strategy with a supervised machine learning approach, in the form of extreme gradient boosting (XGBoost), to leverage both types of context for imputation purposes. We evaluated the methodology through a series of experiments on approximately 8,200 patients' records in the MIMIC-III dataset.

Result

The results demonstrate that the new model outperforms baseline and state-of-the-art models on 13 commonly collected laboratory test variables. In terms of the normalized root mean square derivation (nRMSD), our model exhibits an imputation improvement by over 20%, on average.

Conclusion

Missing data imputation on the temporal variables can be largely improved via prefilling strategy and the supervised training technique, which leverages both the longitudinal and cross-sectional context simultaneously.

Text-mining in electronic healthcare records can be used as efficient tool for screening and data collection in cardiovascular trials: a multicenter validation study

OBJECTIVE

This study aimed to validate trial patient eligibility screening and baseline data collection using text-mining in electronic healthcare records (EHRs), comparing the results to those of an international trial.

STUDY DESIGN AND SETTING

In three medical centers with different EHR vendors, EHR-based text-mining was used to automatically screen patients for trial eligibility and extract baseline data on nineteen characteristics. First, the yield of screening with automated EHR text-mining search was compared with manual screening by research personnel. Second, the accuracy of extracted baseline data by EHR text mining was compared to manual data entry by research personnel.

RESULTS

Of the 92,466 patients visiting the out-patient cardiology departments, 568 (0.6%) were enrolled in the trial during its recruitment period using manual screening methods. Automated EHR data screening of all patients showed that the number of patients needed to screen could be reduced by 73,863 (79.9%). The remaining 18,603 (20.1%) contained 458 of the actual participants (82.4% of participants). In trial participants, automated EHR text-mining missed a median of 2.8% (Interquartile range [IQR] across all variables 0.4-8.5%) of all data points compared to manually collected data. The overall accuracy of automatically extracted data was 88.0% (IQR 84.7-92.8%).

CONCLUSION

Automatically extracting data from EHRs using text-mining can be used to identify trial participants and to collect baseline information.

Perinatal epidemiology: Issues, challenges, and potential solutions

Perinatal epidemiology research is concerned with identifying the effects of events during pregnancy on pregnancy outcomes that include maternal, fetal, and neonatal health outcomes. Randomized trials in perinatal research face many challenges, including randomization difficulties, ethical considerations, and inadequate statistical power due to the small number of subjects eligible for participation. For these reasons, most epidemiological studies conducted in this research field are observational and include different types of bias. This review describes the key methodological difficulties in the design and analysis of randomized and observational studies in perinatal epidemiology, and provides potential corrective approaches.

Five analytic challenges in working with electronic health records data to support clinical trials with some solutions

Electronic health records data are becoming a key data resource in clinical research. Owing to issues of data efficiency, electronic health records data are being used for clinical trials. This includes both large-scale pragmatic trails and smaller—more focused—point-of-care trials. While electronic health records data open up a number of scientific opportunities, they also present a number of analytic challenges. This article discusses five particular challenges related to organizing electronic health records data for analytic purposes. These are as follows: (1) data are not organized for research purposes, (2) data are both densely and irregularly observed, (3) we don’t have all data elements we may want or need, (4) data are both cross-sectional and longitudinal, and (5) data may be informatively observed. While laying out these challenges, the article notes how many of these challenges can be addressed by careful and thoughtful study design as well as by integration of clinicians and informaticians into the analytic team.

Computable Phenotype Implementation for a National, Multicenter Pragmatic Clinical Trial: Lessons Learned From ADAPTABLE

BACKGROUND

Many large-scale cardiovascular clinical trials are plagued with escalating costs and low enrollment. Implementing a computable phenotype, which is a set of executable algorithms, to identify a group of clinical characteristics derivable from electronic health records or administrative claims records, is essential to successful recruitment in large-scale pragmatic clinical trials. This methods paper provides an overview of the development and implementation of a computable phenotype in ADAPTABLE (Aspirin Dosing: a Patient-Centric Trial Assessing Benefits and Long-Term Effectiveness)-a pragmatic, randomized, open-label clinical trial testing the optimal dose of aspirin for secondary prevention of atherosclerotic cardiovascular disease events.

METHODS AND RESULTS

A multidisciplinary team developed and tested the computable phenotype to identify adults ≥18 years of age with a history of atherosclerotic cardiovascular disease without safety concerns around using aspirin and meeting trial eligibility criteria. Using the computable phenotype, investigators identified over 650 000 potentially eligible patients from the 40 participating sites from Patient-Centered Outcomes Research Network-a network of Clinical Data Research Networks, Patient-Powered Research Networks, and Health Plan Research Networks. Leveraging diverse recruitment methods, sites enrolled 15 076 participants from April 2016 to June 2019. During the process of developing and implementing the ADAPTABLE computable phenotype, several key lessons were learned. The accuracy and utility of a computable phenotype are dependent on the quality of the source data, which can be variable even with a common data model. Local validation and modification were required based on site factors, such as recruitment strategies, data quality, and local coding patterns. Sustained collaboration among a diverse team of researchers is needed during computable phenotype development and implementation.

CONCLUSIONS

The ADAPTABLE computable phenotype served as an efficient method to recruit patients in a multisite pragmatic clinical trial. This process of development and implementation will be informative for future large-scale, pragmatic clinical trials. Registration: URL: https://www.clinicaltrials.gov; Unique identifier: NCT02697916.

Conceptual Design, Implementation, and Evaluation of Generic and Standard-Compliant Data Transfer into Electronic Health Records

Objectives  The objective of this study is the conceptual design, implementation and evaluation of a system for generic, standard-compliant data transfer into electronic health records (EHRs). This includes patient data from clinical research and medical care that has been semantically annotated and enhanced with metadata. The implementation is based on the single-source approach. Technical and clinical feasibilities, as well as cost-benefit efficiency, were investigated in everyday clinical practice.

Methods  Münster University Hospital is a tertiary care hospital with 1,457 beds and 10,823 staff who treated 548,110 patients in 2018. Single-source metadata architecture transformation (SMA:T) was implemented as an extension to the EHR system. This architecture uses Model Driven Software Development (MDSD) to generate documentation forms according to the Clinical Data Interchange Standards Consortium (CDISC) operational data model (ODM). Clinical data are stored in ODM format in the EHR system database. Documentation forms are based on Google's Material Design Standard. SMA:T was used at a total of five clinics and one administrative department in the period from March 1, 2018 until March 31, 2019 in everyday clinical practice.

Results  The technical and clinical feasibility of SMA:T was demonstrated in the course of the study. Seventeen documentation forms including 373 data items were created with SMA:T. Those were created for 2,484 patients by 283 users in everyday clinical practice. A total of 121 documentation forms were examined retrospectively. The Constructive cost model (COCOMO II) was used to calculate cost and time savings. The form development mean time was reduced by 83.4% from 3,357 to 557 hours. Average costs per form went down from EUR 953 to 158.

Conclusion  Automated generic transfer of standard-compliant data and metadata into EHRs is technically and clinically feasible, cost efficient, and a useful method to establish comprehensive and semantically annotated clinical documentation. Savings of time and personnel resources are possible.

A systematic review identifying common data items in neonatal trials and assessing their completeness in routinely recorded United Kingdom national neonatal data

Background

We aimed to test whether a common set of key data items reported across high-impact neonatal clinical trials could be identified, and to quantify their completeness in routinely recorded United Kingdom neonatal data held in the National Neonatal Research Database (NNRD).

Methods

We systematically reviewed neonatal clinical trials published in four high-impact medical journals over 10 years (2006–2015) and extracted baseline characteristics, stratification items and potential confounders used to adjust primary outcomes. Completeness was examined using data held in the NNRD for identified data items, for infants admitted to neonatal units in 2015. The NNRD is a repository of routinely recorded data extracted from neonatal Electronic Patient Records (EPR) of all admissions to National Health Service (NHS) Neonatal Units in England, Wales and Scotland. We defined missing data as an empty field or an implausible value. We reported common data items as frequencies and percentages alongside percentages of completeness.

Results

We identified 44 studies involving 32,095 infants and 126 data items. Fourteen data items were reported by more than 20% of studies. Gestational age (95%), sex (93%) and birth weight (91%) were the most common baseline data items. The completeness of data in the NNRD was high for these data with greater than 90% completeness found for 9 of the 14 most common items.

Conclusion

High-impact neonatal clinical trials share common data items. In the United Kingdom, these items can be obtained at a high level of completeness from routinely recorded data held in the NNRD. The feasibility and efficiency using routinely recorded EPR data, such as that held in the NNRD, for clinical trials, rather than collecting these items anew, should be examined.

Trial registration

PROSPERO registration number CRD42016046138. Registered prospectively on 17 August 2016.

Accuracy of the medication list in the electronic health record-implications for care, research, and improvement

Objective

Electronic medication lists may be useful in clinical decision support and research, but their accuracy is not well described. Our aim was to assess the completeness of the medication list compared to the clinical narrative in the electronic health record.

Methods

We reviewed charts of 30 patients with inflammatory bowel disease (IBD) from each of 6 gastroenterology centers. Centers compared IBD medications from the medication list to the clinical narrative.

Results

We reviewed 379 IBD medications among 180 patients. There was variation by center, from 90% patients with complete agreement between the medication list and clinical narrative to 50% agreement.

Conclusions

There was a range in the accuracy of the medication list compared to the clinical narrative. This information may be helpful for sites seeking to improve data quality and those seeking to use medication list data for research or clinical decision support.

Development of Patient Databases for Endocrinological Clinical and Pharmaceutical Trials: A Survey

BACKGROUND

According to European legislation, a clinical trial is a research involving patients, which also includes a research end-product. The main objective of the clinical trial is to prove that the research product, i.e. a proposed medication or treatment, is effective and safe for patients. The implementation, development, and operation of a patient database, which will function as a matrix of samples with the appropriate parameterization, may provide appropriate tools to generate samples for clinical trials.

AIMS

The aim of the present work is to review the literature with respect to the up-to-date progress on the development of databases for clinical trials and patient recruitment using free and open-source software in the field of endocrinology.

METHODS

An electronic literature search was conducted by the authors from 1984 to June 2019. Original articles and systematic reviews selected, and the titles and abstracts of papers screened to determine whether they met the eligibility criteria, and full texts of the selected articles were retrieved.

RESULTS

The present review has indicated that the electronic health records are related with both the patient recruitment and the decision support systems in the domain of endocrinology. The free and open-source software provides integrated solutions concerning electronic health records, patient recruitment, and the decision support systems.

CONCLUSION

The patient recruitment relates closely to the electronic health record. There is maturity at the academic and research level, which may lead to good practices for the deployment of the electronic health record in selecting the right patients for clinical trials.

Common Data Elements for Acute Coronary Syndrome: Analysis Based on the Unified Medical Language System

BACKGROUND

Standardization in clinical documentation can increase efficiency and can save time and resources.

OBJECTIVE

The objectives of this work are to compare documentation forms for acute coronary syndrome (ACS), check for standardization, and generate a list of the most common data elements using semantic form annotation with the Unified Medical Language System (UMLS).

METHODS

Forms from registries, studies, risk scores, quality assurance, official guidelines, and routine documentation from four hospitals in Germany were semantically annotated using UMLS. This allowed for automatic comparison of concept frequencies and the generation of a list of the most common concepts.

RESULTS

A total of 3710 forms items from 86 sources were semantically annotated using 842 unique UMLS concepts. Half of all medical concept occurrences were covered by 60 unique concepts, which suggests the existence of a core dataset of relevant concepts. Overlap percentages between forms were relatively low, hinting at inconsistent documentation structures and lack of standardization.

CONCLUSIONS

This analysis shows a lack of standardized and semantically enriched documentation for patients with ACS. Efforts made by official institutions like the European Society for Cardiology have not yet been fully implemented. Utilizing a standardized and annotated core dataset of the most important data concepts could make export and automatic reuse of data easier. The generated list of common data elements is an exemplary implementation suggestion of the concepts to use in a standardized approach.

Clinical Workflow and Substance Use Screening, Brief Intervention, and Referral to Treatment Data in the Electronic Health Records: A National Drug Abuse Treatment Clinical Trials Network Study

Introduction:

The use of electronic health records (EHR) data in research to inform recruitment and outcomes is considered a critical element for pragmatic studies. However, there is a lack of research on the availability of substance use disorder (SUD) treatment data in the EHR to inform research.

Methods:

This study recruited providers who used an EHR for patient care and whose facilities were affiliated with the National Institute on Drug Abuse’s National Drug Abuse Treatment Clinical Trials Network (NIDA CTN). Data about providers’ use of an EHR and other methods to support and document clinical tasks for Substance use screening, Brief Intervention, and Referral to Treatment (SBIRT) were collected.

Results:

Participants (n = 26) were from facilities across the country (South 46.2%, West 23.1%, Midwest 19.2 percent, Northeast 11.5 percent), representing 26 different health systems/facilities at various settings: primary care (30.8 percent), ambulatory other/specialty (26.9 percent), mixed setting (11.5 percent), hospital outpatient (11.5 percent), emergency department (7.7 percent), inpatient (3.8 percent), and other (7.7 percent). Validated tools were rarely used for substance use screen and SUD assessment. Structured and unstructured EHR fields were commonly used to document SBIRT. The following tasks had high proportions of using unstructured EHR fields: substance use screen, treatment exploration, brief intervention, referral, and follow-up.

Conclusion:

This study is the first of its kind to investigate the documentation of SBIRT in the EHR outside of unique settings (e.g., Veterans Health Administration). While results are descriptive, they emphasize the importance of developing EHR features to collect structured data for SBIRT to improve health care quality evaluation and SUD research.

Establishing an electronic health record-supported approach for outreach to and recruitment of persons at high risk of type 2 diabetes in clinical trials: The vitamin D and type 2 diabetes (D2d) study experience

AIMS

To establish recruitment approaches that leverage electronic health records in multicenter prediabetes/diabetes clinical trials and compare recruitment outcomes between electronic health record-supported and conventional recruitment methods.

METHODS

Observational analysis of recruitment approaches in the vitamin D and type 2 diabetes (D2d) study, a multicenter trial in participants with prediabetes. Outcomes were adoption of electronic health record-supported recruitment approaches by sites, number of participants screened, recruitment performance (proportion screened who were randomized), and characteristics of participants from electronic health record-supported versus non-electronic health record methods.

RESULTS

In total, 2423 participants were randomized: 1920 from electronic health record (mean age of 60 years, 41% women, 68% White) and 503 from non-electronic health record sources (mean age of 56.9 years, 58% women, 61% White). Electronic health record-supported recruitment was adopted by 21 of 22 sites. Electronic health record-supported recruitment was associated with more participants screened versus non-electronic health record methods (4969 vs 2166 participants screened), higher performance (38.6% vs 22.7%), and more randomizations (1918 vs 505). Participants recruited via electronic health record were older, included fewer women and minorities, and reported higher use of dietary supplements. Electronic health record-supported recruitment was incorporated in diverse clinical environments, engaging clinicians either at the individual or the healthcare system level.

CONCLUSION

Establishing electronic health record-supported recruitment approaches across a multicenter prediabetes/diabetes trial is feasible and can be adopted by diverse clinical environments.

An outcome model approach to transporting a randomized controlled trial results to a target population

OBJECTIVE

Participants enrolled into randomized controlled trials (RCTs) often do not reflect real-world populations. Previous research in how best to transport RCT results to target populations has focused on weighting RCT data to look like the target data. Simulation work, however, has suggested that an outcome model approach may be preferable. Here, we describe such an approach using source data from the 2 × 2 factorial NAVIGATOR (Nateglinide And Valsartan in Impaired Glucose Tolerance Outcomes Research) trial, which evaluated the impact of valsartan and nateglinide on cardiovascular outcomes and new-onset diabetes in a prediabetic population.

MATERIALS AND METHODS

Our target data consisted of people with prediabetes serviced at the Duke University Health System. We used random survival forests to develop separate outcome models for each of the 4 treatments, estimating the 5-year risk difference for progression to diabetes, and estimated the treatment effect in our local patient populations, as well as subpopulations, and compared the results with the traditional weighting approach.

RESULTS

Our models suggested that the treatment effect for valsartan in our patient population was the same as in the trial, whereas for nateglinide treatment effect was stronger than observed in the original trial. Our effect estimates were more efficient than the weighting approach and we effectively estimated subgroup differences.

CONCLUSIONS

The described method represents a straightforward approach to efficiently transporting an RCT result to any target population.

A review of the impact of utilising electronic medical records for clinical research recruitment

INTRODUCTION

Recruitment is an important aspect of clinical research, as poor recruitment could undermine the scientific value of a trial or delay the development process of new treatments. The development of electronic medical records provides a new way to identify potential participants for trials by matching the eligibility criteria with patients' data within electronic medical records.

METHODS

A literature search was performed to examine the effectiveness and efficiency of the electronic medical record recruitment method using MEDLINE, PubMed, PubMed Central, CINAHL Plus with Full Text, ScienceDirect and Cochrane Library databases. These searches generated 11 articles that met the eligibility criteria, and handsearching reference lists generated two additional articles bringing the total number of articles to 13. These articles were subjected to critical appraisal utilising the Effective Public Health Practice Project tool.

RESULTS

Out of the 13 included articles, 11 provided quantitative data on recruitment effectiveness while seven articles provided quantitative data on recruitment efficiency. The automation in screening and patient identification by using alerts, a notification system, to notify research staff of a potential participant, was observed to contribute to higher recruitment yield and reduced workload due to its specificity on participant screening. The use of electronic medical record alerts was found to be associated with better recruitment outcomes when they were sent to dedicated research staff rather than physicians. Using electronic medical records for recruitment was found to be effective due to its capability for patient identification outside working hours and fast processing time, which was particularly useful for clinical trials in acute conditions. Several challenges may hinder the impact of the electronic medical record recruitment method, including the lack of conformity of clinical trial eligibility criteria and electronic medical record data structure and missing data. 'Alert fatigue' could also impact on the effectiveness of this method in the long term.

CONCLUSION

The results from this review supports electronic medical record being an effective and efficient method for clinical trial recruitment. Recommendations were made in order to maximise the potential of the electronic medical record recruitment method and also for future research in order to improve the quality of evidence to support this strategy for recruitment.

A Curated Cancer Clinical Outcomes Database (C3OD) for accelerating patient recruitment in cancer clinical trials

Data used to determine patient eligibility for cancer clinical trials often come from disparate sources that are typically maintained by different groups within an institution, use differing technologies, and are stored in different formats. Collecting data and resolving inconsistencies across sources increase the time it takes to screen eligible patients, potentially delaying study completion. To address these challenges, the Biostatistics and Informatics Shared Resource at The University of Kansas Cancer Center developed the Curated Cancer Clinical Outcomes Database (C3OD). C3OD merges data from the electronic medical record, tumor registry, bio-specimen and data registry, and allows querying through a single unified platform. By centralizing access and maintaining appropriate controls, C3OD allows researchers to more rapidly obtain detailed information about each patient in order to accelerate eligibility screening. This case report describes the design of this informatics platform as well as initial assessments of its reliability and usability.

MIRACUM: Medical Informatics in Research and Care in University Medicine

INTRODUCTION

This article is part of the Focus Theme of Methods of Information in Medicine on the German Medical Informatics Initiative. Similar to other large international data sharing networks (e.g. OHDSI, PCORnet, eMerge, RD-Connect) MIRACUM is a consortium of academic and hospital partners as well as one industrial partner in eight German cities which have joined forces to create interoperable data integration centres (DIC) and make data within those DIC available for innovative new IT solutions in patient care and medical research.

OBJECTIVES

Sharing data shall be supported by common interoperable tools and services, in order to leverage the power of such data for biomedical discovery and moving towards a learning health system. This paper aims at illustrating the major building blocks and concepts which MIRACUM will apply to achieve this goal.

GOVERNANCE AND POLICIES

Besides establishing an efficient governance structure within the MIRACUM consortium (based on the steering board, a central administrative office, the general MIRACUM assembly, six working groups and the international scientific advisory board), defining DIC governance rules and data sharing policies, as well as establishing (at each MIRACUM DIC site, but also for MIRACUM in total) use and access committees are major building blocks for the success of such an endeavor.

ARCHITECTURAL FRAMEWORK AND METHODOLOGY

The MIRACUM DIC architecture builds on a comprehensive ecosystem of reusable open source tools (MIRACOLIX), which are linkable and interoperable amongst each other, but also with the existing software environment of the MIRACUM hospitals. Efficient data protection measures, considering patient consent, data harmonization and a MIRACUM metadata repository as well as a common data model are major pillars of this framework. The methodological approach for shared data usage relies on a federated querying and analysis concept.

USE CASES

MIRACUM aims at proving the value of their DIC with three use cases: IT support for patient recruitment into clinical trials, the development and routine care implementation of a clinico-molecular predictive knowledge tool, and molecular-guided therapy recommendations in molecular tumor boards.

RESULTS

Based on the MIRACUM DIC release in the nine months conceptual phase first large scale analysis for stroke and colorectal cancer cohorts have been pursued.

DISCUSSION

Beyond all technological challenges successfully applying the MIRACUM tools for the enrichment of our knowledge about diagnostic and therapeutic concepts, thus supporting the concept of a Learning Health System will be crucial for the acceptance and sustainability in the medical community and the MIRACUM university hospitals.

Improving the Patient-Clinician Interface of Clinical Trials through Health Informatics Technologies

The current state of clinical trials underscores a need for timely interventions to reduce the cost and length of the average trial. Newly developed health informatics technologies-including electronic health records, telemedicine systems, and mobile health applications-have recently been employed in a wide range of clinical trials in an effort to improve different aspects of the clinical trial process. The current review will focus on the observed benefits and drawbacks of using such technology to improve various patient-centered aspects of the clinical trial process, namely its potential to improve patient recruitment, patient retention, and data collection. Broad future challenges and opportunities in the field as a whole will also be covered.

Possible Sources of Bias in Primary Care Electronic Health Record Data Use and Reuse

BACKGROUND

Enormous amounts of data are recorded routinely in health care as part of the care process, primarily for managing individual patient care. There are significant opportunities to use these data for other purposes, many of which would contribute to establishing a learning health system. This is particularly true for data recorded in primary care settings, as in many countries, these are the first place patients turn to for most health problems.

OBJECTIVE

In this paper, we discuss whether data that are recorded routinely as part of the health care process in primary care are actually fit to use for other purposes such as research and quality of health care indicators, how the original purpose may affect the extent to which the data are fit for another purpose, and the mechanisms behind these effects. In doing so, we want to identify possible sources of bias that are relevant for the use and reuse of these type of data.

METHODS

This paper is based on the authors' experience as users of electronic health records data, as general practitioners, health informatics experts, and health services researchers. It is a product of the discussions they had during the Translational Research and Patient Safety in Europe (TRANSFoRm) project, which was funded by the European Commission and sought to develop, pilot, and evaluate a core information architecture for the learning health system in Europe, based on primary care electronic health records.

RESULTS

We first describe the different stages in the processing of electronic health record data, as well as the different purposes for which these data are used. Given the different data processing steps and purposes, we then discuss the possible mechanisms for each individual data processing step that can generate biased outcomes. We identified 13 possible sources of bias. Four of them are related to the organization of a health care system, whereas some are of a more technical nature.

CONCLUSIONS

There are a substantial number of possible sources of bias; very little is known about the size and direction of their impact. However, anyone that uses or reuses data that were recorded as part of the health care process (such as researchers and clinicians) should be aware of the associated data collection process and environmental influences that can affect the quality of the data. Our stepwise, actor- and purpose-oriented approach may help to identify these possible sources of bias. Unless data quality issues are better understood and unless adequate controls are embedded throughout the data lifecycle, data-driven health care will not live up to its expectations. We need a data quality research agenda to devise the appropriate instruments needed to assess the magnitude of each of the possible sources of bias, and then start measuring their impact. The possible sources of bias described in this paper serve as a starting point for this research agenda.

A Text Structuring Method for Chinese Medical Text Based on Temporal Information

Chinese Electronic Medical Records (EMRs) contains a large number of complex medical free text which includes a variety of information, such as temporal information, patients’ symptoms and laboratory data. However, as an important knowledge base, these unstructured text data in EMR are hard to process directly by computer to support further medical research. This paper proposes a novel text structuring method to extract knowledge from EMR texts and reorganize them in chronological order according to the temporal information in the text. By implementing some entropy-based algorithms as contrast, experiments evaluate the performance of the proposed method, which indicates the new method can significantly reduce the complexity of EMR text. This work is significant in structuring the EMR free text into temporal-structured data for further medical analysis.

Clinical Research Informatics: Recent Advances and Future Directions

OBJECTIVES

To summarize significant developments in Clinical Research Informatics (CRI) over the past two years and discuss future directions.

METHODS

Survey of advances, open problems and opportunities in this field based on exploration of current literature.

RESULTS

Recent advances are structured according to three use cases of clinical research: Protocol feasibility, patient identification/ recruitment and clinical trial execution.

DISCUSSION

CRI is an evolving, dynamic field of research. Global collaboration, open metadata, content standards with semantics and computable eligibility criteria are key success factors for future developments in CRI.

Stepwise strategies to successfully recruit diabetes patients in a large research study in Mexican population

AIMS

Describe stepwise strategies (electronic chart review, patient preselection, call-center, personnel dedicated to recruitment) for the successful recruitment of >5000 type 2 diabetes patients in four months.

METHODS

Twenty-five family medicine clinics from Mexico City and the State of Mexico participated: 13 usual care, 6 specialized diabetes care and 6 chronic disease care. Appointments were scheduled from 11/3/2015 to 3/31/2016. Phone calls were generated automatically from an electronic database. A telephone questionnaire verified inclusion criteria, and scheduled an appointment, with a daily report of appointments, patient attendance, acceptance rate, and questionnaire completeness. Another recruitment log reviewed samples collected. Absolute number (percentage) of patients are reported. Means and standard deviations were estimated for continuous variables, χ² test and independent "t" tests were used. OR and 95% CI were estimated.

RESULTS

14,358 appointments were scheduled, 9146 (63.7%) attended their appointment: 5710 (62.4%) fulfilled inclusion criteria and 5244 agreed to participate (91.8% acceptance). Those accepting participation were more likely women, younger and with longer disease duration (p<0.05). The cost of the call-center service was $3,010,000.00 Mexican pesos (∼$31.70 USD per recruited patient).

CONCLUSIONS

Stepwise strategies recruit a high number of patients in a short time. Call centers offer a low cost per patient.

Association of admission and patient characteristics with quality of discharge letters: posthoc analysis of a retrospective study

Background

A complete, correct and timely discharge letter can communicate important information from the hospital to the general practitioner. The adequacy of the letter may vary with the patient and admission characteristics of the patient. Insight in the association between these characteristics and the presence and quality of the discharge letter will give rise to improvement activities for a better continuity of care after discharge.

The objective was to determine the presence, correctness and timeliness of admission information in discharge letters and to determine the association between patient and admission characteristics, including unplanned readmissions and the quality of the discharge letter.

Methods

A post-hoc analysis of a two-staged retrospective patient record review study was performed in 4048 patient records in a random sample of 20 hospitals.

Results

Nearly ten percent of the discharge letters are lacking in patient records in Dutch hospitals. In 59.1% of the discharge letters, one or more relevant components are missing. Important laboratory results, relevant information about consultations, answers to the questions of the referrer, changes in medication and follow up are often lacking. Discharge letters are more likely to be missing in elective patient admissions to a hospital, with a shorter length of stay, less comorbidity, and in readmissions. There was a significant variation in missing discharge letters between hospitals and between hospital departments.

Conclusions

The quality of discharge letters varies with patient and admission characteristics.

Leveraging the EHR4CR platform to support patient inclusion in academic studies: challenges and lessons learned

Background

The development of Electronic Health Records (EHRs) in hospitals offers the ability to reuse data from patient care activities for clinical research. EHR4CR is a European public-private partnership aiming to develop a computerized platform that enables the re-use of data collected from EHRs over its network. However, the reproducibility of queries may depend on attributes of the local data. Our objective was 1/ to describe the different steps that were achieved in order to use the EHR4CR platform and 2/ to identify the specific issues that could impact the final performance of the platform.

Methods

We selected three institutional studies covering various medical domains. The studies included a total of 67 inclusion and exclusion criteria and ran in two University Hospitals. We described the steps required to use the EHR4CR platform for a feasibility study. We also defined metrics to assess each of the steps (including criteria complexity, normalization quality, and data completeness of EHRs).

Results

We identified 114 distinct medical concepts from a total of 67 eligibility criteria Among the 114 concepts: 23 (20.2%) corresponded to non-structured data (i.e. for which transformation is needed before analysis), 92 (81%) could be mapped to terminologies used in EHR4CR, and 86 (75%) could be mapped to local terminologies. We identified 51 computable criteria following the normalization process. The normalization was considered by experts to be satisfactory or higher for 64.2% (43/67) of the computable criteria. All of the computable criteria could be expressed using the EHR4CR platform.

Conclusions

We identified a set of issues that could affect the future results of the platform: (a) the normalization of free-text criteria, (b) the translation into computer-friendly criteria and (c) issues related to the execution of the query to clinical data warehouses. We developed and evaluated metrics to better describe the platforms and their result. These metrics could be used for future reports of Clinical Trial Recruitment Support Systems assessment studies, and provide experts and readers with tools to insure the quality of constructed dataset.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-017-0299-3) contains supplementary material, which is available to authorized users.

Automatic data source identification for clinical trial eligibility criteria resolution

Clinical trial coordinators refer to both structured and unstructured sources of data when evaluating a subject for eligibility. While some eligibility criteria can be resolved using structured data, some require manual review of clinical notes. An important step in automating the trial screening process is to be able to identify the right data source for resolving each criterion. In this work, we discuss the creation of an eligibility criteria dataset for clinical trials for patients with two disparate diseases, annotated with the preferred data source for each criterion (i.e., structured or unstructured) by annotators with medical training. The dataset includes 50 heart-failure trials with a total of 766 eligibility criteria and 50 trials for chronic lymphocytic leukemia (CLL) with 677 criteria. Further, we developed machine learning models to predict the preferred data source: kernel methods outperform simpler learning models when used with a combination of lexical, syntactic, semantic, and surface features. Evaluation of these models indicates that the performance is consistent across data from both diagnoses, indicating generalizability of our method. Our findings are an important step towards ongoing efforts for automation of clinical trial screening.

Completitude das notificações de violência perpetrada contra adolescentes em Pernambuco, Brasil

Resumo O objetivo deste trabalho foi analisar a tendência da completitude dos dados de violência perpetrada contra adolescentes registrados em Pernambuco, em 2009-2012. Estudo transversal, com 5.259 adolescentes vítimas de violência notificadas no SINAN-VIVA da Secretaria Estadual de Saúde de Pernambuco. Utilizou regressão linear simples para investigar a tendência de completitude das variáveis. Os percentuais de completitude foram considerados como variáveis dependentes (Y) e os anos da série, como independentes (X). Os resultados mostram um incremento significativo de 204% no número de notificações. Porém, das 34 variáveis analisadas, 27 (79,4%) apresentaram tendência Estacionária, 6 (17,6%) Decrescente e apenas uma (2,9%) Crescente. A completitude foi considerada ‘Muito Ruim’ para as variáveis: Escolaridade (47,3%), Complemento (21,3%), Hora da Ocorrência (38,0%) e Uso de Álcool Pelo Agressor (47,0%). Portanto, apesar do grande incremento no numero de notificações, a qualidade dos dados permaneceu comprometida, dificultando uma análise mais realista neste grupo.

Common data elements for secondary use of electronic health record data for clinical trial execution and serious adverse event reporting

BACKGROUND

Data capture is one of the most expensive phases during the conduct of a clinical trial and the increasing use of electronic health records (EHR) offers significant savings to clinical research. To facilitate these secondary uses of routinely collected patient data, it is beneficial to know what data elements are captured in clinical trials. Therefore our aim here is to determine the most commonly used data elements in clinical trials and their availability in hospital EHR systems.

METHODS

Case report forms for 23 clinical trials in differing disease areas were analyzed. Through an iterative and consensus-based process of medical informatics professionals from academia and trial experts from the European pharmaceutical industry, data elements were compiled for all disease areas and with special focus on the reporting of adverse events. Afterwards, data elements were identified and statistics acquired from hospital sites providing data to the EHR4CR project.

RESULTS

The analysis identified 133 unique data elements. Fifty elements were congruent with a published data inventory for patient recruitment and 83 new elements were identified for clinical trial execution, including adverse event reporting. Demographic and laboratory elements lead the list of available elements in hospitals EHR systems. For the reporting of serious adverse events only very few elements could be identified in the patient records.

CONCLUSIONS

Common data elements in clinical trials have been identified and their availability in hospital systems elucidated. Several elements, often those related to reimbursement, are frequently available whereas more specialized elements are ranked at the bottom of the data inventory list. Hospitals that want to obtain the benefits of reusing data for research from their EHR are now able to prioritize their efforts based on this common data element list.

Optimizing the Use of Electronic Data Sources in Clinical Trials: The Landscape, Part 1

BACKGROUND

TransCelerate BioPharma has created the eSource Initiative with the intent to facilitate the industry's movement toward optimal usage of electronic data sources. Although guidance and standards have been in place for some time, data collection methods and technology have not been utilized to their fullest capability, and transcription between electronic systems continues to be the norm.

METHODS

The TransCelerate approach for the eSource Initiative is to understand the current landscape and highlight factors that are influencing the adoption of new technologies. As a preliminary step in this process, TransCelerate surveyed member companies regarding eSource usage and barriers.

RESULTS

Literature review, stakeholder engagement, and the member survey have provided insight into the current landscape, which will help TransCelerate to develop proposals for best practices for industry utilization of electronic data collection tools and methods to benefit all stakeholders.

CONCLUSIONS

Based on survey results, companies generally have taken steps to leverage current eSource technologies and prepare for optimal utilization of electronic data sources. The TransCelerate eSource Initiative will continue to evaluate the technology, regulatory, standards, and health care landscape to support the goal of improving global clinical science and global clinical trial execution. Forthcoming publications will focus on technology landscape, future vision, and demonstration projects.

The completeness of electronic medical record data for patients with Type 2 Diabetes in primary care and its implications for computer modelling of predicted clinical outcomes

AIM

To describe the completeness of routinely collected primary care data that could be used by computer models to predict clinical outcomes among patients with Type 2 Diabetes (T2D).

METHODS

Data on blood pressure, weight, total cholesterol, HDL-cholesterol and glycated haemoglobin levels for regular patients were electronically extracted from the medical record software of 12 primary care practices in Australia for the period 2000-2012. The data was analysed for temporal trends and for associations between patient characteristics and completeness. General practitioners were surveyed to identify barriers to recording data and strategies to improve its completeness.

RESULTS

Over the study period data completeness improved up to around 80% complete although the recording of weight remained poorer at 55%. T2D patients with Ischaemic Heart Disease were more likely to have their blood pressure recorded (OR 1.6, p=0.02). Practitioners reported not experiencing any major barriers to using their computer medical record system but did agree with some suggested strategies to improve record completeness.

CONCLUSION

The completeness of routinely collected data suitable for input into computerised predictive models is improving although other dimensions of data quality need to be addressed.

Electronic health records to facilitate clinical research

Electronic health records (EHRs) provide opportunities to enhance patient care, embed performance measures in clinical practice, and facilitate clinical research. Concerns have been raised about the increasing recruitment challenges in trials, burdensome and obtrusive data collection, and uncertain generalizability of the results. Leveraging electronic health records to counterbalance these trends is an area of intense interest. The initial applications of electronic health records, as the primary data source is envisioned for observational studies, embedded pragmatic or post-marketing registry-based randomized studies, or comparative effectiveness studies. Advancing this approach to randomized clinical trials, electronic health records may potentially be used to assess study feasibility, to facilitate patient recruitment, and streamline data collection at baseline and follow-up. Ensuring data security and privacy, overcoming the challenges associated with linking diverse systems and maintaining infrastructure for repeat use of high quality data, are some of the challenges associated with using electronic health records in clinical research. Collaboration between academia, industry, regulatory bodies, policy makers, patients, and electronic health record vendors is critical for the greater use of electronic health records in clinical research. This manuscript identifies the key steps required to advance the role of electronic health records in cardiovascular clinical research.

A Comparison of Health Plan- and Provider-Delivered Chronic Care Management Models on Patient Clinical Outcomes

BACKGROUND

The real world implementation of chronic care management model varies greatly. One aspect of this variation is the delivery mode. Two contrasting strategies include provider-delivered care management (PDCM) and health plan-delivered care management (HPDCM).

OBJECTIVE

We aimed to compare the effectiveness of PDCM vs. HPDCM on improving clinical outcomes for patients with chronic diseases.

DESIGN

We used a quasi-experimental two-group pre-post design using the difference-in-differences method.

PATIENTS

Commercially insured patients, with any of the five chronic diseases-congestive heart failure, chronic obstructive pulmonary disease, coronary heart disease, diabetes, or asthma, who were outreached to and engaged in either PDCM or HPDCM were included in the study.

MAIN MEASURES

Outreached patients were those who received an attempted or actual contact for enrollment in care management; and engaged patients were those who had one or more care management sessions/encounters with a care manager. Effectiveness measures included blood pressure, low density lipoprotein (LDL), weight loss, and hemoglobin A1c (for diabetic patients only). Primary endpoints were evaluated in the first year of follow-up.

KEY RESULTS

A total of 4,000 patients were clustered in 165 practices (31 in PDCM and 134 in HPDCM). The PDCM approach demonstrated a statistically significant improvement in the proportion of outreached patients whose LDL was under control: the proportion of patients with LDL < 100 mg/dL increased by 3 % for the PDCM group (95 % CI: 1 % to 6 %) and 1 % for the HPDCM group (95 % CI: -2 % to 5 %). However, the 2 % difference in these improvements was not statistically significant (95 % CI: -2 % to 6 %). The HPDCM approach showed 3 % [95 % CI: 2 % to 6 %] improvement in overall diabetes care among outreached patients and significant reduction in obesity rates compared to PDCM (4 %, 95 % CI: 0.3 % to 8 %).

CONCLUSIONS

Both care management delivery modes may be viable options for improving care for patients with chronic diseases. In this commercially insured population, neither PDCM nor HPDCM resulted in substantial improvement in patients' clinical indicators in the first year. Different care management strategies within the provider-delivered programs need further investigation.

Temporal bone radiology report classification using open source machine learning and natural langue processing libraries

Background

Radiology reports are a rich resource for biomedical research. Prior to utilization, trained experts must manually review reports to identify discrete outcomes. The Audiological and Genetic Database (AudGenDB) is a public, de-identified research database that contains over 16,000 radiology reports. Because the reports are unlabeled, it is difficult to select those with specific abnormalities. We implemented a classification pipeline using a human-in-the-loop machine learning approach and open source libraries to label the reports with one or more of four abnormality region labels: inner, middle, outer, and mastoid, indicating the presence of an abnormality in the specified ear region.

Methods

Trained abstractors labeled radiology reports taken from AudGenDB to form a gold standard. These were split into training (80 %) and test (20 %) sets. We applied open source libraries to normalize and convert every report to an n-gram feature vector. We trained logistic regression, support vector machine (linear and Gaussian), decision tree, random forest, and naïve Bayes models for each ear region. The models were evaluated on the hold-out test set.

Results

Our gold-standard data set contained 726 reports. The best classifiers were linear support vector machine for inner and outer ear, logistic regression for middle ear, and decision tree for mastoid. Classifier test set accuracy was 90 %, 90 %, 93 %, and 82 % for the inner, middle, outer and mastoid regions, respectively. The logistic regression method was very consistent, achieving accuracy scores within 2.75 % of the best classifier across regions and a receiver operator characteristic area under the curve of 0.92 or greater across all regions.

Conclusions

Our results indicate that the applied methods achieve accuracy scores sufficient to support our objective of extracting discrete features from radiology reports to enhance cohort identification in AudGenDB. The models described here are available in several free, open source libraries that make them more accessible and simplify their utilization as demonstrated in this work. We additionally implemented the models as a web service that accepts radiology report text in an HTTP request and provides the predicted region labels. This service has been used to label the reports in AudGenDB and is freely available.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-016-0306-3) contains supplementary material, which is available to authorized users.

The value of structured data elements from electronic health records for identifying subjects for primary care clinical trials

BACKGROUND

An increasing number of clinical trials are conducted in primary care settings. Making better use of existing data in the electronic health records to identify eligible subjects can improve efficiency of such studies. Our study aims to quantify the proportion of eligibility criteria that can be addressed with data in electronic health records and to compare the content of eligibility criteria in primary care with previous work.

METHODS

Eligibility criteria were extracted from primary care studies downloaded from the UK Clinical Research Network Study Portfolio. Criteria were broken into elemental statements. Two expert independent raters classified each statement based on whether or not structured data items in the electronic health record can be used to determine if the statement was true for a specific patient. Disagreements in classification were discussed until 100 % agreement was reached. Statements were also classified based on content and the percentages of each category were compared to two similar studies reported in the literature.

RESULTS

Eligibility criteria were retrieved from 228 studies and decomposed into 2619 criteria elemental statements. 74 % of the criteria elemental statements were considered likely associated with structured data in an electronic health record. 79 % of the studies had at least 60 % of their criteria statements addressable with structured data likely to be present in an electronic health record. Based on clinical content, most frequent categories were: "disease, symptom, and sign", "therapy or surgery", and "medication" (36 %, 13 %, and 10 % of total criteria statements respectively). We also identified new criteria categories related to provider and caregiver attributes (2.6 % and 1 % of total criteria statements respectively).

CONCLUSIONS

Electronic health records readily contain much of the data needed to assess patients' eligibility for clinical trials enrollment. Eligibility criteria content categories identified by our study can be incorporated as data elements in electronic health records to facilitate their integration with clinical trial management systems.

Textual inference for eligibility criteria resolution in clinical trials

Clinical trials are essential for determining whether new interventions are effective. In order to determine the eligibility of patients to enroll into these trials, clinical trial coordinators often perform a manual review of clinical notes in the electronic health record of patients. This is a very time-consuming and exhausting task. Efforts in this process can be expedited if these coordinators are directed toward specific parts of the text that are relevant for eligibility determination. In this study, we describe the creation of a dataset that can be used to evaluate automated methods capable of identifying sentences in a note that are relevant for screening a patient's eligibility in clinical trials. Using this dataset, we also present results for four simple methods in natural language processing that can be used to automate this task. We found that this is a challenging task (maximum F-score=26.25), but it is a promising direction for further research.

Better big data

By 2018, Medicare payments will be tied to quality of care. The Centers for Medicare and Medicaid Services currently use quality-based metric for some reimbursements through their different programs. Existing and future quality metrics will rely on risk adjustment to avoid unfairly punishing those who see the sickest, highest-risk patients. Despite the limitations of the data used for risk adjustment, there are potential solutions to improve the accuracy of these codes by calibrating data by merging databases and compiling information collected for multiple reporting programs to improve accuracy. In addition, healthcare staff should be informed about the importance of risk adjustment for quality of care assessment and reimbursement. As the number of encounters tied to value-based reimbursements increases in inpatient and outpatient care, coupled with accurate data collection and utilization, the methods used for risk adjustment could be expanded to better account for differences in the care delivered in diverse settings.