A distribution-based method for assessing the differences between clinical trial target populations and patient populations in electronic health records

Computational Phenomapping of Randomized Clinical Trials to Enable Assessment of their Real-world Representativeness and Personalized Inference

Importance

Randomized clinical trials (RCTs) are the standard for defining an evidence-based approach to managing disease, but their generalizability to real-world patients remains challenging to quantify.

Objective

To develop a multidimensional patient variable mapping algorithm to quantify the similarity and representation of electronic health record (EHR) patients corresponding to an RCT and estimate the putative treatment effects in real-world settings based on individual treatment effects observed in an RCT.

Design

A retrospective analysis of the Treatment of Preserved Cardiac Function Heart Failure with an Aldosterone Antagonist Trial (TOPCAT; 2006-2012) and a multi-hospital patient cohort from the electronic health record (EHR) in the Yale New Haven Hospital System (YNHHS; 2015-2023).

Setting

A multicenter international RCT (TOPCAT) and multi-hospital patient cohort (YNHHS).

Participants

All TOPCAT participants and patients with heart failure with preserved ejection fraction (HFpEF) and ≥1 hospitalization within YNHHS.

Exposures

63 pre-randomization characteristics measured across the TOPCAT and YNNHS cohorts.

Main Outcomes and Measures

Real-world generalizability of the RCT TOPCAT using a multidimensional phenotypic distance metric between TOPCAT and YNHHS cohorts. Estimation of the individualized treatment effect of spironolactone use on all-cause mortality within the YNHHS cohort based on phenotypic distance from the TOPCAT cohort.

Results

There were 3,445 patients in TOPCAT and 11,712 HFpEF patients across five hospital sites. Across the 63 TOPCAT variables mapped by clinicians to the EHR, there were larger differences between TOPCAT and each of the 5 EHR sites (median SMD 0.200, IQR 0.037-0.410) than between the 5 EHR sites (median SMD 0.062, IQR 0.010-0.130). The synthesis of these differences across covariates using our multidimensional similarity score also suggested substantial phenotypic dissimilarity between the TOPCAT and EHR cohorts. By phenotypic distance, a majority (55%) of TOPCAT participants were closer to each other than any individual EHR patient. Using a TOPCAT-derived model of individualized treatment benefit from spironolactone, those predicted to derive benefit and receiving spironolactone in the EHR cohorts had substantially better outcomes compared with predicted benefit and not receiving the medication (HR 0.74, 95% CI 0.62-0.89).

Conclusions and Relevance

We propose a novel approach to evaluating the real-world representativeness of RCT participants against corresponding patients in the EHR across the full multidimensional spectrum of the represented phenotypes. This enables the evaluation of the implications of RCTs for real-world patients.

KEY POINTS

Question: How can we examine the multi-dimensional generalizability of randomized clinical trials (RCT) to real-world patient populations?Findings: We demonstrate a novel phenotypic distance metric comparing an RCT to real-world populations in a large multicenter RCT of heart failure patients and the corresponding patients in multisite electronic health records (EHRs). Across 63 pre-randomization characteristics, pairwise assessments of members of the RCT and EHR cohorts were more discordant from each other than between members of the EHR cohort (median standardized mean difference 0.200 [0.037-0.410] vs 0.062 [0.010-0.130]), with a majority (55%) of RCT participants closer to each other than any individual EHR patient. The approach also enabled the quantification of expected real world outcomes based on effects observed in the RCT.Meaning: A multidimensional phenotypic distance metric quantifies the generalizability of RCTs to a given population while also offering an avenue to examine expected real-world patient outcomes based on treatment effects observed in the RCT.

Longitudinal natural history studies based on real-world data in rare diseases: Opportunity and a novel approach

Growing interest in therapeutic development for rare diseases necessitate a systematic approach to the collection and curation of natural history data that can be applied consistently across this group of heterogenous rare diseases. In this study, we discuss the challenges facing natural history studies for leukodystrophies and detail a novel standardized approach to creating a longitudinal natural history study using existing medical records. Prospective studies are uniquely challenging for rare diseases. Delays in diagnosis and overall rarity limit the timely collection of natural history data. When feasible, prospective studies are often cross-sectional rather than longitudinal and are unlikely to capture pre- or early- symptomatic disease trajectories, limiting their utility in characterizing the full natural history of the disease. Therapeutic development in leukodystrophies is subject to these same obstacles. The Global Leukodystrophy Initiative Clinical Trials Network (GLIA-CTN) comprises of a network of research institutions across the United States, supported by a multi-center biorepository protocol, to map the longitudinal clinical course of disease across leukodystrophies. As part of GLIA-CTN, we developed Standard Operating Procedures (SOPs) that delineated all study processes related to staff training, source documentation, and data sharing. Additionally, the SOP detailed the standardized approach to data extraction including diagnosis, clinical presentation, and medical events, such as age at gastrostomy tube placement. The key variables for extraction were selected through face validity, and common electronic case report forms (eCRF) across leukodystrophies were created to collect analyzable data. To enhance the depth of the data, clinical notes are extracted into "original" and "imputed" encounters, with imputed encounter referring to a historic event (e.g., loss of ambulation 3 months prior). Retrospective Functional Assessments were assigned by child neurologists, using a blinded dual-rater approach and score discrepancies were adjudicated by a third rater. Upon completion of extraction, data source verification is performed. Data missingness was evaluated using statistics. The proposed methodology will enable us to leverage existing medical records to address the persistent gap in natural history data within this unique disease group, allow for assessment of clinical trajectory both pre- and post-formal diagnosis, and promote recruitment of larger cohorts.

A review of research on eligibility criteria for clinical trials

The purpose of this paper is to systematically sort out and analyze the cutting-edge research on the eligibility criteria of clinical trials. Eligibility criteria are important prerequisites for the success of clinical trials. It directly affects the final results of the clinical trials. Inappropriate eligibility criteria will lead to insufficient recruitment, which is an important reason for the eventual failure of many clinical trials. We have investigated the research status of eligibility criteria for clinical trials on academic platforms such as arXiv and NIH. We have classified and sorted out all the papers we found, so that readers can understand the frontier research in this field. Eligibility criteria are the most important part of a clinical trial study. The ultimate goal of research in this field is to formulate more scientific and reasonable eligibility criteria and speed up the clinical trial process. The global research on the eligibility criteria of clinical trials is mainly divided into four main aspects: natural language processing, patient pre-screening, standard evaluation, and clinical trial query. Compared with the past, people are now using new technologies to study eligibility criteria from a new perspective (big data). In the research process, complex disease concepts, how to choose a suitable dataset, how to prove the validity and scientific of the research results, are challenges faced by researchers (especially for computer-related researchers). Future research will focus on the selection and improvement of artificial intelligence algorithms related to clinical trials and related practical applications such as databases, knowledge graphs, and dictionaries.

Rudin R, Sulca Flores J, Goolkasian G, Sousa J, Rodriguez J, Lipsitz S, Foer D, K. Dalal A. Assessing Equitable Recruitment in a Digital Health Trial for Asthma

OBJECTIVE

This study aimed to assess a multipronged strategy using primarily digital methods to equitably recruit asthma patients into a clinical trial of a digital health intervention.

METHODS

We approached eligible patients using at least one of eight recruitment strategies. We recorded approach dates and the strategy that led to completion of a web-based eligibility questionnaire that was reported during the verbal consent phone call. Study team members conducted monthly sessions using a structured guide to identify recruitment barriers and facilitators. The proportion of participants who reported being recruited by a portal or nonportal strategy was measured as our outcomes. We used Fisher's exact test to compare outcomes by equity variable, and multivariable logistic regression to control for each covariate and adjust effect size estimates. Using grounded theory, we coded and extracted themes regarding recruitment barriers and facilitators.

RESULTS

The majority (84.4%) of patients who met study inclusion criteria were patient portal enrollees. Of 6,366 eligible patients who were approached, 627 completed the eligibility questionnaire and were less frequently Hispanic, less frequently Spanish-speaking, and more frequently patient portal enrollees. Of 445 patients who consented to participate, 241 (54.2%) reported completing the eligibility questionnaire after being contacted by a patient portal message. In adjusted analysis, only race (odds ratio [OR]: 0.46, 95% confidence interval [CI]: 0.28-0.77, p = 0.003) and college education (OR: 0.60, 95% CI: 0.39-0.91, p = 0.016) remained significant. Key recruitment barriers included technology issues (e.g., lack of email access) and facilitators included bilingual study staff, Spanish-language recruitment materials, targeted phone calls, and clinician-initiated "1-click" referrals.

CONCLUSION

A primarily digital strategy to recruit patients into a digital health trial is unlikely to achieve equitable participation, even in a population overrepresented by patient portal enrollees. Nondigital recruitment methods that address racial and educational disparities and less active portal enrollees are necessary to ensure equity in clinical trial enrollment.

Calibrating a network meta-analysis of diabetes trials of sodium glucose cotransporter 2 inhibitors, glucagon-like peptide-1 receptor analogues and dipeptidyl peptidase-4 inhibitors to a representative routine population: a systematic review protocol

INTRODUCTION

Participants in randomised controlled trials (trials) are generally younger and healthier than many individuals encountered in clinical practice. Consequently, the applicability of trial findings is often uncertain. To address this, results from trials can be calibrated to more representative data sources. In a network meta-analysis, using a novel approach which allows the inclusion of trials whether or not individual-level participant data (IPD) is available, we will calibrate trials for three drug classes (sodium glucose cotransporter 2 (SGLT2) inhibitors, glucagon-like peptide-1 (GLP1) receptor analogues and dipeptidyl peptidase-4 (DPP4) inhibitors) to the Scottish diabetes register.

METHODS AND ANALYSIS

Medline and EMBASE databases, the US clinical trials registry (clinicaltrials.gov) and the Chinese Clinical Trial Registry (chictr.org.cn) will be searched from 1 January 2002. Two independent reviewers will apply eligibility criteria to identify trials for inclusion. Included trials will be phase 3 or 4 trials of SGLT2 inhibitors, GLP1 receptor analogues or DPP4 inhibitors, with placebo or active comparators, in participants with type 2 diabetes, with at least one of glycaemic control, change in body weight or major adverse cardiovascular event as outcomes. Unregistered trials will be excluded.We have identified a target population from the population-based Scottish diabetes register. The chosen cohort comprises people in Scotland with type 2 diabetes who either (1) require further treatment due to poor glycaemic control where any of the three drug classes may be suitable, or (2) who have adequate glycaemic control but are already on one of the three drug classes of interest or insulin.

ETHICS AND DISSEMINATION

Ethical approval for IPD use was obtained from the University of Glasgow MVLS College Ethics Committee (Project: 200160070). The Scottish diabetes register has approval from the Scottish A Research Ethics Committee (11/AL/0225) and operates with Public Benefit and Privacy Panel for Health and Social Care approval (1617-0147).

PROSPERO REGISTRATION NUMBER

CRD42020184174.

A Framework for Systematic Assessment of Clinical Trial Population Representativeness Using Electronic Health Records Data

BACKGROUND

Clinical trials are the gold standard for generating robust medical evidence, but clinical trial results often raise generalizability concerns, which can be attributed to the lack of population representativeness. The electronic health records (EHRs) data are useful for estimating the population representativeness of clinical trial study population.

OBJECTIVES

This research aims to estimate the population representativeness of clinical trials systematically using EHR data during the early design stage.

METHODS

We present an end-to-end analytical framework for transforming free-text clinical trial eligibility criteria into executable database queries conformant with the Observational Medical Outcomes Partnership Common Data Model and for systematically quantifying the population representativeness for each clinical trial.

RESULTS

We calculated the population representativeness of 782 novel coronavirus disease 2019 (COVID-19) trials and 3,827 type 2 diabetes mellitus (T2DM) trials in the United States respectively using this framework. With the use of overly restrictive eligibility criteria, 85.7% of the COVID-19 trials and 30.1% of T2DM trials had poor population representativeness.

CONCLUSION

This research demonstrates the potential of using the EHR data to assess the clinical trials population representativeness, providing data-driven metrics to inform the selection and optimization of eligibility criteria.

Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review

OBJECTIVE

Real-world data (RWD), defined as routinely collected healthcare data, can be a potential catalyst for addressing challenges faced in clinical trials. We performed a scoping review of database-specific RWD applications within clinical trial contexts, synthesizing prominent uses and themes.

MATERIALS AND METHODS

Querying 3 biomedical literature databases, research articles using electronic health records, administrative claims databases, or clinical registries either within a clinical trial or in tandem with methodology related to clinical trials were included. Articles were required to use at least 1 US RWD source. All abstract screening, full-text screening, and data extraction was performed by 1 reviewer. Two reviewers independently verified all decisions.

RESULTS

Of 2020 screened articles, 89 qualified: 59 articles used electronic health records, 29 used administrative claims, and 26 used registries. Our synthesis was driven by the general life cycle of a clinical trial, culminating into 3 major themes: trial process tasks (51 articles); dissemination strategies (6); and generalizability assessments (34). Despite a diverse set of diseases studied, <10% of trials using RWD for trial process tasks evaluated medications or procedures (5/51). All articles highlighted data-related challenges, such as missing values.

DISCUSSION

Database-specific RWD have been occasionally leveraged for various clinical trial tasks. We observed underuse of RWD within conducted medication or procedure trials, though it is subject to the confounder of implicit report of RWD use.

CONCLUSION

Enhanced incorporation of RWD should be further explored for medication or procedure trials, including better understanding of how to handle related data quality issues to facilitate RWD use.

A knowledge base of clinical trial eligibility criteria

OBJECTIVE

We present the Clinical Trial Knowledge Base, a regularly updated knowledge base of discrete clinical trial eligibility criteria equipped with a web-based user interface for querying and aggregate analysis of common eligibility criteria.

MATERIALS AND METHODS

We used a natural language processing (NLP) tool named Criteria2Query (Yuan et al., 2019) to transform free text clinical trial eligibility criteria from ClinicalTrials.gov into discrete criteria concepts and attributes encoded using the widely adopted Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) and stored in a relational SQL database. A web application accessible via RESTful APIs was implemented to enable queries and visual aggregate analyses. We demonstrate CTKB's potential role in EHR phenotype knowledge engineering using ten validated phenotyping algorithms.

RESULTS

At the time of writing, CTKB contained 87,504 distinctive OMOP CDM standard concepts, including Condition (47.82%), Drug (23.01%), Procedure (13.73%), Measurement (24.70%) and Observation (5.28%), with 34.78% for inclusion criteria and 65.22% for exclusion criteria, extracted from 352,110 clinical trials. The average hit rate of criteria concepts in eMERGE phenotype algorithms is 77.56%.

CONCLUSION

CTKB is a novel comprehensive knowledge base of discrete eligibility criteria concepts with the potential to enable knowledge engineering for clinical trial cohort definition, clinical trial population representativeness assessment, electronical phenotyping, and data gap analyses for using electronic health records to support clinical trial recruitment.

How and when informative visit processes can bias inference when using electronic health records data for clinical research

OBJECTIVE

Electronic health records (EHR) data have become a central data source for clinical research. One concern for using EHR data is that the process through which individuals engage with the health system, and find themselves within EHR data, can be informative. We have termed this process informed presence. In this study we use simulation and real data to assess how the informed presence can impact inference.

MATERIALS AND METHODS

We first simulated a visit process where a series of biomarkers were observed informatively and uninformatively over time. We further compared inference derived from a randomized control trial (ie, uninformative visits) and EHR data (ie, potentially informative visits).

RESULTS

We find that only when there is both a strong association between the biomarker and the outcome as well as the biomarker and the visit process is there bias. Moreover, once there are some uninformative visits this bias is mitigated. In the data example we find, that when the "true" associations are null, there is no observed bias.

DISCUSSION

These results suggest that an informative visit process can exaggerate an association but cannot induce one. Furthermore, careful study design can, mitigate the potential bias when some noninformative visits are included.

CONCLUSIONS

While there are legitimate concerns regarding biases that "messy" EHR data may induce, the conditions for such biases are extreme and can be accounted for.

Using Real-World Data to Rationalize Clinical Trials Eligibility Criteria Design: A Case Study of Alzheimer's Disease Trials

Low trial generalizability is a concern. The Food and Drug Administration had guidance on broadening trial eligibility criteria to enroll underrepresented populations. However, investigators are hesitant to do so because of concerns over patient safety. There is a lack of methods to rationalize criteria design. In this study, we used data from a large research network to assess how adjustments of eligibility criteria can jointly affect generalizability and patient safety (i.e the number of serious adverse events [SAEs]). We first built a model to predict the number of SAEs. Then, leveraging an a priori generalizability assessment algorithm, we assessed the changes in the number of predicted SAEs and the generalizability score, simulating the process of dropping exclusion criteria and increasing the upper limit of continuous eligibility criteria. We argued that broadening of eligibility criteria should balance between potential increases of SAEs and generalizability using donepezil trials as a case study.

Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations

Suicide is a leading cause of death that defies prediction and challenges prevention efforts worldwide. Artificial intelligence (AI) and machine learning (ML) have emerged as a means of investigating large datasets to enhance risk detection. A systematic review of ML investigations evaluating suicidal behaviors was conducted using PubMed/MEDLINE, PsychInfo, Web-of-Science, and EMBASE, employing search strings and MeSH terms relevant to suicide and AI. Databases were supplemented by hand-search techniques and Google Scholar. Inclusion criteria: (1) journal article, available in English, (2) original investigation, (3) employment of AI/ML, (4) evaluation of a suicide risk outcome. N = 594 records were identified based on abstract search, and 25 hand-searched reports. N = 461 reports remained after duplicates were removed, n = 316 were excluded after abstract screening. Of n = 149 full-text articles assessed for eligibility, n = 87 were included for quantitative synthesis, grouped according to suicide behavior outcome. Reports varied widely in methodology and outcomes. Results suggest high levels of risk classification accuracy (>90%) and Area Under the Curve (AUC) in the prediction of suicidal behaviors. We report key findings and central limitations in the use of AI/ML frameworks to guide additional research, which hold the potential to impact suicide on broad scale.

Differences in patient characteristics and external validity of randomized clinical trials on pain management following total hip and knee arthroplasty: a systematic review

Background

The external validity of randomized controlled trials (RCTs) is critical for the relevance of trial results in a clinical setting. We aimed to assess the external validity of RCTs investigating postoperative pain treatment after total hip and knee arthroplasty (THA and TKA) by comparing patient characteristics in these trials with a clinical cohort. Further, we assessed the use of exclusion criteria of the included RCTs.

Methods

We searched PubMed, Embase, and Cochrane Central Register of Controlled Trials for relevant RCTs up to June 2019. Data on patient characteristics from this research population were compared with an unselected clinical cohort from the Danish Hip and Knee Arthroplasty Registries in the period 2005–2019. Trends in patient characteristics and the use of exclusion criteria were assessed with control charts.

Results

In total, 550 RCTs with 48 962 participants were included in the research cohort. The clinical cohort included 101 439 THA patients and 90 505 TKA patients. Patient characteristics (age, body mass index (BMI), American Society of Anesthesiologists (ASA) score and sex distribution) in the research cohort resembled those of the clinical cohort. Age, BMI and ASA scores did not change over time in the research cohort. In the clinical cohort, age increased among both THA and TKA patients, and BMI and ASA scores increased among TKA patients. Most commonly used exclusion criteria in the RCTs were high ASA score (62%), older age (45%), obesity (32%) and chronic opioid use (41%). Exclusion of chronic opioid users and individuals with obesity increased over time.

Conclusion

Patient characteristics in research trials investigating postoperative pain management after THA and TKA currently resemble those of a clinical cohort. However, individuals in the clinical cohort are getting older, and TKA patients more obese with increasing ASA scores. Concomitantly, RCTs increase the tendency to exclude patients with older age, obesity, chronic pain and/or opioid use. This trending discrepancy can hinder the generalizability of future research results, and therefore increased focus on pragmatic trials resembling real-world conditions are needed.

PROSPERO registration number

CRD42019125691

Five analytic challenges in working with electronic health records data to support clinical trials with some solutions

Electronic health records data are becoming a key data resource in clinical research. Owing to issues of data efficiency, electronic health records data are being used for clinical trials. This includes both large-scale pragmatic trails and smaller—more focused—point-of-care trials. While electronic health records data open up a number of scientific opportunities, they also present a number of analytic challenges. This article discusses five particular challenges related to organizing electronic health records data for analytic purposes. These are as follows: (1) data are not organized for research purposes, (2) data are both densely and irregularly observed, (3) we don’t have all data elements we may want or need, (4) data are both cross-sectional and longitudinal, and (5) data may be informatively observed. While laying out these challenges, the article notes how many of these challenges can be addressed by careful and thoughtful study design as well as by integration of clinicians and informaticians into the analytic team.

Assessing the Use and Perception of Dietary Supplements Among Obese Patients with National Health and Nutrition Examination Survey

Complementary alternative medicine, especially dietary supplements (DS), has gained increasing popularity for weight loss due to its availability without prescription, price, and ease of use. Besides weight loss, there are various perceived, potential benefits linked to DS use. However, health consumers with limited health literacy may not adequately know the benefits and risk of overdose for DS. In this project, we aim to gain a better understanding of the use of DS products among obese people as well as the perceived benefits of these products. We identified obese adults after combining the National Health and Nutrition Examination Survey data collected from 2003 to 2014. We found that there is a knowledge gap between the reported benefits of major DS by obese adults and the existing DS knowledge base and label database. This gap may inform the design of patient education material on DS usage in the future.

Precision Dosing Priority Criteria: Drug, Disease, and Patient Population Variables

The administered dose of a drug modulates whether patients will experience optimal effectiveness, toxicity including death, or no effect at all. Dosing is particularly important for diseases and/or drugs where the drug can decrease severe morbidity or prolong life. Likewise, dosing is important where the drug can cause death or severe morbidity. Since we believe there are many examples where more precise dosing could benefit patients, it is worthwhile to consider how to prioritize drug-disease targets. One key consideration is the quality of information available from which more precise dosing recommendations can be constructed. When a new more precise dosing scheme is created and differs significantly from the approved label, it is important to consider the level of proof necessary to either change the label and/or change clinical practice. The cost and effort needed to provide this proof should also be considered in prioritizing drug-disease precision dosing targets. Although precision dosing is being promoted and has great promise, it is underutilized in many drugs and disease states. Therefore, we believe it is important to consider how more precise dosing is going to be delivered to high priority patients in a timely manner. If better dosing schemes do not change clinical practice resulting in better patient outcomes, then what is the use? This review paper discusses variables to consider when prioritizing precision dosing candidates while highlighting key examples of precision dosing that have been successfully used to improve patient care.

Clinical Trial Generalizability Assessment in the Big Data Era: A Review

Clinical studies, especially randomized, controlled trials, are essential for generating evidence for clinical practice. However, generalizability is a long‐standing concern when applying trial results to real‐world patients. Generalizability assessment is thus important, nevertheless, not consistently practiced. We performed a systematic review to understand the practice of generalizability assessment. We identified 187 relevant articles and systematically organized these studies in a taxonomy with three dimensions: (i) data availability (i.e., before or after trial (a priori vs. a posteriori generalizability)); (ii) result outputs (i.e., score vs. nonscore); and (iii) populations of interest. We further reported disease areas, underrepresented subgroups, and types of data used to profile target populations. We observed an increasing trend of generalizability assessments, but < 30% of studies reported positive generalizability results. As a priori generalizability can be assessed using only study design information (primarily eligibility criteria), it gives investigators a golden opportunity to adjust the study design before the trial starts. Nevertheless, < 40% of the studies in our review assessed a priori generalizability. With the wide adoption of electronic health records systems, rich real‐world patient databases are increasingly available for generalizability assessment; however, informatics tools are lacking to support the adoption of generalizability assessment practice.

Assessing the Validity of a a priori Patient-Trial Generalizability Score using Real-world Data from a Large Clinical Data Research Network: A Colorectal Cancer Clinical Trial Case Study

Existing trials had not taken enough consideration of their population representativeness, which can lower the effectiveness when the treatment is applied in real-world clinical practice. We analyzed the eligibility criteria of Bevacizumab colorectal cancer treatment trials, assessed their a priori generalizability, and examined how it affects patient outcomes when applied in real-world clinical settings. To do so, we extracted patient-level data from a large collection of electronic health records (EHRs) from the OneFlorida consortium. We built a zero-inflated negative binomial model using a composite patient-trial generalizability (cPTG) score to predict patients' clinical outcomes (i.e., number of serious adverse events, [SAEs]). Our study results provide a body of evidence that 1) the cPTG scores can predict patient outcomes; and 2) patients who are more similar to the study population in the trials that were used to develop the treatment will have a significantly lower possibility to experience serious adverse events.

Patient Engagement in Medical Research Among Older Adults: Analysis of the Health Information National Trends Survey

Background

By 2035, it is expected that older adults (aged 65 years and older) will outnumber children and will represent 78 million people in the US population. As the aging population continues to grow, it is critical to reduce disparities in their representation in medical research.

Objective

This study aimed to describe sociodemographic characteristics and health and information behaviors as factors that influence US adults’ interest in engaging in medical research, beyond participation as study subjects.

Methods

Nationally representative cross-sectional data from the 2014 Health Information National Trends Survey (N=3677) were analyzed. Descriptive statistics and weighted multivariable logistic regression analyses were performed to assess predictors of one’s interest in patient engagement in medical research. The independent variables included age, general health, income, race and ethnicity, education level, insurance status, marital status, and health information behaviors.

Results

We examined the association between the independent variables and patient interest in engaging in medical research (PTEngage_Interested). Patient interest in engaging in medical research has a statistically significant association with age (adjusted P<.01). Younger adults (aged 18-34 years), lower middle-aged adults (aged 35-49 years), and higher middle-aged adults (aged 50-64 years) indicated interest at relatively the same frequency (29.08%, 29.56%, and 25.12%, respectively), but older adults (aged ≥65 years) expressed less interest (17.10%) than the other age groups. After the multivariate model was run, older adults (odds ratio 0.738, 95% CI 0.500-1.088) were found to be significantly less likely to be interested in engaging in medical research than adults aged 50 to 64 years. Regardless of age, the strongest correlation was found between interest in engaging in medical research and actively looking for health information (P<.001). Respondents who did not seek health information were significantly less likely than those who did seek health information to be interested in engaging in medical research.

Conclusions

Patients’ interest in engaging in medical research vary by age and information-seeking behaviors. As the aging population continues to grow, it is critical to reduce disparities in their representation in medical research. Interest in participatory research methods may reflect an opportunity for consumer health informatics technologies to improve the representation of older adults in future medical research.

Learning Personalized Treatment Rules from Electronic Health Records Using Topic Modeling Feature Extraction

To address substantial heterogeneity in patient response to treatment of chronic disorders and achieve the promise of precision medicine, individualized treatment rules (ITRs) are estimated to tailor treatments according to patient-specific characteristics. Randomized controlled trials (RCTs) provide gold standard data for learning ITRs not subject to confounding bias. However, RCTs are often conducted under stringent inclusion/exclusion criteria, and participants in RCTs may not reflect the general patient population. Thus, ITRs learned from RCTs lack generalizability to the broader real world patient population. Real world databases such as electronic health records (EHRs) provide new resources as complements to RCTs to facilitate evidence-based research for personalized medicine. However, to ensure the validity of ITRs learned from EHRs, a number of challenges including confounding bias and selection bias must be addressed. In this work, we propose a matching-based machine learning method to estimate optimal individualized treatment rules from EHRs using interpretable features extracted from EHR documentation of medications and ICD diagnoses codes. We use a latent Dirichlet allocation (LDA) model to extract latent topics and weights as features for learning ITRs. Our method achieves confounding reduction in observational studies through matching treated and untreated individuals and improves treatment optimization by augmenting feature space with clinically meaningful LDA-based features. We apply the method to EHR data collected at New York Presbyterian Hospital clinical data warehouse in studying optimal second-line treatment for type 2 diabetes (T2D) patients. We use cross validation to show that ITRs outperforms uniform treatment strategies (i.e., assigning same treatment to all individuals), and including topic modeling features leads to more reduction of post-treatment complications.

The next generation of precision medicine: observational studies, electronic health records, biobanks and continuous monitoring

Precision medicine can utilize new techniques in order to more effectively translate research findings into clinical practice. In this article, we first explore the limitations of traditional study designs, which stem from (to name a few): massive cost for the assembly of large patient cohorts; non-representative patient data; and the astounding complexity of human biology. Second, we propose that harnessing electronic health records and mobile device biometrics coupled to longitudinal data may prove to be a solution to many of these problems by capturing a 'real world' phenotype. We envision that future biomedical research utilizing more precise approaches to patient care will utilize continuous and longitudinal data sources.

An outcome model approach to transporting a randomized controlled trial results to a target population

OBJECTIVE

Participants enrolled into randomized controlled trials (RCTs) often do not reflect real-world populations. Previous research in how best to transport RCT results to target populations has focused on weighting RCT data to look like the target data. Simulation work, however, has suggested that an outcome model approach may be preferable. Here, we describe such an approach using source data from the 2 × 2 factorial NAVIGATOR (Nateglinide And Valsartan in Impaired Glucose Tolerance Outcomes Research) trial, which evaluated the impact of valsartan and nateglinide on cardiovascular outcomes and new-onset diabetes in a prediabetic population.

MATERIALS AND METHODS

Our target data consisted of people with prediabetes serviced at the Duke University Health System. We used random survival forests to develop separate outcome models for each of the 4 treatments, estimating the 5-year risk difference for progression to diabetes, and estimated the treatment effect in our local patient populations, as well as subpopulations, and compared the results with the traditional weighting approach.

RESULTS

Our models suggested that the treatment effect for valsartan in our patient population was the same as in the trial, whereas for nateglinide treatment effect was stronger than observed in the original trial. Our effect estimates were more efficient than the weighting approach and we effectively estimated subgroup differences.

CONCLUSIONS

The described method represents a straightforward approach to efficiently transporting an RCT result to any target population.

EliIE: An open-source information extraction system for clinical trial eligibility criteria

OBJECTIVE

To develop an open-source information extraction system called Eligibility Criteria Information Extraction (EliIE) for parsing and formalizing free-text clinical research eligibility criteria (EC) following Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.0.

MATERIALS AND METHODS

EliIE parses EC in 4 steps: (1) clinical entity and attribute recognition, (2) negation detection, (3) relation extraction, and (4) concept normalization and output structuring. Informaticians and domain experts were recruited to design an annotation guideline and generate a training corpus of annotated EC for 230 Alzheimer's clinical trials, which were represented as queries against the OMOP CDM and included 8008 entities, 3550 attributes, and 3529 relations. A sequence labeling-based method was developed for automatic entity and attribute recognition. Negation detection was supported by NegEx and a set of predefined rules. Relation extraction was achieved by a support vector machine classifier. We further performed terminology-based concept normalization and output structuring.

RESULTS

In task-specific evaluations, the best F1 score for entity recognition was 0.79, and for relation extraction was 0.89. The accuracy of negation detection was 0.94. The overall accuracy for query formalization was 0.71 in an end-to-end evaluation.

CONCLUSIONS

This study presents EliIE, an OMOP CDM-based information extraction system for automatic structuring and formalization of free-text EC. According to our evaluation, machine learning-based EliIE outperforms existing systems and shows promise to improve.

The representativeness of eligible patients in type 2 diabetes trials: a case study using GIST 2.0

Objective

The population representativeness of a clinical study is influenced by how real-world patients qualify for the study. We analyze the representativeness of eligible patients for multiple type 2 diabetes trials and the relationship between representativeness and other trial characteristics.

Methods

Sixty-nine study traits available in the electronic health record data for 2034 patients with type 2 diabetes were used to profile the target patients for type 2 diabetes trials. A set of 1691 type 2 diabetes trials was identified from ClinicalTrials.gov, and their population representativeness was calculated using the published Generalizability Index of Study Traits 2.0 metric. The relationships between population representativeness and number of traits and between trial duration and trial metadata were statistically analyzed. A focused analysis with only phase 2 and 3 interventional trials was also conducted.

Results

A total of 869 of 1691 trials (51.4%) and 412 of 776 phase 2 and 3 interventional trials (53.1%) had a population representativeness of <5%. The overall representativeness was significantly correlated with the representativeness of the Hba1c criterion. The greater the number of criteria or the shorter the trial, the less the representativeness. Among the trial metadata, phase, recruitment status, and start year were found to have a statistically significant effect on population representativeness. For phase 2 and 3 interventional trials, only start year was significantly associated with representativeness.

Conclusions

Our study quantified the representativeness of multiple type 2 diabetes trials. The common low representativeness of type 2 diabetes trials could be attributed to specific study design requirements of trials or safety concerns. Rather than criticizing the low representativeness, we contribute a method for increasing the transparency of the representativeness of clinical trials.

Assessing the population representativeness of colorectal cancer treatment clinical trials

The generalizability (external validity) of clinical trials has long been a concern for both clinical research community as well as the general public. Results of trials that do not represent the target population may not be applicable to the broader patient population. In this study, we used a previously published metric Generalizability Index for Study Traits (GIST) to assess the population representativeness of colorectal cancer (CRC) treatment trials. Our analysis showed that the quantitative eligibility criteria of CRC trials are in general not restrictive. However, the qualitative eligibility criteria in these trials are with moderate or strict restrictions, which may impact their population representativeness of the real-world patient population.

Computer-aided assessment of the generalizability of clinical trial results

BACKGROUND

The effects of an intervention on patients from populations other than that included in a trial may vary as a result of differences in population features, treatment administration, or general setting. Determining the generalizability of a trial to a target population is important in clinical decision making at both the individual practitioner and policy-making levels. However, awareness to the challenges associated with the assessment of generalizability of trials is low and tools to facilitate such assessment are lacking.

METHODS

We review the main factors affecting the generalizability of a clinical trial results beyond the trial population. We then propose a framework for a standardized evaluation of parameters relevant to determining the external validity of clinical trials to produce a "generalizability score". We then apply this framework to populations of patients with heart failure included in trials, cohorts and registries to demonstrate the use of the generalizability score and its graphic representation along three dimensions: participants' demographics, their clinical profile and intervention setting. We use the generalizability score to compare a single trial to multiple "target" clinical scenarios. Additionally, we present the generalizability score of several studies with regard to a single "target" population.

RESULTS

Similarity indices vary considerably between trials and target population, but inconsistent reporting of participant characteristics limit head-to-head comparisons.

CONCLUSION

We discuss the challenges involved in performing automatic assessment of trial generalizability at scale and propose the adoption of a standard format for reporting the characteristics of trial participants to enable better interpretation of their results.

Comparing and Contrasting A Priori and A Posteriori Generalizability Assessment of Clinical Trials on Type 2 Diabetes Mellitus

Clinical trials are indispensable tools for evidence-based medicine. However, they are often criticized for poor generalizability. Traditional trial generalizability assessment can only be done after the trial results are published, which compares the enrolled patients with a convenience sample of real-world patients. However, the proliferation of electronic data in clinical trial registries and clinical data warehouses offer a great opportunity to assess the generalizability during the design phase of a new trial. In this work, we compared and contrasted a priori (based on eligibility criteria) and a posteriori (based on enrolled patients) generalizability of Type 2 diabetes clinical trials. Further, we showed that comparing the study population selected by the clinical trial eligibility criteria to the real-world patient population is a good indicator of the generalizability of trials. Our findings demonstrate that the a priori generalizability of a trial is comparable to its a posteriori generalizability in identifying restrictive quantitative eligibility criteria.

Learning Optimal Individualized Treatment Rules from Electronic Health Record Data

Medical research is experiencing a paradigm shift from "one-size-fits-all" strategy to a precision medicine approach where the right therapy, for the right patient, and at the right time, will be prescribed. We propose a statistical method to estimate the optimal individualized treatment rules (ITRs) that are tailored according to subject-specific features using electronic health records (EHR) data. Our approach merges statistical modeling and medical domain knowledge with machine learning algorithms to assist personalized medical decision making using EHR. We transform the estimation of optimal ITR into a classification problem and account for the non-experimental features of the EHR data and confounding by clinical indication. We create a broad range of feature variables that reflect both patient health status and healthcare data collection process. Using EHR data collected at Columbia University clinical data warehouse, we construct a decision tree for choosing the best second line therapy for treating type 2 diabetes patients.

Clinical Research Informatics for Big Data and Precision Medicine

OBJECTIVES

To reflect on the notable events and significant developments in Clinical Research Informatics (CRI) in the year of 2015 and discuss near-term trends impacting CRI.

METHODS

We selected key publications that highlight not only important recent advances in CRI but also notable events likely to have significant impact on CRI activities over the next few years or longer, and consulted the discussions in relevant scientific communities and an online living textbook for modern clinical trials. We also related the new concepts with old problems to improve the continuity of CRI research.

RESULTS

The highlights in CRI in 2015 include the growing adoption of electronic health records (EHR), the rapid development of regional, national, and global clinical data research networks for using EHR data to integrate scalable clinical research with clinical care and generate robust medical evidence. Data quality, integration, and fusion, data access by researchers, study transparency, results reproducibility, and infrastructure sustainability are persistent challenges.

CONCLUSION

The advances in Big Data Analytics and Internet technologies together with the engagement of citizens in sciences are shaping the global clinical research enterprise, which is getting more open and increasingly stakeholder-centered, where stakeholders include patients, clinicians, researchers, and sponsors.

Correlating eligibility criteria generalizability and adverse events using Big Data for patients and clinical trials

Randomized controlled trials can benefit from proactive assessment of how well their participant selection strategies during the design of eligibility criteria can influence the study generalizability. In this paper, we present a quantitative metric called generalizability index for study traits 2.0 (GIST 2.0) to assess the a priori generalizability (based on population representativeness) of a clinical trial by accounting for the dependencies among multiple eligibility criteria. The metric was evaluated on 16 sepsis trials identified from ClinicalTrials.gov, with their adverse event reports extracted from the trial results sections. The correlation between GIST scores and adverse events was analyzed. We found that the GIST 2.0 score was significantly correlated with total adverse events and serious adverse events (weighted correlation coefficients of 0.825 and 0.709, respectively, with P < 0.01). This study exemplifies the promising use of Big Data in electronic health records and ClinicalTrials.gov for optimizing eligibility criteria design for clinical studies.

GIST 2.0: A scalable multi-trait metric for quantifying population representativeness of individual clinical studies

The design of randomized controlled clinical studies can greatly benefit from iterative assessments of population representativeness of eligibility criteria. We propose a multi-trait metric - GIST 2.0 that can compute the a priori generalizability based on the population representativeness of a clinical study by explicitly modeling the dependencies among all eligibility criteria. We evaluate this metric on twenty clinical studies of two diseases and analyze how a study's eligibility criteria affect its generalizability (collectively and individually). We statistically analyze the effects of trial setting, trait selection and trait summarizing technique on GIST 2.0. Finally we provide theoretical as well as empirical validations for the expected properties of GIST 2.0.

Optimizing Clinical Research Participant Selection with Informatics

Clinical research participants are often not reflective of real-world patients due to overly restrictive eligibility criteria. Meanwhile, unselected participants introduce confounding factors and reduce research efficiency. Biomedical informatics, especially Big Data increasingly made available from electronic health records, offers promising aids to optimize research participant selection through data-driven transparency.

Prolonged dual antiplatelet therapy in stable coronary disease: comparative observational study of benefits and harms in unselected versus trial populations

OBJECTIVE

 To estimate the potential magnitude in unselected patients of the benefits and harms of prolonged dual antiplatelet therapy after acute myocardial infarction seen in selected patients with high risk characteristics in trials.

DESIGN

 Observational population based cohort study.

SETTING

 PEGASUS-TIMI-54 trial population and CALIBER (ClinicAl research using LInked Bespoke studies and Electronic health Records).

PARTICIPANTS

 7238 patients who survived a year or more after acute myocardial infarction.

INTERVENTIONS

 Prolonged dual antiplatelet therapy after acute myocardial infarction.

MAIN OUTCOME MEASURES

 Recurrent acute myocardial infarction, stroke, or fatal cardiovascular disease. Fatal, severe, or intracranial bleeding.

RESULTS

 1676/7238 (23.1%) patients met trial inclusion and exclusion criteria ("target" population). Compared with the placebo arm in the trial population, in the target population the median age was 12 years higher, there were more women (48.6% v 24.3%), and there was a substantially higher cumulative three year risk of both the primary (benefit) trial endpoint of recurrent acute myocardial infarction, stroke, or fatal cardiovascular disease (18.8% (95% confidence interval 16.3% to 21.8%) v 9.04%) and the primary (harm) endpoint of fatal, severe, or intracranial bleeding (3.0% (2.0% to 4.4%) v 1.26% (TIMI major bleeding)). Application of intention to treat relative risks from the trial (ticagrelor 60 mg daily arm) to CALIBER's target population showed an estimated 101 (95% confidence interval 87 to 117) ischaemic events prevented per 10 000 treated per year and an estimated 75 (50 to 110) excess fatal, severe, or intracranial bleeds caused per 10 000 patients treated per year. Generalisation from CALIBER's target subgroup to all 7238 real world patients who were stable at least one year after acute myocardial infarction showed similar three year risks of ischaemic events (17.2%, 16.0% to 18.5%), with an estimated 92 (86 to 99) events prevented per 10 000 patients treated per year, and similar three year risks of bleeding events (2.3%, 1.8% to 2.9%), with an estimated 58 (45 to 73) events caused per 10 000 patients treated per year.

CONCLUSIONS

 This novel use of primary-secondary care linked electronic health records allows characterisation of "healthy trial participant" effects and confirms the potential absolute benefits and harms of dual antiplatelet therapy in representative patients a year or more after acute myocardial infarction.

Visual assessment of the similarity between a patient and trial population: Is This Clinical Trial Applicable to My Patient?

BACKGROUND

A critical consideration when applying the results of a clinical trial to a particular patient is the degree of similarity of the patient to the trial population. However, similarity assessment rarely is practical in the clinical setting. Here, we explore means to support similarity assessment by clinicians.

METHODS

A scale chart was developed to represent the distribution of reported clinical and demographic characteristics of clinical trial participant populations. Constructed for an individual patient, the scale chart shows the patient's similarity to the study populations in a graphical manner. A pilot test case was conducted using case vignettes assessed by clinicians. Two pairs of clinical trials were used, each addressing a similar clinical question. Scale charts were manually constructed for each simulated patient. Clinicians were asked to estimate the degree of similarity of each patient to the populations of a pair of trials. Assessors relied on either the scale chart, a summary table (aligning characteristics of 2 trial populations), or original trial reports. Assessment time and between-assessor agreement were compared. Population characteristics considered important by assessors were recorded.

RESULTS

Six assessors evaluated 6 cases each. Using a visual scale chart, agreement between physicians was higher and the time required for similarity assessment was comparable.

CONCLUSION

We suggest that further research is warranted to explore visual tools facilitating the choice of the most applicable clinical trial to a specific patient. Automating patient and trial population characteristics extraction is key to support this effort.

Characterizing treatment pathways at scale using the OHDSI network

Observational research promises to complement experimental research by providing large, diverse populations that would be infeasible for an experiment. Observational research can test its own clinical hypotheses, and observational studies also can contribute to the design of experiments and inform the generalizability of experimental research. Understanding the diversity of populations and the variance in care is one component. In this study, the Observational Health Data Sciences and Informatics (OHDSI) collaboration created an international data network with 11 data sources from four countries, including electronic health records and administrative claims data on 250 million patients. All data were mapped to common data standards, patient privacy was maintained by using a distributed model, and results were aggregated centrally. Treatment pathways were elucidated for type 2 diabetes mellitus, hypertension, and depression. The pathways revealed that the world is moving toward more consistent therapy over time across diseases and across locations, but significant heterogeneity remains among sources, pointing to challenges in generalizing clinical trial results. Diabetes favored a single first-line medication, metformin, to a much greater extent than hypertension or depression. About 10% of diabetes and depression patients and almost 25% of hypertension patients followed a treatment pathway that was unique within the cohort. Aside from factors such as sample size and underlying population (academic medical center versus general population), electronic health records data and administrative claims data revealed similar results. Large-scale international observational research is feasible.

Multivariate analysis of the population representativeness of related clinical studies

OBJECTIVE

To develop a multivariate method for quantifying the population representativeness across related clinical studies and a computational method for identifying and characterizing underrepresented subgroups in clinical studies.

METHODS

We extended a published metric named Generalizability Index for Study Traits (GIST) to include multiple study traits for quantifying the population representativeness of a set of related studies by assuming the independence and equal importance among all study traits. On this basis, we compared the effectiveness of GIST and multivariate GIST (mGIST) qualitatively. We further developed an algorithm called "Multivariate Underrepresented Subgroup Identification" (MAGIC) for constructing optimal combinations of distinct value intervals of multiple traits to define underrepresented subgroups in a set of related studies. Using Type 2 diabetes mellitus (T2DM) as an example, we identified and extracted frequently used quantitative eligibility criteria variables in a set of clinical studies. We profiled the T2DM target population using the National Health and Nutrition Examination Survey (NHANES) data.

RESULTS

According to the mGIST scores for four example variables, i.e., age, HbA1c, BMI, and gender, the included observational T2DM studies had superior population representativeness than the interventional T2DM studies. For the interventional T2DM studies, Phase I trials had better population representativeness than Phase III trials. People at least 65years old with HbA1c value between 5.7% and 7.2% were particularly underrepresented in the included T2DM trials. These results confirmed well-known knowledge and demonstrated the effectiveness of our methods in population representativeness assessment.

CONCLUSIONS

mGIST is effective at quantifying population representativeness of related clinical studies using multiple numeric study traits. MAGIC identifies underrepresented subgroups in clinical studies. Both data-driven methods can be used to improve the transparency of design bias in participation selection at the research community level.

Valx: A System for Extracting and Structuring Numeric Lab Test Comparison Statements from Text

OBJECTIVES

To develop an automated method for extracting and structuring numeric lab test comparison statements from text and evaluate the method using clinical trial eligibility criteria text.

METHODS

Leveraging semantic knowledge from the Unified Medical Language System (UMLS) and domain knowledge acquired from the Internet, Valx takes seven steps to extract and normalize numeric lab test expressions: 1) text preprocessing, 2) numeric, unit, and comparison operator extraction, 3) variable identification using hybrid knowledge, 4) variable - numeric association, 5) context-based association filtering, 6) measurement unit normalization, and 7) heuristic rule-based comparison statements verification. Our reference standard was the consensus-based annotation among three raters for all comparison statements for two variables, i.e., HbA1c and glucose, identified from all of Type 1 and Type 2 diabetes trials in ClinicalTrials.gov.

RESULTS

The precision, recall, and F-measure for structuring HbA1c comparison statements were 99.6%, 98.1%, 98.8% for Type 1 diabetes trials, and 98.8%, 96.9%, 97.8% for Type 2 diabetes trials, respectively. The precision, recall, and F-measure for structuring glucose comparison statements were 97.3%, 94.8%, 96.1% for Type 1 diabetes trials, and 92.3%, 92.3%, 92.3% for Type 2 diabetes trials, respectively.

CONCLUSIONS

Valx is effective at extracting and structuring free-text lab test comparison statements in clinical trial summaries. Future studies are warranted to test its generalizability beyond eligibility criteria text. The open-source Valx enables its further evaluation and continued improvement among the collaborative scientific community.

Prediction of black box warning by mining patterns of Convergent Focus Shift in clinical trial study populations using linked public data

OBJECTIVE

To link public data resources for predicting post-marketing drug safety label changes by analyzing the Convergent Focus Shift patterns among drug testing trials.

METHODS

We identified 256 top-selling prescription drugs between 2003 and 2013 and divided them into 83 BBW drugs (drugs with at least one black box warning label) and 173 ROBUST drugs (drugs without any black box warning label) based on their FDA black box warning (BBW) records. We retrieved 7499 clinical trials that each had at least one of these drugs for intervention from the ClinicalTrials.gov. We stratified all the trials by pre-marketing or post-marketing status, study phase, and study start date. For each trial, we retrieved drug and disease concepts from clinical trial summaries to model its study population using medParser and SNOMED-CT. Convergent Focus Shift (CFS) pattern was calculated and used to assess the temporal changes in study populations from pre-marketing to post-marketing trials for each drug. Then we selected 68 candidate drugs, 18 with BBW warning and 50 without, that each had at least nine pre-marketing trials and nine post-marketing trials for predictive modeling. A random forest predictive model was developed to predict BBW acquisition incidents based on CFS patterns among these drugs. Pre- and post-marketing trials of BBW and ROBUST drugs were compared to look for their differences in CFS patterns.

RESULTS

Among the 18 BBW drugs, we consistently observed that the post-marketing trials focused more on recruiting patients with medical conditions previously unconsidered in the pre-marketing trials. In contrast, among the 50 ROBUST drugs, the post-marketing trials involved a variety of medications for testing their associations with target intervention(s). We found it feasible to predict BBW acquisitions using different CFS patterns between the two groups of drugs. Our random forest predictor achieved an AUC of 0.77. We also demonstrated the feasibility of the predictor for identifying long-term BBW acquisition events without compromising prediction accuracy.

CONCLUSIONS

This study contributes a method for post-marketing pharmacovigilance using Convergent Focus Shift (CFS) patterns in clinical trial study populations mined from linked public data resources. These signals are otherwise unavailable from individual data resources. We demonstrated the added value of linked public data and the feasibility of integrating ClinicalTrials.gov summaries and drug safety labels for post-marketing surveillance. Future research is needed to ensure better accessibility and linkage of heterogeneous drug safety data for efficient pharmacovigilance.

IDENTIFICATION OF QUESTIONABLE EXCLUSION CRITERIA IN MENTAL DISORDER CLINICAL TRIALS USING A MEDICAL ENCYCLOPEDIA

Precision medicine requires precise evidence-based practice and precise definition of the patients included in clinical studies for evidence generalization. Clinical research exclusion criteria define confounder patient characteristics for exclusion from a study. However, unnecessary exclusion criteria can weaken patient representativeness of study designs and generalizability of study results. This paper presents a method for identifying questionable exclusion criteria for 38 mental disorders. We extracted common eligibility features (CEFs) from all trials on these disorders from ClinicalTrials.gov. Network Analysis showed scale-free property of the CEF network, indicating uneven usage frequencies among CEFs. By comparing these CEFs' term frequencies in clinical trials' exclusion criteria and in the PubMed Medical Encyclopedia for matching conditions, we identified unjustified potential overuse of exclusion CEFs in mental disorder trials. Then we discussed the limitations in current exclusion criteria designs and made recommendations for achieving more patient-centered exclusion criteria definitions.

Improving Bridging from Informatics Practice to Theory

BACKGROUND

In 1962, Methods of Information in Medicine ( MIM ) began to publish papers on the methodology and scientific fundamentals of organizing, representing, and analyzing data, information, and knowledge in biomedicine and health care. Considered a companion journal, Applied Clinical Informatics ( ACI ) was launched in 2009 with a mission to establish a platform that allows sharing of knowledge between clinical medicine and health IT specialists as well as to bridge gaps between visionary design and successful and pragmatic deployment of clinical information systems. Both journals are official journals of the International Medical Informatics Association.

OBJECTIVES

As a follow-up to prior work, we set out to explore congruencies and interdependencies in publications of ACI and MIM. The objectives were to describe the major topics discussed in articles published in ACI in 2014 and to determine if there was evidence that theory in 2014 MIM publications was informed by practice described in ACI publications in any year. We also set out to describe lessons learned in the context of bridging informatics practice and theory and offer opinions on how ACI editorial policies could evolve to foster and improve such bridging.

METHODS

We conducted a retrospective observational study and reviewed all articles published in ACI during the calendar year 2014 (Volume 5) for their main theme, conclusions, and key words. We then reviewed the citations of all MIM papers from 2014 to determine if there were references to ACI articles from any year. Lessons learned in the context of bridging informatics practice and theory and opinions on ACI editorial policies were developed by consensus among the two authors.

RESULTS

A total of 70 articles were published in ACI in 2014. Clinical decision support, clinical documentation, usability, Meaningful Use, health information exchange, patient portals, and clinical research informatics emerged as major themes. Only one MIM article from 2014 cited an ACI article. There are several lessons learned including the possibility that there may not be direct links between MIM theory and ACI practice articles. ACI editorial policies will continue to evolve to reflect the breadth and depth of the practice of clinical informatics and articles received for publication. Efforts to encourage bridging of informatics practice and theory may be considered by the ACI editors.

CONCLUSIONS

The lack of direct links from informatics theory-based papers published in MIM in 2014 to papers published in ACI continues as was described for papers published during 2012 to 2013 in the two companion journals. Thus, there is little evidence that theory in MIM has been informed by practice in ACI.

Simulation-based Evaluation of the Generalizability Index for Study Traits

The Generalizability Index for Study Traits (GIST) has been proposed recently for assessing the population representativeness of a set of related clinical trials using eligibility features (e.g., age or BMI), one each time. However, GIST has not yet been evaluated. To bridge this knowledge gap, this paper reports a simulation-based validation study for GIST. Using the National Health and Nutrition Examination Survey (NHANES) data, we demonstrated the effectiveness of GIST at quantifying the population representativeness of a set of related trials that differ in disease domains, study phases, sponsor types, and study designs, respectively. We also showed that among seven example medical conditions, the GIST of age increases from Phase I trials to Phase III trials in the seven disease domains and is the lowest in asthma trials. We concluded that GIST correlates with simulation-based generalizability results and is a valid metric for quantifying population representativeness of related clinical trials.

Patient-reported outcomes in a large community-based pain medicine practice: evaluation for use in phenotype modeling

Background

An academic, community medicine partnership was established to build a phenotype-to-outcome model targeting chronic pain. This model will be used to drive clinical decision support for pain medicine in the community setting. The first step in this effort is an examination of the electronic health records (EHR) from clinics that treat chronic pain. The biopsychosocial components provided by both patients and care providers must be of sufficient scope to populate the spectrum of patient types, treatment modalities, and possible outcomes.

Methods

The patient health records from a large Midwest pain medicine practice (Michigan Pain Consultants, PC) contains physician notes, administrative codes, and patient-reported outcomes (PRO) on over 30,000 patients during the study period spanning 2010 to mid-2014. The PRO consists of a regularly administered Pain Health Assessment (PHA), a biopsychosocial, demographic, and symptomology questionnaire containing 163 items, which is completed approximately every six months with a compliance rate of over 95 %. The biopsychosocial items (74 items with Likert scales of 0–10) were examined by exploratory factor analysis and descriptive statistics to determine the number of independent constructs available for phenotypes and outcomes. Pain outcomes were examined both in the aggregate and the mean of longitudinal changes in each patient.

Results

Exploratory factor analysis of the intake PHA revealed 15 orthogonal factors representing pain levels; physical, social, and emotional functions; the effects of pain on these functions; vitality and health; and measures of outcomes and satisfaction. Seven items were independent of the factors, offering unique information. As an exemplar of outcomes from the follow-up PHAs, patients reported approximately 60 % relief in their pain. When examined in the aggregate, patients showed both a decrease in pain levels and an increase in coping skills with an increased number of visits. When examined individually, 80-85 % of patients presenting with the highest pain levels reported improvement by approximately two points on an 11-point pain scale.

Conclusions

We conclude that the data available in a community practice can be a rich source of biopsychosocial information relevant to the phenotypes of chronic pain. It is anticipated that phenotype linkages to best treatments and outcomes can be constructed from this set of records.

Research Data Explorer: Lessons Learned in Design and Development of Context-based Cohort Definition and Selection

Research Data eXplorer (RedX) was designed to support self-service research data queries and cohort identification from clinical research databases. The primary innovation of RedX was the electronic health record view of patient data, to provide better contextual understanding for non-technical users in building complex data queries. The design of RedX around this need identified multiple functions that would use individual patient views to better understand population-based data, and vice-versa. During development, the more necessary and valuable components of RedX were refined, leading to a functional self-service query and cohort identification tool. However, with the improved capabilities and extensibility of other applications for data querying and navigation, our long-term implementation and dissemination plans have moved towards consolidation and alignment of RedX functions as enhancements in these other initiatives.

Visual aggregate analysis of eligibility features of clinical trials

OBJECTIVE

To develop a method for profiling the collective populations targeted for recruitment by multiple clinical studies addressing the same medical condition using one eligibility feature each time.

METHODS

Using a previously published database COMPACT as the backend, we designed a scalable method for visual aggregate analysis of clinical trial eligibility features. This method consists of four modules for eligibility feature frequency analysis, query builder, distribution analysis, and visualization, respectively. This method is capable of analyzing (1) frequently used qualitative and quantitative features for recruiting subjects for a selected medical condition, (2) distribution of study enrollment on consecutive value points or value intervals of each quantitative feature, and (3) distribution of studies on the boundary values, permissible value ranges, and value range widths of each feature. All analysis results were visualized using Google Charts API. Five recruited potential users assessed the usefulness of this method for identifying common patterns in any selected eligibility feature for clinical trial participant selection.

RESULTS

We implemented this method as a Web-based analytical system called VITTA (Visual Analysis Tool of Clinical Study Target Populations). We illustrated the functionality of VITTA using two sample queries involving quantitative features BMI and HbA1c for conditions "hypertension" and "Type 2 diabetes", respectively. The recruited potential users rated the user-perceived usefulness of VITTA with an average score of 86.4/100.

CONCLUSIONS

We contributed a novel aggregate analysis method to enable the interrogation of common patterns in quantitative eligibility criteria and the collective target populations of multiple related clinical studies. A larger-scale study is warranted to formally assess the usefulness of VITTA among clinical investigators and sponsors in various therapeutic areas.

Assessing the Collective Population Representativeness of Related Type 2 Diabetes Trials by Combining Public Data from ClinicalTrials.gov and NHANES

Randomized controlled trials generate high-quality medical evidence. However, the use of unjustified inclusion/exclusion criteria may compromise the external validity of a study. We have introduced a method to assess the population representativeness of related clinical trials using electronic health record (EHR) data. As EHR data may not perfectly represent the real-world patient population, in this work, we further validated the method and its results using the National Health and Nutrition Examination Survey (NHANES) data. We visualized and quantified the differences in the distributions of age, HbA1c, and BMI among the target population of Type 2 diabetes trials, diabetics in NHANES databases, and a convenience sample of patients enrolled in selected Type 2 diabetes trials. The results are consistent with the previous study.

A method for analyzing commonalities in clinical trial target populations

ClinicalTrials.gov presents great opportunities for analyzing commonalities in clinical trial target populations to facilitate knowledge reuse when designing eligibility criteria of future trials or to reveal potential systematic biases in selecting population subgroups for clinical research. Towards this goal, this paper presents a novel data resource for enabling such analyses. Our method includes two parts: (1) parsing and indexing eligibility criteria text; and (2) mining common eligibility features and attributes of common numeric features (e.g., A1c). We designed and built a database called "Commonalities in Target Populations of Clinical Trials" (COMPACT), which stores structured eligibility criteria and trial metadata in a readily computable format. We illustrate its use in an example analytic module called CONECT using COMPACT as the backend. Type 2 diabetes is used as an example to analyze commonalities in the target populations of 4,493 clinical trials on this disease.