Applicability Area: A novel utility-based approach for evaluating predictive models, beyond discrimination

Translating prediction models into practice and supporting clinicians' decision-making demand demonstration of clinical value. Existing approaches to evaluating machine learning models emphasize discriminatory power, which is only a part of the medical decision problem. We propose the Applicability Area (ApAr), a decision-analytic utility-based approach to evaluating predictive models that communicate the range of prior probability and test cutoffs for which the model has positive utility; larger ApArs suggest a broader potential use of the model. We assess ApAr with simulated datasets and with three published medical datasets. ApAr adds value beyond the typical area under the receiver operating characteristic curve (AUROC) metric analysis. As an example, in the diabetes dataset, the top model by ApAr was ranked as the 23rd best model by AUROC. Decision makers looking to adopt and implement models can leverage ApArs to assess if the local range of priors and utilities is within the respective ApArs.

Autonomous artificial intelligence increases screening and follow-up for diabetic retinopathy in youth: the ACCESS randomized control trial

Diabetic retinopathy can be prevented with screening and early detection. We hypothesized that autonomous artificial intelligence (AI) diabetic eye exams at the point-of-care would increase diabetic eye exam completion rates in a racially and ethnically diverse youth population. AI for Children's diabetiC Eye ExamS (NCT05131451) is a parallel randomized controlled trial that randomized youth (ages 8-21 years) with type 1 and type 2 diabetes to intervention (autonomous artificial intelligence diabetic eye exam at the point of care), or control (scripted eye care provider referral and education) in an academic pediatric diabetes center. The primary outcome was diabetic eye exam completion rate within 6 months. The secondary outcome was the proportion of participants who completed follow-through with an eye care provider if deemed appropriate. Diabetic eye exam completion rate was significantly higher (100%, 95%CI: 95.5%, 100%) in the intervention group (n = 81) than the control group (n = 83) (22%, 95%CI: 14.2%, 32.4%)(p < 0.001). In the intervention arm, 25/81 participants had an abnormal result, of whom 64% (16/25) completed follow-through with an eye care provider, compared to 22% in the control arm (p < 0.001). Autonomous AI increases diabetic eye exam completion rates in youth with diabetes.

Advancing Toward a Common Data Model in Ophthalmology: Gap Analysis of General Eye Examination Concepts to Standard Observational Medical Outcomes Partnership (OMOP) Concepts

Purpose

Evaluate the degree of concept coverage of the general eye examination in one widely used electronic health record (EHR) system using the Observational Health Data Sciences and Informatics Observational Medical Outcomes Partnership (OMOP) common data model (CDM).

Design

Study of data elements.

Participants

Not applicable.

Methods

Data elements (field names and predefined entry values) from the general eye examination in the Epic foundation system were mapped to OMOP concepts and analyzed. Each mapping was given a Health Level 7 equivalence designation-equal when the OMOP concept had the same meaning as the source EHR concept, wider when it was missing information, narrower when it was overly specific, and unmatched when there was no match. Initial mappings were reviewed by 2 graders. Intergrader agreement for equivalence designation was calculated using Cohen's kappa. Agreement on the mapped OMOP concept was calculated as a percentage of total mappable concepts. Discrepancies were discussed and a final consensus created. Quantitative analysis was performed on wider and unmatched concepts.

Main Outcome Measures

Gaps in OMOP concept coverage of EHR elements and intergrader agreement of mapped OMOP concepts.

Results

A total of 698 data elements (210 fields, 488 values) from the EHR were analyzed. The intergrader kappa on the equivalence designation was 0.88 (standard error 0.03, P < 0.001). There was a 96% agreement on the mapped OMOP concept. In the final consensus mapping, 25% (1% fields, 31% values) of the EHR to OMOP concept mappings were considered equal, 50% (27% fields, 60% values) wider, 4% (8% fields, 2% values) narrower, and 21% (52% fields, 8% values) unmatched. Of the wider mapped elements, 46% were missing the laterality specification, 24% had other missing attributes, and 30% had both issues. Wider and unmatched EHR elements could be found in all areas of the general eye examination.

Conclusions

Most data elements in the general eye examination could not be represented precisely using the OMOP CDM. Our work suggests multiple ways to improve the incorporation of important ophthalmology concepts in OMOP, including adding laterality to existing concepts. There exists a strong need to improve the coverage of ophthalmic concepts in source vocabularies so that the OMOP CDM can better accommodate vision research.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Associations between sleep health and obesity and weight change in adults: The Daily24 Multisite Cohort Study

OBJECTIVES

To examine cross-sectional and longitudinal associations of individual sleep domains and multidimensional sleep health with current overweight or obesity and 5-year weight change in adults.

METHODS

We estimated sleep regularity, quality, timing, onset latency, sleep interruptions, duration, and napping using validated questionnaires. We calculated multidimensional sleep health using a composite score (total number of "good" sleep health indicators) and sleep phenotypes derived from latent class analysis. Logistic regression was used to examine associations between sleep and overweight or obesity. Multinomial regression was used to examine associations between sleep and weight change (gain, loss, or maintenance) over a median of 1.66 years.

RESULTS

The sample included 1016 participants with a median age of 52 (IQR = 37-65), who primarily identified as female (78%), White (79%), and college-educated (74%). We identified 3 phenotypes: good, moderate, and poor sleep. More regularity of sleep, sleep quality, and shorter sleep onset latency were associated with 37%, 38%, and 45% lower odds of overweight or obesity, respectively. The addition of each good sleep health dimension was associated with 16% lower adjusted odds of having overweight or obesity. The adjusted odds of overweight or obesity were similar between sleep phenotypes. Sleep, individual or multidimensional sleep health, was not associated with weight change.

CONCLUSIONS

Multidimensional sleep health showed cross-sectional, but not longitudinal, associations with overweight or obesity. Future research should advance our understanding of how to assess multidimensional sleep health to understand the relationship between all aspects of sleep health and weight over time.

Strengths, weaknesses, opportunities, and threats for the nation's public health information systems infrastructure: synthesis of discussions from the 2022 ACMI Symposium

OBJECTIVE

The annual American College of Medical Informatics (ACMI) symposium focused discussion on the national public health information systems (PHIS) infrastructure to support public health goals. The objective of this article is to present the strengths, weaknesses, threats, and opportunities (SWOT) identified by public health and informatics leaders in attendance.

MATERIALS AND METHODS

The Symposium provided a venue for experts in biomedical informatics and public health to brainstorm, identify, and discuss top PHIS challenges. Two conceptual frameworks, SWOT and the Informatics Stack, guided discussion and were used to organize factors and themes identified through a qualitative approach.

RESULTS

A total of 57 unique factors related to the current PHIS were identified, including 9 strengths, 22 weaknesses, 14 opportunities, and 14 threats, which were consolidated into 22 themes according to the Stack. Most themes (68%) clustered at the top of the Stack. Three overarching opportunities were especially prominent: (1) addressing the needs for sustainable funding, (2) leveraging existing infrastructure and processes for information exchange and system development that meets public health goals, and (3) preparing the public health workforce to benefit from available resources.

DISCUSSION

The PHIS is unarguably overdue for a strategically designed, technology-enabled, information infrastructure for delivering day-to-day essential public health services and to respond effectively to public health emergencies.

CONCLUSION

Most of the themes identified concerned context, people, and processes rather than technical elements. We recommend that public health leadership consider the possible actions and leverage informatics expertise as we collectively prepare for the future.

Effectiveness of artificial intelligence screening in preventing vision loss from diabetes: a policy model

The effectiveness of using artificial intelligence (AI) systems to perform diabetic retinal exams ('screening') on preventing vision loss is not known. We designed the Care Process for Preventing Vision Loss from Diabetes (CAREVL), as a Markov model to compare the effectiveness of point-of-care autonomous AI-based screening with in-office clinical exam by an eye care provider (ECP), on preventing vision loss among patients with diabetes. The estimated incidence of vision loss at 5 years was 1535 per 100,000 in the AI-screened group compared to 1625 per 100,000 in the ECP group, leading to a modelled risk difference of 90 per 100,000. The base-case CAREVL model estimated that an autonomous AI-based screening strategy would result in 27,000 fewer Americans with vision loss at 5 years compared with ECP. Vision loss at 5 years remained lower in the AI-screened group compared to the ECP group, in a wide range of parameters including optimistic estimates biased toward ECP. Real-world modifiable factors associated with processes of care could further increase its effectiveness. Of these factors, increased adherence with treatment was estimated to have the greatest impact.

Human-Centered Design of a Clinical Decision Support for Anemia Screening in Children with Inflammatory Bowel Disease

BACKGROUND

Inflammatory bowel disease (IBD) commonly leads to iron deficiency anemia (IDA). Rates of screening and treatment of IDA are often low. A clinical decision support system (CDSS) embedded in an electronic health record could improve adherence to evidence-based care. Rates of CDSS adoption are often low due to poor usability and fit with work processes. One solution is to use human-centered design (HCD), which designs CDSS based on identified user needs and context of use and evaluates prototypes for usefulness and usability.

OBJECTIVES

this study aimed to use HCD to design a CDSS tool called the IBD Anemia Diagnosis Tool, IADx.

METHODS

Interviews with IBD practitioners informed creation of a process map of anemia care that was used by an interdisciplinary team that used HCD principles to create a prototype CDSS. The prototype was iteratively tested with "Think Aloud" usability evaluation with clinicians as well as semi-structured interviews, a survey, and observations. Feedback was coded and informed redesign.

RESULTS

Process mapping showed that IADx should function at in-person encounters and asynchronous laboratory review. Clinicians desired full automation of clinical information acquisition such as laboratory trends and analysis such as calculation of iron deficit, less automation of clinical decision selection such as laboratory ordering, and no automation of action implementation such as signing medication orders. Providers preferred an interruptive alert over a noninterruptive reminder.

CONCLUSION

Providers preferred an interruptive alert, perhaps due to the low likelihood of noticing a noninterruptive advisory. High levels of desire for automation of information acquisition and analysis with less automation of decision selection and action may be generalizable to other CDSSs designed for chronic disease management. This underlines the ways in which CDSSs have the potential to augment rather than replace provider cognitive work.

Healthcare utilization is a collider: an introduction to collider bias in EHR data reuse

OBJECTIVES

Collider bias is a common threat to internal validity in clinical research but is rarely mentioned in informatics education or literature. Conditioning on a collider, which is a variable that is the shared causal descendant of an exposure and outcome, may result in spurious associations between the exposure and outcome. Our objective is to introduce readers to collider bias and its corollaries in the retrospective analysis of electronic health record (EHR) data.

TARGET AUDIENCE

Collider bias is likely to arise in the reuse of EHR data, due to data-generating mechanisms and the nature of healthcare access and utilization in the United States. Therefore, this tutorial is aimed at informaticians and other EHR data consumers without a background in epidemiological methods or causal inference.

SCOPE

We focus specifically on problems that may arise from conditioning on forms of healthcare utilization, a common collider that is an implicit selection criterion when one reuses EHR data. Directed acyclic graphs (DAGs) are introduced as a tool for identifying potential sources of bias during study design and planning. References for additional resources on causal inference and DAG construction are provided.

Electronic Health Record-Based Recruitment and Retention and Mobile Health App Usage: Multisite Cohort Study

BACKGROUND

To address the obesity epidemic, there is a need for novel paradigms, including those that address the timing of eating and sleep in relation to circadian rhythms. Electronic health records (EHRs) are an efficient way to identify potentially eligible participants for health research studies. Mobile health (mHealth) apps offer available and convenient data collection of health behaviors, such as timing of eating and sleep.

OBJECTIVE

The aim of this descriptive analysis was to report on recruitment, retention, and app use from a 6-month cohort study using a mobile app called Daily24.

METHODS

Using an EHR query, adult patients from three health care systems in the PaTH clinical research network were identified as potentially eligible, invited electronically to participate, and instructed to download and use the Daily24 mobile app, which focuses on eating and sleep timing. Online surveys were completed at baseline and 4 months. We described app use and identified predictors of app use, defined as 1 or more days of use, versus nonuse and usage categories (ie, immediate, consistent, and sustained) using multivariate regression analyses.

RESULTS

Of 70,661 patients who were sent research invitations, 1021 (1.44%) completed electronic consent forms and online baseline surveys; 4 withdrew, leaving a total of 1017 participants in the analytic sample. A total of 53.79% (n=547) of the participants were app users and, of those, 75.3% (n=412), 50.1% (n=274), and 25.4% (n=139) were immediate, consistent, and sustained users, respectively. Median app use was 28 (IQR 7-75) days over 6 months. Younger age, White race, higher educational level, higher income, having no children younger than 18 years, and having used 1 to 5 health apps significantly predicted app use (vs nonuse) in adjusted models. Older age and lower BMI predicted early, consistent, and sustained use. About half (532/1017, 52.31%) of the participants completed the 4-month online surveys. A total of 33.5% (183/547), 29.3% (157/536), and 27.1% (143/527) of app users were still using the app for at least 2 days per month during months 4, 5, and 6 of the study, respectively.

CONCLUSIONS

EHR recruitment offers an efficient (ie, high reach, low touch, and minimal participant burden) approach to recruiting participants from health care settings into mHealth research. Efforts to recruit and retain less engaged subgroups are needed to collect more generalizable data. Additionally, future app iterations should include more evidence-based features to increase participant use.

Harmonizing units and values of quantitative data elements in a very large nationally pooled electronic health record (EHR) dataset

Objective

The goals of this study were to harmonize data from electronic health records (EHRs) into common units, and impute units that were missing.

Materials and Methods

The National COVID Cohort Collaborative (N3C) table of laboratory measurement data—over 3.1 billion patient records and over 19 000 unique measurement concepts in the Observational Medical Outcomes Partnership (OMOP) common-data-model format from 55 data partners. We grouped ontologically similar OMOP concepts together for 52 variables relevant to COVID-19 research, and developed a unit-harmonization pipeline comprised of (1) selecting a canonical unit for each measurement variable, (2) arriving at a formula for conversion, (3) obtaining clinical review of each formula, (4) applying the formula to convert data values in each unit into the target canonical unit, and (5) removing any harmonized value that fell outside of accepted value ranges for the variable. For data with missing units for all the results within a lab test for a data partner, we compared values with pooled values of all data partners, using the Kolmogorov-Smirnov test.

Results

Of the concepts without missing values, we harmonized 88.1% of the values, and imputed units for 78.2% of records where units were absent (41% of contributors’ records lacked units).

Discussion

The harmonization and inference methods developed herein can serve as a resource for initiatives aiming to extract insight from heterogeneous EHR collections. Unique properties of centralized data are harnessed to enable unit inference.

Conclusion

The pipeline we developed for the pooled N3C data enables use of measurements that would otherwise be unavailable for analysis.

Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative

OBJECTIVE

In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations.

MATERIALS AND METHODS

We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements.

RESULTS

Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback.

DISCUSSION

We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate.

CONCLUSION

By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

Applying Decision Science to the Prioritization of Healthcare-Associated Infection Initiatives

OBJECTIVES

Improving patient quality remains a top priority from the perspectives of both patient outcomes and cost of care. The continuing threat to patient safety has resulted in an increasing number of options for patient safety initiatives, making choices more difficult because of competing priorities. This study provides a proof of concept for using low-cost decision science methods for prioritizing initiatives.

METHODS

Using multicriteria decision analysis, we developed a decision support model for aiding the prioritization of the four most common types of healthcare-associated infections: surgical site infections, central line-associated bloodstream infections, ventilator-associated events, and catheter-associated urinary tract infections. In semistructured interviews, we elicited structure and parameter values of a candidate model, which was then validated by six participants with different roles across three urban teaching and nonteaching hospitals in the Baltimore, Maryland area.

RESULTS

Participants articulated the following structural attributes of concern: patient harm, monetary costs, patient mortality, reputational effects, and patient satisfaction. A quantitative decision-making model with an associated uncertainty report for prioritizing initiatives related to the four most common types of healthcare-associated infections was then created.

CONCLUSIONS

A decision support methodology such as our proof of concept could aid hospital executives in prioritizing the quality improvement initiatives within their hospital, with more complete data. Because hospitals continue to struggle in improving quality of care with tighter budgets, a formal decision support mechanism could be used to objectively prioritize patient safety and quality initiatives.

Clinical Characterization and Prediction of Clinical Severity of SARS-CoV-2 Infection Among US Adults Using Data From the US National COVID Cohort Collaborative

Importance

The National COVID Cohort Collaborative (N3C) is a centralized, harmonized, high-granularity electronic health record repository that is the largest, most representative COVID-19 cohort to date. This multicenter data set can support robust evidence-based development of predictive and diagnostic tools and inform clinical care and policy.

Objectives

To evaluate COVID-19 severity and risk factors over time and assess the use of machine learning to predict clinical severity.

Design, Setting, and Participants

In a retrospective cohort study of 1 926 526 US adults with SARS-CoV-2 infection (polymerase chain reaction >99% or antigen <1%) and adult patients without SARS-CoV-2 infection who served as controls from 34 medical centers nationwide between January 1, 2020, and December 7, 2020, patients were stratified using a World Health Organization COVID-19 severity scale and demographic characteristics. Differences between groups over time were evaluated using multivariable logistic regression. Random forest and XGBoost models were used to predict severe clinical course (death, discharge to hospice, invasive ventilatory support, or extracorporeal membrane oxygenation).

Main Outcomes and Measures

Patient demographic characteristics and COVID-19 severity using the World Health Organization COVID-19 severity scale and differences between groups over time using multivariable logistic regression.

Results

The cohort included 174 568 adults who tested positive for SARS-CoV-2 (mean [SD] age, 44.4 [18.6] years; 53.2% female) and 1 133 848 adult controls who tested negative for SARS-CoV-2 (mean [SD] age, 49.5 [19.2] years; 57.1% female). Of the 174 568 adults with SARS-CoV-2, 32 472 (18.6%) were hospitalized, and 6565 (20.2%) of those had a severe clinical course (invasive ventilatory support, extracorporeal membrane oxygenation, death, or discharge to hospice). Of the hospitalized patients, mortality was 11.6% overall and decreased from 16.4% in March to April 2020 to 8.6% in September to October 2020 (P = .002 for monthly trend). Using 64 inputs available on the first hospital day, this study predicted a severe clinical course using random forest and XGBoost models (area under the receiver operating curve = 0.87 for both) that were stable over time. The factor most strongly associated with clinical severity was pH; this result was consistent across machine learning methods. In a separate multivariable logistic regression model built for inference, age (odds ratio [OR], 1.03 per year; 95% CI, 1.03-1.04), male sex (OR, 1.60; 95% CI, 1.51-1.69), liver disease (OR, 1.20; 95% CI, 1.08-1.34), dementia (OR, 1.26; 95% CI, 1.13-1.41), African American (OR, 1.12; 95% CI, 1.05-1.20) and Asian (OR, 1.33; 95% CI, 1.12-1.57) race, and obesity (OR, 1.36; 95% CI, 1.27-1.46) were independently associated with higher clinical severity.

Conclusions and Relevance

This cohort study found that COVID-19 mortality decreased over time during 2020 and that patient demographic characteristics and comorbidities were associated with higher clinical severity. The machine learning models accurately predicted ultimate clinical severity using commonly collected clinical data from the first 24 hours of a hospital admission.

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment

Objective

Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers.

Materials and Methods

The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics.

Results

Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access.

Conclusions

The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

Cost-effectiveness of Autonomous Point-of-Care Diabetic Retinopathy Screening for Pediatric Patients With Diabetes

Importance

Screening for diabetic retinopathy is recommended for children with type 1 diabetes (T1D) and type 2 diabetes (T2D), yet screening rates remain low. Point-of-care diabetic retinopathy screening using autonomous artificial intelligence (AI) has become available, providing immediate results in the clinic setting, but the cost-effectiveness of this strategy compared with standard examination is unknown.

Objective

To assess the cost-effectiveness of detecting and treating diabetic retinopathy and its sequelae among children with T1D and T2D using AI diabetic retinopathy screening vs standard screening by an eye care professional (ECP).

Design, Setting, and Participants

In this economic evaluation, parameter estimates were obtained from the literature from 1994 to 2019 and assessed from March 2019 to January 2020. Parameters included out-of-pocket cost for autonomous AI screening, ophthalmology visits, and treating diabetic retinopathy; probability of undergoing standard retinal examination; relative odds of undergoing screening; and sensitivity, specificity, and diagnosability of the ECP screening examination and autonomous AI screening.

Main Outcomes and Measures

Costs or savings to the patient based on mean patient payment for diabetic retinopathy screening examination and cost-effectiveness based on costs or savings associated with the number of true-positive results identified by diabetic retinopathy screening.

Results

In this study, the expected true-positive proportions for standard ophthalmologic screening by an ECP were 0.006 for T1D and 0.01 for T2D, and the expected true-positive proportions for autonomous AI were 0.03 for T1D and 0.04 for T2D. The base case scenario of 20% adherence estimated that use of autonomous AI would result in a higher mean patient payment ($8.52 for T1D and $10.85 for T2D) than conventional ECP screening ($7.91 for T1D and $8.20 for T2D). However, autonomous AI screening was the preferred strategy when at least 23% of patients adhered to diabetic retinopathy screening.

Conclusions and Relevance

These results suggest that point-of-care diabetic retinopathy screening using autonomous AI systems is effective and cost saving for children with diabetes and their caregivers at recommended adherence rates.

The National COVID Cohort Collaborative: Clinical Characterization and Early Severity Prediction

Background:

The majority of U.S. reports of COVID-19 clinical characteristics, disease course, and treatments are from single health systems or focused on one domain. Here we report the creation of the National COVID Cohort Collaborative (N3C), a centralized, harmonized, high-granularity electronic health record repository that is the largest, most representative U.S. cohort of COVID-19 cases and controls to date. This multi-center dataset supports robust evidence-based development of predictive and diagnostic tools and informs critical care and policy.

Methods and Findings:

In a retrospective cohort study of 1,926,526 patients from 34 medical centers nationwide, we stratified patients using a World Health Organization COVID-19 severity scale and demographics; we then evaluated differences between groups over time using multivariable logistic regression. We established vital signs and laboratory values among COVID-19 patients with different severities, providing the foundation for predictive analytics. The cohort included 174,568 adults with severe acute respiratory syndrome associated with SARS-CoV-2 (PCR >99% or antigen <1%) as well as 1,133,848 adult patients that served as lab-negative controls. Among 32,472 hospitalized patients, mortality was 11.6% overall and decreased from 16.4% in March/April 2020 to 8.6% in September/October 2020 (p = 0.002 monthly trend). In a multivariable logistic regression model, age, male sex, liver disease, dementia, African-American and Asian race, and obesity were independently associated with higher clinical severity. To demonstrate the utility of the N3C cohort for analytics, we used machine learning (ML) to predict clinical severity and risk factors over time. Using 64 inputs available on the first hospital day, we predicted a severe clinical course (death, discharge to hospice, invasive ventilation, or extracorporeal membrane oxygenation) using random forest and XGBoost models (AUROC 0.86 and 0.87 respectively) that were stable over time. The most powerful predictors in these models are patient age and widely available vital sign and laboratory values. The established expected trajectories for many vital signs and laboratory values among patients with different clinical severities validates observations from smaller studies, and provides comprehensive insight into COVID-19 characterization in U.S. patients.

Conclusions:

This is the first description of an ongoing longitudinal observational study of patients seen in diverse clinical settings and geographical regions and is the largest COVID-19 cohort in the United States. Such data are the foundation for ML models that can be the basis for generalizable clinical decision support tools. The N3C Data Enclave is unique in providing transparent, reproducible, easily shared, versioned, and fully auditable data and analytic provenance for national-scale patient-level EHR data. The N3C is built for intensive ML analyses by academic, industry, and citizen scientists internationally. Many observational correlations can inform trial designs and care guidelines for this new disease.

Understanding Primary Care Patients' Self-weighing Habits: Cohort Analysis from the PaTH Clinical Data Research Network

BACKGROUND

Greater than 60% of adults have overweight or obesity. Self-weighing is an effective weight loss and weight maintenance tool. However, little is known about self-weighing habits among the primary care patient population. Our objective was to examine the frequency of patient-reported self-weighing, and to evaluate the associations of self-weighing with demographic characteristics and self-monitoring behaviors.

METHODS

We conducted an analysis of survey data collected as part of the PaTH Clinical Data Research Network, which recruited a cohort of 1,021 primary care patients at 4 academic medical centers. Patients of all body mass index (BMI) categories were included.

RESULTS

Response rate of 6-month survey was 727 (71%). The mean age was 56 years, and most were female (68%), White (78%), college graduates (66%), and employed/retired (85%). The mean BMI was 30.2 kg/m², 80% of participants had a BMI ≧ 25 kg/m². Of patients with BMI ≧ 25 kg/m², 35% of participants self-weighed weekly and 23% daily. Participants who reported self-weighing at least weekly were more likely to be older (59 vs 54 years, p < 0.01), married (p = 0.01), college graduates (p = 0.03), White (p < 0.01), and employed vs disabled/unemployed (p < 0.01). Patients who self-weighed daily had a lower BMI (29 kg/m² vs 31 kg/m², p = 0.04). Patients who tracked exercise or food intake were more likely to self-weigh daily (p < 0.01), as were patients wanting to lose or maintain weight (p < 0.01).

CONCLUSIONS

Despite its potential for primary and secondary obesity prevention, only 35% of primary care patients with overweight or obesity engage in self-weighing weekly and less than a quarter (23%) self-weigh daily. Socioeconomic status appears to be a factor influencing regular self-weighing in this population, potentially contributing to greater health disparities in obesity rates. Patients who self-weighed daily had a lower BMI, suggesting that it may play a role in primary prevention of obesity. More work is needed to explore self-weighing among patients.

Evaluation of multidisciplinary collaboration in pediatric trauma care using EHR data

OBJECTIVES

The study sought to identify collaborative electronic health record (EHR) usage patterns for pediatric trauma patients and determine how the usage patterns are related to patient outcomes.

MATERIALS AND METHODS

A process mining-based network analysis was applied to EHR metadata and trauma registry data for a cohort of pediatric trauma patients with minor injuries at a Level I pediatric trauma center. The EHR metadata were processed into an event log that was segmented based on gaps in the temporal continuity of events. A usage pattern was constructed for each encounter by creating edges among functional roles that were captured within the same event log segment. These patterns were classified into groups using graph kernel and unsupervised spectral clustering methods. Demographics, clinical and network characteristics, and emergency department (ED) length of stay (LOS) of the groups were compared.

RESULTS

Three distinct usage patterns that differed by network density were discovered: fully connected (clique), partially connected, and disconnected (isolated). Compared with the fully connected pattern, encounters with the partially connected pattern had an adjusted median ED LOS that was significantly longer (242.6 [95% confidence interval, 236.9-246.0] minutes vs 295.2 [95% confidence, 289.2-297.8] minutes), more frequently seen among day shift and weekday arrivals, and involved otolaryngology, ophthalmology services, and child life specialists.

DISCUSSION

The clique-like usage pattern was associated with decreased ED LOS for the study cohort, suggesting greater degree of collaboration resulted in shorter stay.

CONCLUSIONS

Further investigation to understand and address causal factors can lead to improvement in multidisciplinary collaboration.

Linking Electronic Health Record and Trauma Registry Data: Assessing the Value of Probabilistic Linkage

BACKGROUND

Electronic health record (EHR) systems contain large volumes of novel heterogeneous data that can be linked to trauma registry data to enable innovative research not possible with either data source alone.

OBJECTIVE

This article describes an approach for linking electronically extracted EHR data to trauma registry data at the institutional level and assesses the value of probabilistic linkage.

METHODS

Encounter data were independently obtained from the EHR data warehouse (n = 1,632) and the pediatric trauma registry (n = 1,829) at a Level I pediatric trauma center. Deterministic linkage was attempted using nine different combinations of medical record number (MRN), encounter identity (ID) (visit ID), age, gender, and emergency department (ED) arrival date. True matches from the best performing variable combination were used to create a gold standard, which was used to evaluate the performance of each variable combination, and to train a probabilistic algorithm that was separately used to link records unmatched by deterministic linkage and the entire cohort. Additional records that matched probabilistically were investigated via chart review and compared against records that matched deterministically.

RESULTS

Deterministic linkage with exact matching on any three of MRN, encounter ID, age, gender, and ED arrival date gave the best yield of 1,276 true matches while an additional probabilistic linkage step following deterministic linkage yielded 110 true matches. These records contained a significantly higher number of boys compared to records that matched deterministically and etiology was attributable to mismatch between MRNs in the two data sets. Probabilistic linkage of the entire cohort yielded 1,363 true matches.

CONCLUSION

The combination of deterministic and an additional probabilistic method represents a robust approach for linking EHR data to trauma registry data. This approach may be generalizable to studies involving other registries and databases.

Interactive Cost-benefit Analysis: Providing Real-World Financial Context to Predictive Analytics

Objective: Clinical implementation of predictive analytics that assess risk of high-cost outcomes are presumed to save money because they help focus interventions designed to avert those outcomes on a subset patients who are most likely to benefit from the intervention. This premise may not always be true. A cost-benefit analysis is necessary to show if a strategy of applying the predictive algorithm is truly favorable to alternative strategies. Methods: We designed and implemented an interactive web-based cost-benefit calculator, enabling specification of accuracy parameters for the predictive model and other clinical and financial factors related to the occurrence of an undesirable outcome. We use the web tool, populated with real-world data to illustrate a cost-benefit analysis of a strategy of applying predictive analytics to select a cohort of high-risk patients to receive interventions to avert readmissions for Congestive Heart Failure (CHF). Results: Application of predictive analytics in clinical care may not always be a cost-saving strategy compared with intervening on all patients. Improving the accuracy of a predictive model may lower costs, but other factors such as the prevalence and cost of the outcome, and the cost and effectiveness of the intervention designed to avert the outcome may be more influential in determining the favored strategy. Conclusion: An interactive cost-benefit analyses provides insights regarding the financial implications of a clinical strategy that implements predictive analytics.

Developing a risk-based composite neurologic outcome for a trial of hydroxyurea in young children with sickle cell disease

BACKGROUND

Studies of interventions to prevent the many neurological complications of sickle cell disease must take into account multiple outcomes of variable severity, with limited sample size. The goals of the studies presented were to use investigator preferences across outcomes to determine an attitude-based weighting of relevant clinical outcomes and to establish a valid composite outcome for a clinical trial.

METHODS

In Study 1, investigators were surveyed about their practice regarding hydroxyurea therapy and opinions about outcomes for the "Hydroxyurea to Prevent the Central Nervous System Complications of Sickle Cell Disease Trial" (HU Prevent), and their minimally acceptable relative risk reduction for the two outcome components, motor and neurocognitive deficits. In Study 2, HU Prevent investigators provided overall weights for these two components. In Study 3, they provided more granular rankings, ratings, and maximum number acceptable to harm. A weighted composite outcome, the Stroke Consequences Risk Score, was constructed that incorporates the major neurologic complications of sickle cell disease. The Stroke Consequences Risk Score represents the 3-year risk of suffering the adverse consequences of stroke. In Study 4, the results of the Optimizing Primary Stroke Prevention in Sickle Cell Anemia (STOP2) and Silent Infarct Transfusion Trials were reanalyzed in light of the composite outcome.

RESULTS

In total, 22 to 27 investigators participated per study. In Study 1, across three samplings between 2009 and 2015, the average minimally acceptable relative risk reduction ranged from 0.36 to 0.50, at or below the target effect size of 0.50. In 2015, 21 (91%) reported that a placebo-controlled trial is reasonable; 23 (100%), that it is ethical; and 22 (96%), that they would change their practice, if the results of the trial were positive. In Studies 2 and 3, the weight elicited for a cognitive decline (of 10 IQ points) from the overall assessment was 0.67 (and for motor deficit, the complementary 0.33); from ranking, 0.6; from rating, 0.58; and from maximal number acceptable to harm, 0.5. Using data from two major clinical trials, Study 4 demonstrated the same conclusions as the original trials using the Stroke Consequences Risk Score, with smaller p-values for both reanalyses. An assessment of acceptability was performed as well.

CONCLUSION

This set of studies provides the rationale, justification, and validation for the use of a weighted composite outcome and confirms the need for the phase III HU Prevent study. Surveys of investigators in multi-center studies can provide the basis of clinically meaningful outcomes that foster the translation of study results into practice while increasing the efficiency of a study.

Desiderata for sharable computable biomedical knowledge for learning health systems

In this commentary, we work out the specific desired functions required for sharing knowledge objects (based on statistical models) presumably to be used for clinical decision support derived from a learning health system, and, in so doing, discuss the implications for novel knowledge architectures. We will demonstrate how decision models, implemented as influence diagrams, satisfy the desiderata. The desiderata include locally validate discrimination, locally validate calibration, locally recalculate thresholds by incorporating local preferences, provide explanation, enable monitoring, enable debiasing, account for generalizability, account for semantic uncertainty, shall be findable, and others as necessary and proper. We demonstrate how formal decision models, especially when implemented as influence diagrams based on Bayesian networks, support both the knowledge artifact itself (the "primary decision") and the "meta-decision" of whether to deploy the knowledge artifact. We close with a research and development agenda to put this framework into place.

Mapping the Flow of Pediatric Trauma Patients Using Process Mining

BACKGROUND

Inhospital pediatric trauma care typically spans multiple locations, which influences the use of resources, that could be improved by gaining a better understanding of the inhospital flow of patients and identifying opportunities for improvement.

OBJECTIVES

To describe a process mining approach for mapping the inhospital flow of pediatric trauma patients, to identify and characterize the major patient pathways and care transitions, and to identify opportunities for patient flow and triage improvement.

METHODS

From the trauma registry of a level I pediatric trauma center, data were extracted regarding the two highest trauma activation levels, Alpha (n = 228) and Bravo (n = 1,713). An event log was generated from the admission, discharge, and transfer data from which patient pathways and care transitions were identified and described. The Flexible Heuristics Miner algorithm was used to generate a process map for the cohort, and separate process maps for Alpha and Bravo encounters, which were assessed for conformance when fitness value was less than 0.950, with the identification and comparison of conforming and nonconforming encounters.

RESULTS

The process map for the cohort was similar to a validated process map derived through qualitative methods. The process map for Bravo encounters had a relatively low fitness of 0.887, and 96 (5.6%) encounters were identified as nonconforming with characteristics comparable to Alpha encounters. In total, 28 patient pathways and 20 care transitions were identified. The top five patient pathways were traversed by 92.1% of patients, whereas the top five care transitions accounted for 87.5% of all care transitions. A larger-than-expected number of discharges from the pediatric intensive care unit (PICU) were identified, with 84.2% involving discharge to home without the need for home care services.

CONCLUSION

Process mining was successfully applied to derive process maps from trauma registry data and to identify opportunities for trauma triage improvement and optimization of PICU use.

A proposed national research and development agenda for population health informatics: summary recommendations from a national expert workshop

OBJECTIVE

The Johns Hopkins Center for Population Health IT hosted a 1-day symposium sponsored by the National Library of Medicine to help develop a national research and development (R&D) agenda for the emerging field of population health informatics (PopHI).

MATERIAL AND METHODS

The symposium provided a venue for national experts to brainstorm, identify, discuss, and prioritize the top challenges and opportunities in the PopHI field, as well as R&D areas to address these.

RESULTS

This manuscript summarizes the findings of the PopHI symposium. The symposium participants' recommendations have been categorized into 13 overarching themes, including policy alignment, data governance, sustainability and incentives, and standards/interoperability.

DISCUSSION

The proposed consensus-based national agenda for PopHI consisted of 18 priority recommendations grouped into 4 broad goals: (1) Developing a standardized collaborative framework and infrastructure, (2) Advancing technical tools and methods, (3) Developing a scientific evidence and knowledge base, and (4) Developing an appropriate framework for policy, privacy, and sustainability. There was a substantial amount of agreement between all the participants on the challenges and opportunities for PopHI as well as on the actions that needed to be taken to address these.

CONCLUSION

PopHI is a rapidly growing field that has emerged to address the population dimension of the Triple Aim. The proposed PopHI R&D agenda is comprehensive and timely, but should be considered only a starting-point, given that ongoing developments in health policy, population health management, and informatics are very dynamic, suggesting that the agenda will require constant monitoring and updating.

After Detection: The Improved Accuracy of Lung Cancer Assessment Using Radiologic Computer-aided Diagnosis

RATIONALE AND OBJECTIVES

The aim of this study was to evaluate the improved accuracy of radiologic assessment of lung cancer afforded by computer-aided diagnosis (CADx).

MATERIALS AND METHODS

Inclusion/exclusion criteria were formulated, and a systematic inquiry of research databases was conducted. Following title and abstract review, an in-depth review of 149 surviving articles was performed with accepted articles undergoing a Quality Assessment of Diagnostic Accuracy Studies (QUADAS)-based quality review and data abstraction.

RESULTS

A total of 14 articles, representing 1868 scans, passed the review. Increases in the receiver operating characteristic (ROC) area under the curve of .8 or higher were seen in all nine studies that reported it, except for one that employed subspecialized radiologists.

CONCLUSIONS

This systematic review demonstrated improved accuracy of lung cancer assessment using CADx over manual review, in eight high-quality observer-performance studies. The improved accuracy afforded by radiologic lung-CADx suggests the need to explore its use in screening and regular clinical workflow.

Brief condom interventions targeting males in clinical settings: a meta-analysis

OBJECTIVE

The objective of this study is to assess the effectiveness of brief clinic-based condom skills interventions that target males.

STUDY DESIGN

We searched PubMed, Cumulative Index of Nursing and Allied Health Literature and PsychInfo for studies published from January 1980 through September 2014, using relevant search terms. We included studies if interventions taught about condoms lasting 60 min or shorter, used randomized or quasi-experimental design, were conducted in a clinical setting and targeted males. Two investigators sequentially reviewed abstracts. We abstracted and reviewed data from 16 studies that met the selection criteria. Where outcomes were poolable, we conducted meta-analyses using a random-effects model and I(2) index to assess heterogeneity. Outcome measures included condom knowledge, attitudes, behaviors, sexually transmitted infections (STIs)/human immunodeficiency virus and unintended pregnancy.

RESULTS

Across studies, teaching about condoms was nested within sexual risk reduction curricula. Most interventions were one on one and conducted in STI clinics. Pooled analyses indicated that intervention receipt was associated with increases in percent of sex acts with condoms (standardized mean difference=0.29 [0.18, 0.41]; 0.19 [0.06, 0.33]) and reductions in STIs at 12-month follow-up or longer {odds ratio (OR)=0.82 [95% confidence interval: 0.67, 0.99]}. One study assessed unintended pregnancy and did not find an intervention effect.

CONCLUSIONS

Study findings hold promise for considering brief condom skills interventions in clinical settings that can result in improvements in males' condom behaviors and possibly biological outcomes.

Public Health and Epidemiology Informatics: Recent Research and Trends in the United States

OBJECTIVES

To survey advances in public health and epidemiology informatics over the past three years.

METHODS

We conducted a review of English-language research works conducted in the domain of public health informatics (PHI), and published in MEDLINE between January 2012 and December 2014, where information and communication technology (ICT) was a primary subject, or a main component of the study methodology. Selected articles were synthesized using a thematic analysis using the Essential Services of Public Health as a typology.

RESULTS

Based on themes that emerged, we organized the advances into a model where applications that support the Essential Services are, in turn, supported by a socio-technical infrastructure that relies on government policies and ethical principles. That infrastructure, in turn, depends upon education and training of the public health workforce, development that creates novel or adapts existing infrastructure, and research that evaluates the success of the infrastructure. Finally, the persistence and growth of infrastructure depends on financial sustainability.

CONCLUSIONS

Public health informatics is a field that is growing in breadth, depth, and complexity. Several Essential Services have benefited from informatics, notably, "Monitor Health," "Diagnose & Investigate," and "Evaluate." Yet many Essential Services still have not yet benefited from advances such as maturing electronic health record systems, interoperability amongst health information systems, analytics for population health management, use of social media among consumers, and educational certification in clinical informatics. There is much work to be done to further advance the science of PHI as well as its impact on public health practice.

Preemptive Bone Marrow Transplantation for FANCD1/BRCA2

Children with biallelic mutations in FANCD1/BRCA2 are at uniquely high risks of leukemia and solid tumors. Preemptive bone marrow transplantation (PE-BMT) has been proposed to avoid the development of leukemia, but empirical study of PE-BMT is unlikely because of the rarity of these children and the unknown benefit of PE-BMT. We used survival analysis to estimate the risks of leukemia and the expected survival if leukemia could be eliminated by curative PE-BMT. We used the results in a decision analysis model to explore the plausibility of PE-BMT for children with variable ages at diagnosis and risks of transplantation-related mortality. For example, PE-BMT at 1 year of age with a 10% risk of transplantation-related mortality increased the mean survival by 1.7 years. The greatest benefit was for patients diagnosed between 1 and 3 years of age, after which the benefit of PE-BMT decreased with age at diagnosis, and the risk of death from solid tumors constituted a relatively greater burden of mortality. Our methods may be used to model survival for other hematologic disorders with limited empirical data and a pressing need for clinical guidance.

Automated detection of retinal disease

Nearly 4 in 10 Americans with diabetes currently fail to undergo recommended annual retinal exams, resulting in tens of thousands of cases of blindness that could have been prevented. Advances in automated retinal disease detection could greatly reduce the burden of labor-intensive dilated retinal examinations by ophthalmologists and optometrists and deliver diagnostic services at lower cost. As the current availability of ophthalmologists and optometrists is inadequate to screen all patients at risk every year, automated screening systems deployed in primary care settings and even in patients' homes could fill the current gap in supply. Expanding screens to all patients at risk by switching to automated detection systems would in turn yield significantly higher rates of detecting and treating diabetic retinopathy per dilated retinal examination. Fewer diabetic patients would develop complications such as blindness, while ophthalmologists could focus on more complex cases.

The Ontology of Clinical Research (OCRe): an informatics foundation for the science of clinical research

To date, the scientific process for generating, interpreting, and applying knowledge has received less informatics attention than operational processes for conducting clinical studies. The activities of these scientific processes - the science of clinical research - are centered on the study protocol, which is the abstract representation of the scientific design of a clinical study. The Ontology of Clinical Research (OCRe) is an OWL 2 model of the entities and relationships of study design protocols for the purpose of computationally supporting the design and analysis of human studies. OCRe's modeling is independent of any specific study design or clinical domain. It includes a study design typology and a specialized module called ERGO Annotation for capturing the meaning of eligibility criteria. In this paper, we describe the key informatics use cases of each phase of a study's scientific lifecycle, present OCRe and the principles behind its modeling, and describe applications of OCRe and associated technologies to a range of clinical research use cases. OCRe captures the central semantics that underlies the scientific processes of clinical research and can serve as an informatics foundation for supporting the entire range of knowledge activities that constitute the science of clinical research.

Caveats for the use of operational electronic health record data in comparative effectiveness research

The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.

Identifying appropriate recipients for CDC infectious risk donor kidneys

Over 10% of deceased donors in 2011 met PHS/CDC criteria for infectious risk donor (IRD), and discard rates are significantly higher for kidneys from these donors. We hypothesized that patient phenotypes exist for whom the survival benefit outweighs the infectious risk associated with IRDs. A patient-oriented Markov decision process model was developed and validated, based on SRTR data and meta-analyses of window period risks among persons with IRD behaviors. The Markov model allows patients to see, for their phenotype, their estimated survival after accepting versus declining an IRD offer, graphed over a 5-year horizon. Estimated 5-year survival differences associated with accepting IRDs ranged from -6.4% to +67.3% for a variety of patient phenotypes. Factors most predictive of the survival difference with IRD transplantation were age, PRA, previous transplant, and the expected time until the next non-IRD deceased donor offer. This study suggests that survival benefit derived from IRD kidneys varies widely by patient phenotype. Furthermore, within the inherent limitations of model-based prediction, this study demonstrates that it is possible to identify those predicted to benefit from IRD kidneys, and illustrates how estimated survival curves based on a clinical decision can be presented to better inform patient and provider decision-making.

Ontology-based federated data access to human studies information

Human studies are one of the most valuable sources of knowledge in biomedical research, but data about their design and results are currently widely dispersed in siloed systems. Federation of these data is needed to facilitate large-scale data analysis to realize the goals of evidence-based medicine. The Human Studies Database project has developed an informatics infrastructure for federated query of human studies databases, using a generalizable approach to ontology-based data access. Our approach has three main components. First, the Ontology of Clinical Research (OCRe) provides the reference semantics. Second, a data model, automatically derived from OCRe into XSD, maintains semantic synchrony of the underlying representations while facilitating data acquisition using common XML technologies. Finally, the Query Integrator issues queries distributed over the data, OCRe, and other ontologies such as SNOMED in BioPortal. We report on a demonstration of this infrastructure on data acquired from institutional systems and from ClinicalTrials.gov.

Healthcare information technology and economics

At the 2011 American College of Medical Informatics (ACMI) Winter Symposium we studied the overlap between health IT and economics and what leading healthcare delivery organizations are achieving today using IT that might offer paths for the nation to follow for using health IT in healthcare reform. We recognized that health IT by itself can improve health value, but its main contribution to health value may be that it can make possible new care delivery models to achieve much larger value. Health IT is a critically important enabler to fundamental healthcare system changes that may be a way out of our current, severe problem of rising costs and national deficit. We review the current state of healthcare costs, federal health IT stimulus programs, and experiences of several leading organizations, and offer a model for how health IT fits into our health economic future.

Active surveillance versus surgery for low risk prostate cancer: a clinical decision analysis

PURPOSE

We assessed the effect of age, health status and patient preferences on outcomes of surgery vs active surveillance for low risk prostate cancer.

MATERIALS AND METHODS

We used Monte Carlo simulation of Markov models of the life courses of 200,000 men diagnosed with low risk prostate cancer and treated with surveillance or radical prostatectomy to calculate quality adjusted life expectancy, life expectancy, prostate cancer specific mortality and years of treatment side effects, with model parameters derived from the literature. We simulated outcomes for men 50 to 75 years old with poor, average or excellent health status (50%, 100% and 150% of average life expectancy, respectively). Sensitivity of outcomes to uncertainties in model parameters was tested.

RESULTS

For 65-year-old men in average health, surgery resulted in 0.3 additional years of life expectancy, 1.6 additional years of impotence or incontinence and a 4.9% decrease in prostate cancer specific mortality compared to surveillance, for a net difference of 0.05 fewer quality adjusted life years. Increased age and poorer baseline health status favored surveillance. With greater than 95% probability, surveillance resulted in net benefits compared to surgery for age older than 74, 67 and 54 years for men in excellent, average and poor health, respectively. Patient preferences toward life under surveillance, biochemical recurrence of disease, treatment side effects and future discount rate affected optimal management choice.

CONCLUSIONS

Older men and men in poor health are likely to have better quality adjusted life expectancy with active surveillance. However, specific individual preferences impact optimal choices and should be a primary consideration in shared decision making.

Adolescent and parental utilities for the health states associated with pelvic inflammatory disease

PURPOSE

There is limited information about how the consumers of adolescent pelvic inflammatory disease (PID) care value health states associated with the disorder. The aim of this study is to determine and compare adolescent and parent PID-related health utilities.

METHODS

Adolescent girls (N=134) and parents (N=121) completed a web-based utility elicitation survey. Participants reviewed five scenarios describing the health states associated with PID (outpatient treatment (mild-moderate disease), inpatient treatment (severe disease), ectopic pregnancy, infertility and chronic abdominal pain). After each scenario, participants were asked to rate health-related quality of life (HRQL) using a Visual Analogue Scale (VAS) and to complete a time trade-off (TTO) assessment. Data were evaluated using multiple linear (VAS) and quantile (TTO) regression analyses.

RESULTS

Adolescents had significantly lower mean valuations (p<0.01) than the parents on the VAS for HRQL in each health state (outpatient (62 vs 76), inpatient (57 vs 74), ectopic (55 vs 73), infertility (59 vs 68) and chronic abdominal pain (48 vs 61)). Using quantile regression analysis, adolescents were also willing to give up more time for health gains indicated by lower median TTO scores (p<0.01) for outpatient treatment (0.98 vs 1.0), inpatient treatment (0.96 vs 1.0) and ectopic pregnancy (0.98 vs 1.0).

CONCLUSIONS

The authors demonstrate that adolescents assign more disutility (lower valuations) than parents for HRQL and three of five of the TTO assessments for PID-related health states. Future economic evaluations using patient-specific preferences to determine resource allocation for PID management in adolescents should include adolescent health outcomes and utilities.

The impact of physician subspecialty training, risk calculation, and patient age on treatment recommendations in ocular hypertension

PURPOSE

To determine whether glaucoma subspecialty training, formal risk estimation, or patient age has an impact on physician treatment recommendations in cases of ocular hypertension.

DESIGN

Experimental study.

METHODS

Members of the American Academy of Ophthalmology (118) and American Glaucoma Society (58) were recruited. Each physician was first asked how many young and old patients with ocular hypertension he or she would treat to prevent someone from progressing to glaucoma (number needed to treat). The physicians then reviewed 100 simulated cases of patients with ocular hypertension and reported their likelihood to treat each case. Half of these cases were presented with an estimated risk of conversion to glaucoma within 5 years and half were presented without an estimate. The treatment recommendations were analyzed to determine whether subspecialty status or the presence of a risk calculation had any impact on treatment recommendations.

RESULTS

Both glaucoma specialists and non-glaucoma specialists were more likely to recommend treatment in cases for which a risk calculation was provided (P = .001). Furthermore, non-glaucoma specialists were more likely to recommend treatment for ocular hypertensive patients than were glaucoma specialists (P < .001). Finally, both groups indicated they were more likely to treat young patients than old.

CONCLUSIONS

Both provision of a risk estimate and lack of glaucoma subspecialty training were associated with physicians being more likely to treat ocular hypertension. These findings have implications with regard to ways in which the treatment of ocular hypertensive patients could be modified and possibly made more consistent with available evidence.

Evaluating information prescriptions in two clinical environments

OBJECTIVE

The research sought to evaluate whether providing personalized information services by libraries can improve satisfaction with information services for specific types of patients.

METHODS

Adult breast cancer (BrCa) clinic patients and mothers of inpatient neonatal intensive care unit (NICU) patients were randomized to receive routine information services (control) or an IRx intervention.

RESULTS

The BrCa trial randomized 211 patients and the NICU trial, 88 mothers. The BrCa trial showed no statistically significant differences in satisfaction ratings between the treatment and control groups. The IRx group in the NICU trial reported higher satisfaction than the control group regarding information received about diagnosis, treatments, respiratory tradeoffs, and medication tradeoffs. BrCa patients posed questions to librarians more frequently than did NICU mothers, and a higher percentage reported using the website. Questions asked of the librarians by BrCa patients were predominantly clinical and focused on the areas of treatment and side effects.

CONCLUSIONS

Study results provide some evidence to support further efforts to both implement information prescription projects in selected settings and to conduct additional research on the costs and benefits of services.

Decision Modeling of Disagreements

Purpose. To identify core disagreements between pediatric hematologists who would treat children with idiopathic thrombocytopenic purpura (ITP) on initial presentation (“Treaters”) and those who would treat by observation (“Observers”), to determine whether each group’s preferred stance was consistent with each individual’s detailed perceptions, and to identify key variables in each stance. Methods. A decision model was constructed in collaboration with experts, and a detailed questionnaire was presented to a nationally representative committee of 25 pediatric hematologists. A full decision tree was specified for each respondent. Results. Nineteen (76%) experts responded; based on preference for initial treatment, 9 were Treaters and 10 Observers. Of the 30 probability/effectiveness variables, 8—almost all concerning treatment effectiveness—had at least one statistically-significant difference between the 2 groups regarding low, best, or high estimates. To convince Observers that treatment is effective would take a clinical trial with between 39 000 and 87 000 participants; to convince Treaters that treatment is not effective enough, between 97 000 and 114 000 participants. Observers’ calculated numbers needed to treat (NNTs) of about 150 000 are more consistent ( P = 0.0023) with their elicited maximum NTTs of about 500. Conclusion. Physicians not specifically trained provided enough data to specify complete individual decision models. From the estimates provided, no practical clinical trial could convince hematologists who would treat children on initial presentation with ITP just to simply observe them or could convince those who would just observe to instead treat with available agents. Perceived burdens could be better characterized, perhaps by including parental perceptions and preferences.

Consumer health informatics: results of a systematic evidence review and evidence based recommendations

An increasing array of technology based tools are available for patient and consumer utilization which claim to facilitate health improvement. The efficacy of these Consumer Health Informatics tools has not previously been systematically reviewed. As such a systematic evidence review of the efficacy of consumer health informatics tools was conducted. This review also sought evidence of any barriers to future widespread utilization of these tools and evidence of economic impact of these tools on health care costs. The findings of this review indicate that while more work needs to be done, the available literature does suggest a positive impact of consumer health informatics tools on select health conditions and outcomes. Many barriers remain that must be overcome prior to widespread utilization of these tools. There was insufficient data regarding economic impact of consumer health informatics tools on healthcare costs.

Applying evidence in practice: A qualitative case study of the factors affecting residents’ decisions

Patient care based on best available evidence is increasingly viewed as the hallmark of good quality medical diagnosis and treatment, yet its uptake is often slow and uneven and the reasons underlying the slow diffusion of evidence-based guidelines remain elusive. The authors report a qualitative study conducted at a major US teaching hospital which sought to discover the reasons why an evidence-based anticoagulation guideline appeared to be applied irregularly, with problematic results. Using a theoretical framework derived from Rogers’ work on the diffusion of innovation, this article describes the ways in which a group of residents evaluated and applied evidence in the context of caring for their patients. Future work in evidence-based practice can benefit from a greater emphasis on studies that use multi-method, qualitative designs to explore the complex ways in which people interact with information and the changes that ensue from its use.

A new method for determining physician decision thresholds using empiric, uncertain recommendations

Background

The concept of risk thresholds has been studied in medical decision making for over 30 years. During that time, physicians have been shown to be poor at estimating the probabilities required to use this method. To better assess physician risk thresholds and to more closely model medical decision making, we set out to design and test a method that derives thresholds from actual physician treatment recommendations. Such an approach would avoid the need to ask physicians for estimates of patient risk when trying to determine individual thresholds for treatment. Assessments of physician decision making are increasingly relevant as new data are generated from clinical research. For example, recommendations made in the setting of ocular hypertension are of interest as a large clinical trial has identified new risk factors that should be considered by physicians. Precisely how physicians use this new information when making treatment recommendations has not yet been determined.

Results

We derived a new method for estimating treatment thresholds using ordinal logistic regression and tested it by asking ophthalmologists to review cases of ocular hypertension before expressing how likely they would be to recommend treatment. Fifty-eight physicians were recruited from the American Glaucoma Society. Demographic information was collected from the participating physicians and the treatment threshold for each physician was estimated. The method was validated by showing that while treatment thresholds varied over a wide range, the most common values were consistent with the 10-15% 5-year risk of glaucoma suggested by expert opinion and decision analysis.

Conclusions

This method has advantages over prior means of assessing treatment thresholds. It does not require physicians to explicitly estimate patient risk and it allows for uncertainty in the recommendations. These advantages will make it possible to use this method when assessing interventions intended to alter clinical decision making.

Development and evaluation of a study design typology for human research

A systematic classification of study designs would be useful for researchers, systematic reviewers, readers, and research administrators, among others. As part of the Human Studies Database Project, we developed the Study Design Typology to standardize the classification of study designs in human research. We then performed a multiple observer masked evaluation of active research protocols in four institutions according to a standardized protocol. Thirty-five protocols were classified by three reviewers each into one of nine high-level study designs for interventional and observational research (e.g., N-of-1, Parallel Group, Case Crossover). Rater classification agreement was moderately high for the 35 protocols (Fleiss' kappa = 0.442) and higher still for the 23 quantitative studies (Fleiss' kappa = 0.463). We conclude that our typology shows initial promise for reliably distinguishing study design types for quantitative human research.

Impact of consumer health informatics applications

OBJECTIVE

The objective of the report is to review the evidence on the impact of consumer health informatics (CHI) applications on health outcomes, to identify the knowledge gaps and to make recommendations for future research.

DATA SOURCES

We searched MEDLINE, EMBASE, The Cochrane Library, Scopus, and CINAHL databases, references in eligible articles and the table of contents of selected journals; and query of experts.

METHODS

Paired reviewers reviewed citations to identify randomized controlled trials (RCTs) of the impact of CHI applications, and all studies that addressed barriers to use of CHI applications. All studies were independently assessed for quality. All data was abstracted, graded, and reviewed by 2 different reviewers.

RESULTS

One hundred forty-six eligible articles were identified including 121 RCTs. Studies were very heterogeous and of variable quality. Four of five asthma care studies found significant positive impact of a CHI application on at least one healthcare process measure. In terms of the impact of CHI on intermediate health outcomes, significant positive impact was demonstrated in at least one intermediate health outcome of; all three identified breast cancer studies, 89 percent of 32 diet, exercise, physical activity, not obesity studies, all 7 alcohol abuse studies, 58 percent of 19 smoking cessation studies, 40 percent of 12 obesity studies, all 7 diabetes studies, 88 percent of 8 mental health studies, 25 percent of 4 asthma/COPD studies, and one of two menopause/HRT utilization studies. Thirteen additional single studies were identified and each found evidence of significant impact of a CHI application on one or more intermediate outcomes. Eight studies evaluated the effect of CHI on the doctor patient relationship. Five of these studies demonstrated significant positive impact of CHI on at least one aspect of the doctor patient relationship. In terms of the impact of CHI on clinical outcomes, significant positive impact was demonstrated in at least one clinical outcome of; one of three breast cancer studies, four of five diet, exercise, or physical activity studies, all seven mental health studies, all three identified diabetes studies. No studies included in this review found any evidence of consumer harm attributable to a CHI application. Evidence was insufficient to determine the economic impact of CHI applications.

CONCLUSIONS

Despite study heterogeneity, quality variability, and some data paucity, available literature suggests that select CHI applications may effectively engage consumers, enhance traditional clinical interventions, and improve both intermediate and clinical health outcomes.

Synergies and distinctions between computational disciplines in biomedical research: perspective from the Clinical andTranslational Science Award programs

Clinical and translational research increasingly requires computation. Projects may involve multiple computationally oriented groups including information technology (IT) professionals, computer scientists, and biomedical informaticians. However, many biomedical researchers are not aware of the distinctions among these complementary groups, leading to confusion, delays, and suboptimal results. Although written from the perspective of Clinical and Translational Science Award (CTSA) programs within academic medical centers, this article addresses issues that extend beyond clinical and translational research. The authors describe the complementary but distinct roles of operational IT, research IT, computer science, and biomedical informatics using a clinical data warehouse as a running example. In general, IT professionals focus on technology. The authors distinguish between two types of IT groups within academic medical centers: central or administrative IT (supporting the administrative computing needs of large organizations) and research IT (supporting the computing needs of researchers). Computer scientists focus on general issues of computation such as designing faster computers or more efficient algorithms, rather than specific applications. In contrast, informaticians are concerned with data, information, and knowledge. Biomedical informaticians draw on a variety of tools, including but not limited to computers, to solve information problems in health care and biomedicine. The paper concludes with recommendations regarding administrative structures that can help to maximize the benefit of computation to biomedical research within academic health centers.