Status Update on Data Required to Build a Learning Health System

Organizational Factors in Clinical Data Sharing for Artificial Intelligence in Health Care

Importance

Limited sharing of data sets that accurately represent disease and patient diversity limits the generalizability of artificial intelligence (AI) algorithms in health care.

Objective

To explore the factors associated with organizational motivation to share health data for AI development.

Design, Setting, and Participants

This qualitative study investigated organizational readiness for sharing health data across the academic, governmental, nonprofit, and private sectors. Using a multiple case studies approach, 27 semistructured interviews were conducted with leaders in data-sharing roles from August 29, 2022, to January 9, 2023. The interviews were conducted in the English language using a video conferencing platform. Using a purposive and nonprobabilistic sampling strategy, 78 individuals across 52 unique organizations were identified. Of these, 35 participants were enrolled. Participant recruitment concluded after 27 interviews, as theoretical saturation was reached and no additional themes emerged.

Main Outcome and Measure

Concepts defining organizational readiness for data sharing and the association between data-sharing factors and organizational behavior were mapped through iterative qualitative analysis to establish a framework defining organizational readiness for sharing clinical data for AI development.

Results

Interviews included 27 leaders from 18 organizations (academia: 10, government: 7, nonprofit: 8, and private: 2). Organizational readiness for data sharing centered around 2 main constructs: motivation and capabilities. Motivation related to the alignment of an organization's values with data-sharing priorities and was associated with its engagement in data-sharing efforts. However, organizational motivation could be modulated by extrinsic incentives for financial or reputational gains. Organizational capabilities comprised infrastructure, people, expertise, and access to data. Cross-sector collaboration was a key strategy to mitigate barriers to access health data.

Conclusions and Relevance

This qualitative study identified sector-specific factors that may affect the data-sharing behaviors of health organizations. External incentives may bolster cross-sector collaborations by helping overcome barriers to accessing health data for AI development. The findings suggest that tailored incentives may boost organizational motivation and facilitate sustainable flow of health data for AI development.

Overview of approaches to estimate real-world disease progression in lung cancer

BACKGROUND

Randomized clinical trials of novel treatments for solid tumors normally measure disease progression using the Response Evaluation Criteria in Solid Tumors. However, novel, scalable approaches to estimate disease progression using real-world data are needed to advance cancer outcomes research. The purpose of this narrative review is to summarize examples from the existing literature on approaches to estimate real-world disease progression and their relative strengths and limitations, using lung cancer as a case study.

METHODS

A narrative literature review was conducted in PubMed to identify articles that used approaches to estimate real-world disease progression in lung cancer patients. Data abstracted included data source, approach used to estimate real-world progression, and comparison to a selected gold standard (if applicable).

RESULTS

A total of 40 articles were identified from 2008 to 2022. Five approaches to estimate real-world disease progression were identified including manual abstraction of medical records, natural language processing of clinical notes and/or radiology reports, treatment-based algorithms, changes in tumor volume, and delta radiomics-based approaches. The accuracy of these progression approaches were assessed using different methods, including correlations between real-world endpoints and overall survival for manual abstraction (Spearman rank ρ = 0.61-0.84) and area under the curve for natural language processing approaches (area under the curve = 0.86-0.96).

CONCLUSIONS

Real-world disease progression has been measured in several observational studies of lung cancer. However, comparing the accuracy of methods across studies is challenging, in part, because of the lack of a gold standard and the different methods used to evaluate accuracy. Concerted efforts are needed to define a gold standard and quality metrics for real-world data.

Risk factors of early mortality in patients with small cell lung cancer: a retrospective study in the SEER database

BACKGROUND

Small cell lung cancer (SCLC) is a highly aggressive neuroendocrine cancer with a high risk of early mortality (i.e., survival time less than 1 month). This study aimed to identify relevant risk factors and predict early mortality in SCLC patients.

METHODS

A total of 27,163 SCLC cases registered between 2010 and 2019 were extracted from the Surveillance, Epidemiology, and End Results (SEER) data. Significant independent risk factors were identified by univariate and multivariate logistic regression analyses. Nomograms for all-causes and cancer-specific early death were constructed and evaluated.

RESULTS

Age, sex, clinical stage, presence of metastasis (liver and lung), and absence of treatment (surgery, radiotherapy and chemotherapy) were identified for significant association with all-causes and cancer-specific early death. Nomograms based on these predictors exhibited high accuracy (area under ROC curve > 0.850) and potential clinical practicality in the prediction of early mortality.

CONCLUSION

We identified a set of factors associated with early mortality from SCLC and developed a clinically useful nomogram to predict high-risk patients. This nomogram could aid oncologists in the administration of individualized treatment regimens, potentially improving clinical outcomes of SCLC patients.

Real word challenges in integrating electronic medical record and administrative health data for regional quality improvement in diabetes: a retrospective cross-sectional analysis

BACKGROUND

Linked electronic medical records and administrative data have the potential to support a learning health system and data-driven quality improvement. However, data completeness and accuracy must first be assessed before their application. We evaluated the processes, feasibility, and limitations of linking electronic medical records and administrative data for the purpose of quality improvement within five specialist diabetes clinics in Edmonton, Alberta, a province known for its robust health data infrastructure.

METHODS

We conducted a retrospective cross-sectional analysis using electronic medical record and administrative data for individuals ≥ 18 years attending the clinics between March 2017 and December 2018. Descriptive statistics were produced for demographics, service use, diabetes type, and standard diabetes benchmarks. The systematic and iterative process of obtaining results is described.

RESULTS

The process of integrating electronic medical record with administrative data for quality improvement was found to be non-linear and iterative and involved four phases: project planning, information generating, limitations analysis, and action. After limitations analysis, questions were grouped into those that were answerable with confidence, answerable with limitations, and not answerable with available data. Factors contributing to data limitations included inaccurate data entry, coding, collation, migration and synthesis, changes in laboratory reporting, and information not captured in existing databases.

CONCLUSION

Electronic medical records and administrative databases can be powerful tools to establish clinical practice patterns, inform data-driven quality improvement at a regional level, and support a learning health system. However, there are substantial data limitations that must be addressed before these sources can be reliably leveraged.

The learning health system for pediatric nephrology: building better systems to improve health

Learning health systems (LHS) align science, informatics, incentives, and culture for continuous improvement and innovation. In this organizational system, best practices are seamlessly embedded in the delivery process, and new knowledge is captured as an integral byproduct of the care delivery experience aimed to transform clinical practice and improve patient outcomes. The objective of this review is to describe how building better health systems that integrate clinical care, improvement, and research as part of an LHS can improve care within pediatric nephrology. This review will provide real-world examples of how this system can be established in a single center and across multiple centers as learning health networks.

Prioritizing Measures that Matter Within a Person-Centered Oncology Learning Health System

Background

Despite progress in developing learning health systems (LHS) and associated metrics of success, a gap remains in identifying measures to guide the implementation and assessment of the impact of an oncology LHS. Our aim was to identify a balanced set of measures to guide a person-centered oncology LHS.

Methods

A modified Delphi process and clinical value compass framework were used to prioritize measures for tracking LHS performance. A multidisciplinary group of 77 stakeholders, including people with cancer and family members, participated in 3 rounds of online voting followed by 50-minute discussions. Participants rated metrics on perceived importance to the LHS and discussed priorities.

Results

Voting was completed by 94% of participants and prioritized 22 measures within 8 domains. Patient and caregiver factors included clinical health (Eastern Cooperative Oncology Group Performance Status, survival by cancer type and stage), functional health and quality of life (Patient Reported Outcomes Measurement Information System [PROMIS] Global-10, Distress Thermometer, Modified Caregiver Strain Index), experience of care (advance care planning, collaboRATE, PROMIS Self-Efficacy Scale, access to care, experience of care, end-of-life quality measures), and cost and resource use (avoidance and delay in accessing care and medications, financial hardship, total cost of care). Contextual factors included team well-being (Well-being Index; voluntary staff turnover); learning culture (Improvement Readiness, compliance with Commission on Cancer quality of care measures); scholarly engagement and productivity (institutional commitment and support for research, academic productivity index); and diversity, equity, inclusion, and belonging (screening and follow-up for social determinants of health, inclusivity of staff and patients).

Conclusions

The person-centered LHS value compass provides a balanced set of measures that oncology practices can use to monitor and evaluate improvement across multiple domains.

Making Diabetes Electronic Medical Record Data Actionable: Promoting Benchmarking and Population Health Improvement Using the T1D Exchange Quality Improvement Portal

This article describes how the T1D Exchange Quality Improvement Collaborative leverages an innovative web platform, the QI Portal, to gather and store electronic medical record (EMR) data to promote benchmarking and population health improvement in a type 1 diabetes learning health system. The authors explain the value of the QI Portal, the process for mapping center-level data from EMRs using standardized data specifications, and the QI Portal's unique features for advancing population health.

Leveraging Clinical Informatics Tools to Extract Cumulative Anthracycline Exposure, Measure Cardiovascular Outcomes, and Assess Guideline Adherence for Children With Cancer

PURPOSE

Cardiovascular disease is a significant cause of late morbidity and mortality in survivors of childhood cancer. Clinical informatics tools could enhance provider adherence to echocardiogram guidelines for early detection of late-onset cardiomyopathy.

METHODS

Cancer registry data were linked to electronic health record data. Structured query language facilitated the construction of anthracycline-exposed cohorts at a single institution. Primary outcomes included the data quality from automatic anthracycline extraction, sensitivity of International Classification of Disease coding for heart failure, and adherence to echocardiogram guideline recommendations.

RESULTS

The final analytic cohort included 385 pediatric oncology patients diagnosed between July 1, 2013, and December 31, 2018, among whom 194 were classified as no anthracycline exposure, 143 had low anthracycline exposure (< 250 mg/m²), and 48 had high anthracycline exposure (≥ 250 mg/m²). Manual review of anthracycline exposure was highly concordant (95%) with the automatic extraction. Among the unexposed group, 15% had an anthracycline administered at an outside institution not captured by standard query language coding. Manual review of echocardiogram parameters and clinic notes yielded a sensitivity of 75%, specificity of 98%, and positive predictive value of 68% for International Classification of Disease coding of heart failure. For patients with anthracycline exposure, 78.5% (n = 62) were adherent to guideline recommendations for echocardiogram surveillance. There were significant association with provider adherence and race and ethnicity (P = .047), and 50% of patients with Spanish as their primary language were adherent compared with 90% of patients with English as their primary language (P = .003).

CONCLUSION

Extraction of treatment exposures from the electronic health record through clinical informatics and integration with cancer registry data represents a feasible approach to assess cardiovascular disease outcomes and adherence to guideline recommendations for survivors.

The OpenDeID corpus for patient de-identification

For research purposes, protected health information is often redacted from unstructured electronic health records to preserve patient privacy and confidentiality. The OpenDeID corpus is designed to assist development of automatic methods to redact sensitive information from unstructured electronic health records. We retrieved 4548 unstructured surgical pathology reports from four urban Australian hospitals. The corpus was developed by two annotators under three different experimental settings. The quality of the annotations was evaluated for each setting. Specifically, we employed serial annotations, parallel annotations, and pre-annotations. Our results suggest that the pre-annotations approach is not reliable in terms of quality when compared to the serial annotations but can drastically reduce annotation time. The OpenDeID corpus comprises 2,100 pathology reports from 1,833 cancer patients with an average of 737.49 tokens and 7.35 protected health information entities annotated per report. The overall inter annotator agreement and deviation scores are 0.9464 and 0.9726, respectively. Realistic surrogates are also generated to make the corpus suitable for distribution to other researchers.

Improving Cancer Data Interoperability: The Promise of the Minimal Common Oncology Data Elements (mCODE) Initiative

PURPOSE

Because of expanding interoperability requirements, structured patient data are increasingly available in electronic health records. Many oncology data elements (eg, staging, biomarkers, documentation of adverse events and cancer outcomes) remain challenging. The Minimal Common Oncology Data Elements (mCODE) project is a consensus data standard created to facilitate transmission of data of patients with cancer.

METHODS

In 2018, mCODE was developed through a work group convened by ASCO, including oncologists, informaticians, researchers, and experts in terminologies and standards. The mCODE specification is organized by 6 high-level domains: patient, laboratory/vital, disease, genomics, treatment, and outcome. In total, 23 mCODE profiles are composed of 90 data elements.

RESULTS

A conceptual model was published for public comment in January 2019 and, after additional refinement, the first public version of the mCODE (version 0.9.1) Fast Healthcare Interoperability Resources (FHIR) implementation guide (IG) was presented at the ASCO Annual Meeting in June 2019. The specification was approved for balloting by Health Level 7 International (HL7) in August 2019. mCODE passed the HL7 ballot in September 2019 with 86.5% approval. The mCODE IG authors worked with HL7 reviewers to resolve all negative comments, leading to a modest expansion in the number of data elements and tighter alignment with FHIR and other HL7 conventions. The mCODE version 1.0 FHIR IG Standard for Trial Use was formally published on March 18, 2020.

CONCLUSION

The mCODE project has the potential to offer tremendous benefits to cancer care delivery and research by creating an infrastructure to better share patient data. mCODE is available free from www.mCODEinitiative.org. Pilot implementations are underway, and a robust community of stakeholders has been assembled across the oncology ecosystem.

Correlation Between Surrogate End Points and Overall Survival in a Multi-institutional Clinicogenomic Cohort of Patients With Non-Small Cell Lung or Colorectal Cancer

IMPORTANCE

Contemporary observational cancer research requires associating genomic biomarkers with reproducible end points; overall survival (OS) is a key end point, but interpretation can be challenging when multiple lines of therapy and prolonged survival are common. Progression-free survival (PFS), time to treatment discontinuation (TTD), and time to next treatment (TTNT) are alternative end points, but their utility as surrogates for OS in real-world clinicogenomic data sets has not been well characterized.

OBJECTIVE

To measure correlations between candidate surrogate end points and OS in a multi-institutional clinicogenomic data set.

DESIGN, SETTING, AND PARTICIPANTS

A retrospective cohort study was conducted of patients with non-small cell lung cancer (NSCLC) or colorectal cancer (CRC) whose tumors were genotyped at 4 academic centers from January 1, 2014, to December 31, 2017, and who initiated systemic therapy for advanced disease. Patients were followed up through August 31, 2020 (NSCLC), and October 31, 2020 (CRC). Statistical analyses were conducted on January 5, 2021.

EXPOSURES

Candidate surrogate end points included TTD; TTNT; PFS based on imaging reports only; PFS based on medical oncologist ascertainment only; PFS based on either imaging or medical oncologist ascertainment, whichever came first; and PFS defined by a requirement that both imaging and medical oncologist ascertainment have indicated progression.

MAIN OUTCOMES AND MEASURES

The primary outcome was the correlation between candidate surrogate end points and OS.

RESULTS

There were 1161 patients with NSCLC (648 women [55.8%]; mean [SD] age, 63 [11] years) and 1150 with CRC (647 men [56.3%]; mean [SD] age, 54 [12] years) identified for analysis. Progression-free survival based on both imaging and medical oncologist documentation was most correlated with OS (NSCLC: ρ = 0.76; 95% CI, 0.73-0.79; CRC: ρ = 0.73; 95% CI, 0.69-0.75). Time to treatment discontinuation was least associated with OS (NSCLC: ρ = 0.45; 95% CI, 0.40-0.50; CRC: ρ = 0.13; 95% CI, 0.06-0.19). Time to next treatment was modestly associated with OS (NSCLC: ρ = 0.60; 0.55-0.64; CRC: ρ = 0.39; 95% CI, 0.32-0.46).

CONCLUSIONS AND RELEVANCE

This cohort study suggests that PFS based on both a radiologist and a treating oncologist determining that a progression event has occurred was the surrogate end point most highly correlated with OS for analysis of observational clinicogenomic data.

The Prospective Dutch Colorectal Cancer (PLCRC) cohort: real-world data facilitating research and clinical care

Real-world data (RWD) sources are important to advance clinical oncology research and evaluate treatments in daily practice. Since 2013, the Prospective Dutch Colorectal Cancer (PLCRC) cohort, linked to the Netherlands Cancer Registry, serves as an infrastructure for scientific research collecting additional patient-reported outcomes (PRO) and biospecimens. Here we report on cohort developments and investigate to what extent PLCRC reflects the “real-world”. Clinical and demographic characteristics of PLCRC participants were compared with the general Dutch CRC population (n = 74,692, Dutch-ref). To study representativeness, standardized differences between PLCRC and Dutch-ref were calculated, and logistic regression models were evaluated on their ability to distinguish cohort participants from the Dutch-ref (AU-ROC 0.5 = preferred, implying participation independent of patient characteristics). Stratified analyses by stage and time-period (2013–2016 and 2017–Aug 2019) were performed to study the evolution towards RWD. In August 2019, 5744 patients were enrolled. Enrollment increased steeply, from 129 participants (1 hospital) in 2013 to 2136 (50 of 75 Dutch hospitals) in 2018. Low AU-ROC (0.65, 95% CI: 0.64–0.65) indicates limited ability to distinguish cohort participants from the Dutch-ref. Characteristics that remained imbalanced in the period 2017–Aug’19 compared with the Dutch-ref were age (65.0 years in PLCRC, 69.3 in the Dutch-ref) and tumor stage (40% stage-III in PLCRC, 30% in the Dutch-ref). PLCRC approaches to represent the Dutch CRC population and will ultimately meet the current demand for high-quality RWD. Efforts are ongoing to improve multidisciplinary recruitment which will further enhance PLCRC’s representativeness and its contribution to a learning healthcare system.