Predictive model algorithms identifying early and advanced stage ER+/HER2- breast cancer in claims data

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

OBJECTIVES

In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices.

RESULTS

We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion.

CONCLUSION

The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

Validity of claims data for identifying cancer incidence in the Japan public health center-based prospective study for the next generation

PURPOSE

This study determined the validity of claims-based definitions for identifying the incidence of total and site-specific cancers in a population-based cohort study.

METHODS

Claims data were obtained for 21 946 participants aged 40-74 years enrolled in the Japan Public Health Center-based Prospective Study for the Next Generation. We defined total and site-specific cancer incidence using combinations of codes from claims data, including diagnosis and procedure codes for cancer therapy. Data from the cancer registry were used as the gold standard to evaluate validity.

RESULTS

Among 21 946 participants, 454 total, 89 stomach, 67 colorectal, 51 lung, 39 breast and 99 prostate invasive cancer cases were newly diagnosed in the cancer registry. For invasive cancer, the sensitivity and specificity of the definition that combined codes for diagnosis and procedures for cancer therapy were 87.0% and 99.4% for total, 88.8% and 99.9% for stomach, 80.6% and 99.9% for colorectal, 86.3% and 99.9% for lung, 100% and 99.9% for breast and 91.9% and 99.9% for prostate cancer, respectively. Furthermore, for invasive and/or in situ cancer, the sensitivity and specificity of the definition were 84.5% and 99.5% for total, 66.7% and 99.9% for colorectal and 100% and 99.9% for breast cancer.

CONCLUSIONS

Our findings suggest that claims-based definitions using diagnosis and procedure codes generally have high validity for total, stomach, lung, breast and prostate cancer incidence, but may underestimate colorectal cancer incidence.

Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review

BACKGROUND

Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology.

METHODS

We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models.

RESULTS

Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available.

CONCLUSIONS

The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.

The emerging role of real-world data in advanced breast cancer therapy: Recommendations for collaborative decision-making

Among stakeholders and decision-makers in advanced breast cancer, the demand for insights from real-world data (RWD) is increasing. Although RWD can be used to support decisions throughout different stages of a breast cancer drug's life cycle, barriers exist to its use and acceptance. We propose a collaborative approach to generating and using RWD that is meaningful to multiple stakeholders, and encourage frameworks toward international guidelines to help standardize RWD methodologies to achieve more efficient use of RWD insights.

The Structured Process to Identify Fit-for-purpose Data (SPIFD): A data feasibility assessment framework

To complement real‐world evidence (RWE) guidelines, the 2019 Structured Preapproval and Postapproval Comparative study design framework to generate valid and transparent real‐world Evidence (SPACE) framework elucidated a process for designing valid and transparent real‐world studies. As an extension to SPACE, here, we provide a structured framework for conducting feasibility assessments—a step‐by‐step guide to identify decision grade, fit‐for‐purpose data, which complements the United States Food and Drug Administration (FDA)’s framework for a RWE program. The process was informed by our collective experience conducting systematic feasibility assessments of existing data sources for pharmacoepidemiology studies to support regulatory decisions. Used with the SPACE framework, the Structured Process to Identify Fit‐For‐Purpose Data (SPIFD) provides a systematic process for conducting feasibility assessments to determine if a data source is fit for decision making, helping ensure justification and transparency throughout study development, from articulation of a specific and meaningful research question to identification of fit‐for‐purpose data and study design.

Development and validation of a predictive model algorithm to identify anaphylaxis in adults with type 2 diabetes in U.S. administrative claims data

PURPOSE

To use medical record adjudication and predictive modeling methods to develop and validate an algorithm to identify anaphylaxis among adults with type 2 diabetes (T2D) in administrative claims.

METHODS

A conventional screening algorithm that prioritized sensitivity to identify potential anaphylaxis cases was developed and consisted of diagnosis codes for anaphylaxis or relevant signs and symptoms. This algorithm was applied to adults with T2D in the HealthCore Integrated Research Database (HIRD) from 2016 to 2018. Clinical experts adjudicated anaphylaxis case status from redacted medical records. We used confirmed case status as an outcome for predictive models developed using lasso regression with 10-fold cross-validation to identify predictors and estimate the probability of confirmed anaphylaxis.

RESULTS

Clinical adjudicators reviewed medical records with sufficient information from 272 adults identified by the anaphylaxis screening algorithm, which had an estimated Positive Predictive Value (PPV) of 65% (95% confidence interval [CI]: 60%-71%). The predictive model algorithm had a c-statistic of 0.95. The model's probability threshold of 0.60 excluded 89% (84/94) of false positives identified by the screening algorithm, with a PPV of 94% (95% CI: 91%-98%). The model excluded very few true positives (15 of 178), and identified 92% (95% CI: 87%-96%) of the cases selected by the screening algorithm.

CONCLUSIONS

Predictive modeling techniques yielded an accurate algorithm with high PPV and sensitivity for identifying anaphylaxis in administrative claims. This algorithm could be considered in future safety studies using similar claims data to reduce potential outcome misclassification.

Data Integration to Improve Real-world Health Outcomes Research for Non-Small Cell Lung Cancer in the United States: Descriptive and Qualitative Exploration

Background

The integration of data from disparate sources could help alleviate data insufficiency in real-world studies and compensate for the inadequacies of single data sources and short-duration, small sample size studies while improving the utility of data for research.

Objective

This study aims to describe and evaluate a process of integrating data from several complementary sources to conduct health outcomes research in patients with non–small cell lung cancer (NSCLC). The integrated data set is also used to describe patient demographics, clinical characteristics, treatment patterns, and mortality rates.

Methods

This retrospective cohort study integrated data from 4 sources: administrative claims from the HealthCore Integrated Research Database, clinical data from a Cancer Care Quality Program (CCQP), clinical data from abstracted medical records (MRs), and mortality data from the US Social Security Administration. Patients with lung cancer who initiated second-line (2L) therapy between November 01, 2015, and April 13, 2018, were identified in the claims and CCQP data. Eligible patients were 18 years or older and received atezolizumab, docetaxel, erlotinib, nivolumab, pembrolizumab, pemetrexed, or ramucirumab in the 2L setting. The main analysis cohort included patients with claims data and data from at least one additional data source (CCQP or MR). Patients without integrated data (claims only) were reported separately. Descriptive and univariate statistics were reported.

Results

Data integration resulted in a main analysis cohort of 2195 patients with NSCLC; 2106 patients had CCQP and 407 patients had MR data. The claims-only cohort included 931 eligible patients. For the main analysis cohort, the mean age was 62.1 (SD 9.27) years, 48.56% (1066/2195) were female, the median length of follow-up was 6.8 months, and for 37.77% (829/2195), death was observed. For the claims-only cohort, the mean age was 66.6 (SD 12.69) years, 52.1% (485/931) were female, the median length of follow-up was 8.6 months, and for 29.3% (273/931), death was observed. The most frequent 2L treatment was immunotherapy (1094/2195, 49.84%), followed by platinum-based regimens (472/2195, 21.50%) and single-agent chemotherapy (441/2195, 20.09%); mean duration of 2L therapy was 5.6 (SD 4.9, median 4) months. We describe challenges and learnings from the data integration process, and the benefits of the integrated data set, which includes a richer set of clinical and outcome data to supplement the utilization metrics available in administrative claims.

Conclusions

The management of patients with NSCLC requires care from a multidisciplinary team, leading to a lack of a single aggregated data source in real-world settings. The availability of integrated clinical data from MRs, health plan claims, and other sources of clinical care may improve the ability to assess emerging treatments.

Real-world safety of palbociclib in breast cancer patients in the United States: a new user cohort study

Background

There is limited real-world safety information on palbociclib for treatment of advanced stage HR+/HER2- breast cancer.

Methods

We conducted a cohort study of breast cancer patients initiating palbociclib and fulvestrant from February 2015 to September 2017 using the HealthCore Integrated Research Database (HIRD), a longitudinal claims database of commercial health plan members in the United States.

The historical comparator cohort comprised patients initiating fulvestrant monotherapy from January 2011 to January 2015. Propensity score matching and Cox regression were used to estimate hazard ratios for various safety events. For acute liver injury (ALI), additional analyses and medical record validation were conducted.

Results

There were 2445 patients who initiated palbociclib including 566 new users of palbociclib-fulvestrant, and 2316 historical new users of fulvestrant monotherapy. Compared to these historical new users of fulvestrant monotherapy, new users of palbociclib-fulvestrant had a greater than 2-fold elevated risk for neutropenia, leukopenia, thrombocytopenia, stomatitis and mucositis, and ALI. Incidence of anemia and QT prolongation were more weakly associated, and incidences of serious infections and pulmonary embolism were similar between groups after propensity score matching. After adjustment for additional ALI risk factors, the elevated risk of ALI in new users of palbociclib-fulvestrant persisted (e.g. primary ALI algorithm hazard ratio (HR) = 3.0, 95% confidence interval (CI) = 1.1–8.4).

Conclusions

This real-world study found increased risks of several adverse events identified in clinical trials, including neutropenia, leukopenia, and thrombocytopenia, but no increased risk of serious infections or pulmonary embolism when comparing new users of palbociclib-fulvestrant to fulvestrant monotherapy. We observed an increased risk of ALI, extending clinical trial findings of significant imbalances in grade 3/4 elevations of alanine aminotransferase (ALT).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-021-07790-z.

Trial designs using real-world data: The changing landscape of the regulatory approval process

Purpose

There is a need to develop hybrid trial methodology combining the best parts of traditional randomized controlled trials (RCTs) and observational study designs to produce real‐world evidence (RWE) that provides adequate scientific evidence for regulatory decision‐making.

Methods

This review explores how hybrid study designs that include features of RCTs and studies with real‐world data (RWD) can combine the advantages of both to generate RWE that is fit for regulatory purposes.

Results

Some hybrid designs include randomization and use pragmatic outcomes; other designs use single‐arm trial data supplemented with external comparators derived from RWD or leverage novel data collection approaches to capture long‐term outcomes in a real‐world setting. Some of these approaches have already been successfully used in regulatory decisions, raising the possibility that studies using RWD could increasingly be used to augment or replace traditional RCTs for the demonstration of drug effectiveness in certain contexts. These changes come against a background of long reliance on RCTs for regulatory decision‐making, which are labor‐intensive, costly, and produce data that can have limited applicability in real‐world clinical practice.

Conclusions

While RWE from observational studies is well accepted for satisfying postapproval safety monitoring requirements, it has not commonly been used to demonstrate drug effectiveness for regulatory purposes. However, this position is changing as regulatory opinions, guidance frameworks, and RWD methodologies are evolving, with growing recognition of the value of using RWE that is acceptable for regulatory decision‐making.