Comparative Effectiveness Research Methods: Symposium Overview and Summary

Agreement in the CARTaGENE cohort between self-reported medication use and claim data

OBJECTIVES

To describe the agreement of self-reported medication use with claim prescription records and to ascertain factors associated with agreement between the two data sources.

METHODS

Baseline data on self-reported medication use was extracted from CARTaGENE, a cohort study in Quebec, Canada, and from the provincial health insurance records (dispensation database) of the same individuals. Kappa statistics were used to estimate concordance beyond chance between the two data sources. Logistic regression models were adjusted to estimate the association between agreement and selected individual's characteristics (sex, age, education, region, income, utilization of health care system, and comorbidities).

RESULTS

Agreement between self-reported medication use and administrative data varied considerably across medication classes (kappa 0.54 for respiratory system and 0.91 for systemic hormonal preparations). Overall, agreement improved when a fixed time window of 90 days was used for exposure measurement. Sex, education level, frequency of health care use and the number of reported medications were associated with agreement.

DISCUSSION

Overall, there was a reasonable agreement between the two data sources, but important variations were found for the different drug classes. These results could be used by researchers to more accurately assess drug exposures using real-world data, which are increasingly important to regulators.

Capturing patient-reported outcomes: paper versus electronic survey administration

Objective: To compare the capture rates and costs of paper patient-reported outcomes (pPRO) administered in-clinic and electronic PROs (ePRO) collected through emails and texts. Design: Retrospective review. Setting: Level 1 trauma center. Patients/Participants: The pPRO program enrolled 2164 patients for postsurgical follow-up in 4 fracture types: ankle, distal radius, proximal humerus, and implant removal from 2012 to 2017. The ePRO program enrolled 3096 patients in 13 fracture types from 2018 to 2020. Among the patients enrolled in the ePRO program, 1296 patients were matched to the 4 original fracture types and time points. Main Outcome Measures: PRO capture rates in 4 fracture types by matched time point and estimated cost of each program per enrolled patient. Results: At first follow-up, pPRO provided a higher capture rate than ePRO for 3 of 4 fracture types except for implant removal (P < 0.05). However, at 6-month and 1-year follow-ups, ePRO demonstrated statistically significant higher capture rates when compared with pPRO for all applicable modules (P < 0.05). The average cost for the pPRO program was $171 per patient versus $56 per patient in the ePRO program. Patients were 1.19 times more likely to complete ePRO compared with pPRO (P = 0.007) after controlling for age, sex, fracture type, and time point. Conclusion: The electronic PRO service has improved long-term capture rates compared with paper PROs, while minimizing cost. A combined program that includes both in-clinic and out of clinic effort may be the ideal model for collection of PROs. Level of Evidence: Level 3.

Comparative effectiveness of intracranial hypertension management guided by ventricular versus intraparenchymal pressure monitoring: a CENTER-TBI study

OBJECTIVE

To compare outcomes between patients with primary external ventricular device (EVD)-driven treatment of intracranial hypertension and those with primary intraparenchymal monitor (IP)-driven treatment.

METHODS

The CENTER-TBI study is a prospective, multicenter, longitudinal observational cohort study that enrolled patients of all TBI severities from 62 participating centers (mainly level I trauma centers) across Europe between 2015 and 2017. Functional outcome was assessed at 6 months and a year. We used multivariable adjusted instrumental variable (IV) analysis with "center" as instrument and logistic regression with covariate adjustment to determine the effect estimate of EVD on 6-month functional outcome.

RESULTS

A total of 878 patients of all TBI severities with an indication for intracranial pressure (ICP) monitoring were included in the present study, of whom 739 (84%) patients had an IP monitor and 139 (16%) an EVD. Patients included were predominantly male (74% in the IP monitor and 76% in the EVD group), with a median age of 46 years in the IP group and 48 in the EVD group. Six-month GOS-E was similar between IP and EVD patients (adjusted odds ratio (aOR) and 95% confidence interval [CI] OR 0.74 and 95% CI [0.36-1.52], adjusted IV analysis). The length of intensive care unit stay was greater in the EVD group than in the IP group (adjusted rate ratio [95% CI] 1.70 [1.34-2.12], IV analysis). One hundred eighty-seven of the 739 patients in the IP group (25%) required an EVD due to refractory ICPs.

CONCLUSION

We found no major differences in outcomes of patients with TBI when comparing EVD-guided and IP monitor-guided ICP management. In our cohort, a quarter of patients that initially received an IP monitor required an EVD later for ICP control. The prevalence of complications was higher in the EVD group.

PROTOCOL

The core study is registered with ClinicalTrials.gov , number NCT02210221, and the Resource Identification Portal (RRID: SCR_015582).

Towards Guideline-Based Management of Tethered Cord Syndrome in Spina Bifida: A Global Health Paradigm Shift in the Era of Prenatal Surgery

An estimated 60% of the world's population lives in Asia, where the incidence of neural tube defects is high. Aware that tethered cord syndrome (TCS) is an important comorbidity, the purpose of this systematic review was to explore the treatment of TCS among individuals living with spina bifida (SB) in Asia. MEDLINE and Embase databases were searched for relevant studies published from January 2000 to June 2018. Search terms such as 'spinal dysraphism,' 'spinabifida,' 'diastematomyelia,' 'lipomeningocele,' 'lypomyelomeningocele,' 'meningomyelocele,' and 'tethered cord syndrome' were used in diverse combinations. Of the 1,290 articles that were identified in accordance with PRISMA (Preferred Items for Systematic Reviews and Meta-Analyses) guidelines, 15 Asia-based studies met the inclusion criteria. Significant differences in the diagnostic criteria and management of TCS were documented. As the surgical techniques for prenatal closure of the spinal defect continue to evolve, their adoption internationally is likely to continue. In this setting, a clear and evidence-based approach to the definition and management of TCS is essential. The recent publication by the Spina Bifida Association of America of their updated care guidelines may serve as a tool used to promote a systematized approach to diagnosing and treating TCS among individuals with SB in the region, as well as globally.

The Glomerular Disease Study and Trial Consortium: A Grassroots Initiative to Foster Collaboration and Innovation

Glomerular kidney disorders account for a significant proportion of chronic kidney disease and end-stage renal disease worldwide. Nevertheless, major obstacles make breakthrough progress in diagnosis and cure an ongoing challenge. Here we report the creation of a "grassroots" initiative that aims to provide new opportunities for nephrologists, pathologists, basic and clinical scientists, patients, and industry partners to collaborate in the field of glomerular kidney disease. Members of the medical community, including trainees, nephrologists, and nephropathologists, can participate in the open-access, Web-based, multidisciplinary clinical video case conferences, which provide "peer-to-peer" exchange of clinical and pathological expertise combined with a formal didactic curriculum. Participants can also join other aspects of the broader initiative. These include the participation in a multisite research study to facilitate enrollment of patients into a longitudinal clinical data and biorepository for glomerular kidney disorders. Items included in this prospective registry include the following: an ontology-based patient medical history, which is regularly updated; interval collection and storage of blood and urine samples; DNA collection; and a contact registry for patients who wish to participate in clinical trials. Participating sites and external scientists can leverage access to the database to pursue genetic, biomarker, epidemiological, and observational clinical effectiveness studies. Patients can independently sign up for a supplementary contact registry to participate in clinical trials if eligible. The broad spectrum of activities within this initiative will foster closer collaboration among trainees, practicing nephrologists, pathologists, and researchers, and may help to overcome some of the barriers to progress in the field of glomerular kidney disease.

Progress in evidence-based medicine: a quarter century on

In response to limitations in the understanding and use of published evidence, evidence-based medicine (EBM) began as a movement in the early 1990s. EBM's initial focus was on educating clinicians in the understanding and use of published literature to optimise clinical care, including the science of systematic reviews. EBM progressed to recognise limitations of evidence alone, and has increasingly stressed the need to combine critical appraisal of the evidence with patient's values and preferences through shared decision making. In another progress, EBM incorporated and further developed the science of producing trustworthy clinical practice guidelines pioneered by investigators in the 1980s. EBM's enduring contributions to clinical medicine include placing the practice of medicine on a solid scientific basis, the development of more sophisticated hierarchies of evidence, the recognition of the crucial role of patient values and preferences in clinical decision making, and the development of the methodology for generating trustworthy recommendations.

Regression-Based Approaches to Patient-Centered Cost-Effectiveness Analysis

Achieving comprehensive patient centricity in cost-effectiveness analyses (CEAs) requires a statistical approach that accounts for patients' preferences and clinical and demographic characteristics. Increased availability and accessibility of patient-level health-related utility data from clinical trials or observational database provide enhanced opportunities to conduct more patient-centered CEA. Regression-based approaches that incorporate patient-level data hold great promise for enhancing CEAs to be more patient centered; this paper provides guidance regarding two CEA approaches that apply regression-based approaches utilizing patient-level health-related utility and costs data. The first approach utilizes patient-reported preferences to determine patient-specific utility. This approach evaluates how individuals' unique clinical and demographic factors affect their utility and cost levels over the course of treatment. The underlying motivation of this approach is to produce CEA estimates that reflect patient-level utilities and costs while adjusting for socio-demographic and clinical factors to aid patient-centered coverage and treatment decision-making. In the second approach, patient utilities are estimated based on the clinically defined health states through which a patient may transition throughout the course of treatment. While this approach is grounded on the widely used Markov transition model, we refine the model to facilitate an enhancement in conducting regression-based analysis to achieve transparent understanding of differences in utilities and costs across diverse patient populations. We discuss the unique statistical challenges of each approach and describe how these analytical strategies are related to non-regression-based models in health services research.

Using a population-based observational cohort study to address difficult comparative effectiveness research questions: the CEASAR study

BACKGROUND

While randomized controlled trials represent the highest level of evidence we can generate in comparative effectiveness research, there are clinical scenarios where this type of study design is not feasible. The Comparative Effectiveness Analyses of Surgery and Radiation in localized prostate cancer (CEASAR) study is an observational study designed to compare the effectiveness and harms of different treatments for localized prostate cancer, a clinical scenario in which randomized controlled trials have been difficult to execute and, when completed, have been difficult to generalize to the population at large.

METHODS

CEASAR employs a population-based, prospective cohort study design, using tumor registries as cohort inception tools. The primary outcome is quality of life after treatment, measured by validated instruments. Risk adjustment is facilitated by capture of traditional and nontraditional confounders before treatment and by propensity score analysis.

RESULTS

We have accrued a diverse, representative cohort of 3691 men in the USA with clinically localized prostate cancer. Half of the men invited to participate enrolled, and 86% of patients who enrolled have completed the 6-month survey.

CONCLUSION

Challenging comparative effectiveness research questions can be addressed using well-designed observational studies. The CEASAR study provides an opportunity to determine what treatments work best, for which patients, and in whose hands.

Barriers to completion of Patient Reported Outcome Measures

Patient Reported Outcomes Measures (PROMs) are commonly used in total joint arthroplasty (TJA) to assess surgical outcomes. However certain patient populations may be underrepresented due to lower survey completion rates. The purpose of this study is to evaluate factors that influence PROM completion rates for 1997 TJA patients between 7/1/2007 and 12/31/2010. Completion rates were lower among patients who were over 75, Hispanic or Black, had Medicare or Medicaid, TKA patients and revision TJA patients (P<0.05 for all comparisons). Having multiple risk factors further reduced completion rates (P<0.001). Overall participation increased significantly during the study period, after electronic data capture methods were introduced. Awareness of these factors may help physicians and researchers improve participation of all patient populations so they are well represented in TJA outcomes research.

Comparisons among Health Behavior Surveys: Implications for the Design of Informatics Infrastructures That Support Comparative Effectiveness Research

Introduction:

To address the electronic health data fragmentation that is a methodological limitation of comparative effectiveness research (CER), the Washington Heights Inwood Informatics Infrastructure for Comparative Effectiveness Research (WICER) project is creating a patient-centered research data warehouse (RDW) by linking electronic clinical data (ECD) from New York Presbyterian Hospital’s clinical data warehouse with ECD from ambulatory care, long-term care, and home health settings and the WICER community health survey (CHS). The purposes of the research were to identify areas of overlap between the WICER CHS and two other surveys that include health behavior data (the Behavioral Risk Factor Surveillance System (BRFSS) Survey and the New York City Community Health Survey (NYC CHS)) and to identify gaps in the current WICER RDW that have the potential to affect patient-centered CER.

Methods:

We compared items across the three surveys at the item and conceptual levels. We also compared WICER RDW (ECD and WICER CHS), BRFSS, and NYC CHS to the County Health Ranking framework.

Results:

We found that 22 percent of WICER items were exact matches with BRFSS and that there were no exact matches between WICER CHS and NYC CHS items not also contained in BRFSS.

Conclusions:

The results suggest that BRFSS and, to a lesser extent, NYC CHS have the potential to serve as population comparisons for WICER CHS for some health behavior-related data and thus may be particularly useful for considering the generalizability of CER study findings. Except for one measure related to health behavior (motor vehicle crash deaths), the WICER RDW’s comprehensive coverage supports the mortality, morbidity, and clinical care measures specified in the County Health Ranking framework but is deficient in terms of some socioeconomic factors and descriptions of the physical environment as captured in BRFSS. Linkage of these data in the WICER RDW through geocoding can potentially facilitate patient-centered CER that integrates important socioeconomic and physical environment influences on health outcomes. The research methods and findings may be relevant to others interested in either integrating health behavior data into RDWs to support patient-centered CER or conducting population-level comparisons.

Comparative effectiveness research: nursing science and health care delivery

Comparative effectiveness research (CER) is an important foundation in the development of scientific evidence that can assist patients, clinicians, and policy makers in making decisions that improve patient and system outcomes, including cost. CER is a part of what is now called dissemination and implementation science, which focuses on translating knowledge into practice by facilitating stakeholder access to more interpretable findings. CER has evolved from a rich history that aims to compare the effectiveness of select clinical treatments, approaches, or programs. This article describes the development of CER, approaches to designing and analyzing this research, and resources useful in generating such knowledge by nurse scientists.

A framework for understanding cancer comparative effectiveness research data needs

OBJECTIVES

Randomized controlled trials remain the gold standard for evaluating cancer intervention efficacy. Randomized trials are not always feasible, practical, or timely and often don't adequately reflect patient heterogeneity and real-world clinical practice. Comparative effectiveness research can leverage secondary data to help fill knowledge gaps randomized trials leave unaddressed; however, comparative effectiveness research also faces shortcomings. The goal of this project was to develop a new model and inform an evolving framework articulating cancer comparative effectiveness research data needs.

STUDY DESIGN AND SETTING

We examined prevalent models and conducted semi-structured discussions with 76 clinicians and comparative effectiveness research researchers affiliated with the Agency for Healthcare Research and Quality's cancer comparative effectiveness research programs.

RESULTS

A new model was iteratively developed and presents cancer comparative effectiveness research and important measures in a patient-centered, longitudinal chronic care model better reflecting contemporary cancer care in the context of the cancer care continuum, rather than a single-episode, acute-care perspective.

CONCLUSION

Immediately relevant for federally funded comparative effectiveness research programs, the model informs an evolving framework articulating cancer comparative effectiveness research data needs, including evolutionary enhancements to registries and epidemiologic research data systems. We discuss elements of contemporary clinical practice, methodology improvements, and related needs affecting comparative effectiveness research's ability to yield findings clinicians, policy makers, and stakeholders can confidently act on.

Using meta-analyses for comparative effectiveness research

Comparative effectiveness research seeks to identify the most effective interventions for particular patient populations. Meta-analysis is an especially valuable form of comparative effectiveness research because it emphasizes the magnitude of intervention effects rather than relying on tests of statistical significance among primary studies. Overall effects can be calculated for diverse clinical and patient-centered variables to determine the outcome patterns. Moderator analyses compare intervention characteristics among primary studies by determining whether effect sizes vary among studies with different intervention characteristics. Intervention effectiveness can be linked to patient characteristics to provide evidence for patient-centered care. Moderator analyses often answer questions never posed by primary studies because neither multiple intervention characteristics nor populations are compared in single primary studies. Thus, meta-analyses provide unique contributions to knowledge. Although meta-analysis is a powerful comparative effectiveness strategy, methodological challenges and limitations in primary research must be acknowledged to interpret findings.

Effectiveness of radiation for prevention of mastectomy in older breast cancer patients treated with conservative surgery

BACKGROUND

A recent clinical trial concluded that radiation therapy (RT) does not lower the risk of mastectomy and, thus, may be omitted in older women with stage I, estrogen receptor (ER)-positive breast cancer who undergo conservative surgery (CS). However, it is not known whether this finding applies to patients outside of clinical trials. Accordingly, we used the Surveillance, Epidemiology, and End Results-Medicare observational cohort to determine the effect of RT on the risk of mastectomy among older women with stage I, ER-positive breast cancer.

METHODS

The authors identified 7403 women ages 70 to 79 years who underwent CS between 1992 and 2002. Claims were used to determine RT status and to identify women who underwent mastectomy subsequent to initial treatment. The Kaplan-Meier method was used to estimate the risk of subsequent mastectomy, and Cox regression analysis was used to determine the effect of RT adjusted for clinical-pathologic covariates.

RESULTS

At a median follow-up of 7.3 years, the risk of subsequent mastectomy within 10 years of diagnosis was 3.2% for patients who received RT versus 6.3% for patients who did not receive RT (P < .001). In adjusted analyses, RT was associated with a lower risk of mastectomy (hazard ratio, 0.33; 95% confidence interval, 0.22-0.48; P < .001). RT provided no benefit for patients ages 75 to 79 years without high-grade tumors who had a pathologic lymph node assessment (P = .80); however, for all other subgroups, RT was associated with an absolute reduction in risk of mastectomy that ranged from 4.3% to 9.8% at 10 years.

CONCLUSIONS

Outside of a clinical trial, the receipt of RT after CS was associated with a greater likelihood of ultimate breast preservation for most older women with early breast cancer.

Reducing bias in the assessment of treatment effectiveness: androgen deprivation therapy for prostate cancer

BACKGROUND

Indication bias is the major challenge in assessing treatment effectiveness in observational studies. We explored the potential advantages of using an instrumental variable approach in the context of primary androgen deprivation therapy (ADT) for prostate cancer.

METHODS

We identified 31,930 men in the linked Surveillance, Epidemiology, and End Results-Medicare database with a diagnosis of prostate cancer who were not treated definitively with radical prostatectomy or radiation in the years 1992 through 2002, with follow-up through 2005. The association between use of primary ADT and overall, prostate cancer-specific, and nonprostate cancer survival was assessed using multivariable regression and instrumental variable methods. Two instrumental variables, based on region and urologist prescribing preference, were constructed and analyzed using exogenous probit models. Prespecified subgroup analyses in patients with lower-risk and higher-risk prostate tumors were also carried out.

RESULTS

In the overall cohort, standard multivariable regression analyses showed a significantly increased risk of prostate cancer-related death, whereas the instrumental variable approaches showed a protective effect of primary ADT, which was significant for the urologist preference instrument (hazard ratio: 0.74; 95% confidence interval, 0.60-0.93). In the high-risk subgroup, using urologist preference for primary ADT as the instrument, there was a significant reduction in overall mortality (hazard ratio: 0.75; 95% confidence interval, 0.57-0.99), driven by a large reduction in prostate cancer-specific mortality.

CONCLUSIONS

Instrumental variable analysis appears to provide better control of bias when assessing the effectiveness of primary ADT for prostate cancer, although the results may be more applicable to policy rather than to clinical decisions.

Comparative effectiveness research for prostate cancer radiation therapy: current status and future directions

Comparative effectiveness research aims to help clinicians, patients and policymakers make informed treatment decisions under real-world conditions. Prostate cancer patients have multiple treatment options, including active surveillance, androgen deprivation therapy, surgery and multiple modalities of radiation therapy. Technological innovations in radiation therapy for prostate cancer have been rapidly adopted into clinical practice despite relatively limited evidence for effectiveness showing the benefit for one modality over another. Comparative effectiveness research has become an essential component of prostate cancer research to help define the benefits, risks and effectiveness of the different radiation therapy modalities currently in use for prostate cancer treatment.

Comparative effectiveness research on patients with acute ischemic stroke using Markov decision processes

Background

Several methodological issues with non-randomized comparative clinical studies have been raised, one of which is whether the methods used can adequately identify uncertainties that evolve dynamically with time in real-world systems. The objective of this study is to compare the effectiveness of different combinations of Traditional Chinese Medicine (TCM) treatments and combinations of TCM and Western medicine interventions in patients with acute ischemic stroke (AIS) by using Markov decision process (MDP) theory. MDP theory appears to be a promising new method for use in comparative effectiveness research.

Methods

The electronic health records (EHR) of patients with AIS hospitalized at the 2^nd Affiliated Hospital of Guangzhou University of Chinese Medicine between May 2005 and July 2008 were collected. Each record was portioned into two "state-action-reward" stages divided by three time points: the first, third, and last day of hospital stay. We used the well-developed optimality technique in MDP theory with the finite horizon criterion to make the dynamic comparison of different treatment combinations.

Results

A total of 1504 records with a primary diagnosis of AIS were identified. Only states with more than 10 (including 10) patients' information were included, which gave 960 records to be enrolled in the MDP model. Optimal combinations were obtained for 30 types of patient condition.

Conclusion

MDP theory makes it possible to dynamically compare the effectiveness of different combinations of treatments. However, the optimal interventions obtained by the MDP theory here require further validation in clinical practice. Further exploratory studies with MDP theory in other areas in which complex interventions are common would be worthwhile.

Perioperative comparative effectiveness research

The goal of comparative effectiveness research (CER) is to improve effectiveness, efficacy and efficiency in health care. While CER seems to present a major opportunity to introduce accountability into health care by identifying and promoting best practices in medicine, many issues surrounding CER remain poorly understood by clinicians and researchers, including what study designs are most appropriate for such research and what analytic tools are most helpful. The goal of this review is therefore to provide background and definitions of what constitutes CER and to discuss the various study designs and their strengths and weaknesses in achieving the stated goals of CER, while relating them to examples relevant to perioperative research. We provide a brief outline of the types of analytic methods particularly useful for CER and connect the reader to references for their practice. Finally, we assess the role of CER in perioperative research and provide some thoughts on future paths.