Using observational studies for comparative effectiveness: finding quality with GRACE

The Intraoperative Adherence to Multimodal Analgesia of Anesthesiologists: A Retrospective Study

Introduction

Multimodal analgesia (MMA) is a critical component of enhanced recovery after surgery (ERAS). However, little research revealed its intraoperative implementation by anesthesiologists, who are on the front line defending against surgical pain. Therefore, the objective of our study is to assess the adherence of anesthesiologists to MMA comprehensively.

Methods

A retrospective study was conducted involving patients undergoing lung resection, knee arthroplasty, and radical mastectomy from pre/post-implementation year of MMA (Jan 1, 2013, to Dec 31, 2013, vs. 2019). Intraoperative analgesia regimens (analgesic mode) and hourly rated morphine milligram equivalents (MME) were compared. In addition, patient characteristics associated with continued opioid use after surgery, surgical types, and position level of anesthesiologists (attending-junior; above attending-senior) were also analyzed.

Results

After MMA initiation, the rate of multimodal analgesic regimen (mode ≥ 2) was significantly increased (post- vs. pre-implementation, 31.57 vs. 21.50%, p < 0.05). However, MME did not show significant difference (post- vs. pre-implementation, 0.402 vs. 0.456, p > 0.05). Patient-level predictors of persistent opioid use after surgery were not related to increased analgesic mode. Lung resection [coefficient, − 0.538; 95% confidence interval (CI), − 0.695 to − 0.383, p < 0.001] and knee arthroplasty (coefficient, − 1.143; 95% CI, − 1.366 to − 0.925, p < 0.001) discouraged multiple analgesic mode, while senior anesthesiologists (coefficient, 0.674; 95% CI 0.548–0.800, p < 0.001) promoted it.

Conclusions

Although anesthesiologists used more analgesics after promoting MMA, the “opioid-sparing” principle was not followed properly. The analgesic mode was not instructed by patients’ characteristics appropriately. In addition, surgeries with cumbersome preparation/process impeded the use of multiple analgesic modes, while senior anesthesiologists preferred multiple analgesic modes.

The importance of the design of observational studies in comparative effectiveness research: Lessons from the GARFIELD-AF and ORBIT-AF registries

Randomized controlled trials (RCTs) are considered the gold standard for estimating the effectiveness of a treatment. However, in many instances they are impractical to conduct because of time limitations, cost restrictions, or ethical reasons. As a consequence, non-randomized observational studies have an important role in comparative effectiveness and safety research since they can address issues that would not be possible using conventional RCT methodology. Observational studies can be strategically designed to reduce the risk of potential sources of bias by emulating the design principles of an equivalent but ideal randomized trial - the target trial - that would answer the research question of interest. In this article, we review some of the necessary components of observational studies required for valid causal inference within the framework of target trial emulation, so as to avoid common methodological pitfalls of study design. We discuss the assumptions of consistency, time-zero specification, exchangeability and positivity. To illustrate these concepts in a context where existing knowledge is well-established through clinical trials, we evaluate and compare the treatment effects of vitamin K antagonists (VKA) against no VKA (No VKA) on the treatment of atrial fibrillation from two real-world observational studies, namely the GARFIELD-AF and ORBIT-AF registries. Results are compared with those of published RCTs.

Perioperative Individualized Goal Directed Therapy for Cardiac Surgery: A Historical-Prospective, Comparative Effectiveness Study

Introduction: Cardiac surgery patients are at increased risk for post-operative complications and prolonged length of stay. Perioperative goal directed therapy (GDT) has demonstrated utility for non-cardiac surgery, however, GDT is not common for cardiac surgery. We initiated a quality improvement (QI) project focusing on the implementation of a GDT protocol, which was applied from the immediate post-bypass period into the intensive care unit (ICU). Our hypothesis was that this novel GDT protocol would decrease ICU length of stay and possibly improve postoperative outcomes. Methods: This was a historical prospective, QI study for patients undergoing cardiac surgery requiring cardiopulmonary bypass (CPB). Integral to the QI project was education towards all associated providers on the concepts related to GDT. The protocol involved identifying patient specific targets for cardiac index and mean arterial pressure. These targets were maintained from the post-CPB period to the first 12 h in the ICU. Statistical comparisons were performed between the year after GDT therapy was launched to the last two years prior to protocol implementation. The primary outcome was ICU length of stay. Results: There was a significant decrease in ICU length of stay when comparing the year after the protocol initiation to years prior, from a median of 6.19 days to 4 days (2017 vs. 2019, p < 0.0001), and a median of 5.88 days to 4 days (2018 vs. 2019, p < 0.0001). Secondary outcomes demonstrated a significant reduction in total administered volumes of inotropic medication(milrinone). All other vasopressors demonstrated no differences across years. Hospital length of stay comparisons did not demonstrate a significant reduction. Conclusion: These results suggest that an individualized goal directed therapy for cardiac surgery patients can reduce ICU length of stay and decrease amount of inotropic therapy.

Online tools to synthesize real-world evidence of comparative effectiveness research to enhance formulary decision making

Results of randomized controlled trials (RCTs) provide valuable comparisons of 2 or more interventions to inform health care decision making; however, many more comparisons are required than available time and resources to conduct them. Moreover, RCTs have limited generalizability. Comparative effectiveness research (CER) using real-world evidence (RWE) can increase generalizability and is important for decision making, but use of nonrandomized designs makes their evaluation challenging. Several tools are available to assist. In this study, we comparatively characterize 5 tools used to evaluate RWE studies in the context of making health care adoption decision making: (1) Good Research for Comparative Effectiveness (GRACE) Checklist, (2) IMI GetReal RWE Navigator (Navigator), (3) Center for Medical Technology Policy (CMTP) RWE Decoder, (4) CER Collaborative tool, and (5) Real World Evidence Assessments and Needs Guidance (REAdi) tool. We describe each and then compare their features along 8 domains: (1) objective/user/context, (2) development/scope, (3) platform/presentation, (4) user design, (5) study-level internal/external validity of evidence, (6) summarizing body of evidence, (7) assisting in decision making, and (8) sharing results/making improvements. Our summary suggests that the GRACE Checklist aids stakeholders in evaluation of the quality and applicability of individual CER studies. Navigator is a collection of educational resources to guide demonstration of effectiveness, a guidance tool to support development of medicines, and a directory of authoritative resources for RWE. The CMTP RWE Decoder aids in the assessment of relevance and rigor of RWE. The CER Collaborative tool aids in the assessment of credibility and relevance. The REAdi tool aids in refinement of the research question, study retrieval, quality assessment, grading the body of evidence, and prompts with questions to facilitate coverage decisions. All tools specify a framework, were designed with stakeholder input, assess internal validity, are available online, and are easy to use. They vary in their complexity and comprehensiveness. The RWE Decoder, CER Collaborative tool, and REAdi tool synthesize evidence and were specifically designed to aid formulary decision making. This study adds clarity on what the tools provide so that the user can determine which best fits a given purpose. DISCLOSURES: This work was supported by the Health Tech Fund, which was provided to the University of Washington School of Pharmacy by its Corporate Advisory Board. This consortium of pharmaceutical and biotech companies supports the research program of the University of Washington School of Pharmacy across the competitive space. The sponsors seeded the idea for the project and contributed to study design and improvement. The authors had full control of all content development, manuscript drafting, and submission for publication. The REAdi tool was developed by the authors. Chen, Bansal, Barthold, Carlson, Veenstra, Basu, Devine, Yun, Ta, and Beal were supported by a training grant from the University of Washington-Allergan Fellowship, unrelated to this work. Basu reports personal fees from Salutis Consulting, unrelated to this work. Graff is an employee of the National Pharmaceutical Council, which was a partner in the development of the CER Collaborative and funding partner for the CMTP RWE Decoder and the GRACE Checklist. A previous version of this work was presented as an invited workshop at AMCP Nexus 2018; October 22-25, 2018; Orlando, FL.

Anesthesiologists as perioperative hospitalists and outcomes in patients undergoing major urologic surgery: a historical prospective, comparative effectiveness study

Background

Perioperative care has been identified as an area of wide variability in quality, with conflicting models, and involving multiple specialties. In 2014, the Loma Linda University Departments of Anesthesiology and Urology implemented a perioperative hospitalist service (PHS), consisting of anesthesiology-trained physicians, to co-manage patients for the entirety of their perioperative period. We hypothesized that implementation of this PHS model would result in an improvement in patient recovery.

Methods

As a quality improvement (QI) initiative, the PHS service was formed of selected anesthesiologists who received training on the core competencies for hospitalist medicine. The service was implemented following a co-management agreement to medically manage patients undergoing major urologic procedures (prostatectomy, cystectomy, and nephrectomy). Impact was assessed by comparisons to data from the year prior to PHS service implementation. Data was compared with and without propensity matching. Primary outcome marker was a reduction in length of stay. Secondary outcome markers included complication rate, return of bowel function, number of consultations, reduction in total direct patient costs, and bed days saved.

Results

Significant reductions in length of stay (p <  0.05) were demonstrated for all surgical procedures with propensity matching and were demonstrated for cystectomy and nephrectomy cases without. Significant reductions in complication rates and ileus were also observed for all surgical procedures post-PHS implementation. Additionally, reductions in total direct patient costs and frequency of consultations were also observed.

Conclusions

Anesthesiologists can safely function as perioperative hospitalists, providing appropriate medical management, and significantly improving both patient recovery and throughput.

Electronic supplementary material

The online version of this article (10.1186/s13741-018-0090-y) contains supplementary material, which is available to authorized users.

Impact of Enhanced Recovery After Surgery and Opioid-Free Anesthesia on Opioid Prescriptions at Discharge From the Hospital: A Historical-Prospective Study

BACKGROUND

The United States is in the midst of an opioid epidemic, and opioid use disorder often begins with a prescription for acute pain. The perioperative period represents an important opportunity to prevent chronic opioid use, and recently there has been a paradigm shift toward implementation of enhanced recovery after surgery (ERAS) protocols that promote opioid-free and multimodal analgesia. The objective of this study was to assess the impact of an ERAS intervention for colorectal surgery on discharge opioid prescribing practices.

METHODS

We conducted a historical-prospective quality improvement study of an ERAS protocol implemented for patients undergoing colorectal surgery with a focus on the opioid-free and multimodal analgesia components of the pathway. We compared patients undergoing colorectal surgery 1 year before implementation (June 15, 2015, to June 14, 2016) and 1 year after implementation (June 15, 2016, to June 14, 2017).

RESULTS

Before the ERAS intervention, opioids at discharge were not significantly increasing (1% per month; 95% confidence interval [CI], -1% to 3%; P = .199). Immediately after the ERAS intervention, opioid prescriptions were not significantly lower (13%; 95% CI, -30% to 3%; P = .110). After the intervention, the rate of opioid prescriptions at discharge did not decrease significantly 1% (95% CI, -3% to 1%) compared to the pre-period rate (P = .399). Subgroup analysis showed that in patients with a combination of low discharge pain scores, no preoperative opioid use, and low morphine milligram equivalents consumption before discharge, the rate of discharge opioid prescription was 72% (95% CI, 61%-83%).

CONCLUSIONS

This study is the first to report discharge opioid prescribing practices in an ERAS setting. Although an ERAS intervention for colorectal surgery led to an increase in opioid-free anesthesia and multimodal analgesia, we did not observe an impact on discharge opioid prescribing practices. The majority of patients were discharged with an opioid prescription, including those with a combination of low discharge pain scores, no preoperative opioid use, and low morphine milligram equivalents consumption before discharge. This observation in the setting of an ERAS pathway that promotes multimodal analgesia suggests that our findings are very likely to also be observed in non-ERAS settings and offers an opportunity to modify opioid prescribing practices on discharge after surgery. For opioid-free anesthesia and multimodal analgesia to influence the opioid epidemic, the dose and quantity of the opioids prescribed should be modified based on the information gathered by in-hospital pain scores and opioid use as well as pain history before admission.

The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness

BACKGROUND

Recognizing the growing need for robust evidence about treatment effectiveness in real-world populations, the Good Research for Comparative Effectiveness (GRACE) guidelines have been developed for noninterventional studies of comparative effectiveness to determine which studies are sufficiently rigorous to be reliable enough for use in health technology assessments.

OBJECTIVE

To evaluate which aspects of the GRACE Checklist contribute most strongly to recognition of quality.

METHODS

We assembled 28 observational comparative effectiveness articles published from 2001 to 2010 that compared treatment effectiveness and/or safety of drugs, medical devices, and medical procedures. Twenty-two volunteers from academia, pharmaceutical companies, and government agencies applied the GRACE Checklist to those articles, providing 56 assessments. Ten senior academic and industry experts provided assessments of overall article quality for the purpose of decision support. We also rated each article based on the number of annual citations and impact factor of the journal in which the article was published. To identify checklist items that were most predictive of quality, classification and regression tree (CART) analysis, a binary, recursive, partitioning methodology, was used to create 3 decision trees, which compared the 56 article assessments with 3 external quality outcomes: (1) expert assessment of overall quality, (2) citation frequency, and (3) impact factor. A fourth tree looked at the composite outcome of all 3 quality indicators.

RESULTS

The best predictors of quality included the following: use of concurrent comparators, limiting the study to new initiators of the study drug, equivalent measurement of outcomes in study groups, collecting data on most if not all known confounders or effect modifiers, accounting for immortal time bias in the analysis, and use of sensitivity analyses to test how much effect estimates depended on various assumptions. Only sensitivity analyses appeared consistently as a predictor of quality in all 4 trees. When a composite outcome of the 3 quality measures was used, the GRACE Checklist showed high sensitivity and specificity (71.43% and 80.95%, respectively).

CONCLUSIONS

The GRACE Checklist stands out from other consensus-driven and expert guidance documents because of its extensive validation efforts. This most recent work shows that the checklist has strong sensitivity and specificity, increasing its utility as a screening tool to identify high-quality observational comparative effectiveness research worthy of in-depth review and applicability for decision support.

DISCLOSURES

No outside funding supported this research. All authors are full-time employees of Quintiles, which provides research and consulting services to the biopharmaceutical industry. The authors have no other disclosures to report. Two of the 3 CART trees were presented at the International Society of Pharmacepidemiology in 2015 ("Article Citations per Year" and "Journal Impact Factor"). The original validation study was published in the March 2014 issue of the Journal of Managed Care & Specialty Pharmacy. The checklist questions and scoring were included using a table that was originally published by this journal in 2014. Study concept and design were primarily contributed by Dreyer and Velentgas, along with Bryant. Bryant took the lead in data collection and analysis, along with Dreyer and Velentgas, and data interpretation was performed by Dreyer, Velentgas, and Bryant. The manuscript was written and revised primarily by Dreyer, along with Bryant and Velentgas.

Perioperative goal-directed therapy and postoperative outcomes in patients undergoing high-risk abdominal surgery: a historical-prospective, comparative effectiveness study

INTRODUCTION

Perioperative goal-directed therapy (PGDT) may improve postoperative outcome in high-risk surgery patients but its adoption has been slow. In 2012, we initiated a performance improvement (PI) project focusing on the implementation of PGDT during high-risk abdominal surgeries. The objective of the present study was to evaluate the effectiveness of this intervention.

METHODS

This is a historical prospective quality improvement study. The goal of this initiative was to standardize the way fluid management and hemodynamic optimization are conducted during high-risk abdominal surgery in the Departments of Anesthesiology and Surgery at the University of California Irvine. For fluid management, the protocol consisted in standardized baseline crystalloid administration of 3 ml/kg/hour and any additional boluses based on PGDT. The impact of the intervention was assessed on the length of stay in the hospital (LOS) and post-operative complications (NSQIP database).

RESULTS

In the 1 year pre- and post-implementation periods, 128 and 202 patients were included. The average volume of fluid administered during the case was 9.9 (7.1-13.0) ml/kg/hour in the pre-implementation period and 6.6 (4.7-9.5) ml/kg/hour in the post-implementation period (p < 0.01). LOS decreased from 10 (6-16) days to 7 (5-11) days (p = 0.0001). Based on the multiple linear regression analysis, the estimated coefficient for intervention was 0.203 (SE = 0.054, p = 0.0002) indicating that, with the other conditions being held the same, introducing intervention reduced LOS by 18% (95% confidence interval 9-27%). The incidence of NSQIP complications decreased from 39% to 25% (p = 0.04).

CONCLUSION

These results suggest that the implementation of a PI program focusing on the implementation of PGDT can transform fluid administration patterns and improve postoperative outcome in patients undergoing high-risk abdominal surgeries.

TRIAL REGISTRATION

Clinicaltrials.gov NCT02057653. Registered 17 December 2013.

Translating comparative effectiveness research into Medicaid payment policy: views from medical and pharmacy directors

Background: As the USA seeks to expand the conduct and dissemination of comparative effectiveness research (CER), views of key stakeholders will help guide the way. Methods: We surveyed 60 medical and pharmacy directors from 46 state Medicaid programs. Results: Over 90% felt that CER would lead to better clinical decision-making and overall value within 5 years and were willing to consider cost–effectiveness in setting medical policy. However, perceived poor quality, inconclusive research, restrictive legislative mandates, lack of budget impact and coverage recommendations, and lack of an independent body to interpret study results were major barriers cited to using CER evidence. Conclusion: Given the significant resources being invested in CER, it is critical that these barriers are overcome to maximize its usefulness for stakeholders.