Bayesian methods for assessing transplant program performance

Impact of institutional variables on centre performance in long-term survival after heart transplant

OBJECTIVES

The gold standard metric for centre-level performance in orthotopic heart transplantation (OHT) is 1-year post-OHT survival. However, it is unclear whether centre performance at 1 year is predictive of longer-term outcomes. This study evaluated factors impacting longer-term centre-level performance in OHT.

METHODS

Patients who underwent OHT in the USA between 2010 and 2021 were identified using the United Network of Organ Sharing data registry. The primary outcome was 5-year survival conditional on 1-year survival following OHT. Multivariable Cox proportional hazard models assessed the impact of centre-level 1-year survival rates on 5-year survival rates. Mixed-effect models were used to evaluate between-centre variability in outcomes.

RESULTS

Centre-level risk-adjusted 5-year mortality conditional on 1-year survival was not associated with centre-level 1-year survival rates [hazard ratio: 0.99 (0.97-1.01, P = 0.198)]. Predictors of 5-year mortality conditional on 1-year survival included black recipient race, pre-OHT serum creatinine, diabetes and donor age. In mixed-effect modelling, there was substantial variability between centres in 5-year mortality rates conditional on 1-year survival, a finding that persisted after controlling for recipient, donor and institutional factors (P < 0.001). In a crude analysis using Kaplan-Meier, the 5-year survival conditional on 1-year survival was: low volume: 86.5%, intermediate volume: 87.5%, high volume: 86.7% (log-rank P = 0.52). These measured variables only accounted for 21.4% of the between-centre variability in 5-year mortality conditional on 1-year survival.

CONCLUSIONS

Centre-level risk-adjusted 1-year outcomes do not correlate with outcomes in the 1- to 5-year period following OHT. Further research is needed to determine what unmeasured centre-level factors contribute to longer-term outcomes in OHT.

Healthcare center clustering for Cox's proportional hazards model by fusion penalty

There has been growing research interest in developing methodology to evaluate healthcare centers' performance with respect to patient outcomes. Conventional assessments can be conducted using fixed or random effects models, as seen in provider profiling. We propose a new method, using fusion penalty to cluster healthcare centers with respect to a survival outcome. Without any prior knowledge of the grouping information, the new method provides a desirable data-driven approach for automatically clustering healthcare centers into distinct groups based on their performance. An efficient alternating direction method of multipliers algorithm is developed to implement the proposed method. The validity of our approach is demonstrated through simulation studies, and its practical application is illustrated by analyzing data from the national kidney transplant registry.

An evaluation of the organ procurement and transplantation network's expanded post-transplant performance metrics

On July 14, 2022, the Organ Procurement and Transplantation Network's (OPTN) Membership and Professional Standards Committee (MPSC) approved bylaws including two new post-transplant performance evaluation metrics, the 90-day (90D) and 1-year conditional on the 90-day (1YC90D) graft survival hazard ratio (HR). These metrics have replaced the previous 1-year (1Y) unconditional, post-transplant graft survival HR and are used to nationally rank and identify programs for MPSC review. The MPSC's policies have major implications for all transplant programs, providers, and patients across the United States. Herein we show two significant limitations with the new evaluation criteria, arbitrary censoring periods and interdependence in the new performance metrics. We have demonstrated a strong and consistent inverse correlation between the new evaluation metrics, thus proving a lack of independence. Moreover, these two evaluation criteria are interdependent even at nominal HRs. Thus, the 90D cohort can be used to accurately predict whether the 1YC90D is above or below a given HR threshold. This could alter practice behaviors and the timing of patient event reporting, which may result in many unintended consequences related to clinical practice. Here we provide the first evidence that this new evaluation system will lead to a significant increase in the number of programs flagged for MPSC review. When this occurs, the cost of operating a transplant program will increase without a clear demonstration of an increased accuracy in identifying problematic programs.

Stakeholders' Perspectives on Transplant Metrics: The 2022 Scientific Registry of Transplant Recipients' Consensus Conference

In July 2022, the Scientific Registry of Transplant Recipients (SRTR) hosted an innovative, multistakeholder consensus conference to identify information and metrics desired by stakeholders in the transplantation system, including patients, living donors, caregivers, deceased donor family members, transplant professionals, organ procurement organization professionals, payers, and regulators. Crucially, patients, caregivers, living donors, and deceased donor family members were included in all aspects of this conference, including serving on the planning committee, participating in preconference focus groups and learning sessions, speaking at the conference, moderating conference sessions and breakout groups, and shaping the conclusions. Patients constituted 24% of the meeting participants. In this report, we document the proceedings and enumerate 160 recommendations, 10 of which have been highly prioritized. SRTR will use the recommendations to develop new presentations of information and metrics requested by stakeholders to support informed decision-making.

Impact of Donor Liver Macrovesicular Steatosis on Deceased Donor Yield and Posttransplant Outcome

BACKGROUND

The Scientific Registry of Transplant Recipients (SRTR) had not traditionally considered biopsy results in risk-adjustment models, yet biopsy results may influence outcomes and thus decisions regarding organ acceptance.

METHODS

Using SRTR data, which includes data on all donors, waitlisted candidates, and transplant recipients in the United States, we assessed (1) the impact of macrovesicular steatosis on deceased donor yield (defined as number of livers transplanted per donor) and 1-y posttransplant graft failure and (2) the effect of incorporating this variable into existing SRTR risk-adjustment models.

RESULTS

There were 21 559 donors with any recovered organ and 17 801 liver transplant recipients included for analysis. Increasing levels of macrovesicular steatosis on donor liver biopsy predicted lower organ yield: ≥31% macrovesicular steatosis on liver biopsy was associated with 87% to 95% lower odds of utilization, with 55% of these livers being discarded. The hazard ratio for graft failure with these livers was 1.53, compared with those with no pretransplant liver biopsy and 0% to 10% steatosis. There was minimal change on organ procurement organization-specific deceased donor yield or program-specific posttransplant outcome assessments when macrovesicular steatosis was added to the risk-adjustment models.

CONCLUSIONS

Donor livers with macrovesicular steatosis are disproportionately not transplanted relative to their risk for graft failure. To avoid undue risk aversion, SRTR now accounts for macrovesicular steatosis in the SRTR risk-adjustment models to help facilitate use of these higher-risk organs. Increased recognition of this variable may also encourage further efforts to standardize the reporting of liver biopsy results.

Transplant program evaluations in the middle of the COVID-19 pandemic

Potential regional variations in effects of COVID-19 on federally mandated, program-specific evaluations by the Scientific Registry of Transplant Recipients (SRTR) have been controversial. SRTR January 2022 program evaluations ended transplant follow-up on March 12, 2020, and excluded transplants performed from March 13, 2020 to June 12, 2020 (the "carve-out"). This study examined the carve-out's impact, and the effect of additionally censoring COVID-19 deaths, on first-year posttransplant outcomes for transplants from July 2018 through December 2020. Program-specific hazard ratios (HRs) for graft failure and death estimated under two alternative scenarios were compared with published HRs: (1) the carve-out was removed; (2) the carve-out was retained, but deaths due to COVID-19 were additionally censored. The HRs estimated by censoring COVID-19 deaths were highly correlated with those estimated with the carve-out alone (r2  = .96). Removal of the carve-out resulted in greater variation in HRs while remaining highly correlated (r2  = .82); however, little geographic impact of the carve-out was observed. The carve-out increased average HR in the Northwest by 0.049; carve-out plus censoring reduced average HR in the Midwest by 0.009. Other regions of the country were not significantly affected. Thus, the current COVID-19 carve-out does not appear to impart substantial bias based on the region of the country.

Textbook Outcome as a Quality Metric in Living and Deceased Donor Kidney Transplantation

BACKGROUND

Quality in kidney transplantation is measured using 1-year patient and graft survival. Because 1-year patient and graft survival exceed 95%, this metric fails to measure a spectrum of quality. Textbook outcomes (TO) are a composite quality metric offering greater depth and resolution. We studied TO after living donor (LD) and deceased donor (DD) kidney transplantation.

STUDY DESIGN

United Network for Organ Sharing data for 69,165 transplant recipients between 2013 and 2017 were analyzed. TO was defined as patient and graft survival of 1 year or greater, 1-year glomerular filtration rate of greater than 40 mL/min, absence of delayed graft function, length of stay of 5 days or less, no readmissions during the first 6 months, and no episodes of rejection during the first year after transplantation. Bivariate analysis identified characteristics associated with TO, and covariates were incorporated into multivariable models. Five-year conditional survival was measured, and center TO rates were corrected for case complexity to allow center-level comparisons.

RESULTS

The national average TO rates were 54.1% and 31.7% for LD and DD transplant recipients. The hazard ratio for death at 5 years for recipients who did not experience TO was 1.92 (95% CI 1.68 to 2.18, p ≤ 0.0001) for LD transplant recipients and 2.08 (95% CI 1.93 to 2.24, p ≤ 0.0001) for DD transplant recipients. Center-level comparisons identify 18% and 24% of centers under-performing in LD and DD transplantation. High rates of TO do not correlate with transplantation center volume.

CONCLUSION

Kidney transplant recipients who experience TO have superior long-term survival. Textbook outcomes add value to the current standards of 1-year patient and graft survival.

A Composite End Point of Graft Status and eGFR at 1 Year to Improve the Scientific Registry of Transplant Recipients' Five-Tier Rating System

BACKGROUND

Performance of kidney transplant programs in the United States is monitored and publicly reported by the Scientific Registry of Transplant Recipients (SRTR). With relatively few allograft failure events per program and increasing homogeneity in program performance, quantifying meaningful differences in program competency based only on 1-year survival rates is challenging.

METHODS

We explored whether the traditional end point of allograft failure at 1 year can be improved by incorporating a measure of allograft function (i.e., eGFR) into a composite end point. We divided SRTR data from 2008 through 2018 into a training and validation set and recreated SRTR tiers, using the traditional and composite end points. The conditional 5-year deceased donor allograft survival and 5-year eGFR were then assessed using each approach.

RESULTS

Compared with the traditional end point, the composite end point of graft failure or eGFR <30 ml/min per 1.73 m² at 1-year post-transplant performed better in stratifying transplant programs based on long-term deceased donor graft survival. For tiers 1 through 5 respectively, the 5-year conditional graft survival was 72.9%, 74.8%, 75.4%, 77.0%, and 79.7% using the traditional end point and 71.1%, 74.4%, 76.9%, 77.0%, and 78.4% with the composite end point. Additionally, with the five-tier system derived from the composite end point, programs in tier 3, tier 4, and tier 5 had significantly higher mean eGFRs at 5 years compared with programs in tier 1. There were no significant eGFR differences among tiers derived from the traditional end point alone.

CONCLUSIONS

This proof-of-concept study suggests that a composite end point incorporating allograft function may improve the post-transplant component of the five-tier system by better differentiating between transplant programs with respect to long-term graft outcomes.

One-year Mortality Is Not a Reliable Indicator of Lung Transplant Center Performance

BACKGROUND

In the United States, the Organ Procurement and Transplant Network uses one-year mortality as the primary measure of transplant center quality. We sought to evaluate the reliability of mortality outcomes in lung transplant and compare statistical methods of program performance evaluation.

METHODS

We used the Standard Transplant Analysis and Research files from the United Network for Organ Sharing to identify lung transplant recipients from 2013-2018 in the United States. We stratified hospitals based on 30-day, 1-year and 5-year survival using risk adjustment, reliability adjustment using empirical Bayes technique, and hierarchical Bayesian mixed-effects models currently used by the OPTN. We measured variation in mortality rates and identification of performance outliers between techniques.

RESULTS

We identified 12,769 recipients in 69 centers. Reliability adjustment reduced variation in hospital outcomes and had a large impact on hospital mortality rankings. For example, with 1-year mortality, 28% (5 hospitals) of the "best" hospitals (top 25%) and 18% (3 hospitals) of the "worst" hospitals (bottom 25%) were reclassified after reliability adjustment. The overall reliability of 1-year mortality was low at 0.42. Compared to the Bayesian method used by the OPTN, reliability adjustment identified fewer outliers. 5-year survival reached a higher reliability plateau with a lower volume of cases required.

CONCLUSIONS

The reliability of 1-year mortality in lung transplantation is low, while 5-year survival estimates may be more reliable at lower case volumes. Reliability adjustment yielded more conservative measures of center performance and fewer outliers compared to current Bayesian methods.

Kidney transplant program specific reporting and transplant metrics

PURPOSE OF REVIEW

Kidney transplantation is a heavily regulated medical procedure with the Secretary of HHS ultimately responsible for oversight and authority derived from the NOTA and the Final Rule. Transplant Programs undergo publicly reported evaluations every 6 months based on outcomes from a 2-and-a-half-year period. The current Bayesian metrics for kidney transplant programs were created such that over ten percentage of programs are deemed underperformers, or 'flag', every 6 months. Newly suggested transplant metrics have been released for public comment in Summer 2021. In addition to graft outcomes, waiting list mortality and organ acceptance rate ratios are proposed.

RECENT FINDINGS

Under the newly proposed kidney transplant metrics, over 10% of programs are expected to be deemed underperformers or 'flagged'. Transplant Center flagging is well correlated with decreased transplantation due to the transplant centres move to more conservative organ and patient acceptance. Death on the waiting list is a proposed metric over which transplant centres have little influence.

SUMMARY

In the USA, the harsh regulation continued by Health Resources and Services Administration (HRSA) through the national organ procurement and transplant network (OPTN) and Scientific Registry for Transplant Recipients (SRTR) leads directly to high organ discard rates and limitations to transplanting patients with perceived unadjusted risks. Instead of loosening regulation in a highly functioning industry that achieves remarkable outcomes in end stage kidney patients, the OPTN with the SRTR persist in increasing potential penalties through more proposed metrics that continue to deem 10% of US kidney transplant programs as underperformers. HRSA must establish a reasonable regulatory environment that allows for innovation and increased transplant opportunities for US end-stage renal disease patients.

Kidney transplant program waitlisting rate as a metric to assess transplant access

Kidney transplant program performance in the United States is commonly measured by posttransplant outcomes. Inclusion of pretransplant measures could provide a more comprehensive assessment of transplant program performance and necessary information for patient decision-making. In this study, we propose a new metric, the waitlisting rate, defined as the ratio of patients who are waitlisted in a center relative to the person-years referred for evaluation to a program. Furthermore, we standardize the waitlisting rate relative to the state average in Georgia, North Carolina, and South Carolina. The new metric was used as a proof-of-concept to assess transplant-program access compared to the existing transplant rate metric. The study cohorts were defined by linking 2017 United States Renal Data System (USRDS) data with transplant-program referral data from the Southeastern United States between January 1, 2012 and December 31, 2016. Waitlisting rate varied across the 9 Southeastern transplant programs, ranging from 10 to 22 events per 100 patient-years, whereas the program-specific waitlisting rate ratio ranged between 0.76 and 1.33. Program-specific waitlisting rate ratio was uncorrelated with the transplant rate ratio (r = -.15, 95% CI, -0.83 to 0.57). Findings warrant collection of national data on early transplant steps, such as referral, for a more comprehensive assessment of transplant program performance and pretransplant access.

Posttransplant outcome assessments at listing: Long-term outcomes are more important than short-term outcomes

Posttransplant outcome assessments are publicly reported for patient and regulatory use. However, the currently reported 1-year posttransplant graft survival assessments are commonly criticized for not identifying clinically meaningful differences between programs, and not providing information about longer-term posttransplant outcomes. We investigated the association of different posttransplant outcome assessments available to patients at the time of listing with subsequent posttransplant graft survival. The posttransplant assessments were from period prevalent, rather than incident, cohorts with more timely 1-, 3-, and 5-year follow-up and 6-, 12-, 18-, 24-, and 30-month cohort windows. The association of these assessments at listing with subsequent posttransplant graft survival included candidates listed between July 12, 2011, and December 15, 2015, who subsequently underwent transplant before December 31, 2018. The assessments with 1-year follow-up had uniformly weaker associations than the assessments with 3- and 5-year follow-up. The assessments with 5-year follow-up had the strongest association in kidney and liver transplantation. For kidney, liver, and lung transplantation, assessment windows of at least 18 months typically had the strongest associations with subsequent graft survival. Posttransplant assessments with 5-year follow-up and 18-30-month cohort windows are better than the current posttransplant assessment with 1-year follow-up, particularly at the time of listing.

Association between changes in Membership and Professional Standards Committee review criteria and use of higher-risk kidneys for transplant

The Organ Procurement and Transplantation Network's Membership and Professional Standards Committee implemented an operational rule on March 1, 2017, intended to increase the number of kidneys transplanted from donors with kidney donor profile index (KDPI) ≥ 85% into recipients with poor estimated posttransplant survival (≥ 80%). Using data from the Scientific Registry of Transplant Recipients, ordinal and logistic regressions estimated, respectively, differences in kidney yield (number of transplanted kidneys per recovered donor) and offer acceptance practices before and after implementation. We included donors recovered January 1, 2016-February 28, 2018. The odds of higher kidney yield for donors with KDPI ≥ 85% were 27% higher after implementation (odds ratio, _1.06 1.27_1.53 ), but odds were also 20% higher for donors with KDPI < 85% (_1.04 1.20_1.38 ). Thus, kidney yield was higher for all donors, with a slightly larger difference for donors with KDPI ≥ 85%. Additionally, the difference in offer acceptance before and after implementation was similar regardless of KDPI (KDPI < 85%, _0.97 1.02_1.07 ; KDPI ≥ 85%, _0.95 1.04_1.14 ). In the first year after implementation, kidney yield increased for donors with KDPI < and ≥ 85%. Thus, kidney yield from higher KDPI donors may have increased without the operational rule.

Comparing Scientific Registry of Transplant Recipients posttransplant program-specific outcome ratings at listing with subsequent recipient outcomes after transplant

To improve accessibility of program-specific reports to patients, the Scientific Registry of Transplant Recipients released a 5-tier system for categorizing 1-year posttransplant program evaluations. Whether this system predicts subsequent posttransplant outcomes at the time patients are waitlisted has been questioned. We investigated the association of tier at listing and the corresponding continuous score used for tier assignment, which ranges from 0 (poor outcomes) to 1 (good outcomes), with eventual 1-year posttransplant graft survival for candidates listed between July 12, 2011, and June 16, 2014, who underwent transplant before December 31, 2016. One additional tier at listing was associated with better 1-year posttransplant outcomes in liver (hazard ratio [HR], 0.93; 95% confidence interval [CI], 0.89-0.97) and lung transplant (HR, 0.90; 95% CI, 0.84-0.97) but not kidney (HR, 0.96; 95% CI, 0.92-1.01) or heart transplant (HR, 1.02; 95% CI, 0.93-1.10). In liver and lung transplant, longer time between listing and transplant was associated with stronger protective effects for high-tier programs. In kidney, liver, and lung transplant, posttransplant evaluations at listing had nonlinear associations with eventual posttransplant outcomes: relatively flat for 5-tier scores <0.5 and decreasing for scores >0.5. After adjustment for measured recipient and donor risk factors, posttransplant evaluations at listing predicted differences in eventual outcomes in liver and lung transplant, providing useful information to patients.

Association of pretransplant and posttransplant program ratings with candidate mortality after listing

The Scientific Registry of Transplant Recipients (SRTR) is responsible for understandable reporting of program metrics, including transplant rate, waitlist mortality, and posttransplant outcomes. SRTR developed five-tier systems for each metric to improve accessibility for the public. We investigated the associations of the five-tier assignments at listing with all-cause candidate mortality after listing, for candidates listed July 12, 2011-June 16, 2014. Transplant rate evaluations with one additional tier were associated with lower mortality after listing in kidney (hazard ratio [HR], _0.93 0.95_0.97 ), liver (HR, _0.87 0.90_0.92 ), and heart (HR, _0.92 0.96_1.00 ) transplantation. For lung transplant patients, mortality after listing was highest at programs with above- and below-average transplant rates and lowest at programs with average transplant rates, suggesting that aggressive acceptance behavior may not always provide a survival benefit. Waitlist mortality evaluations with one additional tier were associated with lower mortality after listing in kidney (HR, _0.94 0.96_0.99 ) transplantation, and posttransplant graft survival evaluations with one additional tier were associated with lower mortality after listing in lung (HR, _0.90 0.94_0.98 ) transplantation. Transplant rate typically had the strongest association with mortality after listing, but the strength of associations differed by organ.

The relationship between the C-statistic and the accuracy of program-specific evaluations

The C-statistic of the risk-adjustment model is often used to judge the accuracy of program evaluations. However, the C-statistic depends on the variability in risk for individual transplants and may be inappropriate for determining the accuracy of program evaluations. A simulation study investigated the association of the C-statistic with several metrics of program evaluation accuracy, including categorizing programs into the 5-tier system and identifying programs for regulatory review. The simulation study used data from deceased donor kidney-alone transplants for adult recipients in the program-specific reports released January 2018. A range of C-statistics was generated by changing the variability in risk for individual transplants. The C-statistic had no association with any metric of program evaluation accuracy. Instead, the number of expected events at a program was the most important factor. For example, Spearman's rho, which is the correlation of ranks, was -0.27 and -0.72 between the true program-specific hazard ratios and assigned tiers for programs with, respectively, <3 and >10 expected events. Presence of unadjusted risk factors did not modify the associations, although the accuracy of program evaluations was systematically lower. Therefore, the C-statistic provides no information on the accuracy of program evaluations.

Comparing dialysis centre mortality outcomes across Australia and New Zealand: identifying unusually performing centres 2008-2013

BACKGROUND

Comparing the mortality profiles of dialysis centres is important to ensure that high standards of care are maintained. We compare the performance of dialysis centres in Australia and New Zealand in their treatment of haemodialysis patients, accounting for the competing risks of kidney transplantation and transfer to peritoneal dialysis.

METHODS

Observational cohort study. We included data from all adult patients (5574 patients) commencing haemodialysis at home or in a facility between 2008 and 2010 across 62 dialysis centres, from the Australia and New Zealand Dialysis and Transplant Registry. Standardised mortality ratios were calculated by estimating mortality probabilities from a pooled random effects logistic regression model, accounting for the competing risk of transplantation using an inverse probability weighting approach. Models were adjusted for patient comorbidities, sex, height, weight, late referral to a nephrologist, age, race, primary kidney disease, smoking status, and serum creatinine (μmol/l).

RESULTS

Two dialysis centres were found to have relatively higher levels of risk-adjusted mortality lying outside the prediction intervals for "usual" performance. Risk adjusted mortality rates were not associated with centres' compliance with guidelines for vascular access and biochemical and haematological targets.

CONCLUSIONS

We demonstrate that standardised mortality ratios are useful to identify facilities that have statistically outlying mortality risk. Our criterion for determining whether a centre has better or worse performance than expected is statistical, and thus analyses such as ours can serve only as a screening tool, and are only one aspect of assessment of "quality" of performance.

Scientific Registry of Transplant Recipients program-specific reports: where we have been and where we are going

PURPOSE OF REVIEW

Reporting provider data on quality to patients and the general public is increasingly common in healthcare. Reporting outcomes in solid organ transplantation has always been controversial and deserves careful consideration to ensure optimal results.

RECENT FINDINGS

As mandated by Federal law, the Scientific Registry of Transplant Recipients publishes program-specific reports on transplant candidates, recipients, donors, and transplant outcomes every 6 months. Recent changes designed to make the results more easily understood by patients and the general public have been well received by patients and controversial among providers. In particular, outcomes are now reported using a five-tier system that distinguishes program results better than the old three-tier system, in which almost all programs were reported "as expected." Metrics that reflect access to transplant are also reported, including transplant rate and survival on the waiting list. Possible measures of longer term outcomes and program rates of accepting organs for transplant are being explored.

SUMMARY

Providing detailed information regarding transplant program practices and outcomes in ways that patients and the general public can understand remains a major focus of the Scientific Registry of Transplant Recipients. Efforts to improve data collection and metrics reported are ongoing.

Reported effects of the Scientific Registry of Transplant Recipients 5-tier rating system on US transplant centers: results of a national survey

In the United States, the Scientific Registry of Transplant Recipients (SRTR) provides publicly available quality report cards. These reports have historically rated transplant programs using a 3-tier system. In 2016, the SRTR temporarily transitioned to a 5-tier system, which classified more programs as under-performing. As part of a larger survey about transplant quality metrics, we surveyed members of the American Society of Transplant Surgeons and American Society of Transplantation (N = 280 respondents) on transplant center experiences with patient and payer responses to the 5-tier SRTR ratings. Over half of respondents (n = 137, 52.1%) reported ≥1 negative effect of the new 5-tier ranking system, including losing patients, losing insurers, increased concern among patients, and increased concern among referring providers. Few respondents (n = 35, 13.7%) reported any positive effects of the 5-tier ranking system. Lower SRTR-reported scores on the 5-tier scale were associated with increased risk of reporting at least one negative effect in a logistic model (P < 0.01). The change to a more granular rating system provoked an immediate response in the transplant community that may have long-term implications for transplant hospital finances and patient options for transplantation.

Expanding clarity or confusion? Volatility of the 5-tier ratings assessing quality of transplant centers in the United States

Outcomes of patients receiving solid organ transplants in the United States are systematically aggregated into bi-annual Program-Specific Reports (PSRs) detailing risk-adjusted survival by transplant center. Recently, the Scientific Registry of Transplant Recipients (SRTR) issued 5-tier ratings evaluating centers based on risk-adjusted 1-year graft survival. Our primary aim was to examine the reliability of 5-tier ratings over time. Using 10 consecutive PSRs for adult kidney transplant centers from June 2012 to December 2016 (n = 208), we applied 5-tier ratings to center outcomes and evaluated ratings over time. From the baseline period (June 2012), 47% of centers had at least a 1-unit tier change within 6 months, 66% by 1 year, and 94% by 3 years. Similarly, 46% of centers had at least a 2-unit tier change by 3 years. In comparison, 15% of centers had a change in the traditional 3-tier rating at 3 years. The 5-tier ratings at 4 years had minimal association with baseline rating (Kappa 0.07, 95% confidence interval [CI] -0.002 to 0.158). Centers had a median of 3 different 5-tier ratings over the period (q1 = 2, q3 = 4). Findings were consistent for center volume, transplant rate, and baseline 5-tier rating. Cumulatively, results suggest that 5-tier ratings are highly volatile, limiting their utility for informing potential stakeholders, particularly transplant candidates given expected waiting times between wait listing and transplantation.

Quality measurement and improvement in liver transplantation

There is growing interest in the quality of health care delivery in liver transplantation. Multiple stakeholders, including patients, transplant providers and their hospitals, payers, and regulatory bodies have an interest in measuring and monitoring quality in the liver transplant process, and understanding differences in quality across centres. This article aims to provide an overview of quality measurement and regulatory issues in liver transplantation performed within the United States. We review how broader definitions of health care quality should be applied to liver transplant care models. We outline the status quo including the current regulatory agencies, public reporting mechanisms, and requirements around quality assurance and performance improvement (QAPI) activities. Additionally, we further discuss unintended consequences and opportunities for growth in quality measurement. Quality measurement and the integration of quality improvement strategies into liver transplant programmes hold significant promise, but multiple challenges to successful implementation must be addressed to optimise value.

Program-specific transplant rate ratios: Association with allocation priority at listing and posttransplant outcomes

The Scientific Registry of Transplant Recipients (SRTR) is considering more prominent reporting of program-specific adjusted transplant rate ratios (TRRs). To enable more useful reporting of TRRs, SRTR updated the transplant rate models to adjust explicitly for components of allocation priority. We evaluated potential associations between TRRs and components of allocation priority that could indicate programs' ability to manipulate TRRs by denying or delaying access to low-priority candidates. Despite a strong association with unadjusted TRRs, we found no candidate-level association between the components of allocation priority and adjusted TRRs. We found a strong program-level association between median laboratory Model for End-stage Liver Disease (MELD) score at listing and program-specific adjusted TRRs (r = .37; P < .001). The program-level association was likely confounded by regional differences in donor supply/demand and listing practices. In kidney transplantation, higher program-specific adjusted TRRs were weakly associated with better adjusted posttransplant outcomes (r = -.14; P = .035) and lower adjusted waitlist mortality rate ratios (r = -.15; P = .022), but these associations were absent in liver, lung, and heart transplantation. Program-specific adjusted TRRs were unlikely to be improved by listing candidates with high allocation priority and can provide useful information for transplant candidates and programs.

A novel patient-centered "intention-to-treat" metric of U.S. lung transplant center performance

Despite the importance of pretransplantation outcomes, 1-year posttransplantation survival is typically considered the primary metric of lung transplant center performance in the United States. We designed a novel lung transplant center performance metric that incorporates both pre- and posttransplantation survival time. We performed an ecologic study of 12 187 lung transplant candidates listed at 56 U.S. lung transplant centers between 2006 and 2012. We calculated an "intention-to-treat" survival (ITTS) metric as the percentage of waiting list candidates surviving at least 1 year after transplantation. The median center-level 1-year posttransplantation survival rate was 84.1%, and the median center-level ITTS was 66.9% (mean absolute difference 19.6%, 95% limits of agreement 4.3 to 35.1%). All but 10 centers had ITTS values that were significantly lower than 1-year posttransplantation survival rates. Observed ITTS was significantly lower than expected ITTS for 7 centers. These data show that one third of lung transplant candidates do not survive 1 year after transplantation, and that 12% of centers have lower than expected ITTS. An "intention-to-treat" survival metric may provide a more realistic expectation of patient outcomes at transplant centers and may be of value to transplant centers and policymakers.

Evaluation of Flagging Criteria of United States Kidney Transplant Center Performance: How to Best Define Outliers?

BACKGROUND

Scientific Registry of Transplant Recipients report cards of US organ transplant center performance are publicly available and used for quality oversight. Low center performance (LP) evaluations are associated with changes in practice including reduced transplant rates and increased waitlist removals. In 2014, Scientific Registry of Transplant Recipients implemented new Bayesian methodology to evaluate performance which was not adopted by Center for Medicare and Medicaid Services (CMS). In May 2016, CMS altered their performance criteria, reducing the likelihood of LP evaluations.

METHODS

Our aims were to evaluate incidence, survival rates, and volume of LP centers with Bayesian, historical (old-CMS) and new-CMS criteria using 6 consecutive program-specific reports (PSR), January 2013 to July 2015 among adult kidney transplant centers.

RESULTS

Bayesian, old-CMS and new-CMS criteria identified 13.4%, 8.3%, and 6.1% LP PSRs, respectively. Over the 3-year period, 31.9% (Bayesian), 23.4% (old-CMS), and 19.8% (new-CMS) of centers had 1 or more LP evaluation. For small centers (<83 transplants/PSR), there were 4-fold additional LP evaluations (52 vs 13 PSRs) for 1-year mortality with Bayesian versus new-CMS criteria. For large centers (>183 transplants/PSR), there were 3-fold additional LP evaluations for 1-year mortality with Bayesian versus new-CMS criteria with median differences in observed and expected patient survival of -1.6% and -2.2%, respectively.

CONCLUSIONS

A significant proportion of kidney transplant centers are identified as low performing with relatively small survival differences compared with expected. Bayesian criteria have significantly higher flagging rates and new-CMS criteria modestly reduce flagging. Critical appraisal of performance criteria is needed to assess whether quality oversight is meeting intended goals and whether further modifications could reduce risk aversion, more efficiently allocate resources, and increase transplant opportunities.

A Five-Tier System for Improving the Categorization of Transplant Program Performance

OBJECTIVE

To better inform health care consumers by better identifying differences in transplant program performance.

DATA SOURCE

Adult kidney transplants performed in the United States, January 1, 2012-June 30, 2014.

STUDY DESIGN

In December 2016, the Scientific Registry of Transplant Recipients instituted a five-tier system for reporting transplant program performance. We compare the differentiation of program performance and the simulated misclassification rate of the five-tier system with the previous three-tier system based on the 95 percent credible interval.

DATA COLLECTION

Scientific Registry of Transplant Recipients database.

PRINCIPAL FINDINGS

The five-tier system improved differentiation and maintained a low misclassification rate of less than 22 percent for programs differing by two tiers.

CONCLUSION

The five-tier system will better inform health care consumers of transplant program performance.

Potential Implications of Recent and Proposed Changes in the Regulatory Oversight of Solid Organ Transplantation in the United States

Every 6 months, the Scientific Registry of Transplant Recipients (SRTR) publishes evaluations of every solid organ transplant program in the United States, including evaluations of 1-year patient and graft survival. The Centers for Medicare & Medicaid Services (CMS) and the Organ Procurement and Transplantation Network (OPTN) Membership and Professional Standards Committee (MPSC) use SRTR's 1-year evaluations for regulatory review of transplant programs. Concern has been growing that the regulatory scrutiny of transplant programs with lower-than-expected outcomes is harmful, causing programs to undertake fewer high-risk transplants and leading to unnecessary organ discards. As a result, CMS raised its threshold for a "Condition-Level Deficiency" designation of observed relative to expected 1-year graft or patient survival from 1.50 to 1.85. Exceeding this threshold in the current SRTR outcomes report and in one of the four previous reports leads to scrutiny that may result in loss of Medicare funding. For its part, OPTN is reviewing a proposal from the MPSC to also change its performance criteria thresholds for program review, to review programs with "substantive clinical differences." We review the details and implications of these changes in transplant program oversight.

Performance Measurement to Demonstrate Value

Anesthesiologists are obligated to demonstrate the value of the care they provide. The Centers for Medicare and Medicaid Services has multiple performance-based payment programs to drive high-value care and motivate integrated care for surgical patients and hospitalized patients. These programs rely on diverse arrays of performance measures and complex reporting rules. Among all specialties, anesthesiology has tremendous potential to effect wide-ranging change on diverse measures. Performance measures deserve scrutiny by anesthesiologists as tools to improve care, the means by which payment is determined, and as a means to demonstrate the value of care to surgeons, hospitals, and patients.

Effects of High-Risk Kidneys on Scientific Registry of Transplant Recipients Program Quality Reports

There is a perception that transplanting high-risk kidneys causes programs to be identified as underperforming, thereby increasing the frequency of discards and diminishing access to transplant. Thus, the Organ Procurement and Transplantation Network (OPTN) has considered excluding transplants using kidneys from donors with high Kidney Donor Profile Index (KDPI) scores (≥0.85) when assessing program performance. We examined whether accepting high-risk kidneys (KDPI ≥0.85) for transplant yields worse outcome evaluations. Despite a clear relationship between KDPI and graft failure and mortality, there was no relationship between a program's use of high-KDPI kidneys and poor performance evaluations after risk adjustment. Excluding high-KDPI donor transplants from the June 2015 evaluations did not alter the programs identified as underperforming, because in every case underperforming programs also had worse-than-expected outcomes among lower-risk donor transplants. Finally, we found that hypothetically accepting and transplanting additional kidneys with KDPI similar to that of kidneys currently discarded would not adversely affect program evaluations. Based on the study findings, there is no evidence that programs that accept higher-KDPI kidneys are at greater risk for low performance evaluations, and risk aversion may limit access to transplant for candidates while providing no measurable benefit to program evaluations.

The clinical relevance of Organ Procurement and Transplantation Network screening criteria for program performance review in the United States

The Organ Procurement and Transplantation Network is charged with overseeing the quality of transplant programs in the United States. However, there has been controversy over whether too many programs are being identified as underperforming. It has also been suggested that dramatic improvements in outcomes throughout the United States have made the thresholds for determining which deceased donor transplant programs are underperforming no longer clinically relevant. The Scientific Registry of Transplant Recipients compared actual and expected 1-y graft survival for transplant programs identified as underperforming in the most recent cohort (transplants from July 1, 2012 to December 31, 2014). For most organs, actual 1-y graft survival was substantially lower for programs identified as underperforming than for programs identified as performing as expected. Differences were smallest for kidney programs: median 1-y graft survival 89.2% vs 95.4% in large-volume programs identified and not identified for Membership and Professional Standards Committee review, respectively. Median expected graft survival was only slightly lower (94.8% vs 95.1%, respectively), suggesting that identified and not identified programs tend to have similar risk tolerances. An excess of 143 grafts were lost from kidney programs identified as underperforming. Transplant programs identified as underperforming generally have reduced 1-y graft survival that stakeholders may consider clinically relevant.

Current state of clinical end-points assessment in transplant: Key points

Solid organ transplantation is the treatment of choice for patients with end-stage organ disease. However, organ transplantation can stress the cardiovascular system and decrease immune surveillance, leading to early mortality and graft loss due to multiple underlying comorbidities. Clinical end-points in transplant include death and graft failure. Thus, generating accurate predictive models through regression models is crucial to test for definitive clinical post-transplantation end-points. Survival predictive models should assemble efficient surrogate markers or prognostic factors to generate a minimal set of variables derived from a proper modeling strategy through regression models. However, a few critical points should be considered when reporting survival analyses and regression models to achieve proper discrimination and calibration of the predictive models. Additionally, population-based risk scores may underestimate risk prediction in transplant. The application of predictive models in these patients should therefore incorporate both classical and non-classical risk factors, as well as community-based health indicators and transplant-specific factors to quantify the outcomes in terms of survival properly. This review focuses on assessment of clinical end-points in transplant through regression models by combining predictive and surrogate variables, and considering key points in these analyses to accurately predict definitive end-points, which could aid clinicians in decision making.