Evaluation of Flagging Criteria of United States Kidney Transplant Center Performance: How to Best Define Outliers?

Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Using the Medical Audit to Improve Practice Performance

Feedback to physicians on their clinical performance is critical to continuous learning and maintenance of skills as well as maintaining patient safety. However, it is fraught with challenges around both implementation and acceptance. Additionally, rewarding of performance improvement is not often done, putting into question the efficacy of the process. Physician audit and feedback have been studied extensively and shown to be beneficial in many fields of medicine. Documenting physician performance and sharing individual and group data have been positively linked to changing physician behavior, ultimately leading to improved patient outcomes. Although casual review of one's own performance is often the easiest approach, it is frequently over- or underestimated by self-evaluation. Objective measures are therefore important to provide concrete data on which physicians can act. A fundamental question remains in mammography: Is reporting the information to the physician and accreditation bodies enough, or should there be consequences for the radiologist and/or facility if there is outlier behavior?

Kidney transplant program specific reporting and transplant metrics

PURPOSE OF REVIEW

Kidney transplantation is a heavily regulated medical procedure with the Secretary of HHS ultimately responsible for oversight and authority derived from the NOTA and the Final Rule. Transplant Programs undergo publicly reported evaluations every 6 months based on outcomes from a 2-and-a-half-year period. The current Bayesian metrics for kidney transplant programs were created such that over ten percentage of programs are deemed underperformers, or 'flag', every 6 months. Newly suggested transplant metrics have been released for public comment in Summer 2021. In addition to graft outcomes, waiting list mortality and organ acceptance rate ratios are proposed.

RECENT FINDINGS

Under the newly proposed kidney transplant metrics, over 10% of programs are expected to be deemed underperformers or 'flagged'. Transplant Center flagging is well correlated with decreased transplantation due to the transplant centres move to more conservative organ and patient acceptance. Death on the waiting list is a proposed metric over which transplant centres have little influence.

SUMMARY

In the USA, the harsh regulation continued by Health Resources and Services Administration (HRSA) through the national organ procurement and transplant network (OPTN) and Scientific Registry for Transplant Recipients (SRTR) leads directly to high organ discard rates and limitations to transplanting patients with perceived unadjusted risks. Instead of loosening regulation in a highly functioning industry that achieves remarkable outcomes in end stage kidney patients, the OPTN with the SRTR persist in increasing potential penalties through more proposed metrics that continue to deem 10% of US kidney transplant programs as underperformers. HRSA must establish a reasonable regulatory environment that allows for innovation and increased transplant opportunities for US end-stage renal disease patients.

Definition and Analysis of Textbook Outcome: A Novel Quality Measure in Kidney Transplantation

BACKGROUND

"Textbook outcome" (TO) is a novel composite quality measure that encompasses multiple postoperative endpoints, representing the ideal "textbook" hospitalization for complex surgical procedures. We defined TO for kidney transplantation using a cohort from a high-volume institution.

METHODS

Adult patients who underwent isolated kidney transplantation at our institution between 2016 and 2019 were included. TO was defined by clinician consensus at our institution to include freedom from intraoperative complication, postoperative reintervention, 30-day intensive care unit or hospital readmission, length of stay > 75th percentile of kidney transplant patients, 90-day mortality, 30-day acute rejection, delayed graft function, and discharge with a Foley catheter. Recipient, operative, financial characteristics, and post-transplant patient, graft, and rejection-free survival were compared between patients who achieved and failed to achieve TO.

RESULTS

A total of 557 kidney transplant patients were included. Of those, 245 (44%) achieved TO. The most common reasons for TO failure were delayed graft function (N = 157, 50%) and hospital readmission within 30 days (N = 155, 50%); the least common was mortality within 90 days (N = 6, 2%). Patient, graft, and rejection-free survival were significantly improved among patients who achieved TO. On average, patients who achieved TO incurred approximately $50,000 less in total inpatient charges compared to those who failed TO.

CONCLUSIONS

TO in kidney transplantation was associated with favorable post-transplant outcomes and significant cost-savings. TO may offer transplant centers a detailed performance breakdown to identify aspects of perioperative care in need of process improvement.

Wait expectations: The impact of delisting as an outcome from the kidney transplant waitlist

BACKGROUND

While kidney transplantation is optimal for the treatment of end-stage kidney disease, available organs do not meet demand. Little is known about the outcomes of patients who are delisted (removed from the waitlist) and unable to benefit from transplant. We describe patients who are delisted and their life expectancy after delisting.

METHODS

Patients ≥ 18 years listed for deceased donor kidney transplant between 01/2003 and 12/2013 were identified in the Scientific Registry of Transplant Recipients and followed through 12/2018. A competing risk model was used to measure the association of demographic and clinical factors with waitlist outcomes of delisting, transplant, and death. Multivariate Cox modeling was used to evaluate factors associated with death after delisting.

RESULTS

Of 324,582 patients listed, 18.0% were delisted, most common reasons were "too sick" or "other." After delisting, half (49.7%) had died by end of follow-up; time to death after removal was 5 years. Increasing age and public insurance were associated with increased risk of death.

CONCLUSIONS

Nearly one in five patients will be delisted from the kidney transplant waitlist. These patients live a surprisingly long time after removal. Much remains unknown about these patients, which could be improved through data collection. Delisting is an important patient outcome that warrants further exploration.

Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre

BACKGROUND

A growing number of research studies have reported inter-observer variability in sizes of tumours measured from CT scans. It remains unclear whether the conventional statistical measures correctly evaluate the CT measurement consistency for optimal treatment management and decision-making. We compared and evaluated the existing measures for evaluating inter-observer variability in CT measurement of cancer lesions.

METHODS

13 board-certified radiologists repeatedly reviewed 10 CT image sets of lung lesions and hepatic metastases selected through a randomisation process. A total of 130 measurements under RECIST 1.1 (Response Evaluation Criteria in Solid Tumors) guidelines were collected for the demonstration. Intraclass correlation coefficient (ICC), Bland-Altman plotting and outlier counting methods were selected for the comparison. The each selected measure was used to evaluate three cases with observed, increased and decreased inter-observer variability.

RESULTS

The ICC score yielded a weak detection when evaluating different levels of the inter-observer variability among radiologists (increased: 0.912; observed: 0.962; decreased: 0.990). The outlier counting method using Bland-Altman plotting with 2SD yielded no detection at all with its number of outliers unchanging regardless of level of inter-observer variability. Outlier counting based on domain knowledge was more sensitised to different levels of the inter-observer variability compared with the conventional measures (increased: 0.756; observed: 0.923; improved: 1.000). Visualisation of pairwise Bland-Altman bias was also sensitised to the inter-observer variability with its pattern rapidly changing in response to different levels of the inter-observer variability.

CONCLUSIONS

Conventional measures may yield weak or no detection when evaluating different levels of the inter-observer variability among radiologists. We observed that the outlier counting based on domain knowledge was sensitised to the inter-observer variability in CT measurement of cancer lesions. Our study demonstrated that, under certain circumstances, the use of standard statistical correlation coefficients may be misleading and result in a sense of false security related to the consistency of measurement for optimal treatment management and decision-making.

Organ utilization - the next hurdle in transplantation?

Nonutilization of organs from consented deceased donors remains a significant factor in limiting patient access to transplantation. Critical to reducing waste is a clear understanding of why organs are not used: accurate metrics are essential to identify the extent and causes of waste but use of these measures as targets or comparators between units/jurisdictions must be done with caution as focus on any one measure may result in unintended adverse consequences. Comparison between centres or countries may be misleading because of variation in definitions, patient or graft characteristics. Two of the most challenging areas to improve appropriate deceased donor organ utilization are appetite for risk and lack of validated tools to help identify an organ that will function appropriately. Currently, the implanting surgeon is widely considered to be accountable for the use of a donated organ so guidelines must be clear to allow and support sensible decisions and recognition that graft failure or inadvertent disease transmission are not necessarily attributable to poor decision-making. Accepting an organ involves balancing risk and benefit for the potential recipient. Novel technologies such as machine perfusion may allow for more robust guidance as to the functioning of the organ.

The structural conundrum of parolees and kidney transplantation

In the United States, there are a large number of incarcerated individuals, resulting in high numbers of previously incarcerated individuals out on parole undergoing reentry into society. An aging prison population translates to an older parolee population and increased incidence of kidney disease requiring either long-term dialysis or transplantation. This paper argues that due to challenges specific to the parolee population as well as societal biases and priorities, Transplant Centers and healthcare professionals face an ethical imperative to attend to the needs of parolees as a class and take steps to address challenges related to access to Centers for renal transplantation evaluation for this disadvantaged group. It will first review the regulatory context of kidney transplantation and highlight the specific ways it effects parolees. The paper will then discuss the broader social context of parolee reentry into society and barriers faced by parolees in this process. This ethical analysis examines the complexity of these issues, and deliberates on ways to balance the competing priorities of justice, respect for this patient population as individuals and as a disadvantaged class, and the societal interests regarding organ allocation and considerable economic burdens of end-stage renal disease on parolees, the justice system, and the public.

A simple risk-based reimbursement system for kidney transplant

Transplant centers were challenged by the Executive Order on Advancing Kidney health to increase access to kidney transplant (KTx) by accepting higher risk patients and organs. However, Medicare reimbursement for KTx does not include adjustment for major complicating comorbidities (MCCs) like other transplants. The prevalence of MCCs was assessed for KTx performed from 10/15 to 10/19 at a single academic center, using Medicare ICD10 MCC criteria exclusive of end-stage kidney disease. KTx hospital resource utilization and estimated margin, assuming Medicare reimbursement, were determined for cases with and without MCC. Among 260 KTx recipients, 49 (19%) had an MCC. Patients with MCCs had longer wait times (1121 days vs 703 days, P < .001); however, there were no differences in age, gender, race, or diagnosis. Donor characteristics associated with an MCC included greater cold ischemic time (1042 vs 670 minutes, P < .001) and fewer living donor KTx (9% vs 32%, P < .001). KTx cost, exclusive of organ acquisition, was 31% higher (MCC: $38 293 vs No MCC: $29 132) and estimated margin was markedly lower (-$7750 vs -$1001, P = .001). In conclusion, KTx with qualifying MCCs resulted in significant financial losses and modification of KTx payment methodology to align with other organ transplants is needed.

Now is the time for the Organ Procurement and Transplantation Network to change regulatory policy to effectively increase transplantation in the United States; Carpe Diem

With the Centers for Medicare and Medicaid Services proposing to remove outcome measures from the transplant centers' renewal for Conditions of Participation an exciting opportunity surfaces for the Organ Procurement and Transplantation Network to make an equally bold change and allow for increased transplantation options for patients in the United States.

Implications of the Advancing American Kidney Health Initiative for kidney transplant centers

The announcement of the Advancing American Kidney Health (AAKH) Initiative on July 10, 2019 was met with a mix of excitement and trepidation, befitting a proposed radical reconfiguration of the delivery of kidney disease care. Aspiring to reduce the incidence of end-stage renal disease, increase the prevalence of home dialysis, and double the number of organs available for transplant, the AAKH payment models primarily focus on incenting behaviors of general nephrologists, though actualizing positive incentives will require the active cooperation of dialysis providers and transplant centers. Here, we review the AAKH initiatives' potential impact on all stakeholders and opine on financial and regulatory pressures on kidney transplant programs, outlining areas of uncertainty and concern, and suggest key points of reflection for clinical and administrative leaders of kidney transplant centers weighing participation in any of the voluntary payment models.

Variation in use of procurement biopsies and its implications for discard of deceased donor kidneys recovered for transplantation

The use of procurement biopsies in deceased donor kidney acceptance is controversial. We analyzed Scientific Registry of Transplant Recipients data (n = 59 328 allografts, 2014-2018) to describe biopsy practices across US organ procurement organizations (OPOs) and examine relationships with discards, using hierarchical modeling to account for OPO and donor factors. Median odds ratios (MORs) provide the median of the odds that allografts with identical reported traits would be biopsied or discarded from 2 randomly drawn OPOs. Biopsies were obtained for 52.7% of kidneys. Biopsy use rose in a graded manner with kidney donor profile index (KDPI). Biopsy rates differed significantly among OPOs (22.8% to 77.5%), even after adjustment for KDPI and other donor factors. Discard rates also varied from 6.6% to 32.1% across OPOs. After adjustment for donor factors and OPO, biopsy was associated with more than 3 times the likelihood of discard (adjusted odds ratio [_95%LCL aOR_95%UCL ], _3.29 3.51_3.76 ). This association was most pronounced for low-risk (KDPI <20) kidneys (aOR, _5.45 6.47_7.69 ), with minimal impact at KDPI >85 (aOR, _0.88 1.15_1.51 ). Adjusted MORs for kidney discard and biopsy were greatest for low-risk kidneys. Reducing the rate of unnecessary biopsy and improving the accuracy of histologic assessments in higher KDPI organs may help reduce graft discard rates.

Quality Metrics in Solid Organ Transplantation: A Systematic Review

BACKGROUND

The best approach for determining whether a transplant program is delivering high-quality care is unknown. This review aims to identify and characterize quality metrics in solid organ transplantation.

METHODS

Medline, Embase, and Cochrane Central Register of Controlled Trials were searched from inception until February 1, 2017. Relevant full text reports and conference abstracts that examined quality metrics in organ transplantation were included. Two reviewers independently extracted study characteristics and quality metrics from 52 full text reports and 24 abstracts. PROSPERO registration: CRD42016035353.

RESULTS

Three hundred seventeen quality metrics were identified and condensed into 114 unique indicators with sufficient detail to be measured in practice; however, many lacked details on development and selection, were poorly defined, or had inconsistent definitions. The process for selecting quality indicators was described in only 5 publications and patient involvement was noted in only 1. Twenty-four reports used the indicators in clinical care, including 12 quality improvement studies. Only 14 quality metrics were assessed against patient and graft survivals.

CONCLUSIONS

More than 300 quality metrics have been reported in transplantation but many lacked details on development and selection, were poorly defined, or had inconsistent definitions. Measures have focused on safety and effectiveness with very few addressing other quality domains, such as equity and patient-centeredness. Future research will need to focus on transparent and objective metric development with proper testing, evaluation, and implementation in practice. Patients will need to be involved to ensure that transplantation quality metrics measure what is important to them.

Measuring transplant center performance: The goals are not controversial but the methods and consequences can be

PURPOSE OF REVIEW

Risks of regulatory scrutiny has generated widespread concern about increasingly risk averse transplant center behaviors regarding both donor and candidate acceptance patterns. To address potential unintended consequences threatening access to care, we discuss recent changes in regulatory metrics and potential improvements in quality oversight of transplant centers.

RECENT FINDINGS

Despite many recent changes to one-year patient and graft survival regulatory criteria, the capacity to accurately identify true underperforming centers and avoiding false positive flagging remains an area of great concern. Numerous studies have demonstrated restrictions in transplant volume and access following transplant center flagging.

SUMMARY

Current regulatory criteria are limited in their capacity to accurately identify poorly performing centers and potentially encourage risk-averse behavior by transplant centers. Efforts to address these concerns should focus on (1) improving risk-adjustment models with better data which captures the acuity of candidate and donor risk, (2) reconsidering primary outcomes measured to assess comprehensive transplant center performance, (3) improving education to address rational or perceived disincentives, and (4) using data more effectively to share best practices.