Individual surgeon mortality rates: can outliers be detected? A national utility analysis

Mortality as an indicator of quality of neurosurgical care in England: a retrospective cohort study

OBJECTIVES

Postoperative mortality is a widely used quality indicator, but it may be unreliable when procedure numbers and/or mortality rates are low, due to insufficient statistical power. The objective was to investigate the statistical validity of postoperative 30-day mortality as a quality metric for neurosurgical practice across healthcare providers.

DESIGN

Retrospective cohort study.

SETTING

Hospital Episode Statistics data from all neurosurgical units in England.

PARTICIPANTS

Patients who underwent neurosurgical procedures between April 2013 and March 2018. Procedures were grouped using the National Neurosurgical Audit Programme classification.

OUTCOMES MEASURED

National 30-day postoperative mortality rates were calculated for elective and non-elective neurosurgical procedural groups. The study estimated the proportion of neurosurgeons and NHS trusts in England that performed sufficient procedures in 3-year and 5-year periods to detect unusual performance (defined as double the national rate of mortality). The actual difference in mortality rates that could be reliably detected based on procedure volumes of neurosurgeons and units over a 5-year period was modelled.

RESULTS

The 30-day mortality rates for all elective and non-elective procedures were 0.4% and 6.1%, respectively. Only one neurosurgeon in England achieved the minimum sample size (n=2402) of elective cases in 5 years needed to detect if their mortality rate was double the national average. All neurosurgical units achieved the minimum sample sizes for both elective (n=2402) and non-elective (n=149) procedures. In several neurosurgical subspecialties, approximately 80% of units (or more) achieved the minimum sample sizes needed to detect if their mortality rate was double the national rate, including elective neuro-oncology (baseline mortality rate=2.3%), non-elective neuro-oncology (rate=5.7%), neurovascular (rate=6.7%) and trauma (rate=11%).

CONCLUSION

Postoperative mortality lacks statistical power as a measure of individual neurosurgeon performance. Neurosurgical units in England performed sufficient procedure numbers overall and in several subspecialty areas to support the use of mortality as a quality indicator.

Reliable consultant level data from an Acute Medical Unit: a powerful tool for improvement

The development of a novel database interrogating the patient management system in the Acute Medical Unit at Forth Valley Royal Hospital, Scotland, has allowed, for the first time, acquisition of reliable individual consultant-level process and outcome data over a 2-year period. These data have a number of uses, including understanding the level of variation between consultant physicians in AMU across key indicators, such as direct discharge percentage (67.5-44.3%), and readmission percentage (4.0-6.8%). Looking at overnight admissions only effectively excluded case mix as a confounder to identify variation in 30-day mortality (0-2.8%). This has allowed benchmarking, and exploring of relationships between volume of work, physician experience, and patient outcomes. For example, no significant relationship was seen between direct discharge percentage and readmission percentage. Furthermore it is extremely useful for individual clinician appraisal and governance. Finally it has practical uses when designing consultant rotas in order to minimise system variation. A key consideration throughout this work has been clear provenance and local clinical ownership of these data, unlike centrally generated data that may not accurately reflect Acute Medical Unit activity.

Impact of surgical volume and surgical outcome assessing registers on the quality of thyroid surgery

The available evidence concerning the relationship between volume and outcome for thyroid surgery is assessed in this article. Morbidity forms the principal surrogate marker of thyroid surgery quality for which postoperative hypocalcaemia and recurrent laryngeal nerve injuries are most commonly reported upon. Whilst there is an abundance of published data for these outcomes, interpretation to recommend annual volume thresholds is challenging. This is due to a lack of consensus on definitions not only for outcomes but high and low volume surgeons. The evidence reviewed in this article supports the notion that high volume surgeons achieve superior outcomes in thyroid surgery quality though it is not possible to recommend minimal annual volumes on the basis of this evidence alone. Every thyroid surgeon should know their own outcomes and how they compare with their peers and engagement in thyroid surgery registries can facilitate this.

The use of an electronic health record system reduces errors in the National Hip Fracture Database

AIM

to compare the validity of data submitted from a UK level 1 trauma centre to the National Hip Fracture Database (NHFD) before and after the introduction of an electronic health record system (EHRS).

PATIENTS AND METHODS

a total of 3224 records were reviewed from July 2009 to July 2017. 2,133 were submitted between July 2009 and October 2014 and 1,091 between October 2014 and July 2017, representing data submitted before and after the introduction of the EHRS, respectively. Data submitted to the NHFD were scrutinised against locally held data.

RESULTS

use of an EHRS was associated with significant reductions in NHFD errors. The operation coding error rate fell significantly from 23.2% (494/2133) to 7.6% (83/1091); P < 0.001. Prior to EHRS introduction, of the 109 deaths recorded in the NHFD, 64 (59%) were incorrect. In the EHRS dataset, all the 112 recorded deaths were correct (P < 0.001). There was no significant difference in the error rate for fracture coding. In the EHRS dataset, after controlling for sample month, entries utilising an operation note template with mandatory fields relevant to NHFD data were more likely to be error free than those not using the template (OR 2.69; 95% CI 1.92-3.78).

CONCLUSION

this study highlights a potential benefit of EHR systems, which offer automated data collection for auditing purposes. However, errors in data submitted to the NHFD remain, particularly in cases where an NHFD-specific operation note template is not used. Clinician engagement with new technologies is vital to avoid human error and ensure database integrity.

Creating Individual Surgeon Performance Assessments in a Statewide Hospital Surgical Quality Improvement Collaborative

BACKGROUND

Surgeon performance profiling is of great interest to surgeons, hospitals, health plans, and the public, yet efforts to date have been contested, with stakeholders at odds over the selection, reliability, and validity of metrics used. We sought to create surgeon-level comparative assessments within the Illinois Surgical Quality Improvement Collaborative.

STUDY DESIGN

American College of Surgeons NSQIP data were obtained for 51 Illinois hospitals covering a 30-month period from 2014 to 2016. Surgeon-level, risk-adjusted outcomes rates were estimated from 3-level crossed random effects logistic regression models and classified as low, as expected, or high for each of 7 postoperative outcomes. Model intra-class correlations and provider-specific reliability statistics were calculated.

RESULTS

A total of 123,141 cases were analyzed for 2,724 surgeons. Median provider case volume was 17 (interquartile range 4 to 54). Overall crude complication rates ranged from 0.62% to 7.14% across the 7 outcomes investigated. Surgeon-level variance estimates were low (intra-class correlation coefficients between 0.007 and 0.074). No performance outliers were detected for 3 of the outcomes measures, while a small number of outliers were identified for any morbidity (11 surgeons), surgical site infection (10 surgeons), death or serious morbidity (8 surgeons), and reoperation (1 surgeon). Among all physicians, median reliability was below 0.1 for each outcome.

CONCLUSIONS

Few individual surgeon performance outliers could be detected in NSQIP clinical registry data for a statewide hospital collaborative over a 30-month period using postoperative patient outcomes. Low surgeon-specific case volumes and minimal variance between surgeons may limit the utility of American College of Surgeons NSQIP outcomes measures for individual profiling. Alternative metrics, such as process measures, patient experience, composite measures, or technical skill assessments should be explored for surgeon-level measurement.