Funnel plots of patient-reported outcomes to evaluate health-care quality: Basic principles, pitfalls and considerations

Optimizing Hospital Performance Evaluation in Total Weight Loss Outcomes After Bariatric Surgery: A Retrospective Analysis to Guide Further Improvement in Dutch Hospitals

INTRODUCTION

Bariatric surgery aims for optimal patient outcomes, often evaluated through the percentage total weight loss (%TWL). Quality registries employ funnel plots for outcome comparisons between hospitals. However, funnel plots are traditionally used for dichotomous outcomes, requiring %TWL to be dichotomized, potentially limiting feedback quality. This study evaluates whether a funnel plot around the median %TWL has better discriminatory performance than binary funnel plots for achieving at least 20% and 25% TWL.

METHODS

All hospitals performing bariatric surgery were included from the Dutch Audit for Treatment of Obesity. A funnel plot around the median was constructed using 5-year %TWL data. Hospitals positioned above the 95% control limit were colored green and those below red. The same hospitals were plotted in the binary funnel plots for 20% and 25% TWL and colored according to their performance in the funnel plot around the median. We explored the hospital's procedural mix in relation to %TWL performance as possible explanatory factors.

RESULTS

The median-based funnel plot identified four underperforming and four outperforming hospitals, while only one underperforming and no outperforming hospitals were found with the binary funnel plot for 20% TWL. The 25% TWL binary funnel plot identified two underperforming and three outperforming hospitals. The proportion of sleeve gastrectomies performed per hospital may explain part of these results as it was negatively associated with median %TWL (β =  - 0.09, 95% confidence interval [- 0.13 to - 0.04]).

CONCLUSION

The funnel plot around the median discriminated better between hospitals with significantly worse and better performance than funnel plots for dichotomized %TWL outcomes.

Age at first sexual intercourse, age at menarche, and age at menopause: a mendelian randomization study on lung cancer risk

Background

There is increasing evidence that sex hormones are involved in the development of lung cancer, but the correlation between the reproductive behavior that changes sex hormone levels and lung cancer is not yet clear. Many previous studies have investigated the association between reproductive factors and lung cancer risk, but the results have been inconsistent. Therefore, we conducted a two-sample Mendelian randomization (MR) analysis to explore the potential relationship between age at first sexual intercourse (AFS), age at menarche, and age at menopause, and lung cancer.

Methods

We performed a MR analysis of the data from the genome-wide association study (GWAS) of European ancestry to evaluate the independent effects of three reproductive behaviors on lung cancer overall (LUCA), lung adenocarcinoma (LUAD), lung squamous cell carcinoma (LUSC), and small cell lung cancer (SCLC). We mainly used the inverse-variance weighting method for the MR analysis. Sensitivity was determined by a MR-pleiotropy residual sum and outlier analysis, a weighted median analysis, a MR-Egger analysis, and a leave-one-out analysis.

Results

The MR analysis results revealed that older AFS had a causal relationship with LUCA [odds ratio (OR) =0.6283, 95% confidence interval (CI): 0.4959-0.7961, P=0.0001), LUAD (OR =0.7042, 95% CI: 0.4967-0.9984, P=0.049), and LUSC (OR =0.6231, 95% CI: 0.4386-0.8853, P=0.0083).

Conclusions

Our results revealed a causal relationship between older AFS and a lower risk of lung cancer. Our findings emphasize the importance of providing sex education, as early sexual intercourse may have undesirable effects. In addition, early psychological treatment is also essential.

The Perils of Misinterpreting and Misusing "Publication Bias" in Meta-analyses: An Education Review on Funnel Plot-Based Methods

Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision and policy making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine, with a focus on the commonly used funnel-plot based methods. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, most statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance) and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ten or more studies. We performed a cross-sectional meta-research investigation of how publication bias was assessed in systematic reviews with meta-analyses published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Because of conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, registered reports, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term "risk of publication bias".

PRISMA Systematic Literature Review, including with Meta-Analysis vs. Chatbot/GPT (AI) regarding Current Scientific Data on the Main Effects of the Calf Blood Deproteinized Hemoderivative Medicine (Actovegin) in Ischemic Stroke

BACKGROUND

Stroke is a significant public health problem and a leading cause of death and long-term disability worldwide. Several treatments for ischemic stroke have been developed, but these treatments have limited effectiveness. One potential treatment for this condition is Actovegin®/AODEJIN, a calf blood deproteinized hemodialysate/ultrafiltrate that has been shown to have pleiotropic/multifactorial and possibly multimodal effects. The actual actions of this medicine are thought to be mediated by its ability to reduce oxidative stress, inflammation, and apoptosis and to enhance neuronal survival and plasticity.

METHODS

To obtain the most up-to-date information on the effects of Actovegin®/AODEJIN in ischemic stroke, we systematically reviewed the literature published in the last two years. This review builds upon our previous systematic literature review published in 2020, which used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method to search for and select related articles over almost two decades, between 1 January 2001 and 31 December 2019. Additionally, we compared the results of our PRISMA search (human intelligence-based) with those obtained from an interrogation of a GPT-based chatbot (ChatGPT) in order to ensure comprehensive coverage of potentially relevant studies.

RESULTS

Our updated review found limited new evidence on the use of Actovegin®/AODEJIN in ischemic stroke, although the number of articles on this subject consistently increased compared to that from our initial systematic literature review. Specifically, we found five articles up to 2020 and eight more until December 2022. While these studies suggest that Actovegin®/AODEJIN may have neuroprotective effects in ischemic stroke, further clinical trials are needed to confirm these findings. Consequently, we performed a funnel analysis to evaluate the potential for publication bias.

DISCUSSION

Our funnel analysis showed no evidence of publication bias, suggesting that the limited number of studies identified was not due to publication bias but rather due to a lack of research in this area. However, there are limitations when using ChatGPT, particularly in distinguishing between truth and falsehood and determining the appropriateness of interpolation. Nevertheless, AI can provide valuable support in conducting PRISMA-type systematic literature reviews, including meta-analyses.

CONCLUSIONS

The limited number of studies identified in our review highlights the need for additional research in this area, especially as no available therapeutic agents are capable of curing central nervous system lesions. Any contribution, including that of Actovegin (with consideration of a positive balance between benefits and risks), is worthy of further study and periodic reappraisal. The evolving advancements in AI may play a role in the near future.

Facing the challenges of PROM implementation in Dutch dialysis care: Patients' and professionals' perspectives

BACKGROUND

Patient Reported Outcome Measures (PROMs) are increasingly used in routine clinical practice to facilitate patients in sharing and discussing health-related topics with their clinician. This study focuses on the implementation experiences of healthcare professionals and patients during the early implementation phase of the newly developed Dutch set of dialysis PROMs and aims to understand the process of early implementation of PROMs from the users' perspectives.

METHODS

This is a qualitative study among healthcare professionals (physicians and nursing staff: n = 13) and patients (n = 14) of which 12 were receiving haemodialysis and 2 peritoneal dialysis. Semi-structured interviews were used to understand the barriers and facilitators that both professionals and patients encounter when starting to implement PROMs.

RESULTS

The early PROM implementation process is influenced by a variety of factors that we divided into barriers and facilitators. We identified four barriers: patient´s indifference to PROMs, scepticism on the benefits of aggregated PROM data, the limited treatment options open to doctors and organizational issues such as mergers, organizational problems and renovations. We also describe four facilitators: professional involvement and patient support, a growing understanding of the use of PROMs during the implementation, quick gains from using PROMs such as receiving instant feedback and a clear ambition on patient care such as a shared view on patient involvement and management support.

CONCLUSIONS

In this qualitative study carried out during the early implementation phase of the Dutch dialysis PROM set, we found that patients did not yet consider the PROM set to be a useful additional tool to share information with their doctor. This was despite the professionals' primary reason for using PROMs being to improve patient-doctor communication. Furthermore, the perceived lack of intervention options was frustrating for some of the professionals. We found that nurses could be important enablers of further implementation because of their intensive relationship with dialysis patients.

The immaturity of patient engagement in value-based healthcare-A systematic review

Introduction

In recent years, Value-Based Healthcare (VBHC) has been gaining traction, particularly in hospitals. A core VBHC element is patient value, i.e., what matters most to the patient and at what cost can this be delivered. This interpretation of value implies patient engagement in patient-doctor communication. Although patient engagement in direct care in the VBHC setting is well described, patient engagement at the organizational level of improving care has hardly been studied. This systematic review maps current knowledge regarding the intensity and impact of patient engagement in VBHC initiatives. We focus on the organizational level of a continuous patient engagement model.

Methods

We performed a systematic review following PRISMA guidelines using five electronic databases. The search strategy yielded 1,546 records, of which 21 studies were eligible for inclusion. Search terms were VBHC and patient engagement, or similar keywords, and we included only empirical studies in hospitals or transmural settings at the organizational level.

Results

We found that consultation, using either questionnaires or interviews by researchers, is the most common method to involve patients in VBHC. Higher levels of patient engagement, such as advisory roles, co-design, or collaborative teams are rare. We found no examples of the highest level of patient engagement such as patients co-leading care improvement committees.

Conclusion

This study included 21 articles, the majority of which were observational, resulting in a limited quality of evidence. Our review shows that patient engagement at the organizational level in VBHC initiatives still relies on low engagement tools such as questionnaires and interviews. Higher-level engagement tools such as advisory roles and collaborative teams are rarely used. Higher-level engagement offers opportunities to improve healthcare and care pathways through co-design with the people being served. We urge VBHC initiatives to embrace all levels of patient engagement to ensure that patient values find their way to the heart of these initiatives.

Use of antenatal corticosteroids among infants with gestational age at 24 to 31 weeks in 57 neonatal intensive care units of China: a cross-sectional study

BACKGROUND

Antenatal corticosteroids (ACS) can significantly improve the outcomes of preterm infants. This study aimed to describe the ACS use rates among preterm infants admitted to Chinese neonatal intensive care units (NICU) and to explore perinatal factors associated with ACS use, using the largest contemporary cohort of very preterm infants in China.

METHODS

This cross-sectional study enrolled all infants born at 24 +0 to 31 +6 weeks and admitted to 57 NICUs of the Chinese Neonatal Network from January 1st, 2019 to December 30th, 2019. The ACS administration was defined as at least one dose of dexamethasone and betamethasone given before delivery. Multiple logistic regressions were applied to determine the association between perinatal factors and ACS usage.

RESULTS

A total of 7828 infants were enrolled, among which 6103 (78.0%) infants received ACS. ACS use rates increased with increasing gestational age (GA), from 177/259 (68.3%) at 24 to 25 weeks' gestation to 3120/3960 (78.8%) at 30 to 31 weeks' gestation. Among infants exposed to ACS, 2999 of 6103 (49.1%) infants received a single complete course, and 33.4% (2039/6103) infants received a partial course. ACS use rates varied from 30.2% to 100% among different hospitals. Multivariate regression showed that increasing GA, born in hospital (inborn), increasing maternal age, maternal hypertension and premature rupture of membranes were associated with higher likelihood to receive ACS.

CONCLUSIONS

The use rate of ACS remained low for infants at 24 to 31 weeks' gestation admitted to Chinese NICUs, with fewer infants receiving a complete course. The use rates varied significantly among different hospitals. Efforts are urgently needed to propose improvement measures and thus improve the usage of ACS.

Funnel plots a graphical instrument for the evaluation of population performance and quality of trauma care: a blueprint of implementation

BACKGROUND

Using patient outcomes to monitor medical centre performance has become an essential part of modern health care. However, classic league tables generally inflict stigmatization on centres rated as "poor performers", which has a negative effect on public trust and professional morale. In the present study, we aim to illustrate that funnel plots, including trends over time, can be used as a method to control the quality of data and to monitor and assure the quality of trauma care. Moreover, we aimed to present a set of regulations on how to interpret and act on underperformance or overperformance trends presented in funnel plots.

METHODS

A retrospective observational cohort study was performed using the Dutch National Trauma Registry (DNTR). Two separate datasets were created to assess the effects of healthy and multiple imputations to cope with missing values. Funnel plots displaying the performance of all trauma-receiving hospitals in 2020 were generated, and in-hospital mortality was used as the main indicator of centre performance. Indirect standardization was used to correct for differences in the types of cases. Comet plots were generated displaying the performance trends of two level-I trauma centres since 2017 and 2018.

RESULTS

Funnel plots based on data using healthy imputation for missing values can highlight centres lacking good data quality. A comet plot illustrates the performance trend over multiple years, which is more indicative of a centre's performance compared to a single measurement. Trends analysis offers the opportunity to closely monitor an individual centres' performance and direct evaluation of initiated improvement strategies.

CONCLUSION

This study describes the use of funnel and comet plots as a method to monitor and assure high-quality data and to evaluate trauma centre performance over multiple years. Moreover, this is the first study to provide a regulatory blueprint on how to interpret and act on the under- or overperformance of trauma centres. Further evaluations are needed to assess its functionality.

LEVEL OF EVIDENCE

Retrospective study, level III.

The psychometric properties of the Turkish version of the Self-Regulatory Fatigue Scale

PURPOSE

The aim of the study was to translate and adapt the Self-Regulatory Fatigue Scale (SRF-18) into Turkish and also demonstrate its reliability and validity in young adults.

DESIGN AND METHODS

A total of 181 participants completed SRF-18, Life Orientation Test-Revised (LOT-R), Brief Self-Control Scale (BSCS), and Multidimensional Fatigue Inventory (MFI). One week later 30 participants refilled SRF-18.

FINDINGS

The internal consistency of SRF-18 was good (α = 0.710). The intraclass correlation coefficient value of the SRF-18 total score was 0.608. SRF-18 was strongly related to MFI and LOT-R (r₁  = -0.632, r₂  = 0.557, p < 0.001). Besides, there was a moderate relationship between SRF-18 and BSCS (r = 0.439, p < 0.001). In addition, SRF-18 had a three-factor structure.

PRACTICE IMPLICATIONS

The Turkish version of the SRF-18 was proven to be a valid and reliable tool.

Funnel plots of patient-reported outcomes to evaluate health-care quality: Basic principles, pitfalls and considerations

A funnel plot is a graphical method to evaluate health-care quality by comparing hospital performances on certain outcomes. So far, in nephrology, this method has been applied to clinical outcomes like mortality and complications. However, patient-reported outcomes (PROs; eg, health-related quality of life [HRQOL]) are becoming increasingly important and should be incorporated into this quality assessment. Using funnel plots has several advantages, including clearly visualized precision, detection of volume-effects, discouragement of ranking hospitals and easy interpretation of results. However, without sufficient knowledge of underlying methods, it is easy to stumble into pitfalls, such as overinterpretation of standardized scores, incorrect direct comparisons of hospitals and assuming a hospital to be in-control (ie, to perform as expected) based on underpowered comparisons. Furthermore, application of funnel plots to PROs is accompanied by additional challenges related to the multidimensional nature of PROs and difficulties with measuring PROs. Before using funnel plots for PROs, high and consistent response rates, adequate case mix correction and high-quality PRO measures are required. In this article, we aim to provide insight into the use and interpretation of funnel plots by presenting an overview of the basic principles, pitfalls and considerations when applied to PROs, using examples from Dutch routine dialysis care.