Inter-rater reliability of distal ureteral diameter ratio compared to grade of VUR

An evaluation of ureteral diameter ratio and vesicoureteral reflux index in the treatment of primary vesicoureteral reflux

Objective

Vesicoureteral reflux (VUR) index is a simple, validated tool that reliably predicts significant improvement and spontaneous resolution of primary reflux in children. The aim of this study was to evaluate and compare the ureter diameter ratio (UDR) and VUR index (VURx) of patients treated with endoscopic injection (EI) and ureteroneocystostomy (UNC) methods in the pediatric age group due to primary VUR.

Methods

Patients under the age of 18 years old who underwent EI and UNC with the diagnosis of primary VUR between January 2011 and September 2021 were determined as the participants. The UDR was assessed using voiding cystourethrography, and the VURx score was determined prior to treatment based on hospital records included in the study.

Results

A total of 255 patients, 60 (23.5%) boys and 195 (76.5%) girls, with a mean age of 76.5 (range 13.0-204.0) months, were included in the study. EI was applied to 130 (51.0%) patients and UNC was applied to 125 (49.0%) patients due to primary VUR. The optimum cut-off for the distal UDR was obtained as 0.17 with sensitivity and specificity of 73.0% and 63.0%, respectively. The positive and negative predictive values were 66.0% and 70.0%, respectively.

Conclusion

When the UDR and VURx score are evaluated together for the surgical treatment of primary VUR in the pediatric age group, it is thought that it may be useful in predicting the clinical course of the disease and evaluating surgical treatment options.

Does the distal ureteral diameter ratio (UDR) matter in the surgical management of vesicoureteral reflux in children?

PURPOSE

To evaluate UDR reliability, sensitivity, specificity and to identify the best treatment basing on UDR among single or double endoscopic injections and ureteral reimplantation.

METHODS

Data of patients affected by primary VUR and treated by endoscopic injection over a 10 years period were retrospectively analyzed. Two radiologist attributed reflux grade and UDR on voiding cystourethrogram twice and blinded. Follow-up focused on resolution after 1 or 2 endoscopic injections. Relation between UDR, reflux grade and outcomes were analyzed.

RESULTS

Patient enrolled were 198. Low grade VUR was present in 24.8%, grade 3 in 41.6%, grade 4-5 in 33.6%. Resolution after one injection was obtained in 88 patients; among 110 not resolved 104 cases had a second injection. Success after 2 injections was reported in 138 cases. UDR showed a higher reliability compared with reflux grade both in intra than inter-reader measurement (ICC > 90%). Success after 1 or 2 injections was reported for UDR < 0.33 and UDR < 0.47 respectively.

CONCLUSION

UDR shows to be a more reliable measurement that allows for an objective estimation of VUR severity and prognosis. It represents a quantitative parameter that might be useful to identify patients who may benefit endoscopic or surgical treatment, avoiding unnecessary under or over-treatment.

Management of Vesicoureteral Reflux: What Have We Learned Over the Last 20 Years?

Vesicoureteral reflux (VUR) is associated with increased risks of urinary tract infection, renal scarring and reflux nephropathy. We review advancements over the last two decades in our understanding and management of VUR. Over time, the condition may resolve spontaneously but it can persist for many years and bladder/bowel dysfunction is often involved. Some factors that increase the likelihood of persistence (e.g., high grade) also increase the risk of renal scarring. Voiding cystourethrography (VCUG) is generally considered the definitive method for diagnosing VUR, and helpful in determining the need for treatment. However, this procedure causes distress and radiation exposure. Therefore, strategies to reduce clinicians' reliance upon VCUG (e.g., after a VUR treatment procedure) have been developed. There are several options for managing patients with VUR. Observation is suitable only for patients at low risk of renal injury. Antibiotic prophylaxis can reduce the incidence of UTIs, but drawbacks such as antibiotic resistance and incomplete adherence mean that this option is not viable for long-term use. Long-term studies of endoscopic injection have helped us understand factors influencing use and the effectiveness of this procedure. Ureteral reimplantation is still performed commonly, and robot-assisted laparoscopic methods are gaining popularity. Over the last 20 years, there has been a shift toward more conservative management of VUR with an individualized, risk-based approach. For continued treatment improvement, better identification of children at risk of renal scarring, robust evidence regarding the available interventions, and an improved VUR grading system are needed.

Reliability in evaluator-based tests: using simulation-constructed models to determine contextually relevant agreement thresholds

BACKGROUND

Indices of inter-evaluator reliability are used in many fields such as computational linguistics, psychology, and medical science; however, the interpretation of resulting values and determination of appropriate thresholds lack context and are often guided only by arbitrary "rules of thumb" or simply not addressed at all. Our goal for this work was to develop a method for determining the relationship between inter-evaluator agreement and error to facilitate meaningful interpretation of values, thresholds, and reliability.

METHODS

Three expert human evaluators completed a video analysis task, and averaged their results together to create a reference dataset of 300 time measurements. We simulated unique combinations of systematic error and random error onto the reference dataset to generate 4900 new hypothetical evaluators (each with 300 time measurements). The systematic errors and random errors made by the hypothetical evaluator population were approximated as the mean and variance of a normally-distributed error signal. Calculating the error (using percent error) and inter-evaluator agreement (using Krippendorff's alpha) between each hypothetical evaluator and the reference dataset allowed us to establish a mathematical model and value envelope of the worst possible percent error for any given amount of agreement.

RESULTS

We used the relationship between inter-evaluator agreement and error to make an informed judgment of an acceptable threshold for Krippendorff's alpha within the context of our specific test. To demonstrate the utility of our modeling approach, we calculated the percent error and Krippendorff's alpha between the reference dataset and a new cohort of trained human evaluators and used our contextually-derived Krippendorff's alpha threshold as a gauge of evaluator quality. Although all evaluators had relatively high agreement (> 0.9) compared to the rule of thumb (0.8), our agreement threshold permitted evaluators with low error, while rejecting one evaluator with relatively high error.

CONCLUSIONS

We found that our approach established threshold values of reliability, within the context of our evaluation criteria, that were far less permissive than the typically accepted "rule of thumb" cutoff for Krippendorff's alpha. This procedure provides a less arbitrary method for determining a reliability threshold and can be tailored to work within the context of any reliability index.