Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol

The voice characterisation checklist: psychometric properties of a brief clinical assessment of voices as social agents

AIM

There is growing interest in tailoring psychological interventions for distressing voices and a need for reliable tools to assess phenomenological features which might influence treatment response. This study examines the reliability and internal consistency of the Voice Characterisation Checklist (VoCC), a novel 10-item tool which assesses degree of voice characterisation, identified as relevant to a new wave of relational approaches.

METHODS

The sample comprised participants experiencing distressing voices, recruited at baseline on the AVATAR2 trial between January 2021 and July 2022 (n = 170). Inter-rater reliability (IRR) and internal consistency analyses (Cronbach's alpha) were conducted.

RESULTS

The majority of participants reported some degree of voice personification (94%) with high endorsement of voices as distinct auditory experiences (87%) with basic attributes of gender and age (82%). While most identified a voice intention (75%) and personality (76%), attribution of mental states (35%) to the voice ('What are they thinking?') and a known historical relationship (36%) were less common. The internal consistency of the VoCC was acceptable (10 items, α = 0.71). IRR analysis indicated acceptable to excellent reliability at the item-level for 9/10 items and moderate agreement between raters' global (binary) classification of more vs. less highly characterised voices, κ = 0.549 (95% CI, 0.240-0.859), p < 0.05.

CONCLUSION

The VoCC is a reliable and internally consistent tool for assessing voice characterisation and will be used to test whether voice characterisation moderates treatment outcome to AVATAR therapy. There is potential wider utility within clinical trials of other relational therapies as well as routine clinical practice.

Differences between physical therapist ratings, self-ratings, and posturographic measures when assessing static balance exercise intensity

Introduction

In order for balance therapy to be successful, the training must occur at the appropriate dosage. However, physical therapist (PT) visual evaluation, the current standard of care for intensity assessment, is not always effective during telerehabilitation. Alternative balance exercise intensity assessment methods have not previously been compared to expert PT evaluations. The aim of this study was therefore to assess the relationship between PT participant ratings of standing balance exercise intensity and balance participant self-ratings or quantitative posturographic measures.

Methods

Ten balance participants with age or vestibular disorder-related balance concerns completed a total of 450 standing balance exercises (three trials each of 150 exercises) while wearing an inertial measurement unit on their lower back. They provided per-trial and per-exercise self-ratings of balance intensity on a scale from 1 (steady) to 5 (loss of balance). Eight PT participants reviewed video recordings and provided a total of 1,935 per-trial and 645 per-exercise balance intensity expert ratings.

Results

PT ratings were of good inter-rater reliability and significantly correlated with exercise difficulty, supporting the use of this intensity scale. Per-trial and per-exercise PT ratings were significantly correlated with both self-ratings (r = 0.77-0.79) and kinematic data (r = 0.35-0.74). However, the self-ratings were significantly lower than the PT ratings (difference of 0.314-0.385). Resulting predictions from self-ratings or kinematic data agreed with PT ratings approximately 43.0-52.4% of the time, and agreement was highest for ratings of a 5.

Discussion

These preliminary findings suggested that self-ratings best indicated two intensity levels (i.e., higher/lower) and sway kinematics were most reliable at intensity extremes.

Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC₁ was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC₁ ranging from 0.071 to 0.535). After training, the multi-rater AC₁ ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC₁: 0.405, 95%-CI 0.149-0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.

Association between abdominal obesity and diabetic retinopathy in patients with diabetes mellitus: A systematic review and meta-analysis

OBJECTIVE

Previous studies have reported different opinions regarding the association between abdominal obesity and diabetic retinopathy (DR) in patients with diabetes mellitus (DM). In this study, we aimed to investigate this problem through a systematic review and meta-analysis to provide a basis for clinical interventions.

METHODS

A comprehensive search was conducted in the PubMed, Embase, and Web of Science databases up to May 1, 2022, for all eligible observational studies. Standardized mean differences (SMD) and 95% confidence intervals (CI) were evaluated using a random-effects model in the Stata software. We then conducted, publication bias assessment, heterogeneity, subgroup and sensitivity analyses.

RESULTS

A total of 5596 DR patients and 17907 non-DR patients were included from 24 studies. The results of the meta-analysis of abdominal obesity parameters showed statistically significant differences between DR and non-DR patients in both type 1 and type 2 diabetes. Waist circumference (WC) was higher in patients with DR than in the non-DR patients. In the waist-hip ratio (WHR) subgroup, the level of WHR was higher in patients with DR than that in non-DR patients. The association between abdominal obesity and mild to moderate nonproliferative DR or vision-threatening DR groups did not show any statistical difference. Subgroup analysis according to ethnicity showed that Caucasians had higher levels of combined abdominal obesity parameters than Asians.

CONCLUSION

We found that abdominal obesity measured by WC and WHR is associated with DR in patients with type 1 and type 2 diabetes. This association is stronger in Caucasians than in Asians, where isolated abdominal obesity might be more related to DR. However, no correlation was found between abdominal obesity and varying degrees of diabetic retinopathy. Further prospective cohort studies with larger sample sizes are yet to be conducted to clarify our findings.

A Study of Improving Independent Walking Outcomes in Children Who Are Blind or Have Low Vision Aged 5 Years and Younger

Walking alone (without assistance) is considered by the World Health Organization as the terminal independent walking milestone, typically achieved by 18 months. Children born with severe visual impairment and blindness (visual impairments) do not achieve independent walking milestones on time. This study examined independent walking in 35 children with visual impairments (VIs) aged 14–69 months. Before treatment, they had limited to no anticipatory control. Wearing the pediatric belt cane (PBC) provided them with consistent, tactile anticipatory control. Thirty-five participants with VIs’ independent walking milestones were analyzed using pre- and post-video recordings provided weekly 1 month before and the 3 months after obtaining the PBC. The PBC consisted of a rectangular frame that connects to a custom-made belt with magnets. The PBC provided no weight support. While none of the participants had motor impairments, 80% of them were receiving physical therapy services prior to obtaining a PBC. Fifty-four percent were receiving orientation and mobility services, and half were equipped with hand-held mobility tools (long canes, adaptive mobility devices, push toys, and reverse walkers). A paired samples t-test indicated a significant difference between the before and with PBC scores ( N = 35, t = −13.0, df = 34, p = .000). Participants demonstrated significantly improved independent walking after wearing their PBC regularly over 3 months. Before PBC, the participants demonstrated delayed independent walking milestones. Most participants walked only with assistance. Those who walked independently experienced unavoidable bodily collisions and used a slow pace with poor gait patterns. After being provided PBCs, these same participants’ independent walking milestones improved, all but three were walking independently and five were observed independently running. Wearing their PBCs allowed them to perform motor skills, like running, often described as needing to be specifically taught to learners with VIs due to their inability to learn incidentally through visual observation.

Assessor burden, inter-rater agreement and user experience of the RoB-SPEO tool for assessing risk of bias in studies estimating prevalence of exposure to occupational risk factors: An analysis from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury

BACKGROUND

As part of the development of the World Health Organization (WHO)/International Labour Organization (ILO) Joint Estimates of the Work-related Burden of Disease and Injury, WHO and ILO carried out several systematic reviews to determine the prevalence of exposure to selected occupational risk factors. Risk of bias assessment for individual studies is a critical step of a systematic review. No tool existed for assessing the risk of bias in prevalence studies of exposure to occupational risk factors, so WHO and ILO developed and pilot tested the RoB-SPEO tool for this purpose. Here, we investigate the assessor burden, inter-rater agreement, and user experience of this new instrument, based on the abovementioned WHO/ILO systematic reviews.

METHODS

Twenty-seven individual experts applied RoB-SPEO to assess risk of bias. Four systematic reviews provided a total of 283 individual assessments, carried out for 137 studies. For each study, two or more assessors independently assessed risk of bias across the eight RoB-SPEO domains selecting one of RoB-SPEO's six ratings (i.e., "low", "probably low", "probably high", "high", "unclear" or "cannot be determined"). Assessors were asked to report time taken (i.e. indicator of assessor burden) to complete each assessment and describe their user experience. To gauge assessor burden, we calculated the median and inter-quartile range of times taken per individual risk of bias assessment. To assess inter-rater reliability, we calculated a raw measure of inter-rater agreement (Pi) for each RoB-SPEO domain, between Pi = 0.00, indicating no agreement and Pi = 1.00, indicating perfect agreement. As subgroup analyses, Pi was also disaggregated by systematic review, assessor experience with RoB-SPEO (≤10 assessments versus > 10 assessments), and assessment time (tertiles: ≤25 min versus 26-66 min versus ≥ 67 min). To describe user experience, we synthesised the assessors' comments and recommendations.

RESULTS

Assessors reported a median of 40 min to complete one assessment (interquartile range 21-120 min). For all domains, raw inter-rater agreement ranged from 0.54 to 0.82. Agreement varied by systematic review and assessor experience with RoB-SPEO between domains, and increased with increasing assessment time. A small number of users recommended further development of instructions for selected RoB-SPEO domains, especially bias in selection of participants into the study (domain 1) and bias due to differences in numerator and denominator (domain 7).

DISCUSSION

Overall, our results indicated good agreement across the eight domains of the RoB-SPEO tool. The median assessment time was comparable to that of other risk of bias tools, indicating comparable assessor burden. However, there was considerable variation in time taken to complete assessments. Additional time spent on assessments may improve inter-rater agreement. Further development of the RoB-SPEO tool could focus on refining instructions for selected RoB-SPEO domains and additional testing to assess agreement for different topic areas and with a wider range of assessors from different research backgrounds.

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

OBJECTIVES

To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE).

STUDY DESIGN AND SETTING

In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC₁ statistic to calculate IRR and ICR.

RESULTS

After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase.

CONCLUSION

Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

Identification and ordering of safety performance indicators using fuzzy TOPSIS: a case study in Indian construction company

Purpose

The purpose of this paper is to propose a practical framework to measure the safety performance of workers in the Indian construction industry. The key safety performance indicators are identified and ordered on the premise that the higher order assignment of an indicator implies a strong indication of an effective safety performance.

Design/methodology/approach

Various indicators of safety performance in the construction industry were identified from extant literature review combined with author's personal viewpoint. The identified variables were inquired for appropriateness for the Indian construction scenario by consultation with experts. Fuzzy Technique for order preference by similarity to ideal solution (TOPSIS) technique was considered for the ranking of the indicators from most to least important.

Findings

The most important highlight of the study was the importance of the role of management by participating in informing workers about the safety rules and compliance toward safety measures. Proper and timely safety training to the workers and equipping them with sophisticated safety equipment for daily activities is perceived to be highly important in ensuring a safe and healthy workplace environment. Controlling the absenteeism rate reduces the burden of extra work on the employees, thereby, encouraging safe work-related behavior.

Originality/value

Senior management should make safety induction programs compulsory at the time of joining of the employees. The guidelines for safety practices, rules and information about the safety equipment should be properly documented and arranged in safety manuals. Periodical drills involving visual demonstration of the safety practices should be followed to ensure safety at workplace.

The MethodologicAl STandards for Epidemiological Research (MASTER) scale demonstrated a unified framework for bias assessment

OBJECTIVE

This paper presents a unified framework for assessment of the methodological quality of analytic study designs.

STUDY DESIGN AND SETTING

A systematic review of 393 methodological quality assessment tools that updated a previous assessment with 100 tools. Tool items were extracted, examined and reworded. Bias domains and finally methodological standards to be fulfilled were defined.

RESULTS

There were 36 unique methodological safeguards that were categorized into seven methodological standards to be fulfilled in the MASTER scale. These methodological standards reflect initial and ongoing equivalence in particular areas, including equal recruitment, equal retention, equal ascertainment, equal implementation, equal prognosis, sufficient analysis, and temporal precedence.

CONCLUSION

This approach unifies existing methods for methodological quality assessment and will be useful for (1) clinical researchers when a bias assessment of clinical research studies is required across analytical designs, (2) promoting a unified framework for bias assessment.