Statistical inference for agreement between multiple raters on a binary scale

Agreement studies often involve more than two raters or repeated measurements. In the presence of two raters, the proportion of agreement and of positive agreement are simple and popular agreement measures for binary scales. These measures were generalized to agreement studies involving more than two raters with statistical inference procedures proposed on an empirical basis. We present two alternatives. The first is a Wald confidence interval using standard errors obtained by the delta method. The second involves Bayesian statistical inference not requiring any specific Bayesian software. These new procedures show better statistical behaviour than the confidence intervals initially proposed. In addition, we provide analytical formulas to determine the minimum number of persons needed for a given number of raters when planning an agreement study. All methods are implemented in the R package simpleagree and the Shiny app simpleagree.

Development, validation, and accuracy of ORCHESTRA emotional exhaustion screening questionnaire among healthcare workers during COVID-19 Pandemic

BACKGROUND

The COVID-19 pandemic has posed unprecedented challenges, particularly for healthcare workers (HCWs). The prolonged struggles exposed the HCWs to a variety of stressors, potentially leading to burnout. Emotional exhaustion is widely recognized as the core component of burnout. This research aims to conceptualize and develop an emotional exhaustion screening questionnaire through literature review, validation, and accuracy testing.

METHOD

A literature review of questionnaires and extraction of items on emotional exhaustion were performed in June 2022. We proceed with the face validity of the items by experts. The items with good content validity ratio and index were selected and reworded to suit the context of HCWs working during the COVID-19 pandemic. A pilot test of the questionnaire was done in the Central University Hospital of Asturias (HUCA) from October to December 2022 with a sample of 148 HCWs from the ORCHESTRA cohort to determine its reliability, convergent validity, and accuracy.

RESULTS

Our literature review identified 15 validated questionnaires. After exclusion, 32 items were sent for content validation by experts, yielding five final items that proceeded with the pilot test. Resulting in a Cronbach's alpha-coefficient of .83 for the scale and .78 for dichotomous responses, demonstrating good internal consistency and convergent validity. The result of our accuracy test yielded sensitivity (90.6%) and specificity (91.6%) for the OEEQ scale; and sensitivity (88.7%) and specificity (89.5%) for OEEQ dichotomous responses.

CONCLUSION

This study developed and validated the ORCHESTRA Emotional Exhaustion Questionnaire, demonstrating the questionnaire's clarity, relevance, and comprehensibility in screening emotional exhaustion among HCWs.

A novel Mendelian randomization method with binary risk factor and outcome

BACKGROUND

Mendelian randomization (MR) applies instrumental variable (IV) methods to observational data using a genetic variant as an IV. Several Monte-Carlo studies investigate the performance of MR methods with binary outcomes, but few consider them in conjunction with binary risk factors.

OBJECTIVE

To develop a novel MR estimator for scenarios with a binary risk factor and outcome; and compare to existing MR estimators via simulations and real data analysis.

METHODS

A bivariate Bernoulli distribution is adapted to the IV setting. Empirical bias and asymptotic coverage probabilities are estimated via simulations. The proposed method is compared to the Wald method, two-stage predictor substitution (2SPS), two-stage residual inclusion (2SRI), and the generalized method of moments (GMM). An analysis is performed using existing data from the CLEAR study to estimate the potential causal effect of smoking on rheumatoid arthritis risk in African Americans.

RESULTS

Bias was low for the proposed method and comparable to 2SPS. The Wald method was often biased towards the null. Coverage was adequate for the proposed method, 2SPS, and 2SRI. Coverage for the Wald and GMM methods was poor in several scenarios. The causal effect of ever smoking on rheumatoid arthritis risk was not statistically significant using a variety of genetic instruments.

CONCLUSIONS

Simulations suggest the proposed MR method is sound with binary risk factors and outcomes, and comparable to 2SPS and 2SRI in terms of bias. The proposed method also provides more natural framework for hypothesis testing compared to 2SPS or 2SRI, which require ad-hoc variance adjustments.

Exploratory Mediation Analysis of Dichotomous Outcomes via Regularization

Exploratory mediation analysis via regularization, or XMed, is a recently developed technique that allows one to identify potential mediators of a process of interest. However, as currently implemented, it can only be applied to continuous outcomes. We extend this method to allow application to dichotomous outcomes, including both mediators and dependent variables. Simulation results show that XMed can achieve the same sensitivity as more conventional methods for mediation analysis such as the Sobel test, percentile bootstrap, and bias-corrected bootstrap, but in general requires only half the sample size to do so. We demonstrate the implementation of this approach using an illustrative example examining the relationship between youth behavioral/emotional problems and alcohol use.

Revised Parallel Analysis With Nonnormal Ability and a Guessing Parameter

Previous work showing that revised parallel analysis can be effective with dichotomous items has used a two-parameter model and normally distributed abilities. In this study, both two- and three-parameter models were used with normally distributed and skewed ability distributions. Relatively minor skew and kurtosis in the underlying ability distribution had almost no effect on Type I error for unidimensional data and reduced power for two-dimensional data slightly with smaller sample sizes of 400. Using a two-parameter model on three-parameter data produced dramatically increased rejection rates for the unidimensional data. Using the correct three-parameter model reduced the unidimensional rejection rates but yielded lower power than the two-parameter data in some conditions.

The Emergence of Visual Awareness: Temporal Dynamics in Relation to Task and Mask Type

One aspect of consciousness phenomena, the temporal emergence of visual awareness, has been subject of a controversial debate. How can visual awareness, that is the experiential quality of visual stimuli, be characterized best? Is there a sharp discontinuous or dichotomous transition between unaware and fully aware states, or does awareness emerge gradually encompassing intermediate states? Previous studies yielded conflicting results and supported both dichotomous and gradual views. It is well conceivable that these conflicting results are more than noise, but reflect the dynamic nature of the temporal emergence of visual awareness. Using a psychophysical approach, the present research tested whether the emergence of visual awareness is context-dependent with a temporal two-alternative forced choice task. During backward masking of word targets, it was assessed whether the relative temporal sequence of stimulus thresholds is modulated by the task (stimulus presence, letter case, lexical decision, and semantic category) and by mask type. Four masks with different similarity to the target features were created. Psychophysical functions were then fitted to the accuracy data in the different task conditions as a function of the stimulus mask SOA in order to determine the inflection point (conscious threshold of each feature) and slope of the psychophysical function (transition from unaware to aware within each feature). Depending on feature-mask similarity, thresholds in the different tasks were highly dispersed suggesting a graded transition from unawareness to awareness or had less differentiated thresholds indicating that clusters of features probed by the tasks quite simultaneously contribute to the percept. The latter observation, although not compatible with the notion of a sharp all-or-none transition between unaware and aware states, suggests a less gradual or more discontinuous emergence of awareness. Analyses of slopes of the fitted psychophysical functions also indicated that the emergence of awareness of single features is variable and might be influenced by the continuity of the feature dimensions. The present work thus suggests that the emergence of awareness is neither purely gradual nor dichotomous, but highly dynamic depending on the task and mask type.

A Comparison of Composite Reliability Estimators: Coefficient Omega Confidence Intervals in the Current Literature

Coefficient omega and alpha are both measures of the composite reliability for a set of items. Unlike coefficient alpha, coefficient omega remains unbiased with congeneric items with uncorrelated errors. Despite this ability, coefficient omega is not as widely used and cited in the literature as coefficient alpha. Reasons for coefficient omega's underutilization include a limited knowledge of its statistical properties. However, consistent efforts to understand the statistical properties of coefficient omega can help improve its utilization in research efforts. Here, six approaches for estimating confidence intervals for coefficient omega with unidimensional congeneric items were evaluated through a Monte Carlo simulation. The evaluations were made through simulation conditions that mimic realistic conditions that investigators are likely to face in applied work, including items that are not normally distributed and small sample size(s). Overall, the normal theory bootstrap confidence interval had the best performance across all simulation conditions that included sample sizes less than 100. However, most methods had sound coverage with sample sizes of 100 or more.

Psychometric features of an assessment instrument with likert and dichotomous response formats

OBJECTIVE

To assess the psychometric properties of a Likert-formatted assessment instrument after altering the responses to a dichotomous format.

DESIGN AND SAMPLE

This methodological study used a 15-item instrument to obtain data from 183 participants who responded in both Likert and dichotomous formats. Response sets from each format were compared.

MEASURES

Each response set underwent factor analysis, Kuder-Richardson 20, Cronbach's α coefficient, item-total correlation, and parallel form equivalence tests.

RESULTS

Factor loads of the instrument varied between .362 and .754 when responses were Likert-formatted and between .370 and .713 when responses were dichotomous. The Cronbach's α coefficient with Likert-formatted responses was .858; the Kuder-Richardson 20 coefficient of the dichotomous responses was .827. Parallel form equivalences were significant at the level of r = .753.

CONCLUSIONS

The instrument had valid results when either Likert or dichotomous responses were obtained.

The graded and dichotomous nature of visual awareness

Is our visual experience of the world graded or dichotomous? Opposite pre-theoretical intuitions apply in different cases. For instance, when looking at a scene, one has a distinct sense that our experience has a graded character: one cannot say that there is no experience of contents that fall outside the focus of attention, but one cannot say that there is full awareness of such contents either. By contrast, when performing a visual detection task, our sense of having perceived the stimulus or not exhibits a more dichotomous character. Such issues have recently been the object of intense debate because different theoretical frameworks make different predictions about the graded versus dichotomous character of consciousness. Here, we review both relevant empirical findings as well as the associated theories (i.e. local recurrent processing versus global neural workspace theory). Next, we attempt to reconcile such contradictory theories by suggesting that level of processing is an often-ignored but highly relevant dimension through which we can cast a novel look at existing empirical findings. Thus, using a range of different stimuli, tasks and subjective scales, we show that processing low-level, non-semantic content results in graded visual experience, whereas processing high-level semantic content is experienced in a more dichotomous manner. We close by comparing our perspective with existing proposals, focusing in particular on the partial awareness hypothesis.