The assumption of a reliable instrument and other pitfalls to avoid when considering the reliability of data

Realization of affiliation goals, interpersonal identity development, and well-being: effects of the implicit affiliation motive among German and Zambian adolescents

Introduction

Across various cultural contexts, success in goal realization relates to individuals' well-being. Moreover, commitment to and successful pursuance of goals are crucial when searching for a meaningful identity in adolescence. However, individuals' goals differ in how much they match their implicit motive dispositions. We hypothesized that successful pursuance of affiliation goals positively relates to commitment-related dimensions of interpersonal identity development (domain: close friends) that, in turn, predict adolescents' level of well-being. However, we further assumed that the links between goal success and identity commitment are particularly pronounced among adolescents who are characterized by a high implicit affiliation motive.

Methods

To scrutinize the generalizability of the assumed relationships, data were assessed among adolescents in individualistic (Germany) and collectivistic (Zambia) cultural contexts.

Results

Regardless of adolescents' cultural background, we found that commitment-related dimensions of interpersonal identity development mediate the link between successful attainment of affiliation goals and well-being, particularly among adolescents with a pronounced implicit affiliation motive; that is, the strength of the implicit affiliation motive moderates the association between goal success and identity commitment.

Conclusion

We discuss findings concerning universal effects of implicit motives on identity commitment and well-being.

Data-driven studies in face identity processing rely on the quality of the tests and data sets

There is growing interest in how data-driven approaches can help understand individual differences in face identity processing (FIP). However, researchers employ various FIP tests interchangeably, and it is unclear whether these tests 1) measure the same underlying ability/ies and processes (e.g., confirmation of identity match or elimination of identity match) 2) are reliable, 3) provide consistent performance for individuals across tests online and in laboratory. Together these factors would influence the outcomes of data-driven analyses. Here, we asked 211 participants to perform eight tests frequently reported in the literature. We used Principal Component Analysis and Agglomerative Clustering to determine factors underpinning performance. Importantly, we examined the reliability of these tests, relationships between them, and quantified participant consistency across tests. Our findings show that participants' performance can be split into two factors (called here confirmation and elimination of an identity match) and that participants cluster according to whether they are strong on one of the factors or equally on both. We found that the reliability of these tests is at best moderate, the correlations between them are weak, and that the consistency in participant performance across tests and is low. Developing reliable and valid measures of FIP and consistently scrutinising existing ones will be key for drawing meaningful conclusions from data-driven studies.

A reassessment of the Resistance to Framing scale

Risky-choice and attribute framing effects are well-known cognitive biases, where choices are influenced by the way information is presented. To assess susceptibility to these framing types, the Resistance to Framing scale is often used, although its performance has rarely been extensively tested. In an online survey among university students from Bulgaria (N = 245) and North America (N = 261), we planned to examine the scale's psychometric properties, structural validity, and measurement invariance. However, some of these examinations were not possible because the scale displayed low and mostly non-significant inter-item correlations as well as low item-total correlations. Followingly, exploratory item response theory analyses indicated that the scale's reliability was low, especially for high levels of resistance to framing. This suggests problems with the scale at a basic level of conceptualization, namely that the items may not represent the same content domain. Overall, the scale in its current version is of limited use, at least in university student samples, due to the identified problems. We discuss potential remedies to these problems, as well as provide open code and data ( https://osf.io/j5n6f ) which facilitates testing the scale in other samples (e.g., general population, different languages and countries) to obtain a comprehensive picture of its performance.

On the Reliability of Wearable Technology: A Tutorial on Measuring Heart Rate and Heart Rate Variability in the Wild

Wearable sensors are quickly making their way into psychophysiological research, as they allow collecting data outside of a laboratory and for an extended period of time. The present tutorial considers fidelity of physiological measurement with wearable sensors, focusing on reliability. We elaborate on why ensuring reliability for wearables is important and offer statistical tools for assessing wearable reliability for between participants and within-participant designs. The framework offered here is illustrated using several brands of commercially available heart rate sensors. Measurement reliability varied across sensors and, more importantly, across the situations tested, and was highest during sleep. Our hope is that by systematically quantifying measurement reliability, researchers will be able to make informed choices about specific wearable devices and measurement procedures that meet their research goals.

Cross-cultural investigation from nine countries on the associations of antisocial traits and the WHO's containment measures for the COVID-19 pandemic

Personality traits play a role in prosocial behavior in relation to containment measures intended to tackle the COVID-19 pandemic. Empirical findings indicated that individuals high in socially aversive traits such as callousness are less compliant with containment measures. This study aimed to add cross-cultural data on the relationship between antisocial traits and adherence to COVID-19 containment measures. The sample consisted of 4,538 adults recruited by convenience in nine countries (Australia, Brazil, England, Iraq, Iran, Italy, Jordan, Saudi Arabia, and the United States). Statistical analyses indicated two latent profiles from our sample, empathic and antisocial, and six COVID-19 containment-measure-related factors using measures covering antisocial traits (PID-5), empathy (ACME), global personality pathology (LPFS-BF), and COVID-19 behaviors and beliefs. Through MANCOVA, the antisocial profile consistently showed less compliance and concern about the COVID-19 containment measures, even when controlling for demographics and local pandemic covariables. The network analysis indicated a lack of empathy and callousness as crucial traits of the predisposition to non-compliant behavior. In elaborating on prosocial campaigns in community emergencies, our cross-cultural findings would need to consider personality traits that focus on antisociality, anticipating similar associations and potential impacts in future disease outbreaks.

Internalized Heterosexism and Substance Use: A Meta-Analysis

Sexual minority individuals report greater rates of substance use compared to their heterosexual counterparts. Internalized heterosexism (IH) is one critical factor underlying this disparity. There has been a proliferation of studies examining IH as a correlate of substance use; however, results of these studies have been mixed. The purpose of the current multilevel meta-analysis was to explain these mixed findings by identifying factors that moderate the overall relationship between IH and substance use. Results from the analysis of 49 studies (209 effect sizes and 162,674 participants) demonstrated a positive overall relationship between IH and substance use ( r = .065). This relationship was significantly moderated by sexual orientation, gender, and type of substance. Increased IH was linked to increased use of heroin ( r = .173), cocaine ( r = .135), tobacco ( r = .100), and alcohol ( r = .070). Implications for research, practice, and policy are discussed.

An Updated Guideline for Assessing Discriminant Validity

Discriminant validity was originally presented as a set of empirical criteria that can be assessed from multitrait-multimethod (MTMM) matrices. Because datasets used by applied researchers rarely lend themselves to MTMM analysis, the need to assess discriminant validity in empirical research has led to the introduction of numerous techniques, some of which have been introduced in an ad hoc manner and without rigorous methodological support. We review various definitions of and techniques for assessing discriminant validity and provide a generalized definition of discriminant validity based on the correlation between two measures after measurement error has been considered. We then review techniques that have been proposed for discriminant validity assessment, demonstrating some problems and equivalencies of these techniques that have gone unnoticed by prior research. After conducting Monte Carlo simulations that compare the techniques, we present techniques called CICFA(sys) and [Formula: see text](sys) that applied researchers can use to assess discriminant validity.

Psychological Intervention and Prevention Programs for Child and Adolescent Exposure to Community Violence: A Systematic Review

The purpose of this article is to systematically review the existing literature regarding intervention and prevention programs that ameliorate the negative effects of exposure to community violence (ECV) on children and adolescents. Using the Preferred Reporting Items for Systematic Review (PRISMA) Guidelines, we conducted a systematic review of the literature aimed at providing a synthesis of the extant Randomized Control Trials (RCT) and peer-reviewed empirical literature on intervention and prevention programs for those affected by ECV. Nine randomized controlled trials were identified: seven studies with elementary school students in the USA, one study with elementary school students in Colombia, and one study with middle school students in the USA. Most trials compared intervention and no-treatment control groups; three studies compared active interventions. The intervention and prevention trials conducted in school settings in the USA showed most impact on reducing internalizing and PTSD symptoms. The evidence base of intervention and prevention programs for child and adolescent exposure to community violence is very limited. This systematic review synthesizes extant evidence of the effectiveness of intervention programs in reducing internalizing and externalizing symptoms, PTSD, exposure to community violence, and in enhancing academic performance. More studies are needed to provide a better understanding of how interventions might ameliorate the adverse consequences of exposure to community violence.

Stability of the Manual Ability Classification System in young children with cerebral palsy

AIM

To examine the stability over time of the Manual Ability Classification System (MACS) levels in children with cerebral palsy (CP) aged 18 to 60 months.

METHOD

This was a prospective longitudinal population-based study of 252 Australian children (160 males [63%] 92 females [37%]; mean age [SD] 41.7mo [14], range 17.2mo-69.2mo) with CP. Children were classified at 18 months (n=70), 24 months (n=131), 30 months (n=173), 36 months (n=209), 48 months (n=226), and 60 months (n=221) of age. Stability of the MACS was examined using the proportion of specific positive agreement and transition proportions, which are measures of agreement.

RESULTS

There were 1030 unique observations, with each of the 252 participants seen between two and six occasions (median=4). Average specific positive agreement over the study period was 76% for MACS level I, 67% for level II, 50% for level III, 51% for level IV, and 83% for level V. MACS levels I and V have the highest degree of stability, while levels III and IV have the lowest. We show how this may be explained by the proportion of children in each MACS level.

INTERPRETATION

Using measures of agreement rather than measures of reliability provides accurate information when measuring stability over time of an ordinal classification system. The relative stability of MACS levels can be explained by the proportion of children in each level.

WHAT THIS PAPER ADDS

Children classified in Manual Ability Classification System (MACS) levels III and IV change levels at next assessment about 50% of the time. Children should be assessed with the MACS regularly, particularly those in levels III and IV. Stability within ordinal classification level categories can be predicted using a measurement error model. Transition proportions or specific agreement is recommended for reporting stability of ordinal classification systems.

Reliability Generalization as a Seal of Quality of Substantive Meta-Analyses: The Case of the VIA Inventory of Strengths (VIA-IS) and Their Relationships to Life Satisfaction

Reliable test scores are essential to interpret the results obtained in statistical analyses correctly. In this study, we used the Values in Action Inventory of Strengths (VIA-IS) as an example of a widely applied assessment instrument to analyze its metric quality in what is known as reliability generalization (RG). In addition, we conducted a meta-analysis of the correlations between character strengths and life satisfaction to examine the potential relationship between the reliability of test scores and the intensity of these correlations. The overall variability of alpha coefficients supports the argument that reliability is sample dependent. Indeed, there were statistically significant mean reliability differences for scores across the 24 scales, with the highest level of reliability observed for Creativity and the lowest for scores on Self-regulation. Significant moderators such as the standard deviation of the scores and the sample type contribute to understand the high variability observed in the reliability estimation. The second meta-analysis showed that Zest, Hope, Gratitude, Curiosity, and Love were the character strengths that were highly related to life satisfaction, while Modesty and Prudence were less related to life satisfaction. Furthermore, the high heterogeneity between samples might be an indicator of the relationship between the variability of reliability of character strengths' scores and the intensity of their correlations with life satisfaction. Those character strengths with high-potential RG are related or unrelated to life satisfaction, whereas character strengths with less-potential RG showed unstable correlation patterns. The results of both studies point out the role of the relationship between the reliability of test scores and substantive studies, such as Pearson's correlations meta-analysis.

Preliminary Checklist for Reporting Observational Studies in Sports Areas: Content Validity

Observational studies are based on systematic observation, understood as an organized recording and quantification of behavior in its natural context. Applied to the specific area of sports, observational studies present advantages when comparing studies based on other designs, such as the flexibility for adapting to different contexts and the possibility of using non-standardized instruments as well as a high degree of development in specific software and data analysis. Although the importance and usefulness of sports-related observational studies have been widely shown, there is no checklist to report these studies. Consequently, authors do not have a guide to follow in order to include all of the important elements in an observational study in sports areas, and reviewers do not have a reference tool for assessing this type of work. To resolve these issues, this article aims to develop a checklist to measure the quality of sports-related observational studies based on a content validity study. The participants were 22 judges with at least 3 years of experience in observational studies, sports areas, and methodology. They evaluated a list of 60 items systematically selected and classified into 12 dimensions. They were asked to score four aspects of each item on 5-point Likert scales to measure the following dimensions: representativeness, relevance, utility, and feasibility. The judges also had an open-format section for comments. The Osterlind index was calculated for each item and for each of the four aspects. Items were considered appropriate when obtaining a score of at least 0.5 in the four assessed aspects. After considering these inclusion criteria and all of the open-format comments, the resultant checklist consisted of 54 items grouped into the same initial 12 dimensions. Finally, we highlight the strengths of this work. We also present its main limitation: the need to apply the resultant checklist to obtain data and, thus, increase quality indicators of its psychometric properties. For this reason, as relevant actions for further development, we encourage expert readers to use it and provide feedback; we plan to apply it to different sport areas.

Cancer and treatment distress psychometric evaluation over time: A BMT CTN 0902 secondary analysis

BACKGROUND

Routine monitoring of cancer-related distress is recognized as essential to quality care and mandated by a major accrediting organization. However, to the authors' knowledge, few cancer-specific measures have been developed to date to assess the multiple cancer-related factors contributing to this distress. In the current study, the authors examined the psychometric properties of the Cancer and Treatment Distress (CTXD) measure over time in hematopoietic cell transplantation (HCT) recipients.

METHODS

As a secondary analysis of a multicenter randomized controlled clinical trial, adult patients undergoing autologous or allogeneic HCT completed patient-reported outcomes including the CTXD and the Medical Outcomes Study Short Form-36 (SF-36) before transplantation and 100 days and 180 days after HCT.

RESULTS

Across 21 transplantation centers, a total of 701 patients consented, underwent transplantation, and were included in the current analyses, 645 of whom were alive at 100 days and 618 of whom were alive at 180 days. Internal consistency reliability was found to be strong for the overall CTXD at the 3 time points: Cronbach alphas (α) were .94, .95, and .95, respectively. Subscale reliability met hypothesized levels of an α>.70 across time, with the lowest reliability noted for the Identity subscale at 180 days (α = .77). Correlations with the SF-36 Mental Health subscale were higher than with the Physical Functioning subscale at each time point, thereby supporting convergent and discriminant validity. Strong correlations of the pretransplantation CTXD with the posttransplantation CTXD and SF-36 Mental Health subscale supported predictive validity.

CONCLUSIONS

The CTXD is reliable and valid as a measure of cancer distress both before and after HCT. It may be a useful tool for measuring dimensions of distress and for defining those patients requiring treatment for distress during and after transplantation. Cancer 2017;123:1416-1423. © 2016 American Cancer Society.

The Development of a Checklist to Enhance Methodological Quality in Intervention Programs

The methodological quality of primary studies is an important issue when performing meta-analyses or systematic reviews. Nevertheless, there are no clear criteria for how methodological quality should be analyzed. Controversies emerge when considering the various theoretical and empirical definitions, especially in relation to three interrelated problems: the lack of representativeness, utility, and feasibility. In this article, we (a) systematize and summarize the available literature about methodological quality in primary studies; (b) propose a specific, parsimonious, 12-items checklist to empirically define the methodological quality of primary studies based on a content validity study; and (c) present an inter-coder reliability study for the resulting 12-items. This paper provides a precise and rigorous description of the development of this checklist, highlighting the clearly specified criteria for the inclusion of items and a substantial inter-coder agreement in the different items. Rather than simply proposing another checklist, however, it then argues that the list constitutes an assessment tool with respect to the representativeness, utility, and feasibility of the most frequent methodological quality items in the literature, one that provides practitioners and researchers with clear criteria for choosing items that may be adequate to their needs. We propose individual methodological features as indicators of quality, arguing that these need to be taken into account when designing, implementing, or evaluating an intervention program. This enhances methodological quality of intervention programs and fosters the cumulative knowledge based on meta-analyses of these interventions. Future development of the checklist is discussed.

Reliability of the adult myopathy assessment tool in individuals with myositis

OBJECTIVE

The Adult Myopathy Assessment Tool (AMAT) is a 13-item performance-based battery developed to assess functional status and muscle endurance. The purpose of this study was to determine the intrarater and interrater reliability of the AMAT in adults with myositis.

METHODS

Nineteen raters (13 physical therapists and 6 physicians) scored videotaped recordings of patients with myositis performing the AMAT for a total of 114 tests and 1,482 item observations per session. Raters rescored the AMAT test and item observations during a followup session (mean ± SD 19 ± 6 days between scoring sessions). All raters completed a single, self-directed, electronic training module prior to the initial scoring session.

RESULTS

Intrarater and interrater reliability correlation coefficients were ≥0.94 for the AMAT functional subscale, endurance subscale, and total score (all P < 0.02 for Ho , ρ ≤0.75). All AMAT items had satisfactory intrarater agreement (kappa statistics with Fleiss-Cohen weights, with values κw = 0.57-1.00). Interrater agreement was acceptable for each AMAT item (κ = 0.56-0.89) except the sit up (κ = 0.16). The standard error of measurement and 95% confidence interval range for the AMAT total scores did not exceed 2 points across all observations (AMAT total score range 0-45).

CONCLUSION

The AMAT is a reliable, domain-specific assessment of functional status and muscle endurance for adult subjects with myositis. Results of this study suggest that physicians and physical therapists may reliably score the AMAT following a single training session. The AMAT functional subscale, endurance subscale, and total score exhibit interrater and intrarater reliability suitable for clinical and research use.

Is functional integration of resting state brain networks an unspecific biomarker for working memory performance?

Is there one optimal topology of functional brain networks at rest from which our cognitive performance would profit? Previous studies suggest that functional integration of resting state brain networks is an important biomarker for cognitive performance. However, it is still unknown whether higher network integration is an unspecific predictor for good cognitive performance or, alternatively, whether specific network organization during rest predicts only specific cognitive abilities. Here, we investigated the relationship between network integration at rest and cognitive performance using two tasks that measured different aspects of working memory; one task assessed visual-spatial and the other numerical working memory. Network clustering, modularity and efficiency were computed to capture network integration on different levels of network organization, and to statistically compare their correlations with the performance in each working memory test. The results revealed that each working memory aspect profits from a different resting state topology, and the tests showed significantly different correlations with each of the measures of network integration. While higher global network integration and modularity predicted significantly better performance in visual-spatial working memory, both measures showed no significant correlation with numerical working memory performance. In contrast, numerical working memory was superior in subjects with highly clustered brain networks, predominantly in the intraparietal sulcus, a core brain region of the working memory network. Our findings suggest that a specific balance between local and global functional integration of resting state brain networks facilitates special aspects of cognitive performance. In the context of working memory, while visual-spatial performance is facilitated by globally integrated functional resting state brain networks, numerical working memory profits from increased capacities for local processing, especially in brain regions involved in working memory performance.

The robust reliability of neuropsychological measures: meta-analyses of test-retest correlations

Test-retest reliability is an important psychometric property relevant to assessment instruments typically used in neuropsychological assessment. This review presents a quantitative summary of test-retest reliability coefficients for a variety of widely used neuropsychological measures. In general, the meta-analytic test-retest reliabilities of the test scores ranged from adequate to high (i.e., r=.7 and higher). Furthermore, the reliability values were largely robust across factors such as age, clinical diagnosis, and the use of alternate forms. The values for some of the memory and executive functioning scores were lower (i.e., less than r=.7). Some of the possible reasons for these lower values include ceiling effects, practice effects, and across time variability in cognitive abilities measured by those tests. In general, neuropsychologists who use these measures in their assessments can be encouraged by the magnitude of the majority of the meta-analytic test-retest correlations obtained.

Statistical assumptions of substantive analyses across the general linear model: a mini-review

The validity of inferences drawn from statistical test results depends on how well data meet associated assumptions. Yet, research (e.g., Hoekstra et al., 2012) indicates that such assumptions are rarely reported in literature and that some researchers might be unfamiliar with the techniques and remedies that are pertinent to the statistical tests they conduct. This article seeks to support researchers by concisely reviewing key statistical assumptions associated with substantive statistical tests across the general linear model. Additionally, the article reviews techniques to check for statistical assumptions and identifies remedies and problems if data do not meet the necessary assumptions.