Reliability generalization: exploring reliability variations on MMPI/MMPI-2 validity scale scores

A reliability generalization meta-analysis of the family resilience assessment scale (FRAS)

PROBLEM

Practitioners and researchers frequently rely on the Family Resilience Assessment Scale (FRAS) to assess family resilience, and previous research reported varying reliability statistics for the measurements with the scale. The present study aims to generalize the reliability of the FRAS based on Cronbach's alpha coefficients reported in the selected studies.

ELIGIBILITY CRITERIA

We selected relevant research on various databases, including Web of Science, PubMed, ProQuest, Scopus, YÖK Thesis Center, DergiPark, and TR Index.

SAMPLE

Satisfying our inclusion criteria, fifty-five studies were included in the present study.

RESULTS

We calculated the reliability generalization coefficients for the FRAS total score to be 0.951 (95% CI [0.942, 0.958]) and 0.949 for Family Communication and Problem Solving, 0.792 for Utilizing Social and Economic Resources, 0.861 for Maintaining a Positive Outlook, 0.635 for Family Connectedness, 0.873 for Family Spirituality, and 0.702 for Ability to Make Meaning of Adversity.

CONCLUSIONS

In a nutshell, our findings secure substantial insights into the reliability of the FRAS and its subscales for prospective researchers and practitioners. In this study, generalized Cronbach's alpha values imply average, good, and acceptable reliability for the FRAS subscales and total score, except for Family Connectedness.

IMPLICATIONS

The distinct contribution of our research may be to reemphasize the significance of avoiding reliability induction and to raise awareness among prospective researchers of evaluating the reliability of any measurement they would obtain.

The agreement of the endoscopic Modified Lund-Kennedy scoring in a clinical research group: An observational study

OBJECTIVES

The main objective was to prove the robustness of the modified Lund-Kennedy staging system and its use in the clinical research group. Secondary objectives were to evaluate the physicians' homogeneity, identify outliers with an unacceptable agreement and define factors for questionable agreement within the group of raters.

MATERIAL AND METHODS

Anonymized endoscopic photos of patients with chronic rhinosinusitis were assessed by independent raters from a clinical research group. The level of agreement between raters was calculated using intra-class correlation and weighted kappa coefficient. Clusters of similarity were identified using Inter-Item Correlation Matrix. The weighted kappa coefficient was calculated for the most homogeneous group and outliers. Age, sex, consultancy years, combined clinical and research work assessed by 5 senior peers were also statistically compared between raters.

RESULTS

Intraclass-correlation coefficients were 0.75 and 0.95 for respectively single and average measures. Single measures value for most homogenous raters was 0.97 (weighted kappa 0.88, (P<0.001). One outlier with less research work score had an unacceptable agreement for single measures coefficient values with the 2 most homogenous raters (respectively 0.59, weighted kappa 0.15, P=0.32 and 0.57, weighted kappa 0.197, P=0.32). Pooled groups were similar in age (P=0.3), sex (P=0.1) and consultancy years (P=0.2) but significantly differentiated in peer-assessed clinical and research work score (P<0.001).

CONCLUSION

Even with a perfect overall agreement, careful examination of correlation matrix revealed an obvious outlier with less than ideal performance. The method may be helpful when studies using endoscopic staging system are designed to involve researchers from different backgrounds. When exploring the most common factors, education and clinical experience play a paramount role.

Empirical Benchmarks for Interpreting Effect Size Variability in Meta-Analysis

Generalization in meta-analyses is not a dichotomous decision (typically encountered in papers using the Q test for homogeneity, the 75% rule, or null hypothesis tests). Inattention to effect size variability in meta-analyses may stem from a lack of guidelines for interpreting credibility intervals. In this commentary, we describe two methods for making practical interpretations and determining whether a particular SDρ represents a meaningful level of variability.

Reliability Generalization

The assessment of positive affect (PA) and negative affect (NA) by means of the Positive Affect and Negative Affect Schedule has received a remarkable popularity in the social sciences. Using a meta-analytic tool—namely, reliability generalization (RG)—population reliability scores of both scales have been investigated on the basis of a random effects model in 147 studies. Correcting for measurement errors, the results demonstrate moderate to high internal consistency coefficients and variations of the PA and NA reliability scores with regard to time frame instructions, language of items, and sample characteristics. The percentage of PA and NA true score variance differs in subpopulations up to 11%. RG analysis of test–retest coefficients illustrates state-like fluctuations and trait-like stability of both scales. Calculations of the fail-safe number point at the robustness of the results. Applications of RG coefficients compared to single-study coefficients highlight the relevance of population coefficients for research and assessment situations.

Reliability reporting across studies using the Buss Durkee Hostility Inventory

Empirical research on anger and hostility has pervaded the academic literature for more than 50 years. Accurate measurement of anger/hostility and subsequent interpretation of results requires that the instruments yield strong psychometric properties. For consistent measurement, reliability estimates must be calculated with each administration, because changes in sample characteristics may alter the scale's ability to generate reliable scores. Therefore, the present study was designed to address reliability reporting practices for a widely used anger assessment, the Buss Durkee Hostility Inventory (BDHI). Of the 250 published articles reviewed, 11.2% calculated and presented reliability estimates for the data at hand, 6.8% cited estimates from a previous study, and 77.1% made no mention of score reliability. Mean alpha estimates of scores for BDHI subscales generally fell below acceptable standards. Additionally, no detectable pattern was found between reporting practices and publication year or journal prestige. Areas for future research are also discussed.

ANALYSIS OF DEMOGRAPHIC FACTORS ON F SCORES AND EVALUATION OF NEW CONFIGURATIONS OF F SCALES

We analyzed the influence of demographic factors on scores of the Infrequency scale of the MMPI (Archer, Gordon, & Kirchner, 1987) administered to male youth in China. The demographic factors, acting as influencing factors of F scores, could be ignored. New configurations of infrequency scales were presented and discussed, which indicated that it was appropriate to choose a 20% item endorsement percentage (IEP) of the Chinese male youth as the criterion for item selection.