Effect Size and Power in Assessing Moderating Effects of Categorical Variables Using Multiple Regression: A 30-Year Review

Dwindling Resources

Applied aviation research can be extremely expensive and time consuming. Generally, aviation researchers use group-designed experimental models, requiring large sample sizes, to observe intervention effects. Currently, in an environment of dwindling resources and the ever-increasing costs associated with participant recruitment/retention, data collection, equipment usage, etc., the use of the appropriate and most efficient experimental design may provide researchers with a less expensive way to test hypotheses and demonstrate effects of the independent variable. This paper offers one such potential alternative that may be suitable for some research studies – the multiple-baseline on subjects single-case research design (MBSCRD), a design requiring only 3–5 participants.

Belief in luck or in skill: which locks people into gambling?

According to the social axioms framework, people's beliefs about how the world functions (i.e., internal or external locus of control) are related to their social behaviors. Previous researchers have attempted to relate locus of control to gambling behavior, but the results have not been clear-cut. The present study speculated that the effects of perceived control (i.e., belief in luck and belief in skill) on gambling behavior are domain-specific and vary with the type of gambling. A total of 306 adult Macau residents ranging in age from 18 to 65 with casino gambling experience were recruited by going door to door. Empirical data on gambling frequency and perceived control relating to 13 types of gambling were collected. Our results demonstrated that the effects of belief in luck or skill on gambling behavior varied across different gambling categories. Specifically, for football lottery, Chinese lottery, and baccarat, it was not belief in skill but rather belief in luck that was a positive significant predictor of gambling frequency. Only for slot machines and stud poker did belief in skill significantly predict gambling frequency. For the remaining eight gambling categories, neither belief in luck nor belief in skill could predict gambling frequency. Our findings indicate that neither internal nor external locus of control can consistently explain people's gambling behaviors. Instead, which factor plays a greater role in a person's gambling behavior is dependent on the gambling type. Therefore, the finding that not all gambles are created equal might be a promising avenue for further research and treatment approaches.

Distinguishing differential susceptibility from diathesis-stress: recommendations for evaluating interaction effects

This report describes the state of the art in distinguishing data generated by differential susceptibility from diathesis-stress models. We discuss several limitations of existing practices for probing interaction effects and offer solutions that are designed to better differentiate differential susceptibility from diathesis-stress models and quantify their corresponding implications. In addition, we demonstrate the utility of these methods by revisiting published evidence suggesting that temperamental difficulty serves as a marker of enhanced susceptibility to early maternal caregiving across a range of outcome domains in the NICHD Study of Early Child Care and Youth Development. We find that, with the exception of mother reports of psychopathology, there is consistent evidence in the Study of Early Child Care and Youth Development that the predictive significance of early sensitivity is moderated by difficult temperament over time. However, differential susceptibility effects emerged primarily for teacher reports of academic skills, social competence, and symptomatology. In contrast, effects more consistent with the diathesis-stress model were obtained for mother reports of social skills and objective tests of academic skills. We conclude by discussing the value of the application of this work to the next wave of Gene × Environment studies focused on early caregiving experiences.

Every cloud has a silver lining: positive effects of deviant coworkers

Drawing on the labeling perspective of deviance, we investigate employee reactions to coworkers perceived as deviants. We look at two positive effects for employees in the presence of a deviant coworker. First, in comparison to a deviant individual, other employees can draw more positive conclusions about themselves; and second, a deviant can be informative about organizational norms, thereby improving employee role clarity. We also examine individual and situational moderators. For the purpose of the study, we developed a measure of the presence of a deviant. The hypothesized relationships were tested in two large samples using multiple regression analyses. The results revealed that in the presence of a deviant coworker, employees reported enhanced self-evaluation. The presence of a deviant coworker was associated with better role clarity only for employees with more interdependent jobs. We also discuss practical implications of the findings and future research directions.

Using the Criterion-Predictor Factor Model to Compute the Probability of Detecting Prediction Bias with Ordinary Least Squares Regression

The study of prediction bias is important and the last five decades include research studies that examined whether test scores differentially predict academic or employment performance. Previous studies used ordinary least squares (OLS) to assess whether groups differ in intercepts and slopes. This study shows that OLS yields inaccurate inferences for prediction bias hypotheses. This paper builds upon the criterion-predictor factor model by demonstrating the effect of selection, measurement error, and measurement bias on prediction bias studies that use OLS. The range restricted, criterion-predictor factor model is used to compute Type I error and power rates associated with using regression to assess prediction bias hypotheses. In short, OLS is not capable of testing hypotheses about group differences in latent intercepts and slopes. Additionally, a theorem is presented which shows that researchers should not employ hierarchical regression to assess intercept differences with selected samples.

Older job seekers' job search intensity: the interplay of proactive personality, age and occupational future time perspective

Long-term unemployment of older people can have severe consequences for individuals, communities and ultimately economies, and is therefore a serious concern in countries with an ageing population. However, the interplay of chronological age and other individual difference characteristics in predicting older job seekers' job search is so far not well understood. This study investigated relationships among age, proactive personality, occupational future time perspective (FTP) and job search intensity of 182 job seekers between 43 and 77 years in Australia. Results were mostly consistent with expectations based on a combination of socio-emotional selectivity theory and the notion of compensatory psychological resources. Proactive personality was positively related to job search intensity and age was negatively related to job search intensity. Age moderated the relationship between proactive personality and job search intensity, such that the relationship was stronger at higher compared to lower ages. One dimension of occupational FTP (perceived remaining time left in the occupational context) mediated this moderating effect, but not the overall relationship between age and job search intensity. Implications for future research, including the interplay of occupational FTP and proactive personality, and some tentative practical implications are discussed.

THE ROLE OF ABSORPTIVE CAPACITY IN FACILITATING "OPEN INNOVATION" OUTCOMES: A STUDY OF AUSTRALIAN SMEs IN THE MANUFACTURING SECTOR

The open innovation approach emphasizes porous knowledge boundaries between firms and upstream suppliers, but tends to ignore questions of transformative efficiency and effectiveness once the knowledge reaches the focal organization. In this paper, we test for the significance of interaction effects between open innovation strategies and absorptive capacity, finding support for the idea that effective knowledge absorption capabilities are of vital importance in the facilitation of innovation effectiveness.

The relationship between breast cancer anxiety and mammography: experiential avoidance as a moderator

Although mammography can aid in the early detection and prevention of breast cancer, many women do not receive annual mammograms. It remains unclear whether anxiety about breast cancer inhibits or promotes mammography rates. The way in which women regulate their anxiety (ie, level of experiential avoidance) may play a role in predicting mammography adherence. A community sample of women (N = 84) completed a questionnaire which assessed mammography rates, experiential avoidance, and breast cancer anxiety. The results suggest that, while controlling for breast cancer anxiety, experiential avoidance (β = .31, p < .01) significantly predicted mammography rates. When examining experiential avoidance as a moderator, a multiple regression analysis approached significance (R2 Δ = .04, p = .07), suggesting that a woman's level of experiential avoidance influences the relationship between anxiety and mammography. These findings will help enable health care practitioners to better identify women at risk of non-adherence to mammography recommendations.

Statistical power and effect sizes of depression research in Japan

AIMS

Few studies have been conducted on the rationales for using interpretive guidelines for effect size, and most of the previous statistical power surveys have covered broad research domains. The present study aimed to estimate the statistical power and to obtain realistic target effect sizes of depression research in Japan.

METHODS

We systematically reviewed 18 leading journals of psychiatry and psychology in Japan and identified 974 depression studies that were mentioned in 935 articles published between 1990 and 2006.

RESULTS

In 392 studies, logistic regression analyses revealed that using clinical populations was independently associated with being a statistical power of <0.80 (odds ratio 5.9, 95% confidence interval 2.9-12.0) and of <0.50 (odds ratio 4.9, 95% confidence interval 2.3-10.5). Of the studies using clinical populations, 80% did not achieve a power of 0.80 or more, and 44% did not achieve a power of 0.50 or more to detect the medium population effect sizes. A predictive model for the proportion of variance explained was developed using a linear mixed-effects model. The model was then used to obtain realistic target effect sizes in defined study characteristics.

CONCLUSION

In the face of a real difference or correlation in population, many depression researchers are less likely to give a valid result than simply tossing a coin. It is important to educate depression researchers in order to enable them to conduct an a priori power analysis.

Methodological criteria for the assessment of moderators in systematic reviews of randomised controlled trials: a consensus study

BACKGROUND

Current methodological guidelines provide advice about the assessment of sub-group analysis within RCTs, but do not specify explicit criteria for assessment. Our objective was to provide researchers with a set of criteria that will facilitate the grading of evidence for moderators, in systematic reviews.

METHOD

We developed a set of criteria from methodological manuscripts (n = 18) using snowballing technique, and electronic database searches. Criteria were reviewed by an international Delphi panel (n = 21), comprising authors who have published methodological papers in this area, and researchers who have been active in the study of sub-group analysis in RCTs. We used the Research ANd Development/University of California Los Angeles appropriateness method to assess consensus on the quantitative data. Free responses were coded for consensus and disagreement. In a subsequent round additional criteria were extracted from the Cochrane Reviewers' Handbook, and the process was repeated.

RESULTS

The recommendations are that meta-analysts report both confirmatory and exploratory findings for sub-groups analysis. Confirmatory findings must only come from studies in which a specific theory/evidence based a-priori statement is made. Exploratory findings may be used to inform future/subsequent trials. However, for inclusion in the meta-analysis of moderators, the following additional criteria should be applied to each study: Baseline factors should be measured prior to randomisation, measurement of baseline factors should be of adequate reliability and validity, and a specific test of the interaction between baseline factors and interventions must be presented.

CONCLUSIONS

There is consensus from a group of 21 international experts that methodological criteria to assess moderators within systematic reviews of RCTs is both timely and necessary. The consensus from the experts resulted in five criteria divided into two groups when synthesising evidence: confirmatory findings to support hypotheses about moderators and exploratory findings to inform future research. These recommendations are discussed in reference to previous recommendations for evaluating and reporting moderator studies.

A moderator-mediator analysis of coronary heart disease mortality

OBJECTIVE

The purpose of this study is to better understand how risk factors for coronary heart disease (CHD) mortality may interact.

METHODS

We conducted a moderator-mediator analysis of a representative national sample of 5027 and 2902 community-dwelling women and men in the first National Health and Nutrition Examination Survey free of CHD in 1982. The outcome was 10-year CHD mortality.

RESULTS

Two hundred sixty-seven subjects experienced CHD mortality. In the complete sample, gender moderated the effect of depressive symptoms, and among women, race-ethnicity moderated the effect of nonleisure activity on CHD mortality, defining three subgroups for further analysis: men, white women, and black/other women. Among men, baseline differences from median age (55 to 64 years), systolic blood pressure (129 to 158 mmHg), or self-rated general health ("good" to "poor") were associated with equivalent increases in 10-year CHD mortality from 2.3% to 5.3% [area-under-the-curve effect size (ES)=0.53]. These factors appeared to mediate the effect of education on CHD mortality. Severe depression in men was associated with higher 10-year CHD mortality than less or no depression, 10.0% vs. 2.5% (ES=0.55). Among white women, baseline differences from median age (51 to 65 years) was also associated with 10-year mortality (1.2 to 13.4%, ES=0.56), as was higher blood pressure (125 to 151 mmHg) or worse self-rated health ("very good" to "fair") to a lesser extent (1.2% to 3.5%, ES=0.51).

CONCLUSION

Moderators (gender, race-ethnicity) defined possible pathways to CHD mortality characterized by varying factors and interactions between factors, highlighting potential utility for targeted interventions among community-dwelling persons.

Debunking Myths and Urban Legends About Meta-Analysis

Meta-analysis is the dominant approach to research synthesis in the organizational sciences. We discuss seven meta-analytic practices, misconceptions, claims, and assumptions that have reached the status of myths and urban legends (MULs). These seven MULs include issues related to data collection (e.g., consequences of choices made in the process of gathering primary-level studies to be included in a meta-analysis), data analysis (e.g., effects of meta-analytic choices and technical refinements on substantive conclusions and recommendations for practice), and the interpretation of results (e.g., meta-analytic inferences about causal relationships). We provide a critical analysis of each of these seven MULs, including a discussion of why each merits being classified as an MUL, their kernels of truth value, and what part of each MUL represents misunderstanding. As a consequence of discussing each of these seven MULs, we offer best-practice recommendations regarding how to conduct meta-analytic reviews.

A meta-analysis of intervention effects on challenging behaviour among persons with intellectual disabilities

BACKGROUND

Persons with intellectual disabilities (ID) often show challenging behaviour. We review distinct interventions that are applied to treat these challenging behaviours, and analyse intervention effects and moderating variables.

METHODS

A literature search was conducted using the databases ERIC, PsycINFO, Web of Science and Medline. A random-effects meta-analysis was carried out, supplemented with sensitivity, subgroup, meta-regression and publication bias analyses.

RESULTS

Eighty potential articles were identified, from which 30 contained sufficient data to enable statistical meta-analysis. From these 30 studies, 18 described a biological, 13 a psychotherapeutic and nine a contextual intervention, either applied alone or combined. The overall standardised mean difference was 0.671 (SD = 0.051). As shown by sensitivity analysis, this summary effect size is robust. Assessed through subgroup and meta-regression analysis, all tested moderators showed no statistically significant association with the treatment effects. After applying a funnel plot-, a fail-safe N-, and Duval's and Tweedie's trim and fill-analysis, we conclude that our meta-analysis does not suffer much from publication bias effects.

CONCLUSIONS

Several biological, psychotherapeutic and contextual interventions effectively reduce challenging behaviours among persons with ID.

On the Misconception of Multicollinearity in Detection of Moderating Effects: Multicollinearity Is Not Always Detrimental

Due to its extensive applicability and computational ease, moderated multiple regression (MMR) has been widely employed to analyze interaction effects between 2 continuous predictor variables. Accordingly, considerable attention has been drawn toward the supposed multicollinearity problem between predictor variables and their cross-product term. This article attempts to clarify the misconception of multicollinearity in MMR studies. The counterintuitive yet beneficial effects of multicollinearity on the ability to detect moderator relationships are explored. Comprehensive treatments and numerical investigations are presented for the simplest interaction model and more complex three-predictor setting. The results provide critical insight that both helps avoid misleading interpretations and yields better understanding for the impact of intercorrelation among predictor variables in MMR analyses.

A Meta-Meta-Analysis: Empirical Review of Statistical Power, Type I Error Rates, Effect Sizes, and Model Selection of Meta-Analyses Published in Psychology

This article uses meta-analyses published in Psychological Bulletin from 1995 to 2005 to describe meta-analyses in psychology, including examination of statistical power, Type I errors resulting from multiple comparisons, and model choice. Retrospective power estimates indicated that univariate categorical and continuous moderators, individual moderators in multivariate analyses, and tests of residual variability within individual levels of categorical moderators had the lowest and most concerning levels of power. Using methods of calculating power prospectively for significance tests in meta-analysis, we illustrate how power varies as a function of the number of effect sizes, the average sample size per effect size, effect size magnitude, and level of heterogeneity of effect sizes. In most meta-analyses many significance tests were conducted, resulting in a sizable estimated probability of a Type I error, particularly for tests of means within levels of a moderator, univariate categorical moderators, and residual variability within individual levels of a moderator. Across all surveyed studies, the median effect size and the median difference between two levels of study level moderators were smaller than Cohen's (1988) conventions for a medium effect size for a correlation or difference between two correlations. The median Birge's (1932) ratio was larger than the convention of medium heterogeneity proposed by Hedges and Pigott (2001) and indicates that the typical meta-analysis shows variability in underlying effects well beyond that expected by sampling error alone. Fixed-effects models were used with greater frequency than random-effects models; however, random-effects models were used with increased frequency over time. Results related to model selection of this study are carefully compared with those from Schmidt, Oh, and Hayes (2009), who independently designed and produced a study similar to the one reported here. Recommendations for conducting future meta-analyses in light of the findings are provided.