Statistical conclusion validity: some common threats and simple remedies

Impact of Tranexamic Acid on Bleeding Outcomes and Complication Rates in Facelift: A Systematic Review and Meta-analysis

BACKGROUND

Rhytidectomy poses a significant risk of bleeding. Several studies have reported the safety profile and efficacy of tranexamic acid (TXA), an antifibrinolytic agent, in minimizing perioperative sequelae, particularly hematoma and bleeding.

OBJECTIVES

The aim of this systematic review was to analyze the effect of TXA administration in facelift surgery, its effect with different routes of administration, and to compare different administration routes in reducing intraoperative blood loss, postoperative edema, and ecchymosis in rhytidectomy.

METHODS

A systematic literature search was conducted to identify studies that reported on TXA in facelift surgery. The primary outcomes of interest were intraoperative blood loss, time to achieve hemostasis, operation duration, and postoperative hematoma, edema, ecchymosis, drain output, and major and minor complications. Meta-analyses of hematoma, operation duration, drain output, and major and minor complications were performed, and the risk of bias was assessed with ROBINS-I for nonrandomized studies, and Cochrane's RoB 2.0, a tool for randomized controlled trials.

RESULTS

In total, 104 articles were included in the initial screening. Out of 388 participants 170 patients were administered TXA, predominantly female (over 91%), with ages from the late 50s to mid-60s. TXA administration varied, with subcutaneous injection being the most common method. The meta-analysis revealed that the pooled prevalence of minor and major hematoma in TXA recipients was remarkably low, with a significant reduction in the risk of minor hematoma (odds ratio [OR] = 0.18, 95% CI 0.05-0.62, P < .001) and no significant difference in major hematoma risk. Interestingly, TXA significantly reduced postoperative drainage compared to the controls (mean difference = -25.59, 95% CI, -30.4--20.77, P < .01). Additionally, neither minor nor major complications were significantly different between the TXA recipients and controls. Specifically, the pooled odds for the incidence of major complications were not significantly different (OR = 1.47, 95% CI, 0.23-9.19, P = .68), and similar results were found for minor complications (OR = 0.59, 95% CI, 0.23-1.48, P = .26).

CONCLUSIONS

TXA significantly reduces postoperative drain output and minor hematomas in facelift surgery without increasing major complications. It also reduces edema, ecchymosis, and intraoperative blood loss. However, further studies are required to explore the efficacy of TXA with different dosages and administration routes.

LEVEL OF EVIDENCE: 3

Interactive Virtual Ankle Movement Controlled by Wrist sEMG Improves Motor Imagery: An Exploratory Study

Virtual reality (VR) techniques can significantly enhance motor imagery training by creating a strong illusion of action for central sensory stimulation. In this article, we establish a precedent by using surface electromyography (sEMG) of contralateral wrist movement to trigger virtual ankle movement through an improved data-driven approach with a continuous sEMG signal for fast and accurate intention recognition. Our developed VR interactive system can provide feedback training for stroke patients in the early stages, even if there is no active ankle movement. Our objectives are to evaluate: 1) the effects of VR immersion mode on body illusion, kinesthetic illusion, and motor imagery performance in stroke patients; 2) the effects of motivation and attention when utilizing wrist sEMG as a trigger signal for virtual ankle motion; 3) the acute effects on motor function in stroke patients. Through a series of well-designed experiments, we have found that, compared to the 2D condition, VR significantly increases the degree of kinesthetic illusion and body ownership of the patients, and improves their motor imagery performance and motor memory. When compared to conditions without feedback, using contralateral wrist sEMG signals as trigger signals for virtual ankle movement enhances patients' sustained attention and motivation during repetitive tasks. Furthermore, the combination of VR and feedback has an acute impact on motor function. Our exploratory study suggests that the sEMG-based immersive virtual interactive feedback provides an effective option for active rehabilitation training for severe hemiplegia patients in the early stages, with great potential for clinical application.

Scaling relationships between the total number of leaves and the total leaf area per culm of two dwarf bamboo species

Total leaf area per plant is an important measure of the photosynthetic capacity of an individual plant that together with plant density drives the canopy leaf area index, that is, the total leaf area per unit ground area. Because the total number of leaves per plant (or per shoot) varies among conspecifics and among mixed species communities, this variation can affect the total leaf area per plant and per canopy but has been little studied. Previous studies have shown a strong linear relationship between the total leaf area per plant (or per shoot) (A T) and the total number of leaves per plant (or per shoot) (N T) on a log-log scale for several growth forms. However, little is known whether such a scaling relationship also holds true for bamboos, which are a group of Poaceae plants with great ecological and economic importance in tropical, subtropical, and warm temperate regions. To test whether the scaling relationship holds true in bamboos, two dwarf bamboo species (Shibataea chinensis Nakai and Sasaella kongosanensis 'Aureostriatus') with a limited but large number of leaves per culm were examined. For the two species, the leaves from 480 and 500 culms, respectively, were sampled and A T was calculated by summing the areas of individual leaves per culm. Linear regression and correlation analyses reconfirmed that there was a significant log-log linear relationship between A T and N T for each species. For S. chinensis, the exponent of the A T versus N T scaling relationship was greater than unity, whereas that of S. kongosanensis 'Aureostriatus' was smaller than unity. The coefficient of variation in individual leaf area increased with increasing N T for each species. The data reconfirm that there is a strong positive power-law relationship between A T and N T for each of the two species, which may reflect adaptations of plants in response to intra- and inter-specific competition for light.

Comparative Outcomes and Failure Rates of Total Femur Replacement in Oncologic and Nononcologic Indications: A Systematic Review and Meta-analysis

BACKGROUND

Total femur replacement (TFR) has become increasingly significant as a salvage procedure for both oncologic reconstruction and complex nononcologic conditions such as revision arthroplasty. Despite its effectiveness in limb salvage, TFR is associated with high complication and failure rates, which vary depending on the underlying indication.

METHODS

This systematic review and meta-analysis adhered to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A comprehensive search of MEDLINE, EMBASE, Web of Science, and Cumulative Index to Nursing and Allied Health Literature databases was conducted, focusing on studies that reported outcomes of TFR in oncologic and nononcologic cases. Primary outcomes included failure mode and rates according to the Henderson classification, functional outcomes scores, and mobility status. Data were analyzed using random-effects models and generalized linear mixed models.

RESULTS

A total of 35 studies involving 1,002 patients were included. The majority of TFRs were performed for oncologic reasons (63.7%). The mean Musculoskeletal Tumor Society (MSTS) score was 66%, with a limb salvage rate of 89%. The meta-analysis revealed a combined failure rate of 34%. For type 4 failures (infection), nononcologic patients exhibited a significantly higher rate at 18% (95% confidence interval [CI], 12%-26%, I2 = 46%, p < 0.01) compared with 8% in oncologic patients (95% CI, 6%-12%, I2 = 0%). Regarding combined types 1 to 4 failures, oncologic patients had a rate of 20% (95% CI, 25%-52%, I2 = 60%), whereas nononcologic patients faced a higher rate of 37% (95% CI, 12%-26%, I2 = 63%) (p < 0.05), indicating a significant difference. There were no significant differences in the MSTS score. In addition, there were no notable differences when comparing failure modes 1, 2, and 3 independently. Mobility analysis showed that approximately 70% of patients required walking aids after surgery.

CONCLUSION

TFR offers a valuable limb salvage option in both oncologic and nononcologic scenarios, despite its high failure rates. Although functional outcomes were similar between groups, the higher failure rate in nononcologic cases and the poor overall quality of evidence warrant further comprehensive assessments into predictors of outcomes to optimize results.

LEVEL OF EVIDENCE

Level III. See Instructions for Authors for a complete description of levels of evidence.

Are alternative variables in a set differently associated with a target variable? Statistical tests and practical advice for dealing with dependent correlations

The analysis of multiple bivariate correlations is often carried out by conducting simple tests to check whether each of them is significantly different from zero. In addition, pairwise differences are often judged by eye or by comparing the p-values of the individual tests of significance despite the existence of statistical tests for differences between correlations. This paper uses simulation methods to assess the accuracy (empirical Type I error rate), power, and robustness of 10 tests designed to check the significance of the difference between two dependent correlations with overlapping variables (i.e., the correlation between X1 and Y and the correlation between X2 and Y). Five of the tests turned out to be inadvisable because their empirical Type I error rates under normality differ greatly from the nominal alpha level of .05 either across the board or within certain sub-ranges of the parameter space. The remaining five tests were acceptable and their merits were similar in terms of all comparison criteria, although none of them was robust across all forms of non-normality explored in the study. Practical recommendations are given for the choice of a statistical test to compare dependent correlations with overlapping variables.

Single versus two-stage management of long-bone chronic osteomyelitis in adults: a systematic review and meta-analysis

BACKGROUND

Chronic osteomyelitis is a debilitating bone infection, characterized by a persistent infection over months to years, poses diagnostic and therapeutic challenges due to its insidious nature and potential for severe bone and soft tissue destruction. This systematic review and meta-analysis aims to review the literature on the treatment of chronic osteomyelitis in long bones and assess cure rates in single versus two-stage surgery.

METHODS

Following the PRISMA guidelines and registered with PROSPERO (ID: CRD42021231237), this review included studies that reported on the management of chronic osteomyelitis in long bones using either a planned one-stage or two-stage surgical approach in adult patients. Databases searched included Medline, Embase, Web of Science, CINAHL, HMIC, and AMED, using keywords related to osteomyelitis, long bones, and surgical management. Eligibility criteria focused on adults with chronic osteomyelitis in long bones, with outcomes reported after a minimum follow-up of 12 months. The meta-analysis utilized the random-effects model to pool cure rates.

RESULTS

The analysis included 42 studies with a total of 1605 patients. The overall pooled cure rate was 91% (CI 95%) with no significant difference observed between single-stage and two-stage surgeries (X2 = 0.76, P > 0.05). Complications were reported in 26.6% of cases in single-stage procedures and 27.6% in two-stage procedures, with prolonged wound drainage noted as a common issue. Dead space management techniques varied across studies, with antibiotic-loaded calcium sulphate beads used in 30.4% of cases.

CONCLUSION

This meta-analysis reveals no significant difference in cure rates between single and two-stage surgical treatments for chronic osteomyelitis in long bones, supporting the efficacy of both approaches. The current treatment strategy should include a combination of debridement, dead space management using local and systematic antibiotics and soft tissue reconstruction if necessary.

Researcher degrees of freedom in statistical software contribute to unreliable results: A comparison of nonparametric analyses conducted in SPSS, SAS, Stata, and R

Researcher degrees of freedom can affect the results of hypothesis tests and consequently, the conclusions drawn from the data. Previous research has documented variability in accuracy, speed, and documentation of output across various statistical software packages. In the current investigation, we conducted Pearson's chi-square test of independence, Spearman's rank-ordered correlation, Kruskal-Wallis one-way analysis of variance, Wilcoxon Mann-Whitney U rank-sum tests, and Wilcoxon signed-rank tests, along with estimates of skewness and kurtosis, on large, medium, and small samples of real and simulated data in SPSS, SAS, Stata, and R and compared the results with those obtained through hand calculation using the raw computational formulas. Multiple inconsistencies were found in the results produced between statistical packages due to algorithmic variation, computational error, and statistical output. The most notable inconsistencies were due to algorithmic variations in the computation of Pearson's chi-square test conducted on 2 × 2 tables, where differences in p-values reported by different software packages ranged from .005 to .162, largely as a function of sample size. We discuss how such inconsistencies may influence the conclusions drawn from the results of statistical analyses depending on the statistical software used, and we urge researchers to analyze their data across multiple packages to check for inconsistencies and report details regarding the statistical procedure used for data analysis.

Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process

JBI offers a suite of critical appraisal instruments that are freely available to systematic reviewers and researchers investigating the methodological limitations of primary research studies. The JBI instruments are designed to be study-specific and are presented as questions in a checklist. The JBI instruments have existed in a checklist-style format for approximately 20 years; however, as the field of research synthesis expands, many of the tools offered by JBI have become outdated. The JBI critical appraisal tools for quantitative studies (eg, randomized controlled trials, quasi-experimental studies) must be updated to reflect the current methodologies in this field. Cognizant of this and the recent developments in risk-of-bias science, the JBI Effectiveness Methodology Group was tasked with updating the current quantitative critical appraisal instruments. This paper details the methods and rationale that the JBI Effectiveness Methodology Group followed when updating the JBI critical appraisal instruments for quantitative study designs. We detail the key changes made to the tools and highlight how these changes reflect current methodological developments in this field.

The mediating role of maladaptive cognitive schemas regarding the relationship between parenting styles and chronic pain in adolescents: a structural equation modelling approach

BACKGROUND

Although there is a growing body of evidence linking parenting styles to health outcomes, little emphasis has been dedicated to how parenting styles affect chronic pain in adolescents. Given the high prevalence of chronic pain in adolescents and taking into consideration the complexity of chronic pain and the factors affecting it, further research is needed to better understand the processes through which parenting styles affect adolescents' pain. The purpose of the present study was to explore the mediating role of maladaptive schemas in the association between different parenting styles and chronic pain.

METHOD

1302 adolescents aged 12 to 21 in Shiraz, Iran, were randomly selected to participate in this study. To identify adolescents with chronic pain, screening questions based on the 11th revision of the International Classification of Diseases were used. Buri's Parental Authority Questionnaire (PAQ), and Young's Schema Questionnaire-Short Form (YSQ-SF) were used to assess the parenting styles and maladaptive cognitive schemas, respectively. The structural equation modeling approach was carried out to evaluate the direct, indirect, and total effects of different parenting styles on chronic pain.

RESULTS

The results in the SEM models revealed that disconnection/ rejection (β = - 0.043, 95%CI =  - 0.07 to  - 0.02), impaired autonomy/ performance (β =  - 0.01, 95%CI =  - 0.02 to -0.003), over-vigilance/inhibition (β =  - 0.007, 95%CI =  - 0.01 to  - 0.008), and impaired limits schemas (β =  - 0.004, 95%CI =  - 0.006 to  - 0.002) significantly mediated the protective effects of the authoritative parenting style on chronic pain. It was also found that the mediating effects of disconnection/ rejection (β = 0.01, 95%CI = 0.01 to 0.02), and over-vigilance/ inhibition (β = 0.002, 95%CI = 0.001 to 0.02) existed in the relationship between the authoritarian style and chronic pain. The permissive style may also affect chronic pain through disconnection/ rejection (β = 0.004, 95%CI = 0.001 to 0.01), other-directedness (β = 0.01, 95%CI = 0.005 to 0.015), and impaired limits schemas (β = 0.05, 95%CI = 0.04 to 0.06).

DISCUSSION

The findings of the present study showed that maladaptive cognitive schemas play a mediating role in the relationship between parenting styles and chronic pain in adolescents. It seems that the interventions that target the effective communication between the parents and the adolescents can be considered as an important part in the chronic pain management in adolescents.

Virtual-Reality Performance-Based Assessment of Cognitive Functions in Adult Patients With Acquired Brain Injury: A Scoping Review

Individuals with acquired brain injury (ABI) commonly present with impairments in cognitive abilities. As these competencies seem to be predictive of patients' abilities to reintegrate into the everyday settings, it is crucial to assess them properly. However, previous research has indicated that patients may perform relatively well on standard tests of cognitive functioning, but may nonetheless encounter significant difficulties in organizing and executing everyday tasks. In order to overcome this issue, virtual reality (VR) methods have been introduced in clinical practice with the aim of creating assessments that simulate real-world activities and thus, provide a clearer picture of patients' functioning in everyday settings. This review offers an overview of VR assessment tools described in the scientific literature between 2010 and 2019. Overall, 38 relevant records describing 31 different tools were found. Among these tools, 16 assessed executive functions and prospective memory, while the other 15 assessed visuo-spatial abilities. Although promising results have been reported, our analysis indicated that about half of the tools deliver tasks that differ from everyday activities, thus limiting the generalizability of patients' performance to the real-world. Moreover, a variety of methodological shortfalls related to study Internal and External Validity have been highlighted, which hamper the possibility of drawing definite recommendations on tool choice. These limitations suggest the importance of putting considerable efforts into the improvement or development of VR tools for patients with ABI for both research and clinical purposes, considering the great potential of this form of assessment.

Conducting proportional meta-analysis in different types of systematic reviews: a guide for synthesisers of evidence

BACKGROUND

Single group data present unique challenges for synthesises of evidence. Proportional meta-analysis is becoming an increasingly common technique employed for the synthesis of single group data. Proportional meta-analysis shares many similarities with the conduct and reporting of comparative, or pairwise, meta-analysis. While robust and comprehensive methods exist detailing how researchers can conduct a meta-analysis that compares two (or more) groups against a common intervention, there is a scarcity of methodological guidance available to assist synthesisers of evidence in the conduct, interpretation, and importance of proportional meta-analysis in systematic reviews.

MAIN BODY

This paper presents an overview targeted to synthesisers of evidence and systematic review authors that details the methods, importance, and interpretation of a proportional meta-analysis. We provide worked examples of how proportional meta-analyses have been conducted in research syntheses previously and consider the methods, statistical considerations, and presentation of this technique.

CONCLUSION

This overview is designed to serve as practical guidance for synthesisers of evidence in the conduct of proportional meta-analyses.

What is the optimum design for my animal experiment?

Within preclinical research, attention has focused on experimental design and how current practices can lead to poor reproducibility. There are numerous decision points when designing experiments. Ethically, when working with animals we need to conduct a harm-benefit analysis to ensure the animal use is justified for the scientific gain. Experiments should be robust, not use more or fewer animals than necessary, and truly add to the knowledge base of science. Using case studies to explore these decision points, we consider how individual experiments can be designed in several different ways. We use the Experimental Design Assistant (EDA) graphical summary of each experiment to visualise the design differences and then consider the strengths and weaknesses of each design. Through this format, we explore key and topical experimental design issues such as pseudo-replication, blocking, covariates, sex bias, inference space, standardisation fallacy and factorial designs. There are numerous articles discussing these critical issues in the literature, but here we bring together these topics and explore them using real-world examples allowing the implications of the choice of design to be considered. Fundamentally, there is no perfect experiment; choices must be made which will have an impact on the conclusions that can be drawn. We need to understand the limitations of an experiment's design and when we report the experiments, we need to share the caveats that inherently exist.

Biochemical, histological, and neuro-physiological effects of long-term aluminum chloride exposure in rats

This study aimed to evaluate the effect of daily sublethal doses of aluminum (Al) on hematological, physiological, biochemical, and behavioral changes in male albino Wistar rats. In addition, Al tissue accumulation and histopathological changes in the cerebral cortex, liver, and kidney were examined. The rats were randomly separated into three groups. Group 1 included rats who received the median deadly dose (LD₅₀) of aluminum chloride (AlCl₃), group 2 served as the control, and group 3 was treated with a non-lethal dose of AlCl₃ (1.5 mg/kg) intraperitoneally for 45 days. At defined time intervals, hepatic and renal specific enzymes and biochemical activity were measured. In addition, we examined Al accumulation, the condition of the liver via histological methods, and the impact on the cerebral cortex. In comparison to the controls, rats treated with AlCl₃ exhibited a rise in AST, ALT, and ALP enzyme activity. We also saw a significant decrease in body weight and a decrease in total protein, lipids, cholesterol, acetylcholinesterase (AChE), RBCs, and Hb levels compared to the control group. Histopathological examination suggested severe changes in the liver, kidney, and cerebral cortex of the rats. The current study indicates that sublethal daily exposure to AlCl₃ causes hazardous effects, as increased Al concentration in the body is shown to induce detrimental biochemical and histological changes as well as decreased body weight. Therefore, careful attention should be given to treatments requiring long exposure in patients and the potential for accumulation via food and drinking.

Additive, Curvilinear, and Interactive Relations of Anxiety and Depression With Indicators of Psychosocial Functioning

This study examines additive, curvilinear, and interactive relations of anxiety and depression with several subjective indicators of intrapersonal (i.e., hope, self-compassion, shame) and interpersonal (i.e., social connectedness, quality of social relationships) functioning in a sample of adults ( N =  547, Mage = 43.37 ± 12.02, female = 56.88%) seeking treatment for psychological difficulties. Results of complementary analyses were largely consistent with the hypothesis that increasing levels of anxiety and depression would correspond with worse psychosocial functioning, although nonlinear relations indicated that the effect of depression progressively attenuated at higher levels of symptom severity. Whereas the findings generally supported additive effects of anxiety and depression, the hypothesis that there would be synergistic effects of anxiety and depression was not supported. Supplementary group comparisons revealed that the functional implications of subsyndromal combinations of anxiety and depression may be comparable to those associated with symptoms that meet more traditional standards (i.e., syndromal or dimensional definitions) of comorbid anxiety–depression. The findings offer further insight into the complex relations of anxiety and depression with psychosocial functioning and emphasize the importance of detecting and offering appropriate treatments for anxiety and depression symptoms that coexist at subsyndromal levels.

Asymptotic versus exact methods in the analysis of contingency tables: Evidence-based practical recommendations

Controversy over the validity of significance tests in the analysis of contingency tables is motivated by the disagreement between asymptotic and exact p values and its dependence on the magnitude of expected frequencies. Variants of Pearson's X² statistic and their asymptotic distributions were proposed to overcome the difficulties, but several approaches also exist to conduct exact tests. This paper shows that discrepant asymptotic and exact results may or may not occur whether expected frequencies are large or small: Eventual inaccuracy of asymptotic p values is instead caused by idiosyncrasies of the discrete distribution of X². More importantly, discrepancies are also artificially created by the hypergeometric sampling model used to perform exact tests. Exact computations under the alternative full-multinomial or product-multinomial models require eliminating nuisance parameters and we propose a novel method that integrates them out. The resultant exact distributions are very accurately approximated by the asymptotic distribution, which eliminates concerns about the accuracy of the latter. We also discuss that the two-stage approach that tests for significance of residuals conditional on a significant X² test is inadvisable and that an alternative single-stage test preserves Type-I error rates and further eliminates concerns about asymptotic accuracy.

Understanding Speech Amid the Jingle and Jangle: Recommendations for Improving Measurement Practices in Listening Effort Research

The latent constructs psychologists study are typically not directly accessible, so researchers must design measurement instruments that are intended to provide insights about those constructs. Construct validation-assessing whether instruments measure what they intend to-is therefore critical for ensuring that the conclusions we draw actually reflect the intended phenomena. Insufficient construct validation can lead to the jingle fallacy-falsely assuming two instruments measure the same construct because the instruments share a name (Thorndike, 1904)-and the jangle fallacy-falsely assuming two instruments measure different constructs because the instruments have different names (Kelley, 1927). In this paper, we examine construct validation practices in research on listening effort and identify patterns that strongly suggest the presence of jingle and jangle in the literature. We argue that the lack of construct validation for listening effort measures has led to inconsistent findings and hindered our understanding of the construct. We also provide specific recommendations for improving construct validation of listening effort instruments, drawing on the framework laid out in a recent paper on improving measurement practices (Flake & Fried, 2020). Although this paper addresses listening effort, the issues raised and recommendations presented are widely applicable to tasks used in research on auditory perception and cognitive psychology.

Social problem solving in social anxiety disorder

Successful social problem solving requires both an adaptive orientation toward the problem and the necessary skills to generate relevant and effective solutions. Surprisingly few studies have examined social problem solving in the context of social anxiety. We examined social problem solving in 38 participants with social anxiety disorder (SAD) in comparison to 30 healthy control (HC) participants with no history of anxiety problems. Participants rated their problem solving attitudes and abilities (i.e., problem orientation) and then generated solutions to hypothetical interpersonal problems from both their own perspective and that of an objective other. These solutions were coded for effectiveness and relevance, as well as the degree to which the solution was active versus passive. Participants with SAD exhibited a more negative problem orientation than HC participants. Furthermore, although SAD and HC participants demonstrated no overall differences in generating relevant and effective solutions to interpersonal problems, utilizing a personal perspective facilitated the generation of more active solutions for HC participants, but less active solutions for those with SAD. Findings illuminate new research directions regarding social problem solving in social anxiety, with potential implications for applied intervention.

Bayesian Estimation with Informative Priors is Indistinguishable from Data Falsification

Criticism of null hypothesis significance testing, confidence intervals, and frequentist statistics in general has evolved into advocacy of Bayesian analyses with informative priors for strong inference. This paper shows that Bayesian analysis with informative priors is formally equivalent to data falsification because the information carried by the prior can be expressed as the addition of fabricated observations whose statistical characteristics are determined by the parameters of the prior. This property of informative priors makes clear that only the use of non-informative, uniform priors in all types of Bayesian analyses is compatible with standards of research integrity. At the same time, though, Bayesian estimation with uniform priors yields point and interval estimates that are identical or nearly identical to those obtained with frequentist methods. At a qualitative level, frequentist and Bayesian outcomes have different interpretations but they are interchangeable when uniform priors are used. Yet, Bayesian interpretations require either the assumption that population parameters are random variables (which they are not) or an explicit acknowledgment that the posterior distribution (which is thus identical to the likelihood function except for a scale factor) only expresses the researcher's beliefs and not any information about the parameter of concern.

Should Psychology Journals Adopt Specialized Statistical Review?

Readers of peer-reviewed research may assume that the reported statistical analyses supporting scientific claims have been closely scrutinized and surpass a high-quality threshold. However, widespread misunderstanding and misuse of statistical concepts and methods suggests that suboptimal or erroneous statistical practice is routinely overlooked during peer review in psychology. Here, we explore whether psychology journals could ameliorate some of the field’s statistical ailments by adopting specialized statistical review: a focused technical assessment, performed by statistical experts, that addresses the analysis and presentation of quantitative information and supplements regular peer review. We discuss evidence from a recent survey of journal editors suggesting that specialized statistical review may be unusual in psychology journals and is regarded by many editors as unnecessary. We contrast these views with those in the biomedical domain, where statistical review has been considered a partial preventive measure against the improper use of statistics since the late 1970s. We suggest that the current “credibility revolution” presents an opportune occasion for psychology journals to consider adopting specialized statistical review.

Nonparametric tests for equality of psychometric functions

Many empirical studies measure psychometric functions (curves describing how observers' performance varies with stimulus magnitude) because these functions capture the effects of experimental conditions. To assess these effects, parametric curves are often fitted to the data and comparisons are carried out by testing for equality of mean parameter estimates across conditions. This approach is parametric and, thus, vulnerable to violations of the implied assumptions. Furthermore, testing for equality of means of parameters may be misleading: Psychometric functions may vary meaningfully across conditions on an observer-by-observer basis with no effect on the mean values of the estimated parameters. Alternative approaches to assess equality of psychometric functions per se are thus needed. This paper compares three nonparametric tests that are applicable in all situations of interest: The existing generalized Mantel-Haenszel test, a generalization of the Berry-Mielke test that was developed here, and a split variant of the generalized Mantel-Haenszel test also developed here. Their statistical properties (accuracy and power) are studied via simulation and the results show that all tests are indistinguishable as to accuracy but they differ non-uniformly as to power. Empirical use of the tests is illustrated via analyses of published data sets and practical recommendations are given. The computer code in MATLAB and R to conduct these tests is available as Electronic Supplemental Material.

Statistical considerations in reporting cardiovascular research

The problem of inadequate statistical reporting is long standing and widespread in the biomedical literature, including in cardiovascular physiology. Although guidelines for reporting statistics have been available in clinical medicine for some time, there are currently no guidelines specific to cardiovascular physiology. To assess the need for guidelines, we determined the type and frequency of statistical tests and procedures currently used in the American Journal of Physiology-Heart and Circulatory Physiology. A PubMed search for articles published in the American Journal of Physiology-Heart and Circulatory Physiology between January 1, 2017, and October 6, 2017, provided a final sample of 146 articles evaluated for methods used and 38 articles for indepth analysis. The t-test and ANOVA accounted for 71% (212 of 300 articles) of the statistical tests performed. Of six categories of post hoc tests, Bonferroni and Tukey tests were used in 63% (62 of 98 articles). There was an overall lack in details provided by authors publishing in the American Journal of Physiology-Heart and Circulatory Physiology, and we compiled a list of recommended minimum reporting guidelines to aid authors in preparing manuscripts. Following these guidelines could substantially improve the quality of statistical reports and enhance data rigor and reproducibility.

Diagnosis checking of statistical analysis in RCTs indexed in PubMed

BACKGROUND

Statistical analysis is essential for reporting of the results of randomized controlled trials (RCTs), as well as evaluating their effectiveness. However, the validity of a statistical analysis also depends on whether the assumptions of that analysis are valid.

OBJECTIVE

To review all RCTs published in journals indexed in PubMed during December 2014 to provide a complete picture of how RCTs handle assumptions of statistical analysis.

METHODS

We reviewed all RCTs published in December 2014 that appeared in journals indexed in PubMed using the Cochrane highly sensitive search strategy. The 2014 impact factors of the journals were used as proxies for their quality. The type of statistical analysis used and whether the assumptions of the analysis were tested were reviewed.

RESULTS

In total, 451 papers were included. Of the 278 papers that reported a crude analysis for the primary outcomes, 31 (27·2%) reported whether the outcome was normally distributed. Of the 172 papers that reported an adjusted analysis for the primary outcomes, diagnosis checking was rarely conducted, with only 20%, 8·6% and 7% checked for generalized linear model, Cox proportional hazard model and multilevel model, respectively. Study characteristics (study type, drug trial, funding sources, journal type and endorsement of CONSORT guidelines) were not associated with the reporting of diagnosis checking.

CONCLUSION

The diagnosis of statistical analyses in RCTs published in PubMed-indexed journals was usually absent. Journals should provide guidelines about the reporting of a diagnosis of assumptions.

Variable criteria sequential stopping rule: Validity and power with repeated measures ANOVA, multiple correlation, MANOVA and relation to Chi-square distribution

The variable criteria sequential stopping rule (vcSSR) is an efficient way to add sample size to planned ANOVA tests while holding the observed rate of Type I errors, α_o, constant. The only difference from regular null hypothesis testing is that criteria for stopping the experiment are obtained from a table based on the desired power, rate of Type I errors, and beginning sample size. The vcSSR was developed using between-subjects ANOVAs, but it should work with p values from any type of F test. In the present study, the α_o remained constant at the nominal level when using the previously published table of criteria with repeated measures designs with various numbers of treatments per subject, Type I error rates, values of ρ, and four different sample size models. New power curves allow researchers to select the optimal sample size model for a repeated measures experiment. The criteria held α_o constant either when used with a multiple correlation that varied the sample size model and the number of predictor variables, or when used with MANOVA with multiple groups and two levels of a within-subject variable at various levels of ρ. Although not recommended for use with χ² tests such as the Friedman rank ANOVA test, the vcSSR produces predictable results based on the relation between F and χ². Together, the data confirm the view that the vcSSR can be used to control Type I errors during sequential sampling with any t- or F-statistic rather than being restricted to certain ANOVA designs.

Thou Shalt Not Bear False Witness Against Null Hypothesis Significance Testing

Null hypothesis significance testing (NHST) has been the subject of debate for decades and alternative approaches to data analysis have been proposed. This article addresses this debate from the perspective of scientific inquiry and inference. Inference is an inverse problem and application of statistical methods cannot reveal whether effects exist or whether they are empirically meaningful. Hence, raising conclusions from the outcomes of statistical analyses is subject to limitations. NHST has been criticized for its misuse and the misconstruction of its outcomes, also stressing its inability to meet expectations that it was never designed to fulfil. Ironically, alternatives to NHST are identical in these respects, something that has been overlooked in their presentation. Three of those alternatives are discussed here (estimation via confidence intervals and effect sizes, quantification of evidence via Bayes factors, and mere reporting of descriptive statistics). None of them offers a solution to the problems that NHST is purported to have, all of them are susceptible to misuse and misinterpretation, and some bring around their own problems (e.g., Bayes factors have a one-to-one correspondence with p values, but they are entirely deprived of an inferential framework). Those alternatives also fail to cover a broad area of inference not involving distributional parameters, where NHST procedures remain the only (and suitable) option. Like knives or axes, NHST is not inherently evil; only misuse and misinterpretation of its outcomes needs to be eradicated.

A Simulation Study of Threats to Validity in Quasi-Experimental Designs: Interrelationship between Design, Measurement, and Analysis

The Campbellian tradition provides a conceptual framework to assess threats to validity. On the other hand, different models of causal analysis have been developed to control estimation biases in different research designs. However, the link between design features, measurement issues, and concrete impact estimation analyses is weak. In order to provide an empirical solution to this problem, we use Structural Equation Modeling (SEM) as a first approximation to operationalize the analytical implications of threats to validity in quasi-experimental designs. Based on the analogies established between the Classical Test Theory (CTT) and causal analysis, we describe an empirical study based on SEM in which range restriction and statistical power have been simulated in two different models: (1) A multistate model in the control condition (pre-test); and (2) A single-trait-multistate model in the control condition (post-test), adding a new mediator latent exogenous (independent) variable that represents a threat to validity. Results show, empirically, how the differences between both the models could be partially or totally attributed to these threats. Therefore, SEM provides a useful tool to analyze the influence of potential threats to validity.

Contextual sensitivity in scientific reproducibility

In recent years, scientists have paid increasing attention to reproducibility. For example, the Reproducibility Project, a large-scale replication attempt of 100 studies published in top psychology journals found that only 39% could be unambiguously reproduced. There is a growing consensus among scientists that the lack of reproducibility in psychology and other fields stems from various methodological factors, including low statistical power, researcher's degrees of freedom, and an emphasis on publishing surprising positive results. However, there is a contentious debate about the extent to which failures to reproduce certain results might also reflect contextual differences (often termed "hidden moderators") between the original research and the replication attempt. Although psychologists have found extensive evidence that contextual factors alter behavior, some have argued that context is unlikely to influence the results of direct replications precisely because these studies use the same methods as those used in the original research. To help resolve this debate, we recoded the 100 original studies from the Reproducibility Project on the extent to which the research topic of each study was contextually sensitive. Results suggested that the contextual sensitivity of the research topic was associated with replication success, even after statistically adjusting for several methodological characteristics (e.g., statistical power, effect size). The association between contextual sensitivity and replication success did not differ across psychological subdisciplines. These results suggest that researchers, replicators, and consumers should be mindful of contextual factors that might influence a psychological process. We offer several guidelines for dealing with contextual sensitivity in reproducibility.

Trust me, it is valid: Research validity in pharmacy education research

Research validity is a complex concept that is often used loosely or conflated with concepts in measurement validity in published quantitative pharmacy education literature. The problem begins with a lack of clarity of the distinction between four types of research validity including measurement, statistical conclusion, internal, and external validity (i.e., generalizability). In many cases published studies provide only incomplete discussions of measurement and external validity. The problem is exacerbated within the context of measurement validity where validation efforts are often reduced to statements about established levels of reliability. Ineffective discussions of research validity make it difficult to interpret study findings. After reading this article, the reader will be able to identify the different types research validity and discuss issues of research validity in quantitative pharmacy education research more completely.

A common control group - optimising the experiment design to maximise sensitivity

Methods for choosing an appropriate sample size in animal experiments have received much attention in the statistical and biological literature. Due to ethical constraints the number of animals used is always reduced where possible. However, as the number of animals decreases so the risk of obtaining inconclusive results increases. By using a more efficient experimental design we can, for a given number of animals, reduce this risk. In this paper two popular cases are considered, where planned comparisons are made to compare treatments back to control and when researchers plan to make all pairwise comparisons. By using theoretical and empirical techniques we show that for studies where all pairwise comparisons are made the traditional balanced design, as suggested in the literature, maximises sensitivity. For studies that involve planned comparisons of the treatment groups back to the control group, which are inherently more sensitive due to the reduced multiple testing burden, the sensitivity is maximised by increasing the number of animals in the control group while decreasing the number in the treated groups.

Research practices that can prevent an inflation of false-positive rates

Recent studies have indicated that research practices in psychology may be susceptible to factors that increase false-positive rates, raising concerns about the possible prevalence of false-positive findings. The present article discusses several practices that may run counter to the inflation of false-positive rates. Taking these practices into account would lead to a more balanced view on the false-positive issue. Specifically, we argue that an inflation of false-positive rates would diminish, sometimes to a substantial degree, when researchers (a) have explicit a priori theoretical hypotheses, (b) include multiple replication studies in a single paper, and (c) collect additional data based on observed results. We report findings from simulation studies and statistical evidence that support these arguments. Being aware of these preventive factors allows researchers not to overestimate the pervasiveness of false-positives in psychology and to gauge the susceptibility of a paper to possible false-positives in practical and fair ways.