Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students' understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

Sleep duration and psychotic experiences in patients at risk of psychosis: A secondary analysis of the EDIE-2 trial

Sleep disturbance is common among individuals at risk of psychosis, yet few studies have investigated the relationship between sleep disturbance and clinical trajectory. The Early Detection and Intervention Evaluation (EDIE-2) trial provides longitudinal data on sleep duration and individual psychotic experiences from a cohort of individuals at risk of psychosis, which this study utilises in an opportunistic secondary analysis. Shorter and more variable sleep was hypothesised to be associated with more severe psychotic experiences and lower psychological wellbeing. Mixed effect models were used to test sleep duration and range as predictors of individual psychotic experiences and psychological wellbeing over the 12-24 months (with assessments every 3 months) in 160 participants. Shorter sleep duration was associated with more severe delusional ideas and hallucinations cross-sectionally and longitudinally. The longitudinal relationships did not remain significant after conservative controls were added for the previous severity of psychotic experiences. No significant relationships were found between the sleep variables and other psychotic experiences (e.g. cognitive disorganisation), or psychological wellbeing. The results support a relationship between shorter sleep duration and delusional ideas and hallucinations. Future studies should focus on improving sleep disturbance measurement, and test whether treating sleep improves clinical trajectory in the at-risk group.

Deception Detection and Truth Detection Are Dependent on Different Cognitive and Emotional Traits: An Investigation of Emotional Intelligence, Theory of Mind, and Attention

Despite evidence that variation exists between individuals in high-stakes truth and deception detection accuracy rates, little work has investigated what differences in individuals' cognitive and emotional abilities contribute to this variation. Our study addressed this question by examining the role played by cognitive and affective theory of mind (ToM), emotional intelligence (EI), and various aspects of attention (alerting, orienting, executive control) in explaining variation in accuracy rates among 115 individuals (87 women; mean age = 27.04 years [ SD = 11.32]) who responded to video clips of truth-tellers and liars in real-world, high-stakes contexts. Faster attentional alerting supported truth detection, and better cognitive ToM and perception of emotion (an aspect of EI) supported deception detection. This evidence indicates that truth and deception detection are distinct constructs supported by different abilities. Future research may address whether interventions targeting these cognitive and emotional traits can also contribute to improving detection skill.

Assessment of metacognitive beliefs in an at risk mental state for psychosis: A validation study of the Metacognitions Questionnaire-30

AIM

The Metacognitions Questionnaire-30 (MCQ-30) has been used to assess metacognitive beliefs in a range of mental health problems. The aim of this study is to assess the validity of the MCQ-30 in people at risk for psychosis.

METHODS

One hundred eighty-five participants meeting criteria for an at risk mental state completed the MCQ-30 as part of their involvement in a randomized controlled trial. Confirmatory and exploratory factor analyses were conducted to assess factor structure and construct validity.

RESULTS

Confirmatory factor analyses confirmed the original five-factor structure of the MCQ-30. Examination of principal component analysis and parallel analysis outputs also suggested a five-factor structure. Correlation analyses including measures of depression, social anxiety, and beliefs about paranoia showed evidence of convergent validity. Discriminant validity was supported using the normalizing subscale of the beliefs about paranoia tool.

CONCLUSIONS

The MCQ-30 demonstrated good fit using the original five-factor model, acceptable to very good internal consistency of items was evident and clinical usefulness in those at risk for psychosis was demonstrated.

Affective theory of mind inferences contextually influence the recognition of emotional facial expressions

The recognition of emotional facial expressions is often subject to contextual influence, particularly when the face and the context convey similar emotions. We investigated whether spontaneous, incidental affective theory of mind inferences made while reading vignettes describing social situations would produce context effects on the identification of same-valenced emotions (Experiment 1) as well as differently-valenced emotions (Experiment 2) conveyed by subsequently presented faces. Crucially, we found an effect of context on reaction times in both experiments while, in line with previous work, we found evidence for a context effect on accuracy only in Experiment 1. This demonstrates that affective theory of mind inferences made at the pragmatic level of a text can automatically, contextually influence the perceptual processing of emotional facial expressions in a separate task even when those emotions are of a distinctive valence. Thus, our novel findings suggest that language acts as a contextual influence to the recognition of emotional facial expressions for both same and different valences.

Components of therapy as mechanisms of change in cognitive therapy for people at risk of psychosis: analysis of the EDIE-2 trial

BACKGROUND

Research suggests that the way in which cognitive therapy is delivered is an important factor in determining outcomes. We test the hypotheses in which the development of a shared problem list, use of case formulation, homework tasks and active intervention strategies will act as process variables.

METHOD

Presence of these components during therapy is taken from therapist notes. The direct and indirect effect of the intervention is estimated by an instrumental variable analysis.

RESULTS

A significant decrease in the symptom score for case formulation (coefficient = -23, 95% CI -44 to -1.7, P = 0.036) and homework (coefficient = -0.26, 95% CI -0.51 to -0.001, P = 0.049) is found. Improvement with the inclusion of active change strategies is of borderline significance (coefficient = -0.23, 95% CI -0.47 to 0.005, P = 0.056).

CONCLUSIONS

There is a greater treatment effect if formulation and homework are involved in therapy. However, high correlation between components means that these may be indicators of overall treatment fidelity.

Internalized stigma, emotional dysfunction and unusual experiences in young people at risk of psychosis

AIMS

To investigate the relationship between internalized stigma, depression, social anxiety and unusual experiences in young people considered to be at risk of developing psychosis.

METHODS

A total of 288 participants meeting criteria for an at-risk mental state were recruited as part of a multisite randomized controlled trial of cognitive behavioural therapy for people meeting criteria for an at risk mental state (ARMS). The sample was assessed at baseline and 6 months using measures of at risk mental states, internalized stigma, depression and social anxiety.

RESULTS

The Personal Beliefs about Experiences Questionnaire was validated for use with an ARMS sample. Correlational analyses at baseline indicated significant relationships between internalized stigma and: (i) depression; (ii) social anxiety; (iii) distress associated with unusual psychological experiences; and (iv) suicidal thinking. Regression analysis indicates negative appraisals of unusual experiences contributed significantly to depression scores at 6-month follow up when controlling for baseline depression and unusual psychological experiences.

CONCLUSIONS

These findings suggest that internalized stigma may contribute to the development and maintenance of depression in young people at risk of psychosis.

Negative cognition, affect, metacognition and dimensions of paranoia in people at ultra-high risk of psychosis: a multi-level modelling analysis

BACKGROUND

Paranoia is one of the commonest symptoms of psychosis but has rarely been studied in a population at risk of developing psychosis. Based on existing theoretical models, including the proposed distinction between ‘poor me’ and ‘bad me’ paranoia, we aimed to test specific predictions about associations between negative cognition, metacognitive beliefs and negative emotions and paranoid ideation and the belief that persecution is deserved (deservedness).

METHOD

We used data from 117 participants from the Early Detection and Intervention Evaluation for people at risk of psychosis (EDIE-2) trial of cognitive–behaviour therapy, comparing them with samples of psychiatric in-patients and healthy students from a previous study. Multi-level modelling was utilized to examine predictors of both paranoia and deservedness, with post-hoc planned comparisons conducted to test whether person-level predictor variables were associated differentially with paranoia or with deservedness.

RESULTS

Our sample of at-risk mental state participants was not as paranoid, but reported higher levels of ‘bad-me’ deservedness, compared with psychiatric in-patients. We found several predictors of paranoia and deservedness. Negative beliefs about self were related to deservedness but not paranoia, whereas negative beliefs about others were positively related to paranoia but negatively with deservedness. Both depression and negative metacognitive beliefs about paranoid thinking were specifically related to paranoia but not deservedness.

CONCLUSIONS

This study provides evidence for the role of negative cognition, metacognition and negative affect in the development of paranoid beliefs, which has implications for psychological interventions and our understanding of psychosis.

Psychopathology and affect dysregulation across the continuum of psychosis: a multiple comparison group study

AIM

There is evidence that psychotic-like phenomena can be detected within the general population and that psychotic experiences lie on a continuum that also spans affective states. We aimed to investigate comparisons of a first-episode psychosis group, an 'at-risk mental state group' and a help-seeking control group with non-patients to explore whether affective states lie on a continuum of psychosis.

METHOD

Measures of psychotic-like experiences, social anxiety and depression were administered to 20 patients experiencing first-episode psychosis (FEP), 113 patients experiencing an 'at-risk' mental state (ARMS), 28 patients who were help-seeking but not experiencing a FEP or ARMS (HSC) and 30 non-clinical participants (NC).

RESULTS

For distress in relation to psychotic-like experiences, the FEP, ARMS and HSC groups scored significantly higher than the NC group for the perceptual abnormalities and non-bizarre ideas. In terms of severity of psychotic experiences, the FEP scored the highest, followed by the ARMS group, followed by the HSC and NC groups. The clinical groups scored significantly higher for depression than the non-clinical group. Interestingly, only the FEP and the ARMS groups scored significantly higher than non-patients for social anxiety.

CONCLUSIONS

These findings suggest that a psychosis continuum exists; however, this does not suggest that both psychosis and affective symptoms lie on the same continuum, rather it would appear the presence of such affective states that may affect help-seeking behaviour and clinical status. The implications of these findings for clinical practice are discussed.

Impact of cognitive therapy on internalised stigma in people with at-risk mental states

BACKGROUND

Internalised stigma in young people meeting criteria for at-risk mental states (ARMS) has been highlighted as an important issue, and it has been suggested that provision of cognitive therapy may increase such stigma.

AIMS

To investigate the effects of cognitive therapy on internalised stigma using a secondary analysis of data from the EDIE-2 trial.

METHOD

Participants meeting criteria for ARMS were recruited as part of a multisite randomised controlled trial of cognitive therapy for prevention and amelioration of psychosis. Participants were assessed at baseline and at 6, 12, 18 and 24 months using measures of psychotic experiences, symptoms and internalised stigma.

RESULTS

Negative appraisals of experiences were significantly reduced in the group assigned to cognitive therapy (estimated difference at 12 months was -1.36 (95% CI -2.69 to -0.02), P = 0.047). There was no difference in social acceptability of experiences (estimated difference at 12 months was 0.46, 95% CI -0.05 to 0.98, P = 0.079).

CONCLUSIONS

These findings suggest that, rather than increasing internalised stigma, cognitive therapy decreases negative appraisals of unusual experiences in young people at risk of psychosis; as such, it is a non-stigmatising intervention for this population.

The development of a population-level clinical screening tool for self-harm repetition and suicide: the ReACT Self-Harm Rule

BACKGROUND

Self-harm is a common reason for Emergency Department (ED) attendance. We aimed to develop a clinical tool to help identify patients at higher risk of repeat self-harm, or suicide, within 6 months of an ED self-harm presentation.

METHOD

The tool, the ReACT Self-Harm Rule, was derived using multicentre data from a prospective cohort study. Binary recursive partitioning was applied to data from two centres, and data from a separate centre were used to test the tool. There were 29 571 self-harm presentations to five hospital EDs between January 2003 and June 2007, involving 18 680 adults aged ⩾16 years. We estimated sensitivity, specificity and positive and negative predictive values to measure the performance of the tool.

RESULTS

A self-harm presentation was classified as higher risk if at least one of the following factors was present: recent self-harm (in the past year), living alone or homelessness, cutting as a method of harm and treatment for a current psychiatric disorder. The rule performed with 95% sensitivity [95% confidence interval (CI) 94-95] and 21% specificity (95% CI 21-22), and had a positive predictive value of 30% (95% CI 30-31) and a negative predictive value of 91% (95% CI 90-92) in the derivation centres; it identified 83/92 of all subsequent suicides.

CONCLUSIONS

The ReACT Self-Harm Rule might be used as a screening tool to inform the process of assessing self-harm presentations to ED. The four risk factors could also be used as an adjunct to in-depth psychosocial assessment to help guide risk formulation. The use of multicentre data helped to maximize the generalizability of the tool, but we need to further verify its external validity in other localities.

Early detection and intervention evaluation for people at risk of psychosis: multisite randomised controlled trial

OBJECTIVE

To determine whether cognitive therapy is effective in preventing the worsening of emerging psychotic symptoms experienced by help seeking young people deemed to be at risk for serious conditions such as schizophrenia.

DESIGN

Multisite single blind randomised controlled trial.

SETTING

Diverse services at five UK sites.

PARTICIPANTS

288 participants aged 14-35 years (mean 20.74, SD 4.34 years) at high risk of psychosis: 144 were assigned to cognitive therapy plus monitoring of mental state and 144 to monitoring of mental state only. Participants were followed-up for a minimum of 12 months and a maximum of 24 months.

INTERVENTION

Cognitive therapy (up to 26 (mean 9.1) sessions over six months) plus monitoring of mental state compared with monitoring of mental state only.

MAIN OUTCOME MEASURES

Primary outcome was scores on the comprehensive assessment of at risk mental states (CAARMS), which provides a dichotomous transition to psychosis score and ordinal scores for severity of psychotic symptoms and distress. Secondary outcomes included emotional dysfunction and quality of life.

RESULTS

Transition to psychosis based on intention to treat was analysed using discrete time survival models. Overall, the prevalence of transition was lower than expected (23/288; 8%), with no significant difference between the two groups (proportional odds ratio 0.73, 95% confidence interval 0.32 to 1.68). Changes in severity of symptoms and distress, as well as secondary outcomes, were analysed using random effects regression (analysis of covariance) adjusted for site and baseline symptoms. Distress from psychotic symptoms did not differ (estimated difference at 12 months -3.00, 95% confidence interval -6.95 to 0.94) but their severity was significantly reduced in the group assigned to cognitive therapy (estimated between group effect size at 12 months -3.67, -6.71 to -0.64, P=0.018).

CONCLUSIONS

Cognitive therapy plus monitoring did not significantly reduce transition to psychosis or symptom related distress but reduced the severity of psychotic symptoms in young people at high risk. Most participants in both groups improved over time. The results have important implications for the at risk mental state concept.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN56283883.

Early detection and intervention evaluation for people at high-risk of psychosis-2 (EDIE-2): trial rationale, design and baseline characteristics

AIMS

Much research has begun to focus on the identification of people who are at high risk of developing psychosis, and clinical services have been initiated for this population. However, only a small number of studies have reported on the efficacy of interventions for preventing or delaying the onset of psychosis. The results of prior work suggest that cognitive therapy (CT) may be an effective, well-tolerated treatment. We report on the rationale and design for a large-scale, multi-site randomized, controlled trial of CT for people who are assessed to be at high risk of psychosis because of either state or state-plus-trait risk factors.

METHODS

The study employs a single-blind design in which all participants receive frequent mental-state monitoring, which will efficiently detect transition to psychosis, and half are randomized to weekly sessions of CT for up to 6 months. Participants will be followed-up for a minimum of 12 months and to a maximum of 2 years.

RESULTS

We report the characteristics of the final sample at baseline (n=288).

CONCLUSIONS

Our study aimed to expand the currently limited evidence base for best practice in interventions for individuals at high risk of psychosis.

Mental state references in psychosis: a pilot study of prompted implicit mentalising during dialogue and its relationship with social functioning

INTRODUCTION

Few studies of psychosis have examined dialogue-based implicit mentalising even though this is likely to reflect the skills required in everyday life better than more typical mentalising tasks. Using a semistructured dialogue task, we predicted that a psychosis sample would be impaired in "prompted" online mentalising (i.e., the frequency and variety of mental and emotional state words as well as references to own mental state) and that performance would relate to social functioning.

METHODS

Eighteen adults with psychosis and nine healthy adults were each recorded during four semistructured dialogues, which were transcribed, coded, scored, and quantitatively analysed. The patients also completed a measure of social functioning.

RESULTS

Compared to controls, the psychosis participants referred to others' mental and emotional states significantly less and with a lower variety of words. These findings were all large effects with sufficient observed power. There was no significant difference in references to own mental state. The relationships between mentalising and social functioning were mostly modest.

CONCLUSIONS

Although prompted implicit mentalising is impoverished in psychosis, self-knowledge appears to be intact. Simulation-based mentalising may be spared in the context of impoverished theory-based mentalising. Also, implicit mentalising contributes to social functioning, corroborating the results of previous work.

Alignment and theory of mind in schizophrenia

INTRODUCTION

We predicted that participants with schizophrenia would be able to successfully "align" during conversation in the context of impaired theory of mind. Alignment is a process by which interlocutors' representations of the conversational situation converge; and it may, in part, explain how people with schizophrenia can often participate successfully in dialogue despite experiencing impaired mentalising.

METHODS

Fifty-nine people with schizophrenia and 38 healthy adults completed a standardised, empirical conversational alignment task with a mentalising component and a measure of current IQ. The patients also completed two independent theory of mind tests. We used ANCOVAs to compare the groups' performances.

RESULTS

The participants with schizophrenia and the healthy participants demonstrated equivalent alignment skills even though the schizophrenia participants displayed clear theory of mind difficulties. Symptom subtype analyses found no differences between subtype groups in alignment, but healthy controls and remitted patients performed significantly better on the mentalising component than the paranoia group.

CONCLUSIONS

These results are consistent with the schizophrenia participants having intact alignment skills alongside mentalising impairments. We propose that this explains why people with schizophrenia can often participate successfully in conversation but have difficulties with more complex dialogues, with resolving misunderstandings, and with untangling ambiguities during conversation.