Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method

There is increasing concern that research is not transparent and that empirical results are often impossible to reproduce. Guidelines for undertaking reproducible research have been proposed in a number of academic areas (e.g. computational economics, psychology and medical research), however currently there are no guidelines for sociological research. This methodological paper provides guidance for undertaking reproducible statistically orientated sociological research. We provide an extended demonstration of the issues associated with reproducing results and undertaking transparent analyses. We draw on suitable concepts and techniques from open research, e-research and computing. We propose a set of Newer Rules of the Sociological Method, for undertaking transparent statistically orientated sociological research that supports reproducibility.

An investigation of social class inequalities in general cognitive ability in two British birth cohorts

The 'Flynn effect' describes the substantial and long-standing increase in average cognitive ability test scores, which has been observed in numerous psychological studies. Flynn makes an appeal for researchers to move beyond psychology's standard disciplinary boundaries and to consider sociological contexts, in order to develop a more comprehensive understanding of cognitive inequalities. In this article we respond to this appeal and investigate social class inequalities in general cognitive ability test scores over time. We analyse data from the National Child Development Study (1958) and the British Cohort Study (1970). These two British birth cohorts are suitable nationally representative large-scale data resources for studying inequalities in general cognitive ability. We observe a large parental social class effect, net of parental education and gender in both cohorts. The overall finding is that large social class divisions in cognitive ability can be observed when children are still at primary school, and similar patterns are observed in each cohort. Notably, pupils with fathers at the lower end of the class structure are at a distinct disadvantage. This is a disturbing finding and it is especially important because cognitive ability is known to influence individuals later in the lifecourse.

The role of administrative data in the big data revolution in social science research

The term big data is currently a buzzword in social science, however its precise meaning is ambiguous. In this paper we focus on administrative data which is a distinctive form of big data. Exciting new opportunities for social science research will be afforded by new administrative data resources, but these are currently under appreciated by the research community. The central aim of this paper is to discuss the challenges associated with administrative data. We emphasise that it is critical for researchers to carefully consider how administrative data has been produced. We conclude that administrative datasets have the potential to contribute to the development of high-quality and impactful social science research, and should not be overlooked in the emerging field of big data.

Ethnicity and ethnic group measures in social survey research

This article is a review of issues associated with measuring ethnicity and using ethnicity measures in social science research. The review is oriented towards researchers who undertake secondary analyses of large-scale multipurpose social science datasets. The article begins with an outline of two main approaches used in social surveys to measure ethnicity, the ‘mutually exclusive category’ approach and the ‘multiple characteristics’ approach. We also describe approaches to the use of ethnicity measures in cross-national comparative research. We emphasise the value of sensitivity analyses. We also encourage researchers to carefully consider the possible relationships between ethnicity and other important variables in order to avoid spurious interpretations of the effects of ethnicity.

A review of educational attainment measures for social survey research

This article is a review of issues associated with measuring education and using educational measures in social science research. The review is orientated towards researchers who undertake secondary analyses of large-scale micro-level social science datasets. The article begins with an outline of important context, which impinges upon the measurement of education. The United Kingdom is the focus of this review, but similar issues apply to other nation states. We provide a critical introduction to the main approaches to measuring education in social survey research, which include measuring years of education, using categorical qualification based measures and scaling approaches. We advocate the use of established education measures to better facilitate comparability and replication. We conclude by making the recommendation that researchers place careful thought into which educational measure they select, and that researchers should routinely engage in appropriate sensitivity analyses.

A Review of occupation-based social classifications for social survey research

This article is a review of issues associated with measuring occupations and using occupation-based socio-economic classifications in social science research. The review is orientated towards researchers who undertake secondary analysis of large-scale micro-level social science datasets. This article begins with an outline of how to handle raw occupational information. This is followed by an introduction to the two main approaches to measuring occupations and a third lesser known but intellectually innovative approach. The three approaches are social class schemes, social stratification scales and the microclass approach. International comparisons are briefly described and a discussion of intersectionality with other key variables such as age and gender is provided. We are careful to emphasise that this article does not advocate the uncritical adoption of any one particular occupation-based socio-economic measure over and above other alternatives. Rather, we are advocating that researchers should choose from the portfolio of existing socio-economic measures in an informed and empirically defensible way, and we strongly advocate undertaking sensitivity analyses. We conclude that researchers should always use existing socio-economic measures that have agreed on and well-documented standards. We strongly advise researchers not to develop their own measures without strong justification nor to use existing measures in an un-prescribed or ad hoc manner.

Statistical modelling of key variables in social survey data analysis

The application of statistical modelling techniques has become a cornerstone of analyses of large-scale social survey data. Bringing this special section on key variables to a close, this final article discusses several important issues relating to the inclusion of key variables in statistical modelling analyses. We outline two, often neglected, issues that are relevant to a great many applications of statistical models based upon social survey data. The first is known as the reference category problem and is related to the interpretation of categorical explanatory variables. The second is the interpretation and comparison of the effects from models for non-linear outcomes. We then briefly discuss other common complexities in using statistical models for social science research; these include the non-linear transformation of variables, and considerations of intersectionality and interaction effects. We conclude by emphasising the importance of two, often overlooked, elements of the social survey data analysis process, sensitivity analysis and documentation for replication. We argue that more attention should routinely be devoted to these issues.

Childhood residential mobility and health in late adolescence and adulthood: findings from the West of Scotland Twenty-07 Study

Background

The relationship between childhood residential mobility and health in the UK is not well established; however, research elsewhere suggests that frequent childhood moves may be associated with poorer health outcomes and behaviours. The aim of this paper was to compare people in the West of Scotland who were residentially stable in childhood with those who had moved in terms of a range of health measures.

Methods

A total of 850 respondents, followed-up for a period of 20 years, were included in this analysis. Childhood residential mobility was derived from the number of addresses lived at between birth and age 18. Multilevel regression was used to investigate the relationship between childhood residential mobility and health in late adolescence (age 18) and adulthood (age 36), accounting for socio-demographic characteristics and frequency of school moves. The authors examined physical health measures, overall health, psychological distress and health behaviours.

Results

Twenty per cent of respondents remained stable during childhood, 59% moved one to two times and 21% moved at least three times. For most health measures (except physical health), there was an increased risk of poor health that remained elevated for frequent movers after adjustment for socio-demographic characteristics and school moves (but was only significant for illegal drug use).

Conclusions

Risk of poor health was elevated in adolescence and adulthood with increased residential mobility in childhood, after adjusting for socio-demographic characteristics and school moves. This was true for overall health, psychological distress and health behaviours, but physical health measures were not associated with childhood mobility.

Data Curation Standards and Social Science Occupational Information Resources

Occupational information resources - data about the characteristics of different occupational positions - are widely used in the social sciences, across a range of disciplines and international contexts. They are available in many formats, most often constituting small electronic files that are made freely downloadable from academic web pages. However there are several challenges associated with how occupational information resources are distributed to, and exploited by, social researchers. In this paper we describe features of occupational information resources, and indicate the role digital curation can play in exploiting them. We report upon the strategies used in the GEODE research project (Grid Enabled Occupational Data Environment1). This project attempts to develop long-term standards for the distribution of occupational information resources, by providing a standardized framework-based electronic depository for occupational information resources, and by providing a data indexing service, based on e-Science middleware, which collates occupational information resources and makes them readily accessible to non-specialist social scientists.

Predictors of academic attainments of young people with Down's syndrome

BACKGROUND

Earlier studies of young people with Down's syndrome have investigated a relatively limited range of variables which may influence their academic attainment. The relative strength of such influences and how they may vary during the school career, has also been under-researched.

AIMS

The aim of the paper is to identify the contemporary and antecedent predictors of the level of academic attainment achieved by a representative sample of young people with Down's syndrome. Sample The paper reports data from three studies of 71 young people with Down's syndrome and their families. Mean IQ at the time of the first study (t1) was 40.4. Mean chronological age was 9 years at t1, 14 at t2, and 21 at t3, when all the young people had left school.

METHODS

The outcome measure was the 58-item Academic Attainments Index (AAI), comprising three sub-scales covering reading, writing and numeracy. Predictors of the outcome were derived from questionnaires and interviews from tutors, mothers and fathers. A path analysis approach was used to investigate the pattern of predictors of the outcome over the three studies.

RESULTS

Factors predicting greater progress in this measure between t2 and t3 were lower chronological age and attendance at mainstream school. Progress from t1 to t2 was also associated with attendance at mainstream school, as well as with higher t1 mental age, mother's practical coping style and higher child attentiveness. Background factors predicting higher t1 AAI scores were higher mental age, attendance at mainstream school and father's internal locus of control. The path analysis model predicted 48% of the variance in t3 outcome scores. Severity of intellectual impairment was by far the most significant predictor.

CONCLUSION

Limitations to the study include evidence of attrition bias towards more able children, and the need to obtain the t3 outcome measure from tutors for some young people and parents for others. Parents may have over-estimated abilities. Results are broadly in agreement with other studies, and confirm the pattern reported earlier with this group. Mainstream school attendance had a modest beneficial effect on AAI scores throughout the school career of the children, independently of level of intellectual disability. Identification of predictors of attainment levels and of improvement over time may help parents, teachers and other professionals involved with families of children and young people with Down's syndrome optimise the attainment of such skills.

Moving and union dissolution

This paper examines the effect of migration and residential mobility on union dissolution among married and cohabiting couples. Moving is a stressful life event, and a large, multidisciplinary literature has shown that family migration often benefits one partner (usually the man) more than the other Even so, no study to date has examined the possible impact of within-nation geographical mobility on union dissolution. We base our longitudinal analysis on retrospective event-history data from Austria. Our results show that couples who move frequently have a significantly higher risk of union dissolution, and we suggest a variety of mechanisms that may explain this.

The '3Rs' model and the concept of alternatives in animal research: a questionnaire survey

It has been 45 years since Russell and Burch first proposed the concept of the '3Rs', yet it remains unclear how those individuals involved in animal research view and implement these concepts. The authors used a questionnaire survey to determine how well-known experts judged issues related to the 3Rs.