Scientific productivity: An exploratory study of metrics and incentives

Responsible research practices could be more strongly endorsed by Australian university codes of research conduct

BACKGROUND

This study aimed to investigate how strongly Australian university codes of research conduct endorse responsible research practices.

METHODS

Codes of research conduct from 25 Australian universities active in health and medical research were obtained from public websites, and audited against 19 questions to assess how strongly they (1) defined research integrity, research quality, and research misconduct, (2) required research to be approved by an appropriate ethics committee, (3) endorsed 9 responsible research practices, and (4) discouraged 5 questionable research practices.

RESULTS

Overall, a median of 10 (IQR 9 to 12) of 19 practices covered in the questions were mentioned, weakly endorsed, or strongly endorsed. Five to 8 of 9 responsible research practices were mentioned, weakly, or strongly endorsed, and 3 questionable research practices were discouraged. Results are stratified by Group of Eight (n = 8) and other (n = 17) universities. Specifically, (1) 6 (75%) Group of Eight and 11 (65%) other codes of research conduct defined research integrity, 4 (50%) and 8 (47%) defined research quality, and 7 (88%) and 16 (94%) defined research misconduct. (2) All codes required ethics approval for human and animal research. (3) All codes required conflicts of interest to be declared, but there was variability in how strongly other research practices were endorsed. The most commonly endorsed practices were ensuring researcher training in research integrity [8 (100%) and 16 (94%)] and making study data publicly available [6 (75%) and 12 (71%)]. The least commonly endorsed practices were making analysis code publicly available [0 (0%) and 0 (0%)] and registering analysis protocols [0 (0%) and 1 (6%)]. (4) Most codes discouraged fabricating data [5 (63%) and 15 (88%)], selectively deleting or modifying data [5 (63%) and 15 (88%)], and selective reporting of results [3 (38%) and 15 (88%)]. No codes discouraged p-hacking or hypothesising after results are known.

CONCLUSIONS

Responsible research practices could be more strongly endorsed by Australian university codes of research conduct. Our findings may not be generalisable to smaller universities, or those not active in health and medical research.

NSF Fellows' perceptions about incentives, research misconduct, and scientific integrity in STEM academia

There is increased concern about perverse incentives, quantitative performance metrics, and hyper-competition for funding and faculty positions in US academia. Recipients of the prestigious National Science Foundation Graduate Research Fellowships (n = 244) from Civil and Environmental Engineering (45.5%) and Computer Science and Engineering (54.5%) were anonymously surveyed to create a baseline snapshot of their perceptions, behaviors and experiences. NSF Fellows ranked scientific advancement as the top metric for evaluating academics followed by publishing in high-impact journals, social impact of research, and publication/citation counts. The self-reported rate of academic cheating was 16.7% and of research misconduct was 3.7%. Thirty-one percent of fellows reported direct knowledge of graduate peers cheating, and 11.9% had knowledge of research misconduct by colleagues. Only 30.7% said they would report suspected misconduct. A majority of fellows (55.3%) felt that mandatory ethics trainings left them unprepared for dealing with ethical issues. Fellows stated academic freedom, flexible schedules and opportunity to mentor students were the most positive aspects of academia, whereas pressures for funding, publication, and tenure were cited as the most negative aspects. These data may be useful in considering how to better prepare STEM graduate trainees for academic careers.

Data Integration for the Study of Outstanding Productivity in Biomedical Research

Our goal is to analyze improvement of scientific performance in a multidimensional outcome space, with a focus on US-based biomedical research. With the growing diversity of research databases, limiting assessment of scientific productivity to bibliometric measures such as number of publications, impact factor of journals and number of citations, is increasingly challenged. Using a wider range of outcomes, from publications through practice improvements to entrepreneurial outcomes, overcomes many current limitations in the study of research growth. However, combining such heterogeneous datasets raise three challenges: 1. gathering in one common place a variety of data shared as csv, xml or xls files, 2. merging and linking this data, that sometimes overlap, 3. assessing the impact of research production and inclusive practices in a multidimensional space, that are often missing from the datasets. We would like to present our solution for the first of those challenges, and discuss our leads for the second and third challenges.

Refinement: Measuring informativeness of ratings in the absence of a gold standard

We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings - entropic refinement - as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which captures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyse refinement in real-world grant-review data, finding evidence that overall merit scores are more refined than criterion scores.

Science Forum: How failure to falsify in high-volume science contributes to the replication crisis

The number of scientific papers published every year continues to increase, but scientific knowledge is not progressing at the same rate. Here we argue that a greater emphasis on falsification – the direct testing of strong hypotheses – would lead to faster progress by allowing well-specified hypotheses to be eliminated. We describe an example from neuroscience where there has been little work to directly test two prominent but incompatible hypotheses related to traumatic brain injury. Based on this example, we discuss how building strong hypotheses and then setting out to falsify them can bring greater precision to the clinical neurosciences, and argue that this approach could be beneficial to all areas of science.

Beauty and Uncertainty as Transformative Factors: A Free Energy Principle Account of Aesthetic Diagnosis and Intervention in Gestalt Psychotherapy

Drawing from field theory, Gestalt therapy conceives psychological suffering and psychotherapy as two intentional field phenomena, where unprocessed and chaotic experiences seek the opportunity to emerge and be assimilated through the contact between the patient and the therapist (i.e., the intentionality of contacting). This therapeutic approach is based on the therapist’s aesthetic experience of his/her embodied presence in the flow of the healing process because (1) the perception of beauty can provide the therapist with feedback on the assimilation of unprocessed experiences; (2) the therapist’s attentional focus on intrinsic aesthetic diagnostic criteria can facilitate the modification of rigid psychopathological fields by supporting the openness to novel experiences. The aim of the present manuscript is to review recent evidence from psychophysiology, neuroaesthetic research, and neurocomputational models of cognition, such as the free energy principle (FEP), which support the notion of the therapeutic potential of aesthetic sensibility in Gestalt psychotherapy. Drawing from neuroimaging data, psychophysiology and recent neurocognitive accounts of aesthetic perception, we propose a novel interpretation of the sense of beauty as a self-generated reward motivating us to assimilate an ever-greater spectrum of sensory and affective states in our predictive representation of ourselves and the world and supporting the intentionality of contact. Expecting beauty, in the psychotherapeutic encounter, can help therapists tolerate uncertainty avoiding impulsive behaviours and to stay tuned to the process of change.

Scholarly Output in Peruvian National Dentistry according to Gender Disparity: A 10-Year Bibliometric Study

Objective. To evaluate Peruvian scientific publications in dentistry according to sex disparity (2011-2020). Methods. This was a retrospective bibliometric study. The unit of analysis was made up of Peruvian dentistry publications indexed in the Scopus database during the last 10 years. Records with metadata (410) corresponding to the period 2011-2020 were downloaded and standardized and refined by analyzing the metadata. The search strategy was developed based on the individual profiles of each Peruvian institution that has a dental school or college. It was evaluated according to the AF-ID of each institution in the Scopus database. In addition, the information provided by the Scopus SciVal tool was used. Finally, publications, impact, and collaboration indicators were used, such as total number per document, per author, average of citations, $h$ -index, collaboration rate, number of institutions, the Source Normalized Impact per Paper indicator, the CiteScore, and the Scopus Field-Weighted Citation Impact. Results. The greatest increase was evident in 2018, with 2019 and 2020 being the maximum peak of scientific publication growth. However, sustained growth has not been evidenced in relation to the female sex. The analysis of coauthorship by the authors revealed four large clusters, of which the first three were represented by male researchers, such as Arriola-Guillen L., Mayta-Tovalino F., and Mendoza-Azpur G., and one by a female, Guerrero María E. Evaluating the national scientific publication in dentistry according to the CiteScore, it was found that most of the publications (145) from Peru were published in Q4 journals, although 90 manuscripts were published in Q1 journals. Conclusions. The Peruvian national dental publication in the last 10 years was mainly supported by male dentists, which invites us to reflect on the need to equalize opportunities so that female researchers can also reduce these gaps.

Research policies and scientific production: A study of 94 Peruvian universities

Studies of research policies and scientific production are essential for strengthening educational systems and achieving objectives such as quality improvement. The aim of this study was to evaluate the influence of research policies on the scientific production of public and private Peruvian universities. An observational, descriptive, secondary analysis study of the research policies of 92 universities and two graduate schools licensed by the National Superintendence of Higher Education of Peru (SUNEDU) was conducted for the period from 2016–2020. Scientific publications from educational institutions were collected from Scopus and Web of Science for the study period, and researchers certified by the National Council of Science and Technology of Peru (CONCYTEC) were divided by group and level. Multiple regression analysis was performed using two models. The analysis indicated that research policies did not influence scientific production in Scopus or Web of Science in either 2019 or 2020 (Model I) but that type of management (p < 0.01), number of National Scientific, Technological, and Technological Innovation Registry (RENACYT) researchers (p < 0.001) and 2016 scientific production (p < 0.001) did influence production when these variables were incorporated into the model (Model II). However, time of licensing and management type had no effects. The number of research policies implemented by Peruvian universities and licensed graduate schools was not large. Therefore, it is recommended that project funding policies, research training, and research collaboration be strengthened and that the management capacity of research centers and institutes be increased.

Bibliometric Study of the National Scientific Production of All Peruvian Schools of Dentistry in Scopus

Objective

To perform a bibliometric analysis of the national scholarly output of all dental schools in Peru in Scopus through a retrospective study after the promulgation of the Peruvian University Law 30220 in 2014.

Methods

This was a descriptive, comparative, retrospective, and cross-sectional study. A search of the Scopus database was performed to identify scholarly output in dentistry between 2014 and 2019. A total of 287 scientific articles with affiliation with the dental faculties of Peruvian public and private universities were evaluated. The data was extracted from Scopus using a complex formula developed from the words of the thesaurus MeSh (Medline) and Emtree (base) with words related to dentistry combined with the AF-IDs of the Peruvian universities.

Results

The Top 10 in the scholarly output of all the Peruvian public and private dental schools were as follows: firstly, the Universidad Peruana Cayetano Heredia (UPCH), with 79 scholarly outputs publications and 5.2 citations per article, followed by the Universidad Nacional Mayor de San Marcos (UNMSM) with 53 scholarly outputs publications and 2.6 citations per article, and in third place, the Universidad Cientifica del Sur (UCSUR) with 49 scholarly outputs publications. In general, it was found that, of the 30 Faculties of Dentistry in Peru, only 10 in the public and private universities have a scholarly output greater than 5 to belong to the top 10. In addition, it was found that, according to the share of publications per journal quartile by the CiteScore Percentile of all the public and private Peruvian Faculties of Dentistry, in 2019, it presented the highest number of scientific publications in all quartiles with 20,33,14 and 43 articles in the quartiles Q1, Q2, Q3, and Q4, respectively. The highest number of scientific publications was produced in 2019 with 20, 33, 14, and 43 articles in quartiles Q1, Q2, Q3, and Q4, respectively.

Conclusion

The UPCH, UNMSM, and UCSUR dental schools were the most productive. Both public and private universities presented an evident increase in their scientific publications in Scopus after the promulgation of the University Law 30220 in 2014.

Proposed methodology for measuring the effectiveness of policies designed to further research

The effects of economic incentives on research have been widely debated in the literature. Some authors deem them to have no or even an adverse impact, particularly, if they are perceived as irrelevant to or an attempt to control researcher activity, whilst others believe they enhance research productivity by inducing new habits such as international collaboration or publication in high-impact journals. In 2007, the Carlos III University of Madrid introduced bonuses associated with research merits to reward research and educational excellence. The policy aims to enhance the quantity and quality of the institution’s scholarly publications. This study analyses whether and to what extent the new policy has had a potential effect on scientific output, impact, and visibility. Scientific activity indicators between 1991 and 2018 were analysed and a state-space model was used to establish possible scenarios (pre- and post-bonus periods) and 3 year predictions. Further to the findings, despite weak growth in researcher staff size, the number of papers rose during the period slightly more than in the pre-bonus simulation. The number of first-quartile papers also rose substantially, attesting to higher impact and visibility of the university’s research. Greater internationalization was also observed. The incentives were found to be highly suggestive of a change in university researchers’ publication habits.

NIH peer review: Criterion scores completely account for racial disparities in overall impact scores

Previous research has found that funding disparities are driven by applications' final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014-2016, we examine assigned reviewers' preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration-the process of combining criterion scores into overall impact scores-disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities-yet do not explain all of the variability-in preliminary overall impact scores.

Turning the tables: A university league-table based on quality not quantity

Background: Universities closely watch international league tables because these tables influence governments, donors and students. Achieving a high ranking in a table, or an annual rise in ranking, allows universities to promote their achievements using an externally validated measure. However, league tables predominantly reward measures of research output, such as publications and citations, and may therefore be promoting poor research practices by encouraging the “publish or perish” mentality.

Methods: We examined whether a league table could be created based on good research practice. We rewarded researchers who cited a reporting guideline, which help researchers report their research completely, accurately and transparently, and were created to reduce the waste of poorly described research. We used the EQUATOR guidelines, which means our tables are mostly relevant to health and medical research. We used Scopus to identify the citations.

Results: Our cross-sectional tables for the years 2016 and 2017 included 14,408 papers with 47,876 author affiliations. We ranked universities and included a bootstrap measure of uncertainty. We clustered universities in five similar groups in an effort to avoid over-interpreting small differences in ranks.

Conclusions: We believe there is merit in considering more socially responsible criteria for ranking universities, and this could encourage better research practice internationally if such tables become as valued as the current quantity-focused tables.

Understanding of researcher behavior is required to improve data reliability

BACKGROUND

A lack of data reproducibility ("reproducibility crisis") has been extensively debated across many academic disciplines.

RESULTS

Although a reproducibility crisis is widely perceived, conclusive data on the scale of the problem and the underlying reasons are largely lacking. The debate is primarily focused on methodological issues. However, examples such as the use of misidentified cell lines illustrate that the availability of reliable methods does not guarantee good practice. Moreover, research is often characterized by a lack of established methods. Despite the crucial importance of researcher conduct, research and conclusive data on the determinants of researcher behavior are widely missing.

CONCLUSION

Meta-research that establishes an understanding of the factors that determine researcher behavior is urgently needed. This knowledge can then be used to implement and iteratively improve measures that incentivize researchers to apply the highest standards, resulting in high-quality data.

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

Background: Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return.

Methods: We examined the peer reviewers' scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures of reviewer disagreement: the standard deviation and range. The outcome variable was the relative citation ratio, which is the number of citations from all publications associated with the application, standardised by field and publication year.

Results: There was a clear increase in relative citations for applications with a better mean. There was no association between relative citations and either of the two measures of disagreement.

Conclusions: We found no evidence that reviewer disagreement was able to identify applications with a higher than average return. However, this is the first study to empirically examine this association, and it would be useful to examine whether reviewer disagreement is associated with research impact in other funding schemes and in larger sample sizes.