Evaluating merit among scientists

REVIEW: RESEARCH INTEREST SCORE IN RESEARCHGATE: THE SILVER BULLET OF SCIENTOMETRICS OR THE EMPEROR’S NEW CLOTHES?

Introduction: Scientists have long searched for the best way to evaluate scientific performance and have come up with numerous indices. Probably, the most famous index is the h-index, a metric that has been used widely in science metrics ever since it appeared. In this study, I evaluated the potential of the new metric Research Interest Score created by ResearchGate (RG). Methods: I analyzed the different metric indices for 88 most cited researchers at the University of Sarajevo. In particular, the number of citations and h-indexes were taken from the Web of Science, Scopus, and Google Scholar databases for these researchers and were correlated with their Research Interest Scores. As a measure of correlation, Pearson correlation coefficients was used. In addition, I conducted a regression analysis to examine how scientometric indices from the Web of Science, Scopus, and Google Scholar predict the Research Interest Score.  Results: The Research Interest Score was most strongly correlated with the number of Google Scholar Citations, followed by the Web of Science citations.  Interestingly, the Research Interest Score was not highly correlated with any of the h-indexes. The regression model was statistically significant and explained 90% of the variance in the Research Interest Scores predicted by the Web of Science, Scopus, and Google Scholar metric indices. The strongest predictor of the Research Interest Score was the number of Google Scholar citations. Conclusions: The Research Interest Score, a new scientometric measure created by ResearchGate, certainly has a potential to be used as a valid measure of scientific impact.

Progressive and degenerative journals: on the growth and appraisal of knowledge in scholarly publishing

Despite continued attention, finding adequate criteria for distinguishing "good" from "bad" scholarly journals remains an elusive goal. In this essay, I propose a solution informed by the work of Imre Lakatos and his methodology of scientific research programmes (MSRP). I begin by reviewing several notable attempts at appraising journal quality - focusing primarily on the impact factor and development of journal blacklists and whitelists. In doing so, I note their limitations and link their overarching goals to those found within the philosophy of science. I argue that Lakatos's MSRP and specifically his classifications of "progressive" and "degenerative" research programmes can be analogized and repurposed for the evaluation of scholarly journals. I argue that this alternative framework resolves some of the limitations discussed above and offers a more considered evaluation of journal quality - one that helps account for the historical evolution of journal-level publication practices and attendant contributions to the growth (or stunting) of scholarly knowledge. By doing so, the seeming problem of journal demarcation is diminished. In the process I utilize two novel tools (the mistake index and scite index) to further illustrate and operationalize aspects of the MSRP.

Citation Metrics in Psychological Science

We present norms for faculty citation counts based on 811 faculty members at 30 PhD-granting psychology departments in the United States across the range of the National Research Council rankings. The metrics were highly skewed, with most scientists having a low to moderate number of citations of their work and a few scientists having extremely high numbers. However, the median per-year citation count was 149, showing widespread scientific contributions across scholars. Some individuals in lower ranked departments are more highly cited than the average scholar in higher ranked departments, with enormous variation in citation counts in both the low- and high-ranking departments. Citation counts overall have risen in recent years, and the citations of early-career scholars are increasing at a faster rate than their senior colleagues did at the same point in their careers. We found that citation counts at the beginning of scientists' careers substantially predict lifetime citation success. Young scholars' citation counts are associated with obtaining positions at higher ranked universities. Finally, we found no significant differences for subfields of psychology. In sum, although a few highly productive scientists have a very large influence, trends reveal that contributions to psychological science are growing over time, widespread, and not limited to a few stars and elite departments.

Measuring research in the big data era: The evolution of performance measurement systems in the Italian teaching hospitals

BACKGROUND

In the healthcare system, Teaching Hospitals(THs) not only provide care, but also train healthcare professionals and carry out research activities. Research is a fundamental pillar of THs' mission and relevant for the healthcare system monitored by Performance Evaluation Systems. Research activities can be measured using citation index services and this paper highlights differences between two services based on bibliometrics, describes opportunities and risks when performance indicators rely on data collected, controlled and validated by external services and discusses the possible impact on health policy at a system and provider level.

METHODS

A bibliometric analysis was done on data between 2014-2016 from ISI Web of Science and Scopus of 18.255 physicians working in 26 Italian THs. Quantity was defined as the number of publications and quality as Impact Factor or Field-Weighted Citation Impact.

RESULTS

Overall, 41.233 and 66.409 documents were extracted from respectively ISI Web of Science and Scopus. While benchmarking results, significant differences in ranked position both in metrics emerged.

DISCUSSION

Utilizing secondary data sources to measure research activities of THs allows benchmarking at an (inter)national level and overcoming self-referment. To utilize indicators for multiple governance purposes at the system and provider level, indicators need to be profoundly understood, require formalizations in data validation, internal analysis and a sharing process among health professionals, management and policymakers.

The Relation of Scientific Creativity and Evaluation of Scientific Impact to Scientific Reasoning and General Intelligence

In many nations, grades and standardized test scores are used to select students for programs of scientific study. We suggest that the skills that these assessments measure are related to success in science, but only peripherally in comparison with two other skills, scientific creativity and recognition of scientific impact. In three studies, we investigated the roles of scientific creativity and recognition of scientific impact on scientific thinking. The three studies described here together involved 219 students at a selective university in the Northeast U.S. Participants received assessments of scientific creativity and recognition of scientific impact as well as a variety of previously used assessments measuring scientific reasoning (generating alternative hypotheses, generating experiments, drawing conclusions) and the fluid aspect of general intelligence (letter sets, number series). They also provided scores from either or both of two college-admissions tests—the SAT and the ACT—as well as demographic information. Our goal was to determine whether the new tests of scientific impact and scientific creativity correlated and factored with the tests of scientific reasoning, fluid intelligence, both, or neither. We found that our new measures tapped into aspects of scientific reasoning as we previously have studied it, although the factorial composition of the test on recognition of scientific impact is less clear than that of the test of scientific creativity. We also found that participants rated high-impact studies as more scientifically rigorous and practically useful than low-impact studies, but also generally as less creative, probably because their titles/abstracts were seemingly less novel for our participants. Replicated findings across studies included the correlation of Letter Sets with Number Series (both measures of fluid intelligence) and the correlation of Scientific Creativity with Scientific Reasoning.