Possibilities for ranking business schools and considerations concerning the stability of such rankings

In this article, we discuss possibilities for ranking business schools and analyse the stability of research rankings using different ranking methods. One focus is set on a comparison of publication-based rankings with citation-based rankings. Our considerations and discussions are based on a (small) case study for which we have examined all (six) business schools at public universities in Austria. The innovative aspect of our article is the chosen mix of methods and the explicit comparison of the results of a publication analysis with those of a citation analysis. In addition, we have developed a new indicator to check the stability of the obtained ranking results with regard to the individual business schools. The results show that the ranks of the individual business schools are quite stable. Nevertheless, we found some differences between publication-based and citation-based rankings. In both cases, however, the choice of the data source as well as switching from full to adjusted counting only have little impact on the ranking results. The main contribution of our approach to research in the field of university rankings is that it shows that focusing on a single (overall) indicator should be avoided, as this can easily lead to bias. Instead, different (partial) indicators should be calculated side by side to provide a more complete picture.

Relationship between bibliometric indicators and university ranking positions

A growing interest for demonstrating prestige and status of higher education institutions has spurred the establishment of several international ranking systems. A major percentage of these rankings include parameters related to scientific productivity. Here, we examined the differences between diverse rankings as well as correlation with bibliometric parameters and disciplines for the top universities. We investigated the top 300 universities from four international rankings, the Times Higher Education World University Ranking (THE), the QS World University Rankings (QS) the ShanghaiRanking-Academic Ranking of World Universities (ARWU) and the U.S.News Best Global Universities Ranking (USNews). The assessed parameters include ranking positions, size related and bibliometrics-related indicators of each selected ranking. The weight of scientometric parameters ranges between 20% (QS) and 75% (USNews). The most important parameters defining ranking positions include citations, international reputation, and the number of researchers, but the correlation strength varies among ranking systems. The absolute number of publications and citations are particularly important in ARWU and USNews rankings, and scientific category normalized (field weighted) citation impact is central in THE and USNews rankings. Our results confirm that universities having outstanding results in rankings using size-independent indicators (QS and THE) compared to others have significantly lower number of students. High impact research can improve position in ARWU and USNews ranking lists. Regarding to different disciplines, the main results show that outstanding universities in THE ranking have higher publication activity in social sciences and universities which perform better in USNews and QS ranking have more publications in science, technology, and medicine fields and lower score in social sciences. In brief, here we present a comprehensive analysis of the correlation between scientometric parameters and university ranking positions, as well as the performance of outstanding universities and their correlation with different disciplines, to help decision makers select parameters for strengthening and to attract the interest of prospective students and their parents via a better understanding of the functions of different ranks.

On the possibilities of presenting the research performance of an institute over a long period of time: the case of the Institute of Information Science at the University of Graz in Austria

In this paper, we demonstrate how the research performance of a university institute (department) over a long period of time can be presented and evaluated. Using the example of an information science institute at a German-speaking university, namely the (former) Institute of Information Science at the University of Graz in Austria, we present the research performance of this institute over the entire duration of its existence (33 years) in different ways. In order to be able to contextualize its performance, we compare it with that of some related institutions from all over the world. Due to the high effort involved in collecting data and the lack of data availability, the comparison must be limited to a period of a few years and—with regard to the institutions from non-German-speaking countries—to the Web of Science as data source. In this international comparison, the institute in the focus of the study shows relatively poor results. As can be seen, the choice of the data source has a major influence on the evaluation results. Especially for institutes from non-English-speaking countries with publications in their respective national languages, an exclusive use of international databases, such as Web of Science or Scopus, cannot fully consider the whole research performance. The use of personal publication lists or local research databases seems almost indispensable in these cases. A major novelty of this article is the handling of a very long evaluation period and the discussion of different ways of subdividing it. With regard to the presentation of the results, in the case of a long observation period, not only should annual and overall results be presented, but also multi-year comparisons be performed. In this way, year-by-year fluctuations can be smoothed out, and longer-term developments can be well represented.

Assessing research excellence: Evaluating the Research Excellence Framework

Performance-based research funding systems have been extensively used around the globe to allocate funds across higher education institutes (HEIs), which led to an increased amount of literature examining their use. The UK’s Research Excellence Framework (REF) uses a peer-review process to evaluate the research environment, research outputs and non-academic impact of research produced by HEIs to produce a more accountable distribution of public funds. However, carrying out such a research evaluation is costly. Given the cost and that it is suggested that the evaluation of each component is subject to bias and has received other criticisms, this article uses correlation and principal component analysis to evaluate REF’s usefulness as a composite evaluation index. As the three elements of the evaluation—environment, impact and output—are highly and positively correlated, the effect of the removal of an element from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings of HEIs. As a result, future evaluations may consider the removal of some elements of the REF or reconsider a new way of evaluating different elements to capture organizational achievement rather than individual achievements.

A longitudinal analysis of university rankings

Pressured by globalization and demand for public organizations to be accountable, efficient, and transparent, university rankings have become an important tool for assessing the quality of higher education institutions. It is therefore important to assess exactly what these rankings measure. Here, the three major global university rankings—the Academic Ranking of World Universities, the Times Higher Education ranking and the Quacquarelli Symonds World University Rankings—are studied. After a description of the ranking methodologies, it is shown that university rankings are stable over time but that there is variation between the three rankings. Furthermore, using principal component analysis and exploratory factor analysis, we demonstrate that the variables used to construct the rankings primarily measure two underlying factors: a university’s reputation and its research performance. By correlating these factors and plotting regional aggregates of universities on the two factors, differences between the rankings are made visible. Last, we elaborate how the results from these analysis can be viewed in light of often-voiced critiques of the ranking process. This indicates that the variables used by the rankings might not capture the concepts they claim to measure. The study provides evidence of the ambiguous nature of university rankings quantification of university performance.

Influential variables in the Journal Impact Factor of Dentistry journals

Objective

The aim of this contribution is to determine what variables influence the position, by quartiles of the impact factor, as a quality indicator of a journal in the field of Dentistry.

Methods

To this end, 24 journals included in Journal Citation Reports, 6 pertaining to each quartile were selected by a stratified sampling and then an ordinal regression model was estimated stepwise considering the journal impact factor quartile as response variable.

Results

The estimation procedure concluded that the average number of papers published yearly by a journal and the percentage of systematic reviews are the most significant variables to be considered, along with the factor representing the journal's degree of adherence to recommendations by the International Committee of Medical Journal Editors.

Conclusions/Clinical significance

Systematic reviews have significant effect on the Journal Impact Factor position of a journal as well as adherence to ICMJE recommendations, while papers publishing clinical trials bear no influence on this factor. Greater yearly average of published papers in a journal means a higher impact factor.

Bibliometrics-based decision trees (BBDTs) based on bibliometrics-based heuristics (BBHs): Visualized guidelines for the use of bibliometrics in research evaluation

Fast-and-frugal heuristics are simple strategies that base decisions on only a few predictor variables. In so doing, heuristics may not only reduce complexity but also boost the accuracy of decisions, their speed, and transparency. In this paper, bibliometrics-based decision trees (BBDTs) are introduced for research evaluation purposes. BBDTs visualize bibliometrics-based heuristics (BBHs), which are judgment strategies solely using publication and citation data. The BBDT exemplar presented in this paper can be used as guidance to find an answer on the question in which situations simple indicators such as mean citation rates are reasonable and in which situations more elaborated indicators (i.e., [sub-]field-normalized indicators) should be applied.