Quantifying long-term scientific impact

An Evaluation of h-Index as a Measure of Research Productivity Among Canadian Academic Plastic Surgeons

Background

Evaluation of research productivity among plastic surgeons can be complex. The Hirsch index (h-index) was recently introduced to evaluate both the quality and quantity of one's research activity. It has been proposed to be valuable in assessing promotions and grant funding within academic medicine, including plastic surgery. Our objective is to evaluate research productivity among Canadian academic plastic surgeons using the h-index.

Methods

A list of Canadian academic plastic surgeons was obtained from websites of academic training programs. The h-index was retrieved using the Scopus database. Relevant demographic and academic factors were collected and their effects on the h-index were analyzed using the t test and Wilcoxon Mann-Whitney U test. Nominal and categorical variables were analyzed using χ² test and 1-way analysis of variance. Univariate and multivariate models were built a priori. All P values were 2 sided, and P < .05 was considered to be significant.

Results

Our study on Canadian plastic surgeons involved 175 surgeons with an average h-index of 7.6. Over 80% of the surgeons were male. Both univariable and multivariable analysis showed that graduate degree (P < .0001), academic rank (P = .03), and years in practice (P < .0001) were positively correlated with h-index. Limitations of the study include that the Scopus database and the websites of training programs were not always up-to-date.

Conclusion

The h-index is a novel tool for evaluating research productivity in academic medicine, and this study shows that the h-index can also serve as a useful metric for measuring research productivity in the Canadian plastic surgery community. Plastic surgeons would be wise to familiarize themselves with the h-index concept and should consider using it as an adjunct to existing metrics such as total publication number.

Using text analysis to quantify the similarity and evolution of scientific disciplines

We use an information-theoretic measure of linguistic similarity to investigate the organization and evolution of scientific fields. An analysis of almost 20 M papers from the past three decades reveals that the linguistic similarity is related but different from experts and citation-based classifications, leading to an improved view on the organization of science. A temporal analysis of the similarity of fields shows that some fields (e.g. computer science) are becoming increasingly central, but that on average the similarity between pairs of disciplines has not changed in the last decades. This suggests that tendencies of convergence (e.g. multi-disciplinarity) and divergence (e.g. specialization) of disciplines are in balance.

Forecasting success via early adoptions analysis: A data-driven study

Innovations are continuously launched over markets, such as new products over the retail market or new artists over the music scene. Some innovations become a success; others don't. Forecasting which innovations will succeed at the beginning of their lifecycle is hard. In this paper, we provide a data-driven, large-scale account of the existence of a special niche among early adopters, individuals that consistently tend to adopt successful innovations before they reach success: we will call them Hit-Savvy. Hit-Savvy can be discovered in very different markets and retain over time their ability to anticipate the success of innovations. As our second contribution, we devise a predictive analytical process, exploiting Hit-Savvy as signals, which achieves high accuracy in the early-stage prediction of successful innovations, far beyond the reach of state-of-the-art time series forecasting models. Indeed, our findings and predictive model can be fruitfully used to support marketing strategies and product placement.

Article-level assessment of influence and translation in biomedical research

Given the vast scale of the modern scientific enterprise, it can be difficult for scientists to make judgments about the work of others through careful analysis of the entirety of the relevant literature. This has led to a reliance on metrics that are mathematically flawed and insufficiently diverse to account for the variety of ways in which investigators contribute to scientific progress. An urgent, critical first step in solving this problem is replacing the Journal Impact Factor with an article-level alternative. The Relative Citation Ratio (RCR), a metric that was designed to serve in that capacity, measures the influence of each publication on its respective area of research. RCR can serve as one component of a multifaceted metric that provides an effective data-driven supplement to expert opinion. Developing validated methods that quantify scientific progress can help to optimize the management of research investments and accelerate the acquisition of knowledge that improves human health.

Power-law citation distributions are not scale-free

We analyze time evolution of statistical distributions of citations to scientific papers published in the same year. While these distributions seem to follow the power-law dependence we find that they are nonstationary and the exponent of the power-law fit decreases with time and does not come to saturation. We attribute the nonstationarity of citation distributions to different longevity of the low-cited and highly cited papers. By measuring citation trajectories of papers we found that citation careers of the low-cited papers come to saturation after 10-15 years while those of the highly cited papers continue to increase indefinitely: The papers that exceed some citation threshold become runaways. Thus, we show that although citation distribution can look as a power-law dependence, it is not scale free and there is a hidden dynamic scale associated with the onset of runaways. We compare our measurements to our recently developed model of citation dynamics based on copying-redirection-triadic closure and find explanations to our empirical observations.

Exploring an Unknown Territory: "Sleeping Beauties" in the Nursing Research Literature

BACKGROUND

Sleeping Beauties (SBs) are publications that are scarcely cited in the years immediately following publication but then suddenly become highly cited later. Such publications have unique citation patterns and can reveal important developments in the field in which they appear.

OBJECTIVES

No holistic analysis of nursing SBs has been done yet. The aim of this study was to identify and analyze the SB phenomenon in the nursing research literature.

METHOD

The corpus for the nursing SB identification was harvested from the Web of Science Core Collection (Thomas Reuters) for the period 1934-2015. Citation histories of 212,239 publications were screened. From those, 3,209 publications with more than 100 citations were selected for analysis. We used our own software and applied the van Raan (2004) and Baumgartner (2010) criteria for SBs-a 5-year sleeping period with at most 10 citations during that time, an average of at least five citations per year after the first 10 years, with at least 100 citations in total. The knowledge context for SBs was determined using citing papers. All citing papers were analyzed with the help of VOSviewer software.

RESULTS

Nine publications were identified as SBs (prevalence of 0.004%). The length of sleep duration ranged from 5 to 10 years (M = 6.8, SD = 2.0), depth of sleep ranged from 0.2 to 0.8 citations (M = 0.6, SD = 0.2), and awake intensity ranged from 6.4 to 15.0 citations (M = 11.0, SD = 3.8). The average number of citations to SBs was 229. Most nursing SBs were produced in the United States (n = 8) from top institutions in journals with high-impact factors. Nursing SBs covered topics including resilience, sampling in qualitative research, metasynthesis, postoperative pain in children, dementia rating scales, care of patients with Alzheimer's disease, nursing theory related to fatigue mechanisms in cancer patients, and family participation during resuscitation. Nursing SBs were cited by authors from a large number of institutions and countries; the number of publications citing nursing SBs is growing exponentially and showing increasing and global interest in the research presented in them.

DISCUSSION

This study demonstrated that SBs in nursing are similar to other scientific disciplines. Existence of SBs suggests that nursing knowledge accumulation is supported by research and professional processes similar to those that emerged in other academic disciplines.

How the fittest compete for leadership: A tale of tails

We investigate how leaders emerge as a consequence of the competitive dynamics between coupled papers in a model citation network. Every paper is allocated an initial fitness depending on its intrinsic quality. Its fitness then involves dynamically as a consequence of the competition between itself and all the other papers in the field. It picks up citations as a result of this adaptive dynamics, becoming a leader if it has the highest citation count at a given time. Extensive analytical and numerical investigations of this model suggest the existence of a universal phase diagram, divided into regions of weak and strong coupling. In the former, we find an "extended" and rather structureless distribution of citation counts among many fit papers; leaders are not necessarily those with the maximal fitness at any given time. By contrast, the strong-coupling region is characterized by a strongly hierarchical distribution of citation counts, that are "localized" among only a few extremely fit papers, and exhibit strong history-to-history fluctuations, as a result of the complex dynamics among papers in the tail of the fitness distribution.

Science and Facebook: The same popularity law!

The distribution of scientific citations for publications selected with different rules (author, topic, institution, country, journal, etc…) collapse on a single curve if one plots the citations relative to their mean value. We find that the distribution of “shares” for the Facebook posts rescale in the same manner to the very same curve with scientific citations. This finding suggests that citations are subjected to the same growth mechanism with Facebook popularity measures, being influenced by a statistically similar social environment and selection mechanism. In a simple master-equation approach the exponential growth of the number of publications and a preferential selection mechanism leads to a Tsallis-Pareto distribution offering an excellent description for the observed statistics. Based on our model and on the data derived from PubMed we predict that according to the present trend the average citations per scientific publications exponentially relaxes to about 4.

Significance and popularity in music production

Creative industries constantly strive for fame and popularity. Though highly desirable, popularity is not the only achievement artistic creations might ever acquire. Leaving a longstanding mark in the global production and influencing future works is an even more important achievement, usually acknowledged by experts and scholars. 'Significant' or 'influential' works are not always well known to the public or have sometimes been long forgotten by the vast majority. In this paper, we focus on the duality between what is successful and what is significant in the musical context. To this end, we consider a user-generated set of tags collected through an online music platform, whose evolving co-occurrence network mirrors the growing conceptual space underlying music production. We define a set of general metrics aiming at characterizing music albums throughout history, and their relationships with the overall musical production. We show how these metrics allow to classify albums according to their current popularity or their belonging to expert-made lists of important albums. In this way, we provide the scientific community and the public at large with quantitative tools to tell apart popular albums from culturally or aesthetically relevant artworks. The generality of the methodology presented here lends itself to be used in all those fields where innovation and creativity are in play.

An exploration of collaborative scientific production at MIT through spatial organization and institutional affiliation

Academic research is increasingly cross-disciplinary and collaborative, between and within institutions. In this context, what is the role and relevance of an individual's spatial position on a campus? We examine the collaboration patterns of faculty at the Massachusetts Institute of Technology, through their academic output (papers and patents), and their organizational structures (institutional affiliation and spatial configuration) over a 10-year time span. An initial comparison of output types reveals: 1. diverging trends in the composition of collaborative teams over time (size, faculty versus non-faculty, etc.); and 2. substantively different patterns of cross-building and cross-disciplinary collaboration. We then construct a multi-layered network of authors, and find two significant features of collaboration on campus: 1. a network topology and community structure that reveals spatial versus institutional collaboration bias; and 2. a persistent relationship between proximity and collaboration, well fit with an exponential decay model. This relationship is consistent for both papers and patents, and present also in exclusively cross-disciplinary work. These insights contribute an architectural dimension to the field of scientometrics, and take a first step toward empirical space-planning policy that supports collaboration within institutions.

A Unified Framework for Complex Networks with Degree Trichotomy Based on Markov Chains

This paper establishes a Markov chain model as a unified framework for describing the evolution processes in complex networks. The unique feature of the proposed model is its capability in addressing the formation mechanism that can reflect the "trichotomy" observed in degree distributions, based on which closed-form solutions can be derived. Important special cases of the proposed unified framework are those classical models, including Poisson, Exponential, Power-law distributed networks. Both simulation and experimental results demonstrate a good match of the proposed model with real datasets, showing its superiority over the classical models. Implications of the model to various applications including citation analysis, online social networks, and vehicular networks design, are also discussed in the paper.

Waves of novelties in the expansion into the adjacent possible

The emergence of novelties and their rise and fall in popularity is an ubiquitous phenomenon in human activities. The coexistence of popular evergreens with novel and sometimes ephemeral trends pervades technological, scientific and artistic production. Though this phenomenon is very intuitively captured by our common sense, a comprehensive explanation of how waves of novelties are not hampered by well established old-comers is still lacking. Here we first quantify this phenomenology by empirically looking at different systems that display innovation at very different levels: the creation of hashtags in Twitter, the evolution of online code repositories, the creation of texts and the listening of songs on online platforms. In all these systems surprisingly similar patterns emerge as the non-trivial outcome of two contrasting forces: the tendency of retracing already explored avenues (exploit) and the inclination to explore new possibilities. These findings are naturally explained in the framework of the expansion of the adjacent possible, a recently introduced theoretical framework that postulates the restructuring of the space of possibilities conditional to the occurrence of innovations. The predictions of our theoretical framework are borne out in all the phenomenologies investigated, paving the way to a better understanding and control of innovation processes.

Uncovering and Predicting the Dynamic Process of Collective Attention with Survival Theory

The subject of collective attention is in the center of this era of information explosion. It is thus of great interest to understand the fundamental mechanism underlying attention in large populations within a complex evolving system. Moreover, an ability to predict the dynamic process of collective attention for individual items has important implications in an array of areas. In this report, we propose a generative probabilistic model using a self-excited Hawkes process with survival theory to model and predict the process through which individual items gain their attentions. This model explicitly captures three key ingredients: the intrinsic attractiveness of an item, characterizing its inherent competitiveness against other items; a reinforcement mechanism based on sum of each previous attention triggers; and a power-law temporal relaxation function, corresponding to the aging in the ability to attract new attentions. Experiments on two population-scale datasets demonstrate that this model consistently outperforms the state-of-the-art methods.

The nearly universal link between the age of past knowledge and tomorrow's breakthroughs in science and technology: The hotspot

Scientists and inventors can draw on an ever-expanding literature for the building blocks of tomorrow's ideas, yet little is known about how combinations of past work are related to future discoveries. Our analysis parameterizes the age distribution of a work's references and revealed three links between the age of prior knowledge and hit papers and patents. First, works that cite literature with a low mean age and high age variance are in a citation "hotspot"; these works double their likelihood of being in the top 5% or better of citations. Second, the hotspot is nearly universal in all branches of science and technology and is increasingly predictive of a work's future citation impact. Third, a scientist or inventor is significantly more likely to write a paper in the hotspot when they are coauthoring than whey they are working alone. Our findings are based on all 28,426,345 scientific papers in the Web of Science, 1945-2013, and all 5,382,833 U.S. patents, 1950-2010, and reveal new antecedents of high-impact science and the link between prior literature and tomorrow's breakthrough ideas.

Stochastic dynamics and the predictability of big hits in online videos

The competition for the attention of users is a central element of the Internet. Crucial issues are the origin and predictability of big hits, the few items that capture a big portion of the total attention. We address these issues analyzing 10^{6} time series of videos' views from YouTube. We find that the average gain of views is linearly proportional to the number of views a video already has, in agreement with usual rich-get-richer mechanisms and Gibrat's law, but this fails to explain the prevalence of big hits. The reason is that the fluctuations around the average views are themselves heavy tailed. Based on these empirical observations, we propose a stochastic differential equation with Lévy noise as a model of the dynamics of videos. We show how this model is substantially better in estimating the probability of an ordinary item becoming a big hit, which is considerably underestimated in the traditional proportional-growth models.

Data-driven predictions in the science of science

The desire to predict discoveries-to have some idea, in advance, of what will be discovered, by whom, when, and where-pervades nearly all aspects of modern science, from individual scientists to publishers, from funding agencies to hiring committees. In this Essay, we survey the emerging and interdisciplinary field of the "science of science" and what it teaches us about the predictability of scientific discovery. We then discuss future opportunities for improving predictions derived from the science of science and its potential impact, positive and negative, on the scientific community.

Growing complex network of citations of scientific papers: Modeling and measurements

We consider the network of citations of scientific papers and use a combination of the theoretical and experimental tools to uncover microscopic details of this network growth. Namely, we develop a stochastic model of citation dynamics based on the copying-redirection-triadic closure mechanism. In a complementary and coherent way, the model accounts both for statistics of references of scientific papers and for their citation dynamics. Originating in empirical measurements, the model is cast in such a way that it can be verified quantitatively in every aspect. Such validation is performed by measuring citation dynamics of physics papers. The measurements revealed nonlinear citation dynamics, the nonlinearity being intricately related to network topology. The nonlinearity has far-reaching consequences including nonstationary citation distributions, diverging citation trajectories of similar papers, runaways or "immortal papers" with infinite citation lifetime, etc. Thus nonlinearity in complex network growth is our most important finding. In a more specific context, our results can be a basis for quantitative probabilistic prediction of citation dynamics of individual papers and of the journal impact factor.

Citation context and impact of 'sleeping beauties' in paediatric research

Objectives 'Sleeping beauties', i.e. publications that are not cited for a long while, present interesting findings in science. This study analysed the citation trends of sleeping beauties in paediatric research. Methods The study used bibliometric software to analyse the papers citing sleeping beauties in paediatric research, to understand the context in which paediatric sleeping beauties were finally cited and the impact of these sleeping beauties on paediatric research. Results Two paediatric sleeping beauties, addressing medical homes and the transition from paediatric to adult health care, respectively, awakened in response to organizational needs. Both presented novel concepts of paediatric service organization that became important because of an increased need for optimization of services. Conclusion All sleeping beauties bring new knowledge that becomes important only after several years. Paediatric sleeping beauties exhibited unique characteristics; however, their presence in paediatric research shows that knowledge acquisition in paediatrics resembles that in other disciplines.

Collective Influence of Multiple Spreaders Evaluated by Tracing Real Information Flow in Large-Scale Social Networks

Identifying the most influential spreaders that maximize information flow is a central question in network theory. Recently, a scalable method called "Collective Influence (CI)" has been put forward through collective influence maximization. In contrast to heuristic methods evaluating nodes' significance separately, CI method inspects the collective influence of multiple spreaders. Despite that CI applies to the influence maximization problem in percolation model, it is still important to examine its efficacy in realistic information spreading. Here, we examine real-world information flow in various social and scientific platforms including American Physical Society, Facebook, Twitter and LiveJournal. Since empirical data cannot be directly mapped to ideal multi-source spreading, we leverage the behavioral patterns of users extracted from data to construct "virtual" information spreading processes. Our results demonstrate that the set of spreaders selected by CI can induce larger scale of information propagation. Moreover, local measures as the number of connections or citations are not necessarily the deterministic factors of nodes' importance in realistic information spreading. This result has significance for rankings scientists in scientific networks like the APS, where the commonly used number of citations can be a poor indicator of the collective influence of authors in the community.

Identification and impact of discoverers in online social systems

Understanding the behavior of users in online systems is of essential importance for sociology, system design, e-commerce, and beyond. Most existing models assume that individuals in diverse systems, ranging from social networks to e-commerce platforms, tend to what is already popular. We propose a statistical time-aware framework to identify the users who differ from the usual behavior by being repeatedly and persistently among the first to collect the items that later become hugely popular. Since these users effectively discover future hits, we refer them as discoverers. We use the proposed framework to demonstrate that discoverers are present in a wide range of real systems. Once identified, discoverers can be used to predict the future success of new items. We finally introduce a simple network model which reproduces the discovery patterns observed in the real data. Our results open the door to quantitative study of detailed temporal patterns in social systems.

Joint estimation of preferential attachment and node fitness in growing complex networks

Complex network growth across diverse fields of science is hypothesized to be driven in the main by a combination of preferential attachment and node fitness processes. For measuring the respective influences of these processes, previous approaches make strong and untested assumptions on the functional forms of either the preferential attachment function or fitness function or both. We introduce a Bayesian statistical method called PAFit to estimate preferential attachment and node fitness without imposing such functional constraints that works by maximizing a log-likelihood function with suitably added regularization terms. We use PAFit to investigate the interplay between preferential attachment and node fitness processes in a Facebook wall-post network. While we uncover evidence for both preferential attachment and node fitness, thus validating the hypothesis that these processes together drive complex network evolution, we also find that node fitness plays the bigger role in determining the degree of a node. This is the first validation of its kind on real-world network data. But surprisingly the rate of preferential attachment is found to deviate from the conventional log-linear form when node fitness is taken into account. The proposed method is implemented in the R package PAFit.

Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.

Model-based evaluation of scientific impact indicators

Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the advantage of avoiding the biases present in real databases, and it allows us to assess which aspects of the model dynamics and which traits of individual researchers a particular indicator actually reflects. We find that the simple average citation count of the authored papers performs well in capturing the intrinsic scientific ability of researchers, regardless of the length of their career. On the other hand, when productivity complements ability in the evaluation process, the notorious h and g indices reveal their potential, yet their normalized variants do not always yield a fair comparison between researchers at different career stages. Notably, the use of logarithmic units for citation counts allows us to build simple indicators with performance equal to that of h and g. Our analysis may provide useful hints for a proper use of bibliometric indicators. Additionally, our framework can be extended by including other aspects of the scientific production process and citation dynamics, with the potential to become a standard tool for the assessment of impact metrics.

The Role of Temporal Trends in Growing Networks

The rich get richer principle, manifested by the Preferential attachment (PA) mechanism, is widely considered one of the major factors in the growth of real-world networks. PA stipulates that popular nodes are bound to be more attractive than less popular nodes; for example, highly cited papers are more likely to garner further citations. However, it overlooks the transient nature of popularity, which is often governed by trends. Here, we show that in a wide range of real-world networks the recent popularity of a node, i.e., the extent by which it accumulated links recently, significantly influences its attractiveness and ability to accumulate further links. We proceed to model this observation with a natural extension to PA, named Trending Preferential Attachment (TPA), in which edges become less influential as they age. TPA quantitatively parametrizes a fundamental network property, namely the network’s tendency to trends. Through TPA, we find that real-world networks tend to be moderately to highly trendy. Networks are characterized by different susceptibilities to trends, which determine their structure to a large extent. Trendy networks display complex structural traits, such as modular community structure and degree-assortativity, occurring regularly in real-world networks. In summary, this work addresses an inherent trait of complex networks, which greatly affects their growth and structure, and develops a unified model to address its interaction with preferential attachment.

Reflection of successful anticancer drug development processes in the literature

The development of cancer drugs is time-consuming and expensive. In particular, failures in late-stage clinical trials are a major cost driver for pharmaceutical companies. This puts a high demand on methods that provide insights into the success chances of new potential medicines. In this study, we systematically analyze publication patterns emerging along the drug discovery process of targeted cancer therapies, starting from basic research to drug approval - or failure. We find clear differences in the patterns of approved drugs compared with those that failed in Phase II/III. Feeding these features into a machine learning classifier allows us to predict the approval or failure of a targeted cancer drug significantly better than educated guessing. We believe that these findings could lead to novel measures for supporting decision making in drug development.