Quantifying long-term scientific impact

Assessing the evolution of scientific publications in orthopedics journals from mainland China, Hong Kong, and Taiwan: a 12-year survey of the literature

Background

In China, the field of orthopedics has experienced significant growth over the past 12 years. However, the recent status of research on orthopedics among individuals in mainland China, Hong Kong, and Taiwan is unknown. In this study, we investigated characteristics and trends of orthopedics publications from these three regions.

Methods

Between 2003 and 2014, all articles published in 63 orthopedics journals originating from mainland China, Hong Kong, and Taiwan were identified via Science Citation Index Expanded (SCIE) database. A survey was conducted to systematically analyze the published orthopedics articles from the three regions according to the numbers of articles, study design, impact factors (IFs), citations, most prolific authors, and institutions. Additionally, we evaluated global trends in orthopedics publications, and ranked top 10 countries in terms of the total number of published articles over 12 years and the number of published articles per year.

Results

A total number of 123,317 articles were published in the 63 orthopedics journals between 2003 and 2014. The worldwide number of annually published orthopedics articles tended to increase during the study period. The total number of orthopedics publications from the three regions, especially in mainland China, increased markedly from 2003 to 2014. The annual number of orthopedics articles from mainland China increased from 6 in 2003 to 813 in 2014, Hong Kong increased from 32 in 2003 to 71 in 2014, and Taiwan increased from 68 in 2003 to 168 in 2014. For accumulated IFs and total citations of articles, mainland China ranked the first place, followed by Taiwan and Hong Kong. However, publications from Taiwan had the highest average citations per article, and publications from Hong Kong had the highest average IFs. Among the top 10 most prolific authors and institutions, 4 authors and 4 institutions were from Taiwan, 3 authors and 4 institutions were from mainland China, and 3 authors and 2 institutions were from Hong Kong.

Conclusions

The quantity of articles published in international orthopedics journals from mainland China presented a remarkable upward trend during the past 12 years. Given the relative size of the populations, it should be emphasized that mainland China still has a long way to go to achieve the academic performance of Hong Kong and Taiwan.

Predicting the future trend of popularity by network diffusion

Conventional approaches to predict the future popularity of products are mainly based on extrapolation of their current popularity, which overlooks the hidden microscopic information under the macroscopic trend. Here, we study diffusion processes on consumer-product and citation networks to exploit the hidden microscopic information and connect consumers to their potential purchase, publications to their potential citers to obtain a prediction for future item popularity. By using the data obtained from the largest online retailers including Netflix and Amazon as well as the American Physical Society citation networks, we found that our method outperforms the accurate short-term extrapolation and identifies the potentially popular items long before they become prominent.

Randomized Controlled Trial of Social Media: Effect of Increased Intensity of the Intervention

Background

A prior randomized controlled trial of social media exposure at Circulation determined that social media did not increase 30‐day page views. Whether insufficient social media intensity contributed to these results is uncertain.

Methods and Results

Original article manuscripts were randomized to social media exposure compared with no social media exposure (control) at Circulation beginning in January 2015. Social media exposure consisted of Facebook and Twitter posts on the journal's accounts. To increase social media intensity, a larger base of followers was built using advertising and organic growth, and posts were presented in triplicate and boosted on Facebook and retweeted on Twitter. The primary outcome was 30‐day page views. Stopping rules were established at the point that 50% of the manuscripts were randomized and had 30‐day follow‐up to compare groups on 30‐day page views. The trial was stopped for futility on September 26, 2015. Overall, 74 manuscripts were randomized to receive social media exposure, and 78 manuscripts were randomized to the control arm. The intervention and control arms were similar based on article type (P=0.85), geographic location of the corresponding author (P=0.33), and whether the manuscript had an editorial (P=0.80). Median number of 30‐day page views was 499.5 in the social media arm and 450.5 in the control arm; there was no evidence of a treatment effect (P=0.38). There were no statistically significant interactions of treatment by manuscript type (P=0.86), by corresponding author (P=0.35), by trimester of publication date (P=0.34), or by editorial status (P=0.79).

Conclusions

A more intensive social media strategy did not result in increased 30‐day page views of original research.

Ethnobotany and herbal medicine in modern complementary and alternative medicine: An overview of publications in the field of I&C medicine 2001-2013

ETHNOPHARMACOLOGICAL RELEVANCE

Ethnobotanical knowledge and traditional medicinal practices from different parts of the world are of global importance and documentation of ethnobotanical, and ethnopharmacological data is a key prerequisite for further research in the area of herbal medicine and its implementation in clinical practice.

AIMS

An attempt was made to evaluate the scientific output of research related to ethnobotany and herbal medicine in journals indexed in the subject area "Integrative and Complementary Medicine" in the period 2001-2013, in order to ascertain research trends in both subdisciplines.

MATERIALS AND METHODS

All articles related to ethnobotany and herbal medicine, extracted from journals included in the field of I&C Medicine and published in the period 2001-2013, have been analyzed for general bibliometric data, and specific data: ethnobotanical data (geographic, floristic, pharmacological, sociological and other relevant data) and phytotherapeutic data (type of applied herbal medicine, plant species studied, pharmacological activity of studied plant species and disease and disorder type studied on a particular model).

RESULTS

In the studied period, the number of articles dealing with ethnobotany and herbal medicine increased 6.3-fold. Articles related to ethnobotanical studies documented medicinal flora from 81 countries, either giving an overview of overall medicinal flora, or presenting the ethnomedicinal aspect of the use of plants for the treatment of ailments typical to the studied area. Additionally, the authors provided significant information on the methods of use and herbal preparations. In herbal medicine studies, plants, traditional plant remedies, herbal medicinal products and active herbal compounds were tested for many of pharmacological activities (146), with the curative activity emerging as most frequently tested. Out of 39 model systems, most of the studies were carried out under controlled in vitro conditions (4589 articles), followed by rat in vivo (2320), human in vivo (1285), mouse in vivo (955), and on agents of pathogenic diseases (887); more than 800 medical disorders were treated.

CONCLUSIONS

The study revealed the regions most studied for new records of floristic and ethnomedicinal diversity, the most frequently studied plant species, and the most promising therapeutic indications for the integration of herbal remedies in the curative process, as ascertained from the selected bibliographic databases.

Understanding the group dynamics and success of teams

Complex problems often require coordinated group effort and can consume significant resources, yet our understanding of how teams form and succeed has been limited by a lack of large-scale, quantitative data. We analyse activity traces and success levels for approximately 150 000 self-organized, online team projects. While larger teams tend to be more successful, workload is highly focused across the team, with only a few members performing most work. We find that highly successful teams are significantly more focused than average teams of the same size, that their members have worked on more diverse sets of projects, and the members of highly successful teams are more likely to be core members or 'leads' of other teams. The relations between team success and size, focus and especially team experience cannot be explained by confounding factors such as team age, external contributions from non-team members, nor by group mechanisms such as social loafing. Taken together, these features point to organizational principles that may maximize the success of collaborative endeavours.

Genealogical Trees of Scientific Papers

Many results have been obtained when studying scientific papers citations databases in a network perspective. Articles can be ranked according to their current in-degree and their future popularity or citation counts can even be predicted. The dynamical properties of such networks and the observation of the time evolution of their nodes started more recently. This work adopts an evolutionary perspective and proposes an original algorithm for the construction of genealogical trees of scientific papers on the basis of their citation count evolution in time. The fitness of a paper now amounts to its in-degree growing trend and a "dying" paper will suddenly see this trend declining in time. It will give birth and be taken over by some of its most prevalent citing "offspring". Practically, this might be used to trace the successive published milestones of a research field.

A Theoretical Model for the Associative Nature of Conference Participation

Participation in conferences is an important part of every scientific career. Conferences provide an opportunity for a fast dissemination of latest results, discussion and exchange of ideas, and broadening of scientists’ collaboration network. The decision to participate in a conference depends on several factors like the location, cost, popularity of keynote speakers, and the scientist’s association with the community. Here we discuss and formulate the problem of discovering how a scientist’s previous participation affects her/his future participations in the same conference series. We develop a stochastic model to examine scientists’ participation patterns in conferences and compare our model with data from six conferences across various scientific fields and communities. Our model shows that the probability for a scientist to participate in a given conference series strongly depends on the balance between the number of participations and non-participations during his/her early connections with the community. An active participation in a conference series strengthens the scientist’s association with that particular conference community and thus increases the probability of future participations.

The Last 50 Years: Mismeasurement and Mismanagement Are Impeding Scientific Research

In the last 50 years, there have been many changes to the substance, conduct, and style of research. Many of these changes have proved disastrous to the life of scientists and to science itself. As a consequence, the near-romantic spirit of adventure and exploration that inspired young scientists of my own and earlier generations has become tarnished. Now, many of us feel beleaguered by bureaucrats and by politicians: they affect our lives profoundly, apparently without an understanding of the way discoveries are made or of the nature of science itself. The core purposes of universities, teaching and research, are being eroded by excessive administration. The number and locations of our publications are counted up like beans and the outcomes are used to rank us, one against another; a process of evaluation that has recast the purposes of publication. Applying for grants takes far too much time from a young scientist's life.

Universality of Citation Distributions for Academic Institutions and Journals

Citations measure the importance of a publication, and may serve as a proxy for its popularity and quality of its contents. Here we study the distributions of citations to publications from individual academic institutions for a single year. The average number of citations have large variations between different institutions across the world, but the probability distributions of citations for individual institutions can be rescaled to a common form by scaling the citations by the average number of citations for that institution. We find this feature seems to be universal for a broad selection of institutions irrespective of the average number of citations per article. A similar analysis for citations to publications in a particular journal in a single year reveals similar results. We find high absolute inequality for both these sets, Gini coefficients being around 0.66 and 0.58 for institutions and journals respectively. We also find that the top 25% of the articles hold about 75% of the total citations for institutions and the top 29% of the articles hold about 71% of the total citations for journals.

Text analysis tools for identification of emerging topics and research gaps in conservation science

Keeping track of conceptual and methodological developments is a critical skill for research scientists, but this task is increasingly difficult due to the high rate of academic publication. As a crisis discipline, conservation science is particularly in need of tools that facilitate rapid yet insightful synthesis. We show how a common text-mining method (latent Dirichlet allocation, or topic modeling) and statistical tests familiar to ecologists (cluster analysis, regression, and network analysis) can be used to investigate trends and identify potential research gaps in the scientific literature. We tested these methods on the literature on ecological surrogates and indicators. Analysis of topic popularity within this corpus showed a strong emphasis on monitoring and management of fragmented ecosystems, while analysis of research gaps suggested a greater role for genetic surrogates and indicators. Our results show that automated text analysis methods need to be used with care, but can provide information that is complementary to that given by systematic reviews and meta-analyses, increasing scientists' capacity for research synthesis.

Reviewing Peer Review at the NIH

Recent reports suggest that peer reviews of National Institutes of Health grant applications are at best imprecise predictors of research projects' scientific impact. But these findings may not mean that peer review is failing.

Ranking nodes in growing networks: When PageRank fails

PageRank is arguably the most popular ranking algorithm which is being applied in real systems ranging from information to biological and infrastructure networks. Despite its outstanding popularity and broad use in different areas of science, the relation between the algorithm's efficacy and properties of the network on which it acts has not yet been fully understood. We study here PageRank's performance on a network model supported by real data, and show that realistic temporal effects make PageRank fail in individuating the most valuable nodes for a broad range of model parameters. Results on real data are in qualitative agreement with our model-based findings. This failure of PageRank reveals that the static approach to information filtering is inappropriate for a broad class of growing systems, and suggest that time-dependent algorithms that are based on the temporal linking patterns of these systems are needed to better rank the nodes.

Universality in survivor distributions: Characterizing the winners of competitive dynamics

We investigate the survivor distributions of a spatially extended model of competitive dynamics in different geometries. The model consists of a deterministic dynamical system of individual agents at specified nodes, which might or might not survive the predatory dynamics: all stochasticity is brought in by the initial state. Every such initial state leads to a unique and extended pattern of survivors and nonsurvivors, which is known as an attractor of the dynamics. We show that the number of such attractors grows exponentially with system size, so that their exact characterization is limited to only very small systems. Given this, we construct an analytical approach based on inhomogeneous mean-field theory to calculate survival probabilities for arbitrary networks. This powerful (albeit approximate) approach shows how universality arises in survivor distributions via a key concept-the dynamical fugacity. Remarkably, in the large-mass limit, the survivor probability of a node becomes independent of network geometry and assumes a simple form which depends only on its mass and degree.

Anatomy of funded research in science

Seeking research funding is an essential part of academic life. Funded projects are primarily collaborative in nature through internal and external partnerships, but what role does funding play in the formulation of these partnerships? Here, by examining over 43,000 scientific projects funded over the past three decades by one of the major government research agencies in the world, we characterize how the funding landscape has changed and its impacts on the underlying collaboration networks across different scales. We observed rising inequality in the distribution of funding and that its effect was most noticeable at the institutional level--the leading universities diversified their collaborations and increasingly became the knowledge brokers in the collaboration network. Furthermore, it emerged that these leading universities formed a rich club (i.e., a cohesive core through their close ties) and this reliance among them seemed to be a determining factor for their research success, with the elites in the core overattracting resources but also rewarding in terms of both research breadth and depth. Our results reveal how collaboration networks organize in response to external driving forces, which can have major ramifications on future research strategy and government policy.

The advantage of short paper titles

Vast numbers of scientific articles are published each year, some of which attract considerable attention, and some of which go almost unnoticed. Here, we investigate whether any of this variance can be explained by a simple metric of one aspect of the paper's presentation: the length of its title. Our analysis provides evidence that journals which publish papers with shorter titles receive more citations per paper. These results are consistent with the intriguing hypothesis that papers with shorter titles may be easier to understand, and hence attract more citations.

Development of obstetrical and gynecological journals, 2007 to 2013: a trend analysis

PURPOSE

To analyze the trends and developments among journals in the specialty of obstetrics and gynecology.

METHODS

Using the Journal Citation Reports from 2007 to 2013, we analyzed the impact factor (IF), Eigenfactor® Score (ES), and Article Influence® Score (AIS) of 43 journals in the field of obstetrics and gynecology published in this time period.

RESULTS

From 78 journals of the Journal Citation Report 2013, 43 were selected for this study. The mean IF grew from 1.68 ± 0.97 in 2007 to 2.12 ± 1.05 in 2013, the ES from 0.0113 ± 0.0169 to 0.0114 ± 0.0140, and the AIS from 0.513 ± 0.302 to 0.663 ± 0.359. Differences in the IF, ES, and AIS between journals from the United States versus Europe could be observed. In most cases, the IF, ES, and AIS increased between 2007 and 2013. Strong correlations could be found between IF, AIS, and ES.

CONCLUSIONS

The overall mean IF for obstetrical and gynecological journals increased over the analyzed time period. The IF remains the standard measure to compare scientific journals. It correlates well with two major alternative measures of scientific impact, the ES and especially the AIS. Other measures are evolving and might show superior usage in the future.

The Association of the H-Index and Academic Rank Among Full-Time Academic Hand Surgeons Affiliated With Fellowship Programs

PURPOSE

To evaluate the association between the Hirsch index (a measure of publications and citations) and academic rank among hand surgeons.

METHODS

This was a cross-sectional study of full-time academic hand surgeons within Accreditation Council for Graduate Medical Education-approved hand surgery fellowship programs in the United States and Canada. The study variables were classified as bibliometric (h-index, I-10 index, total number of publications, total number of citations, maximum number of citations for a single work) and demographics (gender, training factors). The outcome was academic rank (instructor, assistant professor, associate professor, professor, endowed professor). Descriptive, bivariate, and multiple regression statistics were computed.

RESULTS

The sample was composed of 366 full-time academic hand surgeons; 86% were male and 98% had formal hand surgery fellowship training. The mean time since completion of surgical training was 17 ± 11 years. The distribution of primary faculty appointments was orthopedic surgery (70%) and plastic surgery (30%). Two hundred fifty surgeons (68%) were members of the American Society for Surgery of the Hand. The mean h-index was 10.2 ± 9.9 and was strongly correlated with academic rank. Gender was not associated with academic rank. Distribution of academic ranks was as follows: instructor (4%), assistant professor (28%), associate professor (40%), professor (22%), and endowed professor (5%). The h-index, years since completion of training, and American Society for Surgery of the Hand membership were associated with academic rank. The h-index had a high sensitivity and specificity for predicting academic rank.

CONCLUSIONS

The h-index is a reliable tool for quantitatively assessing research productivity and should be considered for use in academic hand surgery.

CLINICAL RELEVANCE

When evaluating candidates for academic promotion in hand surgery, the h-index is a potentially valuable tool for assessing research productivity and impact.

Cross-correlations of American baby names

The quantitative description of cultural evolution is a challenging task. The most difficult part of the problem is probably to find the appropriate measurable quantities that can make more quantitative such evasive concepts as, for example, dynamics of cultural movements, behavioral patterns, and traditions of the people. A strategy to tackle this issue is to observe particular features of human activities, i.e., cultural traits, such as names given to newborns. We study the names of babies born in the United States from 1910 to 2012. Our analysis shows that groups of different correlated states naturally emerge in different epochs, and we are able to follow and decrypt their evolution. Although these groups of states are stable across many decades, a sudden reorganization occurs in the last part of the 20th century. We unambiguously demonstrate that cultural evolution of society can be observed and quantified by looking at cultural traits. We think that this kind of quantitative analysis can be possibly extended to other cultural traits: Although databases covering more than one century (such as the one we used) are rare, the cultural evolution on shorter timescales can be studied due to the fact that many human activities are usually recorded in the present digital era.

Defining and identifying Sleeping Beauties in science

A Sleeping Beauty (SB) in science refers to a paper whose importance is not recognized for several years after publication. Its citation history exhibits a long hibernation period followed by a sudden spike of popularity. Previous studies suggest a relative scarcity of SBs. The reliability of this conclusion is, however, heavily dependent on identification methods based on arbitrary threshold parameters for sleeping time and number of citations, applied to small or monodisciplinary bibliographic datasets. Here we present a systematic, large-scale, and multidisciplinary analysis of the SB phenomenon in science. We introduce a parameter-free measure that quantifies the extent to which a specific paper can be considered an SB. We apply our method to 22 million scientific papers published in all disciplines of natural and social sciences over a time span longer than a century. Our results reveal that the SB phenomenon is not exceptional. There is a continuous spectrum of delayed recognition where both the hibernation period and the awakening intensity are taken into account. Although many cases of SBs can be identified by looking at monodisciplinary bibliographic data, the SB phenomenon becomes much more apparent with the analysis of multidisciplinary datasets, where we can observe many examples of papers achieving delayed yet exceptional importance in disciplines different from those where they were originally published. Our analysis emphasizes a complex feature of citation dynamics that so far has received little attention, and also provides empirical evidence against the use of short-term citation metrics in the quantification of scientific impact.

Quantifying the consistency of scientific databases

Science is a social process with far-reaching impact on our modern society. In recent years, for the first time we are able to scientifically study the science itself. This is enabled by massive amounts of data on scientific publications that is increasingly becoming available. The data is contained in several databases such as Web of Science or PubMed, maintained by various public and private entities. Unfortunately, these databases are not always consistent, which considerably hinders this study. Relying on the powerful framework of complex networks, we conduct a systematic analysis of the consistency among six major scientific databases. We found that identifying a single "best" database is far from easy. Nevertheless, our results indicate appreciable differences in mutual consistency of different databases, which we interpret as recipes for future bibliometric studies.

Temporal effects in trend prediction: identifying the most popular nodes in the future

Prediction is an important problem in different science domains. In this paper, we focus on trend prediction in complex networks, i.e. to identify the most popular nodes in the future. Due to the preferential attachment mechanism in real systems, nodes' recent degree and cumulative degree have been successfully applied to design trend prediction methods. Here we took into account more detailed information about the network evolution and proposed a temporal-based predictor (TBP). The TBP predicts the future trend by the node strength in the weighted network with the link weight equal to its exponential aging. Three data sets with time information are used to test the performance of the new method. We find that TBP have high general accuracy in predicting the future most popular nodes. More importantly, it can identify many potential objects with low popularity in the past but high popularity in the future. The effect of the decay speed in the exponential aging on the results is discussed in detail.

Modeling the citation network by network cosmology

Citation between papers can be treated as a causal relationship. In addition, some citation networks have a number of similarities to the causal networks in network cosmology, e.g., the similar in-and out-degree distributions. Hence, it is possible to model the citation network using network cosmology. The casual network models built on homogenous spacetimes have some restrictions when describing some phenomena in citation networks, e.g., the hot papers receive more citations than other simultaneously published papers. We propose an inhomogenous causal network model to model the citation network, the connection mechanism of which well expresses some features of citation. The node growth trend and degree distributions of the generated networks also fit those of some citation networks well.

On a heuristic point of view concerning the citation distribution: introducing the Wakeby distribution

The paper proposes a heuristic approach to modeling the cumulative distribution of citations of papers in scientific journals by means of the Wakeby distribution. The Markov process of citation leading to the Wakeby distribution is analyzed using the terminal time formalism. The Wakeby distribution is derived in the paper from the simple and general inhomogeneous Choquet–Deny convolution equation for a non-probability measure. We give statistical evidence that the Wakeby distribution is a reasonable approximation of the empirical citation distributions.

AMS Subject Classification: 91D30; 91D99

Anatomy of scientific evolution

The quest for historically impactful science and technology provides invaluable insight into the innovation dynamics of human society, yet many studies are limited to qualitative and small-scale approaches. Here, we investigate scientific evolution through systematic analysis of a massive corpus of digitized English texts between 1800 and 2008. Our analysis reveals great predictability for long-prevailing scientific concepts based on the levels of their prior usage. Interestingly, once a threshold of early adoption rates is passed even slightly, scientific concepts can exhibit sudden leaps in their eventual lifetimes. We developed a mechanistic model to account for such results, indicating that slowly-but-commonly adopted science and technology surprisingly tend to have higher innate strength than fast-and-commonly adopted ones. The model prediction for disciplines other than science was also well verified. Our approach sheds light on unbiased and quantitative analysis of scientific evolution in society, and may provide a useful basis for policy-making.

Where should I send it? Optimizing the submission decision process

How do scientists decide where to submit manuscripts? Many factors influence this decision, including prestige, acceptance probability, turnaround time, target audience, fit, and impact factor. Here, we present a framework for evaluating where to submit a manuscript based on the theory of Markov decision processes. We derive two models, one in which an author is trying to optimally maximize citations and another in which that goal is balanced by either minimizing the number of resubmissions or the total time in review. We parameterize the models with data on acceptance probability, submission-to-decision times, and impact factors for 61 ecology journals. We find that submission sequences beginning with Ecology Letters, Ecological Monographs, or PLOS ONE could be optimal depending on the importance given to time to acceptance or number of resubmissions. This analysis provides some guidance on where to submit a manuscript given the individual-specific values assigned to these disparate objectives.

Network-driven reputation in online scientific communities

The ever-increasing quantity and complexity of scientific production have made it difficult for researchers to keep track of advances in their own fields. This, together with growing popularity of online scientific communities, calls for the development of effective information filtering tools. We propose here an algorithm which simultaneously computes reputation of users and fitness of papers in a bipartite network representing an online scientific community. Evaluation on artificially-generated data and real data from the Econophysics Forum is used to determine the method's best-performing variants. We show that when the input data is extended to a multilayer network including users, papers and authors and the algorithm is correspondingly modified, the resulting performance improves on multiple levels. In particular, top papers have higher citation count and top authors have higher h-index than top papers and top authors chosen by other algorithms. We finally show that our algorithm is robust against persistent authors (spammers) which makes the method readily applicable to the existing online scientific communities.

High-ranked social science journal articles can be identified from early citation information

Do citations accumulate too slowly in the social sciences to be used to assess the quality of recent articles? I investigate whether this is the case using citation data for all articles in economics and political science published in 2006 and indexed in the Web of Science. I find that citations in the first two years after publication explain more than half of the variation in cumulative citations received over a longer period. Journal impact factors improve the correlation between the predicted and actual future ranks of journal articles when using citation data from 2006 alone but the effect declines sharply thereafter. Finally, more than half of the papers in the top 20% in 2012 were already in the top 20% in the year of publication (2006).

Predictability of extreme events in social media

It is part of our daily social-media experience that seemingly ordinary items (videos, news, publications, etc.) unexpectedly gain an enormous amount of attention. Here we investigate how unexpected these extreme events are. We propose a method that, given some information on the items, quantifies the predictability of events, i.e., the potential of identifying in advance the most successful items. Applying this method to different data, ranging from views in YouTube videos to posts in Usenet discussion groups, we invariantly find that the predictability increases for the most extreme events. This indicates that, despite the inherently stochastic collective dynamics of users, efficient prediction is possible for the most successful items.

Assessing the long-term impact of public investments in comparative effectiveness research: conceptual framework and lessons learned

The American Recovery and Reinvestment Act (ARRA) of 2009 directed US$1.1 billion to the US Department of Health and Human Services for support of comparative effectiveness research (CER). As part of this investment, US Department of Health and Human Services commissioned a midstream evaluation of the ARRA CER portfolio. One goal of the evaluation was to identify issues to consider for a future evaluation of the long-term impact of this portfolio and other CER investments. In planning the ARRA CER evaluation, we developed and revised a conceptual framework and related policy research questions that may be useful to future efforts to assess the impact of CER or patient-centered outcomes research investments. In addition, we explored methodological challenges related to designing an evaluation to assess investments in CER that may be informative to any future plans to evaluate the long-term impact of ARRA CER as well subsequent investments made from the Patient-Centered Outcomes Research Trust Fund.

Ranking scientific publications: the effect of nonlinearity

Ranking the significance of scientific publications is a long-standing challenge. The network-based analysis is a natural and common approach for evaluating the scientific credit of papers. Although the number of citations has been widely used as a metric to rank papers, recently some iterative processes such as the well-known PageRank algorithm have been applied to the citation networks to address this problem. In this paper, we introduce nonlinearity to the PageRank algorithm when aggregating resources from different nodes to further enhance the effect of important papers. The validation of our method is performed on the data of American Physical Society (APS) journals. The results indicate that the nonlinearity improves the performance of the PageRank algorithm in terms of ranking effectiveness, as well as robustness against malicious manipulations. Although the nonlinearity analysis is based on the PageRank algorithm, it can be easily extended to other iterative ranking algorithms and similar improvements are expected.

Measuring long-term impact based on network centrality: unraveling cinematic citations

Traditional measures of success for film, such as box-office revenue and critical acclaim, lack the ability to quantify long-lasting impact and depend on factors that are largely external to the craft itself. With the growing number of films that are being created and large-scale data becoming available through crowd-sourced online platforms, an endogenous measure of success that is not reliant on manual appraisal is of increasing importance. In this article we propose such a ranking method based on a combination of centrality indices. We apply the method to a network that contains several types of citations between more than 40,000 international feature films. From this network we derive a list of milestone films, which can be considered to constitute the foundations of cinema. In a comparison to various existing lists of ‘greatest’ films, such as personal favourite lists, voting lists, lists of individual experts, and lists deduced from expert polls, the selection of milestone films is more diverse in terms of genres, actors, and main creators. Our results shed light on the potential of a systematic quantitative investigation based on cinematic influences in identifying the most inspiring creations in world cinema. In a broader perspective, we introduce a novel research question to large-scale citation analysis, one of the most intriguing topics that have been at the forefront of scientific enquiries for the past fifty years and have led to the development of various network analytic methods. In doing so, we transfer widely studied approaches from citation analysis to the the newly emerging field of quantification efforts in the arts. The specific contribution of this paper consists in modelling the multidimensional cinematic references as a growing multiplex network and in developing a methodology for the identification of central films in this network.

Network-based statistical comparison of citation topology of bibliographic databases

Modern bibliographic databases provide the basis for scientific research and its evaluation. While their content and structure differ substantially, there exist only informal notions on their reliability. Here we compare the topological consistency of citation networks extracted from six popular bibliographic databases including Web of Science, CiteSeer and arXiv.org. The networks are assessed through a rich set of local and global graph statistics. We first reveal statistically significant inconsistencies between some of the databases with respect to individual statistics. For example, the introduced field bow-tie decomposition of DBLP Computer Science Bibliography substantially differs from the rest due to the coverage of the database, while the citation information within arXiv.org is the most exhaustive. Finally, we compare the databases over multiple graph statistics using the critical difference diagram. The citation topology of DBLP Computer Science Bibliography is the least consistent with the rest, while, not surprisingly, Web of Science is significantly more reliable from the perspective of consistency. This work can serve either as a reference for scholars in bibliometrics and scientometrics or a scientific evaluation guideline for governments and research agencies.

The Matthew effect in empirical data

The Matthew effect describes the phenomenon that in societies, the rich tend to get richer and the potent even more powerful. It is closely related to the concept of preferential attachment in network science, where the more connected nodes are destined to acquire many more links in the future than the auxiliary nodes. Cumulative advantage and success-breads-success also both describe the fact that advantage tends to beget further advantage. The concept is behind the many power laws and scaling behaviour in empirical data, and it is at the heart of self-organization across social and natural sciences. Here, we review the methodology for measuring preferential attachment in empirical data, as well as the observations of the Matthew effect in patterns of scientific collaboration, socio-technical and biological networks, the propagation of citations, the emergence of scientific progress and impact, career longevity, the evolution of common English words and phrases, as well as in education and brain development. We also discuss whether the Matthew effect is due to chance or optimization, for example related to homophily in social systems or efficacy in technological systems, and we outline possible directions for future research.

Predicting the evolution of spreading on complex networks

Due to the wide applications, spreading processes on complex networks have been intensively studied. However, one of the most fundamental problems has not yet been well addressed: predicting the evolution of spreading based on a given snapshot of the propagation on networks. With this problem solved, one can accelerate or slow down the spreading in advance if the predicted propagation result is narrower or wider than expected. In this paper, we propose an iterative algorithm to estimate the infection probability of the spreading process and then apply it to a mean-field approach to predict the spreading coverage. The validation of the method is performed in both artificial and real networks. The results show that our method is accurate in both infection probability estimation and spreading coverage prediction.