How has healthcare research performance been assessed?: a systematic review

The Impact of the Dermal Matrix in Tissue Reconstruction: A Bibliometric Perspective in Plastic Surgery

In the vast field of medical scientific research, few topics have managed to attract as much attention and mobilise academic resources as the use of dermal matrices in the reconstruction of soft tissue defects. In this study, we used bibliographic metrics such as co-authorship, keyword co-occurrence, and citations per document to analyse the relationship between the use of dermal matrices to reconstruct soft tissue defects caused by burns, tumours, and trauma. In addition, keyword analysis has highlighted the crucial role of technology in recent studies and the innovation brought about by the use of dermal matrices in the reconstruction of soft tissue defects. Keywords used in recent studies have revealed the critical role of technology in the development of the field. We extracted a set of 1329 research papers from the Web of Science Core Collection database between 2010 and 2024 that met our criteria. Through keyword analysis, we identified technology as a significant factor in recent studies. Our results showed that there is very little collaboration between authors on the topic and that most of them are from Asia. A significant number of articles on this topic come from the USA, China, Japan, Germany, the UK, and France. We discovered the top ten most cited sources analysing the use of dermal matrices in the reconstruction of soft tissue defects. Finally, we think that this study will be beneficial for our further research.

What predicts citation counts and translational impact in headache research? A machine learning analysis

BACKGROUND

We aimed to develop the first machine learning models to predict citation counts and the translational impact, defined as inclusion in guidelines or policy documents, of headache research, and assess which factors are most predictive.

METHODS

Bibliometric data and the titles, abstracts, and keywords from 8600 publications in three headache-oriented journals from their inception to 31 December 2017 were used. A series of machine learning models were implemented to predict three classes of 5-year citation count intervals (0-5, 6-14 and, >14 citations); and the translational impact of a publication. Models were evaluated out-of-sample with area under the receiver operating characteristics curve (AUC).

RESULTS

The top performing gradient boosting model predicted correct citation count class with an out-of-sample AUC of 0.81. Bibliometric data such as page count, number of references, first and last author citation counts and h-index were among the most important predictors. Prediction of translational impact worked optimally when including both bibliometric data and information from the title, abstract and keywords, reaching an out-of-sample AUC of 0.71 for the top performing random forest model.

CONCLUSION

Citation counts are best predicted by bibliometric data, while models incorporating both bibliometric data and publication content identifies the translational impact of headache research.

Australian radiation therapist's research capacity and culture: investigating the extent of assistance required to build research capacity

Introduction

Building research capacity within the radiation therapist workforce/profession is essential to guarantee research is embedded into core practices. Assessment of current capacity levels within organisation, department and individual domains needs to occur to establish a baseline and ensure research capacity building (RCB) strategies will be targeted successfully. This exploratory study aimed to identify the areas within each domain where radiation therapists would benefit from extra research assistance, that being research support and process changes, particularly in relation to the workplace and health sector.

Methods

Practising radiation therapists (RTs) within Australia were recruited through professional organisations and invited to complete the online Research Capacity Culture Tool (RCCT). The survey was conducted using Qualtrics with data exported to SPSS.V27 for analysis. Descriptive statistics and the Industry Specific Item Interpretation for Policy and Strategy were used to analyse and report the results.

Results

Survey participation rate was 121/2640 (4.6%). Within the Organisation and Department domains, the most research assistance was required in infrastructure (n = 92–37) and support (n = 66–45) categories, respectively. Participants from private sector (Organisation: 42.7%, Department: 53.7%) and metropolitan centres (Organisation: 32.6%, Department: 47.5%) required a higher rate of assistance when compared to their counterparts in both categories. The individual domain showed similar levels of assistance required across health sectors and work locations. Workplaces showed similar levels of complexity of research activity; private sector (62.5%) recorded the highest level of no research activity.

Conclusion

This study has provided insight into how the research capacity and culture of organisations and departments in which individuals' work will influence their abilities and opportunities to perform research.

Impact of international research fellows in neurosurgery: results from a single academic center

OBJECTIVE

International research fellows have been historically involved in academic neurosurgery in the United States (US). To date, the contribution of international research fellows has been underreported. Herein, the authors aimed to quantify the academic output of international research fellows in the Department of Neurosurgery at The Johns Hopkins University School of Medicine.

METHODS

Research fellows with Doctor of Medicine (MD), Doctor of Philosophy (PhD), or MD/PhD degrees from a non-US institution who worked in the Hopkins Department of Neurosurgery for at least 6 months over the past decade (2010-2020) were included in this study. Publications produced during fellowship, number of citations, and journal impact factors (IFs) were analyzed using ANOVA. A survey was sent to collect information on personal background, demographics, and academic activities.

RESULTS

Sixty-four international research fellows were included, with 42 (65.6%) having MD degrees, 17 (26.6%) having PhD degrees, and 5 (7.8%) having MD/PhD degrees. During an average 27.9 months of fellowship, 460 publications were produced in 136 unique journals, with 8628 citations and a cumulative journal IF of 1665.73. There was no significant difference in total number of publications, first-author publications, and total citations per person among the different degree holders. Persons holding MD/PhDs had a higher number of citations per publication per person (p = 0.027), whereas those with MDs had higher total IFs per person (p = 0.048). Among the 43 (67.2%) survey responders, 34 (79.1%) had nonimmigrant visas at the start of the fellowship, 16 (37.2%) were self-paid or funded by their country of origin, and 35 (81.4%) had mentored at least one US medical student, nonmedical graduate student, or undergraduate student.

CONCLUSIONS

International research fellows at the authors' institution have contributed significantly to academic neurosurgery. Although they have faced major challenges like maintaining nonimmigrant visas, negotiating cultural/language differences, and managing self-sustainability, their scientific productivity has been substantial. Additionally, the majority of fellows have provided reciprocal mentorship to US students.

Determinants of societal and academic recognition: Evidence from randomised controlled trials

Given the increasing importance of recognition in academia and the vital role of randomised controlled trials (RCTs) in medical research and clinical decisions, this study verifies how RCTs’ academic and societal impacts are affected by visibility factors, subjects and methodological validity. This study concentrated on a sample of 446 RCTs indexed in Scopus and evaluated by Cochrane reviewers in terms of their methodological validity. The altmetrics, bibliometric and bibliographical information were extracted from Altmetric.com and Scopus, and the contributing countries’ development ranks were obtained from the United Nations Development report. The linear regression analyses revealed that citations and altmetrics depend on some subjects. They are also affected by publication year and journals’ previous reputation. Citations are also affected by keyword counts and reference counts. Keyword counts and contributing countries’ developmental rank also predict the tweet counts. While none of the methodological validity dimensions were found to predict citations, ‘Incomplete Outcome Data’ and ‘Random Sequence Generation’ significantly, though slightly, affect Mendeley Readership and tweets, respectively. By confirming the dependence of RCTs’ recognition on some methodological validity features and attention-inducing characteristics, the study provides further evidence on the interaction of quality and visibility dynamisms in the recognition network and the complementary role of societal mentions for academic citation.

Development and implementation of standardized study performance metrics for a VA healthcare system clinical research consortium

The cost of conducting clinical trials is continuously increasing and is driven in large part by the time and resources required to activate trials and reach accrual targets. The impact of low enrollment in a clinical trial can negatively affect the validity of study results and delay its generalizability to the broader population. Quality is a multidimensional concept which could relate to the design, conduct, and analysis of a trial, its clinical relevance, protection/safety of study participants, or quality of reporting. Furthermore, the quality of controlled trials is of obvious relevance to systematic reviews and if the "raw material" or "data" is flawed then the conclusions of systematic reviews cannot be trusted. To date, the literature surrounding the establishment of standardized study enrollment and quality metrics to assess site performance in clinical trial consortiums is scarce. The lack of these metrics presents challenges to study site teams, sponsors, and other clinical research enterprise key stakeholders for adequately monitoring and evaluating study site performance as it relates to fulfilling trial enrollment and quality goals. The Department of Veterans Affairs (VA) Cooperative Studies Program (CSP) Network of Dedicated Enrollment Sites (NODES) undertook an effort to determine the feasibility of establishing and implementing standardized study enrollment and quality metrics for a clinical research consortium (NODES) as a tool to evaluate its performance. In this manuscript, we describe the development and implementation of standardized study enrollment and quality metrics to assess site performance across studies in our clinical research consortium.

Beyond the metrics of health research performance in African countries

While it is important to be able to evaluate and measure a country's performance in health research (HR), HR systems are complex and multifaceted in nature. As such, attempts at measurement can suffer several limitations which risk leading to inadequate indices or representations. In this study, we critically review common indicators of HR capacity and performance and explore their strengths and limitations. The paper is informed by review of data sources and documents, combined with interviews and peer-to-peer learning activities conducted with officials working in health and education ministries in a set of nine African countries. We find that many metrics that can assess HR performance have gaps in the conceptualisation or fail to address local contextual realities, which makes it a challenge to interpret them in relation to other theoretical constructs. Our study identified several concepts that are excluded from current definitions of indicators and systems of metrics for HR performance. These omissions may be particularly important for interpreting HR performance within the context and processes of HR in African countries, and thus challenging the relevance, utility, appropriateness and acceptability of universal measures of HR in the region. We discuss the challenges that scholars may find in conceptualising such a complex phenomenon-including the different and competing viewpoints of stakeholders, in setting objectives of HR measurement work, and in navigating the realities of empirical measurement where missing or partial data may necessitate that proxies or alternative indicators may be chosen. These findings are important to ensure that the global health community does not rely on over-simplistic evaluations of HR when analysing and planning for improvements in low-income and middle-income countries.

Comparative bibliometric analysis of publications by past Royal Australasian College of Surgeons research scholarship recipients

BACKGROUND

The Royal Australasian College of Surgeons awards scholarships to surgeons, surgical trainees and recipients focused on developing their clinical knowledge and improving outcomes for patients. A bibliometric analysis of research scholarship recipients publications and h-index scores was conducted to understand the benefits of receiving these scholarships.

METHODS

A bibliometric analysis of Royal Australasian College of Surgeons scholarship recipients in 2015 was performed using Open Researcher and Contributor ID (ORCID), Scopus, Google Scholar, ResearchGate, LinkedIn and PubMed to identify the number of publications, h-index scores, field-weighted citation impact and the relative citation ratio.

RESULTS

Nineteen research scholarship recipients authored 842 publications, with 491 (58%) published after completion of their scholarship. Seven recipients published 50% or more of their articles in the 5 years since completion. Five recipients have each published more than 45 articles since 2015. H-index scores varied between Scopus and Google Scholar (overall range: 4-34). Scopus identified the most publications, followed by ResearchGate. Determining publication numbers for recipients was problematic due to self-reporting in some databases (i.e. Google Scholar, ResearchGate), variations in author names (i.e. maiden to married name), duplication of publications and the inclusion of supplementary material (i.e. extra tables) in self-reporting databases. Field-weighted citation impact and relative citation ratio values exceeded 1 on 12 occasions demonstrating recipients are more cited than the global average.

CONCLUSION

Continuous tracking of publication rates and h-index scores of scholarship recipients demonstrates recipients' continuing interest in advancing and disseminating medical knowledge to improve patient outcomes. The 2015 scholarship recipients publication numbers continued to increase after their scholarship tenure.

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

BACKGROUND

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

METHODS

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

RESULTS

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

CONCLUSIONS

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

Seven decades of chemotherapy clinical trials: a pan-cancer social network analysis

Clinical trials establish the standard of cancer care, yet the evolution and characteristics of the social dynamics between the people conducting this work remain understudied. We performed a social network analysis of authors publishing chemotherapy-based prospective trials from 1946 to 2018 to understand how social influences, including the role of gender, have influenced the growth and development of this network, which has expanded exponentially from fewer than 50 authors in 1946 to 29,197 in 2018. While 99.4% of authors were directly or indirectly connected by 2018, our results indicate a tendency to predominantly connect with others in the same or similar fields, as well as an increasing disparity in author impact and number of connections. Scale-free effects were evident, with small numbers of individuals having disproportionate impact. Women were under-represented and likelier to have lower impact, shorter productive periods (P < 0.001 for both comparisons), less centrality, and a greater proportion of co-authors in their same subspecialty. The past 30 years were characterized by a trend towards increased authorship by women, with new author parity anticipated in 2032. The network of cancer clinical trialists is best characterized as strategic or mixed-motive, with cooperative and competitive elements influencing its appearance. Network effects such as low centrality, which may limit access to high-profile individuals, likely contribute to the observed disparities.

Consolidation in a crisis: Patterns of international collaboration in early COVID-19 research

This paper seeks to understand whether a catastrophic and urgent event, such as the first months of the COVID-19 pandemic, accelerates or reverses trends in international collaboration, especially in and between China and the United States. A review of research articles produced in the first months of the COVID-19 pandemic shows that COVID-19 research had smaller teams and involved fewer nations than pre-COVID-19 coronavirus research. The United States and China were, and continue to be in the pandemic era, at the center of the global network in coronavirus related research, while developing countries are relatively absent from early research activities in the COVID-19 period. Not only are China and the United States at the center of the global network of coronavirus research, but they strengthen their bilateral research relationship during COVID-19, producing more than 4.9% of all global articles together, in contrast to 3.6% before the pandemic. In addition, in the COVID-19 period, joined by the United Kingdom, China and the United States continued their roles as the largest contributors to, and home to the main funders of, coronavirus related research. These findings suggest that the global COVID-19 pandemic shifted the geographic loci of coronavirus research, as well as the structure of scientific teams, narrowing team membership and favoring elite structures. These findings raise further questions over the decisions that scientists face in the formation of teams to maximize a speed, skill trade-off. Policy implications are discussed.

Ranking Multi-Metric Scientific Achievements Using a Concept of Pareto Optimality

The ranking of multi-metric scientific achievements is a challenging task. For example, the scientific ranking of researchers utilizes two major types of indicators; namely, number of publications and citations. In fact, they focus on how to select proper indicators, considering only one indicator or combination of them. The majority of ranking methods combine several indicators, but these methods are faced with a challenging concern—the assignment of suitable/optimal weights to the targeted indicators. Pareto optimality is defined as a measure of efficiency in the multi-objective optimization which seeks the optimal solutions by considering multiple criteria/objectives simultaneously. The performance of the basic Pareto dominance depth ranking strategy decreases by increasing the number of criteria (generally speaking, when it is more than three criteria). In this paper, a new, modified Pareto dominance depth ranking strategy is proposed which uses some dominance metrics obtained from the basic Pareto dominance depth ranking and some sorted statistical metrics to rank the scientific achievements. It attempts to find the clusters of compared data by using all of indicators simultaneously. Furthermore, we apply the proposed method to address the multi-source ranking resolution problem which is very common these days; for example, there are several world-wide institutions which rank the world’s universities every year, but their rankings are not consistent. As our case studies, the proposed method was used to rank several scientific datasets (i.e., researchers, universities, and countries) for proof of concept.

Measuring the academic value of academic medical centers: describing a methodology for developing an evaluation model at one Academic Medical Center

Background

Academic Medical Centers (AMCs) must simultaneously serve different purposes:

Delivery of high quality healthcare services to patients, as the main mission, supported by other core missions such as academic activities, i.e., researching, teaching and tutoring, while maintaining solvency.

This study aims to develop a methodology for constructing models evaluating the academic value provided by AMCs and implementing it at the largest AMC in Israel.

Methods

Thirty five practiced educators and researchers, academic experts, faculty members and executives, all employed by a metropolitan 1500-bed AMC, were involved in developing academic quality indicators. First, an initial list of AMCs’ academic quality indicators was drafted, using a literature review and consulting scholars. Afterwards, additional data and preferences were collected by conducting semi-structured interviews, complemented by a three-round Delphi Panel. Finally, the methodology for constructing a model evaluating the academic value provided by the AMC was developed.

Results

The composite academic quality indicators methodology consists of nine indicators (relative weight in parentheses): ‘Scientific Publications Value’ (18.7%), ‘Completed Studies’ (13.5%), ‘Authors Value’ (13.0%), ‘Residents Quality’ (11.3%), ‘Competitive Grants Budget’ (10.2%), ‘Academic Training’ (8.7%), ‘Academic Positions’ (8.3%), ‘Number of Studies’ (8.3%) and ‘Academic Supervision’ (8.0%).

These indicators were grouped into three core categories: ‘Education’, ‘Research’ and ‘Publications’, having almost the same importance on a scale from zero to one (0–1), i.e., 0.363, 0.320, and 0.317, respectively. The results demonstrated a high level of internal consistency (Cronbach-alpha range: 0.79–0.86).

Conclusions

We have found a gap in the ability to measure academic value provided by AMCs. The main contribution of this research is the development of methodology for constructing evaluation models for AMCs academic performance. Further studies are needed to further test the validity and reliability of the proposed methodology at other sites.

Electronic supplementary material

The online version of this article (10.1186/s13584-019-0334-4) contains supplementary material, which is available to authorized users.

Collaborative patterns, authorship practices and scientific success in biomedical research: a network analysis

OBJECTIVE

To investigate the relationship between biomedical researchers' collaborative and authorship practices and scientific success.

DESIGN

Longitudinal quantitative analysis of individual researchers' careers over a nine-year period.

SETTING

A leading biomedical research institution in the United Kingdom.

PARTICIPANTS

Five hundred and twenty-five biomedical researchers who were in employment on 31 December 2009.

MAIN OUTCOME MEASURES

We constructed the co-authorship network in which nodes are the researchers, and links are established between any two researchers if they co-authored one or more articles. For each researcher, we recorded the position held in the co-authorship network and in the bylines of all articles published in each three-year interval and calculated the number of citations these articles accrued until January 2013. We estimated maximum likelihood negative binomial panel regression models.

RESULTS

Our analysis suggests that collaboration sustained success, yet excessive co-authorship did not. Last positions in non-alphabetised bylines were beneficial for higher academic ranks but not for junior ones. A professor could witness a 20.57% increase in the expected citation count if last-listed non-alphabetically in one additional publication; yet, a lecturer suffered from a 13.04% reduction. First positions in alphabetised bylines were positively associated with performance for junior academics only. A lecturer could experience a 8.78% increase in the expected citation count if first-listed alphabetically in one additional publication. While junior researchers amplified success when brokering among otherwise disconnected collaborators, senior researchers prospered from socially cohesive networks, rich in third-party relationships.

CONCLUSIONS

These results help biomedical scientists shape successful careers and research institutions develop effective assessment and recruitment policies that will ultimately sustain the quality of biomedical research and patient care.

A Decade of Translational Bioinformatics: A Retrospective Analysis of "Year-in-Review" Presentations

For the past 11 years, the year-in-review (YIR) keynote presentation at the AMIA Informatics summit has been a perennial highlight. We hypothesized that the presented material from these keynotes could be used to assess both the recent trajectory of topics in informatics-especially translational bioinformatics (TBI)-as well as the scientific merit of the crowd-sourced process used to nominate, review, and select the papers presented at the YIR. We compare YIR articles to a background set of non-YIR articles from informatics journals using structured metadata and qualitative thematic analysis, paying specific attention to trends and popularity over time. These trends were inspected both internally (comparing the YIR sessions to each other) and externally (comparing them to the overall content of scientific literature for the same time period). In doing so, we identified some unexpected patterns that suggest important opportunities for TBI research in the future.

Evaluation of the research capacity and culture of allied health professionals in a large regional public health service

Purpose

The first aim of this study was to evaluate the current research capacity and culture among allied health professionals (AHPs) working in a large regional health service. The second aim of this study was to undertake principal component analyses (PCAs) to determine key components influencing our research capacity and culture.

Patients and methods

As part of a cross-sectional observational study, the Research Capacity and Culture (RCC) tool was administered to AHPs working in Gold Coast Health to measure self-reported research capacity and culture across Organization, Team, and Individual domains, including barriers to and motivators for performing research. An exploratory PCA was performed to identify key components influencing research capacity and culture in each of the three domains, and the results were compared with the findings of a previous study performed in a large metropolitan health district.

Results

This study found moderate levels of research capacity and culture across all domains, with higher scores (median, IQR) reported for the Organization domain (7,5–8) compared to the Team (6,3–8) and Individual domains (5,2–7). Two components were identified in each domain. Components in the Organization domain included “research culture” and “research infrastructure”; components in the Team domain included “valuing and sharing research” and “supporting research”; and components in the Individual domain included “skills for conducting research” and “skills for searching and critiquing the literature”. These components were found to be highly correlated with each other, with correlations between components within each domain ranging from 0.459 to 0.702.

Conclusion

The results of this study reinforce the need for an integrated “whole of system” approach to research capacity building. Ongoing investment in tailored support and infrastructure is required to maintain current areas of strengths and build on identified areas of weakness at the level of organizations, teams, and individual AHPs, and consideration should also be given as to how support across these three levels is integrated.

Organisational benefits of a strong research culture in a health service: a systematic review

Objective The aim of the present study was to determine whether there is an association between having research culture in a health service and better organisational performance. Methods Using systematic review methods, databases were searched, inclusion criteria applied and study quality appraised. Data were extracted from selected studies and the results were synthesised descriptively. Results Eight studies were selected for review. Five studies compared health services with high versus low levels of research activity among the workforce. Three studies evaluated the effect of specific interventions focused on the health workforce. All studies reported a positive association between research activity and organisational performance. Improved organisational performance included lower patient mortality rates (two of two studies), higher levels of patient satisfaction (one of one study), reduced staff turnover (two of two studies), improved staff satisfaction (one of two studies) and improved organisational efficiency (four of five studies). Conclusions A stronger research culture appears to be associated with benefits to patients, staff and the organisation. What is known about this topic? Research investment in the health workforce can increase research productivity of the health workforce. In addition, investment in clinical research can lead to positive health outcomes. However, it is not known whether a positive research culture among the health workforce is associated with improved organisational performance. What does this paper add? The present systematic review of the literature provides evidence that a positive research culture and interventions directed at the health workforce are associated with patient, staff and organisational benefits. What are the implications for practitioners? For health service managers and policy makers, one interpretation of the results could be to provide support for initiatives directed at the health workforce to increase a research culture in health services. However, because association does not imply causation, managers need to interpret the results with caution and evaluate the effect of any initiatives to increase the research culture of the health workforce on the performance of their organisation.

Assessing the impact of healthcare research: A systematic review of methodological frameworks

BACKGROUND

Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

METHODS AND FINDINGS

Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) 'primary research-related impact', (2) 'influence on policy making', (3) 'health and health systems impact', (4) 'health-related and societal impact', and (5) 'broader economic impact'. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

CONCLUSIONS

The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

Neurosurgeon academic impact is associated with clinical outcomes after clipping of ruptured intracranial aneurysms

BACKGROUND

Surgeon-dependent factors such as experience and volume are associated with patient outcomes. However, it is unknown whether a surgeon's research productivity could be related to outcomes. The main aim of this study is to investigate the association between the surgeon's academic productivity and clinical outcomes following neurosurgical clipping of ruptured aneurysms.

METHODS

We performed a post-hoc analysis of 3567 patients who underwent clipping of ruptured intracranial aneurysms in the randomized trials of tirilazad mesylate from 1990 to 1997. These trials included 162 centers and 156 surgeons from 21 countries. Primary and secondary outcomes were: Glasgow outcome scale score and mortality, respectively. Total publications, H-index, and graduate degrees were used as academic indicators for each surgeon. The association between outcomes and academic factors were assessed using a hierarchical logistic regression analysis, adjusting for patient covariates.

RESULTS

Academic profiles were available for 147 surgeons, treating a total of 3307 patients. Most surgeons were from the USA (62, 42%), Canada (18, 12%), and Germany (15, 10%). On univariate analysis, the H-index correlated with better functional outcomes and lower mortality rates. In the multivariate model, patients under the care of surgeons with higher H-indices demonstrated improved neurological outcomes (p = 0.01) compared to surgeons with lower H-indices, without any significant difference in mortality. None of the other academic indicators were significantly associated with outcomes.

CONCLUSION

Although prognostication following surgery for ruptured intracranial aneurysms primarily depends on clinical and radiological factors, the academic impact of the operating neurosurgeon may explain some heterogeneity in surgical outcomes.

Benchmarking the research track record and level of appointment of Australian occupational therapy academics

BACKGROUND

Academic faculty are active contributors to the evidence base of the profession. Little is known about the research productivity of Australian occupational therapy academics.

METHODS

A bibliographic analysis of Australian occupational therapy faculty websites and corresponding Scopus citation database profiles was conducted. A description of current research productivity benchmarks and relationships with holding a doctorate, academic appointment level and institutional characteristics were explored.

RESULTS

The median H-index of Clinical and Associate Lecturers was 0, 1 for Lecturers, 4 for Senior Lecturers, 7.5 for Associate Professors and 13 for Professors. The number of publications and number of citing documents follows a Pareto tendency, where 20% of researchers are responsible for around 80% of the research track record of Australian occupational therapy academics.

CONCLUSION

The findings provide a benchmark for Australian occupational therapy faculty research track records and associations with appointment levels. This benchmark differs across countries and disciplines. The results raise considerations for enhancing the overall research track record of occupational therapy academics.

Research impact of systems-level long-term care research: a multiple case study

BACKGROUND

Traditional reporting of research outcomes and impacts, which tends to focus on research product publications and grant success, does not capture the value, some contributions, or the complexity of research projects. The purpose of this study was to understand the contributions of five systems-level research projects as they were unfolding at the Bruyère Centre for Learning, Research and Innovation (CLRI) in long-term care (LTC) in Ottawa, Ontario, Canada. The research questions were, (1) How are partnerships with research end-users (policymakers, administrators and other public/private organisations) characterised? (2) How have interactions with the CLRI Management Committee and Steering Committee influenced the development of research products? (3) In what way have other activities, processes, unlinked actors or organisations been influenced by the research project activities?

METHODS

The study was guided by Kok and Schuit's concept of research impacts, using a multiple case study design. Data were collected through focus groups and interviews with research teams, a management and a steering committee, research user partners, and unlinked actors. Documents were collected and analysed for contextual background.

RESULTS

Cross-case analysis revealed four major themes: (1) Benefits and Perceived Tensions: Working with Partners; (2) Speaking with the LTC Community: Interactions with the CLRI Steering Committee; (3) The Knowledge Broker: Interactions with the Management Committee; and (4) All Forms of Research Contributions.

CONCLUSIONS

Most contributions were focused on interactions with networks and stimulating important conversations in the province about LTC issues. These contributions were well-supported by the Steering and Management Committees' research-to-action platform, which can be seen as a type of knowledge brokering model. It was also clear that researcher-user partnerships were beneficial and important.

Development and use of a research productivity assessment tool for clinicians in low-resource settings in the Pacific Islands: a Delphi study

Background

Research performance assessments have proliferated, but research indicators for use amongst clinicians in poorly resourced countries have been ill-defined. The aims of the present paper were to determine a set of indicators as determined by clinician participants from the Pacific Islands and a panel of research experts for use in the performance assessment of clinicians.

Methods

Two focus group discussions, one for nurses and one for doctors, were used to obtain the views of 28 Pacific Island clinicians of the BRRACAP Study about what the research indicators should be. A modified Delphi survey was used to obtain a consensus amongst 19 research experts, with Pacific Island research experience, as to what the indicators should be and then to rank these in terms of importance. A survey of the participants obtained data on the research tasks/actions performed 20 months after the initial research workshop. A resultant tool comprising of 21 indicators was used to assess the performance of 18 Pacific participants.

Results

The Pacific Island clinicians determined that research was important and that performance should be measured. They identified research indicators that could be used in their settings and ranked their importance using a points system. The panel of experts identified implementation of research findings, collaborations and actual change in practice as more important, with bibliometric measurements low down in the scale. Although only 64 % of the 28 BRRACAP Study participants returned the questionnaire, 39 % of those performed more than half of the 21 indicators used. Of the 18 Pacific clinicians assessed, 7 (39 %) performed 10 or more tasks.

Conclusions

A research performance assessment tool was developed using process and output indicators identified by Pacific clinicians and a panel of research experts. The tool, which placed emphasis on process and outputs that were not bibliometric based, proved useful in assessing the performance of Pacific clinicians working in a low-resource setting.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0077-4) contains supplementary material, which is available to authorized users.

Research capacity and culture of the Victorian public health allied health workforce is influenced by key research support staff and location

OBJECTIVE

The aim of the present study was to identify and understand the self-rated research capacity and culture of the allied health workforce. METHODS. The present study was a cross-sectional survey. The Research Capacity and Culture tool was disseminated to all Victorian public health allied health departments. General demographic data were also collected, including the presence of an organisational allied health research lead.

RESULTS

Five hundred and twenty fully completed surveys were returned by participants; all allied health disciplines and all grades were represented. One hundred and eighty-six participants had an organisational allied health research lead and 432 were located in a metropolitan-based health service. There were significant differences (P < 0.05) within all organisational and team research skills between those with and without a research lead, together with those in different service locations (metropolitan vs non-metropolitan). Higher self-ratings in individual research skills (P < 0.05) were primarily associated with more senior and metropolitan-located clinicians.

CONCLUSION

The allied health workforce identifies as a group that is ready to build the evidence to support clinical practice yet requires a whole-systems approach to do so. The results of the present study suggest that the development of key people to build capacity at a higher organisational level has a flow-down effect on research capacity and culture.

Tackling the "so what" problem in scientific research: a systems-based approach to resource and publication tracking

Peer-reviewed publications are one measure of scientific productivity. From a project, program, or institutional perspective, publication tracking provides the quantitative data necessary to guide the prudent stewardship of federal, foundation, and institutional investments by identifying the scientific return for the types of support provided. In this article, the authors describe the Vanderbilt Institute for Clinical and Translational Research's (VICTR's) development and implementation of a semiautomated process through which publications are automatically detected in PubMed and adjudicated using a "just-in-time" workflow by a known pool of researchers (from Vanderbilt University School of Medicine and Meharry Medical College) who receive support from Vanderbilt's Clinical and Translational Science Award. Since implementation, the authors have (1) seen a marked increase in the number of publications citing VICTR support, (2) captured at a more granular level the relationship between specific resources/services and scientific output, (3) increased awareness of VICTR's scientific portfolio, and (4) increased efficiency in complying with annual National Institutes of Health progress reports. They present the methodological framework and workflow, measures of impact for the first 30 months, and a set of practical lessons learned to inform others considering a systems-based approach for resource and publication tracking. They learned that contacting multiple authors from a single publication can increase the accuracy of the resource attribution process in the case of multidisciplinary scientific projects. They also found that combining positive (e.g., congratulatory e-mails) and negative (e.g., not allowing future resource requests until adjudication is complete) triggers can increase compliance with publication attribution requests.

Effect of the economic crisis on the production of immunology patents managed through the Patent Cooperation Treaty agreement from 2004-2011

OBJECTIVES

To determine the evolution of patents in immunology, as a result of research and innovation in the years 2004-2011.

DESIGN

The search for patents published internationally in immunology was made by using the SCOPUSTM database. SCOPUS gives information about over 23 million patents. The extracted data from patents were: inventors and applicants; their nationalities; sections, classes and subclasses of the International Patent Classification.

PARTICIPANTS

89 countries.

SETTING

Data have been obtained from the database SCOPUS. It has been used for the international patent classification.

MAIN OUTCOME MEASURES

Patents by country, Productive sectors, Productive areas.

RESULTS

A total of 17,281 patents were applied for immunology during 2004-2011 of which 16,811 were from 30 Organisation for Economic Cooperation and Development countries, and 5326 from 28 countries in the European Union. These patents were granted in 89 countries and 13,699 of them were submitted by researchers from only one country. Private entities applied for 62.45% of all patents, universities 17.48%, hospitals 3.40% and public research organisations and private applicants applied for the rest. The university that made more applications was the University of California with 315 and the company was Genentech Inc. (US) with 302. The reduction in the number of applications of international patents in all disciplines of science also affected the area of immunology.

CONCLUSIONS

Collaboration in immunology between universities, companies and hospitals is hard because their interests are different. It is shown in patent applications that the majority of patents in immunology are applied for by only one entity. Patents in immunology are developed, mainly, in aspects such as medical preparations, peptides, mutation or genetic engineering, therapeutic activity of chemical compounds and analysing materials by determining their chemical or physical properties.

Poorly cited articles in peer-reviewed cardiovascular journals from 1997 to 2007: analysis of 5-year citation rates

BACKGROUND

The extent to which articles are cited is a surrogate of the impact and importance of the research conducted; poorly cited articles may identify research of limited use and potential wasted investments. We assessed trends in the rates of poorly cited articles and journals in the cardiovascular literature from 1997 to 2007.

METHODS AND RESULTS

We identified original articles published in cardiovascular journals and indexed in the Scopus citation database from 1997 to 2007. We defined poorly cited articles as those with ≤5 citations in the 5 years following publication and poorly cited journals as those with >75% of journal content poorly cited. We identified 164 377 articles in 222 cardiovascular journals from 1997 to 2007. From 1997 to 2007, the number of cardiovascular articles and journals increased by 56.9% and 75.2%, respectively. Of all the articles, 75 550 (46.0%) were poorly cited, of which 25 650 (15.6% overall) had no citations. From 1997 to 2007, the proportion of poorly cited articles declined slightly (52.1%-46.2%, trend P<0.001), although the absolute number of poorly cited articles increased by 2595 (trend P<0.001). At a journal level, 44% of cardiovascular journals had more than three-fourths of the journal's content poorly cited at 5 years.

CONCLUSION

Nearly half of all peer-reviewed articles published in cardiovascular journals are poorly cited 5 years after publication, and many are not cited at all. The cardiovascular literature and the number of poorly cited articles both increased substantially from 1997 to 2007. The high proportion of poorly cited articles and journals suggests inefficiencies in the cardiovascular research enterprise.

[Trends and evolutions of French breast cancer research: a bibliometric study]

This article presents a bibliometric study carried out in order to describe the trends and evolutions of French breast cancer research from 2003 to 2013. The results show an increase in the number of publications, especially international publications coordinated by non-French institutions. The most visible topics, in terms of number of publications by keywords, are related to biology, clinical trials and genetics. Most publications are written by authors affiliated to comprehensive cancer centres, followed by universities, research centres, university hospitals and governmental agencies. The importance of publications by topic varies throughout the years: there has been an increase of the number of publications related to targeted therapies or genomics. The importance of institutions or country affiliation of authors varies with the topics. This study, especially the analysis by keywords, enables the coordinators of research programs to identify the predominant actors and themes.

Measuring the outcome of biomedical research: a systematic literature review

BACKGROUND

There is an increasing need to evaluate the production and impact of medical research produced by institutions. Many indicators exist, yet we do not have enough information about their relevance. The objective of this systematic review was (1) to identify all the indicators that could be used to measure the output and outcome of medical research carried out in institutions and (2) enlist their methodology, use, positive and negative points.

METHODOLOGY

We have searched 3 databases (Pubmed, Scopus, Web of Science) using the following keywords: [Research outcome* OR research output* OR bibliometric* OR scientometric* OR scientific production] AND [indicator* OR index* OR evaluation OR metrics]. We included articles presenting, discussing or evaluating indicators measuring the scientific production of an institution. The search was conducted by two independent authors using a standardised data extraction form. For each indicator we extracted its definition, calculation, its rationale and its positive and negative points. In order to reduce bias, data extraction and analysis was performed by two independent authors.

FINDINGS

We included 76 articles. A total of 57 indicators were identified. We have classified those indicators into 6 categories: 9 indicators of research activity, 24 indicators of scientific production and impact, 5 indicators of collaboration, 7 indicators of industrial production, 4 indicators of dissemination, 8 indicators of health service impact. The most widely discussed and described is the h-index with 31 articles discussing it.

DISCUSSION

The majority of indicators found are bibliometric indicators of scientific production and impact. Several indicators have been developed to improve the h-index. This indicator has also inspired the creation of two indicators to measure industrial production and collaboration. Several articles propose indicators measuring research impact without detailing a methodology for calculating them. Many bibliometric indicators identified have been created but have not been used or further discussed.

The research capacity and culture of Australian podiatrists

Background

Best practice clinical health care is widely recognised to be founded on evidence based practice. Enhancing evidence based practice via the rapid translation of new evidence into every day clinical practice is fundamental to the success of health care and in turn health care professions. There is little known about the collective research capacity and culture of the podiatry profession across Australia. Thus, the aim of this study was to investigate the research capacity and culture of the podiatry profession within Australia and determine if there were any differences between podiatrists working in different health sectors and workplaces.

Method

All registered podiatrists were eligible to participate in a cross-sectional online survey. The Australian Podiatry Associations disseminated the survey and all podiatrists were encouraged to distribute it to colleagues. The Research Capacity and Culture (RCC) tool was used to collect all research capacity and culture item variables using a 10-point scale (1 = lowest; 10 = highest). Additional demographic, workplace and health sector data variables were also collected. Mann–Whitney-U, Kruskal–Wallis and logistic regression analyses were used to determine any difference between health sectors and workplaces. Word cloud analysis was used for qualitative responses of individual motivators and barriers to research culture.

Results

There were 232 fully completed surveys (6% of Australian registered podiatrists). Overall respondents reported low success or skills (Median rating < 4) on the majority of individual success or skill items. Podiatrists working in multi-practitioner workplaces reported higher individual success or skills in the majority of items compared with sole practitioners (p < 0.05). Non-clinical and public health sector podiatrists reported significantly higher post-graduate study enrolment or completion, research activity participation, provisions to undertake research and individual success or skill than those working privately.

Conclusions

This study suggests that podiatrists in Australia report similar low levels of research success or skill to those reported in other allied health professions. The workplace setting and health sector seem to play key roles in self reported research success and skills. This is important knowledge for podiatrists and researchers aiming to translate research evidence into clinical practice.

Disease specific productivity of american cancer hospitals

CONTEXT

Research-oriented cancer hospitals in the United States treat and study patients with a range of diseases. Measures of disease specific research productivity, and comparison to overall productivity, are currently lacking.

HYPOTHESIS

Different institutions are specialized in research of particular diseases.

OBJECTIVE

To report disease specific productivity of American cancer hospitals, and propose a summary measure.

METHOD

We conducted a retrospective observational survey of the 50 highest ranked cancer hospitals in the 2013 US News and World Report rankings. We performed an automated search of PubMed and Clinicaltrials.gov for published reports and registrations of clinical trials (respectively) addressing specific cancers between 2008 and 2013. We calculated the summed impact factor for the publications. We generated a summary measure of productivity based on the number of Phase II clinical trials registered and the impact factor of Phase II clinical trials published for each institution and disease pair. We generated rankings based on this summary measure.

RESULTS

We identified 6076 registered trials and 6516 published trials with a combined impact factor of 44280.4, involving 32 different diseases over the 50 institutions. Using a summary measure based on registered and published clinical trails, we ranked institutions in specific diseases. As expected, different institutions were highly ranked in disease-specific productivity for different diseases. 43 institutions appeared in the top 10 ranks for at least 1 disease (vs 10 in the overall list), while 6 different institutions were ranked number 1 in at least 1 disease (vs 1 in the overall list).

CONCLUSION

Research productivity varies considerably among the sample. Overall cancer productivity conceals great variation between diseases. Disease specific rankings identify sites of high academic productivity, which may be of interest to physicians, patients and researchers.

How variable are the journal impact measures?

Purpose

– The purpose of this paper is to examine the variability of the impact factor (IF) and additional metrics in biomedical journals to provide some clues to the reliability of journal citation indicators.

Design/methodology/approach

– Having used ISI Journal Citation Reports, from 2005 to 2011, the authors extracted 62 subject categories related to biomedical sciences. The category lists and citation profile for each journal were then downloaded and extracted. Coefficient of variation was applied to estimate the overall variability of the journal citation indicators.

Findings

– Total citation indicators for 3,411 journals were extracted and examined. The overall variability of IFs and other journal citation measures in basic, clinical or translational, open access or subscription journals decreased while the quality and prestige of those journals developed. Interestingly, journal citation measures produced dissimilar variability trends and thus highlighted the importance of using multiple instead of just one measure in evaluating the performance and influence of biomedical journals. Eigenfactor™, Article's Influence and Cited Half Life proposed as more reliable indicators.

Originality/value

– The relative variability of the journal citation measures in biomedical journals would decrease with a development in the impact and quality of journals. Eigenfactor™ and Cited Half Life are suggested as more reliable measures indicating few changes during the study period and across different impact level journals. These findings will be useful for librarians, researchers and decision makers who need to use citation measures as evaluative tools.

Trends in performance indicators of neuroimaging anatomy research publications: a bibliometric study of major neuroradiology journal output over four decades based on web of science database

Quantitative, qualitative, and innovative application of bibliometric research performance indicators to anatomy and radiology research and education can enhance cross-fertilization between the two disciplines. We aim to use these indicators to identify long-term trends in dissemination of publications in neuroimaging anatomy (including both productivity and citation rates), which has subjectively waned in prestige during recent years. We examined publications over the last 40 years in two neuroradiological journals, AJNR and Neuroradiology, and selected and categorized all neuroimaging anatomy research articles according to theme and type. We studied trends in their citation activity over time, and mathematically analyzed these trends for 1977, 1987, and 1997 publications. We created a novel metric, "citation half-life at 10 years postpublication" (CHL-10), and used this to examine trends in the skew of citation numbers for anatomy articles each year. We identified 367 anatomy articles amongst a total of 18,110 in these journals: 74.2% were original articles, with study of normal anatomy being the commonest theme (46.7%). We recorded a mean of 18.03 citations for each anatomy article, 35% higher than for general neuroradiology articles. Graphs summarizing the rise (upslope) in citation rates after publication revealed similar trends spanning two decades. CHL-10 trends demonstrated that more recently published anatomy articles were likely to take longer to reach peak citation rate. Bibliometric analysis suggests that anatomical research in neuroradiology is not languishing. This novel analytical approach can be applied to other aspects of neuroimaging research, and within other subspecialties in radiology and anatomy, and also to foster anatomical education.

Characterizing scientific production of Italian Oral Surgery professionals through evaluation of bibliometric indices

The aim of this study was to characterize the scientific production of Italian Oral Surgery professionals by evaluating different bibliometric indices. The bibliometric evaluation was conducted on the Scopus Database upon all the Active Members joining three important Italian scientific societies in Oral Surgery (SIdCO, SIO, and SICOI). The scientific production was analysed by considering the number of total publications, number of total citations, h-index, and hc-index. Moreover, the overall sample was divided into two groups (Academics and Not Academics), according to the fact the professionals had or not a university position, and then into sub-groups according to the different career lengths. Statistical analyses were performed to compare the scientific productivity amongst groups. For all the considered parameters a lack of homogeneity between groups was reported, and significantly greater mean values were recorded for the Academics compared to the Not Academics Group. Moreover, the h-index values increased more regularly as the career length progressed than the hc-index values, even if the last seemed to be less variable. h- and hc-indices are both stable bibliometric parameters, but as the hc-index values are related not only to the number of citation but also to their age, it seems to be less influenced by the authors' career length. Bibliometric analysis of the scientific production in dentistry may facilitate the recognition of factors that may further enhance research activity and clinical performance and be useful for a comparative assessment of authors or research groups in terms of quality and quantity of the scientific production.

Research and collaboration overview of Institut Pasteur International Network: a bibliometric approach toward research funding decisions

BACKGROUND

Institut Pasteur International Network (IPIN), which includes 32 research institutes around the world, is a network of research and expertise to fight against infectious diseases. A scientometric approach was applied to describe research and collaboration activities of IPIN.

METHODS

Publications were identified using a manual search of IPIN member addresses in Science Citation Index Expanded (SCIE) between 2006 and 2011. Total publications were then subcategorized by geographic regions. Several scientometric indicators and the H-index were employed to estimate the scientific production of each IPIN member. Subject and geographical overlay maps were also applied to visualize the network activities of the IPIN members.

RESULTS

A total number of 12667 publications originated from IPIN members. Each author produced an average number of 2.18 papers and each publication received an average of 13.40 citations. European Pasteur Institutes had the largest amount of publications, authored papers, and H-index values. Biochemistry and molecular biology, microbiology, immunology and infectious diseases were the most important research topics, respectively. Geographic mapping of IPIN publications showed wide international collaboration among IPIN members around the world.

CONCLUSION

IPIN has strong ties with national and international authorities and organizations to investigate the current and future health issues. It is recommended to use scientometric and collaboration indicators as measures of research performance in IPIN future policies and investment decisions.

Research Impact: A United Kingdom Occupational Therapy Research Foundation Perspective

Introduction:

The United Kingdom Occupational Therapy Research Foundation provides profession-specific funding for research, but what is its impact? An exploratory study was undertaken to gather intelligence on the impact of completed projects.

Method:

Eleven grant holders were invited to complete a research impact assessment form based on the multidimensional Becker Medical Library Model. Four domains of impact were included: research outputs and advancement of knowledge, clinical implementation, community and public benefit, and economic benefit.

Results:

Eight impact assessment forms were returned (73%); these reflected grants awarded across the four funding streams, although there were no returns from grant holders receiving over £50,000. Clearly evident was that most researchers were seeking publication in a range of journals, and disseminating findings at conferences. Other notable impacts included the extent to which participants were using findings in educational activities; the apparent importance of the collaborative partnerships in terms of the clinical application of findings post project; the active engagement of practitioners and service users; and the opportunities arising for follow-on funding or projects.

Conclusion:

Capturing research impact is complex, but vital. The project provided a strong argument to adopt a prospective approach to impact reporting throughout, and beyond, the life of a grant.

Citation analysis may severely underestimate the impact of clinical research as compared to basic research

Background

Citation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research.

Methodology

A visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact.

Results

Visualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields: Cardiac & cardiovascular systems, Clinical neurology, and Surgery. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.

Conclusions

Popular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.