|
1
|
Braithwaite J, Herkes J, Churruca K, Long JC, Pomare C, Boyling C, Bierbaum M, Clay-Williams R, Rapport F, Shih P, Hogden A, Ellis LA, Ludlow K, Austin E, Seah R, McPherson E, Hibbert PD, Westbrook J. Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models. BMJ Open 2019; 9:e025320. [PMID: 30928941 PMCID: PMC6475357 DOI: 10.1136/bmjopen-2018-025320] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/10/2018] [Revised: 02/04/2019] [Accepted: 02/06/2019] [Indexed: 12/20/2022] Open
Abstract
OBJECTIVES Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model. DESIGN We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework. DATA SOURCES All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements. DATA EXTRACTION AND SYNTHESIS Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed. RESULTS Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others. CONCLUSIONS All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.
Collapse
Affiliation(s)
- Jeffrey Braithwaite
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Jessica Herkes
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Kate Churruca
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Janet C Long
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Chiara Pomare
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Claire Boyling
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Mia Bierbaum
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Robyn Clay-Williams
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Frances Rapport
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Patti Shih
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Anne Hogden
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Louise A Ellis
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Kristiana Ludlow
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Elizabeth Austin
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Rebecca Seah
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Elise McPherson
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| | - Peter D Hibbert
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
- Division of Health Sciences, University of South Australia, Adelaide, South Australia, Australia
| | - Johanna Westbrook
- Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
| |
Collapse
|