Evaluating peer reviews. Pilot testing of a grading instrument

Tools used to assess the quality of peer review reports: a methodological systematic review

Background

A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer review reports in biomedical research.

Methods

We conducted a methodological systematic review by searching PubMed, EMBASE (via Ovid) and The Cochrane Methodology Register (via The Cochrane Library) as well as Google® for all reports in English describing a tool for assessing the quality of a peer review report in biomedical research. Data extraction was performed in duplicate using a standardized data extraction form. We extracted information on the structure, development and validation of each tool. We also identified quality components across tools using a systematic multi-step approach and we investigated quality domain similarities among tools by performing hierarchical, complete-linkage clustering analysis.

Results

We identified a total number of 24 tools: 23 scales and 1 checklist. Six tools consisted of a single item and 18 had several items ranging from 4 to 26. None of the tools reported a definition of ‘quality’. Only 1 tool described the scale development and 10 provided measures of validity and reliability. Five tools were used as an outcome in a randomized controlled trial (RCT). Moreover, we classified the quality components of the 18 tools with more than one item into 9 main quality domains and 11 subdomains. The tools contained from two to seven quality domains. Some domains and subdomains were considered in most tools such as the detailed/thorough (11/18) nature of reviewer’s comments. Others were rarely considered, such as whether or not the reviewer made comments on the statistical methods (1/18).

Conclusion

Several tools are available to assess the quality of peer review reports; however, the development and validation process is questionable and the concepts evaluated by these tools vary widely. The results from this study and from further investigations will inform the development of a new tool for assessing the quality of peer review reports in biomedical research.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0688-x) contains supplementary material, which is available to authorized users.

Validity of a Structured Method of Selecting Abstracts for a Plastic Surgical Scientific Meeting

In 1999, the European Association of Plastic Surgeons accepted a structured method to assess and select the abstracts that are submitted for its yearly scientific meeting. The two criteria used to evaluate whether such a selection method is accurate were reliability and validity. The authors previously established this method to be reliable on the basis of a prospective evaluation of the selection process used for the 2000 meeting of the European Association of Plastic Surgeons. It is more difficult to assess the validity of this method because there is no objective standard of quality of a scientific abstract against which the accuracy of selection can be assessed. This study statistically evaluated the accuracy of the meeting participants' assessment of presentations made during the meeting as the criterion standard for abstract selection on the basis of data obtained from the 2002 selection process. The authors evaluated the interobserver repeatability among five meeting participants of selecting the best presentations, the validity of the method of abstract selection after this criterion standard had been established, and the validity of reviewers' rating of abstracts as indicators of the scientific value of the actual presentations. The authors conclude that the assessment of platform presentations at a plastic surgical meeting is reliable. Accepting this assessment as the criterion standard, however, they could not prove the validity of their selection method or the validity of the reviewers' rating of abstracts as an indicator of the scientific value of the actual presentations.