Effect of blinding and unmasking on the quality of peer review

The objectives of this study were to see whether, in the opinion of authors, blinding or unmasking or a combination of the two affects the quality of reviews and to compare authors' and editors' assessments. In a trial conducted in the British Medical Journal, 527 consecutive manuscripts were randomized into one of three groups, and each was sent to two reviewers, who were randomized to receive a blinded or an unblinded copy of the manuscript. Review quality was assessed by two editors and the corresponding author. There was no significant difference in assessment between groups or between editors and authors. Reviews recommending publication were scored more highly than those recommending rejection.

Evidence-based surgery: A passing fad?

Recent years have witnessed the development of a new movement within health care: the promotion of "evidence-based medicine" (EBM). EBM is about integrating individual clinical expertise and the best external evidence derived from scientific research. Advocates claim that much medical practice is based too much on opinion and experience and insufficiently on research evidence. Their approach would increase the quality of care and its efficiency. This paper describes the principal steps in the evidence-based approach-systematic reviews of the literature and meta-analyses-and its shortcomings in surgery. These include the reliance of EBM on randomized trials, the lack of generalizability of scientific evidence to individual patients, the lack of attention to third party interests, the threat to the "art" of medicine, and the dangers of an oversimplistic approach. Although EBM clearly has a place, it does not have all the answers.

What observational studies can offer decision makers

Observational studies, for example cohort and case-control studies in which patients are allocated treatment on a non-random basis, are thought by some investigators to be flawed. This view results from the fact that, unlike experimental methods (randomized controlled trials; RCTs), the results of such observational studies are vulnerable to confounding. However, this view assumes that satisfactory adjustment of differences in risk or prognosis between treatment groups is impossible and it ignores some of the limitations of RCTs. While many of the problems involved in conducting RCTs could be overcome, the practical implications for researchers and funding bodies mean this is often not possible. In such circumstances, observational studies offer an alternative to an absence of any scientific evidence. While making use of observational methods, researchers must acknowledge the associated limitations: the inevitable inability to take unknown confounders into account, non-blinding of practitioners and patients, and the inclusion of practitioners' and patients' treatment preferences.

Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts

Research on the value of peer review is limited by the lack of a validated instrument to measure the quality of reviews. The aim of this study was to develop a simple, reliable, and valid scale that could be used in studies of peer review. A Review Quality Instrument (RQI) that assesses the extent to which a reviewer has commented on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and on two aspects of the review (constructiveness and substantiation of comments) was devised and tested. Its internal consistency was high (Cronbach's alpha 0.84). The mean total score (based on the seven items each scored on a 5-point Likert scale from 1 to 5) had good test-retest (Kw = 1.00) and inter-rater (Kw = 0.83) reliability. There was no evidence of floor or ceiling effects, construct validity was evident, and the respondent burden was acceptable (2-10 minutes). Although improvements to the RQI should be pursued, the instrument can be recommended for use in the study of peer review.

Threats to applicability of randomised trials: exclusions and selective participation

BACKGROUND

Although the randomised controlled trial (RCT) is regarded as the 'gold standard' in terms of evaluating the effectiveness of interventions, it is susceptible to challenges to its external validity if those participating are unrepresentative of the reference population for whom the intervention in question is intended. In the past, reporting on numbers and types of potential subjects that have been excluded by design, and centres, clinicians or patients that have elected not to participate, has generally been poor, and the threat to inference posed by possible selection bias is unclear.

METHODS

A systematic review was undertaken, based largely on MEDLINE and EMBASE with follow-up of cited references, to assess the extent, nature and importance of excluding potential subjects or the unwillingness of particular centres, clinicians or patients to participate.

RESULTS

RCTs vary widely in the extent to which potential future recipients of treatment are included. The reasons cited for excluding certain categories of patient may be medical or scientific. Medical reasons include a high risk of adverse effects and the belief that benefit will be relatively small or absent (or has already been established) in the groups in question. Scientific reasons include more precise estimates of treatment effect because of a relatively homogeneous sample and the reduction of potential bias by excluding those individuals most likely to be lost to follow-up. Many RCTs have blanket exclusions, such as the elderly, women and ethnic minorities, but reasons for these exclusions are seldom given. Evaluative research is undertaken predominantly in university or teaching centres. Non-randomised studies are more likely than RCTs to include non-teaching centres. The effect of patient non-participation appears to depend on whether the RCT is concerned with treatment of an existing condition or with disease prevention. Participants in treatment trials tend to be more severely ill than those who do not participate. In contrast, those who participate in prevention trials are more likely to have adopted a healthy lifestyle than those who decline. Most evaluative studies fail to document adequately the characteristics of those who, while eligible, do not participate. However, subjects included in RCTs (i.e. eligible and participating) tend to have a different prognosis than patients identified from clinical databases.

CONCLUSIONS

Narrow inclusion criteria may offer benefits such as increased precision and reduced loss to follow-up, but there are important disadvantages, such as uncertainty about extrapolation of results, which may result in denial of effective treatment to groups who might benefit, and delay in obtaining definitive results because of reduced recruitment rate. Selective participation by teaching centres and sicker patients in treatment RCTs may exaggerate the measured treatment effect. Prevention trials, on the other hand, may underestimate effects as participants have less capacity to benefit.

Predicting coronary disease risk based on short-term RR interval measurements: a neural network approach

Coronary heart disease is a multifactorial disease and it remains the most common cause of death in many countries. Heart rate variability has been used for non-invasive measurement of parasympathetic activity and prediction of cardiac death. Patterns of heart rate variability associated with respiratory sinus arrhythmia have recently been considered as possible indicators of coronary heart disease risk in asymptomatic subjects. The aim of this work is to detect individuals at varying risk of coronary heart disease based on short-term heart rate variability measurements under controlled respiration. Artificial neural networks are used to recognise Poincaré-plot-encoded heart rate variability patterns related to coronary heart disease risk. The results indicate a relatively coarse binary representation of Poincaré plots could be superior to an analogue encoding which, in principle, carries more information.

Where do UK health services researchers publish their findings?

Health services research has emerged as the third vital requirement for understanding and improving health care, alongside basic science and clinical research. This has coincided with more stringent management of research, in particular by funding bodies. The latter are seeking to use bibliographic databases to aid the monitoring of the output of their investments. The principal source of data in the UK is the Research Outputs Database (ROD) set up by the Wellcome Trust primarily to monitor basic and clinical research. Health services researchers' output is difficult to monitor in view of the large number and wide variety of journals in which they publish. In addition, nearly half the journals (representing 35% of the articles) are not currently covered by the ROD. Funding bodies will underestimate the quantity of health services researchers' output unless they take these findings into account.

Outcome of total hip replacement in Japan and England. Comparison of two retrospective cohorts

Two retrospective cohorts of patients who had undergone a primary total hip replacement (THR) were studied in Japan and England. We analyzed information from hospital case notes and self-administered questionnaires. THR was highly effective in both countries, with significant improvements in morbidity and health status. There were, however, important differences in clinical management and outcome. Japanese patients were more likely to be treated under regional anesthesia by means of an anterior/anterolateral incision, to be transfused, to have a prosthesis implanted without cement, and to stay longer in the hospital. A higher rate of serious in-hospital complications occurred in England, mostly due to the high frequency of hypotension. The incidences of minor complications were consistent with known differences in risks between the two countries. The persistent postoperative perception of limping reported by the English patients was unexpected.

Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial

OBJECTIVES

To examine the effect on peer review of asking reviewers to have their identity revealed to the authors of the paper.

DESIGN

Randomised trial. Consecutive eligible papers were sent to two reviewers who were randomised to have their identity revealed to the authors or to remain anonymous. Editors and authors were blind to the intervention.

MAIN OUTCOME MEASURES

The quality of the reviews was independently rated by two editors and the corresponding author using a validated instrument. Additional outcomes were the time taken to complete the review and the recommendation regarding publication. A questionnaire survey was undertaken of the authors of a cohort of manuscripts submitted for publication to find out their views on open peer review.

RESULTS

Two editors' assessments were obtained for 113 out of 125 manuscripts, and the corresponding author's assessment was obtained for 105. Reviewers randomised to be asked to be identified were 12% (95% confidence interval 0.2% to 24%) more likely to decline to review than reviewers randomised to remain anonymous (35% v 23%). There was no significant difference in quality (scored on a scale of 1 to 5) between anonymous reviewers (3.06 (SD 0.72)) and identified reviewers (3.09 (0.68)) (P=0.68, 95% confidence interval for difference - 0.19 to 0.12), and no significant difference in the recommendation regarding publication or time taken to review the paper. The editors' quality score for reviews (3.05 (SD 0.70)) was significantly higher than that of authors (2.90 (0.87)) (P<0.005, 95%confidence interval for difference - 0.26 to - 0.03). Most authors were in favour of open peer review.

CONCLUSIONS

Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.

Effect of blinding and unmasking on the quality of peer review: a randomized trial

CONTEXT

Little research has been conducted into the quality of peer review and, in particular, the effects of blinding peer reviewers to authors' identities or masking peer reviewers' identities.

OBJECTIVE

To determine whether concealing authors' identities from reviewers (blinding) and/or revealing the reviewer's identity to a coreviewer (unmasking) affects the quality of reviews, the time taken to carry out reviews, and the recommendation regarding publication.

DESIGN AND SETTING

Randomized trial of 527 consecutive manuscripts submitted to BMJ, which were randomized and each sent to 2 peer reviewers.

INTERVENTIONS

Manuscripts were randomized as to whether the reviewers were unmasked, masked, or uninformed that a study was taking place. Two reviewers for each manuscript were randomized to receive either a blinded or an unblinded version.

MAIN OUTCOME MEASURES

Mean total quality score, time taken to carry out the review, and recommendation regarding publication.

RESULTS

Of the 527 manuscripts entered into the study, 467 (89%) were successfully randomized and followed up. The mean total quality score was 2.87. There was little or no difference in review quality between the masked and unmasked groups (scores of 2.82 and 2.96, respectively) and between the blinded and unblinded groups (scores of 2.87 and 2.90, respectively). There was no apparent Hawthorne effect. There was also no significant difference between groups in the recommendations regarding publication or time taken to review.

CONCLUSIONS

Blinding and unmasking made no editorially significant difference to review quality, reviewers' recommendations, or time taken to review. Other considerations should guide decisions as to the form of peer review adopted by a journal, and improvements in the quality of peer review should be sought via other means.

What makes a good reviewer and a good review for a general medical journal?

CONTEXT

Selecting peer reviewers who will provide high-quality reviews is a central task of editors of biomedical journals.

OBJECTIVES

To determine the characteristics of reviewers for a general medical journal who produce high-quality reviews and to describe the characteristics of a good review, particularly in terms of the time spent reviewing and turnaround time.

DESIGN, SETTING, AND PARTICIPANTS

Surveys of reviewers of the 420 manuscripts submitted to BMJ between January and June 1997.

MAIN OUTCOME MEASURES

Review quality was assessed independently by 2 editors and by the corresponding author using a newly developed 7-item review quality instrument.

RESULTS

Of the 420 manuscripts, 345 (82%) had 2 reviews completed, for a total of 690 reviews. Authors' assessments of review quality were available for 507 reviews. The characteristics of reviewers had little association with the quality of the reviews they produced (explaining only 8% of the variation), regardless of whether editors or authors defined the quality of the review. In a logistic regression analysis, the only significant factor associated with higher-quality ratings by both editors and authors was reviewers trained in epidemiology or statistics. Younger age also was an independent predictor for editors' quality assessments, while reviews performed by reviewers who were members of an editorial board were rated of poorer quality by authors. Review quality increased with time spent on a review, up to 3 hours but not beyond.

CONCLUSIONS

The characteristics of reviewers we studied did not identify those who performed high-quality reviews. Reviewers might be advised that spending longer than 3 hours on a review on average did not appear to increase review quality as rated by editors and authors.

Development and validation of the Menorrhagia Outcomes Questionnaire

OBJECTIVE

To develop and evaluate the acceptability, reliability and validity of a short, patient-based questionnaire for assessing the outcomes of surgical treatment for menorrhagia due to benign disease.

DESIGN

A psychometric study by postal survey.

SETTING

Five hospitals in southeast England.

SAMPLE

One hundred and eleven women undergoing hysterectomy for menorrhagia due to benign disease.

METHODS

Data from a long research questionnaire used in the North West Thames Hysterectomy Study were analysed using standard psychometric methods to identify the subset of items which were the most scientifically sound indicators of outcome. The Menorrhagia Outcomes questionnaire is a 26-item questionnaire which covers symptoms, post-operative complications, quality of life, and women's satisfaction with outcome. The questionnaire was field tested for acceptability, reliability and validity by postal survey.

RESULTS

The Menorrhagia Outcomes Questionnaire was found to be highly acceptable to women and showed excellent internal consistency, test-retest reliability, criterion and construct validity.

CONCLUSION

The Menorrhagia Outcomes Questionnaire is a practical and scientifically sound measure of outcome from the woman's perspective following surgical treatment for menorrhagia due to benign disease. It takes less than five minutes to complete, is appropriate for use with different surgical treatments for menorrhagia (eg. hysterectomy, endometrial resection, laser ablation) and is feasible for routine monitoring of large numbers of women by postal survey. Most importantly, this questionnaire has been shown to perform well from a scientific point view, having met standard psychometric criteria for reliability and validity.

Development and validation of an audit instrument: the Prostate Outcomes Questionnaire

OBJECTIVE

To develop a short, patient-based questionnaire for auditing the outcomes of treatment for benign prostatic hyperplasia (BPH) and to evaluate the acceptability, reliability and validity of the measure.

PATIENTS AND METHODS

Data from a long research questionnaire used in the Oxford/North West Thames Prostatectomy Study were analysed to identify the subset of items which contained the most scientifically sound indicators of outcome. Items were selected on the basis of standard psychometric analyses to develop a short questionnaire, the Prostate Outcomes Questionnaire (POQ), a 27-item instrument covering urinary symptoms, complications after surgery, quality of life and patient satisfaction with outcome. The POQ was field tested for acceptability, reliability and validity in a postal survey of 125 men undergoing transurethral resection of the prostate for BPH from seven hospitals in south-east England.

RESULTS

The POQ was highly acceptable to patients, as indicated by a 95% response rate, the absence of 'floor' and 'ceiling' effects and a low proportion of missing data. It showed excellent internal consistency reliability (Cronbach's alpha of 0.82-0.93 for the five summary scales and a mean item-total correlation of 0.56) and test-retest reliability (test-retest correlations for items were 0.40-1.00, with a mean test-retest correlation of 0.85, and for summary scales of 0.91-0.95). Construct validity was confirmed by: high intercorrelations between the five summary scales and the total score (0.88-0.95), with a pattern of intercorrelations among specific subscales that showed item-convergent and discriminant validity; higher scores for patients who reported an improvement after surgery than in those who were not improved (P < 0.001); the expected gradient of scores for patients reporting different levels of symptom distress (P < 0.001); high correlations with the longer parent questionnaire (0.75-0.88); and low to moderate correlations with the Nottingham Health Profile and Activities of Daily Living scores (0.15-0.67), with the pattern of correlations providing evidence of scale convergent and discriminant validity. Correlations with age and social class were low, suggesting that responses were not biased by sociodemographic factors.

CONCLUSION

The POQ is a practical and scientifically sound patient-based measure of outcome after treatment for BPH which can be used for routine audit. It takes < 5 min to complete, is feasible for routine monitoring of large numbers of patients by postal survey, and is accompanied by a Users' Manual which provides practical help in conducting a local patient survey, and a computer program for scoring data. Most importantly, the POQ has been shown scientifically to perform well, having met standard psychometric criteria for reliability and validity.

The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

OBJECTIVE

To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity.

DESIGN

A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non-randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden.

MAIN RESULTS

The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes).

CONCLUSIONS

This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.

Does comorbidity affect the outcome of surgery? Total hip replacement in the UK and Japan

OBJECTIVES

To assess the impact of comorbidity on the outcome of surgery in the UK and in Japan; to determine the predictive ability of a new measure of comorbidity, the Index of Co-Existent Diseases (ICED); and to see if its predictive power could be improved.

DESIGN

Logistic regression using data from two retrospective cohorts with prospective outcome data collection.

SETTING

Six hospitals (three teaching, three non-teaching) in the UK and 15 (12 teaching, three non-teaching) hospitals in Japan.

STUDY PARTICIPANTS

Patients undergoing total hip replacement (THR) surgery in the UK (n = 268) and in Japan (n = 249).

MAIN OUTCOME MEASURES

Serious complications before hospital discharge and change in three measures of general health status [basic activities of daily living (ADL); instrumental ADL; social activities].

RESULTS

The distribution of levels of comorbidity differed between the UK and Japan: none (26.1 versus 42.2%); mild (30.6 versus 43.0%); moderate (23.5 versus 12.0%); and severe (19.8 versus 2.8%). In the UK, the incidence of serious complications was higher in patients with moderate (27.0%) or severe (26.4%) comorbidity than in those with no (14.3%) or mild (13.4%) comorbidity (P<0.001). In contrast, no significant association was found in Japan. The relationship between comorbidity and change in health status was weak in the UK and non-significant in Japan. Logistic regression confirmed that comorbidity was a significant predictor of serious complications in the UK. The only other significant factor was surgical approach (anterior/antero-lateral; odds ratio 2.16, P<0.05). Attempts to improve the predictive power of the ICED by modifying its structure and by reclassifying complications was successful in achieving a linear (rather than dichotomous) relationship. The predictive power, however, was poor.

CONCLUSIONS

Comorbidity is a significant determinant of serious complications following THR but not of changes in functional or health status. Comparisons of clinical performance using post-operative complications must take levels of comorbidity into account if they are to be meaningful. The ICED is of less validity in the UK and Japan than in its country of origin, the USA. Further work to develop better instruments for the UK and Japan is needed.