Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews

Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials

BACKGROUND

Homeopathy seems scientifically implausible, but has widespread use. We aimed to assess whether the clinical effect reported in randomised controlled trials of homeopathic remedies is equivalent to that reported for placebo.

METHODS

We sought studies from computerised bibliographies and contracts with researchers, institutions, manufacturers, individual collectors, homeopathic conference proceedings, and books. We included all languages. Double-blind and/or randomised placebo-controlled trials of clinical conditions were considered. Our review of 185 trials identified 119 that met the inclusion criteria. 89 had adequate data for meta-analysis, and two sets of trial were used to assess reproducibility. Two reviewers assessed study quality with two scales and extracted data for information on clinical condition, homeopathy type, dilution, "remedy", population, and outcomes.

FINDINGS

The combined odds ratio for the 89 studies entered into the main meta-analysis was 2.45 (95% CI 2.05, 2.93) in favour of homeopathy. The odds ratio for the 26 good-quality studies was 1.66 (1.33, 2.08), and that corrected for publication bias was 1.78 (1.03, 3.10). Four studies on the effects of a single remedy on seasonal allergies had a pooled odds ratio for ocular symptoms at 4 weeks of 2.03 (1.51, 2.74). Five studies on postoperative ileus had a pooled mean effect-size-difference of -0.22 standard deviations (95% CI -0.36, -0.09) for flatus, and -0.18 SDs (-0.33, -0.03) for stool (both p < 0.05).

INTERPRETATION

The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo. However, we found insufficient evidence from these studies that homeopathy is clearly efficacious for any single clinical condition. Further research on homeopathy is warranted provided it is rigorous and systematic.

Language bias in randomised controlled trials published in English and German

BACKGROUND

Some randomised controlled trials (RCTs) done in German-speaking Europe are published in international English-language journals and others in national German-language journals. We assessed whether authors are more likely to report trials with statistically significant results in English than in German.

METHODS

We studied pairs of RCT reports, matched for first author and time of publication, with one report published in German and the other in English. Pairs were identified from reports round in a manual search of five leading German-language journals and from reports published by the same authors in English found on Medline. Quality of methods and reporting were assessed with two different scales by two investigators who were unaware of authors' identities, affiliations, and other characteristics of trial reports. Main study endpoints were selected by two investigators who were unaware of trial results. Our main outcome was the number of pairs of studies in which the levels of significance (shown by p values) were discordant.

FINDINGS

62 eligible pairs of reports were identified but 19 (31%) were excluded because they were duplicate publications. A further three pairs (5%) were excluded because no p values were given. The remaining 40 pairs were analysed. Design characteristics and quality features were similar for reports in both languages. Only 35% of German-language articles, compared with 62% of English-language articles, reported significant (p < 0.05) differences in the main endpoint between study and control groups (p = 0.002 by McNemar's test). Logistic regression showed that the only characteristic that predicted publication in an English-language journal was a significant result. The odds ratio for publication of trials with significant results in English was 3.75 (95% CI 1.25-11.3).

INTERPRETATION

Authors were more likely to publish RCTs in an English-language journal if the results were statistically significant. English language bias may, therefore, be introduced in reviews and meta-analyses if they include only trials reported in English. The effort of the Cochrane Collaboration to identify as many controlled trials as possible, through the manual search of many medical journals published in different languages will help to reduce such bias.

Meta-analysis and endocrinology

It is likely that more studies that use meta-analysis will be published in the endocrinologic literature. Major strengths of meta-analysis are the systematic ascertainment of research on a given topic and the explicit delineation of reasons for accepting or rejecting studies as a basis for drawing conclusions. The tendency of meta-analysis to focus on a single estimate of effect and to ignore heterogeneity are problems both with the conduct of meta-analysis and the way in which it is interpreted. Meta-analysis cannot overcome bias in the original studies. It is difficult to perform a good meta-analysis and easy to perform a bad one. The critical reader should not be overawed by the results of a meta-analysis. Reading a meta-analysis should not substitute for careful reading of the primary studies on which the meta-analysis is based. Meta-analysis should not be used to stifle the conduct of original research.

The French clinical guidelines and medical references programme: development of 48 guidelines for private practice over a period of 18 months

The French medical profession and health insurance organizations have jointly committed themselves to a concept termed the 'medical regulation' of care. They decided to promote the quality of health care, judging that an approach based on quality was the best option for reducing the increase in health costs. The Clinical Guidelines and Medical References programme was entrusted to ANDEM (Agence Nationale pour le Développement de L'Evaluation Médicale). Fifty working groups (669 experts) and 50 reading groups (1643 experts) met from June 1994 to November 1995 to produce guidelines. Learned societies were involved to propose experts. Hospital practitioners, doctors who specialized in the topic in question and those who did not had equal representation in the groups. The method consisted of a review of the literature to determine the level of scientific evidence. ANDEM's Scientific Council suggested modifications to the groups and agreed to disseminate 48 of the 50 texts. Careful observation of the operation of the groups identified factors that can positively influence the nature of the discussions and help avoid conflict: an abundance of high-quality literature, an understanding of clinical research methodology, the existence of guidelines from different institutions or different countries tending towards the same conclusions, good initial work carried out by the chairperson and the project manager, a limit to the number of questions asked of the group, the chairperson having good people skills and meeting-management skills, and an absence of professional and financial consequences for the participants. Good management of working groups is an additional factor in ensuring success. The regulatory medical references programme has led to changes of behaviour within the medical profession.