Should systematic reviews include non-randomized and uncontrolled studies? The case of acupuncture for chronic headache

We aimed to investigate: (1) whether patient and intervention characteristics, design-independent quality aspects, and response rates differ between randomized and non-randomized trials of acupuncture for chronic headache; (2) whether non-randomized studies provide useful additional information (regarding long-term effects, prognostic factors, adverse effects, and generalizability); (3) reasons for potential differences in response rates. Studies including at least five patients and reporting clinical outcome data were identified through searches in Medline, Embase, the Cochrane Controlled Trials Register, other databases and checking of bibliographies. Twenty-four randomized trials and 35 non-randomized studies (five non-randomized controlled cohort studies, 10 prospective uncontrolled studies, 10 case series, and 10 cross-sectional surveys) met the inclusion criteria. Studies were heterogeneous regarding patients, interventions, outcome measurements and results. On average, randomized trials had smaller sample sizes, met more quality criteria, and had lower response rates (0.59 [95% confidence interval 0.48-0.69] vs. 0.78 [0.72-0.83]). Whether randomized or not, studies meeting more quality criteria had lower response rates. Non-randomized studies did not have significantly longer follow-up periods, three included an analysis of prognostic variables, only one reported on adverse effects, and the degree of generalizability was unclear. In the case of acupuncture for chronic headache, non-randomized studies confirmed the finding of a systematic review of randomized trials that the treatment is likely to be effective but provided little relevant additional information on long-term effects, prognostic factors, and adverse effects.