Assessing the quality of reports of randomized trials in pediatric complementary and alternative medicine

Methodological and statistical characteristics of meta-analyses on spinal cord stimulation for chronic pain: a systematic review

BACKGROUND

A growing number of meta-analyses (MA) have investigated the use of spinal cord stimulation (SCS) as a treatment modality for chronic pain. The quality of these MAs has not been assessed by validated appraisal tools.

OBJECTIVE

To examine the methodological characteristics and quality of MAs related to the use of SCS for chronic pain syndromes.

EVIDENCE REVIEW

An online literature search was conducted in Ovid MEDLINE(R), Ovid EMBASE, Ovid Cochrane Database of Systematic Reviews, and Scopus databases (January 1, 2000 through June 30, 2023) to identify MAs that investigated changes in pain intensity, opioid consumption, and/or physical function after SCS for the treatment of chronic pain. MA quality was assessed using A Measurement Tool to Assess Systematic Reviews (AMSTAR-2) critical appraisal tool.

FINDINGS

Twenty-five MAs were appraised in the final analysis. Three were considered "high" quality, three "low" quality, and 19 "critically low" quality, per the AMSTAR-2 criteria. There was no association between the publication year and AMSTAR-2 overall quality (β 0.043; 95% CI -0.008 to 0.095; p=0.097). There was an association between the impact factor and AMSTAR-2 overall quality (β 0.108; 95% CI 0.044 to 0.172; p=0.002), such that studies published in journals with higher impact factors were associated with higher overall quality. There was no association between the effect size and AMSTAR-2 overall quality (β -0.168; 95% CI -0.518 to 0.183; p=0.320).According to our power analysis, three studies were adequately powered (>80%) to reject the null hypothesis, while the remaining studies were underpowered (<80%).

CONCLUSIONS

The study demonstrates a critically low AMSTAR-2 quality for most MAs published on the use of SCS for treating chronic pain. Future MAs should improve study quality by implementing the AMSTAR-2 checklist items.

PROSPERO REGISTRATION NUMBER

CRD42023431155.

A Critical Review to Identify the Domains Used to Measure the Effect and Outcome of Adaptogenic Herbal Medicines

Background: Phytoadaptogens are considered to be herbal medicines with a multi-target effect that strengthen organ systems compromised by stress. Although animal and laboratory studies have identified numerous molecular targets associated with adaptogenic activity, the non-specific characteristic of these herbal medicines has meant there is no known methods to accurately determine efficacy of adaptogens in humans. This critical review of the evidence aims to identify domains which have been used to measure the effect of adaptogens in humans, in order to create pathways for translating laboratory, animal, and clinical studies on adaptogens into practical applications in the future. Methods: EMBASE, AMED, PubMed, Cochrane Library, and WHO ICTRP databases were searched for randomized trials which examined known physiological actions of adaptogens. Results: Twenty-four studies were identified and critically appraised using the Jadad scale. The findings identified three broad categories of outcome measures, including cognitive, mood and biological measures. Conclusions: There was a great heterogeneity in data making it difficult to draw conclusions as to the most effective measurement tools to capture the holistic activity in humans. Cognitive measures hold promise as a reliable measurement tool when used in conjunction with other relevant tools. Further investigation is necessary to determine the most appropriate and diverse tools to measure the complex multi-target action of adaptogens.

A review of 17 years of application of partnership care model on the consequences of chronic diseases: Describing and assessing the quality of the methodology of papers

Background : Regarding the widespread prevalence of chronic diseases, nurses need to understand the choices, priorities, and abilities of patients in reality, their communication, and the social context in order to play their professional role and responsibility. This review study was conducted with two purposes: determining the effect of partnership-care-model (PCM) on the outcomes of chronic diseases, and describing and assessing the quality of methodology of published papers in the field. Methods : The systematic review was conducted from inception to the year 2019. A systematic search was conducted using PRISMA guidelines to retrieve all national and international PCM studies. To assess the quality of the methodology of studies, four instruments were used, including JADAD, Consort, New Castle-Ottawa, and Cochrane. Results : In the initial search, 11509 papers were retrieved, which according to the PRISMA guidelines, 23 relevant papers remained. The results of the reviewed papers indicated the effectiveness of the PCM. Describing and assessing the quality of the methodology of published papers of PCM was at an acceptable level. The final papers were classified and reviewed according to the authors, year, sampling characteristics, data collection methods, and final PCM conclusion. Conclusion : According to the review of the studies and their effectiveness in the different outcomes, also, the acceptable quality of the methodology of published papers, it can be concluded that the PCM is an effective, context-based, simple, efficient, and reliable model and has the ability to be used in promoting and improving the various dimensions of chronic diseases.

Quality analysis of randomized controlled trials reporting in the field of pediatrics by Indonesian researchers

OBJECTIVE

To determine the quality of reports of the randomized controlled trial (RCT) in the field of pediatrics conducted by Indonesian investigators.

METHODS

All pediatric RCTs conducted by Indonesian researchers were sourced from international and national (local) publications. The assessment was done using both the Consolidated Standards of Reporting Trial (CONSORT) 2010 statement and Jadad Scale. Overall scores from each assessment are reported with a comparison of overall scores between studies in international and local publications.

RESULTS

A total of 91 pediatric randomized control trials by Indonesian authors were gathered. National publications yielded a total of 44 studies (48.4%) whilst international publications yielded 47 studies (51.6%). Using the CONSORT statement the percentage of good reports was 38.3% in international journals and 33.3% in national journals. Using Jadad scale the percentage of good reports was 43.6% (international journals) and 37.0% (national journals). Both were not statistically significant.

CONCLUSIONS

Even though Indonesian investigators still need to be familiarized with good RCT reporting, the overall quality of the reports is fairly satisfactory. There is no significant difference in quality between studies published in international or national journals.

Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update

PURPOSE

Reporting guidelines (eg, Consolidated Standards of Reporting Trials [CONSORT] statement) are intended to improve reporting standards and enhance the transparency and reproducibility of research findings. Despite accessibility of such guidelines, researchers are not required to adhere to them. Our goal was to determine the current status of reporting quality in the medical literature and examine whether adherence of reporting guidelines has improved since the inception of reporting guidelines.

MATERIALS AND METHODS

Eight reporting guidelines, such as CONSORT, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), STrengthening the Reporting of OBservational studies in Epidemiology (STROBE), Quality of Reporting of Meta-analysis (QUOROM), STAndards for Reporting of Diagnostic accuracy (STARD), Animal Research: Reporting In Vivo Experiments (ARRIVE), Consolidated Health Economic Evaluation Reporting Standards (CHEERS), and Meta-analysis of Observational Studies in Epidemiology (MOOSE) were examined. Our inclusion criteria included reviews published between January 1996 to September 2016 which investigated the adherence to reporting guidelines in the literature that addressed clinical trials, systematic reviews, observational studies, meta-analysis, diagnostic accuracy, economic evaluations, and preclinical animal studies that were in English. All reviews were found on Web of Science, Excerpta Medical Database (EMBASE), MEDLINE, and Cumulative Index to Nursing and Allied Health Literature (CINAHL).

RESULTS

Among the general searching of 26,819 studies by using the designed searching method, 124 studies were included post screening. We found that 87.9% of the included studies reported suboptimal adherence to reporting guidelines. Factors associated with poor adherence included non-pharmacological interventions, year of publication, and trials concluding with significant results. Improved adherence was associated with better study designs such as allocation concealment, random sequence, large sample sizes, adequately powered studies, multiple authorships, and being published in journals endorsing guidelines.

CONCLUSION

We conclude that the level of adherence to reporting guidelines remains suboptimal. Endorsement of reporting guidelines by journals is important and recommended.

Reducing waste in pediatric clinical research

The importance of reducing waste and increasing value when conducting research has been emphasized by a series of articles published in the Lancet in 2014. A survey indicates that, one year later, these articles have not influenced how research is conducted. In this review, we explore four stages described by Moher et al. in research production that lead to waste. We show that all four stages including, questions relevant to users, appropriate design conduct and analysis, accessible full research, unbiased and usable reports, efficient research regulation and management of biomedical research are also producing an important waste in pediatric research. We conclude that methods to improve research quality and limit waste need to be implemented in pediatric research and recognized by authorities as a priority.

Trends in the number and the quality of trial protocols involving children submitted to a French Institutional Review Board

Background

There is a great need for high quality clinical research for children. The European Pediatric Regulation aimed to improve the quality of clinical trials in order to increase the availability of treatments for children. The main purpose of this study was to assess the evolution of both the number and the quality of pediatric trial protocols that were submitted to a French Institutional Review Board (IRB00009118) before and after the initiation of the EU Pediatric Regulation.

Methods

All protocols submitted to the IRB00009118 between 2003 and 2014 and conducting research on subjects under eighteen years of age were eligible. The quality of randomized clinical trials was assessed according to the guidelines developed by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network and ranked using the Jadad score.

Results

Out of 622 protocols submitted to the Institutional Review Board (IRB), 21% (133/622) included children. Among these 133 pediatric protocols, the number of submitted pediatric protocols doubled between the two studied periods. From 2003 to 2008, 47 protocols including 21 institutionally sponsored were submitted to the IRB and from 2009 until 2014, 86 protocols including 48 institutionally sponsored were submitted. No significant trend was observed on the quality of RCTs. The overall median score of RCTs on the Jadad scale was high (3.5), 70.0% of protocols had a Jadad score ≥ 3, and 30.0% had a score < 3.

Conclusion

Following the EU Pediatric Regulation, the number of pediatric protocols submitted to the IRB00009118 tends to increase, but no change was noticed regarding their quality.

Quality of reporting in surgical randomized clinical trials

BACKGROUND

RCTs testing surgical interventions can change clinical practice. The adequate reporting of surgical trials is an important issue.

METHODS

A cross-sectional survey was undertaken by searching PubMed for two-arm parallel randomized trials assessing surgical interventions published in 2003 or 2013. Quality of reporting was evaluated against the CONSORT checklist containing 29 items (standard CONSORT plus CONSORT Extension for Trials Assessing Non-Pharmacological Treatments). Univariable and multivariable linear regression was undertaken to explore factors associated with quality of reporting measured with CONSORT scores.

RESULTS

Some 120 trials were identified and included. The mean(s.d.) CONSORT score was 12·7(4·0). Trials published in 2013 achieved a higher CONSORT score than those published in 2003 (mean 14·5(3·8) versus 10·8(3·4) respectively; P < 0·001). The extent to which these trials met the requirement for CONSORT reporting differed substantially across items: four of 29 items were reported adequately across trials, and seven were reported adequately in less than 20 per cent of trials. Less than 40 per cent of the trials described the additional items related to surgical interventions and care providers (such as nursing care and anaesthetic management). In multivariable regression analysis, trials published in 2013 (coefficient 3·05, 95 per cent c.i. 1·89 to 4·20) and multicentre studies (coefficient 2·08, 0·79 to 3·37) were associated with statistically higher quality of reporting.

CONCLUSION

The quality of reporting in surgical trials has improved in the past decade. Overall quality, however, remains suboptimal, particularly in relation to details regarding surgical interventions and management.

Recommendations and evidence for reporting items in pediatric clinical trial protocols and reports: two systematic reviews

BACKGROUND

Complete and transparent reporting of clinical trial protocols and reports ensures that these documents are useful to all stakeholders, that bias is minimized, and that the research is not wasted. However, current studies repeatedly conclude that pediatric trial protocols and reports are not appropriately reported. Guidelines like SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and CONSORT (Consolidated Standards of Reporting Trials) may improve reporting, but do not offer guidance on issues unique to pediatric trials. This paper reports two systematic reviews conducted to build the evidence base for the development of pediatric reporting guideline extensions: 1) SPIRIT-Children (SPIRIT-C) for pediatric trial protocols, and 2) CONSORT-Children (CONSORT-C) for pediatric trial reports.

METHOD

MEDLINE, the Cochrane Methodology Register, and reference lists of included studies were searched. Publications of any type were eligible if they included explicit recommendations or empirical evidence for the reporting of potential items in a pediatric protocol (SPIRIT-C systematic review) or trial report (CONSORT-C systematic review). Study characteristics, recommendations and evidence for pediatric extension items were extracted. Recurrent themes in the recommendations and evidence were identified and synthesized. All steps were conducted by two reviewers.

RESULTS

For the SPIRIT-C and CONSORT-C systematic reviews 366 and 429 publications were included, respectively. Recommendations were identified for 48 of 50 original reporting items and sub-items from SPIRIT, 15 of 20 potential SPIRIT-C reporting items, all 37 original CONSORT items and sub-items, and 16 of 22 potential CONSORT-C reporting items. The following overarching themes of evidence to support or refute the utility of reporting items were identified: transparency; reproducibility; interpretability; usefulness; internal validity; external validity; reporting bias; publication bias; accountability; scientific soundness; and research ethics.

CONCLUSION

These systematic reviews are the first to systematically gather evidence and recommendations for the reporting of specific items in pediatric protocols and trials. They provide useful and translatable evidence on which to build pediatric extensions to the SPIRIT and CONSORT reporting guidelines. The resulting SPIRIT-C and CONSORT-C will provide guidance to the authors of pediatric protocols and reports, respectively, helping to alleviate concerns of inappropriate and inconsistent reporting, and reduce research waste.

The Role of Salivary Neuropeptides in Pediatrics: Potential Biomarkers for Integrated Therapies

INTRODUCTION

Objective measures of symptom response to integrated complementary approaches in pediatrics are evolving. The purpose of this study was to document the concentration range of salivary neuropeptides in healthy controls and in children with cancer, to explore correlations between serum and salivary measurements for Calcitonin Gene-Related Peptide (CGRP) and Vasoactive Intestinal Polypeptide (VIP), and to determine whether there is a change in these salivary neuropeptide levels in response to integrated mind-body therapies.

METHODS

A non-randomized pragmatic study with three phases: Phase 1- Healthy Control Saliva-10 healthy controls provided saliva samples; Phase 2- Cancer Diagnosis Serum-Saliva- 16 mixed-type cancer patients provided blood and saliva samples; Phase 3- Acute Lymphocytic Leukemia (ALL) Saliva Intervention- 12 patients with ALL provided pre- and post-complementary intervention saliva samples.

INTERVENTIONS

20-minutes of structured touch or scripted relaxation breathing were administered to patients in Phase 3; Phase 1 and 2 patients did not receive this intervention.

OUTCOME MEASURES

cortisol, CGRP, VIP, State/Trait Anxiety Scale, visual analogue scale, vital signs.

RESULTS

Salivary CGRP and VIP were similar for children in Phases 1 and 2. There was a correlation between serum and salivary VIP in the mixed cancer group, though not between serum and salivary CGRP. In Phase 3 children, following a complementary intervention, salivary CGRP, heart rate, and systolic blood pressure decreased.

DISCUSSION/CONCLUSIONS

These data provide evidence of a decrease in sympathetic output after integrative/complementary therapy intervention in children with cancer. The study underscores the potential role of salivary neuropeptides as non-invasive biomarkers for integrated therapies in pediatrics.

Complementary and alternative medicine use by pediatric specialty outpatients

OBJECTIVE

Complementary and alternative medicine (CAM) use is high among children and youth with chronic illnesses. The objective of this study was to assess the prevalence and patterns of CAM use in 10 subspecialty clinics in Canada and to compare CAM use between 2 geographically diverse locations.

METHODS

This survey was carried out at 1 Children's Hospital in western Canada (Edmonton) and 1 Children's Hospital in central Canada (Ottawa). Questionnaires were completed by parents in either French or English.

RESULTS

Although demographic characteristics of the 2 populations were similar, CAM use at the western hospital was 71% (n = 704) compared with 42% (n = 222) at the central hospital (P < .0001). Most respondents agreed or strongly agreed that they feel comfortable discussing CAM in their clinic. The most common CAM products currently used were multivitamins/minerals, herbal products, and homeopathic remedies. The most common CAM practices currently used were massage, chiropractic, relaxation, and aromatherapy. Eighty adverse effects were reported, and 55 (68.8%) of these were self-assessed as minor.

CONCLUSIONS

Results of this study indicate that CAM use is high among pediatric specialty clinic outpatients and is much greater in the western than in the central hospital. Most respondents felt that their CAM use was helpful with few or no harms associated. Many patients, using CAM alongside their conventional medicines, are still not discussing their CAM use with their physicians and are increasing the likelihood for potential interactions and preventable harms.

The efficacy of auditory perceptual training for tinnitus: a systematic review

Auditory perceptual training affects neural plasticity and so represents a potential strategy for tinnitus management. We assessed the effects of auditory perceptual training on tinnitus perception and/or its intrusiveness via a systematic review of published literature. An electronic database search using the keywords ‘tinnitus and learning’ or ‘tinnitus and training’ was conducted, updated by a hand search. The ten studies identified were reviewed independently by two reviewers, data were extracted, study quality was assessed according to a number of specific criteria and the information was synthesised using a narrative approach. Nine out of the ten studies reported some significant change in either self-reported or psychoacoustic outcome measures after auditory training. However, all studies were quality rated as providing low or moderate levels of evidence for an effect. We identify a need for appropriately randomised and controlled studies that will generate high-quality unbiased and generalisable evidence to ascertain whether or not auditory perceptual training has a clinically relevant effect on tinnitus.

Systematic review of randomized controlled trial quality in pediatric kidney transplantation

Limited pediatric-specific research can lead to sub-standard evidence for clinical decision making in children. We sought to systematically evaluate the methodological quality and the reporting standards of randomized controlled trials (RCTs) of transplantation trials in children. We included RCTs of kidney transplant recipients that had enrolled at least one child (aged 17 years or less) and that were reported in English language, peer reviewed journals from 2000 onward in the Cochrane Renal Group's specialized register. Trial reports were assessed against the 22 item checklist of the CONsolidated Standards Of Reporting Trials (CONSORT) statement. Twenty-seven RCTs were included. The reporting of the essential components of the methods, results and discussion domains was unsatisfactory. Mean CONSORT criteria score for the pediatric trials was 67% and 66% for trials including both adults and children (p value for the difference = 1.00). Trial reporting quality in pediatric transplantation trials is not different from trials involving adults. It is evident that the reporting standards of RCTs in both adult and pediatric transplantation require major improvements. This work bench-marks current standards for future quality improvement.

Chiropractic care for paediatric and adolescent Attention-Deficit/Hyperactivity Disorder: A systematic review

BACKGROUND

Psychostimulants are first line of therapy for paediatric and adolescent AD/HD. The evidence suggests that up to 30% of those prescribed stimulant medications do not show clinically significant outcomes. In addition, many children and adolescents experience side-effects from these medications. As a result, parents are seeking alternate interventions for their children. Complementary and alternative medicine therapies for behavioural disorders such as AD/HD are increasing with as many as 68% of parents having sought help from alternative practitioners, including chiropractors.

OBJECTIVE

The review seeks to answer the question of whether chiropractic care can reduce symptoms of inattention, impulsivity and hyperactivity for paediatric and adolescent AD/HD.

METHODS

Electronic databases (Cochrane CENTRAL register of Controlled Trials, Cochrane Database of Systematic reviews, MEDLINE, PsycINFO, CINAHL, Scopus, ISI Web of Science, Index to Chiropractic Literature) were searched from inception until July 2009 for English language studies for chiropractic care and AD/HD. Inclusion and exclusion criteria were applied to select studies. All randomised controlled trials were evaluated using the Jadad score and a checklist developed from the CONSORT (Consolidated Standards of Reporting Trials) guidelines.

RESULTS

The search yielded 58 citations of which 22 were intervention studies. Of these, only three studies were identified for paediatric and adolescent AD/HD cohorts. The methodological quality was poor and none of the studies qualified using inclusion criteria.

CONCLUSIONS

To date there is insufficient evidence to evaluate the efficacy of chiropractic care for paediatric and adolescent AD/HD. The claim that chiropractic care improves paediatric and adolescent AD/HD, is only supported by low levels of scientific evidence. In the interest of paediatric and adolescent health, if chiropractic care for AD/HD is to continue, more rigorous scientific research needs to be undertaken to examine the efficacy and effectiveness of chiropractic treatment. Adequately-sized RCTs using clinically relevant outcomes and standardised measures to examine the effectiveness of chiropractic care verses no-treatment/placebo control or standard care (pharmacological and psychosocial care) are needed to determine whether chiropractic care is an effective alternative intervention for paediatric and adolescent AD/HD.

Reporting quality and information consistency of randomized, controlled trials presented as abstracts at the American Urological Association annual meetings

PURPOSE

We assessed the quality of randomized, controlled trial reporting in abstracts from the annual meetings of the American Urological Association and determined whether the information provided is consistent with subsequent full text publications.

MATERIALS AND METHODS

All randomized, controlled trials presented in abstract form at the 2002 and 2003 American Urological Association annual meetings were identified for review. A systematic PubMed search based on authorship and key words from the study title was done to identify all subsequent full text publications. A standardized evaluation form was developed based on the published literature, pilot tested in a separate sample and applied by 2 independent reviewers.

RESULTS

A total of 126 randomized, controlled trials were identified for review, including 56 in 2002 and 70 in 2003. Approximately a third of the trials (43 or 34.1%) identified the study design as a randomized, controlled trial in the abstract title. The method of randomization, allocation concealment and blinding was reported in 0% (0), 0% (0) and 40.5% (51) of studies, respectively. Mean/median followup was provided in 27.0% of studies (34). Of 126 randomized, controlled trials presented in abstract form 62.7% (79) were subsequently published as full text articles. Study sample size and the number of randomized subjects differed in 24.1% and 28.9% of abstracts, respectively. From the small proportion of randomized, controlled trials (23 or 29.1%) that identified a single primary end point results differed in 9 of 23 (39.1%).

CONCLUSIONS

Most abstracts fail to provide the necessary information to assess methodological quality. Organizers of urological meetings should consider implementing a more structured abstract format that requires authors to provide the necessary study details, thereby allowing urologists to critically appraise study validity.

Quality of surgical randomized controlled trials for acute cholecystitis: assessment based on CONSORT and additional check items

BACKGROUND/PURPOSE

In this study, we conducted a limited survey of reports of surgical randomized controlled trials, using the consolidated standards of reporting trials (CONSORT) statement and additional check items to clarify problems in the evaluation of surgical reports.

METHODS

A total of 13 randomized trials were selected from two latest review articles on biliary surgery. Each randomized trial was evaluated according to 28 quality measures that comprised items from the CONSORT statement plus additional items. Analysis focused on relationships between the quality of each study and the estimated effect gap ("pooled estimate in meta-analysis" -- "estimated effect of each study").

RESULTS

No definite relationships were found between individual study quality and the estimated effect gap. The following items could have been described but were not provided in almost all the surgical RCT reports: "clearly defined outcomes"; "details of randomization"; "participant flow charts"; "intention-to-treat analysis"; "ancillary analyses"; and "financial conflicts of interest". The item, "participation of a trial methodologist in the study" was not found in any of the reports.

CONCLUSIONS

Although the quality of reporting trials is not always related to a biased estimation of treatment effect, the items used for quality measures must be described to enable readers to evaluate the quality and applicability of the reporting. Further development of an assessment tool is needed for items specific to surgical randomized controlled trials.

A systematic evaluation of the impact of STRICTA and CONSORT recommendations on quality of reporting for acupuncture trials

BACKGROUND

We investigated whether there had been an improvement in quality of reporting for randomised controlled trials of acupuncture since the publication of the STRICTA and CONSORT statements. We conducted a before-and-after study, comparing ratings for quality of reporting following the publication of both STRICTA and CONSORT recommendations.

METHODOLOGY AND PRINCIPAL FINDINGS

Ninety peer reviewed journal articles reporting the results of acupuncture trials were selected at random from a wider sample frame of 266 papers. Papers published in three distinct time periods (1994-1995, 1999-2000 and 2004-2005) were compared. Assessment criteria were developed directly from CONSORT and STRICTA checklists. Papers were independently assessed for quality of reporting by two assessors, one of whom was blind to information which could have introduced systematic bias (e.g. date of publication). We detected a statistically significant increase in the reporting of CONSORT items for papers published in each time period measured. We did not, however, find a difference between the number of STRICTA items reported in journal articles published before and 3 to 4 years following the introduction of STRICTA recommendations.

CONCLUSIONS AND SIGNIFICANCE

The results of this study suggest that general standards of reporting for acupuncture trials have significantly improved since the introduction of the CONSORT statement in 1996, but that quality in reporting details specific to acupuncture interventions has yet to change following the more recent introduction of STRICTA recommendations. Wider targeting and revision of the guidelines is recommended.

Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity

BACKGROUND

It is often stated that external validity is not sufficiently considered in the assessment of clinical studies. Although tools for its evaluation have been established, there is a lack of awareness of their significance and application. In this article, a comprehensive checklist is presented addressing these relevant criteria.

METHODS

The checklist was developed by listing the most commonly used assessment criteria for clinical studies. Additionally, specific lists for individual applications were included. The categories of biases of internal validity (selection, performance, attrition and detection bias) correspond to structural, treatment-related and observational differences between the test and control groups. Analogously, we have extended these categories to address external validity and model validity, regarding similarity between the study population/conditions and the general population/conditions related to structure, treatment and observation.

RESULTS

A checklist is presented, in which the evaluation criteria concerning external validity and model validity are systemized and transformed into a questionnaire format.

CONCLUSION

The checklist presented in this article can be applied to both planning and evaluating of clinical studies. We encourage the prospective user to modify the checklists according to the respective application and research question. The higher expenditure needed for the evaluation of clinical studies in systematic reviews is justified, particularly in the light of the influential nature of their conclusions on therapeutic decisions and the creation of clinical guidelines.

Quality of reporting of randomized controlled trials of herbal medicine interventions

BACKGROUND

Public interest in herbal medicines has generated an increasing number of trials evaluating their efficacy. Trials with poor methodologic quality have exaggerated estimates of treatment effect, and incomplete reporting of trials causes difficulties in assessing trial methodologic quality. The objective of this project was to examine the quality of reporting of randomized controlled intervention trials of herbal medicine.

METHODS

MEDLINE (1966 to September 2003) was searched for randomized controlled trials of 10 herbal medicines. Two individuals (J. G. and J. D.) independently assessed trials using the Consolidated Standard of Reporting Trials checklist. Disagreements were resolved by consensus. The mean number of checklist items reported across all and for individual herbal medicines was calculated. The influence of decade of publication and species of herbal medicine tested was explored using an analysis of variance.

RESULTS

A total of 206 randomized controlled trials of herbal medicine were included. Interrater reliability on reporting quality assessment was high. A total of 45% of items were reported across all trials. The quality of reporting improved across decades from the 1970s to the 2000s. Individual herbal species differed in the total number of items reported, with echinacea, ginkgo, St. John's wort, and kava trials reporting the most items.

CONCLUSIONS

Important methodologic components of randomized controlled trials of herbal medicines are incompletely reported including allocation concealment, method used to generate the allocation sequence, and whether an intention-to-treat analysis was used. Also, key information unique to these trials may be missing, such as percentage of active constituents and type or form of the herbal medicine preparation. We suggest trialists consult a recent extension of the Consolidated Standard of Reporting Trials statement specific to herbal medicine trials when designing and reporting randomized controlled intervention trials of herbal medicines.

The Cochrane Fertility Regulation Group: synthesizing the best evidence about family planning

The Fertility Regulation Group of the Cochrane Collaboration has been assessing the best available evidence on fertility regulation, family size and birth spacing. By the end of 2005, this group had published 32 systematic reviews and 12 protocols; most reviews were on contraception. Because of suboptimal trial quality, firm conclusions could be made in only five reviews. Threats to internal validity in published trials include the absence of description of allocation concealment, intentional exclusion of participants after randomization, failure to use intention-to-treat analyses and lack of treatment blinding. The precision of results has been limited by small sample sizes. The finding that most trials of oral contraceptives were conducted by pharmaceutical companies raises concerns about potential commercial bias. Of necessity, most information about fertility regulation effectiveness and adverse effects comes from observational studies, which vary widely in quality. Systematic reviews of evidence, with an emphasis on randomized controlled trials when available, will continue to improve fertility regulation in the years ahead.

The efficacy and safety of exogenous melatonin for primary sleep disorders. A meta-analysis

BACKGROUND

Exogenous melatonin has been increasingly used in the management of sleep disorders.

PURPOSE

To conduct a systematic review of the efficacy and safety of exogenous melatonin in the management of primary sleep disorders.

DATA SOURCES

A number of electronic databases were searched. We reviewed the bibliographies of included studies and relevant reviews and conducted hand-searching.

STUDY SELECTION

Randomized controlled trials (RCTs) were eligible for the efficacy review, and controlled trials were eligible for the safety review.

DATA EXTRACTION

One reviewer extracted data, while the other verified data extracted. The Random Effects Model was used to analyze data.

DATA SYNTHESIS

Melatonin decreased sleep onset latency (weighted mean difference [WMD]: -11.7 minutes; 95% confidence interval [CI]: -18.2, -5.2)); it was decreased to a greater extent in people with delayed sleep phase syndrome (WMD: -38.8 minutes; 95% CI: -50.3, -27.3; n=2) compared with people with insomnia (WMD: -7.2 minutes; 95% CI: -12.0, -2.4; n=12). The former result appears to be clinically important. There was no evidence of adverse effects of melatonin.

CONCLUSIONS

There is evidence to suggest that melatonin is not effective in treating most primary sleep disorders with short-term use (4 weeks or less); however, additional large-scale RCTs are needed before firm conclusions can be drawn. There is some evidence to suggest that melatonin is effective in treating delayed sleep phase syndrome with short-term use. There is evidence to suggest that melatonin is safe with short-term use (3 months or less).

For randomized controlled trials, the quality of reports of complementary and alternative medicine was as good as reports of conventional medicine

OBJECTIVE

To compare the quality of reporting of reports randomized controlled trials (RCTs) published in English and in languages other than English (LOE), and to determine whether there were differences between conventional medicine (CM) and complementary and alternative medicine (CAM) reports.

STUDY DESIGN AND SETTING

We examined more than 600 RCTs associated with 125 systematic reviews. We extracted characteristics of each RCT using a standardized data collection form. We assessed quality using the Jadad scale and the adequacy of allocation concealment.

RESULTS

There were only minor differences in the quality of reports of RCTs published in English compared with other languages (median quality score of 3 vs. 2, P=.10), and the quality of reports of CAM RCTs was similar to the CM reports (median score of 3 vs. 2, P=.14). There was no effect of language of publication on quality of reporting for CM trials (median score of 2 vs. 2, P=.12). Among CAM trials, however, overall quality scores were higher for reports in English than for reports in other languages (median score of 3 vs. 2, P=.04).

CONCLUSION

The overall quality of reports published in languages other than English is similar to that of English-language reports. Moreover, the overall quality of reporting of RCTs of CAM interventions is as good as that for CM interventions.

Complementary and alternative medicine in Canadian children: A call for action

Complementary and alternative medicine (CAM) use, including paediatric use, is common. The Hospital for Sick Children Foundation organized a priority-setting forum regarding paediatrics and the use of CAM/natural health products in November 2001. Four priority areas were identified: the creation of a national paediatric research network and/or centre of excellence in CAM; support for experiential learning, education, and training; completion of a needs/use assessment; and facilitation of knowledge transfer. These priority areas are discussed in the context of research, education, knowledge transfer and health systems. The present article represents a call for action for Canadian funding agencies to support the development of an evidence base in paediatric CAM.

NCCAM support for the Cochrane Collaboration CAM Field

Americans use CAM therapies now more than ever before, yet there is a shortage of accessible high-quality evidence from clinical trials to show which CAM therapies work best and for which conditions. The need for such evidence is great, not only among people with serious, chronic health conditions, but also among healthcare providers, who often require clinical trial evidence before recommending a medical treatment. For the last few years, the University of Maryland Center for Integrative Medicine has been working steadily to meet this need for information by identifying published clinical trials of alternative therapies, summarizing these into rigorous scientific reviews, and disseminating these reviews to the general public and to healthcare providers. In May 2003, the Center was awarded a large 4-year research grant from the National Institutes of Health's National Center for Complementary and Alternative Medicine (NCCAM) to support and extend this important work.

Larger effect sizes were associated with higher quality ratings in complementary and alternative medicine randomized controlled trials

OBJECTIVE

To determine if the quality of complementary and alternative medicine (CAM) placebo controlled randomized controlled trials (RCTs) is related to the effect sizes they produce when the type of interventions and outcome variables are controlled.

STUDY DESIGN AND SETTING

A sample of 25 CAM meta-analyses was identified from MEDLINE and hand searches containing published effect sizes for at least three efficacy trials employing placebo control groups. From these 25 reviews, 26 pairs of trials were selected: the one reporting the largest effect size and the matching trial reporting the smallest effect size. Quality and publication characteristics were then abstracted from each trial.

RESULTS

Unlike the preponderance of past evidence examining the relationship between quality and effect size, the present study found that trials possessing the largest effect sizes within a meta-analysis were also associated with higher quality ratings than their counterparts possessing the lowest effect sizes (P=.019).

CONCLUSIONS

Possible theoretical reasons for this unexpected positive relationship include (a) sampling error, (b) reduced within group variation, (c) fraudulent reporting, and (d) the restriction of the analyses to placebo controlled trial.

A clarification of critical appraisal, evidence, and education

This is a revision of an earlier draft letter which was published erroneously. The author was provided with this revised letter and chose to not respond further.

Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found?

BACKGROUND

The safety and effectiveness of CAM interventions are of great relevance to pediatric health care providers. The objective of this study is to identify sources of reported randomized controlled trials (RCTs) in the field of pediatric complementary and alternative medicine (CAM).

METHODS

Reports of RCTs were identified by searching Medline and 12 additional bibliographic databases and by reviewing the reference lists of previously identified pediatric CAM systematic reviews.

RESULTS

We identified 908 reports of RCTs that included children under 18 and investigated a CAM therapy. Since 1965, there has been a steady growth in the number of these trials that are being published. The four journals that published the most reported RCTs are The American Journal of Clinical Nutrition, Pediatrics, Journal of Pediatrics, and Lancet. Medline, CAB Health, and Embase were the best database sources for identifying these studies; they indexed 93.2%, 58.4% and 42.2 % respectively of the journals publishing reports of pediatric CAM RCTs.

CONCLUSIONS

Those working or interested in the field of pediatric CAM should routinely search Medline, CAB Health and Embase for literature in the field. The four core journals identified above should be included in their collection.

Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine

OBJECTIVE

To examine the quality of reports of complementary and alternative medicine (CAM) systematic reviews in the pediatric population. We also examined whether there were differences in the quality of reports of a subset of CAM reviews compared to reviews using conventional interventions.

METHODS

We assessed the quality of reports of 47 CAM systematic reviews and 19 reviews evaluating a conventional intervention. The quality of each report was assessed using a validated 10-point scale.

RESULTS

Authors were particularly good at reporting: eligibility criteria for including primary studies, combining the primary studies for quantitative analysis appropriately, and basing their conclusions on the data included in the review. Reviewers were weak in reporting: how they avoided bias in the selection of primary studies, and how they evaluated the validity of the primary studies. Overall the reports achieved 43% (median = 3) of their maximum possible total score. The overall quality of reporting was similar for CAM reviews and conventional therapy ones.

CONCLUSIONS

Evidence based health care continues to make important contributions to the well being of children. To ensure the pediatric community can maximize the potential use of these interventions, it is important to ensure that systematic reviews are conducted and reported at the highest possible quality. Such reviews will be of benefit to a broad spectrum of interested stakeholders.