Content of reports on clinical trials: a critical review

Current Research and Development in Hyperthermic Intraperitoneal Chemotherapy (HIPEC)-A Cross-Sectional Analysis of Clinical Trials Registered on ClinicalTrials.gov

INTRODUCTION

Over the past two decades, cytoreductive surgery and HIPEC has improved outcomes for selected patients with peritoneal metastasis from various origins. This is a cross-sectional study with descriptive analyses of HIPEC trials registered on ClinicalTrials.gov. This study aimed to characterize clinical trials on HIPEC registered on ClinicalTrials.gov with the primary objective of identifying a trial focus and to examine whether trial results were published.

METHODS

The search included trials registered from 1 January 2001 to 14 March 2022. We examined the associations of exposure variables and other trial features with two primary outcomes: therapeutic focus and results reporting.

RESULTS

In total, 234 clinical trials were identified; 26 (11%) were already published, and 15 (6%) trials have reported their results but have not been published as full papers. Among ongoing nonpublished trials, 81 (39%) were randomized, 30 (14%) were blinded, n = 39 (20%) were later phase trials (i.e., phases 3 and 4), n = 152 (73%) were from a single institution, and 91 (44%) had parallel groups. Most of the trials were recruiting at the time of this analysis (75, 36%), and 39 (20%) were completed but had yet to publish results. In total, 68% of the trials focused on treatment strategies, and 53% investigated the oncological outcome. The most studied neoplasms for HIPEC trials were peritoneally metastasized colorectal cancer (32%), gastric cancer (29%), and ovarian cancer (26%). Twenty different drugs were analyzed in these clinical trials.

CONCLUSIONS

Many study results are awaited from ongoing HIPEC trials. Most HIPEC trials focused on gastric, colorectal, or ovarian cancer. Many clinical trials were identified involving multiple entities and chemotherapeutic agents.

Clustering by Health Professionals in Individually Randomised Controlled Trials

Purpose: The aim of this study was to investigate the prevalence of clustering by health professionals in individually randomised controlled trials (iRCT), and its adjustment in both the sample size calculation estimates and the analysis of the data collected in iRCT (that is, trials that randomise individuals only). As a result, cluster randomised controlled trials will not be the part of this review study. Additionally, the authors aimed to discover the prevalence of the various forms of clustering in iRCT. Methods: iRCT, in which the intervention was delivered by a health professional, were electronically searched in three medical journals. The dates searched were from 1st January 2000–31st August 2009. The retrieved trials were then screened to exclude those with complex designs and trials with more than two parallel arms. The selected trials were then fully reviewed for the presence of clustering effects and any corresponding adjustment. Data about the sample size calculation in the selected trials were also included. A basic form was generated for the purpose of data extraction from each of the selected trials. Results: Of the 130 iRCT reviewed, clustering of outcomes was present in 127 (98%) trials. Only 61 trials (47%) had adjusted for the clustering effects in their design and analysis, while 53% of the trials had ignored the clustering effect, and hence no adjustment had been made in the trial design or analysis. Regarding the various forms of clustering, clustering by centre in multicentre trials was found in 79 trials (60%), followed by natural clustering in 26 trials (20%), and clustering imposed by the design of the study in 23 trials (18%). Conclusion: Potential clustering of outcomes exists in almost all iRCT; however, this review found that <50% of iRCT took clustering into account and adjusted the sample size calculation and statistical analysis of this data for clustering. Almost half of the reviewed iRCT ignored the clustering effect. As a result, inaccurate and nongeneralisable results could have been generated.

Nonadherence to treatment protocol in published randomised controlled trials: a review

This review aimed to ascertain the extent to which nonadherence to treatment protocol is reported and addressed in a cohort of published analyses of randomised controlled trials (RCTs). One hundred publications of RCTs, randomly selected from those published in BMJ, New England Journal of Medicine, the Journal of the American Medical Association and The Lancet during 2008, were reviewed to determine the extent and nature of reported nonadherence to treatment protocol, and whether statistical methods were used to examine the effect of such nonadherence on both benefit and harms analyses. We also assessed the quality of trial reporting of treatment protocol nonadherence and the quality of reporting of the statistical analysis methods used to investigate such nonadherence. Nonadherence to treatment protocol was reported in 98 of the 100 trials, but reporting on such nonadherence was often vague or incomplete. Forty-two publications did not state how many participants started their randomised treatment. Reporting of treatment initiation and completeness was judged to be inadequate in 64% of trials with short-term interventions and 89% of trials with long-term interventions. More than half (51) of the 98 trials with treatment protocol nonadherence implemented some statistical method to address this issue, most commonly based on per protocol analysis (46) but often labelled as intention to treat (ITT) or modified ITT (23 analyses in 22 trials). The composition of analysis sets for their benefit outcomes were not explained in 57% of trials, and 62% of trials that presented harms analyses did not define harms analysis populations. The majority of defined harms analysis populations (18 out of 26 trials, 69%) were based on actual treatment received, while the majority of trials with undefined harms analysis populations (31 out of 43 trials, 72%) appeared to analyse harms using the ITT approach. Adherence to randomised intervention is poorly considered in the reporting and analysis of published RCTs. The majority of trials are subject to various forms of nonadherence to treatment protocol, and though trialists deal with this nonadherence using a variety of statistical methods and analysis populations, they rarely consider the potential for bias introduced. There is a need for increased awareness of more appropriate causal methods to adjust for departures from treatment protocol, as well as guidance on the appropriate analysis population to use for harms outcomes in the presence of such nonadherence.

Achieving high research reporting quality through the use of computational ontologies

Systematic reviews and meta-analyses constitute one of the central pillars of evidence-based medicine. However, clinical trials are poorly reported which delays meta-analyses and consequently the translation of clinical research findings to clinical practice. We propose a Center of Excellence in Research Reporting in Neurosurgery (CERR-N) and the creation of a clinically significant computational ontology to encode Randomized Controlled Trials (RCT) studies in neurosurgery. A 128 element strong computational ontology was derived from the Trial Bank ontology by omitting classes which were not required to perform meta-analysis. Three researchers from our team tagged five randomly selected RCT's each, published in the last 5 years (2004-2008), in the Journal of Neurosurgery (JoN), Neurosurgery Journal (NJ) and Journal of Neurotrauma (JoNT). We evaluated inter and intra observer reliability for the ontology using percent agreement and kappa coefficient. The inter-observer agreement was 76.4%, 75.97% and 74.9% and intra-observer agreement was 89.8%, 80.8% and 86.56% for JoN, NJ and JoNT respectively. The inter-observer kappa coefficient was 0.60, 0.54 and 0.53 and the intra-observer kappa coefficient was 0.79, 0.82 and 0.79 for JoN, NJ and JoNT journals respectively. The high degree of inter and intra-observer agreement confirms tagging consistency in sections of a given scientific manuscript. Standardizing reporting for neurosurgery articles can be reliably achieved through the integration of a computational ontology within the context of a CERR-N. This approach holds potential for the overall improvement in the quality of reporting of RCTs in neurosurgery, ultimately streamlining the translation of clinical research findings to improvement in patient care.

Reporting of sample size calculation in randomised controlled trials: review

OBJECTIVES

To assess quality of reporting of sample size calculation, ascertain accuracy of calculations, and determine the relevance of assumptions made when calculating sample size in randomised controlled trials.

DESIGN

Review.

DATA SOURCES

We searched MEDLINE for all primary reports of two arm parallel group randomised controlled trials of superiority with a single primary outcome published in six high impact factor general medical journals between 1 January 2005 and 31 December 2006. All extra material related to design of trials (other articles, online material, online trial registration) was systematically assessed. Data extracted by use of a standardised form included parameters required for sample size calculation and corresponding data reported in results sections of articles. We checked completeness of reporting of the sample size calculation, systematically replicated the sample size calculation to assess its accuracy, then quantified discrepancies between a priori hypothesised parameters necessary for calculation and a posteriori estimates.

RESULTS

Of the 215 selected articles, 10 (5%) did not report any sample size calculation and 92 (43%) did not report all the required parameters. The difference between the sample size reported in the article and the replicated sample size calculation was greater than 10% in 47 (30%) of the 157 reports that gave enough data to recalculate the sample size. The difference between the assumptions for the control group and the observed data was greater than 30% in 31% (n=45) of articles and greater than 50% in 17% (n=24). Only 73 trials (34%) reported all data required to calculate the sample size, had an accurate calculation, and used accurate assumptions for the control group.

CONCLUSIONS

Sample size calculation is still inadequately reported, often erroneous, and based on assumptions that are frequently inaccurate. Such a situation raises questions about how sample size is calculated in randomised controlled trials.

Epidemiology and reporting of randomised trials published in PubMed journals

Although randomised trials are important for evidence-based medicine, little is known about their overall characteristics. We assessed the epidemiology and reporting of methodological details for all 519 PubMed-indexed randomised trials published in December, 2000 (383 [74%] parallel-group, 116 [22%] crossover). 482 (93%) were published in specialty journals. A median of 80 participants (10th-90th percentile 25-369) were recruited for parallel-group trials. 309 (60%) were blinded. Power calculation, primary outcomes, random sequence generation, allocation concealment, and handling of attrition were each adequately described in less than half of publications. The small sample sizes are worrying, and poor reporting of methodological characteristics will prevent reliable quality assessment of many published trials.

An ontology of randomized controlled trials for evidence-based practice: content specification and evaluation using the competency decomposition method

Randomized controlled trials (RCTs) are one of the least biased sources of clinical research evidence, and are therefore a critical resource for the practice of evidence-based medicine. With over 10,000 new RCTs indexed in Medline each year, knowledge systems are needed to help clinicians translate evidence into practice. Common ontologies for RCTs and other domains would facilitate the development of these knowledge systems. However, no standard method exists for developing domain ontologies. In this paper, we describe a new systematic approach to specifying and evaluating the conceptual content of ontologies. In this method, called competency decomposition, the target task for an ontology is hierarchically decomposed into subtasks and methods, and the ontology content is specified by identifying the domain information required to complete each of the subtasks. We illustrate the use of this competency decomposition approach for the content specification and evaluation of an RCT ontology for evidence-based practice.

Publication of clinical trials: accountability and accessibility

Publication of findings from clinical trials is a necessary step in the research continuum, to provide a record of the work done, convey information to the community, and support translation of research into clinical practice. Systematic reviews of randomized controlled trials are now widely regarded as the highest level of evidence in determining the effect of an intervention on an outcome. They largely depend on internationally accessible, published reports of all trials undertaken. Investigators and their institutions or organizations have responsibility for reporting their clinical trials accurately and completely, including disclosure of potential conflicts of interest. To ensure evidence-based health care, issues relating to accessibility and accountability of clinical trial results require immediate action.

When can a clinical trial be called 'randomized'?

It is widely recognized that, in the context of the evaluation of medical interventions, randomized clinical trials constitute the gold standard. This is because randomization tends to balance both measured and unmeasured baseline characteristics, allows for masking, and provides a basis for inference. It is understandable, then, that investigators would wish to utilize this methodology whenever it is feasible to do so. Unfortunately, some studies are labeled as randomized when in fact they are not. These studies then receive more credibility, and influence medical practice, more than they deserve to. After reviewing the benefits of randomization, paying particular attention to the specific aspects of randomization that confer each benefit, we will explore the issue of what constitutes a randomized study.

Evidence-based practice for mere mortals: the role of informatics and health services research

The poor translation of evidence into practice is a well-known problem. Hopes are high that information technology can help make evidence-based practice feasible for mere mortal physicians. In this paper, we draw upon the methods and perspectives of clinical practice, medical informatics, and health services research to analyze the gap between evidence and action, and to argue that computing systems for bridging this gap should incorporate both informatics and health services research expertise. We discuss 2 illustrative systems--trial banks and a web-based system to develop and disseminate evidence-based guidelines (alchemist)--and conclude with a research and training agenda.

Sample size slippages in randomised trials: exclusions and the lost and wayward

Proper randomisation means little if investigators cannot include all randomised participants in the primary analysis. Participants might ignore follow-up, leave town, or take aspartame when instructed to take aspirin. Exclusions before randomisation do not bias the treatment comparison, but they can hurt generalisability. Eligibility criteria for a trial should be clear, specific, and applied before randomisation. Readers should assess whether any of the criteria make the trial sample atypical or unrepresentative of the people in which they are interested. In principle, assessment of exclusions after randomisation is simple: none are allowed. For the primary analysis, all participants enrolled should be included and analysed as part of the original group assigned (an intent-to-treat analysis). In reality, however, losses frequently occur. Investigators should, therefore, commit adequate resources to develop and implement procedures to maximise retention of participants. Moreover, researchers should provide clear, explicit information on the progress of all randomised participants through the trial by use of, for instance, a trial profile. Investigators can also do secondary analyses on, for instance, per-protocol or as-treated participants. Such analyses should be described as secondary and non-randomised comparisons. Mishandling of exclusions causes serious methodological difficulties. Unfortunately, some explanations for mishandling exclusions intuitively appeal to readers, disguising the seriousness of the issues. Creative mismanagement of exclusions can undermine trial validity.

Electronic trial banks: a complementary method for reporting randomized trials

BACKGROUND

Randomized clinical trial (RCT) results are often difficult to find, interpret, or apply to clinical care. The authors propose that RCTs be reported into electronic knowledge bases-trial banks-in addition to being reported in text. What information should these trial-bank reports contain?

METHODS

Using the competency decomposition method, the authors specified the ideal trial-bank contents as the information necessary and sufficient for completing the task of systematic reviewing.

RESULTS

They decomposed the systematic reviewing task into four top-level tasks and 62 subtasks. 162 types of trial information were necessary and sufficient for completing these subtasks. These items relate to a trial's design, execution, administration, and results.

CONCLUSION

Trial-bank publishing of these 162 items would capture into computer-understandable form all the trial information needed for critically appraising and synthesizing trial results. Decision-support systems that access shared, up-to-date trial banks could help clinicians manage, synthesize, and apply RCT evidence more effectively.

Methods used to evaluate the quality of evidence underlying the National Kidney Foundation-Dialysis Outcomes Quality Initiative Clinical Practice Guidelines: description, findings, and implications

This report describes the approach the National Kidney Foundation-Dialysis Outcomes Quality Initiative (NKF-DOQI) used to assess the strength of published evidence pertinent to individual NKF-DOQI Clinical Practice Guidelines, as well as the relationship between that approach and methods used by the US Preventive Services Task Force, the Cochrane Collaboration, and the Agency for Health Care Policy and Research to rate the quality and/or strength of evidence. We also present the results of an analysis of the strength of evidence underlying the NKF-DOQI Guidelines showing that one cannot infer the quality of evidence reported in a study (rated either on a 0-to-1 scale or categorically as excellent, very good, good, fair, or poor) simply by knowing the type of study design used (randomized trial, nonrandomized trial, natural experiment, cohort study, cross-sectional study, case-control study, case report). Issues related to assessment of the strength of evidence underlying a practice guideline opposed to that reported in an individual study are highlighted.

A critical review of papers from clinical cancer research

A review of 75 original articles from clinical cancer research in Norway is presented. Articles published in 1993 and with at least one Norwegian author were included in the review. Sixty papers were observational, whereas 15 were experimental. Of the observational studies 44 were retrospective. Most of the papers concerned prognostic factors. Prior hypotheses were explicitly defined in 16 papers only, and less than half of the articles described inclusion and exclusion criteria. Sample size calculations were performed in four papers only. The choice of statistical method was considered to be suitable in 22 of the 58 articles presenting statistical inferences. Problems related to multiple significance testing were rarely addressed, although the median number of p-values reported was as high as 8. Confidence intervals for main findings were presented in 14 papers. For proper planning of studies, as well as for analysis and interpretation of study results, statistical advice is indeed required.

Are community health interventions evaluated appropriately? A review of six journals

OBJECTIVES

To determine if Randomized Controlled Trial (RCT) methodology was used appropriately in community health, we: (1) determined the proportion of non-randomized studies that should have been RCTs, and (2) assessed the quality of the RCTs.

METHODS

The 1992 issues of six community health journals were manually searched. Intervention studies were analyzed. Studies that did not use randomization were analyzed for feasibility and practicality of RCT methods; RCTs were analyzed for quality using a checklist. RCTs were compared with community health RCTs from The New England Journal of Medicine. The proportion of studies meeting each criterion was determined.

RESULTS

Fourteen percent of 603 studies were interventions and 4% were RCTs. Of those not using randomization, 42% should have. Mean RCT scores were significantly lower for the community health journals than for The New England Journal of Medicine. Many criteria important to quality scored poorly.

CONCLUSIONS

RCTs are underused and lack methodologic rigor in community health. Conclusions regarding the effectiveness of interventions are therefore suspect.

Methodologic standards in surgical trials

BACKGROUND

Concerns have been raised that flaws in the design and analysis of trials will hinder the interpretation of their relevance to clinical practice. The objective of this study was to review the nature and methodologic standards of surgical trails published in 10 prestigious journals between January 1988 and December 1994.

METHODS

We evaluated the demography and methodologic standards of 364 trials. Each article was independently scrutinized by two assessors with documentation of the interassessor variation.

RESULTS

Less than 50% of the trials made comment about an unbiased assessment of outcome, gave an adequate description of the randomization technique, or provided a prospective estimate of the sample size. Economic factors were declared in 6.5% of the trials. Only 2% of the trials attempted to measure the effect of an intervention on the quality of life patients.

CONCLUSIONS

Readers should be cautious when interpreting the results of surgical trials.

Methodology citations and the quality of randomized controlled trials in obstetrics and gynecology

OBJECTIVES

Randomized controlled trials offer the best chance for valid treatment comparisons, yet most trials are of poor quality. This may reflect a lack of awareness of the requirements for conducting and reporting this type of research. If so, then citation of methodology references might indicate knowledge of how to conduct these studies and vice versa. Our study tests the hypothesis that the methodologic quality of published trials is related to citation of methodology references.

STUDY DESIGN

We performed a hand search of the American Journal of Obstetrics and Gynecology, the British Journal of Obstetrics and Gynaecology, the Journal of Obstetrics and Gynaecology, and Obstetrics and Gynecology to identify all randomized controlled trials published in 1990 and 1991 (N = 206). We reviewed the reference lists of all reports of randomized controlled trials and evaluated the adequacy of randomization methods by accepted criteria.

RESULTS

Most reports (81.6%) cited no methodology text or article. Although lack of any methodology reference was not significantly related to failure to report an adequate random method of sequence generation, this was highly related (p < 0.001) to failure to report adequate allocation concealment. Scanning the reference list of reports took a mean of 16 seconds and identified most poorly done trials.

CONCLUSIONS

Investigators who conduct randomized controlled trials should be thoroughly familiar with this type of research or should get expert help. Poorly done trials are wasteful and often misleading.

Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology

OBJECTIVE

To assess the methodological quality of approaches to blind ing and to handling of exclusions as reported in randomised trials from one medical specialty.

DESIGN

Survey of published, parallel group randomised controlled trials.

DATA SOURCES

A random sample of 110 reports in which allocation was described as randomised from 1990 and 1991 volumes of four journals of obstetrics and gynaecology.

MAIN OUTCOME MEASURES

The adequacy of the descriptions of double blinding and exclusions after randomisation.

RESULTS

Through 31 trials reported being double blind, about twice as many could have been. Of the 31 trials only eight (26%) provided information on the protection of the allocation schedule and only five (16%) provided some written assurance of successful implementation of double blinding. Of 38 trials in which the authors provided sufficient information for readers to infer that no exclusions after randomisation had occurred, six (16%) reported adequate allocation concealment and none stated that an intention to treat analysis had been performed. That compared with 14 (27%) and six (12%), respectively, for the 52 trials that reported exclusions.

CONCLUSIONS

Investigators could have double blinded more often. When they did double blind, they reported poorly and rarely evaluated it. Paradoxically, trials that reported exclusions seemed generally of a higher methodological standard than those that had no apparent exclusions. Exclusions from analysis may have been made in some of the trials in which no exclusions were reported. Editors and readers of reports of randomised trials should understand that flawed reporting of exclusions may often provide a misleading impression of the quality of the trial.

Development of Design-a-Trial, a knowledge-based critiquing system for authors of clinical trial protocols

Many published clinical trials are poorly designed, suggesting that the protocol was incomplete, disorganised or contained errors. This fact, doctors' limited statistical skills and the shortage of medical statisticians, prompted us to develop a knowledge-based aid, Design-a-Trial, for authors of clinical trial protocols. This interviews a physician, prompts them with suitable design options, comments on the statistical rigour and feasibility of their proposed design and generates a 6-page draft protocol document. This paper outlines the process used to develop Design-a-Trial, presents preliminary evaluation results, and discusses lessons we learned which may apply to the developed of other medical decision-aids.

Review of standards for reporting results of treatment of exocrine pancreatic cancer

From January 1989 to June 1992, 263 publications from professional international journals on investigations of results of treatment of exocrine pancreatic cancers were studied. The data on patient's selection were, in two-thirds of the studies, not satisfactory, and histopathological verification were missing or periampullary cancers were included in more than half of them. The number of patients in each study ranged between 5 and 442. The follow-up of at least 2 yr was in one-third of the surgically treated cases. In about every sixth study patients were lost to the follow-up, most often in of surgically treated cases. Complete or partial response to antineoplastic agents were reported in 11%, but in few of them long-term survival were noted. In about 20% of the studies the main comparison was made with historical controls. Measurements of quality of life was done in most studies before treatment, but in fewer afterward. Adverse effects were correctly reported in almost all cases on a short-term basis, but in only 38% in longer follow-ups. The overall data calls for improvement in recording and follow-up of patients treated for pancreatic cancer.

Frontiers of operative laparoscopy: a review and critique of the evidence

OBJECTIVE

This report reviews the evidence supporting the use of operative laparoscopy for a number of expanded indications in gynecology.

STUDY DESIGN

Studies published in English were identified through a Medline search back to 1966 for literature listed under the medical subject heading of "peritoneoscopy"; this search was supplemented by reviews of reference lists and discussions with experts.

RESULTS

Fair evidence exists to recommend use of laparoscopy for ectopic pregnancy, ovarian biopsy, and treatment of polycystic ovarian syndrome resistant to clomiphene citrate therapy. Evidence concerning other indications is inadequate to allow recommendations.

CONCLUSIONS

Mechanisms are urgently needed to evaluate surgical innovations in gynecology with the same degree of scientific rigor currently afforded medical innovations. Without surgical technology assessment, the current double standards in gynecologic therapy will persist.

Impact of random assignment on study outcome: an empirical examination

Sixty research investigations published in the biomedical literature were analyzed to examine the effect of design attributes on outcome. All 60 studies included a controlled trial involving a pretest, a therapeutic intervention, and a posttest across at least two groups. Thirty of the trials used random assignment of participants to treatment or control conditions and 30 trials employed some nonrandom method of subject assignment. Trial results were aggregated and evaluated by comparing effect sizes for the primary statistical test of the hypothesis. Data analysis revealed that the trial results, as measured by effect size, did not vary across therapeutic trials using random assignment versus those using nonrandom allocation of subjects. The impact of design attributes in the interpretation of multiple clinical trials addressing a similar research question is examined. The argument is made that various design attributes frequently associated with methodological quality should be considered as important moderator variables and their influence on trial outcome should not be assumed a priori but rather examined empirically.

Randomisation and baseline comparisons in clinical trials

80 reports of randomised clinical trials in four leading general medical journals were reviewed. The reporting of the methodology of randomisation was inadequate. In 30% of trials there was no clear evidence that the groups had been randomised. Among trials that used simple randomisation the sample sizes in the two groups were too often similar, and there was an unexpected small bias in favour of there being fewer patients in the experimental group. The handling of comparisons of baseline characteristics was inadequate in 41% of the trials. Suggestions are made for improving standards.

Meta-analysis and evidence

Meta-analysis is the science of combining evidence from different studies, but traditional statistical techniques contain neither a formal definition nor a measure of evidence. It is argued in this paper that the log-likelihood ratio, as a measure of the "weight of evidence," can be a very useful tool in the meta-analysis. The mathematics and the philosophy behind the use of this index are introduced. The construction and interpretation of "support curves" in fixed and random-effects models are presented. The application of evidential techniques is illustrated on six trials of aspirin therapy previously presented by Canner. The possible dangers of focusing on statistical error rates instead of evidence are discussed.

Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis

Important design aspects were decreasingly reported in NSAID trials over the years, whereas the quality of statistical analysis improved. In half of the trials, the effect variables in the methods and results sections were not the same, and the interpretation of the erythrocyte sedimentation rate in the reports seemed to depend on whether a significant difference was found. Statistically significant results appeared in 93 reports (47%). In 73 trials they favored only the new drug, and in 8 only the active control. All 39 trials with a significant difference in side effects favored the new drug. Choice of dose, multiple comparisons, wrong calculation, subgroup and within-groups analyses, wrong sampling units (in 63% of trials for effect variables, in 23% for side effects), change in measurement scale before analysis, baseline difference, and selective reporting of significant results were some of the verified or possible causes for the large proportion of results that favored the new drug. Doubtful or invalid statements were found in 76% of the conclusions or abstracts. Bias consistently favored the new drug in 81 trials, and the control in only one trial. It is not obvious how a reliable meta-analysis could be done in these trials.

The retrieval of randomized clinical trials in liver diseases from the medical literature: manual versus MEDLARS searches

Recently, the usefulness of MEDLARS computer searches in biomedical research was questioned; the conclusion was drawn that to completely capture a specific population of articles, the MEDLARS system was inadequate. This conclusion was based on a comparison of MEDLARS and manual searches for articles on random clinical trials in liver disease for the period 1966-1982. The present study re-evaluated the validity of this conclusion. In the initial revised search strategy, the recall of valid articles was not significantly improved, as compared with the original MEDLARS search strategy, but precision of this revised search was significantly increased. The number of valid articles recalled was significantly increased by broadening the MEDLARS search strategy criteria, but not without a simultaneous and significant decrease in precision. A limiting capture rate of about 79% of the population of articles on random clinical trials was reached when all logical expansions of the search strategy were exhausted. Strategies for increasing the efficiency of MEDLARS searches in general are discussed citing limitations specific to the population of random clinical trial articles.

Review: clinical trials in peptic ulcer disease--problems of methodology and interpretation

This review focuses on the methodology and interpretation of drug trials in peptic ulcer disease. The problems of planning and conduct that are discussed include the ethics of using placebo, eligibility criteria, estimations of sample size, stopping rules, randomization, blinding, and efficacy criteria, that is, ulcer healing and pain relief in the short term and prevention of relapse and complications in the long term. Statistical topics covered include confidence intervals, evaluation of survival type data, post-stratification, and sub-group analysis. The difference between clinical and statistical significance is discussed, major problems being overemphasis on P-value, type II errors, and post hoc power determinations. Explanatory and pragmatic questions are based on compliance-to-protocol and intention-to-treat cohorts, respectively, and involve problems of compliance testing, evaluation of withdrawals, and the use of fixed-dose regimens. The rather slow process for clinical trials to gain acceptance is described, and it is proposed to rely on disease-related and behavioural barriers, lack of knowledge of the inherent limitations of methodology, and overemphasis on the subject of peptic ulcer healing, in addition to some concern at the relevance of assessing long-term drug intervention by repeated endoscopies rather than by studying symptoms and the incidence of complications. We foresee an increased impact of clinical trials on ulcer research and therapeutic decision making, provided physicians are able to develop the proper methodology to answer the relevant questions.

Loss of patients in clinical trials that measure long-term survival following myocardial infarction

Loss of patients from clinical trials can nullify adequate randomization if the loss is unequally distributed among treatment groups. This study was designed to assess the magnitude of the problem in randomized control trials evaluating long-term therapy for survivors of myocardial infarction (MI). Only 19 of 52 trials reported having an explicit policy on withdrawals in the design stage; only 2 reported blinding the decision for withdrawal and only 7 reported accounting for withdrawals in sizing. In addition, only 16 gave the reader enough information to calculate the effect of withdrawals on trial results. In 2 of these 16 trials a p less than 0.05 result obtained by including withdrawals (intention to treat method) was reduced to p less than 0.05 when withdrawals were excluded. It is evident that many long-term trials do not contain adequate data on withdrawals. Readers of published trials are seldom able to judge whether or not withdrawals might affect the final results.

Cumulating quality of life results in controlled trials of coronary artery bypass graft surgery

Many studies evaluating the effectiveness of coronary artery bypass graft surgery allude to the quality of life benefit resulting from surgery. However, no comprehensive empirical estimate of the absolute or relative magnitude of this benefit is currently available. This paper presents a data synthesis of the research literature on bypass surgery to derive such an estimate. It uses follow-up measures of the percent of patients who were angina-free within both the surgical and medical groups of 14 controlled trials to estimate the quality of life benefit following surgery. Results based on the longest reported follow-up period suggest that the chances are approximately 25 to 40% greater that patients will be angina-free if they receive surgery rather than medical treatment. Estimates of benefit are about 15% less in randomized controlled trials compared to controlled trials that used a matching strategy. These results are unlikely to be affected by related factors such as the percentage of patients who crossover from the medical group to the surgical group or the specific method of calculating anginal relief used in this research report. However, differential patients selection may account for the observed design effect.