Effect of study criteria on recruitment and generalizability of the results

Associations Between Childhood Trauma Characteristics and Theory of Mind in Adults: Results From a Large, Diverse Sample

Theory of mind (ToM) is an essential social cognitive process encompassing abilities to represent and understand others' mental states. Although previous reports linked childhood trauma to social cognitive deficits, how characteristics of trauma exposure, such as subtype or timing, affect ToM remains unaddressed. Using data from a diverse adult sample (n = 2200), we tested whether exposure type and first exposure timing of common childhood trauma associated with ToM. Neither interpersonal loss (β = - 0.25, p = 0.170, [- 0.61, 0.10]) nor child maltreatment (β = - 0.21, p = 0.369, [- 0.66, 0.25]) was associated with lower ToM. There was no effect of timing of age at which trauma was experienced (F = 2.19, p = 0.087). While we did not identify age-dependent effects, future studies should examine links between timing or chronicity of prospectively reported childhood trauma and social cognition. Understanding of how childhood experiences shape ToM could reveal mechanisms underlying social cognition development and inform prevention efforts.

Effect of Eligibility Criteria on Patients' Survival and Serious Adverse Events in Colorectal Cancer Drug Trials

This study investigates the impact of clinical trial eligibility criteria on patient survival and serious adverse events (SAEs) in colorectal cancer (CRC) drug trials using real-world data. We utilized the OneFlorida+ network's data repository, conducting a retrospective analysis of CRC patients receiving FDA-approved first-line metastatic treatments. Propensity score matching created balanced case-control groups, which were evaluated using survival analysis and machine learning algorithms to assess the effects of eligibility criteria. Our study included 68,375 patients, with matched case-control groups comprising 1,126 patients each. Survival analysis revealed ethnicity and race, along with specific medical history (eligibility criteria), as significant survival outcome predictors. Machine learning models, particularly the XgBoost regressor, were employed to analyze SAEs, indicating that age and study groups were notable factors in SAEs occurrence. The study's findings highlight the importance of considering patient demographics and medical history in CRC trial designs.

Quadriceps Tendon Autograft Versus Bone-Patellar Tendon-Bone and Hamstring Tendon Autografts for Anterior Cruciate Ligament Reconstruction: A Systematic Review and Meta-analysis

BACKGROUND

The best type of autograft for anterior cruciate ligament (ACL) reconstruction remains debatable.

HYPOTHESIS

Compared with bone-patellar tendon-bone (BPTB) and hamstring tendon (HT) autografts, the quadriceps tendon (QT) autograft has comparable graft survival as well as clinical function and pain outcomes.

STUDY DESIGN

Meta-analysis; Level of evidence, 4.

METHODS

A systematic literature search was conducted in PubMed, Embase, Scopus, and the Cochrane Library to July 2020. Randomized controlled trials (RCTs) and observational studies reporting comparisons of QT versus BPTB or HT autografts for ACL reconstruction were included. All analyses were stratified according to study design: RCTs or observational studies.

RESULTS

A total of 24 studies were included: 7 RCTs and 17 observational studies. The 7 RCTs included 388 patients, and the 17 observational studies included 19,196 patients. No significant differences in graft failure (P = .36), the International Knee Documentation Committee (IKDC) subjective score (P = .39), or the side-to-side difference in stability (P = .60) were noted between QT and BPTB autografts. However, a significant reduction in donor site morbidity was noted in the QT group compared with the BPTB group (risk ratio [RR], 0.17 [95% CI, 0.09-0.33]; P < .001). No significant differences in graft failure (P = .57), the IKDC subjective score (P = .25), or the side-to-side stability difference (P = .98) were noted between QT and HT autografts. However, the QT autograft was associated with a significantly lower rate of donor site morbidity than the HT autograft (RR, 0.60 [95% CI, 0.39-0.93]; P = .02). A similar graft failure rate between the QT and control groups was observed after both early and late full weightbearing, after early and late full range of motion, and after using the QT autograft with a bone plug and all soft tissue QT grafts. However, a significantly lower rate of donor site morbidity was observed in the QT group compared with the control group after both early and late full weightbearing, after early and late full range of motion, and after using the QT autograft with a bone plug and all soft tissue QT grafts. No difference in effect estimates was seen between RCTs and observational studies.

CONCLUSION

The QT autograft had comparable graft survival, functional outcomes, and stability outcomes compared with BPTB and HT autografts. However, donor site morbidity was significantly lower with the QT autograft than with BPTB and HT autografts.

Conservative vs. operative treatment for humeral shaft fractures: a meta-analysis and systematic review of randomized clinical trials and observational studies

BACKGROUND

This meta-analysis aimed to compare conservative vs. operative treatment for humeral shaft fractures in terms of the nonunion rate, reintervention rate, permanent radial nerve palsy rate, and functional outcomes. Secondarily, effect estimates from observational studies were compared with estimates of randomized clinical trials (RCTs).

METHODS

The PubMed/Medline, Embase, CENTRAL (Cochrane Central Register of Controlled Trials), and CINAHL (Cumulative Index to Nursing and Allied Health Literature) databases were searched for both RCTs and observational studies comparing conservative with operative treatment for humeral shaft fractures.

RESULTS

A total of 2 RCTs (150 patients) and 10 observational studies (1262 patients) were included. The pooled nonunion rate of all studies was higher in patients treated conservatively (15.3%) vs. operatively (6.4%) (risk difference, 8%; odds ratio [OR], 2.9; 95% confidence interval [CI], 1.8-4.5; I² = 0%). The reintervention rate was also higher for conservative treatment (14.3%) than for operative treatment (8.9%) (risk difference, 6%; OR, 1.9; 95% CI, 1.1-3.5; I² = 30%). The higher reintervention rate was predominantly attributable to the higher nonunion rate in patients treated conservatively. The permanent radial nerve palsy rate was equal in both groups (OR, 0.6; 95% CI, 0.2-1.9; I² = 18%). There appeared to be no difference in mean time to union and mean Disabilities of the Arm, Shoulder and Hand scores between the treatment groups. No difference was found between effect estimates form observational studies and RCTs.

CONCLUSION

This systematic review shows that satisfactory results can be achieved with both conservative and operative management; however, operative treatment reduces the risk of nonunion compared with conservative treatment, with comparable reintervention rates (for indications other than nonunion). Furthermore, operative treatment results in a similar permanent radial nerve palsy rate, despite its inherent additional surgery-related risks. No difference in mean time-to-union and short-term functional results was detected.

Operative vs Nonoperative Treatment of Distal Radius Fractures in Adults: A Systematic Review and Meta-analysis

IMPORTANCE

No consensus has been reached to date regarding the optimal treatment for distal radius fractures. The international rate of operative treatment has been increasing, despite higher costs and limited functional outcome evidence to support this shift.

OBJECTIVES

To compare functional, clinical, and radiologic outcomes after operative vs nonoperative treatment of distal radius fractures in adults.

DATA SOURCES

The PubMed/MEDLINE, Embase, CENTRAL (Cochrane Central Register of Controlled Trials), and CINAHL (Cumulative Index to Nursing and Allied Health Literature) databases were searched from inception to June 15, 2019, for studies comparing operative vs nonoperative treatment of distal radius fractures.

STUDY SELECTION

Randomized clinical trials (RCTs) and observational studies reporting on the following: acute distal radius fracture with operative treatment (internal or external fixation) vs nonoperative treatment (cast immobilization, splinting, or bracing); patients 18 years or older; and functional outcome. Studies in a language other than English or reporting treatment for refracture were excluded.

DATA EXTRACTION AND SYNTHESIS

Data extraction was performed independently by 2 reviewers. Effect estimates were pooled using random-effects models and presented as risk ratios (RRs) or mean differences (MDs) with 95% CIs. Data were analyzed in September 2019.

MAIN OUTCOMES AND MEASURES

The primary outcome measures included medium-term functional outcome measured with the Disabilities of the Arm, Shoulder and Hand questionnaire (DASH) and the overall complication rate after operative and nonoperative treatment.

RESULTS

A total of 23 unique studies were included, consisting of 8 RCTs and 15 observational studies, that described 2254 unique patients. Among the studies that presented sex data, 1769 patients were women [80.6%]. Overall weighted mean age was 67 [range, 22-90] years). The RCTs included 656 patients (29.1%); observational studies, 1598 patients (70.9%). The overall pooled effect estimates the showed a significant improvement in medium-term (≤1 year) DASH score after operative treatment compared with nonoperative treatment (MD, -5.22 [95% CI, -8.87 to -1.57]; P = .005; I2 = 84%). No difference in complication rate was observed (RR, 1.03 [95% CI, 0.69-1.55]; P = .87; I2 = 62%). A significant improvement in grip strength was noted after operative treatment, measured in kilograms (MD, 2.73 [95% CI, 0.15-5.32]; P = .04; I2 = 79%) and as a percentage of the unaffected side (MD, 8.21 [95% CI, 2.26-14.15]; P = .007; I2 = 76%). No improvement in medium-term DASH score was found in the subgroup of studies that only included patients 60 years or older (MD, -0.98 [95% CI, -3.52 to 1.57]; P = .45; I2 = 34%]), compared with a larger improvement in medium-term DASH score after operative treatment in the other studies that included patients 18 years or older (MD, -7.50 [95% CI, -12.40 to -2.60]; P = .003; I2 = 77%); the difference between these subgroups was statically significant (test for subgroup differences, P = .02).

CONCLUSIONS AND RELEVANCE

This meta-analysis suggests that operative treatment of distal radius fractures improves the medium-term DASH score and grip strength compared with nonoperative treatment in adults, with no difference in overall complication rate. The findings suggest that operative treatment might be more effective and have a greater effect on the health and well-being of younger, nonelderly patients.

A comparison of neurocognition and functioning in first episode psychosis populations: do research samples reflect the real world?

PURPOSE

The current study evaluates the demographic, clinical, and neurocognitive characteristics of a recruited FEP research sample, a research control group, and a FEP clinic sample that were assessed and treated within the same center and time period.

METHODS

This study utilized data collected through an observational study and a retrospective chart review. Samples were ascertained in the Longitudinal Assessment and Monitoring of Clinical Status and Brain Function in Adolescents and Adults study and the Prevention and Recovery in Early Psychosis clinic. FEP clinic patients (n = 77), FEP research participants (n = 44), and age-matched controls (n = 38) were assessed using the MATRICS consensus cognitive battery and global functioning social and role scales. Between-group differences were assessed via one-way ANOVA and Chi-square analyses.

RESULTS

No significant differences were observed between groups with regard to age and gender. The FEP research sample had a higher proportion of white participants, better social and role functioning, and better neurocognitive performance when compared with the FEP clinical population. The clinic sample also had more diagnostic variability and higher prevalence of substance use disorders relative to the FEP research sample.

CONCLUSIONS

Researchers should be aware of how study design and recruitment practices may impact the representativeness of samples, with particular concern for equal representation of racial minorities and patients with more severe illness. Studies should be designed to minimize burden to promote a wider range of participation.

Operative treatment versus nonoperative treatment of Achilles tendon ruptures: systematic review and meta-analysis

OBJECTIVES

To compare re-rupture rate, complication rate, and functional outcome after operative versus nonoperative treatment of Achilles tendon ruptures; to compare re-rupture rate after early and late full weight bearing; to evaluate re-rupture rate after functional rehabilitation with early range of motion; and to compare effect estimates from randomised controlled trials and observational studies.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

PubMed/Medline, Embase, CENTRAL, and CINAHL databases were last searched on 25 April 2018 for studies comparing operative versus nonoperative treatment of Achilles tendon ruptures.

STUDY SELECTION CRITERIA

Randomised controlled trials and observational studies reporting on comparison of operative versus nonoperative treatment of acute Achilles tendon ruptures.

DATA EXTRACTION

Data extraction was performed independently in pairs, by four reviewers, with the use of a predefined data extraction file. Outcomes were pooled using random effects models and presented as risk difference, risk ratio, or mean difference, with 95% confidence interval.

RESULTS

29 studies were included-10 randomised controlled trials and 19 observational studies. The 10 trials included 944 (6%) patients, and the 19 observational studies included 14 918 (94%) patients. A significant reduction in re-ruptures was seen after operative treatment (2.3%) compared with nonoperative treatment (3.9%) (risk difference 1.6%; risk ratio 0.43, 95% confidence interval 0.31 to 0.60; P<0.001; I²=22%). Operative treatment resulted in a significantly higher complication rate than nonoperative treatment (4.9% v 1.6%; risk difference 3.3%; risk ratio 2.76, 1.84 to 4.13; P<0.001; I²=45%). The main difference in complication rate was attributable to the incidence of infection (2.8%) in the operative group. A similar reduction in re-rupture rate in favour of operative treatment was seen after both early and late full weight bearing. No significant difference in re-rupture rate was seen between operative and nonoperative treatment in studies that used accelerated functional rehabilitation with early range of motion (risk ratio 0.60, 0.26 to 1.37; P=0.23; I²=0%). No difference in effect estimates was seen between randomised controlled trials and observational studies.

CONCLUSIONS

This meta-analysis shows that operative treatment of Achilles tendon ruptures reduces the risk of re-rupture compared with nonoperative treatment. However, re-rupture rates are low and differences between treatment groups are small (risk difference 1.6%). Operative treatment results in a higher risk of other complications (risk difference 3.3%). The final decision on the management of acute Achilles tendon ruptures should be based on patient specific factors and shared decision making. This review emphasises the potential benefits of adding high quality observational studies in meta-analyses for the evaluation of objective outcome measures after surgical treatment.

Fixation of flail chest or multiple rib fractures: current evidence and how to proceed. A systematic review and meta-analysis

Purpose

The aim of this systematic review and meta-analysis was to present current evidence on rib fixation and to compare effect estimates obtained from randomized controlled trials (RCTs) and observational studies.

Methods

MEDLINE, Embase, CENTRAL, and CINAHL were searched on June 16th 2017 for both RCTs and observational studies comparing rib fixation versus nonoperative treatment. The MINORS criteria were used to assess study quality. Where possible, data were pooled using random effects meta-analysis. The primary outcome measure was mortality. Secondary outcome measures were hospital length of stay (HLOS), intensive care unit length of stay (ILOS), duration of mechanical ventilation (DMV), pneumonia, and tracheostomy.

Results

Thirty-three studies were included resulting in 5874 patients with flail chest or multiple rib fractures: 1255 received rib fixation and 4619 nonoperative treatment. Rib fixation for flail chest reduced mortality compared to nonoperative treatment with a risk ratio of 0.41 (95% CI 0.27, 0.61, p < 0.001, I² = 0%). Furthermore, rib fixation resulted in a shorter ILOS, DMV, lower pneumonia rate, and need for tracheostomy. Results from recent studies showed lower mortality and shorter DMV after rib fixation, but there were no significant differences for the other outcome measures. There was insufficient data to perform meta-analyses on rib fixation for multiple rib fractures. Pooled results from RCTs and observational studies were similar for all outcome measures, although results from RCTs showed a larger treatment effect for HLOS, ILOS, and DMV compared to observational studies.

Conclusions

Rib fixation for flail chest improves short-term outcome, although the indication and patient subgroup who would benefit most remain unclear. There is insufficient data regarding treatment for multiple rib fractures. Observational studies show similar results compared with RCTs.

Electronic supplementary material

The online version of this article (10.1007/s00068-018-1020-x) contains supplementary material, which is available to authorized users.

A Review of the Impact of Exclusion Criteria on the Generalizability of Schizophrenia Treatment Research

Treatment research studies employ criteria that determine which patients are eligible to participate and which are not. When such exclusion criteria produce a treatment research sample that is a small and unrepresentative subset of all patients with a particular disease, clinicians may be hesitant to apply the research results in front-line clinical practice. Accordingly, the present paper reviews the English-language literature on exclusion criteria in schizophrenia treatment research and draws initial conclusions about their impact. Empirically derived estimates of the rate of exclusion vary widely (31.0-98.2%), but the best available evidence suggests that about 4 in 5 patients with schizophrenia would be ineligible to enroll in a typical treatment research study. Women are particularly likely to be excluded from schizophrenia treatment research, which is problematic from both a clinical and social justice viewpoint. Excluded patients also tend to be older than eligible patients, and, though it has been examined in only a few studies, they also tend to have more severe problems at baseline and different outcomes over time than patients who are allowed to participate in research. More limited use of exclusion criteria in schizophrenia treatment research would be beneficial in terms of increasing generalizability, but would also potentially involve costs, particularly a need for larger samples. More modest steps that would improve treatment outcome research reports include requiring a full description of the rationale for, and nature of, any exclusion criteria, and, having a designated place in the discussion section which draws attention to the proper scope of generalization.

A systematic literature review on the efficacy-effectiveness gap: comparison of randomized controlled trials and observational studies of glucose-lowering drugs

Aim

To identify a potential efficacy–effectiveness gap and possible explanations (drivers of effectiveness) for differences between results of randomized controlled trials (RCTs) and observational studies investigating glucose-lowering drugs.

Methods

A systematic literature review was conducted in English language articles published between 1 January, 2000 and 31 January, 2015 describing either RCTs or observational studies comparing glucagon-like peptide-1 analogs (GLP-1) with insulin or comparing dipeptidyl peptidase-4 inhibitors (DPP-4i) with sulfonylurea, all with change in glycated hemoglobin (HbA1c) as outcome. Medline, Embase, Current Content, and Biosis were searched. Information on effect estimates, baseline characteristics of the study population, publication year, study duration, and number of patients, and for observational studies, characteristics related to confounding adjustment and selection- and information bias were extracted.

Results

From 312 hits, 11 RCTs and 7 observational studies comparing GLP-1 with insulin, and from 474 hits, 16 RCTs and 4 observational studies comparing DPP-4i with sulfonylurea were finally included. No differences were observed in baseline characteristics of the study populations (age, sex, body mass index, time since diagnosis of type 2 diabetes mellitus, and HbA1c) or effect sizes across study designs. Mean effect sizes ranged from −0.43 to 0.91 and from −0.80 to 1.13 in RCTs and observational studies, respectively, comparing GLP-1 with insulin, and from −0.13 to 2.70 and −0.20 to 0.30 in RCTs and observational studies, respectively, comparing DPP-4i and sulfonylurea. Generally, the identified observational studies held potential flaws with regard to confounding adjustment and selection- and information bias.

Conclusions

Neither potential drivers of effectiveness nor an efficacy–effectiveness gap were identified. However, the limited number of studies and potential problems with confounding adjustment, selection- and information bias in the observational studies, may have hidden a true efficacy-effectiveness gap.

Selection criteria limit generalizability of smoking pharmacotherapy studies differentially across clinical trials and laboratory studies: A systematic review on varenicline

BACKGROUND

The selection criteria used in clinical trials for smoking cessation and in laboratory studies that seek to understand mechanisms responsible for treatment outcomes may limit their generalizability to one another and to the general population.

METHODS

We reviewed studies on varenicline versus placebo and compared eligibility criteria and participant characteristics of clinical trials (N=23) and laboratory studies (N=22) across study type and to nationally representative survey data on adult, daily USA smokers (2014 National Health Interview Survey; 2014 National Survey on Drug Use and Health).

RESULTS

Relative to laboratory studies, clinical trials more commonly reported excluding smokers who were unmotivated to quit and for specific medical conditions (e.g., cardiovascular disease, COPD), although both study types frequently reported excluding for general medical or psychiatric reasons. Laboratory versus clinical samples smoked less, had lower nicotine dependence, were younger, and more homogeneous with respect to smoking level and nicotine dependence. Application of common eligibility criteria to national survey data resulted in considerable elimination of the daily-smoking population for both clinical trials (≥47%) and laboratory studies (≥39%). Relative to the target population, studies in this review recruited participants who smoked considerably more and had a later smoking onset age, and were under-representative of Caucasians.

CONCLUSIONS

Results suggest that selection criteria of varenicline studies limit generalizability in meaningful ways, and differences in criteria across study type may undermine efforts at translational research. Recommendations for improvements in participant selection and reporting standards are discussed.

A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results

Randomized controlled trials (RCTs) are conducted under idealized and rigorously controlled conditions that may compromise their external validity. A literature review was conducted of published English language articles that reported the findings of studies assessing external validity by a comparison of the patient sample included in RCTs reporting on pharmaceutical interventions with patients from everyday clinical practice. The review focused on publications in the fields of cardiology, mental health, and oncology. A range of databases were interrogated (MEDLINE; EMBASE; Science Citation Index; Cochrane Methodology Register). Double-abstract review and data extraction were performed as per protocol specifications. Out of 5,456 de-duplicated abstracts, 52 studies met the inclusion criteria (cardiology, n = 20; mental health, n = 17; oncology, n = 15). Studies either performed an analysis of the baseline characteristics (demographic, socioeconomic, and clinical parameters) of RCT-enrolled patients compared with a real-world population, or assessed the proportion of real-world patients who would have been eligible for RCT inclusion following the application of RCT inclusion/exclusion criteria. Many of the included studies concluded that RCT samples are highly selected and have a lower risk profile than real-world populations, with the frequent exclusion of elderly patients and patients with co-morbidities. Calculation of ineligibility rates in individual studies showed that a high proportion of the general disease population was often excluded from trials. The majority of studies (n = 37 [71.2 %]) explicitly concluded that RCT samples were not broadly representative of real-world patients and that this may limit the external validity of the RCT. Authors made a number of recommendations to improve external validity. Findings from this review indicate that there is a need to improve the external validity of RCTs such that physicians treating patients in real-world settings have the appropriate evidence on which to base their clinical decisions. This goal could be achieved by trial design modification to include a more representative patient sample and by supplementing RCT evidence with data generated from observational studies. In general, a thoughtful approach to clinical evidence generation is required in which the trade-offs between internal and external validity are considered in a holistic and balanced manner.

How Commonly Used Inclusion and Exclusion Criteria in Antidepressant Registration Trials Affect Study Enrollment

In clinical trials, each specific inclusion and exclusion criterion eliminates a percentage of the potentially eligible population from trial participation and thus increases the time and effort needed for enrollment in a study. Drug developers often do not have data on how these criteria affect the pool of potentially eligible subjects for their trials and, hence, they cannot factor in the impact of these criteria when designing a study and planning the time needed to complete it. Consequently, drug developers often have ambitious timelines that are unrealistic and can lead to actions that may interfere with the ability to separate the efficacy of drug versus placebo. To investigate the effects of inclusion and exclusion criteria on study enrollment, the authors quantified the effects of the inclusion and exclusion criteria commonly used in antidepressant registration trials (ARTs) by applying these criteria to the population treated in the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study. In essence, the STAR*D study population was used as a surrogate for the general population of individuals with major depressive disorder. The effect of each criterion commonly used in ARTs was assessed in terms of the percentage of the STAR*D population that would have been excluded individually and collectively (i.e., when all criteria were applied at once). For continuous criteria such as age and severity of depression, the resulting effects have been presented graphically. Collectively, the typical inclusion and exclusion criteria used in ARTs would have eliminated at least 82% of the STAR*D population. This result means that more than 5 times the number of subjects would have to be screened to find a population that would meet the typical inclusion and exclusion criteria for an ART, directly determining the screening effort required in terms of both resources and time. Thus, developers of antidepressant drugs can use the data from this study to plan the recruitment effort required and to weigh any potential benefit of each criterion alone and in aggregate versus their cost in terms of recruitment support and time. These data also graphically illustrate for prescribers how restrictive the population likely to be enrolled in ARTs is relative to the patients whom they treat with such medications.

The role of observational studies in optimizing the clinical management of chronic myeloid leukemia

Survival has increased dramatically for patients with chronic phase chronic myeloid leukemia (CP-CML) using BCR-ABL targeted tyrosine kinase inhibitors, such that life expectancy is expected to approximate that of patients without CP-CML. Randomized controlled trials (RCTs) and observational studies provide valuable insights into the management of chronic diseases such as CP-CML. RCTs are undoubtedly the backbone of clinical research, and the 'gold standard' for evaluating the efficacy and safety of new therapies. However, many questions surrounding the optimal management of patients with CML remain unanswered, and it is widely accepted that these questions will be best answered by evaluating the use of available therapies in clinical practice. Observational studies can extend the knowledge base beyond the clinical trial setting and thus capture a more accurate picture of everyday clinical practice, particularly patients' experiences of long-term CML treatment. There is therefore growing interest in and appreciation of the value of observational research. This review article will examine the relative merits of RCTs and observational studies in the setting of CML, highlighting those factors - such as the advancing age of the CML patient population and growing importance of patient-reported outcomes - that mean that observational studies should play an important role in shaping clinical practice. This article also provides an overview of what observational studies have told us thus far about the optimal management of patients with CML, outlines some of the key remaining unanswered clinical questions in CML, and summarizes ongoing observational studies designed to provide answers to these key questions.

Are participants in pharmacological and psychotherapy treatment trials for social anxiety disorder representative of patients in real-life settings?

BACKGROUND

The present study sought to quantify the generalizability of clinical trial results in individuals with a Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, diagnosis of social anxiety disorder (SAD) to a large representative community sample.

METHODS

Data were derived from the 2004-2005 National Epidemiologic Survey on Alcohol and Related Conditions, a large nationally representative sample of 34,653 adults from the US population. We applied a standard set of exclusion criteria representative of pharmacological and psychotherapy clinical trials to all adults with a Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, diagnosis of SAD (n = 965) in the past 12 months and then to a subsample of participants seeking treatment (n = 363). Our aim was to assess how many participants with SAD would fulfill typical eligibility criteria.

RESULTS

We found that more than 7 of 10 respondents from the overall SAD sample in a typical pharmacological efficacy trial and more than 6 of 10 participants in a typical psychotherapy efficacy trial would have been excluded by at least 1 criterion. In addition, more than 8 of 10 respondents seeking treatment for SAD would have been excluded from participation in a typical pharmacological or psychotherapy efficacy trial. Having a current major depression explained a large proportion of ineligibility.

CONCLUSIONS

Clinical trials should carefully consider the impact of exclusion criteria on the generalizability of their results and explain the rationale for their use. For SAD treatment trials to adequately inform clinical practice, the eligibility rate must be increased through a general relaxation of overly stringent eligibility criteria.

Assessing the external validity of a randomized controlled trial of anthelminthics in mothers and their children in Entebbe, Uganda

Background

The ‘external validity’ of randomized controlled trials is an important measure of quality, but is often not formally assessed. Trials concerning mass drug administration for helminth control are likely to guide public health policy and careful interpretation of their context is needed. We aimed to determine how representative participants in one such trial were of their community. We explore implications for trial interpretation and resulting public health recommendations.

Methods

The trial assessed was the Entebbe Mother and Baby Study (EMaBS), a trial of anthelminthic treatment during pregnancy and early childhood. In a novel approach for assessing external validity, we conducted a two-stage cluster sample community survey within the trial catchment area and compared characteristics of potentially-eligible community children with characteristics of children participating in the trial.

Results

A total of 173 children aged three to five-years-old were surveyed from 480 households. Of children surveyed, we estimated that mothers of 60% would have been eligible for recruitment, and of these, 31% had actually been enrolled. Children surveyed were compared to 199 trial children in the same age group reviewed at annual trial visits during the same time period. There were significant differences in ethnicity between the trial participants and the community children, and in socioeconomic status, with those in the trial having, on average, more educated parents and higher maternal employment. Trial children were less likely to have barefoot exposure and more likely to use insecticide-treated bed nets. There were no significant differences in numbers of reported illness events over the last year.

Conclusions

The trial had not enrolled all eligible participants, and those enrolled were of higher socioeconomic status, and had lower risk of exposure to the parasitic infections targeted by the trial interventions. It is possible the trial may have underestimated the absolute effects of anthelminthic treatment during pregnancy and early childhood, although the fact that there were no differences in reported incidence of common infectious diseases (one of the primary outcomes of EMaBS) between the two groups provides reassurance. Concurrent community surveys may be an effective way to test the external validity of trials.

EMaBS Trial registration

ISRCTN32849447, registered 22 July 2005

Electronic supplementary material

The online version of this article (doi:10.1186/1745-6215-15-310) contains supplementary material, which is available to authorized users.

Representativeness of obstetric patients who participate in perinatal depression research: findings from the Women's Mental Health and Infants Program (WMHIP) integrated dataset

The aims of this study were to evaluate the feasibility of integrating archival datasets from depression projects involving pregnant women recruited from obstetric clinics and then assess the representativeness of the integrated dataset. Datasets from six studies were standardized and integrated. Chi-square, t-, and Wilcoxon rank-sum tests were used to compare characteristics between women who completed a depression screening questionnaire (DSQ) and were (1) eligible and ineligible for research participation and (2) eligible women who accepted and declined participation. The integrated dataset comprises 9,112 pregnant women, of whom 71.0 % (n = 6,472) were ineligible for participation because their DSQ scores indicated no-to-minimal depressive symptoms (NDS). Among the 23.9 % (2,176) of women identified as eligible, in part, because their DSQ scores indicated elevated levels of depressive symptoms (EDS), 29.6 % (644) of women participated (P-EDS) and 47.6 % (1,036) of women did not participate (D-EDS). While the NDS and EDS groups were significantly different on almost all variables, the P-EDS and D-EDS groups were significantly different on only a few variables. Compared to the D-EDS group, the P-EDS group was earlier in pregnancy and, on the Edinburgh Postnatal Depression Screen, was more likely to endorse impaired "ability to laugh" and "enjoy oneself", and endorse at greater severity "ability to laugh." It is a reasonable and feasible strategy to integrate thematically similar datasets to increase statistical power. Additionally, typical recruitment strategies for minimal risk perinatal depression research at obstetric clinics, during routine prenatal care visits, appear to produce an externally valid study cohort.

Veteran subjects willingness to participate in schizophrenia clinical trials

Predictive characteristics of subjects agreeing to be randomized into clinical trials for the treatment of schizophrenia and schizoaffective disorder have been little studied. In this study, we used data from the recruitment phase of a randomized trial that compared long acting injectable (LAI) risperidone to oral antipsychotic medications. Basic socio-demographic and clinical data were gathered from eligible patients and clinicians at the time of screening for trial entry. Bivariate comparisons and multivariate logistic regression were used to compare those who agreed to participate and those who refused. Altogether 446 veterans were eligible on preliminary screening, of these 382 (86 %) agreed to participate and 64 (14 %) declined. Eligible patients who agreed to be randomized were more willing to change medications without regard to their level of satisfaction with their current medication. Subjects reported as currently taking LAI medication and taking risperidone, in particular, were more likely to agree to participate. Factors that did not significantly predict participation included age, years on current medication, reported medication compliance, race, and gender. Veterans with schizophrenia or schizoaffective disorder who were actually more satisfied with their current medications and who were currently taking the experimental agent were more likely to agree to participate in this randomized clinical trial in contrast to expectations that individuals who are unsatisfied with their current treatment would be more likely to enroll in such studies.

Are subjects in treatment trials of panic disorder representative of patients in routine clinical practice? Results from a national sample

BACKGROUND

Research on the generalizability of clinical trials in panic disorder is limited. The present study sought to quantify the generalizability of clinical trials' results of individuals with DSM-IV panic disorder (PD) to a large community sample.

METHODS

Data were derived from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC), a large national representative sample of 43,093 adults of the United States population. We applied a standard set of eligibility criteria representative of PD clinical trials to all adults with past 12 months PD (n=907), and then to a subgroup of participants seeking treatment (n=105). Our aim was to determine the proportion of participants with PD who would have been excluded by typical eligibility criteria.

RESULTS

We found that more than 8 out of ten participants (80.52%; 95% CI=77.13-83.52%) with PD were excluded by at least one criterion. In the subgroup of participants who sought treatment, the exclusion rate by at least one criterion was higher (92.40%; 95% CI=84.60-96.42%). For the full sample and the treatment-seeking subsample, having currently a depression and a diagnosis of alcohol or drug abuse/dependence were the criteria excluding the highest percentage of participants. Having a lifetime history of bipolar disorder and a current significant medical condition also excluded a substantial proportion of individuals in both samples. Exclusion rates were similar when considering panic disorder with and without agoraphobia.

CONCLUSIONS

Clinical trials, that exclude a majority of adults with panic disorder, should carefully consider the impact of eligibility criteria on the generalizability of their results. As required by CONSORT guidelines, reporting exclusion rate estimate and reasons of eligibility should be mandatory in both clinical trials and meta-analyses.

Generalizability of clinical trial results for generalized anxiety disorder to community samples

BACKGROUND

There has been little research on the generalizability of clinical trials for generalized anxiety disorder (GAD). The present study examines the generalizability of pharmacological and psychotherapy clinical trials' results of individuals with DSM-IV GAD to a large community sample.

METHODS

Data were drawn from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC), a large national representative face-to-face sample of 43,093 adults of the United States population. We applied a standard set of eligibility criteria representative of GAD pharmacological and psychotherapy clinical trials to all adults with past 12 months GAD (n = 894), and to a subgroup of participants seeking treatment (n = 329). Our aim was to assess how many participants with GAD would fulfil typical eligibility criteria.

RESULTS

We found that more than seven out of 10 participants with GAD were excluded by at least one criterion. In the subgroup of GAD participants who sought treatment, the exclusion rate by at least one criterion raised to more than eight out of 10 participants with GAD. For the overall sample and the treatment-seeking subsample, having a current depression was the criterion excluding the highest percentage of individuals. Having a lifetime history of bipolar disorder, a current significant medical condition, a current diagnosis of alcohol abuse or dependence, and a social or specific phobia also excluded a substantial proportion of individuals in both samples.

CONCLUSIONS

Clinical trials exclude a majority of adults with GAD. Clinical trials should carefully consider the impact of eligibility criteria on the generalizability of their results.

Predictors of physician referral for patient recruitment to Alzheimer disease clinical trials

BACKGROUND

Inadequate recruitment into Alzheimer disease clinical trials is an important threat to the validity and generalizability of the studies. The majority of dementia patients are first evaluated by community-based physicians; however, physician perceptions of clinical research are largely unknown.

METHODS

A survey was distributed to 3123 physicians in 3 states; 370 were returned. Survey items assessed attitudes, perceived benefits of and barriers to referral to clinical research, and physicians use of the internet for medical information.

RESULTS

The mean age of the respondents was 50.6+/-10.8 years; 70% were male, 78% white, 61% were primary care providers; 63% used the internet > or =3 times/week. No demographic or medical specialty differences existed between those who were likely (n=193) and unlikely (n=162) to refer patients to clinical trials. Differences were discovered in perceived benefits reported by physicians who were more likely to refer, whereas differences in perceived barriers existed in primary care compared with specialists. Referral to clinical trials is predicted by close proximity to a research center [odds ratio (OR): 4.0; 95% confidence interval (CI), 1.1-15.6] and availability of internet information regarding diagnostic evaluation (OR: 2.3; 95% CI, 1.1-4.7). Primary barriers included concerns about exposure of patients to uncomfortable procedures (OR: 4.7; 95% CI, 1.2-18.7) and lack of time to discuss research participation (OR: 6.8; 95% CI, 1.4-32.3).

CONCLUSIONS

Proximity to a research center and availability of diagnostic clinical tools are strong predictors of clinical trial referral. Concern over risks to patients and lack of time are strong barriers. These results suggest that dementia outreach education targeted to physicians should emphasize the importance of clinical trials with a focus on discussing research participation in a time-efficient manner and increasing awareness of risk reduction and the safety of research protocols. Providing easy access to up-to-date, user-friendly educational materials on dementia diagnosis and research via the internet are likely to improve referrals of patients to Alzheimer disease clinical trials from community physicians.

Outliers on the dose-response curve: how to minimize this problem using therapeutic drug monitoring, an underutilized tool in psychiatry

This column continues the discussion of outliers on the dose-response curve begun in earlier columns. It focuses on therapeutic drug monitoring (TDM) as an underutilized tool in psychiatry to minimize this problem. The scientific rationale for dose adjustment based on TDM is presented and its efficiency is contrasted with dose adjustment based on clinical assessment of response. In current practice, the use of TDM with psychiatric drugs is generally restricted to drugs with narrow therapeutic windows or drugs imported into psychiatry from neurology where TDM is more commonly used. Examples of each of these types of drugs are cited.

From randomized controlled trials to observational studies

Randomized controlled trials are considered the gold standard in the hierarchy of research designs for evaluating the efficacy and safety of a treatment intervention. However, their results can have limited applicability to patients in clinical settings. Observational studies using large health care databases can complement findings from randomized controlled trials by assessing treatment effectiveness in patients encountered in day-to-day clinical practice. Results from these designs can expand upon outcomes of randomized controlled trials because of the use of larger and more diverse patient populations with common comorbidities and longer follow-up periods. Furthermore, well-designed observational studies can identify clinically important differences among therapeutic options and provide data on long-term drug effectiveness and safety.

Learning from past mistakes: assessing trial quality, power and eligibility in non-renal systemic lupus erythematosus randomized controlled trials

OBJECTIVES

To evaluate the post hoc study power of randomized controlled trials (RCTs) in the treatment of non-renal SLE and to determine the generalizability of these RCTs using an SLE database.

METHODS

RCTs in non-renal SLE were identified using PubMed (1975-2007). Inclusion/exclusion criteria, trial quality (5-point scale) and results of each study were recorded. The inclusion/exclusion criteria were compared with an SLE database to determine the proportion of patients from the database who would theoretically be eligible for these trials. For each negative study, we calculated the post hoc study power. We also looked for temporal improvements of trials in the literature and examined if pharmaceutical involvement influenced trial quality.

RESULTS

Sixty-four articles were included; the mean power of 30 negative studies was 24.6 +/- s.e.m. 3.9% (range 2.5-81.1%). Only one study had a power > 80%. Overall, potential eligibility of SLE patients in the database was 45.1 +/- s.e.m. 3.6%. Only 14 studies (21.9%) were of good quality. Fortunately, RCT quality is improving over time (trials <1995, compared with 1996-2002 and >2003; P < 0.001). Trials with pharmaceutical involvement had a significantly higher number of enrollees and better study quality.

CONCLUSIONS

Negative RCTs in SLE were mostly underpowered but the generalizability of these trials was high. Determination of study power and the impact of eligibility criteria on generalizability of study results are crucial in the design of clinical trials to ensure applicability to clinical practice.

Assessment of unmet needs and the lack of generalizability in the design of randomized controlled trials for scleroderma treatment

OBJECTIVE

To determine the generalizability of randomized controlled trials (RCTs) in the treatment of systemic sclerosis (SSc) using the Canadian Scleroderma Research Group (CSRG) database.

METHODS

We identified articles related to SSc published from 1958 to 2006. Key points on trial design were recorded. The inclusion/exclusion criteria were used in conjunction with the CSRG database to determine the proportion of patients with SSc who would theoretically be eligible for these trials. Articles were classified into subcategories according to the target system. The CSRG database contains 438 patients with SSc from 14 Canadian centers. Results were in median (%) and mean (%) with 95% confidence intervals (95% CIs).

RESULTS

In total, 210 articles were evaluated and 73 were selected for inclusion in this study. The mean percentage of eligible patients with SSc associated with other conditions was 35% (95% CI 17-53) for Raynaud's phenomenon, 24% (95% CI 1-47) for digital ulcers, 48% (95% CI 27-68) for gastrointestinal (GI) involvement, 32% (95% CI 20-43) for overall disease modification, 6% (95% CI 4-8) for pulmonary arterial hypertension, 2% (95% CI 0-4) for interstitial lung disease, and 38% (95% CI 12-64) for other categories.

CONCLUSION

Except for GI trials, <38% of the identified patients with SSc would have been suitable to enter the RCTs. Although some patients would be ineligible because they lack certain organ involvement, RCTs designed to include appropriate patients with SSc are needed; there are few proven treatments and trials typically do not include the majority of those who could potentially benefit from the intervention.