Comparison between treatment effects in a randomised controlled trial and an observational study using propensity scores in primary care

Postoperative recreational sports and subjective shoulder function: a comprehensive analysis following intramedullary stabilization of displaced midshaft clavicular fractures

Introduction Sports-related outcomes and the role of recreational sports activities for shoulder function after intramedullary stabilization (IMS) of displaced midshaft clavicular fractures (DMCFs) in the general population are not well known. In this study, we aimed to determine the sport-related outcomes (return-to-sports [RTS] rate, type of sports, time until RTS, and intensity) and to explore the role of sports after IMS of DMCFs. Materials and Methods This single-center, retrospective, cohort study included patients who underwent IMS of DMCFs between 2009 and 2022 at a Level II trauma center in Germany, experienced no major complications, and had completed at least 1 year of follow-up. Propensity score matching was conducted to obtain a balanced sample of patients who did not engage (cases) and engaged (controls) in postoperative sports activities by adjusting for age and fracture complexity. Groups were compared to assess the impact of regular sports activities on subjective shoulder functioning at follow-up, as evaluated using the Disabilities of the Arm, Shoulder, and Hand and Oxford Shoulder Score (OSS) questionnaires, after controlling for the patient- (i.e., sex and smoking) and treatment- (i.e., surgery duration and physical therapy) factors. Results Among the 199 patients included, the RTS rate was 97.5%, and 160 patients practiced regular postoperative activity, mostly in the same sport and intensity. In the matched cohort (39 cases and 39 controls), practicing regular sports activities postoperatively was the only independent factor associated with a higher OSS in the multiple regression analysis (unstandardized regression coefficient = 2.40; Bias-corrected and accelerated 95% confidence interval [0.28, 4.69]). Conclusions The sport-related outcomes after IMS of DMCFs in our cohort were comparable to those achieved after plate osteosynthesis, and IMS reported in the literature. Recreational sports activities benefitted subjective shoulder function, thereby encouraging further research and potentially influencing management policies. Level of evidence Grade IV - a retrospective observational cohort study.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

BACKGROUND

Researchers and decision-makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up-to-date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question.

OBJECTIVES

To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types.

SEARCH METHODS

We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews.

SELECTION CRITERIA

We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case-control and cross-sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation.

DATA COLLECTION AND ANALYSIS

Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach.

MAIN RESULTS

We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty-four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs - for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case-control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty-three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: - pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); - RCTs and observational studies with substantial or high heterogeneity; that is, I2 ≥ 50% (ratio of ratios 1.11, 95% CI 1.04 to 1.18); - no use (ratio of ratios 1.07, 95% CI 1.03 to 1.11) or unclear use (ratio of ratios 1.13, 95% CI 1.03 to 1.25) of propensity score adjustment in observational studies; and - observational studies without further specification of the study design (ratio of ratios 1.06, 95% CI 0.96 to 1.18). We detected no clear difference in other subgroup analyses.

AUTHORS' CONCLUSIONS

We found no difference or a very small difference between effect estimates from RCTs and observational studies. These findings are largely consistent with findings from recently published research. Factors other than study design need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies, such as differences in the population, intervention, comparator, and outcomes investigated in the respective studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding is needed of how these factors might yield estimates reflective of true effectiveness.

Comparison of median sternotomy closure-related complication rates using orthopedic wire or suture in dogs: A multi-institutional observational treatment effect analysis

Objective

To determine and compare median sternotomy (MS) closure‐related complication rates using orthopedic wire or suture in dogs.

Study design

Multi‐institutional, retrospective observational study with treatment effect analysis.

Animals

331 client‐owned dogs, of which 68 were excluded.

Methods

Medical records of dogs with MS were examined across nine referral centers (2004–2020). Signalment, weight, clinical presentation, surgical details, complications, and outcomes were recorded. Follow‐up was performed using patient records and email/telephone contact. Descriptive statistics, treatment effect analysis and logistic regression were performed.

Results

Median sternotomy closure was performed with wire in 115 dogs and suture in 148. Thirty‐seven dogs experienced closure‐related complications (14.1%), 20 in the wire group and 17 in the suture group. Twenty‐three were listed as mild, four as moderate and 10 as severe. Treatment effect analysis showed a mean of 2.3% reduction in closure‐related complications associated with using suture versus wire (95% CI: −9.1% to +4.5%). In multivariable logistic regression, the only factor associated with increased risk of closure‐related complications was dog size (p = .01). This effect was not modified by the type of closure used (interaction term: OR = 0.99 [95% CI: 0.96/1.01]).

Conclusion

The incidence of closure‐related complication after MS was low compared to previous reports. The likelihood of developing a closure‐related complication was equivalent between sutures and wires, independent of dog size, despite a higher proportion of complications seen in larger dogs (≥20 kg).

Clinical significance

Use of either orthopedic wire or suture appear to be an appropriate closure method for sternotomy in dogs of any size.

Use of propensity score matching to create counterfactual group to assess potential HIV prevention interventions

The design of HIV prevention trials in the context of effective HIV preventive methods is a challenge. Alternate designs, including using non-randomised 'observational control arms' have been proposed. We used HIV simulated vaccine efficacy trials (SiVETs) to show pitfalls that may arise from using such observational controls and suggest how to conduct the analysis in the face of the pitfalls. Two SiVETs were nested within previously established observational cohorts of fisherfolk (FF) and female sex workers (FSW) in Uganda. SiVET participants received a licensed Hepatitis B vaccine in a schedule (0, 1 and 6 months) similar to that for a possible HIV vaccine efficacy trial. All participants received HIV counselling and testing every quarter for one year to assess HIV incidence rate ratio (IRR) between SiVET and non-SiVET (observational data). Propensity scores, conditional on baseline characteristics were calculated for SiVET participation and matched between SiVET and non-SiVET in the period before and during the SiVET study. We compared IRR before and after propensity score matching (PSM). In total, 3989 participants were enrolled into observational cohorts prior to SiVET, (1575 FF prior to Jul 2012 and 2414 FSW prior to Aug 2014). SiVET enrolled 572 participants (Jul 2012 to Apr 2014 in FF and Aug 2014 to Apr 2017 in FSW), with 953 non-SiVET participants observed in the SiVET concurrent period and 2928 from the pre-SiVET period (before Jul 2012 in FF or before Apr 2014 in FSW). Imbalances in baseline characteristics were observed between SiVET and non-SiVET participants in both periods before PSM. Similarly, HIV incidence was lower in SiVET than non-SiVET; SiVET-concurrent period, IRR = 0.59, 95% CI 0.31-0.68, p = 0.033 and pre-SiVET period, IRR = 0.77, 95% CI 0.43-1.29, p = 0.161. After PSM, participants baseline characteristics were comparable and there were minimal differences in HIV incidence between SiVET and non-SiVET participants. The process of screening for eligibility for efficacy trial selects participants with baseline characteristics different from the source population, confounding any observed differences in HIV incidence. Propensity score matching can be a useful tool to adjust the imbalance in the measured participants' baseline characteristics creating a counterfactual group to estimate the effect of interventions on HIV incidence.

Clinical effectiveness of a cardiology outpatient management plan to reduce inefficiency in consultations

Objectives: Longer delays in carrying out complementary tests in cardiology services have resulted in patients arriving for consultation without these tests being performed (inefficient consultations). To ameliorate this situation, a management-based intervention was designed, optimizing the available resources and modifying the appointment system. Therefore, our objective was to determine the effectiveness of this intervention to reduce the number of inefficient consultations and improve the clinical care process.Methods: A non-randomized experimental study comparing two periods (pre- and post-intervention) was designed, analyzing a total of 473 outpatients attending cardiology consultations in a Spanish region in February 2014 (pre-intervention) and 441 patients attending cardiology consultations in November 2014 (post-intervention). The outcome of management measures aimed at optimizing coordination in outpatient care to reduce inefficient consultations was analyzed. After the visit, treatment modifications, requests for new examinations or tests, outpatient discharges, and new diagnoses were evaluated.Results: In the pre-intervention period, 37.2% of the patients had not had the tests performed, while in the post-intervention period, this figure dropped to 10.7% (p < 0.001). When the patients had all the tests completed, there was an increase in the number of new examinations (p < 0.001), outpatient discharges (p < 0.001) and new diagnoses (p = 0.004). Treatment modifications were not significant (p = 0.223).Conclusions: The intervention proved effective, clinically relevant, and statistically significant in reducing the proportion of inefficient consultations, thereby enabling continuation of the clinical care process.

The premature closure of ROMPA clinical trial: mortality reduction in septic shock by plasma adsorption

OBJECTIVES

Coupled Plasma Filtration and Adsorption (CPFA) use in septic shock remains controversial. The objective is to clarify whether the application of high doses of CPFA in addition to the current clinical practice could reduce hospital mortality in septic shock patients in Intensive Care Units at 28 days and at 90 days follow-up.

DESIGN

We designed a prospective randomised clinical trial, Reducción de la Mortalidad Plasma-Adsorción (ROMPA), to demonstrate an absolute mortality reduction of 20% (α=0.05; 1-β=0.8; n=190 (95×2)).

SETTING

Being aware of the pitfalls associated with previous medical device trials, we developed a training programme to improve CPFA use (especially clotting problems). The protocol was approved by the ethics committees of all participating centres. Circumstances beyond our control produced a change in recruitment conditions unacceptable to ROMPA researchers and the trial was discontinued.

PARTICIPANTS

By closure, five centres from an initial 10 fulfilled the necessary trial criteria, with 49 patients included, 30 in the control group (CG) and 19 in the intervention group (IG).

INTERVENTION

CPFA.

MAIN OUTCOME MEASURES

Hospital mortality at 28 days and 90 days follow-up.

RESULTS

After 28 days, 14 patients died (46.7%) from the CG and 11 (57.9%) from the IG, not reaching statistical significance (p=0.444). At 90 days, 19 patients had died (63.3%) from the CG and 11 patients (57.9%) from the IG, (p=0.878). The adjustment by propensity score or the use of the Kaplan-Meier technique failed to achieve statistical difference, neither by Intention to Treat nor by the Actual Intervention Received.

CONCLUSION

We herewith present the results gained from the prematurely closed trial. The results are inconclusive due to low statistical power but we consider that this data is of interest for the scientific community and potentially necessary for any ensuing debate.

REGISTER

NCT02357433 in clinicaltrials.gov.

Efficacy, Safety and Long-term Outcomes of Nicorandil Use in Patients with Stable Ischemic Heart Disease According to the Results of Randomized and Observational Studies

Working Group of the NIKEA Study. Yekaterinburg: Akulina E.N., Reznik I.I.; Izhevsk: Grebnev S.A., Yezhov A., Shinkareva S.E.; Krasnodar: Kudryashov E.A., Skibitsky A.V., Skibitsky V.V., Fendrikova A.V.; Krasnoyarsk: Altayev V.D., Matyushin G.V., Nemik D.B., Pitaev R.R., Samokhvalov E.V., Stolbikov Yu.Yu.; Moscow: Balashov I.S., Voronina V.P., Gaisenok O.V., Dmitrieva N.A., Zagrebelny A.V., Zakharova A.V., Zelenova T.I., Kolganova E.V., Leonov A.S., Lerman O.V., Maximova M.A., Sladkova T.A., Shestakova G.N.; Novosibirsk: Kuimov A.D., Shurkevich A.A.; Omsk: Goodilin V.A., Loginova E.N., Nechaeva G.I.; Orel: Zhuravleva L.L., Lobanova G.N., Luneva M.M., Mitroshina T.N.; Orenburg: Kondratenko V.Yu., Libis R.A.; Rostov-on-Don: Dubishcheva N.F., Kalacheva N.M., Kolomatskaya O.E., Romadina G., Skarzhinskaya N.S., Chesnikova A.I., Chugunova I.B.; Ryazan: Dobrynina N.V., Nikolaev A.S., Trofimova Ya.M., Yakushin S.S.; Tula: Berberfish L.D., Gomova T.A., Gorina G.I., Dabizha V.G., Zubareva L.A., Nadezhkina K.N., Nikitina V.F., Renko I.E., Soin I.A., Yunusova K.N.

Background. Nicorandil is an antianginal drug for which, the ability to positively influence the prognosis of patients (pts) with stable ischemic heart disease (IHD) was confirmed in a randomized controlled trial (RCT) of IONA (the Impact Of Nicorandil in Angina). To study whether the results of RCTs are reproduced in real clinical practice seems to be an actual scientific and practical task.

Aim. To compare the data on the effectiveness and safety of nicorandil in pts with stable IHD obtained in the NIKEA observational study (OS) and in the IONA randomized study.

Material and methods. 590 pts with IHD and stable angina pectoris were included in the OS NIKEA. All pts were recommended to take nicorandil in addition to the standard antiischemic therapy. 21 months after being included in the study, 524 pts received a phone call. During the telephone contact with pts or their relatives, the life status of pts was determined. According to these results of the survey data were obtained, that 15 people died and 509 pts were alive. The events included in the primary combined endpoint (PCEP) were also determined: death from all causes, new cases of acute myocardial infarction and acute cerebrovascular accident, unscheduled operations of myocardial revascularization, hospitalization for decompensation of chronic heart failure, atrial fibrillation, unstable angina, information on taking nicorandil and other drug therapy, adverse events of drug treatment have been reported. A comparative analysis of the results of the OS NIKEA and RCT IONA was carried out. The results of the IONA study were taken according to the publication in the Lancet 2002. A comparative analysis of the results of the effectiveness of nicorandil in real practice (according to the OS results) was performed with the data obtained in the RCT: the nicorandil/placebo groups in the RCT were compared with the adherent/non-adherent nicorandil groups in the OS.

Results. The follow-up duration in both studies was similar and averaged 1.6±0.5 years at RCT IONA and 1.8±0.4 years at NIKEA study. The average age of pts was 67,0±8,0 years in RCT and 65.1±9.6 years in OS. In pts of OS more pronounced comorbidity was noted (cardiovascular diseases, diabetes mellitus). Drugs that favorably affect the prognosis in pts with IHD were more often prescribed to NIKEA study pts (p<0.05). In both RCTs and OS, the antianginal effectiveness of nicorandil was confirmed. According to the OS results, a reduction in the number of angina attacks and a decrease in the need for short-acting nitrates were demonstrated. The frequency of PCEP components was higher in RCT.

Conclusion. Long-term outcomes according to the NIKEA observational program for various components of the PCEP turned out to be similar to the results of RCT IONA. It is demonstrated the efficacy of nicorandil in real clinical practice. 

Venous Thromboembolism and Bleeding Adverse Events in Lower Leg, Ankle, and Foot Orthopaedic Surgery with and without Anticoagulants

BACKGROUND

Currently, there is insufficient knowledge about the benefits of anticoagulant use for primary prevention of venous thromboembolism (VTE) and its inherent risk of bleeding adverse events in patients undergoing surgery distal to the knee.

METHODS

The study included patients who had undergone an orthopaedic procedure distal to the tibial articular surface when they were 18 years of age or older. Using retrospective information from a tertiary care referral center, we compared patient demographics, clinical findings, diagnostic reports, procedures performed, and the rate of symptomatic VTE and bleeding adverse events between patients who had and those who had not received anticoagulant prophylaxis. Propensity score matching was used to minimize selection bias due to prophylactic treatment allocation.

RESULTS

A total of 5,286 patients who had received anticoagulant prophylaxis for below-the-knee surgery were successfully matched with 5,286 patients who had not received anticoagulant prophylaxis for such surgery. After propensity score matching, the standardized difference between the groups was <0.1 for all baseline characteristics, indicating a negligible difference between the groups. Patients who received anticoagulant prophylaxis had a significantly lower risk of developing a VTE compared with patients who did not (39 patients [0.7%] versus 99 patients [1.9%]), with an odds ratio (OR) of 0.38 (95% confidence interval [CI], 0.25 to 0.56; p < 0.001). In contradistinction, patients who received anticoagulant prophylaxis had a significantly higher risk of developing a bleeding adverse event than those who did not (115 [2.2%] versus 55 [1.0%]; OR, 2.18 [95% CI, 1.55 to 3.09]; p < 0.001).

CONCLUSIONS

Anticoagulant prophylaxis reduced the risk of VTE after surgery distal to the tibial articular surface by 3-fold but resulted in a concomitant 2-fold increase in the risk of a bleeding adverse event. Large-scale, prospective studies are necessary to better understand the true incidence of such events, associated patient-specific risk factors, efficacy of various thromboprophylactic regimens, and patient-reported implications of such events.

LEVEL OF EVIDENCE

Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.

A narrative review of data collection and analysis guidelines for comparative effectiveness research in chronic pain using patient-reported outcomes and electronic health records

Chronic pain is a widespread and complex set of conditions that are often difficult and expensive to treat. Comparative effectiveness research (CER) is an evolving research method that is useful in determining which treatments are most effective for medical conditions such as chronic pain. An underutilized mechanism for conducting CER in pain medicine involves combining patient-reported outcomes (PROs) with electronic health records (EHRs). Patient-reported pain and mental and physical health outcomes are increasingly collected during clinic visits, and these data can be linked to EHR data that are relevant to the treatment of a patient's pain, such as diagnoses, medications ordered, and medical comorbidities. When aggregated, this information forms a data repository that can be used for high-quality CER. This review provides a blueprint for conducting CER using PROs combined with EHRs. As an example, the University of Pittsburgh's patient outcomes repository for treatment is described. This system includes PROs collected via the Collaborative Health Outcomes Information Registry software and cross-linked data from the University of Pittsburgh Medical Center EHR. The requirements, best practice guidelines, statistical considerations, and caveats for performing CER with this type of data repository are also discussed.

Impact of antibiotics for children presenting to general practice with cough on adverse outcomes: secondary analysis from a multicentre prospective cohort study

Background

Clinicians commonly prescribe antibiotics to prevent major adverse outcomes in children presenting in primary care with cough and respiratory symptoms, despite limited meaningful evidence of impact on these outcomes.

Aim

To estimate the effect of children’s antibiotic prescribing on adverse outcomes within 30 days of initial consultation.

Design and setting

Secondary analysis of 8320 children in a multicentre prospective cohort study, aged 3 months to <16 years, presenting in primary care across England with acute cough and other respiratory symptoms.

Method

Baseline clinical characteristics and antibiotic prescribing data were collected, and generalised linear models were used to estimate the effect of antibiotic prescribing on adverse outcomes within 30 days (subsequent hospitalisations and reconsultation for deterioration), controlling for clustering and clinicians’ propensity to prescribe antibiotics.

Results

Sixty-five (0.8%) children were hospitalised and 350 (4%) reconsulted for deterioration. Clinicians prescribed immediate and delayed antibiotics to 2313 (28%) and 771 (9%), respectively. Compared with no antibiotics, there was no clear evidence that antibiotics reduced hospitalisations (immediate antibiotic risk ratio [RR] 0.83, 95% confidence interval [CI] = 0.47 to 1.45; delayed RR 0.70, 95% CI = 0.26 to 1.90, overall P = 0.44). There was evidence that delayed (rather than immediate) antibiotics reduced reconsultations for deterioration (immediate RR 0.82, 95% CI = 0.65 to 1.07; delayed RR 0.55, 95% CI = 0.34 to 0.88, overall P = 0.024).

Conclusion

Most children presenting with acute cough and respiratory symptoms in primary care are not at risk of hospitalisation, and antibiotics may not reduce the risk. If an antibiotic is considered, a delayed antibiotic prescription may be preferable as it is likely to reduce reconsultation for deterioration.