Value of surgical pilot and feasibility study protocols

Pilot and feasibility trials in surgery are incompletely reported according to the CONSORT checklist: a meta-research study

OBJECTIVES

Given the key role that pilot and feasibility (PAF) trials play in addressing the challenges of surgical trials, adequate reporting completeness is essential. Our aim was to assess completeness of reporting and evaluate if the items of the Consolidated Standards of Reporting Trials (CONSORT) extension for PAF trials have been reported in surgical PAF trials.

STUDY DESIGN

This is a metaresearch study reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Studies were included if they were pilot or feasibility randomized trials evaluating a surgical intervention. The primary outcome was overall adherence to the CONSORT statement extension to randomized PAF trials checklist. A Poisson regression was performed to explore the association between research factors and reporting completeness.

SETTING

MEDLINE, Embase, and Cochrane Central Register of Controlled Trials were searched from January 1-December 31, 2011 and 2021.

RESULTS

After screening 1991 citations, 38 studies from 2011 to 34 studies from 2021 were included. The mean CONSORT reporting score across all included studies was 21.5 (standard deviation 6.3). After excluding items that were not applicable to all studies, a mean of 20.1 (standard deviation 6.1) of 34 items (0.59) were reported. Studies published in 2021 (vs 2011) did not have a greater number of CONSORT items reported (incidence rate ratio [IRR] 1.01, 95% confidence interval [CI] 0.89-1.15). Studies registered in a clinical trial registry (IRR 1.29, 95% CI 1.12-1.48) and randomizing more than 50 patients (IRR 1.16, 95% CI 1.04-1.30) were associated with more CONSORT items reported.

CONCLUSION

The reporting completeness of surgical PAF trials is poor and has not improved after the publication of the CONSORT extension.

Adaptive clinical trials in surgery: A scoping review of methodological and reporting quality

IMPORTANCE

Adaptive surgical trials are scarce, but adopting these methods may help elevate the quality of surgical research when large-scale RCTs are impractical.

OBJECTIVE

Randomized-controlled trials (RCTs) are the gold standard for evidence-based healthcare. Despite an increase in the number of RCTs, the number of surgical trials remains unchanged. Adaptive clinical trials can streamline trial design and time to trial reporting. The advantages identified for ACTs may help to improve the quality of future surgical trials. We present a scoping review of the methodological and reporting quality of adaptive surgical trials.

EVIDENCE REVIEW

We performed a search of Ovid, Web of Science, and Cochrane Collaboration for all adaptive surgical RCTs performed from database inception to October 12, 2023. We included any published trials that had at least one surgical arm. All review and abstraction were performed in duplicate. Risk of bias (RoB) was assessed using the RoB 2.0 instrument and reporting quality was evaluated using CONSORT ACE 2020. All results were analyzed using descriptive methods.

FINDINGS

Of the 1338 studies identified, six trials met inclusion criteria. Trials were performed in cardiothoracic, oral, orthopedic, and urological surgery. The most common type of adaptive trial was group sequential design with pre-specified interim analyses planned for efficacy, futility, and/or sample size re-estimation. Two trials did use statistical simulations. Our risk of bias evaluation identified a high risk of bias in 50% of included trials. Reporting quality was heterogeneous regarding trial design and outcome assessment and details in relation to randomization and blinding concealment.

CONCLUSION AND RELEVANCE

Surgical trialists should consider implementing adaptive components to help improve patient recruitment and reduce trial duration. Reporting of future adaptive trials must adhere to existing CONSORT ACE 2020 guidelines. Future research is needed to optimize standardization of adaptive methods across medicine and surgery.

REporting quality of PilOt randomised controlled trials in surgery (REPORTS): a methodological survey protocol

INTRODUCTION

The aim of this methodological review is to evaluate the completeness of reporting of surgical pilot and feasibility randomised trials as per the Consolidated Standards of Reporting Trials (CONSORT) extension to randomised pilot and feasibility trials. Moreover, we aim to assess for the presence of spin reporting and inconsistency between abstract and main text reporting in surgical pilot and feasibility randomised trials.

METHODS AND ANALYSIS

A comprehensive, electronic search strategy will be used to identify studies indexed in Medline, Embase and Cochrane Central Register of Controlled Trials (CENTRAL) databases. Studies will be included if they are pilot or feasibility randomised trials of surgical interventions. The primary outcome will be overall CONSORT statement extension to randomised pilot and feasibility trials checklist completeness. This will be defined as trials reporting each of the 40 items in the CONSORT statement extension to randomised pilot and feasibility trials checklist. Secondary outcomes will include the reporting of individual studies as per the CONSORT extension to randomised pilot and feasibility trials, the use of spin reporting strategies, trial factors associated with reporting quality and spin strategy use, and consistency between abstract and main text reporting. Poisson and logistic regressions will be performed to explore the association between trial factors and completeness of reporting as measured by the number of reported CONSORT items.

ETHICS AND DISSEMINATION

This is a methodological survey that has been registered a priori on the International Prospective Register for Systematic Reviews (PROSPERO) (CRD42023475512). Local ethics approval is not required. We plan to disseminate study results through peer-reviewed publication and conference presentations.

Recommendations for optimising pilot and feasibility work in surgery

BACKGROUND

Surgical trials are recognised as inherently challenging. Pilot and feasibility studies (PAFS) are increasingly acknowledged as a key method to optimise the design and conduct of randomised trials but remain limited in surgery. We used a mixed methods approach to develop recommendations for how surgical PAFS could be optimised.

METHODS

The findings from a quantitative analysis of funded surgical PAFS over a 10-year period and in-depth qualitative interviews with surgeons, methodologists and funders were triangulated and synthesised with available methodological guidance on PAFS.

RESULTS

The synthesis informed the development of an explanatory model describing root causes and compounding challenges that contribute to how and why surgical PAFS is not currently optimised. The four root causes identified include issues relating to (i) understanding the full scope of PAFS; (ii) design and conduct of PAFS; (iii) reporting of PAFS; and (iv) lack of appreciation of the value of PAFS by all stakeholder groups. Compounding challenges relate to both cultural issues and access to and interpretation of available methodological PAFS guidance. The study findings and explanatory model were used to inform the development of a practical guidance tool for surgeons and study teams to improve research practice.

CONCLUSIONS

Optimisation of PAFS in surgery requires a cultural shift in research practice amongst funders, academic institutions, regulatory bodies and journal editors, as well as amongst surgeons. Our 'Top Tips' guidance tool offers an accessible framework for surgeons designing PAFS. Adoption and utilisation of these recommendations will optimise surgical PAFS, facilitating successful and efficient future surgical trials.

How do pilot and feasibility studies inform randomised placebo-controlled trials in surgery? A systematic review

INTRODUCTION

Randomised controlled trials (RCTs) with a placebo comparator are considered the gold standard study design when evaluating healthcare interventions. These are challenging to design and deliver in surgery. Guidance recommends pilot and feasibility work to optimise main trial design and conduct; however, the extent to which this occurs in surgery is unknown.

METHOD

A systematic review identified randomised placebo-controlled surgical trials. Articles published from database inception to 31 December 2020 were retrieved from Ovid-MEDLINE, Ovid-EMBASE and CENTRAL electronic databases, hand-searching and expert knowledge. Pilot/feasibility work conducted prior to the RCTs was then identified from examining citations and reference lists. Where studies explicitly stated their intent to inform the design and/or conduct of the future main placebo-controlled surgical trial, they were included. Publication type, clinical area, treatment intervention, number of centres, sample size, comparators, aims and text about the invasive placebo intervention were extracted.

RESULTS

From 131 placebo surgical RCTs included in the systematic review, 47 potentially eligible pilot/feasibility studies were identified. Of these, four were included as true pilot/feasibility work. Three were original articles, one a conference abstract; three were conducted in orthopaedic surgery and one in oral and maxillofacial surgery. All four included pilot RCTs, with an invasive surgical placebo intervention, randomising 9-49 participants in 1 or 2 centres. They explored the acceptability of recruitment and the invasive placebo intervention to patients and trial personnel, and whether blinding was possible. One study examined the characteristics of the proposed invasive placebo intervention using in-depth interviews.

CONCLUSION

Published studies reporting feasibility/pilot work undertaken to inform main placebo surgical trials are scarce. In view of the difficulties of undertaking placebo surgical trials, it is recommended that pilot/feasibility studies are conducted, and more are reported to share key findings and optimise the design of main RCTs.

PROSPERO REGISTRATION NUMBER

CRD42021287371.

Methods of Recruitment for Surgical and Perioperative Randomized Controlled Trials: A Rapid Review

Due to the complex nature of surgical randomized controlled trials (RCTs), reaching target recruitment can be challenging. The primary objective was to report on characteristics of successful pilot surgical and perioperative RCTs and the methodological strategies implemented to optimize recruitment. The secondary objective was to provide recommendations for successful recruitment strategies for future surgical RCTs. Ovid MEDLINE, Ovid EMBASE, and Web of Science (via Ovid) databases were searched from 2012 to 2022. This review included surgical and perioperative pilot studies that met their recruitment targets. Study and recruitment characteristics were summarized, and potential relationships between study design and recruitment rate were assessed. Optimized recruitment strategies were extracted when reported. Of 4156 total articles identified, 255 underwent full-text screening, and 52 articles were included. Of the included pilot studies, 21% (n = 11) did not indicate a target sample size or recruitment rate. Recruitment methods were minimally reported in pilot studies for perioperative or surgical RCTs. Strategies to optimize recruitment included internal iterative evaluations of the recorded recruitment appointments and staged introduction of the study. Recruitment rate was not associated with invasiveness of intervention or burden of participation. Patient involvement is absent from current reports on methodological design and offers valuable opportunity to optimize recruitment. Recruitment strategies in perioperative and surgical RCTs can be optimized with iterative qualitative evaluation of the recruitment methods with input from the interdisciplinary research team.

Determining the Effectiveness of Fibrin Sealants in Reducing Complications in Patients Undergoing Lateral Neck Dissection (DEFeND): A Randomised External Pilot Trial

OBJECTIVES

High-quality randomised controlled trials (RCT) to support the use of Fibrin Sealants (FS) in neck dissection (ND) are lacking. The DEFeND trial assessed critical pilot/feasibility questions and signals from clinical outcomes to inform a future definitive trial.

PATIENTS AND METHODS

The study design piloted was a blinded surgical RCT. All participants underwent unilateral ND for head and neck cancer. Interventional arm: ND with application of FS.

CONTROL ARM

ND alone. Feasibility outcomes included recruitment, effectiveness of blinding, protocol adherence and evaluating administrative processes. Clinical outcomes included surgical complications (primary outcome), drainage volume, time to drain removal, length of hospital stay, pain and the Neck Dissection Impairment Index.

RESULTS

Recruitment completed ahead of time. Fifty-three patients were recruited, and 48 were randomised at a rate of 5.3 patients/month. Blinding of patients, research nurses and outcome assessors was effective. Two protocol deviations occurred. Two patients were lost to follow-up. The mean (SD) Comprehensive Complication Index in the interventional arm was 6.5 (12.8), and it was 9.9 (14.2) in the control arm. The median (IQR) time to drain removal (days) was shorter in the interventional arm (2.67 (2.42, 3.58) vs. 3.40 (2.50, 4.27)). However, this did not translate to a clinically significant reduction in median (IQR) length of hospital stay in days (intervention: 3.48 (2.64, 4.54), control: 3.74 (3.11, 4.62)).

CONCLUSION

The proposed trial design was effective, and a definitive surgical trial is feasible. Whilst there was a tendency for FS to improve clinical outcomes, the effect size did not reach clinical or statistical significance. (ISRCTN99181100).

Randomized controlled trials and alternative study designs in surgical oncology

Surgery is central to the cure of most solid cancers and an integral part of modern multimodal cancer management for early and advanced stage cancers. Decisions made by surgeons and multidisciplinary team members are based on best available knowledge for the defined clinical situation at hand. While surgery is both an art and a science, good decision-making requires data that are robust, valid, representative and, applicable to most if not all patients with a specific cancer. Such data largely comes from clinical observations and registries, and more preferably from trials conducted with the specific purpose of arriving at new answers. As part of the ESSO core curriculum development an increased focus has been put on the need to enhance research literacy among surgical candidates. As an expansion of the curriculum catalogue list and to enhance the educational value, we here present a set of principles and emerging concepts which applies to surgical oncologist for reading, understanding, planning and contributing to future surgeon-led cancer trials.

Inclusion of progression criteria in external randomised pilot trials: a cross-sectional study of funding applications submitted to the NIHR Research for Patient Benefit Programme

BACKGROUND

External randomised pilot trials aim to assess whether a future definitive randomised controlled trial (RCT) is feasible. Pre-specified progression criteria help guide the interpretation of pilot trial findings to decide whether, and how, a definitive trial should be conducted. We aimed to examine how researchers report and plan to assess progression criteria in external pilot trial funding applications submitted to the NIHR Research for Patient Benefit Programme.

METHODS

We conducted a cross-sectional study of progression criteria inclusion in Stage 1 (outline) and corresponding Stage 2 (full) funding applications for external randomised external pilot trials submitted to NIHR RfPB between July 2017 and July 2019.

RESULTS

Of the 100 Stage 1 outline applications assessed, 95 were eligible for inclusion (of these, 52 were invited to Stage 2 full application; 43 were rejected) and 49/52 were eligible for inclusion at Stage 2 full application (of these, 35 were awarded funding; 14 were rejected). Over half of applications assessed at Stage 1 (48/95, 51%), and 73% of those assessed at Stage 2 (36/49) included progression criteria in their research plans. Progression criteria were most often reported in a stop-go format, often with additional specified factors that should be considered when determining feasibility (Stage 1 33/48, 69%; Stage 2 21/36, 58%). Recruitment and retention were the most frequent indicators of feasibility to inform progression criteria. One-third of applications provided some justification or rationale for their targets (Stage 1 16/48, 33%; Stage 2 12/36, 33%). Funding committee feedback mentioned progression criteria in over 20% of applications (Stage 1 22/95, 23%; Stage 2 11/49, 22%) to either request the addition of progression criteria or provide justification for the criteria stipulated.

CONCLUSIONS

Our findings indicate that researchers do not always include progression criteria in external randomised pilot trial applications submitted to research funders. This can result in a lack of transparency in the assessment of randomised pilot trial feasibility.

TRIAL REGISTRATION

Open Science Framework osf.io/89ap7, registered 29th June 2021.

Intensive physical therapy after emergency laparotomy: Pilot phase of the Incidence of Complications following Emergency Abdominal surgery Get Exercising randomized controlled trial

BACKGROUND

Postoperative pneumonia and delayed physical recovery are significant problems after emergency laparotomy. No randomized controlled trial has assessed the feasibility, safety, or effectiveness of intensive postoperative physical therapy in this high-risk acute population.

METHODS

The internal pilot phase of the Incidence of Complications after Emergency Abdominal Surgery: Get Exercising (ICEAGE) trial was a prospective, randomized controlled trial that evaluated the feasibility, safety, and clinical trial processes of providing intensive physical therapy immediately following emergency laparotomy. Fifty consecutive patients were recruited at the principal participating hospital and randomly assigned to standard-care or intensive physical therapy of twice daily coached breathing exercises for 2 days and 30 minutes of daily supervised rehabilitation over the first 5 postoperative days.

RESULTS

Interventions were provided exactly as per protocol in 35% (78 of 221 patients) of planned treatment sessions. Main barriers to protocol delivery were physical therapist unavailability on weekends (59 of 221 patients [27%]), awaiting patient consent (18 of 99 patients [18%]), and patient fatigue (26 of 221 patients [12%]). Despite inhibitors to treatment delivery, the intervention group still received twice as many breathing exercise sessions and four times the amount of physical therapy over the first 5 postoperative days (23 minutes [interquartile range, 12-29 minutes] vs. 86 minutes [interquartile range, 53-121 minutes]; p < 0.001). One adverse event was reported from 78 rehabilitation sessions (1.3%), which resolved fully on cessation of activity without escalation of medical care.

CONCLUSION

Intensive postoperative physical therapy can be delivered safely and successfully to patients in the first week after emergency laparotomy. The ICEAGE trial protocol resulted in intervention group participants receiving more coached breathing exercises and spending significantly more time physically active over the first 5 days after surgery compared with standard care. It was therefore recommended to progress into the multicenter phase of ICEAGE to definitively test the effect of intensive physical therapy to prevent pneumonia and improve physical recovery after emergency laparotomy.

LEVEL OF EVIDENCE

Therapeutic/Care Management; Level II.

The use of cognitive task analysis in clinical and health services research — a systematic review

Background

At times, clinical case complexity and different types of uncertainty present challenges to less experienced clinicians or the naive application of clinical guidelines where this may not be appropriate. Cognitive task analysis (CTA) methods are used to elicit, document and transfer tacit knowledge about how experts make decisions.

Methods

We conducted a methodological review to describe the use of CTA methods in understanding expert clinical decision-making. We searched MEDLINE, EMBASE and PsycINFO from inception to 2019 for primary research studies which described the use of CTA methods to understand how qualified clinicians made clinical decisions in real-world clinical settings.

Results

We included 81 articles (80 unique studies) from 13 countries, published from 1993 to 2019, most commonly from surgical and critical care settings. The most common aims were to understand expert decision-making in particular clinical scenarios, using expert decision-making in the development of training programmes, understanding whether decision support tools were warranted and understanding procedural variability and error identification or reduction. Critical decision method (CDM) and CTA interviews were most frequently used, with hierarchical task analysis, task knowledge structures, think-aloud protocols and other methods less commonly used. Studies used interviews, observation, think-aloud exercises, surveys, focus groups and a range of more CTA-specific methodologies such as the systematic human error reduction and prediction approach. Researchers used CTA methods to investigate routine/typical (n = 64), challenging (n = 13) or more uncommon, rare events and anomalies (n = 3).

Conclusions

In conclusion, the elicitation of expert tacit knowledge using CTA has seen increasing use in clinical specialties working under challenging time pressures, complexity and uncertainty. CTA methods have great potential in the development, refinement, modification or adaptation of complex interventions, clinical protocols and practice guidelines.

Registration

PROSPERO ID CRD42019128418.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-01002-6.

Are some feasibility studies more feasible than others? A review of the outcomes of feasibility studies on the ISRCTN registry

Background

Feasibility studies are often conducted before committing to a randomised controlled trial (RCT), yet there is little published evidence to inform how useful feasibility studies are, especially in terms of adding or reducing waste in research. This study attempted to examine how many feasibility studies demonstrated that the full trial was feasible and whether some feasibility studies were inherently likely to be feasible or not feasible, based on the topic area and/or research setting.

Methods

Keyword searches were conducted on the International Standard Randomised Controlled Trials Number (ISRCTN) registry to identify all completed feasibility studies which had been conducted in the UK.

Results

A total of 625 records from the 1933 identified were reviewed before it became evident that it would be futile to continue. Of 329 feasibility studies identified, 160 had a known outcome (49%), 133 (83%) trials were deemed to be feasible and only 27 (17%) were reported to be non-feasible. There were therefore too few studies to allow the intended comparison of differences in non-feasible studies by topic and/or setting.

Conclusions

There were too few studies reported as non-feasible to draw any useful conclusions on whether topic and/or setting had an effect. However, the high feasibility rate (83%) may suggest that non-feasible studies are subject to publication bias or that many feasible studies are redundant and may be adding waste to the research pathway.

Methodological quality for systematic reviews of adverse events with surgical interventions: a cross-sectional survey

BACKGROUND

An increasing number of systematic reviews assessed the safety of surgical interventions over time. How well these systematic reviews were designed and conducted determines the reliability of evidence. In this study, we aimed to assess the methodological quality of systematic reviews on the safety of surgical interventions.

METHODS

We searched PubMed for systematic reviews of surgical interventions with safety as the exclusive outcome from 1st-Jan, 2015 to 1st-Jan, 2020. The methodological quality of eligible systematic reviews was evaluated according to the AMSTAR 2.0 instrument. The primary outcomes were the number of methodological weaknesses and the global methodological quality. The proportion of each methodological weakness among eligible systematic reviews was compared by three pre-defined stratification variables. The absolute difference of the proportion (PD) was used as the effect estimator, with the two-tailed z-test for the significance.

RESULTS

We identified 127 systematic reviews from 18,636 records. None (n = 0, 0.00%) of them could be rated as "high" in terms of the global methodological quality; in contrast, they were either rated as "low" (n = 18, 14.17%) or as "critically low" (n = 109, 85.83%). The median number of methodological weaknesses of these systematic reviews was 8 (interquartile range, IQR: 6 to 9), in which 4 (IQR: 2 to 4) were critical weaknesses. Systematic reviews that used any reporting guideline (e.g., domain 13, PD = -0.22, 95% CI: - 0.39, - 0.06; p = 0.01) and developed a protocol in advance (e.g., domain 6, PD = -0.20, 95% CI: - 0.39, - 0.01; p = 0.04) were less likely to have methodological weakness in some domains but not for the rest (e.g., domain 8, PD = 0.04, 95% CI: - 0.14, 0.21; p = 0.68; with protocol vs. without).

CONCLUSIONS

The methodological quality of current systematic reviews of adverse events with surgical interventions was poor. Further efforts, for example, encouraging researchers to develop a protocol in advance, are needed to enhance the methodological quality of these systematic reviews.

Progression from external pilot to definitive randomised controlled trial: a methodological review of progression criteria reporting

OBJECTIVES

Prespecified progression criteria can inform the decision to progress from an external randomised pilot trial to a definitive randomised controlled trial. We assessed the characteristics of progression criteria reported in external randomised pilot trial protocols and results publications, including whether progression criteria were specified a priori and mentioned in prepublication peer reviewer reports.

STUDY DESIGN

Methodological review.

METHODS

We searched four journals through PubMed: British Medical Journal Open, Pilot and Feasibility Studies, Trials and Public Library of Science One. Eligible publications reported external randomised pilot trial protocols or results, were published between January 2018 and December 2019 and reported progression criteria. We double data extracted 25% of the included publications. Here we report the progression criteria characteristics.

RESULTS

We included 160 publications (123 protocols and 37 completed trials). Recruitment and retention were the most frequent indicators contributing to progression criteria. Progression criteria were mostly reported as distinct thresholds (eg, achieving a specific target; 133/160, 83%). Less than a third of the planned and completed pilot trials that included qualitative research reported how these findings would contribute towards progression criteria (34/108, 31%). The publications seldom stated who established the progression criteria (12/160, 7.5%) or provided rationale or justification for progression criteria (44/160, 28%). Most completed pilot trials reported the intention to proceed to a definitive trial (30/37, 81%), but less than half strictly met all of their progression criteria (17/37, 46%). Prepublication peer reviewer reports were available for 153/160 publications (96%). Peer reviewer reports for 86/153 (56%) publications mentioned progression criteria, with peer reviewers of 35 publications commenting that progression criteria appeared not to be specified.

CONCLUSIONS

Many external randomised pilot trial publications did not adequately report or propose prespecified progression criteria to inform whether to proceed to a future definitive randomised controlled trial.

Fluorescence or X-ray cholangiography in elective laparoscopic cholecystectomy: a randomized clinical trial

BACKGROUND

Safe laparoscopic cholecystectomy may necessitate biliary imaging, and non-invasive fluorescence cholangiography may have advantages over contrast X-ray cholangiography. This trial compared fluorescence and X-ray cholangiography for visualization of the critical junction between the cystic, common hepatic and common bile ducts.

METHODS

This non-inferiority blinded RCT included patients who had either intraoperative fluorescence cholangiography using 0·05 mg/kg indocyanine green or X-ray cholangiography during elective laparoscopic cholecystectomy.

RESULTS

Between March 2015 and August 2018, a total of 120 patients were randomized (60 in each group). There were no drop-outs and 30-day follow-up data were available for all patients. In intention-to-treat analysis, there was no difference between the fluorescence and X-ray cholangiography groups in ability to visualize the critical junction (49 of 60 versus 51 of 60 respectively; P = 0·230). Fluorescence cholangiography was faster by a few minutes: median 2·0 (range 0·5-5·0) versus 4·8 (1·3-17·6) min (P < 0·001).

CONCLUSION

Fluorescence cholangiography was confirmed to be non-inferior to X-ray cholangiography in visualizing the critical junction during laparoscopic cholecystectomy. Registration number: NCT02344654 ( http://www.clinicaltrials.gov).