Assessment and classification of protocol deviations

Quality Management in Clinical and Public Health Research: A Panacea for Minimising and Eliminating Protocol Deviations in Research Operations

A quality management system for clinical and public health research operations is indispensable because it ensures the integrity and reliability of research outcomes. By implementing a robust quality management practice in research implementation and operation, research teams can uphold the highest standard of research conduct, thereby enhancing the credibility and trustworthiness of research findings. This paper elucidates the significance and role of a quality management system in clinical and public health research operations and its efficacy in minimising and eliminating protocol deviations and highlights the key steps in setting up a quality management system for research operations.

Preventing Potential Patient Harm Through Clinical Content Interventions During Oncology Clinical Trial Implementation

OBJECTIVES

Clinical trial documents are complex and may have inconsistencies, leading to potential site implementation errors and may compromise participant safety. This study characterizes the frequency and type of administrative and potential patient safety interventions (PPSIs) made during the review of oncology trial documents for clinical trial implementation by centralized clinical content specialists.

METHODS

A dedicated group of centralized clinical content specialists reviewed trial documents, including the protocol, laboratory manual, and pharmacy/cellular therapy manual, and collected intervention data over a 1-year period. Each trial was categorized by study phase and sponsor type, and multiple interventions could be identified per trial. Interventions were deemed administrative or PPSIs, with PPSIs further subcategorized as medication, laboratory, procedure related, or other.

RESULTS

Of 585 clinical trials reviewed, 269 (46%) required intervention(s). Among 1001 interventions, 171 (17.1%) were PPSIs. Most PPSIs were medication related (45.6%), with drug dosing interventions most frequently identified (53.8%). Phase 1 trials had the highest proportion of PPSIs (0.35:1) and administrative interventions (2:1) per trial compared with all other phases. Investigator-initiated trials saw the highest proportion of PPSIs per trial (0.44:1) of all sponsor types.

CONCLUSIONS

This study demonstrates a gap in patient safety when assessing trial documents for clinical trial implementation. One solution to address this gap is the utilization of a centralized team of clinical specialists to preemptively review trial documents, thereby enhancing patient safety during clinical trial conduct.

Unreported protocol deviations - The tip of the research-berg

Introduction

Failure to stay within an ethics committee (EC)-approved protocol limits is termed protocol deviation or violation (PD/PV), depending on the seriousness of the transgression and its attendant risks and/or harms. PD/PVs arise in the post-approval phase of the research and are often missed. Current guidelines expect ECs to detect, report and recommend suitable actions such that research participants' risks and harms are mitigated, to the extent possible.

Objective

Yenepoya Ethics Committee-1 conducted an internal audit of ongoing postgraduate dissertations involving human participants to assess the occurrence of PD/PVs.

Materials and Methods

54 out of 80 postgraduates responded to our request for filling out a self-reported checklist. These responses were followed up with physical verification of the protocol-related documents.

Results

Protocol transgressions were classified as non-compliance (administrative issues), protocol deviations (minor transgressions with minimal or less than minimal increase in attendant risk to participants) and protocol violations (serious transgressions with more than minimal increase in attendant risk to participants). The non-compliances included non-reporting for audit and non-reporting of PDs. Protocol deviations included non-conformance to EC validity, sample size, approved methodology, informed consent process and documentation and suboptimal data storage. No protocol violations were observed.

Conclusion

We report PD/PVs from these 54 protocols - with our assessment on the negative impact it may have on scientific validity, harm to participants, EC functioning and credibility of the institution - in the hope that our readers appreciate this important aspect of the post-approval process in the functioning of an EC.

A Review of the Metrics Used to Assess Auto-Contouring Systems in Radiotherapy

Auto-contouring could revolutionise future planning of radiotherapy treatment. The lack of consensus on how to assess and validate auto-contouring systems currently limits clinical use. This review formally quantifies the assessment metrics used in studies published during one calendar year and assesses the need for standardised practice. A PubMed literature search was undertaken for papers evaluating radiotherapy auto-contouring published during 2021. Papers were assessed for types of metric and the methodology used to generate ground-truth comparators. Our PubMed search identified 212 studies, of which 117 met the criteria for clinical review. Geometric assessment metrics were used in 116 of 117 studies (99.1%). This includes the Dice Similarity Coefficient used in 113 (96.6%) studies. Clinically relevant metrics, such as qualitative, dosimetric and time-saving metrics, were less frequently used in 22 (18.8%), 27 (23.1%) and 18 (15.4%) of 117 studies, respectively. There was heterogeneity within each category of metric. Over 90 different names for geometric measures were used. Methods for qualitative assessment were different in all but two papers. Variation existed in the methods used to generate radiotherapy plans for dosimetric assessment. Consideration of editing time was only given in 11 (9.4%) papers. A single manual contour as a ground-truth comparator was used in 65 (55.6%) studies. Only 31 (26.5%) studies compared auto-contours to usual inter- and/or intra-observer variation. In conclusion, significant variation exists in how research papers currently assess the accuracy of automatically generated contours. Geometric measures are the most popular, however their clinical utility is unknown. There is heterogeneity in the methods used to perform clinical assessment. Considering the different stages of system implementation may provide a framework to decide the most appropriate metrics. This analysis supports the need for a consensus on the clinical implementation of auto-contouring.

A review of reproducible and transparent research practices in urology publications from 2014 to2018

Background

Reproducibility is essential for the integrity of scientific research. Reproducibility is measured by the ability of different investigators to replicate the outcomes of an original publication using the same materials and procedures. Unfortunately, reproducibility is not currently a standard being met by most scientific research.

Methods

For this review, we sampled 300 publications in the field of urology to assess for 14 indicators of reproducibility including material availability, raw data availability, analysis script availability, pre-registration information, links to protocols, and if the publication was available free to the public. Publications were also assessed for statements about conflicts of interest and funding sources.

Results

Of the 300 sample publications, 171 contained empirical data available for analysis of reproducibility. Of the 171 articles with empirical data to analyze, 0.58% provided links to protocols, 4.09% provided access to raw data, 3.09% provided access to materials, and 4.68% were pre-registered. None of the studies provided analysis scripts. Our review is cross-sectional in nature, including only PubMed indexed journals-published in English-and within a finite time period. Thus, our results should be interpreted in light of these considerations.

Conclusion

Current urology research does not consistently provide the components needed to reproduce original studies. Collaborative efforts from investigators and journal editors are needed to improve research quality while minimizing waste and patient risk.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12894-022-01059-8.

Development of the CLARIFY (CheckList stAndardising the Reporting of Interventions For Yoga) guidelines: a Delphi study

BACKGROUND

The use of yoga as a therapeutic modality is increasing; however, a lack of transparent intervention reporting is restricting the dissemination and implementation of yoga research into clinical and community practice. The aim of this study was to develop a yoga-specific reporting guideline as an extension to existing reporting guidelines for randomised controlled trials, observational studies and case reports.

METHODS

Recognised international stakeholders in the design and conduct of yoga research were invited to contribute to the electronic Delphi survey. A four-round Delphi was conducted, whereby panellists rated selected items for their importance in the inclusion of yoga reporting guidelines, according to a 5-step Likert scale. A priori consensus for item inclusion was agreement of items as 'Very important' or 'Extremely important' by ≥80% of panellists. Non-consensus items were forwarded to subsequent rounds for re-rating.

RESULTS

53 experts in yoga research from 11 countries, primarily identifying as researchers (50%), allied health professionals (18.8%) and yoga professionals (12.5%), consented to participate in the Delphi. Of these, 48 completed Round 1 (91%), 43 completed Round 2 (81%), 39 completed Round 3 (74%) and 32 completed Round 4 (60%). Panellists reached consensus for inclusion on 21 items, grouped under 10 domains reflective of more generic intervention-based guidelines.

CONCLUSIONS

The consensus-based 21-item CLARIFY (CheckList stAndardising the Reporting of Interventions For Yoga) checklist provides a minimum reporting template for researchers across a range of methodology designs. Use of these yoga-specific guidelines, in conjunction with the CLARIFY explanation and elaboration guidelines, will standardise the minimum level of detail required for transparent yoga intervention, facilitating the replication, dissemination and implementation of yoga research. Ongoing research will assess the uptake and impact of CLARIFY, to ensure these guidelines retain their relevance to the internationally growing field of yoga research.

Compliance of Clinical Trial Protocols for Foods with Function Claims (FFC) in Japan: Consistency between Clinical Trial Registrations and Published Reports

Background: A new type of foods with a health claims notification system, the Foods with Function Claims (FFC), was introduced in Japan in April 2015. This cross-sectional study sought to clarify compliance of clinical trial protocols reported as the scientific basis of efficacy in the FFC system. Methods: All articles based on clinical trials published on the Consumer Affairs Agency website from 1 July 2018 to 30 June 2021 were reviewed. Items assessed included first author characteristics (for-profit or academia), journal name, year published, journal impact factor in 2020, article language, name of clinical trial registration (CTR), and seven compliance items (Title: T, Participant: P, Intervention: I, Comparison: C, Outcome: O, Study design: S, and Institutional Review Board, IRB). Among studies that conducted CTR, consistency with these seven compliance items was evaluated. Results: Out of 136 studies that met all inclusion criteria, 103 (76%) performed CTR, and CTR was either not performed or not specified for 33 (24%). Compliance between the protocol and the text was high (≥96%) for items P and S, but considerably lower for items T, I, C, O, and IRB (52%, 15%, 13%, 69%, and 27%, respectively). Furthermore, 43% of protocols did not include functional ingredients or food names in items T or I. The total score was 3.7 ± 1.1 pts (out of 7). Conclusions: Some CTs had no protocol registration, and even registered protocols were suboptimal in transparency. In addition to selective reporting, a new problem identified was that the content of the intervention (test food) was intentionally concealed.

Reducing Events of Noncompliance in Neurology Human Subjects Research: the Effect of Human Subjects Research Protection Training and Site Initiation Visits

In an effort to minimize protocol noncompliance in neurological research studies that can potentially compromise patient safety, delay completion of the study, and result in premature termination and added costs, we determined the effect of investigator trainings and site initiation visits (SIVs) on the occurrence of noncompliance events. Results of protocol audits conducted at the National Institute of Neurological Disorders and Stroke from 2003 to 2019 on 97 research protocols were retrospectively analyzed. Based on the depth of auditing and provision of investigator research training, audit data were separated into four arms: 1) Early Period, 2003 to 2012; 2) Middle Period, 2013 to 2016; and Late Period, 2017 to 2019, further divided into 3) Late Period without SIVs; and 4) Late Period with SIVs. Events of noncompliance were classified by the type of protocol deviation, the category, and the cause. In total, 952 events occurred across 1080 participants. Protocols audited during the Middle Period, compared to the Early Period, showed a decrease in the percentage of protocols with at least 1 noncompliance event. Protocols with SIVs had a further decrease in major, minor, procedural, eligibility, and policy events. Additionally, protocols audited during the Early Period had on average 0.46 major deviations per participant, compared to 0.26 events in protocols audited during the Middle Period, and 0.08 events in protocols audited during the Late Period with SIVs. Protocol deviations and noncompliance events in neurological clinical trials can be reduced by targeted investigator trainings and SIVs. These measures have major impacts on the integrity, safety, and effectiveness of human subjects research in neurology.

Text Classification for Clinical Trial Operations: Evaluation and Comparison of Natural Language Processing Techniques

The ability to detect patterns and trends across protocol deviations (PDs) is key to ensure high data quality and sufficient oversight of patient safety. In clinical trial operations, some business processes and work instructions limit efficient protocol deviation trending because a majority of protocol deviations are left unclassified. When this occurs, it restricts clinical teams from determining systemic issues or signals in the data. The unstructured text in protocol deviation descriptions is an important component of trial operation knowledge. Natural language processing (NLP) can make protocol deviation descriptions more accessible and can support information extraction and trending analysis. This paper reviews how the natural language processing techniques of Term-Frequency Inverse-Document-Frequency (TF-IDF) combined with the supervised machine learning model of Support Vector Machines (SVM) and word embedding approaches such as word2vec can be used to categorize/label protocol deviations across multiple therapeutic areas. NLP is a key tool that will lead to more data driven decisions in clinical trial operations.

Ten simple rules for providing effective bioinformatics research support

Life scientists are increasingly turning to high-throughput sequencing technologies in their research programs, owing to the enormous potential of these methods. In a parallel manner, the number of core facilities that provide bioinformatics support are also increasing. Notably, the generation of complex large datasets has necessitated the development of bioinformatics support core facilities that aid laboratory scientists with cost-effective and efficient data management, analysis, and interpretation. In this article, we address the challenges-related to communication, good laboratory practice, and data handling-that may be encountered in core support facilities when providing bioinformatics support, drawing on our own experiences working as support bioinformaticians on multidisciplinary research projects. Most importantly, the article proposes a list of guidelines that outline how these challenges can be preemptively avoided and effectively managed to increase the value of outputs to the end user, covering the entire research project lifecycle, including experimental design, data analysis, and management (i.e., sharing and storage). In addition, we highlight the importance of clear and transparent communication, comprehensive preparation, appropriate handling of samples and data using monitoring systems, and the employment of appropriate tools and standard operating procedures to provide effective bioinformatics support.

Improving the Quality of Provided Care: Lessons Learned From Auditing Neonatal Stabilization

Video and physiological parameter recording of neonatal stabilization was implemented at the Neonatal Intensive Care Unit (NICU) of the Leiden University Medical Center. In order to improve documentation and the quality of care provided during neonatal transition, we implemented weekly plenary audits reviewing recordings of neonatal stabilization in 2014. In audits, provided care is reviewed, discussing, among others, mask technique, compliance to the prevailing local guideline, and clinical decision making and alternative treatment options. In this perspective, we argue that auditing neonatal stabilization is a valuable tool to improve patient safety and the quality of care provided during neonatal stabilization. We, therefore, report lessons learned and areas for improvement that could be identified and addressed during audits conducted at our NICU. Important areas for improvement were guideline compliance, documentation, the usage of medical devices, the conduct of delivery room studies, and clinical decision making. By reporting our experiences, we hope to encourage other NICUs to also implement regular audit meetings, fitting to their improvement needs.

Clinical Trial Amendments: Development of a Process for Consistent Implementation

Clinical trials are responsible for the advancements made in cancer care. An interdepartmental work group at a major academic cancer hospital developed a process for the consistent communication and implementation of clinical trial amendments. This process ensures improved patient safety, as well as high-quality research. 
.

Investigation of the Study Characteristics Affecting Clinical Trial Quality Using the Protocol Deviations Leading to Exclusion of Subjects From the Per Protocol Set Data in Studies for New Drug Application: A Retrospective Analysis

BACKGROUND

The concept of the risk-based approach has been introduced as an effort to secure the quality of clinical trials. In the risk-based approach, identification and evaluation of risk in advance are considered important. For recently completed clinical trials, we investigated the relationship between study characteristics and protocol deviations leading to the exclusion of subjects from Per Protocol Set (PPS) efficacy analysis.

METHODS

New drugs approved in Japan in the fiscal year 2014-2015 were targeted in the research. The reasons for excluding subjects from the PPS efficacy analysis were described in 102 trials out of 492 in the summary of new drug application documents, which was publicly disclosed after the drug's regulatory approval. The author extracted these reasons along with the numbers of the cases and the study characteristics of each clinical trial. Then, the direct comparison, univariate regression analysis, and multivariate regression analysis was carried out based on the exclusion rate.

RESULT

The study characteristics for which exclusion of subjects from the PPS efficacy analysis were frequently observed was multiregional clinical trials in study region; inhalant and external use in administration route; Anti-infective for systemic use; Respiratory system, Dermatologicals, and Nervous system in therapeutic drug under the Anatomical Therapeutic Chemical Classification. In the multivariate regression analysis, the clinical trial variables of inhalant, Respiratory system, or Dermatologicals were selected as study characteristics leading to a higher exclusion rate.

CONCLUSIONS

The characteristics of the clinical trial that is likely to cause protocol deviations that will affect efficacy analysis were suggested. These studies should be considered for specific attention and priority observation in the trial protocol or its monitoring plan and execution, such as a clear description of inclusion/exclusion criteria in the protocol, development of training materials to site staff, and/or trial subjects as specific risk-alleviating measures.