Reducing waste from incomplete or unusable reports of biomedical research

Assessing the completeness of patient-reported outcomes reporting in congestive heart failure clinical trials

OBJECTIVE

We aim to evaluate the quality of patient-reported outcomes included in randomized control trials for the treatment of congestive heart failure using the International Society for Quality of Life Research (ISOQOL) checklist, a validated tool for critically appraising the quality of patient-reported outcomes.

DESIGN

We performed a cross-sectional analysis of 65 randomized control trials with patient-reported outcomes for drug intervention trials for treating congestive heart failure.

SETTING

N/A.

PARTICIPANTS

N/A.

MAIN OUTCOME MEASURES

The primary outcome of this study was to evaluate the reporting completeness of patient-reported outcomes in congestive heart failure clinical trials with drug interventions according to the ISOQOL checklist.

RESULTS

Our search returned 1114 studies, of which, 65 are included in the analysis. The average completion of the ISOQOL reporting standards was 44.51%. Higher completion of the ISOQOL patient-reported outcome standards was observed in the clinical trials with patient-reported outcomes as primary endpoints compared to the clinical trials with patient-reported outcomes as a secondary endpoint. The multivariable regression model showed that clinical trials with patient-reported outcomes as a primary endpoint had a 21.46% better completion percentage (t = 4.45, p ⩽ 0.001) when controlling for PRO recording duration and trial registration. Eight (8/65, 12.31%) of the clinical trials met the satisfaction criteria of completing two-thirds of the ISOQOL patient-reported outcomes reporting standards. All of these RCTs had a patient-reported outcome as a primary endpoint.

CONCLUSION

Our analysis of the reporting of patient-reported outcomes in congestive heart failure clinical trials with drug interventions suggests that the quality of reporting is suboptimal. This evidence of substandard reporting of patient-reported outcomes is disconcerting as it reduces the transparency of randomized control trials, which are considered the foundation of evidenced-based medicine. Inadequate reporting may result in clinicians implementing misrepresented or incomplete evidence into clinical practice. Validated reporting tools, such as the ISOQOL, can be used by trialists and clinicians alike to improve and critically appraise the reporting of patient-reported outcomes in randomized control trials.

TRIAL REGISTRATION

N/A.

Increased endorsement of TRIPOD and other reporting guidelines by high impact factor journals: survey of instructions to authors

OBJECTIVES

To assess the endorsement of reporting guidelines by high impact factor journals over the period 2017-2022, with a specific focus on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

STUDY DESIGN AND SETTING

We searched the online 'instructions to authors' of high impact factor medical journals in February 2017 and in January 2022 for any reference to reporting guidelines and TRIPOD in particular.

RESULTS

In 2017, 205 out of 337 (61%) journals mentioned any reporting guideline in their instructions to authors and in 2022 this increased to 245 (73%) journals. A reference to TRIPOD was provided by 27 (8%) journals in 2017 and 67 (20%) in 2022. Of those journals mentioning TRIPOD in 2022, 22% provided a link to the TRIPOD website and 60% linked to TRIPOD information on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network website. Twenty-five percent of the journals required adherence to TRIPOD.

CONCLUSION

About three-quarters of high-impact medical journals endorse the use of reporting guidelines and 20% endorse TRIPOD. Transparent reporting is important in enhancing the usefulness of health research and endorsement by journals plays a critical role in this.

A 6-minute protocol, combining mental imagery practices and breathing exercises, promotes hand-grip strength in firefighters: A series of "N-of-1" trials

BACKGROUND

To carry out their victim rescue and fire-fighting missions, firefighters perform high levels of physical exertion and among them strenuous muscular activity. A specific mental preparation protocol that could induce better strength performance throughout their 24-hour schedule is a major issue for firefighters.

OBJECTIVE

This case report aims to examine whether a 6-minute Intervention combining mental imagery practices and breathing exercises, specifically designed to be used while travelling between the fire station and the mission site, would be able to promote maximum strength performance.

METHODS

A series of three Single Case Experimental Designs (SCED) was conducted in single blind design to investigate the effects of repeated challenge-withdrawals between the Intervention and the Baseline on the maximum voluntary isometric contraction (MVIC) strength of the hand-grip.

RESULTS

Data analyses revealed that 62.5% to 100% of the hand-grip strength values during the Intervention periods were greater than or equal to the maximum data point recorded in the Baseline periods. The effect sizes of these highlighted increases of the hand-grip strength performance revealed by the percentage of non-overlapping data (PND) were 75% i.e., moderately effective on average.

CONCLUSIONS

Such a "psyching-up" practice before an upcoming muscular activity can promote muscular strength in firefighters. These results have led French firefighter departments to integrate the teaching of these practices into the initial instruction of firefighters, and remains to be confirmed by a randomised control trial.

Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations

OBJECTIVES

To investigate the extent to which articles of economic evaluations of healthcare interventions indexed in MEDLINE incorporate research practices that promote transparency, openness, and reproducibility.

STUDY DESIGN AND SETTING

We evaluated a random sample of health economic evaluations indexed in MEDLINE during 2019. We included articles written in English reporting an incremental cost-effectiveness ratio in terms of costs per life years gained, quality-adjusted life years, and/or disability-adjusted life years. Reproducible research practices, openness, and transparency in each article were extracted in duplicate. We explored whether reproducible research practices were associated with self-report use of a guideline.

RESULTS

We included 200 studies published in 147 journals. Almost half were published as open access articles (n = 93; 47%). Most studies (n = 150; 75%) were model-based economic evaluations. In 109 (55%) studies, authors self-reported use a guideline (e.g., for study conduct or reporting). Few studies (n = 31; 16%) reported working from a protocol. In 112 (56%) studies, authors reported the data needed to recreate the incremental cost-effectiveness ratio for the base case analysis. This percentage was higher in studies using a guideline than studies not using a guideline (72/109 [66%] with guideline vs. 40/91 [44%] without guideline; risk ratio 1.50, 95% confidence interval 1.15-1.97). Only 10 (5%) studies mentioned access to raw data and analytic code for reanalyses.

CONCLUSION

Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations. This study provides baseline data to compare future progress in the field.

Evaluation of 'implications for research' statements in systematic reviews of interventions in advanced cancer patients - a meta-research study

BACKGROUND

Implications for research (IfR) sections are an important part of systematic reviews (SRs) to inform health care researchers and policy makers. PRISMA 2020 recommends reporting IfR, while Cochrane Reviews require a separate chapter on IfR. However, it is unclear to what extent SRs discuss IfR. We aimed i) to assess whether SRs include an IfR statement and ii) to evaluate which elements informed IfR statements.

METHODS

We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). As suggested in the Cochrane Handbook, we assessed if the following predefined variables were referred to in IfR statements: patient, intervention, control, outcome (PICO) and study design; concepts underlying Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions.

RESULTS

We included 261 SRs. The majority evaluated a pharmacological intervention (n = 244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR statement (n = 210, 80.5%). IfR statements commonly addressed 'intervention' (n = 121, 57.6%), 'patient ' (n = 113, 53.8%), and 'study design' (n = 107, 51.0%). The most frequent PICO and study design combinations were 'patient and intervention ' (n = 71, 33.8%) and 'patient, intervention and study design ' (n = 34, 16.2%). Concepts underlying GRADE domains were rarely used for informing IfR recommendations: 'risk of bias ' (n = 2, 1.0%), and 'imprecision ' (n = 1, 0.5%), 'inconsistency ' (n = 1, 0.5%). Additional elements informing IfR were considerations on cost effectiveness (n = 9, 4.3%), reporting standards (n = 4, 1.9%), and individual patient data meta-analysis (n = 4, 1.9%).

CONCLUSION

Although about 80% of SRs included an IfR statement, the reporting of PICO elements varied across SRs. Concepts underlying GRADE domains were rarely used to derive IfR. Further work needs to assess the generalizability beyond SRs in advanced cancer patients. We suggest that more specific guidance on which and how IfR elements to report in SRs of interventions needs to be developed. Utilizing PICO elements and concepts underlying GRADE according to the Cochrane Handbook to state IfR seems to be a reasonable approach in the interim.

REGISTRATION

CRD42019134904.

Pilot test of Consensus Reporting Items for Studies in Primary care (CRISP)

The Consensus Reporting Items for Studies in Primary care (CRISP) provides a new research reporting guideline to meet the needs of the producers and users of primary care (PC) research. Developed through an iterative program of research, including investigators, practicing clinicians, patients, community representatives, and educators, the CRISP Checklist guides PC researchers across the spectrum of research methods, study designs, and topics. This pilot test included a variety of team members using the CRISP Checklist for writing, revising, and reviewing PC research reports. All or most of the 15 participants reported that the checklist was easy to use, improved research reports, and should be recommended by PC research journals. The checklist is adaptable to different study types; not all items apply to all reports. The CRISP Checklist can help meet the needs of PC research when used in parallel with existing guidelines that focus on specific methods and limited topics.

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

Research Impact Assessment (RIA) represents one of a suite of policies intended to improve the impact generated from investment in health and medical research (HMR). Positivist indicator-based approaches to RIA are widely implemented but increasingly criticised as theoretically problematic, unfair, and burdensome. This commentary proposes there are useful outcomes that emerge from the process of applying an indicator-based RIA framework, separate from those encapsulated in the metrics themselves. The aim for this commentary is to demonstrate how the act of conducting an indicator-based approach to RIA can serve to optimise the productive gains from the investment in HMR. Prior research found that the issues regarding RIA are less about the choice of indicators/metrics, and more about the discussions prompted and activities incentivised by the process. This insight provides an opportunity to utilise indicator-based methods to purposely optimise the research impact. An indicator-based RIA framework specifically designed to optimise research impacts should: focus on researchers and the research process, rather than institution-level measures; utilise a project level unit of analysis that provides control to researchers and supports collaboration and accountability; provide for prospective implementation of RIA and the prospective orientation of research; establish a line of sight to the ultimate anticipated beneficiaries and impacts; Include process metrics/indicators to acknowledge interim steps on the pathway to final impacts; integrate 'next' users and prioritise the utilisation of research outputs as a critical measure; Integrate and align the incentives for researchers/research projects arising from RIA, with those existing within the prevailing research system; integrate with existing peer-review processes; and, adopt a system-wide approach where incremental improvements in the probability of translation from individual research projects, yields higher impact across the whole funding portfolio.Optimisation of the impacts from HMR investment represents the primary purpose of Research Impact policy. The process of conducting an indicator-based approach to RIA, which engages the researcher during the inception and planning phase, can directly contribute to this goal through improvements in the probability that an individual project will generate interim impacts. The research project funding process represents a promising forum to integrate this approach within the existing research system.

Quality of Reporting Randomized Controlled Trials Published in Three of the Most Citable Periodontal Journals from 2018 to 2022

This study aimed to evaluate the reporting quality of randomized clinical trials (RCTs) in periodontology. Three leading periodontology journals, the Journal of Periodontology (JOP), the Journal of Clinical Periodontology (JOCP), and the Journal of Periodontal Research (JOPR), were selected for this investigation. The RCTs were identified by manually searching for human trial articles published in these three journals. Two authors independently conducted the literature search, and a pre-piloted extraction sheet was used to screen the potential RCTs. The CONSORT checklist guidelines were employed to calculate the score value. Intra-examiner reliability was assessed by scoring a random sample of 10% of the papers in a second round conducted by the first examiner three months after the initial data collection. A search of abstracts published over a five-year period yielded 176 articles that reported RCTs, accounting for 11.7% of all articles published in the three journals. The highest number of RCTs was published in 2020, and more than half of the included RCTs (51%) originated from Europe. Many of the analyzed RCTs inadequately reported almost half of the items on the CONSORT checklist. Furthermore, univariate analysis revealed significant associations between certain factors and the overall CONSORT score, such as publication in JOP (p = 0.048), publication year of 2019 (p = 0.041) and 2021 (p = 0.042), first author from North America (p = 0.016), and RCTs with more than six authors (p = 0.042). Clinical trial research in periodontics has made significant progress in the past five years. However, there is room for improvement in adhering to the CONSORT guidelines.

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Challenges for research uptake for health policymaking and practice in low- and middle-income countries: a scoping review

BACKGROUND

An estimated 85% of research resources are wasted worldwide, while there is growing demand for context-based evidence-informed health policymaking. In low- and middle-income countries (LMICs), research uptake for health policymaking and practice is even lower, while little is known about the barriers to the translation of health evidence to policy and local implementation. We aimed to compile the current evidence on barriers to uptake of research in health policy and practice in LMICs using scoping review.

METHODS

The scoping review followed the Preferred Reporting Items for Systematic Review and Meta-Analyses-extension for Scoping Reviews (PRISMA-ScR) and the Arksey and O'Malley framework. Both published evidence and grey literature on research uptake were systematically searched from major databases (PubMed, Cochrane Library, CINAHL (EBSCO), Global Health (Ovid)) and direct Google Scholar. Literature exploring barriers to uptake of research evidence in health policy and practice in LMICs were included and their key findings were synthesized using thematic areas to address the review question.

RESULTS

A total of 4291 publications were retrieved in the initial search, of which 142 were included meeting the eligibility criteria. Overall, research uptake for policymaking and practice in LMICs was very low. The challenges to research uptake were related to lack of understanding of the local contexts, low political priority, poor stakeholder engagement and partnership, resource and capacity constraints, low system response for accountability and lack of communication and dissemination platforms.

CONCLUSION

Important barriers to research uptake, mainly limited contextual understanding and low participation of key stakeholders and ownership, have been identified. Understanding the local research and policy context and participatory evidence production and dissemination may promote research uptake for policy and practice. Institutions that bridge the chasm between knowledge formation, evidence synthesis and translation may play critical role in the translation process.

Recommendations for Cell-Free DNA Assay Validations: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists

Diagnosing, selecting therapy for, and monitoring cancer in patients using a minimally invasive blood test represents a significant advance in precision medicine. Wide variability exists in how circulating tumor DNA (ctDNA) assays are developed, validated, and reported in the literature, which hinders clinical adoption and may negatively impact patient care. Standardization is needed for factors affecting ctDNA assay performance and reporting, including pre-analytical variables, analytical considerations, and elements of laboratory assay reporting. The Association for Molecular Pathology Clinical Practice Committee's Liquid Biopsy Working Group (LBxWG), including organizational representation from the American Society of Clinical Oncology and the College of American Pathologists, has undertaken a full-text data extraction of 1228 ctDNA publications that describe assays performed in patients with lymphoma and solid tumor malignancies. With an emphasis on clinical assay validation, the LBxWG has developed a set of 13 best practice consensus recommendations for validating, reporting, and publishing clinical ctDNA assays. Recommendations include reporting key pre-analytical considerations and assay performance metrics; this analysis demonstrates these elements are inconsistently included in publications. The LBxWG recommendations are intended to assist clinical laboratories with validating and reporting ctDNA assays and to ensure high-quality data are included in publications. It is expected that these recommendations will need to be updated as the body of literature continues to mature.

Saldanha I, Smith V, Toomey E, Williamson P. Using behavioural science to enhance use of core outcome sets in trials: protocol

Background

Core outcome sets (COS) represent agreed-upon sets of outcomes, which are the minimum that should be measured and reported in all trials in specific health areas. Use of COS can reduce outcome heterogeneity, selective outcome reporting, and research waste, and can facilitate evidence syntheses. Despite benefits of using COS, current use of COS in trials is low. COS use can be understood as a behaviour, in that it is something trialists do, or not do, adequately. The aim of this study is to identify strategies, informed by behaviour change theory, to increase COS use in trials.

Methods

The project will be conducted in two stages, informed by the behaviour change wheel (BCW). The BCW is a theoretically based framework that can be used to classify, identify, and develop behaviour change strategies. In Stage 1, barriers and enablers to COS use will be extracted from published studies that examined trialist's use of COS. Barriers and facilitators will be mapped to the components of COM-B model (capability, opportunity, and motivation), which forms part of the BCW framework. Stage 2 will build on Stage 1 findings to identify and select intervention functions and behaviour change techniques to enhance COS use in trials.

Discussion

The findings of this study will provide an understanding of the behavioural factors that influence COS use in trials and what strategies might be used to target these factors to increase COS use in trials.

Saldanha I, Smith V, Toomey E, Williamson P. Using behavioural science to enhance use of core outcome sets in trials: protocol

BACKGROUND

Core outcome sets (COS) represent agreed-upon sets of outcomes, which are the minimum that should be measured and reported in all trials in specific health areas. Use of COS can reduce outcome heterogeneity, selective outcome reporting, and research waste, and can facilitate evidence syntheses. Despite benefits of using COS, current use of COS in trials is low. COS use can be understood as a behaviour, in that it is something trialists do, or not do, adequately. The aim of this study is to identify strategies, informed by behaviour change theory, to increase COS use in trials.

METHODS

The project will be conducted in two stages, informed by the behaviour change wheel (BCW). The BCW is a theoretically based framework that can be used to classify, identify, and develop behaviour change strategies. In Stage 1, barriers and enablers to COS use will be extracted from published studies that examined trialist's use of COS. Barriers and facilitators will be mapped to the components of COM-B model (capability, opportunity, and motivation), which forms part of the BCW framework. Stage 2 will build on Stage 1 findings to identify and select intervention functions and behaviour change techniques to enhance COS use in trials.

DISCUSSION

The findings of this study will provide an understanding of the behavioural factors that influence COS use in trials and what strategies might be used to target these factors to increase COS use in trials.

"The project did not come to us with a solution": Perspectives of research teams on implementing a study about electronic health record-embedded individualized pain plans for emergency department treatment of vaso-occlusive episodes in adults with sickle cell disease

BACKGROUND

This study aimed to capture the implementation process of the ALIGN Study, (An individualized Pain Plan with Patient and Provider Access for Emergency Department care of Sickle Cell Disease). ALIGN aimed to embed Individualized Pain Plans in the electronic health record (E-IPP) and provide access to the plan for both adult patients with sickle cell disease (SCD) and emergency department providers when a person with SCD comes to the emergency department in vaso-occlusive crises.

METHODS

Semi-structured interviews were conducted with research teams from the 8 participating sites from the ALIGN study. Seventeen participants (principal investigators and study coordinators) shared their perspectives about the implementation of ALIGN in their sites. Data were analyzed in three phases using open coding steps adapted from grounded theory and qualitative content analysis.

RESULTS

A total of seven overarching themes were identified: (1) the E-IPP structure (location and upkeep) and collaboration with the informatics team, (2) the role of ED champion, (3) the role of research coordinators, (4) research team communication, and communication between research team and clinical team, (5) challenges with the study protocol, (6) provider feedback: addressing over-utilizers, patient mistrust, and the positive feedback about the intervention, and (7) COVID-19 and its effects on study implementation.

CONCLUSIONS

Findings from this study contribute to learning how to implement E-IPPs for adult patients with SCD in ED. The study findings highlight the importance of early engagement with different team members, a champion from the emergency department, study coordinators with different skills and enhancement of communication and trust among team members. Further recommendations are outlined for hospitals aiming to implement E-IPP for patients with SCD in ED.

Gynecological trials frequently exclude people based on their symptoms rather than their condition: a systematic review of Cochrane reviews and their component trials

OBJECTIVES

To identify strategies used in recent randomized controlled trials (RCTs) and their associated Cochrane Reviews where patients with the same gynecological condition present with different symptoms but would plausibly benefit from a common intervention.

STUDY DESIGN AND SETTING

We searched the Cochrane library (February 2022) for reviews in polycystic ovarian syndrome (PCOS) and endometriosis. Reviews were included if the intervention was intended to treat all condition-specific symptoms. For each trial we recorded the strategy used and the number of potentially eligible participants excluded as a direct result of the chosen strategy. For each review we recorded the numbers of RCTs and participants excluded on the basis of symptoms experienced.

RESULTS

There were 89 distinct PCOS trials in 13 reviews, and 13 Endometriosis trials in 11 reviews. Most trials restricted their eligibility to participants with specific symptoms (55% PCOS, 46% endometriosis). The second most common strategy was to measure and analyze clinical outcomes that were not relevant to all participants (38% PCOS, 31% endometriosis). Reviews excluded 27% of trials in participants evaluating the same intervention in participants experiencing the same condition based on the outcomes measured in the trials.

CONCLUSION

Most gynecological trials exclude patients who could benefit from treatment or measure outcomes not relevant to all participants. We introduce a taxonomy to describe trial design strategies for conditions with heterogeneous symptoms.

The PRISMATIC project: protocol for a research programme on novel methods to improve reporting and peer review of systematic reviews of health evidence

BACKGROUND

Incomplete reporting about what systematic reviewers did and what they found prevents users of the report from being able to fully interpret the findings and understand the limitations of the underlying evidence. Reporting guidelines such as the PRISMA statement and its extensions are designed to improve reporting. However, there are important inconsistencies across the various PRISMA reporting guidelines, which causes confusion and misinterpretation. Coupled with this, users might need to consult multiple guidelines to gain a full understanding of the guidance. Furthermore, the current passive strategy of implementing PRISMA has not fully brought about needed improvements in the completeness of systematic review reporting.

METHODS

The PRISMATIC ('PRISMA, Technology, and Implementation to enhance reporting Completeness') project aims to use novel methods to enable more efficient and effective translation of PRISMA reporting guidelines into practice. We will establish a working group who will develop a unified PRISMA statement that harmonises content across the main PRISMA guideline and several of its extensions. We will then develop a web application that generates a reporting template and checklist customised to the characteristics and methods of a systematic review ('PRISMA-Web app') and conduct a randomised trial to evaluate its impact on authors' reporting. We will also develop a web application that helps peer reviewers appraise systematic review manuscripts ('PRISMA-Peer app') and conduct a diagnostic accuracy study to evaluate its impact on peer reviewers' detection of incomplete reporting.

DISCUSSION

We anticipate the novel guidance and web-based apps developed throughout the project will substantively enhance the completeness of reporting of systematic reviews of health evidence, ultimately benefiting users who rely on systematic reviews to inform health care decision-making.

Toward a global harmonization of service infrastructure in academic clinical trial units: an international survey

Background

Clinicians around the world perform clinical research in addition to their high workload. To meet the demands of high quality Investigator Initiated Trials (IITs), Clinical Trial Units (CTUs) (as part of Academic Research Institutions) are implemented worldwide. CTUs increasingly hold a key position in facilitating the international mutual acceptance of clinical research data by promoting clinical research practices and infrastructure according to international standards.

Aim

In this project, we aimed to identify services that established and internationally operating CTUs - members of the International Clinical Trial Center Network (ICN) - consider most important to ensure the smooth processing of a clinical trial while meeting international standards. We thereby aim to drive international harmonization by providing emerging and growing CTUs with a resource for informed service range set-up.

Methods

Following the AMEE Guide, we developed a questionnaire, addressing the perceived importance of different CTU services. Survey participants were senior representatives of CTUs and part of the ICN with long-term experience in their field and institution.

Results

Services concerning quality and coordination of a research project were considered to be most essential, i.e., Quality management, Monitoring and Project management, followed by Regulatory & Legal affairs, Education & Training, and Data management. Operative services for conducting a research project, i.e., Study Nurse with patient contact and Study Nurse without patient contact, were considered to be least important.

Conclusion

To balance the range of services offered while meeting high international standards of clinical research, emerging CTUs should focus on offering (quality) management services and expertise in regulatory and legal affairs. Additionally, education and training services are required to ensure clinicians are well trained on GCP and legislation. CTUs should evaluate whether the expertise and resources are available to offer operative services.

Machine learning models in clinical practice for the prediction of postoperative complications after major abdominal surgery

Complications after surgery have a major impact on short- and long-term outcomes, and decades of technological advancement have not yet led to the eradication of their risk. The accurate prediction of complications, recently enhanced by the development of machine learning algorithms, has the potential to completely reshape surgical patient management. In this paper, we reflect on multiple issues facing the implementation of machine learning, from the development to the actual implementation of machine learning models in daily clinical practice, providing suggestions on the use of machine learning models for predicting postoperative complications after major abdominal surgery.

Efficacy and safety of traditional Chinese medicine adjuvant therapy for severe pneumonia: evidence mapping of the randomized controlled trials, systematic reviews, and meta-analyses

Background and Objective: Severe pneumonia is a critical respiratory disease with high mortality. There is insufficient evidence on the efficacy and safety of traditional Chinese medicine (TCM) adjuvant therapy for severe pneumonia. This study aims to identify, describe, assess, and summarize the currently available high-quality design evidence on TCM adjuvant therapy for severe pneumonia to identify evidence gaps using the evidence mapping approach. Methods: Systematic searches were performed on English and Chinese online databases (PubMed, EMBASE, Cochrane Library, Web of Science, CNKI, WanFang Data, CQVIP, and SinoMed) to identify papers from inception until August 2023 for inclusion into the review. Randomized controlled trials (RCTs), systematic reviews (SRs), and meta-analyses concerning TCM adjuvant therapy for severe pneumonia or its complications in adults were included. The risk of bias in RCTs was evaluated by using the Cochrane Handbook ROB tool. The Assessment of Multiple Systematic Reviews 2 (AMSTAR-2), the Risk of Bias in Systematic Review (ROBIS) tool, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system were used to assess the methodological quality, risk of bias, and evidence quality of SRs or meta-analyses, respectively. Then, a bubble plot was designed to visually display information in four dimensions. Results: A total of 354 RCTs and 17 SRs or meta-analyses met the inclusion criteria. The published RCTs had several flaws, such as unreasonable design, limited sample size, insufficient attention to non-drug therapy studies and syndrome differentiation, improper selection or use of outcome indicators, and failure to provide high-quality evidence. Sixteen SRs or meta-analyses of methodological quality scored "Critically Low" confidence. Twelve SRs or meta-analyses were rated as "High Risk." Most outcomes were rated as "Low" evidence quality. We found that TCM combined with conventional treatment could improve the clinical total effective rate and the TCM syndromes efficacy. The combined approach could also shorten mechanical ventilation time, infection control time, and length of hospital and ICU stay; significantly reduce temperature, respiratory rate, heart rate, white blood cell counts, levels of C-reactive protein, procalcitonin, blood inflammatory factors, bacteriological response, and D-dimer; decrease CPIS, APACHE II score, and PSI score; improve pulmonary imaging features, arterial blood gas indicators (including arterial oxygen pressure, arterial oxygen saturation, and oxygen index), and lung function (including forced vital capacity and forced expiratory volume in the first second) for severe pneumonia compared with conventional treatment only (p < 0.05). There was no significant difference in adverse reactions and incidence of adverse events (p > 0.05). In addition, compared with conventional treatment only, most SRs or meta-analyses concluded that TCM combined with conventional treatment was "Beneficial" or "Probably beneficial." Conclusion: TCM combined with conventional treatment had advantages in efficacy, clinical signs, laboratory results, and life quality outcomes of severe pneumonia, with no difference in safety outcomes compared with conventional treatment only. QingJin Huatan decoction is the most promising target, and Xuanbai Chengqi decoction has a "Probably beneficial" conclusion. XueBiJing injection and TanReQing injection are two commonly used Chinese herbal injections for treating severe pneumonia, and both are "Probably beneficial." However, there was a need for multicenter RCTs with large sample sizes and high methodological quality in the future. In addition, the methodological design and quality of SRs or meta-analyses should be improved to form high-quality, evidence-based medical evidence and provide evidence for the effectiveness and safety of TCM adjuvant therapy for severe pneumonia.

Influence of research evidence on the use of cardiovascular clinical prediction rules in primary care: an exploratory qualitative interview study

BACKGROUND

Cardiovascular clinical prediction rules (CPRs) are widely used in primary care. They accumulate research evidence through derivation, external validation, and impact studies. However, existing knowledge about the influence of research evidence on the use of CPRs is limited. Therefore, we explored how primary care clinicians' perceptions of and experiences with research influence their use of cardiovascular CPRs.

METHODS

We conducted an exploratory qualitative interview study with thematic analysis. Primary care clinicians were recruited from the WWAMI (Washington, Wyoming, Alaska, Montana and Idaho) region Practice and Research Network (WPRN). We used purposeful sampling to ensure maximum variation within the participant group. Data were collected by conducting semi-structured online interviews. We analyzed data using inductive thematic analysis to identify commonalities and differences within themes.

RESULTS

Of 29 primary care clinicians who completed the questionnaire, 15 participated in the interview. We identified two main themes relating to the influence of clinicians' perceptions of and experiences with cardiovascular CPR research on their decisions about using cardiovascular CPRs: "Seek and judge" and "be acquainted and assume." When clinicians are familiar with, trust, and feel confident in using research evidence, they might actively search and assess the evidence, which may then influence their decisions about using cardiovascular CPRs. However, clinicians, who are unfamiliar with, distrust, or find it challenging to use research evidence, might be passively acquainted with evidence but do not make their own judgment on the trustworthiness of such evidence. Therefore, these clinicians might not rely on research evidence when making decisions about using cardiovascular CPRs.

CONCLUSIONS

Clinicians' perceptions and experiences could influence how they use research evidence in decisions about using cardiovascular CPRs. This implies, when promoting evidence-based decisions, it might be useful to target clinicians' unfamiliarity, distrust, and challenges regarding the use of research evidence rather than focusing only on their knowledge and skills. Further, because clinicians often rely on evidence-unrelated factors, guideline developers and policymakers should recommend cardiovascular CPRs supported by high-quality evidence.

Reporting of Observational Studies Explicitly Aiming to Emulate Randomized Trials: A Systematic Review

Importance

Observational (nonexperimental) studies that aim to emulate a randomized trial (ie, the target trial) are increasingly informing medical and policy decision-making, but it is unclear how these studies are reported in the literature. Consistent reporting is essential for quality appraisal, evidence synthesis, and translation of evidence to policy and practice.

Objective

To assess the reporting of observational studies that explicitly aimed to emulate a target trial.

Evidence Review

We searched Medline, Embase, PsycINFO, and Web of Science for observational studies published between March 2012 and October 2022 that explicitly aimed to emulate a target trial of a health or medical intervention. Two reviewers double-screened and -extracted data on study characteristics, key predefined components of the target trial protocol and its emulation (eligibility criteria, treatment strategies, treatment assignment, outcome[s], follow-up, causal contrast[s], and analysis plan), and other items related to the target trial emulation.

Findings

A total of 200 studies that explicitly aimed to emulate a target trial were included. These studies included 26 subfields of medicine, and 168 (84%) were published from January 2020 to October 2022. The aim to emulate a target trial was explicit in 70 study titles (35%). Forty-three studies (22%) reported use of a published reporting guideline (eg, Strengthening the Reporting of Observational Studies in Epidemiology). Eighty-five studies (43%) did not describe all key items of how the target trial was emulated and 113 (57%) did not describe the protocol of the target trial and its emulation.

Conclusion and Relevance

In this systematic review of 200 studies that explicitly aimed to emulate a target trial, reporting of how the target trial was emulated was inconsistent. A reporting guideline for studies explicitly aiming to emulate a target trial may improve the reporting of the target trial protocols and other aspects of these emulation attempts.

Methods for Heart Rate Variability Biofeedback (HRVB): A Systematic Review and Guidelines

Heart Rate Variability Biofeedback (HRVB) has been widely used to improve cardiovascular health and well-being. HRVB is based on breathing at an individual's resonance frequency, which stimulates respiratory sinus arrhythmia (RSA) and the baroreflex. There is, however, no methodological consensus on how to apply HRVB, while details about the protocol used are often not well reported. Thus, the objectives of this systematic review are to describe the different HRVB protocols and detect methodological concerns. PsycINFO, CINALH, Medline and Web of Science were searched between 2000 and April 2021. Data extraction and quality assessment were based on PRISMA guidelines. A total of 143 studies were finally included from any scientific field and any type of sample. Three protocols for HRVB were found: (i) "Optimal RF" (n = 37), each participant breathes at their previously detected RF; (ii) "Individual RF" (n = 48), each participant follows a biofeedback device that shows the optimal breathing rate based on cardiovascular data in real time, and (iii) "Preset-pace RF" (n = 51), all participants breathe at the same rate rate, usually 6 breaths/minute. In addition, we found several methodological differences for applying HRVB in terms of number of weeks, duration of breathing or combination of laboratory and home sessions. Remarkably, almost 2/3 of the studies did not report enough information to replicate the HRVB protocol in terms of breathing duration, inhalation/exhalation ratio, breathing control or body position. Methodological guidelines and a checklist are proposed to enhance the methodological quality of future HRVB studies and increase the information reported.

Development and validation of a Medication Adherence Universal Questionnaire: the MAUQ

BACKGROUND

Different questionnaires assess self-reported medication adherence and others quantify aspects of patients attitudes towards medication, but not together in a single instrument. Gathering these two aspects in a single instrument could reduce patients survey burden.

AIM

The aim of this study was to develop the Medication Adherence Universal Questionnaire (MAUQ) using the Maastricht Utrecht Adherence in Hypertension short version (MUAH-16) factorial structure as the hypothesized model.

METHOD

A multistep process started with the modification of the MUAH-16 to obtain the MAUQ. Patients using at least one antihypertensive medicine were recruited. The two questionnaires, the MUAH-16 and MAUQ, were applied. A confirmatory factor analysis (CFA) was performed using the initial MUAH-16 s-order 4-factor model. An additional bifactor model with four uncorrelated factors and an overall score was tested. The comparative fit index (CFI), root mean square error of approximation (RMSEA) with confidence intervals (CIs), and standardized root mean squared residual (SRMR) were used to assess both models.

RESULTS

A sample of 300 hypertensive patients completed the instruments. The CFA with the second-order 4-factor solution resulted in similar results for the MUAH-16 and MAUQ: CFIs of 0.934 and 0.930, RMSEAs of 0.043 [CI 0.030-0.056] and 0.045 [CI 0.031-0.057] and SRMRs of 0.060 and 0.061, respectively. The CFA with the bifactor model showed slightly better results for both the MUAH-16 and MAUQ: CFIs of 0.974 and 0.976, RMSEAs of 0.030 [CI 0.005-0.046] and 0.028 [CI 0.001-0.044], and SRMRs of 0.043 and 0.044, respectively.

CONCLUSION

CFA demonstrated that the MAUQ presented a better fit to both models than the MUAH-16, obtaining a robust universal free instrument to assess medicine-taking behaviour and four medicine beliefs components.

The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences

BACKGROUND

The number of preliminary studies conducted and published has increased in recent years. However, there are likely many preliminary studies that go unpublished because preliminary studies are typically small and may not be perceived as methodologically rigorous. The extent of publication bias within preliminary studies is unknown but can prove useful to determine whether preliminary studies appearing in peer-reviewed journals are fundamentally different than those that are unpublished. The purpose of this study was to identify characteristics associated with publication in a sample of abstracts of preliminary studies of behavioral interventions presented at conferences.

METHODS

Abstract supplements from two primary outlets for behavioral intervention research (Society of Behavioral Medicine and International Society of Behavioral Nutrition and Physical Activity) were searched to identify all abstracts reporting findings of behavioral interventions from preliminary studies. Study characteristics were extracted from the abstracts including year presented, sample size, design, and statistical significance. To determine if abstracts had a matching peer-reviewed publication, a search of authors' curriculum vitae and research databases was conducted. Iterative logistic regression models were used to estimate odds of abstract publication. Authors with unpublished preliminary studies were surveyed to identify reasons for nonpublication.

RESULTS

Across conferences, a total of 18,961 abstracts were presented. Of these, 791 were preliminary behavioral interventions, of which 49% (388) were published in a peer-reviewed journal. For models with main effects only, preliminary studies with sample sizes greater than n = 24 were more likely to be published (range of odds ratios, 1.82 to 2.01). For models including interactions among study characteristics, no significant associations were found. Authors of unpublished preliminary studies indicated small sample sizes and being underpowered to detect effects as barriers to attempting publication.

CONCLUSIONS

Half of preliminary studies presented at conferences go unpublished, but published preliminary studies appearing in peer-reviewed literature are not systematically different from those that remain unpublished. Without publication, it is difficult to assess the quality of information regarding the early-stage development of interventions. This inaccessibility inhibits our ability to learn from the progression of preliminary studies.

Methods and results of studies on reporting guideline adherence are poorly reported: a meta-research study

OBJECTIVES

We investigated recent meta-research studies on adherence to four reporting guidelines to determine the proportion that provided (1) an explanation for how adherence to guideline items was rated and (2) results from all included individual studies. We examined conclusions of each meta-research study to evaluate possible repetitive and similar findings.

STUDY DESIGN AND SETTING

A cross-sectional meta-research study. MEDLINE (Ovid) was searched on July 5, 2022 for studies that used any version of the Consolidated Standards of Reporting Trials, Preferred Reporting Items for Systematic Reviews and Meta-Analyses, Standards for the Reporting of Diagnostic Accuracy Studies, or Strengthening the Reporting of Observational Studies in Epidemiology reporting guidelines or their extensions to evaluate reporting.

RESULTS

Of 148 included meta-research studies published between August 2020 and June 2022, 14 (10%, 95% confidence interval [CI] 6%-15%) provided a fully replicable explanation of how they coded the adherence ratings and 49 (33%, 95% CI 26%-41%) completely reported individual study results. Of 90 studies that classified reporting as adequate or inadequate in the study abstract, six (7%, 95% CI 3%-14%) concluded that reporting was adequate, but none of those six studies provided information on how items were coded or provided item-level results for included studies.

CONCLUSION

Almost all included meta-research studies found that reporting in health research is suboptimal. However, few of these reported enough information for verification or replication.

Reporting and data-sharing level of acupuncture randomised controlled trials: a cross-sectional study protocol

INTRODUCTION

Randomised controlled trials (RCTs) play an important role in evidence-based medicine. However, an article with low reporting quality may mislead both experts and the general public into an erroneous decision. Data sharing can contribute to the truthfulness and transparency of trials. Acupuncture RCTs have been increasing rapidly these years, but the reporting quality and data-sharing level of acupuncture RCTs are not clear. Thus, this study will provide the current status of the reporting quality and data-sharing level of acupuncture RCTs.

METHODS AND ANALYSIS

A cross-sectional study will be conducted. The seven databases including MEDLINE, EMBASE, CENTRAL, CBM, CNKI, Wanfang Database and VIP will be searched between 1 January 2012 and 15 October 2022 to identify acupuncture RCTs. The basic characteristics of included trials will be summarised. The reporting quality for included RCTs will be assessed by the Consolidated Standards for Reporting Trials 2010 statement and the Standards for Reporting Interventions in Controlled Trials of Acupuncture. The data-sharing level will be assessed by open science practices.

ETHICS AND DISSEMINATION

Ethical approval is not required for this study. This protocol has been registered in Open Science Framework Registries. The findings of this study will be submitted to a peer-reviewed academic journal.

AgroEcoList 1.0: A checklist to improve reporting standards in ecological research in agriculture

Many publications lack sufficient background information (e.g. location) to be interpreted, replicated, or reused for synthesis. This impedes scientific progress and the application of science to practice. Reporting guidelines (e.g. checklists) improve reporting standards. They have been widely taken up in the medical sciences, but not in ecological and agricultural research. Here, we use a community-centred approach to develop a reporting checklist (AgroEcoList 1.0) through surveys and workshops with 23 experts and the wider agroecological community. To put AgroEcoList in context, we also assessed the agroecological community's perception of reporting standards in agroecology. A total of 345 researchers, reviewers, and editors, responded to our survey. Although only 32% of respondents had prior knowledge of reporting guidelines, 76% of those that had said guidelines improved reporting standards. Overall, respondents agreed on the need of AgroEcolist 1.0; only 24% of respondents had used reporting guidelines before, but 78% indicated they would use AgroEcoList 1.0. We updated AgroecoList 1.0 based on respondents' feedback and user-testing. AgroecoList 1.0 consists of 42 variables in seven groups: experimental/sampling set-up, study site, soil, livestock management, crop and grassland management, outputs, and finances. It is presented here, and is also available on github (https://github.com/AgroecoList/Agroecolist). AgroEcoList 1.0 can serve as a guide for authors, reviewers, and editors to improve reporting standards in agricultural ecology. Our community-centred approach is a replicable method that could be adapted to develop reporting checklists in other fields. Reporting guidelines such as AgroEcoList can improve reporting standards and therefore the application of research to practice, and we recommend that they are adopted more widely in agriculture and ecology.

Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models

OBJECTIVES

We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques.

STUDY DESIGN AND SETTING

We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty.

RESULTS

We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies.

CONCLUSION

Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.

Is our public research money well spent? Publication of research outputs from Health Research Council of New Zealand-funded studies: a cross-sectional study

OBJECTIVE

To evaluate the reporting of results from the projects and programmes funded by the Health Research Council (HRC) New Zealand.

DESIGN

A cross-sectional analysis.

SETTING

Research projects and programmes funded by the HRC New Zealand from 2006 to 2014.

PARTICIPANTS

Publicly available data provided by the HRC.

MAIN OUTCOME MEASURES

The number and proportion with evidence of publication and dissemination of a research output from HRC grants and the time taken to disseminate the results.

RESULTS

Of the 374 HRC grants from 2006 to 2014, there was no evidence of publication or reporting of any research output for 48 studies (13%). Of the 326 (87%) grants with research outputs, there was a mean dissemination time of 4.73 years (SD 2.37). The total funding provided by the HRC was NZ$471 663 336, while the 48 grants with no evidence of dissemination represented NZ$47 095 727 (10%).

CONCLUSIONS

Thirteen per cent of the HRC projects and programmes from 2006 to 2014 have not contributed to the healthcare evidence as their results remain unknown.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Do Case Reports on Total Mandibular, Extended, and Bilateral Total Temporomandibular Joint Reconstruction Prostheses Adhere to the Surgical CAse REport Guidelines?

PURPOSE

The Surgical CAse REport (SCARE) guidelines are a standardized format for reporting surgical cases. The aim of this study was to evaluate the completeness of case reports documenting alloplastic reconstruction of large craniomaxillofacial defects involving total mandibular, bilateral, and extended temporomandibular joint in major high-quality craniomaxillofacial journals, based on the SCARE guidelines.

METHODS

An extensive online search was performed according to the Priority Reporting Items for Systematic Reviews and Meta-Analyses statement in PubMed, Embase, Scopus, Google Scholar, and Dimensions databases to identify relevant case reports. Each selected case report was assessed on 16 topics (38 items) of the SCARE guidelines, using a scoring scale of "0" (No/noncompliance), "1" (Yes/compliance), and 2" (unclear). The completeness of reporting (COR) score was calculated as the ratio of "yes" responses to "total" (ie, yes + no + unclear) responses. Adequacy of case reporting was denoted by a COR score of 70% or more.

RESULTS

A total of 35 case reports were selected, where the male to female patients ratio was 3:4 cases, mean ± standard deviation (SD) age: 34.9 ± 16.7 years, mean ± SD follow-up duration: 17.0 ± 12.9 months, and number of patients with left, right, and bilateral temporomandibular joint reconstruction prostheses were 16, 10, and 09, respectively. The mean ± SD COR score for all 35 case reports and the individual item of the SCARE guidelines was 70.2 ± 10.5% and 66.5 ± 31.2%, respectively. The minimum and maximum COR score was found for "Keywords" (0.0%) and "Introduction" (100%) and "Clinical Findings" (100%), respectively. Adequate reporting was found for 20/35 (57%) case reports.

CONCLUSIONS

This study revealed that case reports in major high-quality craniomaxillofacial journals suffer from insufficient reporting. Widespread adoption of available standards, such as SCARE guidelines, is proposed to improve the quality and robustness of case reporting.

Completeness of Reporting in Diet- and Nutrition-Related Randomized Controlled Trials and Systematic Reviews With Meta-Analysis: Protocol for 2 Independent Meta-Research Studies

BACKGROUND

Journal articles describing randomized controlled trials (RCTs) and systematic reviews with meta-analysis of RCTs are not optimally reported and often miss crucial details. This poor reporting makes assessing these studies' risk of bias or reproducing their results difficult. However, the reporting quality of diet- and nutrition-related RCTs and meta-analyses has not been explored.

OBJECTIVE

We aimed to assess the reporting completeness and identify the main reporting limitations of diet- and nutrition-related RCTs and meta-analyses of RCTs, estimate the frequency of reproducible research practices among these RCTs, and estimate the frequency of distorted presentation or spin among these meta-analyses.

METHODS

Two independent meta-research studies will be conducted using articles published in PubMed-indexed journals. The first will include a sample of diet- and nutrition-related RCTs; the second will include a sample of systematic reviews with meta-analysis of diet- and nutrition-related RCTs. A validated search strategy will be used to identify RCTs of nutritional interventions and an adapted strategy to identify meta-analyses in PubMed. We will search for RCTs and meta-analyses indexed in 1 calendar year and randomly select 100 RCTs (June 2021 to June 2022) and 100 meta-analyses (July 2021 to July 2022). Two reviewers will independently screen the titles and abstracts of records yielded by the searches, then read the full texts to confirm their eligibility. The general features of these published RCTs and meta-analyses will be extracted into a research electronic data capture database (REDCap; Vanderbilt University). The completeness of reporting of each RCT will be assessed using the items in the CONSORT (Consolidated Standards of Reporting Trials), its extensions, and the TIDieR (Template for Intervention Description and Replication) statements. Information about practices that promote research transparency and reproducibility, such as the publication of protocols and statistical analysis plans will be collected. There will be an assessment of the completeness of reporting of each meta-analysis using the items in the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) statement and collection of information about spin in the abstracts and full-texts. The results will be presented as descriptive statistics in diagrams or tables. These 2 meta-research studies are registered in the Open Science Framework.

RESULTS

The literature search for the first meta-research retrieved 20,030 records and 2182 were potentially eligible. The literature search for the second meta-research retrieved 10,918 records and 850 were potentially eligible. Among them, random samples of 100 RCTs and 100 meta-analyses were selected for data extraction. Data extraction is currently in progress, and completion is expected by the beginning of 2023.

CONCLUSIONS

Our meta-research studies will summarize the main limitation on reporting completeness of nutrition- or diet-related RCTs and meta-analyses and provide comprehensive information regarding the particularities in the reporting of intervention studies in the nutrition field.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/43537.

The importance of establishing a core outcome set for endodontic clinical trials and outcomes studies

Endodontic therapy aims to preserve teeth by preventing and treating apical disease, therefore, evaluation of treatment outcome in clinical trials and outcomes studies should effectively assess if it achieves these aims. Traditionally, treatment outcomes have been reported by clinicians after history and clinical examination in what is known as clinician-reported outcomes (CROs). Much less commonly employed however, are patient-reported outcomes (PROs) in which patients directly report on their condition. Endodontic treatment outcome reporting is evolving from a focus on CROs to increasing consideration for patient and disease-focused outcomes, with different criteria being proposed for assessment of treatment outcomes. Unfortunately, this has led to considerable variability and a lack of consensus on the definition, appropriate measurement and reporting of these outcomes. Heterogeneity in outcome reporting in clinical research provides a significant major barrier to conduct meta-analysis, guidelines development, clinical decision making, and ultimately affecting patient care. These effects could, however, be reduced by the establishment of a core outcome set (COS) in endodontics, which is defined as an agreed, standardized set of outcomes that should be included, measured and reported as a minimum in all trials and outcome studies. COS development is a regulated and validated process requiring involvement of appropriate stakeholders as well as a rigorous methodology. To date, COS has been developed for the management of traumatic dental injuries, orthodontic and periodontal treatment and is currently being developed for endodontic treatment. The aim of this review is to discuss the importance of COS in endodontics with focus on the evidence for and impact of heterogeneity in reporting endodontic treatment outcomes. An overview of an ongoing process for development of COS for different endodontic treatment modalities will also be provided.

Study Protocol: Multi-level Determinants of Implementation and Sustainment in the Education Sector

Evidence-based programs (EBPs) delivered in elementary schools show great promise in reducing risk for emotional and behavioral disorders (EBDs). However, efforts to sustain EBPs in school face barriers. Improving EBP sustainment thus represents a priority, but little research exists to inform the development of sustainment strategies. To address this gap, the Sustaining Evidenced-Based Innovations through Multi-level Implementation Constructs (SEISMIC) project will: (a) Determine if malleable individual, intervention, and organizational factors predict EBP treatment fidelity and modifications during implementation, sustainment, or both; (b) Assess the impact of EBP fidelity and modifications on child outcomes during implementation and sustainment; and (c) Explore the mechanisms through which individual, intervention, and organizational factors influence sustainment outcomes. This protocol paper describes SEISMIC, which builds upon a federally-funded RCT evaluating BEST in CLASS, a teacher-delivered program for K-3rd grade children at risk for EBDs. The sample will include 96 teachers, 384 children, and 12 elementary schools. A multi-level, interrupted time series design will be used to examine the relationship between baseline factors, treatment fidelity, modifications, and child outcomes, followed by a mixed-method approach to elucidate the mechanisms that influence sustainment outcomes. Findings will be used to create a strategy to improve EBP sustainment in schools.

Evaluation of Risk of Bias in Neuroimaging-Based Artificial Intelligence Models for Psychiatric Diagnosis: A Systematic Review

IMPORTANCE

Neuroimaging-based artificial intelligence (AI) diagnostic models have proliferated in psychiatry. However, their clinical applicability and reporting quality (ie, feasibility) for clinical practice have not been systematically evaluated.

OBJECTIVE

To systematically assess the risk of bias (ROB) and reporting quality of neuroimaging-based AI models for psychiatric diagnosis.

EVIDENCE REVIEW

PubMed was searched for peer-reviewed, full-length articles published between January 1, 1990, and March 16, 2022. Studies aimed at developing or validating neuroimaging-based AI models for clinical diagnosis of psychiatric disorders were included. Reference lists were further searched for suitable original studies. Data extraction followed the CHARMS (Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies) and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. A closed-loop cross-sequential design was used for quality control. The PROBAST (Prediction Model Risk of Bias Assessment Tool) and modified CLEAR (Checklist for Evaluation of Image-Based Artificial Intelligence Reports) benchmarks were used to systematically evaluate ROB and reporting quality.

FINDINGS

A total of 517 studies presenting 555 AI models were included and evaluated. Of these models, 461 (83.1%; 95% CI, 80.0%-86.2%) were rated as having a high overall ROB based on the PROBAST. The ROB was particular high in the analysis domain, including inadequate sample size (398 of 555 models [71.7%; 95% CI, 68.0%-75.6%]), poor model performance examination (with 100% of models lacking calibration examination), and lack of handling data complexity (550 of 555 models [99.1%; 95% CI, 98.3%-99.9%]). None of the AI models was perceived to be applicable to clinical practices. Overall reporting completeness (ie, number of reported items/number of total items) for the AI models was 61.2% (95% CI, 60.6%-61.8%), and the completeness was poorest for the technical assessment domain with 39.9% (95% CI, 38.8%-41.1%).

CONCLUSIONS AND RELEVANCE

This systematic review found that the clinical applicability and feasibility of neuroimaging-based AI models for psychiatric diagnosis were challenged by a high ROB and poor reporting quality. Particularly in the analysis domain, ROB in AI diagnostic models should be addressed before clinical application.

Validity of age estimation methods and reproducibility of bone/dental maturity indices for chronological age estimation

AIM

This systematic review sought to assess the validity of age estimation methods based on bone or dental maturity indices and their reproducibility through a meta-analysis of validation and reproducibility studies.

DATA SOURCES

A systematic online search was conducted in PubMed and Google Scholar.

STUDY SELECTION

Cross-sectional studies were included. The authors excluded articles without information on validity and reproducibility outcomes, articles not written in English or Italian, and those where it was impossible to obtain pooled reproducibility estimates of Cohen's kappa or the intraclass correlation coefficient (ICC) due to a lack of information on the variability measure.

DATA EXTRACTION AND SYNTHESIS

The authors tried to follow the preferred reporting items for systematic reviews and meta-analyses (PRISMA) protocol. They also considered the PICOS/PECOS strategy to assess the research questions in their included studies; nevertheless, no particular guideline was reported to be consistently followed in their study.

RESULTS

Twenty-three (23) studies were selected for data extraction and critical appraisal. The pooled male mean error of the age prediction was 0.08 years (95% CI: -0.12; 0.29), and the pooled female mean error was 0.09 years (95% CI: -0.12; 0.30). Studies using Nolla's method had a mean error closest to zero with a slight overestimation: mean male age prediction error of 0.02 (95% CI: -0.37; 0.41) and mean female age prediction error of 0.03 (95% CI: -0.34; 0.41). Haavikko's method had a mean error of -1.12 (95% CI: -2.29; 0.06) and -1.33 (95% CI: -2.54; -0.13) for males and females, respectively. Cameriere's method also underestimated the chronological age and was the only method with a higher absolute mean error for males than females (males: -0.22 [95% CI: -0.44; 0.00]; females: -0.17 [95% CI: -0.34; -0.01]). Overall, Demirjian's and Willems's methods tended to overestimate chronological age in both males (Demirjian: 0.59 [95% CI: 0.28; 0.91]; Willems: 0.07 [95% CI: -0.17; 0.31]) and females (Demirjian: 0.64 [95% CI 0.38; 0.90]; Willems: 0.09 [95% CI: -0.13; 0.31]). The prediction intervals (PI) overlapped zero for all methods, rendering the difference between estimated and chronological ages not statistically significant for males and females. Cameriere's method showed the smallest PI for both biological genders, while the Haavikko and other methods had the widest intervals. No heterogeneity was observed in inter-examiner (heterogeneity: Q = 5.78, p = 0.888) and intra-examiner (heterogeneity: Q = 9.11, p = 0.611) agreement, so a fixed-effects model was used. For inter-examiner agreement, the ICC ranged from 0.89 to 0.99, and the meta-analytic pooled ICC was 0.98 (95% CI 0.97; 1.00), which was near-perfect reliability. Concerning intra-examiner agreement, the ICCs ranged from 0.90 to 1.00, and the meta-analytic pooled ICC was 0.99 (95% CI 0.98; 1.00), which was also close to perfect reliability.

CONCLUSIONS

This study recommended the Nolla and Cameriere methods as preferred approaches while mentioning that the Cameriere method was validated on a smaller sample size than Nolla's, thus requiring further testing on additional populations to better assess the mean error estimates by sex. However, the evidence in this paper is of very low quality and offers no certainty.

The Potential of Medicinal Plants and Natural Products in the Treatment of Burns and Sunburn-A Review

Treating burns remains a challenge for modern medicine, especially in developing countries that cannot afford expensive, advanced therapies. This review article summarises clinical and animal model studies of botanical preparations and their mixtures in treating burn wounds and sunburn. Articles available in electronic databases such as PubMed, Scopus, Web of Science, Science Direct and Google Scholar, published in English in 2010-2022, were considered. In the described clinical trials, it was shown that some herbal preparations have better effectiveness in treating burn wounds, including shortening the healing time and reducing inflammation, than the conventional treatment used hitherto. These herbal preparations contained extracts from Albizia julibrissin, Alkanna tinctoria, Aloe vera, Arnebia euchroma, Betula pendula and Betula pubescens, Centella asiatica, Hippophaë rhamnoides, Juglans regia, Lawsonia inermis, and mixtures of Matricaria chamomilla and Rosa canina. Research on animal models shows that many extracts may potentially benefit the treatment of burn wounds and sunburn. Due to the diverse mechanism of action, antibacterial activity, the safety of use and cost-effectiveness, herbal preparations can compete with conventional treatment. The growing interest in alternative medicine and herbal medicine encourages further research. Not only single preparations but also their mixtures should be taken into account because the research conducted so far often suggests a synergistic effect of the ingredients.

[Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extensionDiretrizes para relatórios de ensaios clínicos com intervenções que utilizam inteligência artificial: a extensão CONSORT-AI]

The CONSORT 2010 statement provides minimum guidelines for reporting randomized trials. Its widespread use has been instrumental in ensuring transparency in the evaluation of new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate impact on health outcomes. The CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence) extension is a new reporting guideline for clinical trials evaluating interventions with an AI component. It was developed in parallel with its companion statement for clinical trial protocols: SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence). Both guidelines were developed through a staged consensus process involving literature review and expert consultation to generate 29 candidate items, which were assessed by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed upon in a two-day consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants). The CONSORT-AI extension includes 14 new items that were considered sufficiently important for AI interventions that they should be routinely reported in addition to the core CONSORT 2010 items. CONSORT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, the human-AI interaction and provision of an analysis of error cases. CONSORT-AI will help promote transparency and completeness in reporting clinical trials for AI interventions. It will assist editors and peer reviewers, as well as the general readership, to understand, interpret and critically appraise the quality of clinical trial design and risk of bias in the reported outcomes.

Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review

OBJECTIVES

To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process.

STUDY DESIGN AND SETTING

Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts.

RESULTS

Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance.

CONCLUSIONS

The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic.

The research status and progress of core outcome set in oral health

The core outcome set (COS) refers to the minimum set of outcomes that should be reported by all clinical trials in a particular health field. The use of COS in clinical studies can reduce the heterogeneity caused by using different outcomes across different clinical studies, facilitate the systematic review of different clinical studies on the same topic, reduce selective reporting bias, and increase the utility of clinical studies. The importance of COS in oral health has recently been recognized. This review summarizes the history, necessity, and key methodological points of COS development, with emphasis on the research status and existing problems in COS development, in the field of oral health.

The reporting of prognostic prediction models for obstetric care was poor: a cross-sectional survey of 10-year publications

BACKGROUND

To investigate the reporting of prognostic prediction model studies in obstetric care through a cross-sectional survey design.

METHODS

PubMed was searched to identify prognostic prediction model studies in obstetric care published from January 2011 to December 2020. The quality of reporting was assessed by the TRIPOD checklist. The overall adherence by study and the adherence by item were calculated separately, and linear regression analysis was conducted to explore the association between overall adherence and prespecified study characteristics.

RESULTS

A total of 121 studies were included, while no study completely adhered to the TRIPOD. The results showed that the overall adherence was poor (median 46.4%), and no significant improvement was observed after the release of the TRIPOD (43.9 to 46.7%). Studies including both model development and external validation had higher reporting quality versus those including model development only (68.1% vs. 44.8%). Among the 37 items required by the TRIPOD, 10 items were reported adequately with an adherence rate over of 80%, and the remaining 27 items had an adherence rate ranging from 2.5 to 79.3%. In addition, 11 items had a report rate lower than 25.0% and even covered key methodological aspects, including blinding assessment of predictors (2.5%), methods for model-building procedures (4.5%) and predictor handling (13.5%), how to use the model (13.5%), and presentation of model performance (14.4%).

CONCLUSIONS

In a 10-year span, prognostic prediction studies in obstetric care continued to be poorly reported and did not improve even after the release of the TRIPOD checklist. Substantial efforts are warranted to improve the reporting of obstetric prognostic prediction models, particularly those that adhere to the TRIPOD checklist are highly desirable.

Developing a knowledge translation program for health practitioners: Allied Health Translating Research into Practice

Background

Front-line health practitioners lack confidence in knowledge translation, yet they are often required to undertake projects to bridge the knowledge-practice gap. There are few initiatives focused on building the capacity of the health practitioner workforce to undertake knowledge translation, with most programs focusing on developing the skills of researchers. This paper reports the development and evaluation of a knowledge translation capacity building program for allied health practitioners located over geographically dispersed locations in Queensland, Australia.

Methods

Allied Health Translating Research into Practice (AH-TRIP) was developed over five years with consideration of theory, research evidence and local needs assessment. AH-TRIP includes five components: training and education; support and networks (including champions and mentoring); showcase and recognition; TRIP projects and implementation; evaluation. The RE-AIM framework (Reach, Effectiveness, Adoption, Implementation Maintenance) guided the evaluation plan, with this paper reporting on the reach (number, discipline, geographical location), adoption by health services, and participant satisfaction between 2019 and 2021.

Results

A total of 986 allied health practitioners participated in at least one component of AH-TRIP, with a quarter of participants located in regional areas of Queensland. Online training materials received an average of 944 unique page views each month. A total of 148 allied health practitioners have received mentoring to undertake their project, including a range of allied health disciplines and clinical areas. Very high satisfaction was reported by those receiving mentoring and attending the annual showcase event. Nine of sixteen public hospital and health service districts have adopted AH-TRIP.

Conclusion

AH-TRIP is a low-cost knowledge translation capacity building initiative which can be delivered at scale to support allied health practitioners across geographically dispersed locations. Higher adoption in metropolitan areas suggests that further investment and targeted strategies are needed to reach health practitioners working in regional areas. Future evaluation should focus on exploring the impact on individual participants and the health service.

Scientific Publishing in Biomedicine: A Brief History of Scientific Journals

Scientific publishing, with about 350-year historical background, has played a central role in advancing science by disseminating new findings, generalizing accepted theories, and sharing novel ideas. The number of scientific journals has exponentially grown from 10 at the end of the 17th century to 100,000 at the end of the 20th century. The publishing landscape has dramatically changed over time from printed journals to online publishing. Although scientific publishing was initially non-commercial, it has become a profitable industry with a significant global financial turnover, reaching $28 billion in annual revenue before the COVID-19 pandemic. However, scientific publishing has encountered several challenges and is suffering from unethical practices and some negative phenomena, like publish-or-perish, driven by the need to survive or get a promotion in academia. Developing a global landscape with collaborative non-commercial journals and platforms is a primary proposed model for the future of scientific publishing. Here, we provide a brief history of the foundation and development of scientific journals and their evolution over time. Furthermore, current challenges and future perspectives of scientific publishing are discussed.

Low Research-Data Availability in Educational-Psychology Journals: No Indication of Effective Research-Data Policies

Research-data availability contributes to the transparency of the research process and the credibility of educational-psychology research and science in general. Recently, there have been many initiatives to increase the availability and quality of research data. Many research institutions have adopted research-data policies. This increased awareness might have raised the sharing of research data in empirical articles. To test this idea, we coded 1,242 publications from six educational-psychology journals and the psychological journal Cognition (as a baseline) published in 2018 and 2020. Research-data availability was low (3.85% compared with 62.74% in Cognition) but has increased from 0.32% (2018) to 7.16% (2020). However, neither the data-transparency level of the journal nor the existence of an official research-data policy on the level of the corresponding author’s institution was related to research-data availability. We discuss the consequences of these findings for institutional research-data-management processes.

Centre for Statistical and Methodological Excellence (CESAME): A Consortium Initiative for Improving Methodology in Randomised Clinical Trials

When conducting randomised clinical trials, the choice of methodology and statistical analyses will influence the results. If the planned methodology is not of optimal quality and predefined in detail, there is a risk of biased trial results and interpretation. Even though clinical trial methodology is already at a very high standard, there are many trials that deliver biased results due to the implementation of inadequate methodology, poor data quality and erroneous or biased analyses. To increase the internal and external validity of randomised clinical trial results, several international institutions within clinical intervention research have formed The Centre for Statistical and Methodological Excellence (CESAME). Based on international consensus, the CESAME initiative will develop recommendations for the proper methodological planning, conduct and analysis of clinical intervention research. CESAME aims to increase the validity of randomised clinical trial results which will ultimately benefit patients worldwide across medical specialities. The work of CESAME will be performed within 3 closely interconnected pillars: (1) planning randomised clinical trials; (2) conducting randomised clinical trials; and (3) analysing randomised clinical trials.