Tool to assess recognition and understanding of elements in Summary of Findings Table for health evidence synthesis: a cross-sectional study

of Findings (SoF) tables concisely present the main findings of evidence synthesis of health evidence, but how users navigate it to understand and interpret the presented information is not clear. We quantified the interaction of medical students with an SoF table while answering a knowledge quiz. Read&Learn tool was used to measure the number of target and non-target table cells visited for each question and the time spent on these cells. Students positively identified target elements for quiz questions and answered simpler questions, but struggled with critical thinking and understanding study outcomes. The question on outcomes with the largest improvement post-intervention had the fewest correct answers, the longest interaction with table cells and the most opened cells before answering. Students spent a median of 72% of the time reading target table cells. A heatmap of the interactions showed that they were mostly answer-oriented. Further development of the tool and metrics is needed to use the tool and the metrics to study the cognitive processes during the assessment of health evidence.

Using evidence to decision frameworks led to guidelines of better quality and more credible and transparent recommendations

BACKGROUND AND OBJECTIVES

To determine whether the use of Evidence to Decision (EtD) frameworks is associated to higher quality of both guidelines and individual recommendations.

METHODS

We identified guidelines recently published by international organizations that have methodological guidance documents for their development. Pairs of researchers independently extracted information on the use of these frameworks, appraised the quality of the guidelines using the Appraisal of Guidelines, Research and Evaluation II Instrument (AGREE-II), and assessed the clinical credibility and implementability of the recommendations with the Appraisal of Guidelines for REsearch & Evaluation Recommendations Excellence (AGREE-REX) tool. We conducted both descriptive and inferential analyses.

RESULTS

We included 66 guidelines from 17 different countries, published in the last 5 years. Thirty guidelines (45%) used an EtD framework to formulate their recommendations. Compared to those that did not use a framework, those using an EtD framework scored higher in all domains of both AGREE-II and AGREE-REX (P < 0.05). Quality scores did not differ between the use of the The Grading of Recommendations Assessment, Development and Evaluation-EtD framework (17 guidelines) or another EtD framework (13 guidelines) (P > 0.05).

CONCLUSION

The use of EtD frameworks is associated with guidelines of better quality, and more credible and transparent recommendations. Endorsement of EtD frameworks by guideline developing organizations will likely increase the quality of their guidelines.

Student Moral Foundations in the Context of Research Misbehavior: An Experimental Study

The study aimed to examine whether the use of words related to different moral foundations can predict the perceived severity of research misconduct. We gave two groups of participants, undergraduate medical students, the same hypothetical scenarios of research misconduct cases containing words related to different aspects of morality, and asked them to assess how inappropriate the described behavior was. Students ranked the described behaviors differently by the inappropriateness of the behavior, but the group wording was not a significant predictor of appropriateness. The reasons for the ranking were not related to any standardized procedures for research integrity but were related to the moral assessment of the students, which was assessed using qualitative approach. The results of this study implicate that personal moral views are an important part in research integrity training.

Perception of organizational climate by university staff and students in medicine and humanities: A qualitative study

Organizational climate and culture are important for research organizations because they foster research integrity and responsible conduct of research, reduce questionable research practices, and improve job satisfaction. The aim of our study was to explore how employees and students perceive organizational climate and its consequences in the university setting. We conducted semi-structured interviews with senior students and employees (teaching and non-teaching staff) from two different university schools: School of Medicine and Faculty of Humanities and Social Sciences. Participants were asked questions regarding perceived climate, working environment, and the role of the institution. The data were analyzed using reflexive thematic analysis approach. Three themes were identified. The first theme addressed the difference in the perception and understanding of organizational climate. The second theme dealt with institutional issues emanating from organizational climate. The third theme described the behavior of stakeholders in the formation of organizational climate. Organizational climate is important concept in academic organizations as it influences both employees, particularly early career researchers, and students. Institutional leadership can strongly influence organizational climate, which can in turn affect job and job satisfaction. Due to the importance of personal morality on everyday decision-making, virtue-based research integrity training could be useful in improving academic institutions' organizational climate.

Web-Based Educational Intervention to Improve Knowledge of Systematic Reviews Among Health Science Professionals: Randomized Controlled Trial

Background

Lack of knowledge of systematic reviews (SRs) could prevent individual health care professionals from using SRs as a source of information in their clinical practice or discourage them from participating in such research.

Objective

In this randomized controlled trial, we evaluated the effect of a short web-based educational intervention on short-term knowledge of SRs.

Methods

Eligible participants were 871 Master’s students of university health sciences studies in Croatia; 589 (67.6%) students who agreed to participate in the trial were randomized using a computer program into 2 groups. Intervention group A (294/589, 49.9%) received a short web-based educational intervention about SR methodology, and intervention group B (295/589, 50.1%) was presented with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist. The participants’ knowledge of SRs was assessed before and after the intervention. The participants could not be blinded because of the nature of the intervention. The primary outcome was the difference in the percentage of correct answers about SR methodology per participant between the groups after the intervention, expressed as relative risk and 95% CI.

Results

Results from 162 and 165 participants in the educational intervention and PRISMA checklist groups, respectively, were available for analysis. Most of them (educational intervention group: 130/162, 80.2%; PRISMA checklist group: 131/165, 79.4%) were employed as health care professionals in addition to being health sciences students. After the intervention, the educational intervention group had 23% (relative risk percentage) more correct answers in the postintervention questionnaire than the PRISMA checklist group (relative risk=1.23, 95% CI 1.17-1.29).

Conclusions

A short web-based educational intervention about SRs is an effective tool for short-term improvement of knowledge of SRs among health care studies students, most of whom were also employed as health care professionals. Further studies are needed to explore the long-term effects of the tested education.

Trial Registration

OSF Registries 10.17605/OSF.IO/RYMVC; https://osf.io/rymvc

Addressing Challenges When Applying GRADE to Public Health Guidelines: A Scoping Review Protocol and Pilot Analysis

This is a protocol for a scoping review that aims to determine how guideline authors using the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) approach have addressed previously identified challenges related to public health. The Joanna Briggs Institute (JBI) methodology for scoping reviews will be followed. We will search and screen titles of guidelines for all languages published in 2013–2021 in: the GIN library, BIGG database, Epistemonikos GRADE guidelines repository, GRADEpro Database, MAGICapp, NICE and WHO websites. Two reviewers will independently screen full texts of the documents identified. The following information will be extracted: methods used for identifying different stakeholders and incorporating their perspectives; methods for identification and prioritization of non-health outcomes; methods for determining thresholds for decision-making; methods for incorporating and grading evidence from non-randomized studies; methods for addressing concerns with conditional recommendations in public health; methods for reaching consensus; additional methodological concerns; and any modifications made to GRADE. A combination of directed content analysis and descriptive statistics will be used for data analysis, and the findings presented narratively in a tabular and graphical form. In this protocol, we present the pilot results from 13 identified eligible guidelines issued between January and August 2021. We will publish the full review results when they become available.

Responsible research and innovation training programs: implementation and evaluation of the HEIRRI project

Responsible research and innovation, or RRI, is a concept that aims to bring together society and science for a better future. There are six key elements of RRI: public engagement, gender equality, science education, open access, ethics and governance.

Higher Education Institutions and Responsible Research and Innovation (HEIRRI) project aimed to bring the concept of RRI into the educational system. Using state-of-the-art review of good practices, HEIRRI team developed 10 training programs on RRI for different higher education institution educational levels, including a summer school and a massive open online course (MOOC). We conducted pilot of the trainings and evaluated participants’ experiences. Satisfaction with HEIRRI training programs on responsible research and innovation was high, both for participants and for the trainers, and trainings raised awareness of RRI. Participants’ feedback was used to identify areas that need improvement and provided for recommendations for final versions of the HEIRRI training programs.

In order to equip researchers with skills to recognize and apply RRI values, RRI should be included in their education. HEIRRI training is suitable for a range of different disciplines, including forensic science, and is free to use and adjust for specific contexts (available from: https://rri-tools.eu/heirri-training-programmes).

Supplemental data for this article is available online at https://doi.org/10.1080/20961790.2021.1970319 .

Antifragility of healthcare systems in Croatia and Bosnia and Herzegovina: Learning from man-made and natural crises

Acute crises, such as a war or a pandemic, are the ultimate tests for health care systems' resilience (temporary response to stress with change and adaptation) and antifragility (permanent benefit from change in response to stress). In this Health Policy paper, we analyse and discuss how the healthcare systems of two European countries - Bosnia and Herzegovina and Croatia - adapted to war as a man-made disaster, and how they adapted to COVID-19 pandemic twenty-five years later. These countries experienced full scale wars in recent history, which significantly changed their political and healthcare systems. This experience prepared the countries for the response to the pandemic, which coincided with two earthquakes in Croatia. We argue that healthcare systems in Croatia and Bosnia and Herzegovina are not only resilient but antifragile, and that they benefited from stressors they were exposed to. The antifragility of the two systems were primarily based on human effort - the strength, adaptability and resilience of health care professionals. We will look at lessons from the wars that were applied to the pandemic and discuss newly recognized opportunities and improvements.

Interventions for Organizational Climate and Culture in Academia: A Scoping Review

Organizational climate and culture may influence different work-related outcomes, including responsible conduct of research and research misconduct in academic or research organizations. In this scoping review we collected evidence on outcomes of interventions to change organizational climate or culture in academic or research settings. Out of 32,093 documents retrieved by the search, we analysed 207 documents in full text, out of which 7 met the eligibility criteria and were included in the final analysis. The included studies measured organizational climate (2 studies), organizational culture (4 studies), or both (1 study) at biomedical faculties (4 studies) or non-academic university departments (3 studies). Four studies had post-test, and three before-and-after study designs. The majority of interventions were face-to-face activities (meetings, different teambuilding activities), and two were based on organizational change. Six studies reported positive changes in organizational climate/culture after the intervention. These positive changes were measured as improvements in score on different questionnaire survey or were described through authors' or external evaluator's narrative reports. However, the methodological quality of the studies was low, both for qualitative and quantitative study designs. Replicable studies, using rigorous methods and clearly defined outcomes are urgently needed if organizations want to achieve a real change in organizational climate or culture for responsible research. The protocol for this scoping review was registered at https://osf.io/7zjqb .

The reporting checklist for public versions of guidelines: RIGHT-PVG

BACKGROUND

Public or patient versions of guidelines (PVGs) are derivative documents that "translate" recommendations and their rationale from clinical guidelines for health professionals into a more easily understandable and usable format for patients and the public. PVGs from different groups and organizations vary considerably in terms of quality of their reporting. In order to address this issue, we aimed to develop a reporting checklist for developers of PVGs and other potential users.

METHODS

First, we collected a list of potential items through reviewing a sample of PVGs, existing guidance for developing and reporting PVGs or other similar evidence-based patient tools, as well as qualitative studies on original studies of patients' needs about the content and/or reporting of information in PVGs or similar evidence-based patient tools. Second, we conducted a two-round Delphi consultation to determine the level of consensus on the items to be included in the final reporting checklist. Third, we invited two external reviewers to provide comments on the checklist.

RESULTS

We generated the initial list of 45 reporting items based on a review of a sample of 30 PVGs, four PVG guidance documents, and 46 relevant studies. After the two-round Delphi consultation, we formed a checklist of 17 items grouped under 12 topics for reporting PVGs.

CONCLUSION

The RIGHT-PVG reporting checklist provides an international consensus on the important criteria for reporting PVGs.

Cognitive levels in testing knowledge in evidence-based medicine: a cross sectional study

BACKGROUND

Knowledge assessment in evidence-based medicine (EBM) is usually performed by the measurement of memorised facts, understanding of EBM concepts and application of learned knowledge in familiar situations, all of which are considered lower-level educational objectives. The aim of this study was to assess EBM knowledge both on higher and lower cognitive levels across EBM topics.

METHODS

In order to assess knowledge on different EBM topics across learning levels, we created a knowledge test (Six Progressive Levels in Testing - SPLIT instrument), which consists of 36 multiple choice items and measures knowledge in EBM at six cognitive levels (Remembering, Understanding, Applying, Analysing, Evaluating and Creating) and addresses six EBM topics (Evidence-based practice, Internal validity, Clinical importance, Study design, Sources of evidence, Diagnostic studies). Three independent assessors defined the minimum passing score (MPS) for the overall test, based on the first-year course content and educational objectives. The instrument was assessed in a sample of first- (n = 119) and third-year medical students (n = 70) and EBM experts (n = 14).

RESULTS

The MPS was 16 correct answers out of total 36 questions, and was achieved by 21 out of 119 first-year students, 14 out of 70 third-year students and 9 out of 14 EBM experts (χ2 = 13.3; P < 0.001, with significantly higher proportion of experts passing compared to students). Although experts had the highest scores overall, none of the groups outperformed others on individual cognitive levels, but the experts outperformed students in EBM topics of Study design and Sources of evidence (P = 0.002 and 0.004, respectively, Kruskal-Wallis test). First- and third-year students performed better on specific course topics taught in that study year (Diagnostic studies and Clinical relevance, respectively).

CONCLUSION

EBM knowledge of students and experts differ according to the specificities of their education/expertise, but neither group had excellent knowledge in all areas. It may be difficult to develop a knowledge test that includes different EBM topics at different cognitive levels to follow the development of specific and general aspects of EBM knowledge.

Perception of Organizational Ethical Climate by University Staff and Students in Medicine and Humanities: A Cross Sectional Study

We assessed students' and employees' perception of ethical climate at a university school of medicine compared to that of social sciences and humanities, as well as temporal changes in the employees' perception of ethical climate. We also explored potential predictors of ethical climate, including moral foundations. This cross-sectional questionnaire study was conducted at the University of Split School of Medicine and the Faculty of Humanities and Social Sciences, in Croatia, from April to September 2019. We used 36-item Ethical Climate Questionnaire and 22-item Moral Foundation Questionnaire to survey employees, senior and doctoral students. We collected responses using ballot boxes as well as online survey. We collected 449 complete responses (response rate 36.8%). The dominant ethical climates at two schools were "Company rules and procedures" and "Laws and professional codes". We compared our results with a study conducted in 2012 and found that the perception of ethical climate had not changed dramatically in last 8 years. Ethical climate, or shared social and work-related behaviours, does not seem to change in these institutions even when students and staff are included with faculty in surveys. We provide further discussion of why this seems to be the case.

Framing the numerical findings of Cochrane plain language summaries: two randomized controlled trials

Background

Cochrane systematic review Plain language Summaries (CSR PLSs should serve as a tool for the evidence translation to non-medical population. However, the evidence of optimal type of numerical presentation in CSR PLSs is still scarce. The aim of this study was to investigate readers’ comprehension and preferences for different presentation of findings, including framing and numerical data, in Cochrane systematic review Plain Language Summaries (CSR PLSs).

Methods

We conducted a parallel randomized trial and a crossover randomized trial at the School of Medicine and family practice offices in Split, Croatia. The participants were students and consumers. We assessed possible differences in comprehension, measured by four questions on PLS content, of CSR PLSs depending on the positive or negative framing of results (n = 91) (Trial 1) or using percentages or frequencies for the presentation of results (n = 245) (Trial 2). The outcome measures were comprehension of PLS content, perceived effectiveness of the treatment and readiness to use the treatment (all on 1–10 scales).

Results

In Trial 1 we found no difference in readers’ perception of the effectiveness of the described treatment, desire that the treatment be offered by their family doctor, readiness to use the treatment, or comprehension when CSR PLS results were presented positively or negatively. In Trial 2 we found no difference in CSR PLS comprehension when results were presented as natural frequencies or percentages (BF₁₀ = 0.62, Bayesian t-test for independent samples).

Conclusions

Numerical presentation and framing direction of results appear to have no significant impact on understanding of messages in CSR PLSs.

Trial registration

The trials were registered in ClinicalTrials.gov. Protocol registration numbers: Trial 1: NCT03442387; Trial 2: NCT03554252.

Health numeracy skills of medical students:cross-sectional and controlled before-and-after study

BACKGROUND

Although numeracy, defined as understanding and handling numbers, is an important skill for the medical profession, it is not clear whether it changes during graduate medical education and whether it can be improved by specific interventions. The objective of this study was to assess objective and subjective numeracy levels at different stages of medical education and explore whether a research methodology/statistics course improves numeracy levels in a longer period.

METHODS

We performed cross-sectional and controlled before-and-after studies. First-year sociology students and first- to sixth-year medical students from the in the cross sectional study and two groups of first-year medical students in a controlled before-and-after study. The intervention was a course on biostatistics and research methodology using blended approach. Numeracy was measured using Subjective Numeracy Scale (Cronbach α = 0.70) and Numeracy Understanding in Medicine instrument (Cronbach α = 0.75).

RESULTS

Whereas first-year medical students did not differ from first-year sociology students in objective numeracy, medicine students had higher results on subjective numeracy. Students from higher years of medical school had generally higher subjective and objective numeracy scores. In the controlled before-and-after study, the intervention group improved more in subjective numeracy (median difference on a 0-8 scale = 0.5, 95% CI 0.3 to 0.7 vs - 0.4, 95% CI - 0.4 to - 0.1, P < 0.001) but not in objective numeracy.

CONCLUSIONS

Although the numeracy levels at the beginning of the medical school are within the range of non-medical population, both objective and subjective numeracy improve during the higher years of medical school. Curriculum during medical school may help in numeracy increase, while research methodology training may help to increase subjective but not objective numeracy skills.

A cross-sectional bibliometric study showed suboptimal journal endorsement rates of STROBE and its extensions

OBJECTIVES

The STrengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement provides guidance on reporting observational studies. Many extensions have been created for specialized methods or fields. We determined endorsement prevalence and typology by journals in extension-related fields.

STUDY DESIGN AND SETTING

A published protocol defined search strategies to identify journals publishing observational studies (2007-2017) across seven fields relating to STROBE extensions. We extracted text regarding STROBE, seven STROBE extensions, reporting guidelines Consolidated Standards of Reporting Trials and Preferred Reporting Items for Systematic Reviews and Meta-Analyses, and transparent reporting documents/groups: International Committee of Medical Journal Editors, Committee on Publication Ethics (COPE), and the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) networks. Relationships between endorsing STROBE, endorsing other guidelines, and journal impact factor were tested using chi square and Mann-Whitney tests.

RESULTS

Of 257 unique journals, 12 (5%) required STROBE on submission, 22 (9%) suggested use, 12 (5%) recommended a "relevant guideline," 72 (28%) mentioned it indirectly (via editorial policies or International Committee of Medical Journal Editors recommendations), and 139 (54%) did not mention STROBE. The relevant extension was required by 2 (<1%) journals; 4 (1%) suggested use. STROBE endorsement was not associated with journal impact indices but was with Consolidated Standards of Reporting Trials and Preferred Reporting Items for Systematic Reviews and Meta-Analyses endorsements.

CONCLUSION

Reporting guideline endorsement rates are low; information is vague and scattered. Unambiguous language is needed to improve adherence to reporting guidelines and increase the quality of reporting.

Reporting quality of European and Croatian health practice guidelines according to the RIGHT reporting checklist

Background

Health practice guidelines (HPGs) are important tools for the translation of evidence into practice. Reporting Items for Practice Guidelines in HealThcare (RIGHT) checklist provides guidance on reporting health practice guidelines (HPGs). We assessed the reporting completeness and quality of a set of national (Croatian) and relevant transnational (European) HPGs.

Methods

The national sample included all HPGs published in the official journal of the Croatian Medical Association in 2014–2016. We searched PubMed to identify relevant European guidelines (n = 24). Two independent reviewers assessed the adherence with the items on the RIGHT checklist. Kappa score was used to measure the level of agreement. Frequentist and Bayes statistics Bayes factor (BF₁₀) was used to evaluate the differences between the national and transnational HPGs.

Results

Overall, Croatian and European HPGs adhered to less than 50% of RIGHT checklist items. Croatian HPGs reported a median of 14.0 (95% CI 13.0–15.0) RIGHT reporting items, and European counterparts reported a median of 16.0 (95% CI 14.0–17.2) out of the total of 35 checklist items (Mann Whitney U test, P = 0.048; BF₁₀ = 1.543). European HPGs were better than Croatian HPGs in reporting stakeholder involvement and values and preferences (BF₁₀ = 80.63), as well as describing the implications of costs and resources (BF₁₀ = 55.15). Croatian HPGs better reported HPGs specified aims (BF₁₀ = 16.90), primary intended users (BF₁₀ = 8.70), and sources of funding (BF₁₀ = 122.90). Most insufficiently reported items for both HPG sets were defining the guideline questions and clear outcomes, quality assurance, management of funding and conflicts of interest, and guideline limitations.

Conclusions

Important methodological details are missing from most published HPGs at national and transnational levels. To ensure better quality and adequate use of HPGs, reporting guidelines should be endorsed and used by developers and users alike.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0828-4) contains supplementary material, which is available to authorized users.

Improving completion rates of students in biomedical PhD programs: an interventional study

BACKGROUND

Analysis of graduation success at the University of Split School of Medicine PhD programs conducted in 2011 revealed that only 11% of students who enrolled and completed their graduate coursework between 1999 and 2011 earned a doctoral degree. In this prospective cohort study we evaluated and compared three PhD programs within the same medical school, where the newest program, called Translational Research in Biomedicine (TRIBE), established in the academic year 2010/11, aimed to increase the graduation rate through an innovative approach.

METHODS

The intervention in the new program was related to three domains: redefined recruitment strategy, strict study regulations, and changes to the curriculum. We compared performance of PhD students between the new and existing programs and analyzed their current status, time to obtain a degree (from enrolment to doctorate), age at doctorate, number of publications on which the thesis was based and the impact factor of journals in which these were published.

RESULTS

These improvement strategies were associated with higher thesis completion rate and reduced time to degree for students enrolled in the TRIBE program. There was no change in the impact factor or number of publications that were the basis for the doctoral theses.

CONCLUSION

Our study describes good practices which proved useful in the design or reform of the PhD training program.