Information overload: what's behind it, what's beyond it?

Critical appraisal of published research papers - A reinforcing tool for research methodology: Questionnaire-based study

Background and Objectives:

Critical appraisal of published research papers is routinely conducted as a journal club (JC) activity in pharmacology departments of various medical colleges across Maharashtra, and it forms an important part of their postgraduate curriculum. The objective of this study was to evaluate the perception of pharmacology postgraduate students and teachers toward use of critical appraisal as a reinforcing tool for research methodology. Evaluation of performance of the in-house pharmacology postgraduate students in the critical appraisal activity constituted secondary objective of the study.

Materials and Methods:

The study was conducted in two parts. In Part I, a cross-sectional questionnaire-based evaluation on perception toward critical appraisal activity was carried out among pharmacology postgraduate students and teachers. In Part II of the study, JC score sheets of 2^nd- and 3^rd-year pharmacology students over the past 4 years were evaluated.

Results:

One hundred and twenty-seven postgraduate students and 32 teachers participated in Part I of the study. About 118 (92.9%) students and 28 (87.5%) faculties considered the critical appraisal activity to be beneficial for the students. JC score sheet assessments suggested that there was a statistically significant improvement in overall scores obtained by postgraduate students (n = 25) in their last JC as compared to the first JC.

Conclusion:

Journal article criticism is a crucial tool to develop a research attitude among postgraduate students. Participation in the JC activity led to the improvement in the skill of critical appraisal of published research articles, but this improvement was not educationally relevant.

WASP (Write a Scientific Paper): Critical appraisal of existing research

Critical appraisal of research involves a systematic process which assesses the question/s posed, study design and execution, statistical evaluation, interpretation of the results, and appropriateness of the conclusions. It necessitates identification of conflicts of interest, analyses the strengths and weaknesses of the study and ultimately the validity, reliability and relevance of the reported findings.

Critical appraisal of published literature

With a large output of medical literature coming out every year, it is impossible for readers to read every article. Critical appraisal of scientific literature is an important skill to be mastered not only by academic medical professionals but also by those involved in clinical practice. Before incorporating changes into the management of their patients, a thorough evaluation of the current or published literature is an important step in clinical practice. It is necessary for assessing the published literature for its scientific validity and generalizability to the specific patient community and reader's work environment. Simple steps have been provided by Consolidated Standard for Reporting Trial statements, Scottish Intercollegiate Guidelines Network and several other resources which if implemented may help the reader to avoid reading flawed literature and prevent the incorporation of biased or untrustworthy information into our practice.

Fuzzy Logic-Based Formalisms for Gynecology Disease Diagnosis

The very basis of the present article is the fact that the medical knowledge consisting of clinical presentation, diagnosis, and treatment of a disease is with imprecision and uncertainty. The overall approach in gynecological disease diagnosis could be divided into three distinct stages, and this was confirmed by seven experienced gynecologists. Stage 1 refers to an initial screening process in order to arrive at a single disease diagnosis for the patients, which is based only on the subjective information provided by patients to the physician. In stage 2, the patient who has not received a single diagnostic label in stage 1 is further investigated for a single disease diagnosis using past history criteria. If stage 2 fails to arrive at a single disease diagnosis for a patient, then physical examination and various tests like imaging tests, blood tests, etc., are conducted, and the test results are processed in stage 3. In stage 1, we have revisited fuzzy relational calculus and mathematically evaluated the perceptions of the domain experts (gynecologists) with respect to 31 gynecological diseases. The paper also presents the research findings with a case study focused on stage 2 using a type 1 fuzzy inference system. Out of 226 patients, 50 are correctly diagnosed for a single disease and 147 for multiple diseases in stage 1. The paper concludes that fuzzy relational calculus is an effective method as an “initial screening” process to arrive at a single disease diagnosis. We have identified 29 out of 226 patients satisfying past history criteria to achieve a single disease diagnosis by stage 2. Investigations for stage 3 are in progress.

Using MEDLINE Elemental Similarity to Assist in the Article Screening Process for Systematic Reviews

Background

Systematic reviews and their implementation in practice provide high quality evidence for clinical practice but are both time and labor intensive due to the large number of articles. Automatic text classification has proven to be instrumental in identifying relevant articles for systematic reviews. Existing approaches use machine learning model training to generate classification algorithms for the article screening process but have limitations.

Objective

We applied a network approach to assist in the article screening process for systematic reviews using predetermined article relationships (similarity). The article similarity metric is calculated using the MEDLINE elements title (TI), abstract (AB), medical subject heading (MH), author (AU), and publication type (PT). We used an article network to illustrate the concept of article relationships. Using the concept, each article can be modeled as a node in the network and the relationship between 2 articles is modeled as an edge connecting them. The purpose of our study was to use the article relationship to facilitate an interactive article recommendation process.

Methods

We used 15 completed systematic reviews produced by the Drug Effectiveness Review Project and demonstrated the use of article networks to assist article recommendation. We evaluated the predictive performance of MEDLINE elements and compared our approach with existing machine learning model training approaches. The performance was measured by work saved over sampling at 95% recall (WSS95) and the F-measure (F₁). We also used repeated analysis over variance and Hommel’s multiple comparison adjustment to demonstrate statistical evidence.

Results

We found that although there is no significant difference across elements (except AU), TI and AB have better predictive capability in general. Collaborative elements bring performance improvement in both F₁ and WSS95. With our approach, a simple combination of TI+AB+PT could achieve a WSS95 performance of 37%, which is competitive to traditional machine learning model training approaches (23%-41% WSS95).

Conclusions

We demonstrated a new approach to assist in labor intensive systematic reviews. Predictive ability of different elements (both single and composited) was explored. Without using model training approaches, we established a generalizable method that can achieve a competitive performance.

Characteristics of evidence-based medicine training in Royal College of Physicians and Surgeons of Canada emergency medicine residencies - a national survey of program directors

BACKGROUND

Recent surveys suggest few emergency medicine (EM) training programs have formal evidence-based medicine (EBM) or journal club curricula. Our primary objective was to describe the methods of EBM training in Royal College of Physicians and Surgeons of Canada (RCPSC) EM residencies. Secondary objectives were to explore attitudes regarding current educational practices including e-learning, investigate barriers to journal club and EBM education, and assess the desire for national collaboration.

METHODS

A 16-question survey containing binary, open-ended, and 5-pt Likert scale questions was distributed to the 14 RCPSC-EM program directors. Proportions of respondents (%), median, and IQR are reported.

RESULTS

The response rate was 93% (13/14). Most programs (85%) had established EBM curricula. Curricula content was delivered most frequently via journal club, with 62% of programs having 10 or more sessions annually. Less than half of journal clubs (46%) were led consistently by EBM experts. Four programs did not use a critical appraisal tool in their sessions (31%). Additional teaching formats included didactic and small group sessions, self-directed e-learning, EBM workshops, and library tutorials. 54% of programs operated educational websites with EBM resources. Program directors attributed highest importance to two core goals in EBM training curricula: critical appraisal of medical literature, and application of literature to patient care (85% rating 5 - "most importance", respectively). Podcasts, blogs, and online journal clubs were valued for EBM teaching roles including creating exposure to literature (4, IQR 1.5) and linking literature to clinical practice experience (4, IQR 1.5) (1-no merit, 5-strong merit). Five of thirteen respondents rated lack of expert leadership and trained faculty educators as potential limitations to EBM education. The majority of respondents supported the creation of a national unified EBM educational resource (4, IQR 1) (1-no support, 5- strongly support).

CONCLUSIONS

RCPSC-EM programs have established EBM teaching curricula and deliver content most frequently via journal club. A lack of EBM expert educators may limit content delivery at certain sites. Program directors supported the nationalization of EBM educational resources. A growing usage of electronic resources may represent an avenue to link national EBM educational expertise, facilitating future collaborative educational efforts.

Journal club for faculty or residents: a model for lifelong learning and maintenance of certification

The journal club offers a model for lifelong learning and maintenance of certification (MOC) for residents and faculty staff. First, it sharpens participants' critical appraisal skills by providing a space to discuss relevant medical literature. Second, it motivates participants to seek new medical literature on their own using technology. Our model sets forth a four-year journal club curriculum that could be used as one continuous curriculum or in bits and pieces. In the first year, the focus is teaching residents how to read an article. The second year focuses on what is of interests to the reader. The third year applies the resident's appraisal skills to assigned articles to test whether they can determine which have reliable and valid findings and which are flawed. In the fourth year residents are asked to distinguish whether articles are well researched and referenced. Our model also motivates participants to read articles in faculty journal clubs throughout their career. In most academic settings category 1 continuing medical education (CME) credits can be awarded so journal club can have the added benefit of satisfying maintenance of certification CME credits. From journal club both residents and faculty can learn what is new and learn to apply this new information in their practice. Finally, because technology creates an overabundance of relevant medical literature, participants using our model can develop strong critical appraisal skills and methods for organizing the information they find that make this information readily available for future use and retrieval.

Do family physicians retrieve synopses of clinical research previously read as email alerts?

Background

A synopsis of new clinical research highlights important aspects of one study in a brief structured format. When delivered as email alerts, synopses enable clinicians to become aware of new developments relevant for practice. Once read, a synopsis can become a known item of clinical information. In time-pressured situations, remembering a known item may facilitate information retrieval by the clinician. However, exactly how synopses first delivered as email alerts influence retrieval at some later time is not known.

Objectives

We examined searches for clinical information in which a synopsis previously read as an email alert was retrieved (defined as a dyad). Our study objectives were to (1) examine whether family physicians retrieved synopses they previously read as email alerts and then to (2) explore whether family physicians purposefully retrieved these synopses.

Methods

We conducted a mixed-methods study in which a qualitative multiple case study explored the retrieval of email alerts within a prospective longitudinal cohort of practicing family physicians. Reading of research-based synopses was tracked in two contexts: (1) push, meaning to read on email and (2) pull, meaning to read after retrieval from one electronic knowledge resource. Dyads, defined as synopses first read as email alerts and subsequently retrieved in a search of a knowledge resource, were prospectively identified. Participants were interviewed about all of their dyads. Outcomes were the total number of dyads and their type.

Results

Over a period of 341 days, 194 unique synopses delivered to 41 participants resulted in 4937 synopsis readings. In all, 1205 synopses were retrieved over an average of 320 days. Of the 1205 retrieved synopses, 21 (1.7%) were dyads made by 17 family physicians. Of the 1205 retrieved synopses, 6 (0.5%) were known item type dyads. However, dyads also occurred serendipitously.

Conclusion

In the single knowledge resource we studied, email alerts containing research-based synopses were rarely retrieved. Our findings help us to better understand the effect of push on pull and to improve the integration of research-based information within electronic resources for clinicians.

How to critically appraise an article

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.