A systematic review of evaluation in formal continuing medical education

A guide to outcome evaluation of simulation-based education programmes in low and middle-income countries

Evaluation is a vital part of any learning activity and is essential to optimize and improve educational programmes. It should be considered and prioritized prior to the implementation of any learning activity. However, comprehensive programme evaluation is rarely conducted, and there are numerous barriers to high-quality evaluation. This review provides a framework for conducting outcome evaluation of simulation-based education programmes in low and middle-income countries (LMICs). The basis of evaluation, including core ideas of theory, purpose and structure are outlined, followed by an examination of the levels and healthcare applications of the Kirkpatrick model of evaluation. Then, methods of conducting evaluation of simulation-based education in LMICs are discussed through the lens of a successful surgical simulation programme in Myanmar, a lower-middle-income country. The programme involved the evaluation of 11 courses over 4 years in Myanmar and demonstrated evaluation at the highest level of the Kirkpatrick model. Reviewing this programme provides a bridge between evaluation theory and practical implementation. A range of evaluation methods are outlined, including surveys, interviews, and clinical outcome measurement. The importance of a mixed-methods approach, enabling triangulation of quantitative and qualitative analysis, is highlighted, as are methods of analysing data, including statistical and thematic analysis. Finally, issues and challenges of conducting evaluation are considered, as well as strategies to overcome these barriers. Ultimately, this review informs readers about evaluation theory and methods, grounded in a practical application, to enable other educators in low-resource settings to evaluate their own activities.

A triangulation model for assessment of change in classroom behavior of medical teachers participating in faculty development program on lecturing skills

BACKGROUND & AIMS

We utilized a triangulation method of a faculty development program's (FDP) evaluation comprising short-course workshops on classroom behaviors and lecturing skills of basic sciences faculty in a medical school.

METHODS & RESULTS

This study utilized data from the pre and post evaluation of classroom lectures by an expert observer. Course participants were observed before the inception of a 4-month FDP and after 6-months of program completion. Findings at 6-month post-FDP interval were supplemented with students' and participant's self-evaluation. Expert evaluation of 15 participants showed that more participants were summarizing lectures at the end of their class (p = 0.021), utilizing more than one teaching tool (p  =  0.008) and showing a well-structured flow of information (p = 0.013). Among the students, majority (95.5%, n  =  728) agreed on "teachers were well-prepared for the lecture", however, a low number (66.1%, n  =  504) agreed on "teachers were able to make the lecture interesting". On self-evaluation (n =  12), majority of the participants (91.7%, n  =  11) thought these FDP workshops had a positive impact on their role as a teacher.

CONCLUSIONS

Gathering feedback from multiple sources can provide a more holistic insight into the impact of an FDP and can provide a robust framework for setting up future FDP targets.

A Conceptual Framework for Continuing Medical Education and Population Health

ISSUE

Health systems have been increasingly called upon to address population health concerns and continuing medical education (CME) is an important means through which clinical practices can be improved. This manuscript elaborates on existing conceptual frameworks in order to support CME practitioners, funders, and policy makers to develop, implement, and evaluate CME vis-a-vis population health concerns.

EVIDENCE

Existing CME conceptual models and conceptions of CME effectiveness require elaboration in order to meet goals of population health improvement. Frameworks for the design, implementation and evaluation of CME consistently reference population health, but do not adequately conceptualize it beyond the aggregation of individual patient health. As a pertinent example, opioid prescribing CME programs use the opioid epidemic to justify their programs, but evaluation approaches are inadequate for demonstrating population health impacts. CME programs that are built to have population health outcomes using frameworks intended primarily for physician performance and patient health outcomes are thus not able to recognize either non-linear associations or negative unintended consequences.

IMPLICATIONS

This proposed conceptual framework draws on the fields of clinical population medicine, the social determinants of health, health equity, and philosophies of population health to build conceptual bridges between the CME outcome levels of physician performance and patient health to population health. The authors use their experience developing, delivering, and evaluating opioid prescribing- and poverty-focused CME programs to argue that population health-focused CME must be re-oriented in at least five ways. These include: 1) scaling effective CME programs while evaluating at population health levels; 2) (re)interpreting evidence for program content from a population perspective; 3) incorporating social determinants of health into clinically-oriented CME activities; 4) explicitly building fluency in population health concepts and practices among health care providers and CME planners; and 5) attending to social inequity in every aspect of CME programs.

Interest and need for continuing medical education in pediatric complementary and integrative medicine: a cross-sectional survey from Switzerland

Background

Pediatric integrative medicine, combining conventional and complementary medical approaches for children and adolescents, is an integral part of the health care system in Switzerland. However, there is still a lack of complementary and integrative medicine topics in training and continuing educational programs. For the first time on a national level, the 2021 annual conference of the Swiss Society of Pediatrics was entirely dedicated to the topic of integrative medicine.

Methods

Using a cross-sectional online survey, this study investigated congress participants’ evaluation and feedback with the aim to assess whether the program had met their objectives and to get empirical data on their attitude, expectations and needs regarding pediatric complementary and integrative medicine. Descriptive methods were used to present the results.

Results

Among 632 participants of the conference, 228 completed the evaluation form (response rate 36%). The overall feedback about the congress and the main theme of pediatric integrative medicine was clearly positive. The majority of respondents had achieved their educational objectives including complementary and integrative medicine issues. 82% were motivated to learn more about complementary and integrative medicine and 66% were stimulated to integrate complementary therapies into their professional practice.

Conclusion

This study from Switzerland confirms the interest in integrative medicine among pediatricians and supports the need for pre- and postgraduate pediatric training on topics related to complementary and integrative medicine. Developing and adapting training and continuing medical education based on evaluations of participant feedback can promote professional development and improve patient care for the benefit of physicians and patients.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12906-022-03581-6.

Piloting a Faculty Development Program in a Rural Haitian Teaching Hospital

Background:

Faculty development for nurse and physician educators has a limited evidence base in high income countries, and very little research from low- and middle-income countries. Health professions educators in many global settings do not receive training on how to educate effectively.

Objective:

To pilot and assess a faculty development program aimed at nurse and physician educators at a teaching hospital in rural Haiti.

Methods:

We developed a program covering a total of 22 topics in health professions education, including applied learning theory as well as nurse and physician targeted topics. We assessed impact through participant assessment of personal growth, participant evaluation of the program, knowledge testing pre and post program, and structured observations of program participants providing teaching during the program.

Findings:

Nineteen out of 37 participants completed the program. While participant reviews were uniformly positive, a pre- and post-test on general educational topics showed no significant change, and the effort to institute observation and feedback of teaching did not succeed.

Conclusions:

Our project showcases some benefits of faculty development, while also demonstrating the challenges of instituting faculty development in situations where participants have limited time and resources. We suspect more benefits may emerge as the program evolves to fit the learners and setting.

Evaluations of Continuing Health Provider Education Focused on Opioid Prescribing: A Scoping Review

PURPOSE

Continuing health provider education (HPE) is an important intervention supported by health policy to counter the opioid epidemic; knowledge regarding appropriate program design and evaluation is lacking. The authors aim to provide a comprehensive understanding of evaluations of opioid-related continuing HPE programs and their appropriateness as interventions to improve population health.

METHOD

In January 2020, the authors conducted a systematic search of 7 databases, seeking studies of HPE programs on opioid analgesic prescribing and overdose prevention. Reviewers independently screened the titles and abstracts of all studies and then assessed the full texts of all studies potentially eligible for inclusion. The authors extracted a range of data using categories for evaluating complex programs: the use of theory, program purpose, inputs, activities, outputs, outcomes, and industry involvement. Results were reported in a narrative synthesis.

RESULTS

Thirty-nine reports on 32 distinct HPE programs met inclusion criteria. Of these 32, 31 (97%) were U.S./Canadian programs and 28 (88%) were reported after 2010. Measurements of changes in knowledge and confidence were common. Performance outcomes were less common and typically self-reported. Most studies (n = 27 [84%]) used concerns of opioid-related harms at the population health level to justify the educational intervention, but only 5 (16%) measured patient- or population-level outcomes directly related to the educational programs. Six programs (19%) had direct or indirect opioid manufacturer involvement.

CONCLUSIONS

Continuing HPE has been promoted as an important means of addressing population-level opioid-related harms by policymakers and educators, yet published evaluations of HPE programs focusing on opioid analgesics inadequately evaluate patient- or population-level outcomes. Instead, they primarily focus on self-reported performance outcomes. Conceptual models are needed to guide the development and evaluation of continuing HPE programs intended to have population health benefits.

Development and Preliminary Evaluation of an Education Program for Primary Care Teams on Discussing Firearms Storage Safety with Veterans

BACKGROUND

Reducing access to lethal means is one of the few empirically supported approaches for lowering suicide rates, and safe firearms storage practices have been associated with reduced risk of death by suicide. Although there is substantial opportunity for primary care to assist in addressing lethal means with veterans, approaches to intervention and educating staff are not well documented. We sought to 1) describe development of an education program for primary care teams to help them discuss firearms storage safety (FSS) with veterans during primary care visits; and 2) conduct a preliminary evaluation of the pilot education program.

METHODS

We used an iterative process involving veterans and primary care staff stakeholders to develop program content, format, and supplemental materials. A grounded theory approach was used to analyze data from focus groups and individual interviews. Following piloting of the program with 71 staff members in two primary care clinics, we analyzed pre- and post-training participant surveys of program satisfaction and attitudes comfort related to firearms safety discussions.

RESULTS

During the development phase, 68 veterans and 107 staff members participated in four veteran focus groups and four primary care focus groups, respectively, and/or individual interviews. The program that was developed, "'Just in Case': Discussing means safety with veterans at elevated risk for suicide," addresses knowledge and skills learning objectives, and includes video demonstrations and skills practice. Survey data obtained just prior to the pilot training sessions showed low self-reported rates of discussing firearms safety with veterans who may be at elevated risk for suicide. Immediate post-training data showed generally high satisfaction with the program and significant improvements in participant self-reported ratings of the importance of, and comfort with FSS.

CONCLUSIONS

This interactive knowledge and skill-based means safety curriculum shows promise as a means for educating primary care staff to deliver messaging about firearms safety to veterans. Additional research is needed to refine and evaluate impacts of this or similar training programs on clinician and veteran behaviors over time.

Continuing Medical Education in Vietnam: A Weighted Analysis from Healthcare Professionals' Perception and Evaluation

PURPOSE

Continuing medical education (CME) is a compulsory requirement for every health professional. However, to date, little is known about the effectiveness of CME in Vietnam. This study assessed CME programs based on attendees' perception and evaluation.

METHODS

A cross-sectional study was conducted during a five-month period in all 62 CME courses at a university hospital. A self-report, anonymous questionnaire was distributed to the participants during the course and was collected at the end of the course. The questionnaire included questions about demographic characteristics, experiences during the course and participants' perception and evaluation as measured by the 19-item Program Evaluation Instrument (PEI). A higher score on the PEI indicates a higher level of positive reaction toward CME programs.

RESULTS

Among 1312 participants in the analysis, the majority were females (58.1%) with a mean age of 34.5 (SD = 10.6) years. Almost all participants had good, positive perceptions toward CME. However, about 5% of participants reported CME a waste of time. Participants reported a high score on the PEI (95.0±8.9) and all four dimensions including program objectives (20.7±2.2), learner's objectives (18.8±2.3), teacher's behavior (25.7±2.7) and program satisfaction (29.7±3.4). While there was no association between demographic characteristics and PEI score, attendance rate during the courses and perceptions toward CME were positively associated with PEI score.

CONCLUSION

CME programs receive positive reaction and evaluation from healthcare professionals and are helpful in providing and updating knowledge, attitude and practice in Vietnam. However, further studies are needed in other settings and specialties to fully understand the effectiveness of CME in Vietnam.

Coaching to enhance qualified surgeons' non-technical skills: a systematic review

INTRODUCTION

The lack of an effective continuing professional development programme for qualified surgeons, specifically one that enhances non-technical skills (NTS), is an issue receiving increased attention. Peer-based coaching, used in multiple professions, is a proposed method to deliver this. The aim of this study was to undertake a systematic review of the literature to summarize the quantity and quality of studies involving surgical coaching of NTS in qualified surgeons.

METHODS

A systematic search of the literature was performed through MEDLINE, EMBASE, Cochrane Collaboration and PsychINFO. Studies were selected based on predefined inclusion and exclusion criteria. Data for the included studies was independently extracted by two reviewers and the quality of the studies evaluated using the Medical Education and Research Study Quality Instrument (MERSQI).

RESULTS

Some 4319 articles were screened from which 19 met the inclusion criteria. Ten studies involved coaching of individual surgeons and nine looked at group coaching of surgeons as part of a team. Group coaching studies used non-surgeons as coaches, included objective assessment of NTS, and were of a higher quality (average MERSQI 13.58). Individual coaching studies focused on learner perception, used experienced surgeons as coaches and were of a lower quality (average MERSQI 11.58). Individual coaching did not show an objective improvement in NTS for qualified surgeons in any study.

CONCLUSION

Surgical coaching of qualified surgeons' NTS in a group setting was found to be effective. Coaching of individual surgeons revealed an overall positive learner perception but did not show an objective improvement in NTS for qualified surgeons.

Continuing education meetings and workshops: effects on professional practice and healthcare outcomes

BACKGROUND

Educational meetings are used widely by health personnel to provide continuing medical education and to promote implementation of innovations or translate new knowledge to change practice within healthcare systems. Previous reviews have concluded that educational meetings can result in small changes in behaviour, but that effects vary considerably. Investigations into which characteristics of educational meetings might lead to greater impact have yielded varying results, and factors that might explain heterogeneity in effects remain unclear. This is the second update of this Cochrane Review.

OBJECTIVES

• To assess the effects of educational meetings on professional practice and healthcare outcomes • To investigate factors that might explain the heterogeneity of these effects SEARCH METHODS: We searched CENTRAL, MEDLINE, Embase, ERIC, Science Citation Index Expanded (ISI Web of Knowledge), and Social Sciences Citation Index (last search in November 2016).

SELECTION CRITERIA

We sought randomised trials examining the effects of educational meetings on professional practice and patient outcomes.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data and assessed risk of bias. One review author assessed the certainty of evidence (GRADE) and discussed with a second review author. We included studies in the primary analysis that reported baseline data and that we judged to be at low or unclear risk of bias. For each comparison of dichotomous outcomes, we measured treatment effect as risk difference adjusted for baseline compliance. We expressed adjusted risk difference values as percentages, and we noted that values greater than zero favour educational meetings. For continuous outcomes, we measured treatment effect as per cent change relative to the control group mean post test, adjusted for baseline performance; we expressed values as percentages and noted that values greater than zero favour educational meetings. We report means and 95% confidence intervals (CIs) and, when appropriate, medians and interquartile ranges to facilitate comparisons to previous versions of this review. We analysed professional and patient outcomes separately and analysed 22 variables that were hypothesised a priori to explain heterogeneity. We explored heterogeneity by using univariate meta-regression and by inspecting violin plots.

MAIN RESULTS

We included 215 studies involving more than 28,167 health professionals, including 142 new studies for this update. Educational meetings as the single intervention or the main component of a multi-faceted intervention compared with no intervention • Probably slightly improve compliance with desired practice when compared with no intervention (65 comparisons, 7868 health professionals for dichotomous outcomes (adjusted risk difference 6.79%, 95% CI 6.62% to 6.97%; median 4.00%; interquartile range 0.29% to 13.00%); 28 comparisons, 2577 health professionals for continuous outcomes (adjusted relative percentage change 44.36%, 95% CI 41.98% to 46.75%; median 20.00%; interquartile range 6.00% to 65.00%)) • Probably slightly improve patient outcomes compared with no intervention (15 comparisons, 2530 health professionals for dichotomous outcomes (adjusted risk difference 3.30%, 95% CI 3.10% to 3.51%; median 0.10%; interquartile range 0.00% to 4.00%); 28 comparisons, 2294 health professionals for continuous outcomes (adjusted relative percentage change 8.35%, 95% CI 7.46% to 9.24%; median 2.00%; interquartile range -1.00% to 21.00%)) The certainty of evidence for this comparison is moderate. Educational meetings alone compared with other interventions • May improve compliance with desired practice when compared with other interventions (6 studies, 1402 health professionals for dichotomous outcomes (adjusted risk difference 9.99%, 95% CI 9.47% to 10.52%; median 16.5%; interquartile range 0.80% to 16.50%); 2 studies, 72 health professionals for continuous outcomes (adjusted relative percentage change 12.00%, 95% CI 9.16% to 14.84%; median 12.00%; interquartile range 0.00% to 24.00%)) No studies met the inclusion criteria for patient outcome measurements. The certainty of evidence for this comparison is low. Interactive educational meetings compared with didactic (lecture-based) educational meetings • We are uncertain of effects on compliance with desired practice (3 studies, 370 health professionals for dichotomous outcomes; 1 study, 192 health professionals for continuous outcomes) or on patient outcomes (1 study, 54 health professionals for continuous outcomes), as the certainty of evidence is very low Any other comparison of different formats and durations of educational meetings • We are uncertain of effects on compliance with desired practice (1 study, 19 health professionals for dichotomous outcomes; 1 study, 20 health professionals for continuous outcomes) or on patient outcomes (1 study, 113 health professionals for continuous outcomes), as the certainty of evidence is very low. Factors that might explain heterogeneity of effects Meta-regression suggests that larger estimates of effect are associated with studies judged to be at high risk of bias, with studies that had unit of analysis errors, and with studies in which the unit of analysis was the provider rather than the patient. Improved compliance with desired practice may be associated with: shorter meetings; poor baseline compliance; better attendance; shorter follow-up; professionals provided with additional take-home material; explicit building of educational meetings on theory; targeting of low- versus high-complexity behaviours; targeting of outcomes with high versus low importance; goal of increasing rather than decreasing behaviour; teaching by opinion leaders; and use of didactic versus interactive teaching methods. Pre-specified exploratory analyses of behaviour change techniques suggest that improved compliance with desired practice may be associated with use of a greater number of behaviour change techniques; goal-setting; provision of feedback; provision for social comparison; and provision for social support. Compliance may be decreased by the use of follow-up prompts, skills training, and barrier identification techniques.

AUTHORS' CONCLUSIONS

Compared with no intervention, educational meetings as the main component of an intervention probably slightly improve professional practice and, to a lesser extent, patient outcomes. Educational meetings may improve compliance with desired practice to a greater extent than other kinds of behaviour change interventions, such as text messages, fees, or office systems. Our findings suggest that multi-strategy approaches might positively influence the effects of educational meetings. Additional trials of educational meetings compared with no intervention are unlikely to change the review findings; therefore we will not further update this review comparison in the future. However, we note that randomised trials comparing different types of education are needed.

A retrospective quantitative implementation evaluation of Safer Opioid Prescribing, a Canadian continuing education program

BACKGROUND

Continuing health professions education (CHPE) is an important policy intervention for the opioid epidemic. Besides effectiveness or impact, health policy implementation should be studied to understand how an intervention was delivered within complex environments. Implementation outcomes can be used to help interpret CHPE effects and impacts, help answer questions of "how" and "why" programs work, and inform transferability. We evaluated Safer Opioid Prescribing (SOP), a national CHPE program, using implementation outcomes of reach, dose, fidelity, and participant responsiveness.

METHODS

We conducted a retrospective quantitative implementation evaluation of the 2014-2017 cohorts of SOP. To measure reach and dose, we examined participation and completion data. We used Ontario physician demographic data, including regulatory status with respect to controlled substances, to examine relevant trends. To measure fidelity and participant responsiveness, we analyzed participant-provided evaluations of bias, active learning, and relevance to practice. We used descriptive statistics and measures of association for both continuous and categorical variables. We used logistic regression to determine predictors of workshop participation and analysis of covariance to examine variation in satisfaction across different-sized sessions.

RESULTS

Reach: In total, there were 472 unique participants, 84.0% of whom were family physicians. Among Ontario physician participants, 90.0% were family physicians with characteristics representative of province-wide demographics. Dose: Webinar completion rate was 86.2% with no differences in completion based on rurality, gender, or controlled substance prescribing status with medical regulatory authorities. Fidelity and participant responsiveness: Nearly all participants rated the three webinars and workshop as balanced, and each element of SOP was also rated as highly relevant to clinical practice.

CONCLUSIONS

This evaluation demonstrates that Safer Opioid Prescribing was implemented as intended. Over a short period and without any external funding, the program reached more than 1% of the Ontario physician workforce. This suggests that the program may be a good model for using virtual CHPE to reach a critical mass of prescribers. This study represents a methodological advance of adapting evaluation methods from health policy and complex interventions for continuing health professions education. Future studies will assess effectiveness and impact on opioid prescribing and utilization within evaluation models of complex interventions.

Performance Change in Treating Tobacco Addiction: An Online, Interprofessional, Facilitated Continuing Education Course (TEACH) Evaluation at Moore's Level 5

INTRODUCTION

Continuing education is essential to build capacity among health care providers (HCPs) to treat people with tobacco addiction. Online, interprofessional training programs are valuable; however, interpretation and comparison of outcomes remain challenging because of inconsistent use of evaluation frameworks. In this study, we used level 5 of Moore's evaluation framework to examine whether an online training program in intensive tobacco cessation counseling achieved sustained performance change among HCPs across multiple health disciplines.

METHODS

The evaluation sample included 62 HCPs with direct clinical duties, who completed the online Training Enhancement in Applied Counseling and Health (TEACH) Core Course in 2015 and 2016. We compared self-reported changes in cessation counseling and clinical practices across eight core competencies from baseline to 6-month follow-up using McNemar's tests and descriptive analyses.

RESULTS

Compared with baseline, significantly more HCPs reported providing cessation counseling at 6-month follow-up (44% versus 81%, P < .001). HCPs also reported significant increases in engagement in six of the eight core competencies.

DISCUSSION

Online training in intensive tobacco cessation treatment can result in sustained performance improvement at 6 months. However, availability of resources and clinical context may influence the extent to which HCPs are able to implement their learned skills. Furthermore, continuing education programs should consider the use of consistent evaluation frameworks to promote cross program comparisons.

Ten caveats of learning analytics in health professions education: A consumer's perspective

A group of 22 medical educators from different European countries, gathered in a meeting in Utrecht in July 2019, discussed the topic of learning analytics (LA) in an open conversation and addressed its definition, its purposes and potential risks for learners and teachers. LA was seen as a significant advance with important potential to improve education, but the group felt that potential drawbacks of using LA may yet be under-exposed in the literature. After transcription and interpretation of the discussion's conclusions, a document was drafted and fed back to the group in two rounds to arrive at a series of 10 caveats educators should be aware of when developing and using LA, including too much standardized learning, with undue consequences of over-efficiency and pressure on learners and teachers, and a decrease of the variety of 'valid' learning resources. Learning analytics may misalign with eventual clinical performance and can run the risk of privacy breaches and inescapability of documented failures. These consequences may not happen, but the authors, on behalf of the full group of educators, felt it worth to signal these caveats from a consumers' perspective.

Criteria for describing and evaluating training interventions in healthcare professions - CRe-DEPTH

BACKGROUND

Descriptions of training intervention in healthcare include a lot of vagueness, leaving the reader in doubt about the intervention and the true value of the reported effects. A set of reporting criteria to improve the transparency and comprehensiveness in reporting training interventions in healthcare is recommended.

OBJECTIVE

This study aimed to develop and content validate a set of reporting criteria for the development and evaluation of training interventions for healthcare professionals.

METHOD

The set of criteria was developed along three stages: (1) a literature review, (2) the design of the criteria, and (3) a Delphi and consensus meeting process with ten experts from different professional disciplines.

RESULTS

The Criteria for Reporting on Development and Evaluation of Professional Training interventions in Healthcare (CRe-DEPTH) set was systematically developed and consisted of 12 items representing four categories: (1) development of the training, (2) characteristics of the training, (3) characteristics of the providers, and (4) assessment of the training outcomes. Each item has a detailed description to support its unambiguous interpretation.

CONCLUSIONS

The developed CRe-DEPTH reporting criteria guide high quality development, evaluation and reporting for complex training interventions, while preserving the flexibility to incorporate various teaching content or teaching methods. The Cre-DEPTH criteria will allow for detailed reporting on the training, its developmental process, as well as on the rationale for underlying choices, and will avoid non-reporting of training details.

Categorising the broad impacts of continuing professional development: a scoping review

CONTEXT

A number of systematic reviews have evaluated the impacts of continuing professional development (CPD). These reviews, due to their focused nature, may fail to capture the full range of impacts of CPD. This scoping review aims to explore the broader impacts of CPD with the intention of developing a categorisation of the types of impact of CPD.

METHODS

The authors searched MEDLINE, CINAHL and ERIC databases for studies published between 2007 and 2017 that looked at the impacts of formal CPD programmes for all health professionals. Studies were independently screened for eligibility; one reviewer charted data for all included studies, a sample of 10% was reviewed by a second reviewer. The charted data were analysed using both qualitative and quantitative content analysis.

RESULTS

The search returned 2750 manuscripts; 192 manuscripts describing 191 studies were included in this review. Most articles were from the USA (78 studies, 41%) and included medical doctors in the population (105 studies, 55%). Twelve categories of impact were generated through conventional content analysis: knowledge, practice change, skill, confidence, attitudes, career development, networking, user outcomes, intention to change, organisational change, personal change and scholarly accomplishments. Knowledge was most commonly measured (103 studies, 54%), whereas measurement of scholarly accomplishments was the least common (10 studies, 5%).

CONCLUSIONS

Existing evidence takes a narrow view when assessing the impacts of CPD. Emphasis on measuring impacts as knowledge, behaviour, confidence, skills and attitudes may be due to the widely accepted four levels of evaluation from the Kirkpatrick Model or because the majority of studies used quantitative methods. The categories proposed in this review may be used to capture a broader view of the impacts of CPD programmes, contributing to the evidence base for their value and translating into CPD programmes that truly transform health professionals, their careers and their practice.

Behavioural change in primary care professionals undertaking online education in dementia care in general practice

A challenge facing general practice is improving the diagnosis, management and care of people with dementia. Training and education for primary care professionals about knowledge and attitudes about dementia is needed. Online resources can provide educational opportunities for health professionals with limited access to dementia training. An online educational resource (four modules over 3 h) was designed to assist primary care practitioners to develop a systematic framework to identify, diagnose and manage patients with dementia within their practice. Interviews and questionnaires (knowledge, attitudes, confidence and behavioural intentions), with practice nurses and international medical graduates working in general practices, were used to evaluate the resource. Participants' knowledge, confidence and attitudes about dementia increased after completing the modules. Participants had strong intentions to apply a systematic framework to identify and manage dementia. In post-module interviews, participants reported increased awareness, knowledge and confidence in assessing and managing people with dementia, corroborating the questionnaire results. This project has demonstrated some early changes in clinical behaviour around dementia care in general practice. Promoting the value of applying a systematic framework with colleagues and co-workers could increase awareness of, and participation in, dementia assessment by other primary care professionals within general practices.

Health Care Providers' Profiles and Evaluations of a Statewide Online Education Program for Dissemination of Clinical Evidence on HIV, Hepatitis C Virus, and Sexually Transmitted Disease: Cross-Sectional Study

BACKGROUND

Timely and effective dissemination of the latest clinical evidence to health care providers is essential for translating biomedical research into routine patient care. Online platforms offer unique opportunities for dissemination of medical knowledge.

OBJECTIVE

In this study, we report the profiles of health care providers participating in the New York State HIV-HCV-STD Clinical Education Initiative online program and their evaluations of the online continuing professional development courses.

METHODS

We compiled professional and personal background information of the clinicians who completed at least one online course. We collected their self-reported program evaluation data with regard to the course content, format, knowledge increase, and impact on clinical practice.

RESULTS

We recorded a total of 4363 completions of 88 online courses by 1976 unique clinicians during a 12-month study period. The clinicians' background was diverse in terms of demographics, education levels, professional disciplines, practice years, employment settings, caseloads, and clinical services. The evaluation of online courses was very positive (usefulness/relevance, 91.08%; easy comprehension, 89.09%; knowledgeable trainer, 92.00%; appropriate format, 84.35%; knowledge increase, 48.52%; intention to use knowledge, 85.26%; and plan to change practice, 21.98%). Comparison with the reference data indicated that the online program successfully reached out to the primary care communities. Both the younger generation and the senior health care providers were attracted to the online program. High-quality multimedia resources, flexibility of access, ease of use, and provision of continuing professional development credits contributed to the initial success of this online clinical education program.

CONCLUSIONS

We have successfully characterized a diverse group of clinicians participating in a statewide online continuing professional development program. The evaluation has shown effective use of online resources to disseminate clinical evidence on HIV, hepatitis C virus, and sexually transmitted disease to primary care clinicians.

Effect of a two-day extensive continuing medical education course on participants' knowledge of clinical and operative urology

OBJECTIVE

Continuing Medical Education (CME) is an established method for facilitating the lifelong learning and developing knowledge, skills and attitudes to ensure delivery of a medical care which is up-to-date, evidence based, safe and patient-centered. An extensive 2 day urology course was conducted to meet the needs of learners. The purpose of the current study was to measure the effect of this CME activity on the knowledge of the participants.

MATERIAL AND METHODS

This quasi-experimental, single group pre-, and post-test study measured the gain in learning as a result of the two-day extensive CME course conducted by the section of Urology at Department of Surgery, the Aga Khan University, Karachi-Pakistan. Gain in knowledge, defined as the difference between the pre-test and the post-test scores, was taken as a measure of course effectiveness. The test comprised of 40 one-best type carefully constructed multiple choice questions (MCQs). Item analyses were also performed.

RESULTS

Forty-five out of a total of 70 participants from within and outside the city completed both pre and post-tests and were included in the study. The mean age of the subjects was 33.3+6.7 years. Of these 45 participants, 68.9% (n=31) of them were trainees at different levels. Mean gain in knowledge was 12.7±6.8% (p<0.01; 95% CI: 4.17-5.79). Mean test scores improved significantly from 37.8±11.3% to 50.3±10.8%. Difference in pre and post scores due to age, gender, practice type or years since start of training was not significant. The reliability of the test using Cronbach's α was 0.634.

CONCLUSION

CME sessions when designed and delivered carefully are effective means of increasing the knowledge significantly. Pre- and post-test is a reliable and valid strategy to measure gain in participants' knowledge.

Developing and evaluating clinical leadership interventions for frontline healthcare providers: a review of the literature

BACKGROUND

The importance of clinical leadership in ensuring high quality patient care is emphasized in health systems worldwide. Of particular concern are the high costs to health systems related to clinical litigation settlements. To avoid further cost, healthcare systems particularly in High-Income Countries invest significantly in interventions to develop clinical leadership among frontline healthcare workers at the point of care. In Low-Income Countries however, clinical leadership development is not well established. This review of the literature was conducted towards identifying a model to inform clinical leadership development interventions among frontline healthcare providers, particularly for improved maternal and newborn care.

METHODS

A structural literature review method was used, articles published between 2004 and 2017 were identified from search engines (Google Scholar and EBSCOhost). Additionally, electronic databases (CINHAL, PubMed, Medline, Academic Search Complete, Health Source: Consumer, Health Source: Nursing/Academic, Science Direct and Ovid®), electronic journals, and reference lists of retrieved published articles were also searched.

RESULTS

Employing pre-selected criteria, 1675 citations were identified. After screening 50 potentially relevant full-text papers for eligibility, 24 papers were excluded because they did not report on developing and evaluating clinical leadership interventions for frontline healthcare providers, 2 papers did not have full text available. Twenty-four papers met the inclusion criteria for review. Interventions for clinical leadership development involved the development of clinical skills, leadership competencies, teamwork, the environment of care and patient care. Work-based learning with experiential teaching techniques is reported as the most effective, to ensure the clinical leadership development of frontline healthcare providers.

CONCLUSIONS

All studies reviewed arose in High-Income settings, demonstrating the need for studies on frontline clinical leadership development in Low-and Middle-Income settings. Clinical leadership development is an on-going process and must target both novice and veteran frontline health care providers. The content of clinical leadership development interventions must encompass a holistic conceptualization of clinical leadership, and should use work-based learning, and team-based approaches, to improve clinical leadership competencies of frontline healthcare providers, and overall service delivery.

Strategies for improving outcomes in the acute management of ischemic stroke in rural emergency departments: a quality improvement initiative in the Stroke Belt

Background

The timely evaluation and initiation of treatment for acute ischemic stroke (AIS) is critical to optimal patient outcomes. However, clinical practice often falls short of guideline-established goals. Hospitals in rural regions of the USA, and notably those in the Stroke Belt, are particularly challenged to meet timing goals since the vast majority of primary stroke centers (PSCs) are concentrated in urban academic institutions.

Methods

Between May 2015 and May 2017, emergency department (ED) teams from 5 non-PSC hospitals in the Stroke Belt participated in a quality improvement (QI) initiative. The intervention included a baseline practice assessment survey, repeat audit-and-feedback cycles with patient data on AIS treatment timing, personalized Continuing Medical Education/Continuing Education-certified grand rounds sessions at each participating site with expert study faculty, targeted reinforcement of best practices, and follow-up to evaluate the benefits and limitations of the intervention.

Results

At the start of the initiative, clinical staff from participating EDs overestimated the proportion of patients with AIS who received alteplase within the guideline-recommended 60-minute door-to-needle window at their facility. At the end of the 6-month intervention period, significantly more patients were treated with alteplase within 60 minutes of ED arrival compared to baseline across the entire sample (1.9% of patients at baseline vs. 5.2% at 6 months; P < 0.01). Similarly, there was a trend toward a decrease in the percentage of patients whose alteplase treatment was initiated more than 60 minutes after their arrival at the ED (67.3% at baseline vs. 22.2% at 6 months).

Conclusion

Structured QI interventions that engage ED care teams to reflect on processes related to AIS diagnosis and treatment and deploy repeat audit-and-feedback cycles with real-time patient data have the potential to support an increase in the number of patients who receive alteplase within the guideline-recommended timeframe of 60 minutes from hospital arrival.

A Methodological Review of Confidence Measurement Scales for Dietitians Working with Individuals

Confidence is frequently used to assess practitioner effectiveness, and its evaluation requires valid and reliable domain-specific tools. However, the quality of existing measures is unknown. This review identifies studies measuring dietitians' confidence in working with individuals; assesses psychometric (measurement) qualities of relevant tools; and identifies areas for future research. Seven electronic databases, the internet, and reference lists were searched to identify the development or use of relevant confidence measurement scales. A quality assessment of psychometric properties was conducted using guidelines developed by Terwee and colleagues. Of the 15 measures reviewed, 4 were subject to factor analysis. Overall, content validity was strong. However, many measures rated poorly due to lack of factor analysis, inadequate sampling, or poor reporting. Of the dietetics-specific instruments, the Dietetic Confidence Scale and Nutrition Counselling Self-Efficacy Scale received the best ratings. The General Self-Efficacy Subscale also rated highly due to validation with the general population. This article highlights the need for dietitians to incorporate evidence-based methods into practice evaluation and instrument development. Dietitians need an awareness of the terminology and key criteria used to evaluate instrument quality to effectively collaborate with statisticians and scale development experts, and critically evaluate the quality of existing measures. Future scale development and reporting must incorporate psychometric evaluation, such as factor analysis, which should be used to explore and/or confirm scale dimensionality. There is broad scope for future methodological research with existing and new measures for nutrition and dietetics practitioners.

Point-of-Care Ultrasound for Jugular Venous Pressure Assessment: Live and Online Learning Compared

Introduction

Point-of-care ultrasound (POCUS) is a novel technique for the assessment of jugular venous pressure. Distance education may allow for efficient dissemination of this technique. We compared online learning to a live course for teaching ultrasonography jugular venous pressure (u-JVP) to determine if these teaching methods yielded different levels of comfort with and use of u-JVP.

Methods

This was an interventional trial of Canadian emergency physicians who had taken a basic POCUS course. The participants were in one of three Groups: online learning (Group OL), live teaching (Group LT), control (Group C). Group LT participants also took an advanced course prior to the study that included instruction in u-JVP. The participants who took the basic course were randomized to Group OL or Group C. Group OL was subject to the intervention, online learning. Group C only received an article citation regarding u-JVP. Questionnaires were completed before and after the intervention. The primary outcome was physician self-reported use and comfort with the technique of u-JVP after online learning compared to live teaching.

Results

Of the 287 advanced course participants, 42 completed the questionnaires (Group LT). Of the 3303 basic course participants, 47 who were assigned to Group OL completed the questionnaires and 47 from Group C completed the questionnaires. Use of u-JVP increased significantly in Group OL (from 15% to 55%) and Group C (from 21% to 47%) with the intervention. The comfort with use did not differ between Group LT and Group OL (p=0.14). The frequency of use remained higher in Group LT than Group OL (p=0.07).

Conclusion

Online learning increases the use and comfort with performing u-JVP for emergency physicians with prior POCUS experience. Although the comfort with use of u-JVP was similar in Groups LT and OL, online learning appears to yield levels of use that are less than those of a live course.

An evaluation of the translation of continuing education into diabetes public health care by pharmacists

Background Pharmacists are assuming greater public health roles and partaking in continuing education to advance knowledge and skills necessary for the provision of this patient care. Objective We sought to determine what conditions in a Middle East context influence how community pharmacists actually incorporate new information into practice. Setting Community pharmacies in Qatar. Methods A continuing professional development (CPD) program regarding the management of fasting diabetes patients during Ramadan was developed and delivered. Participants then maintained a record of their patient encounters when attempting to screen fasting diabetes patients for risk and offer medication, lifestyle, and monitoring advice. Diary entries were coded using inductive methods and follow-up focus group discussion was conducted to further corroborate the thematic analysis. Main outcome measure Facilitators and barriers to care. Results Forty-one pharmacists attended the CPD program and 35 subsequently made at least one diary entry during the 3-weeks preceding and during Ramadan. One-hundred and forty-eight submitted records and the transcript of one focus group (n = 6) were analyzed. Three main factors were found to influence pharmacists' ability to engage use new knowledge and skills: situational, patient, and pharmacist. Patient reception was the overwhelming influence whereby positive interactions encouraged pharmacists to continue screening and counseling attempts, but difficult encounters were negative reinforcing stimuli in almost equal measure. Conclusion In this Middle East setting, environmental factors play a considerable role in the pharmacists' ability to engage in public health care and reinforce that continuing education for health professionals must be closely aligned with the realities of practice and purposefully considered as part of its evaluation.

Evaluating investment in quality improvement capacity building: a systematic review

PURPOSE

Leading health systems have invested in substantial quality improvement (QI) capacity building, but little is known about the aggregate effect of these investments at the health system level. We conducted a systematic review to identify key steps and elements that should be considered for system-level evaluations of investment in QI capacity building.

METHODS

We searched for evaluations of QI capacity building and evaluations of QI training programmes. We included the most relevant indexed databases in the field and a strategic search of the grey literature. The latter included direct electronic scanning of 85 relevant government and institutional websites internationally. Data were extracted regarding evaluation design and common assessment themes and components.

RESULTS

48 articles met the inclusion criteria. 46 articles described initiative-level non-economic evaluations of QI capacity building/training, while 2 studies included economic evaluations of QI capacity building/training, also at the initiative level. No system-level QI capacity building/training evaluations were found. We identified 17 evaluation components that fit within 5 overarching dimensions (characteristics of QI training; characteristics of QI activity; individual capacity; organisational capacity and impact) that should be considered in evaluations of QI capacity building. 8 key steps in return-on-investment (ROI) assessments in QI capacity building were identified: (1) planning-stakeholder perspective; (2) planning-temporal perspective; (3) identifying costs; (4) identifying benefits; (5) identifying intangible benefits that will not be included in the ROI estimation; (6) discerning attribution; (7) ROI calculations; (8) sensitivity analysis.

CONCLUSIONS

The literature on QI capacity building evaluation is limited in the number and scope of studies. Our findings, summarised in a Framework to Guide Evaluations of QI Capacity Building, can be used to start closing this knowledge gap.

A needs assessment of education research topics among surgical educators in the United States

BACKGROUND

There are currently no courses that focus specifically on surgical education research. A needs assessment of surgical educators is required to best design these courses.

METHODS

A cross-sectional survey-based study on all faculty members of the Association for Surgical Education was done to determine their education research needs.

RESULTS

The overall response rate was 15% and the majority of the 78 respondents were physicians (63%) in their mid- to late career stage (65%). Participants thought research topics should be taught at an advanced level in a workshop format. Senior educators were less interested than junior educators in learning to create conceptual frameworks (p = 0.038) and presenting their research at national meetings (p = 0.014).

CONCLUSIONS

Surgical educators desire more training in education research techniques that are taught in a workshop format at a national surgical education meeting. These workshops may lay the groundwork for a nationally recognized certificate in surgical education research.

Recruitment and retention strategies for public sector optometrists in KwaZulu-Natal Province, South Africa

Background: In South Africa, there is a paucity of optometrists serving the needs of the larger public sector. KwaZulu-Natal is one of the most densely populated provinces and home to several of the poorest districts. Despite an optometry school in the province, and with a lack of compulsory community service for new graduates, more optometrists are needed to serve the public sector. While studies on the recruitment and retention of medical and allied health professionals have been conducted, limited evidence exists on work trends of public sector optometrists.Methods: A cross-sectional study design using both quantitative and qualitative data collection methods was used. All public sector optometrists and local district health co-ordinators in the province were contacted, with an 80% (41 out of 51) and 75% (9 out of 12) response rate, respectively. Questionnaires containing demographic, recruitment, retention and open-ended questions were distributed by post, fax and email and via an online survey to both groups. Telephonic interviews were also conducted using semi-structured techniques. Frequency distributions, Fisher’s exact test and odds ratios were used to statistically describe the demographic data, while qualitative responses were recorded and analysed for commonly occurring themes.Results: The present public sector optometry workforce comprises mainly young (73%), black (70%), women (66%). They chose to work in the public sector to ‘make a difference’ and were attracted by ‘good working hours’ and ‘job security’. Fifty-three percent of optometrists work in the public sector due to a study bursary, for which there was a statistically significant association for race (p = 0.01), gender (p = 0.05) and background origin (p = 0.05). To aid their retention in public service, improved salaries, career progression, recognition, improved management relations and improved instrumentation were ranked highest by these optometrists.Conclusion: The demographic profile of presently serving public sector optometrists poses many human resource (HR) challenges and opportunities. Universities should pay attention to rural origin of students and provide exposure to rural clinical experiences during study. Departments of Health use study bursary incentives to recruit optometrists, but need to consider financial and non-financial incentives for their retention. At hospital level, a responsive HR management system should be implemented with emphasis on career management, recognition, improving infrastructure and supporting professional development.

Learning from a role model: A cascade or whirlpool effect?

BACKGROUND

Continuing Professional Development (CPD) and Faculty Development (FD) courses have been designed in the expectation that a cascade effect will occur, consisting of a conveyance of information from the courses to clinical trainers to daily practice and/or to trainees by means of role modeling.

PURPOSE

The aims of this study were to gain insight into factors that encourage clinical trainers to incorporate what they have learned in CPD/FD into their role model function and the factors that influence conveyance from master to apprentice.

METHOD

We conducted a qualitative study using semi-structured interviews with GP trainers and their trainees.

RESULTS

Twenty-four GP trainers who completed a CPD/FD course and sixteen of their trainees participated in the study. Analysis of their statements enabled the identification of factors that affect the amplification of the competences of clinical trainers and their awareness of being a role model, the applicability in training practice and conveyance to the trainee.

CONCLUSIONS

As a result of interactions between the trainer, trainee and patient, it seems more accurate to represent the conveyance of competences from master to apprentice using the image of a whirlpool rather than a cascade, with the influential factors and interactions functioning as filters, causing a decline in the effectiveness of CPD/FD. Using the filters as a basis for turning-points for improvements around the whirlpool could increase the effectiveness of CPD/FD.

Proposal of a linear rather than hierarchical evaluation of educational initiatives: the 7Is framework

Extensive resources are expended attempting to change clinical practice; however, determining the effects of these interventions can be challenging. Traditionally, frameworks to examine the impact of educational interventions have been hierarchical in their approach. In this article, existing frameworks to examine medical education initiatives are reviewed and a novel '7Is framework' discussed. This framework contains seven linearly sequenced domains: interaction, interface, instruction, ideation, integration, implementation, and improvement. The 7Is framework enables the conceptualization of the various effects of an intervention, promoting the development of a set of valid and specific outcome measures, ultimately leading to more robust evaluation.

Can didactic continuing education improve clinical decision making and reduce cost of quality? Evidence from a case study

INTRODUCTION

Administration of human serum albumin (HSA) solutions for the resuscitation of critically ill patients remains controversial. The objective of this study was to assess the effect of continuing medical education (CME) on health care professionals' clinical decision making with regard to HSA administration and the costs of quality (COQ). A quasi-experimental study of time series association of CME intervention with COQ and use of HSA solution was conducted at the Surgery Department of the Hospital Valjevo, Serbia. The CME contained evidence-based criteria for HSA solution administration in surgical patients. The preintervention period was defined as January 2009 to May 2011. CME was provided in June 2011, with the postintervention period June 2011 to May 2012.

METHODS

Total mortality rate, the rate of nonsurgical mortality, the rate of surgical mortality, the rate of sepsis patient mortality, index of irrational use of HSA solutions, and number of hospital days per hospitalized patient were collected for each month as quality indicators. Statistical analysis was performed by multivariate autoregressive integrated moving average (MARIMA) modeling. The specification of the COQ was performed according to a traditional COQ model.

RESULTS

The CME intervention resulted in an average monthly reduction of the hospital days per hospitalized patient, the rate of sepsis patient mortality, index of irrational use of HSA solutions, and COQ for $593,890.77 per year.

DISCUSSION

Didactic CME presenting evidence-based criteria for HSA administration was associated with improvements in clinical decisions and COQ. In addition, this study demonstrates that models combining MARIMA and traditional COQ models can be useful in the evaluation of CME interventions aimed at reducing COQ.

Encouraging Reflection and Change in Clinical Practice: Evolution of a Tool

This article describes the systematic development and gradual transformation of a tool to guide participants in a continuing medical education program to reflect on their current practices and to make commitments to change. The continuous improvement of this tool was influenced by evolving needs of the program, reviews of relevant educational literature, feedback from periodic program surveys, interviews with group facilitators, and results from educational research studies. As an integral component of the educational process used in the Practice Based Small Group Learning Program, the current tool is designed to help family physicians think about what has been learned during each educational session and examine issues related to the implementation of evidence-based changes into their clinical practice. Lessons learned will be highlighted. Both the developmental processes employed and the practice reflection tool itself have applicability to other educational environments that focus on continuing professional development.

The impact of CME on physician performance and patient health outcomes: an updated synthesis of systematic reviews

INTRODUCTION

Since 1977, many systematic reviews have asked 2 fundamental questions: (1) Does CME improve physician performance and patient health outcomes? and (2) What are the mechanisms of action that lead to positive changes in these outcomes? The article's purpose is to synthesize the systematic review literature about CME effectiveness published since 2003.

METHODS

We identified 8 systematic reviews of CME effectiveness published since 2003 in which primary research studies in CME were reviewed and physicians' performance and/or patient health outcomes were included as outcome measures.

RESULTS

Five systematic reviews addressed the question of "Is CME Effective?" using primary studies employing randomized controlled trials (RCTs) or experimental design methods and concluded: (1) CME does improve physician performance and patient health outcomes, and (2) CME has a more reliably positive impact on physician performance than on patient health outcomes. The 8 systematic reviews support previous research showing CME activities that are more interactive, use more methods, involve multiple exposures, are longer, and are focused on outcomes that are considered important by physicians lead to more positive outcomes.

DISCUSSION

Future research on CME effectiveness must take account of the wider social, political, and organizational factors that play a role in physician performance and patient health outcomes. We now have 39 systematic reviews that present an evidence-based approach to designing CME that is more likely to improve physician performance and patient health outcomes. These insights from the scientific study of CME effectiveness should be incorporated in ongoing efforts to reform systems of CME and health care delivery.

Long-term effects of an educational seminar on antibiotic prescribing by GPs: a randomised controlled trial

BACKGROUND

High levels of outpatient antibiotic use remain observed in many European countries. Several studies have shown a strong relationship between antibiotic use and bacterial resistance.

AIM

To assess the long-term effect of a standardised educational seminar on antibiotic prescriptions by GPs.

DESIGN AND SETTING

Randomised controlled trial of 171 GPs (of 203 initially randomised) in France.

METHOD

GPs in the control group (n = 99) received no antibiotic prescription recommendation. Intervention group GPs (n = 72) attended an interactive seminar presenting evidence-based guidelines on antibiotic prescription for respiratory infections. The proportion of prescriptions containing an antibiotic in each group and related costs were compared to the baseline up to 30 months following the intervention. Data were obtained from the National Health Insurance System database.

RESULTS

In the intervention group, 4-6 months after the intervention, there was a significant decrease in the proportion of prescriptions containing an antibiotic from 15.2 ± 5.4% to 12.3 ± 5.8% (-2.8% [95% CI = -3.8 to -1.9], P<0.001). By contrast, an increase was observed in controls from 15.3 ± 6.0 to 16.4 ± 6.7% (+1.1% [95% CI = +0.4 to +1.8], P<0.01), resulting in a between-group difference of 3.93% ([95% CI = 2.75 to 5.11], P<0.001). The between-group difference was maintained 30 months after intervention (1.99% [95% CI = 0.56 to 3.42], P<0.01). Persistence of the intervention effect over the entire study period was confirmed in a hierarchical multivariate analysis.

CONCLUSION

This randomised trial shows that a standardised and interactive educational seminar results in a long-term reduction in antibiotic prescribing and could justify a large-scale implementation of this intervention.

Performance assessment through pre- and post-training evaluation of continuing medical education courses in prevention and management of cardio-vascular diseases in primary health care facilities of Armenia

To assess the difference of pre- and post-training performance evaluation of continuing medical education (CME) courses in cardio-vascular diseases (CVD) management among physicians at primary health care facilities of Armenian regions we conducted an evaluation survey. 212 medical records were surveyed on assessment of performance before and after the training courses through a self-employed structured questionnaire. Analysis of survey revealed statistically significant differences (p < 0.05) in a number of variables: threefold increased recording of lipids and body mass index (p = 0.001); moderate increased recording of comorbidities and aspirin prescription (p < 0.012); eightfold increased recording of dyslipidemia management plan, twofold increased recording for CVD management plan and fivefold increased recording for CVD absolute risk (p = 0.000). Missing records of electrocardiography and urine/creatinine analyses decreased statistically significantly (p < 0.05). Statistically significant decrease was observed in prescription of thiazides and angiotensin receptor blockers/angiotensin converting enzyme inhibitors (p < 0.005), while prescription of statins and statins with diet for dyslipidemia management showed increased recording (p < 0.05). Similarly, we observed increased records for counseling of rehabilitation physical activity (p = 0.006). In this survey most differences in pre- and post-evaluation of performance assessment may be explained by improved and interactive training modes, more advanced methods of demonstration of modeling. Current findings may serve a basis for future planning of CME courses for physicians of remote areas facing challenges in upgrading their knowledge, as well as expand the experience of performance assessment along with evaluation of knowledge scores.

Decreasing the ceiling effect in assessing meeting quality at an academic professional meeting

BACKGROUND

The psychometric properties of evaluations at academic meetings have not been well studied.

PURPOSE

To explore the ceiling effect in the evaluation of quality of a professional meeting and whether a change in the scale labels would decrease the ceiling effect.

METHODS

Cross-sectional study at two national meetings (2009-2010), attendees completed the evaluation on paper forms or online (5-point Likert scale).

RESULTS

Of 1,064 evaluations, the mean session ratings was higher among respondents to the paper version in 2009 (4.2; 95% confidence interval [CI], 4.1 to 4.3) as compared to online responders in 2009 (3.0; 95% CI, 2.9 to 3.1) or online responders in 2010 (3.0; 95% CI, 2.9 to 3.1)(p < 0.001).

CONCLUSION

A ceiling effect was present in the evaluation of an academic meeting. A change in the evaluation scale labels decreased the ceiling effect and increased evaluation variability.

Towards a conceptual framework demonstrating the effectiveness of audiovisual patient descriptions (patient video cases): a review of the current literature

BACKGROUND

Technological advances have enabled the widespread use of video cases via web-streaming and online download as an educational medium. The use of real subjects to demonstrate acute pathology should aid the education of health care professionals. However, the methodology by which this effect may be tested is not clear.

METHODS

We undertook a literature review of major databases, found relevant articles relevant to using patient video cases as educational interventions, extracted the methodologies used and assessed these methods for internal and construct validity.

RESULTS

A review of 2532 abstracts revealed 23 studies meeting the inclusion criteria and a final review of 18 of relevance. Medical students were the most commonly studied group (10 articles) with a spread of learner satisfaction, knowledge and behaviour tested. Only two of the studies fulfilled defined criteria on achieving internal and construct validity. The heterogeneity of articles meant it was not possible to perform any meta-analysis.

CONCLUSIONS

Previous studies have not well classified which facet of training or educational outcome the study is aiming to explore and had poor internal and construct validity. Future research should aim to validate a particular outcome measure, preferably by reproducing previous work rather than adopting new methods. In particular cognitive processing enhancement, demonstrated in a number of the medical student studies, should be tested at a postgraduate level.

Faculty development in medical education research

This 2012 Academic Emergency Medicine consensus conference breakout session was devoted to the task of identifying the history and current state of faculty development in education research in emergency medicine (EM). The participants set a future agenda for successful faculty development in education research. A number of education research and content experts collaborated during the session. This article summarizes existing academic and medical literature, expert opinions, and audience consensus to report our agreement and findings related to the promotion of faculty development.

Factor analysis methods and validity evidence: a review of instrument development across the medical education continuum

PURPOSE

Instrument development consistent with best practices is necessary for effective assessment and evaluation of learners and programs across the medical education continuum. The author explored the extent to which current factor analytic methods and other techniques for establishing validity are consistent with best practices.

METHOD

The author conducted electronic and hand searches of the English-language medical education literature published January 2006 through December 2010. To describe and assess current practices, she systematically abstracted reliability and validity evidence as well as factor analysis methods, data analysis, and reported evidence from instrument development articles reporting the application of exploratory factor analysis and principal component analysis.

RESULTS

Sixty-two articles met eligibility criteria. They described 64 instruments and 95 factor analyses. Most studies provided at least one source of evidence based on test content. Almost all reported internal consistency, providing evidence based on internal structure. Evidence based on response process and relationships with other variables was reported less often, and evidence based on consequences of testing was not identified. Factor analysis findings suggest common method selection errors and critical omissions in reporting.

CONCLUSIONS

Given the limited reliability and validity evidence provided for the reviewed instruments, educators should carefully consider the available supporting evidence before adopting and applying published instruments. Researchers should design for, test, and report additional evidence to strengthen the argument for reliability and validity of these measures for research and practice.

Impact of web-based case conferencing on cancer genetics training outcomes for community-based clinicians

INTRODUCTION

Technology and market forces are driving the demand for cancer risk assessment services in the community setting, where few clinicians are trained to order and interpret predictive genetic tests. City of Hope conducts a three-phase course in genetic cancer risk assessment (GCRA) for community-based clinicians, comprised of distance didactics, face-to-face workshops, and 12 months of professional development. As designed, the course cannot meet increasing demands for GCRA training. Action research identified face-to-face workshops as a barrier to increasing course capacity. This study compared the learning effectiveness of Web-based case conferencing to face-to-face training.

METHODS

A quasi-experimental design compared pre- to post-knowledge, skills, and professional self-efficacy outcomes from 2009 to 2010 course cohorts (n = 96). The intervention group (n = 52) engaged in Web-based case conferences during distance learning; the comparison group (n = 44) participated in the course as originally designed.

RESULTS

Both groups and all practice disciplines demonstrated significant pre- to post-increases on all measures. Knowledge increases were higher for the intervention group (p < 0.015); skills and self-efficacy increases were comparable between groups (p < 0.33 and p < 0.30, respectively).

DISCUSSION

Findings support the learning utility of Web-based case conferencing. Further studies may inform the development of tools to assess the impact of Web-based case conferencing on practice change and patient outcomes, in alignment with the highest standards of continuing professional development.