A self-instructional model to teach systems-based practice and practice-based learning and improvement

A Systematic Review on the Effectiveness of Systems-Based Practice Curricula in Health Professions Education

This systematic review aims to evaluate the effectiveness of systems-based practice (SBP) curricula from the perspective of health professions students and workers. A total of 8468 citations were sourced from six electronic databases and manual searches conducted independently by two researchers, of which 44 studies were eventually included. A meta-analysis using a random effects model and a meta-synthesis using the thematic synthesis approach were conducted. Most studies targeted medical students, residents, and resident physicians from various clinical specialties. Almost half of all studies focused on didactic or knowledge-based interventions to teach SBP. About a third of all studies measured non-self-evaluated knowledge change, clinical abilities, and clinical outcomes. Both meta-analysis and meta-synthesis results revealed positive outcomes of increased knowledge of SBP, increased recognition of SBP as a core competency in one's profession, and increased application of SBP knowledge in one's profession. Meta-synthesis results also revealed negative outcomes at the institutional and teacher/health professions level. This review highlights the importance of SBP education and supports the effectiveness of SBP curricula. There is a need to address the negative outcomes at the institutional and teacher/health professions level. Moreover, future studies could investigate the integration of self-assessment outcomes with comparison to some external standard.

A realist synthesis of quality improvement curricula in undergraduate and postgraduate medical education: what works, for whom, and in what contexts?

BACKGROUND

With the integration of quality improvement (QI) into competency-based models of physician training, there is an increasing requirement for medical students and residents to demonstrate competence in QI. There may be factors that commonly facilitate or inhibit the desired outcomes of QI curricula in undergraduate and postgraduate medical education. The purpose of this review was to synthesise attributes of QI curricula in undergraduate and postgraduate medical education associated with curricular outcomes.

METHODS

A realist synthesis of peer-reviewed and grey literature was conducted to identify the common contexts, mechanisms, and outcomes of QI curricula in undergraduate and postgraduate medical education in order to develop a programme theory to articulate what works, for whom, and in what contexts.

RESULTS

18854 records underwent title and abstract screening, full texts of 609 records were appraised for eligibility, data were extracted from 358 studies, and 218 studies were included in the development and refinement of the final programme theory. Contexts included curricular strategies, levels of training, clinical settings, and organisational culture. Mechanisms were identified within the overall QI curricula itself (eg, clear expectations and deliverables, and protected time), in the didactic components (ie, content delivery strategies), and within the experiential components (eg, topic selection strategies, working with others, and mentorship). Mechanisms were often associated with certain contexts to promote educational and clinical outcomes.

CONCLUSION

This research describes the various pedagogical strategies for teaching QI to medical learners and highlights the contexts and mechanisms that could potentially account for differences in educational and clinical outcomes of QI curricula. Educators may benefit from considering these contexts and mechanisms in the design and implementation of QI curricula to optimise the outcomes of training in this competency area.

Validity evidence for Quality Improvement Knowledge Application Tool Revised (QIKAT-R) scores: consequences of rater number and type using neurology cases

OBJECTIVES

To develop neurology scenarios for use with the Quality Improvement Knowledge Application Tool Revised (QIKAT-R), gather and evaluate validity evidence, and project the impact of scenario number, rater number and rater type on score reliability.

METHODS

Six neurological case scenarios were developed. Residents were randomly assigned three scenarios before and after a quality improvement (QI) course in 2015 and 2016. For each scenario, residents crafted an aim statement, selected a measure and proposed a change to address a quality gap. Responses were scored by six faculty raters (two with and four without QI expertise) using the QIKAT-R. Validity evidence from content, response process, internal structure, relations to other variables and consequences was collected. A generalisability (G) study examined sources of score variability, and decision analyses estimated projected reliability for different numbers of raters and scenarios and raters with and without QI expertise.

RESULTS

Raters scored 163 responses from 28 residents. The mean QIKAT-R score was 5.69 (SD 1.06). G-coefficient and Phi-coefficient were 0.65 and 0.60, respectively. Interrater reliability was fair for raters without QI expertise (intraclass correlation = 0.53, 95% CI 0.30 to 0.72) and acceptable for raters with QI expertise (intraclass correlation = 0.66, 95% CI 0.02 to 0.88). Postcourse scores were significantly higher than precourse scores (6.05, SD 1.48 vs 5.22, SD 1.5; p < 0.001). Sufficient reliability for formative assessment (G-coefficient > 0.60) could be achieved by three raters scoring six scenarios or two raters scoring eight scenarios, regardless of rater QI expertise.

CONCLUSIONS

Validity evidence was sufficient to support the use of the QIKAT-R with multiple scenarios and raters to assess resident QI knowledge application for formative or low-stakes summative purposes. The results provide practical information for educators to guide implementation decisions.

Radiology Research in Quality and Safety: Current Trends and Future Needs

Promoting quality and safety research is now essential for radiology as reimbursement is increasingly tied to measures of quality, patient safety, efficiency, and appropriateness of imaging. This article provides an overview of key features necessary to promote successful quality improvement efforts in radiology. Emphasis is given to current trends and future opportunities for directing research. Establishing and maintaining a culture of safety is paramount to organizations wishing to improve patient care. The correct culture must be in place to support quality initiatives and create accountability for patient care. Focused educational curricula are necessary to teach quality and safety-related skills and behaviors to trainees, staff members, and physicians. The increasingly complex healthcare landscape requires that organizations build effective data infrastructures to support quality and safety research. Incident reporting systems designed specifically for medical imaging will benefit quality improvement initiatives by identifying and learning from system errors, enhancing knowledge about safety, and creating safer systems through the implementation of standardized practices and standards. Finally, validated performance measures must be developed to accurately reflect the value of the care we provide for our patients and referring providers. Common metrics used in radiology are reviewed with focus on current and future opportunities for investigation.

Validity and Reliability of a Tool to Assess Quality Improvement Knowledge and Skills in Pediatrics Residents

BACKGROUND

Residency programs are expected to educate residents in quality improvement (QI). Effective assessments are needed to ensure residents gain QI knowledge and skills. Limitations of current tools include poor interrater reliability and requirement for scorer training.

OBJECTIVE

To provide evidence for the validity of the Assessment of Quality Improvement Knowledge and Skills (AQIKS), which is a new tool that provides a summative assessment of pediatrics residents' ability to recall QI concepts and apply them to a clinical scenario.

METHODS

We conducted a quasi-experimental study to measure the AQIKS performance in 2 groups of pediatrics residents: postgraduate year (PGY) 2 residents who participated in a 1-year longitudinal QI curriculum, and a concurrent control group of PGY-1 residents who received no formal QI training. The curriculum included 20 hours of didactics and participation in a resident-led QI project. Three faculty members with clinical QI experience, who were not involved in the curriculum and received no additional training, scored the AQIKS.

RESULTS

Complete data were obtained for 30 of 37 residents (81%) in the intervention group, and 36 of 40 residents (90%) in the control group. After completing a QI curriculum, the intervention group's mean score was 40% higher than at baseline (P < .001), while the control group showed no improvement (P = .29). Interrater reliability was substantial (κ = 0.74).

CONCLUSIONS

The AQIKS detects an increase in QI knowledge and skills among pediatrics residents who participated in a QI curriculum, with better interrater reliability than currently available assessment tools.

Patient Safety and Quality Improvement Education in Otolaryngology Residency: Preliminary Look at a Module-Based Approach

Since publication of the Institute of Medicine's report To Err Is Human in 1999, patient safety and health care quality have become hot topics in the parlance of modern medical care. The Accreditation Council for Graduate Medical Education now requires integration of these topics into resident education, with evidence of trainee involvement in Patient Safety and Quality Improvement (PSQI) projects. Research in other disciplines indicates that interactive, experiential learning leads to the highest quality PSQI education. Otolaryngology as a field has been slow to adopt these changes into its residency curricula due to competing educational demands and lack of faculty expertise. The author reports preliminary experience with integration of an online module-based curriculum that addresses both of these issues.

Programmatic Assessment of a Comprehensive Quality Improvement Curriculum in an Otolaryngology Residency

To date, there is minimal literature discussing quality improvement curricula in otolaryngology. Herein, we present our program, including didactic, web-based, and experiential learning, developed in the setting of a preexisting institutional quality and safety policy. Nine otolaryngology residents were evaluated with assessments focused on learner satisfaction, learner attitudes, and knowledge acquisition according to the Kirkpatrick framework. Wilcoxon signed-rank test was used to compare results. While the total score increased across all assessments, it was significant for only the Quality Improvement Knowledge Application Tool Revised ( P < .05). We find our initial learning outcomes encouraging and hope that our comprehensive curriculum can serve as a resource to other programs, which can be adapted to fit within the context of variable training environments. Furthermore, it is imperative to consider continuous assessment and refinement of any educational program, using the same quality improvement principles that we endeavor to teach.

The unmet need for a national surgical quality improvement curriculum: a systematic review

INTRODUCTION

The Accreditation Council for Graduate Medical Education Next Accreditation System will require general surgery training programs to demonstrate outstanding clinical outcomes and education in quality improvement (QI). The American College of Surgeons-National Surgical Quality Improvement Project Quality In-Training Initiative reports the results of a systematic review of the literature investigating the availability of a QI curriculum.

METHODS

Using defined search terms, a systematic review was conducted in Embase, PubMed, and Google Scholar (January 2000-March 2013) to identify a surgical QI curriculum. Bibliographies from selected articles and other relevant materials were also hand searched. Curriculum was defined as an organized program of learning complete with content, instruction, and assessment for use in general surgical residency programs. Two independent observers graded surgical articles on quality of curriculum presented.

RESULTS

Overall, 50 of 1155 references had information regarding QI in graduate medical education. Most (n = 24, 48%) described QI education efforts in nonsurgical fields. A total of 31 curricular blueprints were identified; 6 (19.4%) were specific to surgery. Targeted learners were most often post graduate year-2 residents (29.0%); only 6 curricula (19.4%) outlined a course for all residents within their respective programs. Plan, Do, Study, Act (n = 10, 32.3%), and Root Cause Analysis (n = 5, 16.1%) were the most common QI content presented, the majority of instruction was via lecture/didactics (n = 26, 83.9%), and only 7 (22.6%) curricula used validated tool kits for assessment.

CONCLUSION

Elements of QI curriculum for surgical education exist; however, comprehensive content is lacking. The American College of Surgeons-National Surgical Quality Improvement Project Quality In-Training Initiative will build on the high-quality components identified in our review and develop data-centered QI content to generate a comprehensive national QI curriculum for use in graduate surgical education.

Where do we go from here? Moving from systems-based practice process measures to true competency via developmental milestones

For many educators it has been challenging to meet the Accreditation Council for Graduate Medical Education's requirements for teaching systems-based practice (SBP). An additional layer of complexity for educators is evaluating competency in SBP, despite milestones and entrustable professional activities (EPAs). In order to address this challenge, the authors present the results of a literature review for how SBP is currently being taught and a series of recommendations on how to achieve competency in SBP for graduate medical trainees with the use of milestones. The literature review included 29 articles and demonstrated that only 28% of the articles taught more than one of the six core principles of SBP in a meaningful way. Only 7% of the articles received the highest grade of A. The authors summarize four guiding principles for creating a competency-based curriculum that is in alignment with the Next Accreditation System (NAS): 1) the curriculum needs to include all of the core principles in that competency, 2) the objectives of the curriculum should be driven by clinical outcomes, 3) the teaching modalities need to be interactive and clinically relevant, and 4) the evaluation process should be able to measure competency and be directly reflective of pertinent milestones and/or EPAs. This literature review and the provided guiding principles can guide other residency educators in their development of competency-based curricula that meets the standards of the NAS.

Engaging residents and fellows to improve institution-wide quality: the first six years of a novel financial incentive program

PURPOSE

Teaching hospitals strive to engage physicians in quality improvement (QI), and graduate medical education (GME) programs must promote trainee competence in systems-based practice (SBP). The authors developed a QI incentive program that engages residents and fellows, providing them with financial incentives to improve quality while simultaneously gaining SBP experience. In this study, they describe and evaluate success in meeting goals set during the program's first six years.

METHOD

During fiscal years (FYs) 2007-2012, QI project goals for all or specific training programs were set collaboratively with residents and fellows at the University of California, San Francisco (UCSF). Data were collected from administrative databases, via chart abstraction, or through independently designed techniques.

RESULTS

Approximately 5,275 residents and fellows were eligible and participated in the program. A total of 55 projects were completed. Among the 18 all-program projects, goals were achieved for 11 (61%) in three domains: patient satisfaction, quality/safety, and operation/utilization. Among the 37 program-specific projects, goals were achieved for 28 (76%) in four categories: patient-level interventions, enhanced communication, workflow improvements, and effective documentation. Residents and fellows earned an average of $800 in bonuses/FY for achieving these goals.

CONCLUSIONS

Thousands of residents and fellows across disciplines participated in real-life, real-time QI during the program's first six years. Participation provided an experience that may promote SBP competence and resulted in improved quality of care across the UCSF Medical Center. Similar programs may assist teaching hospitals and GME programs in meeting current and future QI and training mandates.

Instituting systems-based practice and practice-based learning and improvement: a curriculum of inquiry

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) requires that training programs integrate system-based practice (SBP) and practice-based learning and improvement (PBLI) into internal medicine residency curricula. CONTEXT AND SETTING: We instituted a seminar series and year-long-mentored curriculum designed to engage internal medicine residents in these competencies.

METHODS

Residents participate in a seminar series that includes assigned reading and structured discussion with faculty who assist in the development of quality improvement or research projects. Residents pursue projects over the remainder of the year. Monthly works in progress meetings, protected time for inquiry, and continued faculty mentorship guide the residents in their project development. Trainees present their work at hospital-wide grand rounds at the end of the academic year. We performed a survey of residents to assess their self-reported knowledge, attitudes and skills in SBP and PBLI. In addition, blinded faculty scored projects for appropriateness, impact, and feasibility.

OUTCOMES

We measured resident self-reported knowledge, attitudes, and skills at the end of the academic year. We found evidence that participants improved their understanding of the context in which they were practicing, and that their ability to engage in quality improvement projects increased. Blinded faculty reviewers favorably ranked the projects' feasibility, impact, and appropriateness. The 'Curriculum of Inquiry' generated 11 quality improvement and research projects during the study period. Barriers to the ongoing work include a limited supply of mentors and delays due to Institutional Review Board approval. Hospital leadership recognizes the importance of the curriculum, and our accreditation manager now cites our ongoing work.

CONCLUSIONS

A structured residency-based curriculum facilitates resident demonstration of SBP and practice-based learning and improvement. Residents gain knowledge and skills though this enterprise and hospitals gain access to trainees who help to solve ongoing problems and meet accreditation requirements.

Assessing systems-based practice

The conceptual definition of systems-based practice (SBP) does not easily translate into directly observable actions or behaviors that can be easily assessed. At the Academic Emergency Medicine consensus conference on education research in emergency medicine (EM), a breakout group presented a review of the literature on existing assessment tools for SBP, discussed the recommendations for research tool development during breakout sessions, and developed a research agenda based on this discussion.

"Learning by doing"--resident perspectives on developing competency in high-quality discharge care

BACKGROUND

Reducing readmissions and post-discharge adverse events by improving the quality of discharge care has become a national priority, yet we have limited understanding about how physicians learn to provide high-quality discharge care.

METHODS

We conducted in-depth, in-person interviews with housestaff physicians with qualitative analysis by a multi-disciplinary team using the constant comparative method to explore learning about high-quality discharge care as a systems-based practice and to identify opportunities to improve training around these concepts.

RESULTS

We analyzed interview transcripts from 29 internal medicine residents: 17 (59 %) were interns (PGY-2 or PGY-3), 12 (41 %) seniors, and 17 (59 %) were female. We identified a recurrent theme of lack of formal training about the discharge process, substantial peer-to-peer instruction, and "learning by doing" on the wards. Within this theme, we identified five specific concepts related to systems-based practice and high-quality discharge care which residents learned during residency: (1) teamwork and the interdisciplinary nature of discharge planning; (2) advanced planning strategies to anticipate challenges in the discharge process; (3) patient safety and the concept of a "safe discharge;" (4) patient continuity of care and learning from post-discharge outcomes and; (5) documentation of discharge plans as a valuable skill.

CONCLUSIONS

Discharge care is an overlooked opportunity to teach concepts of systems-based practice explicitly as learning about discharge care is unstructured and individual experiences may vary considerably. Educational interventions to standardize learning about discharge care may improve the development of systems-based practice during residency and help improve the overall quality of discharge care at teaching hospitals.

Reshaping orthopaedic resident education in systems-based practice

BACKGROUND

Despite advances in understanding the "systems-based practice" competency in resident education, this topic has remained difficult to teach, assess, and document. The goal of this study was to perform a needs assessment and an analysis of the current state of systems-based practice education in orthopaedic residency programs across the U.S. and within our own institution.

METHODS

A sample of orthopaedic educators and residents from across the U.S. who were attending the 2010 American Orthopaedic Association (AOA) Effective Orthopaedic Educator Course, AOA Resident Leadership Forum, and AOA Council of Residency Directors meeting were surveyed to determine (1) which aspects of systems-based practice, if any, were being taught; (2) how systems-based practice is being taught; and (3) how residency programs are assessing systems-based practice. In addition, an in-depth case study of these issues was performed by means of seven semi-structured focus group sessions with diverse stakeholders who participated in the care of musculoskeletal patients at the authors' institution. A quantitative approach was used to analyze the survey data. The focus group data were analyzed with procedures associated with grounded theory, relying on a constant comparative method to develop salient themes arising from the discussion.

RESULTS

"Clinical observation" (33%) and "didactic case-based learning" (23%) were reported by the survey respondents as the most commonly used teaching methods, but specific topics were taught inconsistently. Competency assessment was reported to occur infrequently, and 36% of respondents reported that systems-based practice areas were not being assessed by any methods. The focus group discussions emphasized the need for standardized experiential learning that was closely linked to the patient's perspective. Orthopaedic faculty members were uncomfortable with their knowledge of this competency and their ability to teach and assess it.

CONCLUSIONS

Teaching the systems-based practice competency occurs inconsistently, and formal assessment occurs infrequently. In addition to formal teaching, learning systems-based practice will be best achieved experientially and from the patient's perspective.

A medical resident post-discharge phone call study

BACKGROUND

Several studies have identified the post-discharge period as a time of vulnerability for patients, and an opportunity exists to improve patient care. Adverse drug events are the most common events leading to complications during the post-discharge period. Recent studies have shown that using a scripted medication reconciliation process improves the quality of patient care.

OBJECTIVE

Does a phone call from a medical resident within 72 hours after hospital discharge improve patient satisfaction and quality of care? Does this exercise result in improved attitudes and competence for practice-based learning, and improvement and system-based practice of participating residents?

MATERIALS AND METHODS

This was a prospective randomized study comparing 1 group of patients that received a medication reconciliation phone call from a medical resident within 72 hours after discharge with a control group that did not receive a call. Adult patients aged ≥ 18 years on a medical resident service for ≥ 2 days and being discharged to home were invited to participate. The primary endpoint of the study was patient satisfaction. Secondary endpoints included readmission rates, emergency department visits, follow-up with the primary care provider, and resident attitudes and competence.

RESULTS

The primary and secondary endpoints did not reach statistical significance. However, a medication reconciliation error occurred in 48% of patients, and 93% of residents agreed that the phone call was beneficial to patient care.

CONCLUSION

Although patient satisfaction was not improved from this exercise, a follow-up call to patients after hospital discharge can identify otherwise missed medication reconciliation errors. Medical residents found the phone call to be worthwhile and gained valuable insight into their own discharge practices as demonstrated by self-reflection and intended change in discharge practices.

Quality improvement in medical education: current state and future directions

CONTEXT

During the last decade, there has been a drive to improve the quality of patient care and prevent the occurrence of avoidable errors. This review describes current efforts to teach or engage trainees in patient safety and quality improvement (QI), summarises progress to date, as well as successes and challenges, and lists our recommendations for the next steps that will shape the future of patient safety and QI in medical education.

CURRENT STATUS

Trainees encounter patient safety and QI through three main groups of activity. First are formal curricula that teach concepts or methods intended to facilitate trainees' participation in QI activities. These curricula increase learner knowledge and may improve clinical processes, but demonstrate limited capacity to modify learner behaviours. Second are educational activities that impart specific skills related to safety or quality which are considered to represent core doctor competencies (e.g. effective patient handover). These are frequently taught effectively, but without emphasis on the general safety or quality principles that inform the relevant skills. Third are real-life QI initiatives that involve trainees as active or passive participants. These innovative approaches expose trainees to safety and quality by integrating QI activities into trainees' day-to-day work. However, this integration can be challenging and can sometimes result in tension with broader educational goals.

FUTURE DIRECTIONS

To prepare the next generation of doctors to make meaningful contributions to the quality mission, we propose the following call to action. Firstly, a major effort to build faculty capacity, especially among teachers of QI, should be instigated. Secondly, accreditation standards and assessment methods, both during training and at end-of-training certification examinations, should explicitly target these competencies. Finally, and perhaps most importantly, we must refocus our attention at all levels of training and instil fundamental, collaborative, open-minded behaviours so that future clinicians are primed to promote a culture of safer, higher-quality care.

Hospital quality and patient safety competencies: development, description, and recommendations for use

BACKGROUND

Hospitalists are poised to have a tremendous impact on improving the quality of care for hospitalized patients. However, many hospitalists are inadequately prepared to engage in efforts to improve quality, because medical schools and residency programs have not traditionally emphasized healthcare quality and patient safety in their curricula.

METHODS

Through a multistep process, the Society of Hospital Medicine (SHM) Quality Improvement Education (QIE) subcommittee developed the Hospital Quality and Patient Safety (HQPS) Competencies to provide a framework for developing and assessing curricula and other professional development experiences. This article describes the development, provides definitions, and makes recommendations on the use of the HQPS Competencies.

RESULTS

The 8 areas of competence include: Quality Measurement and Stakeholder Interests, Data Acquisition and Interpretation, Organizational Knowledge and Leadership Skills, Patient Safety Principles, Teamwork and Communication, Quality and Safety Improvement Methods, Health Information Systems, and Patient Centeredness. Reflecting differing levels of hospitalist involvement in healthcare quality, 3 levels of expertise within each area of competence have been established: basic, intermediate, and advanced. Standards for each competency area use carefully selected action verbs to reflect educational goals for hospitalists at each level.

CONCLUSIONS

Formal incorporation of the HQPS Competencies into professional development programs, and innovative educational initiatives and curricula, will help provide current hospitalists and the next generations of hospitalists with the needed skills to be successful.

A multidisciplinary approach for teaching systems-based practice to internal medicine residents

BACKGROUND

Rapid growth in the complexity of the health care environment (including monitoring systems for health care quality and patient safety) may result in graduating internists not being adequately prepared for the demands the system places on them. In response, the Residency Review Committee for Internal Medicine created the Educational Innovations Project (EIP) to encourage select residency training programs to develop new strategies and methods to meet changing demands in graduate medical education.

METHODS

As part of the EIP, our program created an innovative administrative internship. This multiyear curriculum provides systems-based practice training and consists of a series of rotations that take place during the 3 years of internal medicine residency. Each session involves close interaction with the nonphysician personnel who are instrumental in making our institution a functional and cohesive unit. To assess the potential impact of the rotations, we survey senior residents, recent graduates, and faculty educators. In conjunction with the Performance and Patient Experience departments of the hospital, we track several systems-based practice metrics for residents, including compliance with core health care measures, length of stay, and patient satisfaction.

RESULTS

Residents recognize the need to develop systems-based practice skills, to readily participate in structured curricula designed to enhance such skills, and to provide leadership in organizing and publishing quality improvement initiatives, and upon graduation, they may lament that they did not receive even more vigorous training in these areas.

CONCLUSION

Although internal medicine residencies continue to improve their training in systems-based practice, our experience suggests that an even greater emphasis on these skills may be warranted.

Pilot study evaluating a practice-based learning and improvement curriculum focusing on the development of system-level quality improvement skills

BACKGROUND

We developed a practice-based learning and improvement (PBLI) curriculum to address important gaps in components of content and experiential learning activities through didactics and participation in systems-level quality improvement projects that focus on making changes in health care processes.

METHODS

We evaluated the impact of our curriculum on resident PBLI knowledge, self-efficacy, and application skills. A quasi-experimental design assessed the impact of a curriculum (PBLI quality improvement systems compared with non-PBLI) on internal medicine residents' learning during a 4-week ambulatory block. We measured application skills, self-efficacy, and knowledge by using the Systems Quality Improvement Training and Assessment Tool. Exit evaluations assessed time invested and experiences related to the team projects and suggestions for improving the curriculum.

RESULTS

The 2 groups showed differences in change scores. Relative to the comparison group, residents in the PBLI curriculum demonstrated a significant increase in the belief about their ability to implement a continuous quality improvement project (P  =  .020), comfort level in developing data collection plans (P  =  .010), and total knowledge scores (P < .001), after adjusting for prior PBLI experience. Participants in the PBLI curriculum also demonstrated significant improvement in providing a more complete aim statement for a proposed project after adjusting for prior PBLI experience (P  =  .001). Exit evaluations were completed by 96% of PBLI curriculum participants who reported high satisfaction with team performance.

CONCLUSION

Residents in our curriculum showed gains in areas fundamental for PBLI competency. The observed improvements were related to fundamental quality improvement knowledge, with limited gain in application skills. This suggests that while heading in the right direction, we need to conceptualize and structure PBLI training in a way that integrates it throughout the residency program and fosters the application of this knowledge and these skills.

Development and preliminary evaluation of a practice-based learning and improvement tool for assessing resident competence and guiding curriculum development

BACKGROUND

Although practice-based learning and improvement (PBLI) is now recognized as a fundamental and necessary skill set, we are still in need of tools that yield specific information about gaps in knowledge and application to help nurture the development of quality improvement (QI) skills in physicians in a proficient and proactive manner. We developed a questionnaire and coding system as an assessment tool to evaluate and provide feedback regarding PBLI self-efficacy, knowledge, and application skills for residency programs and related professional requirements.

METHODS

Five nationally recognized QI experts/leaders reviewed and completed our questionnaire. Through an iterative process, a coding system based on identifying key variables needed for ideal responses was developed to score project proposals. The coding system comprised 14 variables related to the QI projects, and an additional 30 variables related to the core knowledge concepts related to PBLI. A total of 86 residents completed the questionnaire, and 2 raters coded their open-ended responses. Interrater reliability was assessed by percentage agreement and Cohen κ for individual variables and Lin concordance correlation for total scores for knowledge and application. Discriminative validity (t test to compare known groups) and coefficient of reproducibility as an indicator of construct validity (item difficulty hierarchy) were also assessed.

RESULTS

Interrater reliability estimates were good (percentage of agreements, above 90%; κ, above 0.4 for most variables; concordances for total scores were R  =  .88 for knowledge and R  =  .98 for application).

CONCLUSION

Despite the residents' limited range of experiences in the group with prior PBLI exposure, our tool met our goal of differentiating between the 2 groups in our preliminary analyses. Correcting for chance agreement identified some variables that are potentially problematic. Although additional evaluation is needed, our tool may prove helpful and provide detailed information about trainees' progress and the curriculum.

Measuring faculty reflection on adverse patient events: development and initial validation of a case-based learning system

BACKGROUND

Critical reflection by faculty physicians on adverse patient events is important for changing physician's behaviors. However, there is little research regarding physician reflection on quality improvement (QI).

OBJECTIVE

To develop and validate a computerized case-based learning system (CBLS) to measure faculty physicians' reflections on adverse patient events.

DESIGN

Prospective validation study.

PARTICIPANTS

Staff physicians in the Department of Medicine at Mayo Clinic Rochester.

MAIN MEASURES

The CBLS was developed by Mayo Clinic information technology, medical education, and QI specialists. The reflection questionnaire, adapted from a previously validated instrument, contained eight items structured on five-point scales. Three cases, representing actual adverse events, were developed based on the most common error types: systems, medication, and diagnostic. In 2009, all Mayo Clinic hospital medicine, non-interventional cardiology, and pulmonary faculty were invited to participate. Faculty reviewed each case, determined the next management step, rated case generalizability and relevance, and completed the reflection questionnaire. Factor analysis and internal consistency reliability were calculated. Associations between reflection scores and characteristics of faculty and patient cases were determined.

KEY RESULTS

Forty-four faculty completed 107 case reflections. The CBLS was rated as average to excellent in 95 of 104 (91.3%) completed satisfaction surveys. Factor analysis revealed two levels of reflection: Minimal and High. Internal consistency reliability was very good (overall Cronbach's α=0.77). Item mean scores ranged from 2.89 to 3.73 on a five-point scale. The overall reflection score was 3.41 (standard deviation 0.64). Reflection scores were positively associated with case generalizability (p=0.001), and case relevance (p=0.02).

CONCLUSIONS

The CBLS is a valid method for stratifying faculty physicians' levels of reflection on adverse patient events. Reflection scores are associated with case generalizability and relevance, indicating that reflection improves with pertinent patient encounters. We anticipate that this instrument will be useful in future research on QI among low versus high-reflecting physicians.

Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new "public interest"

In 1910, in his recommendations for reforming medical education, Abraham Flexner responded to what he deemed to be the "public interest." Now, 100 years later, to respond to the current needs of society, the education of physicians must once again change. In addition to understanding the biological basis of health and disease, and mastering technical skills for treating individual patients, physicians will need to learn to navigate in and continually improve complex systems in order to improve the health of the patients and communities they serve. Physicians should not be mere participants in, much less victims of, such systems. Instead, they ought to be prepared to help lead those systems toward ever-higher-quality care for all. A number of innovative programs already exist for students and residents to help integrate improvement skills into professional preparation, and that goal is enjoying increasing support from major professional organizations and accrediting bodies. These experiences have shown that medical schools and residency programs will need to both teach the scientific foundations of system performance and provide opportunities for trainees to participate in team-based improvement of the real-world health systems in which they work. This significant curricular change, to meet the social need of the 21st century, will require educators and learners to embrace new core values, in addition to those held by the profession for generations. These include patient-centeredness, transparency, and stewardship of limited societal resources for health care.

Teaching quality improvement and patient safety to trainees: a systematic review

PURPOSE

To systematically review published quality improvement (QI) and patient safety (PS) curricula for medical students and/or residents to (1) determine educational content and teaching methods, (2) assess learning outcomes achieved, and (3) identify factors promoting or hindering curricular implementation.

METHOD

Data sources included Medline (to January 2009), EMBASE, HealthSTAR, and article bibliographies. Studies selected reported curricula outlining specific educational content and teaching format. For articles with an evaluative component, the authors abstracted methodological features, such as study design. For all articles, they conducted a thematic analysis to identify factors influencing successful implementation of the included curricula.

RESULTS

Of 41 curricula that met the authors' criteria, 14 targeted medical students, 24 targeted residents, and 3 targeted both. Common educational content included continuous QI, root cause analysis, and systems thinking. Among 27 reports that included an evaluation, curricula were generally well accepted. Most curricula demonstrated improved knowledge. Thirteen studies (32%) successfully implemented local changes in care delivery, and seven (17%) significantly improved target processes of care. Factors that affected the successful curricular implementation included having sufficient numbers of faculty familiar with QI and PS content, addressing competing educational demands, and ensuring learners' buy-in and enthusiasm. Participants in some curricula also commented on discrepancies between curricular material and local institutional practice or culture.

CONCLUSIONS

QI and PS curricula that target trainees usually improve learners' knowledge and frequently result in changes in clinical processes. However, successfully implementing such curricula requires attention to a number of learner, faculty, and organizational factors.

Teaching systems-based competency in anesthesiology residency: development of an education and assessment tool

BACKGROUND

The Accreditation Council for Graduate Medical Education requires programs to educate and evaluate residents in 6 competencies, including systems-based practice. We designed a survey and assessment tool to address the competency as it pertains to anesthetic drug costs in an academic center.

METHODS

Residents, certified registered nurse anesthetists, and faculty were asked to complete an anesthetic drug-cost survey without relying on reference materials. After a combination of compulsory in-class didactic sessions and web-based education, the participants were asked to design an anesthetic, give example cases, and determine costs. The initial task was repeated 1 year later.

RESULTS

Our preintervention survey revealed that most practitioners knew very little about anesthetic drug costs, regardless of level of training or degree. All residents completed the mandatory online education tool, more than 80% attended the departmental grand rounds program, and 100% met the goal of designing an anesthetic for all cases within the preset price limit. A repeat of the cost estimate produced an improvement in cost estimates with reduction in variability (P < .05, Student unpaired t test), although estimates of volatile anesthetic and reversal agent costs did not achieve significance at the .05 level for any of the 3 cases.

CONCLUSION

Introducing a formalized teaching and assessment tool has improved our residents' understanding of anesthetic drug costs, and improved our ability to teach and assess the systems-based practice competency.

SUPPORTING PHYSICIANS' PRACTICE-BASED LEARNING AND IMPROVEMENT (PBLI) AND QUALITY IMPROVEMENT THROUGH EXPLORATION OF POPULATION-BASED MEDICAL DATA

Participating in self-assessment activities may stimulate improvement in practice behaviors. However, it is unclear how best to support the development of self-assessment skills, particularly in the health care domain. Exploration of population-based data is one method to enable health care providers to identify deficiencies in overall practice behavior that can motivate quality improvement initiatives. At the University of Virginia, we are developing a decision support tool to integrate and present population-based patient data to health care providers related to both clinical outcomes and non-clinical measures (e.g., demographic information). By enabling users to separate their direct impact on clinical outcomes from other factors out of their control, we may enhance the self-assessment process.

Integrating quality improvement and residency education: insights from the AIAMC National Initiative about the roles of the designated institutional official and program director

Active engagement of both the designated institutional official (DIO) and the program director (PD) is essential to implement any change in graduate medical education (GME). Strategies that are established by the Accreditation Council for Graduate Medical Education or other entities are, in the end, effective only as implemented at the individual program level. The interpretation of national standards or guidelines, and the specific adaptation to the vagaries of individual institutions and programs, can lead to significant variability in implementation and potentially in outcomes. Variability occurs between programs within the same institution and between some specialty programs at different institutions. The National Initiative, sponsored by the Alliance of Independent Academic Medical Centers, was launched in 2007 to demonstrate the effectiveness of GME as a key driver to improve quality, patient safety, and cost-effectiveness of care. This report addresses (1) the key roles of both the DIO and the PD in achieving the goals of the National Initiative, (2) the challenges these goals presented to each role, and (3) some of the tactics drawn from the experiences of the National Initiative in overcoming those challenges. The experience of the National Initiative underscored the synergies of the DIO and PD roles to improve patient care while simultaneously fulfilling their critical responsibilities as institutional and program leaders in GME with even greater effectiveness.

Development of the objective structured system-interaction examination

STUDY OBJECTIVE

The purpose of this study was to develop an objective method of evaluating resident competency in systems-based practice.

STUDY DESIGN

Faculty developed a 12-station examination, the Objective Structured System-Interaction Examination (OSSIE), patterned after the Objective Structured Clinical Examinations (OSCEs), to evaluate residents' ability to effectively work within the complex medical system of care. Scenarios consisted of multiple situations, such as patient hand-offs, consultations, complicated discharges, and family meetings, in which residents interacted with simulated professionals, simulated patients, and simulated family members to demonstrate the systems-based skills. Twelve second-year residents participated in the OSSIE.

FINDINGS

Along with the standardized professionals, a faculty member provided the resident with immediate feedback and completed an evaluation form designed specifically to assess systems-based practice. Residents, faculty, and staff evaluated the OSSIE and felt it provided a rich learning experience and was a beneficial means of formative assessment. The residents' third-year learning experiences were adapted to meet their needs, and suggestions were offered for curriculum revision.

DISCUSSION

The OSSIE is unique in that it uses standardized professionals, involves scenarios in a variety of settings, and incorporates current technology, including an electronic health record and a state-of-the-art simulation laboratory, into the examination. Challenges to implementation include faculty time, scheduling of residents, and availability of resources.

CONCLUSION

By using the OSSIE, faculty are able to assess, provide constructive feedback, and tailor training opportunities to improve resident competence in systems-based practice. Reliability and validity of an instrument developed for use with the OSSIE are currently being determined.