The importance of measuring competency-based outcomes: standard evaluation measures are not surrogates for clinical performance of internal medicine residents

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

BACKGROUND

Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM).

OBJECTIVE

To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods.

DESIGN

The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward.

PARTICIPANTS

NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members.

MAIN MEASURES

The number and description of RSQMs generated during this process.

KEY RESULTS

Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission.

CONCLUSIONS

Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

A Progressive, Collaborative Process to Improve a Curriculum and Define an Assessment Program

OBJECTIVE

To revise the University of Oklahoma College of Pharmacy's professional program outcomes and create an assessment map using results from previous peer review and mapping of all professional courses and curricular streams of knowledge, skills, and attitudes (KSAs).

DESIGN

After consolidating 15 original program outcomes into 11 more precise outcome statements, defining KSAs for each, and getting faculty approval of them, the committee detailed measurable program expectations upon graduation for each outcome and created an assessment map identifying where KSAs were taught, how they were to be assessed, and the expected ability level (novice, competent, proficient) for each across the curriculum.

ASSESSMENT

The committee's work identified deficits, inconsistencies, and disproportionalities in professional program assessment. It recommended assessments to capture student achievement of each outcome, identified performance levels and criteria to measure outcomes progressively in each professional year, and outlined a process to provide students periodic reports on their progress in achieving each outcome.

CONCLUSION

This work establishes a firm foundation for ongoing efforts to measure effectiveness of the professional program, especially in light of Accreditation Council for Pharmacy Education's (ACPE) revised accreditation standards.

Realizing the promise and importance of performance-based assessment

Work-based assessment (WBA) is the assessment of trainees and physicians across the educational continuum of day-to-day competencies and practices in authentic, clinical environments. What distinguishes WBA from other assessment modalities is that it enables the evaluation of performance in context. In this perspective, we describe the growing importance, relevance, and evolution of WBA as it relates to competency-based medical education, supervision, and entrustment. Although a systematic review is beyond the purview of this perspective, we highlight specific methods and needed shifts to WBA that (a) consider patient outcomes, (b) use nonphysician assessors, and (c) assess the care provided to populations of patients. We briefly describe strategies for the effective implementation of WBA and identify outstanding research questions related to its use.

A novel educational strategy targeting health care workers in underserved communities in Central America to integrate HIV into primary medical care

Background

Current educational strategies to integrate HIV care into primary medical care in Central America have traditionally targeted managers or higher-level officials, rather than local health care workers (HCWs). We developed a complementary online and on-site interactive training program to reach local HCWs at the primary care level in underserved communities.

Methods

The training program targeted physicians, nurses, and community HCWs with limited access to traditional onsite training in Panama, Nicaragua, Dominican Republic, and Guatemala. The curriculum focused on principles of HIV care and health systems using a tutor-supported blended educational approach of an 8-week online component, a weeklong on-site problem-solving workshop, and individualized project-based interventions.

Results

Of 258 initially active participants, 225 (225/258 = 87.2%) successfully completed the online component and the top 200 were invited to the on-site workshop. Of those, 170 (170/200 = 85%) attended the on-site workshop. In total, 142 completed all three components, including the project phase. Quantitative and qualitative evaluation instruments included knowledge assessments, reflexive essays, and acceptability surveys. The mean pre and post-essay scores demonstrating understanding of social determinants, health system organization, and integration of HIV services were 70% and 87.5%, respectively, with an increase in knowledge of 17.2% (p<0.001). The mean pre- and post-test scores evaluating clinical knowledge were 70.9% and 90.3%, respectively, with an increase in knowledge of 19.4% (p<0.001). A survey of Likert scale and open-ended questions demonstrated overwhelming participant satisfaction with course content, structure, and effectiveness in improving their HIV-related knowledge and skills.

Conclusion

This innovative curriculum utilized technology to target HCWs with limited access to educational resources. Participants benefited from technical skills acquired through the process, and could continue working within their underserved communities while participating in the online component and then implement interventions that successfully converted theoretical knowledge to action to improve integration of HIV care into primary care.

Achieving the AAAs of Ambulatory Care: Aptitude, Appeal, and Appreciation

BACKGROUND

In the current health care environment more patient care has moved from in-hospital care to the ambulatory primary care settings; however, fewer internal medicine residents are pursuing primary care careers. Barriers to residents developing a sense of competency and enjoyment in ambulatory medicine include the complexity of practice-based systems, patients with multiple chronic diseases, and the limited time that residents spend in the outpatient setting.

OBJECTIVE

In an effort to accelerate residents' ambulatory care competence and enhance their satisfaction with ambulatory practice, we sought to change the learning environment. Interns were provided a series of intensive, focused, ambulatory training sessions prior to beginning their own continuity clinic sessions. The sessions were designed to enable them to work confidently and effectively in their continuity clinic from the beginning of the internship year, and it was hoped this would have a positive impact on their perception of the desirability of ambulatory practice.

METHODS

Improvement needs assessment after a performance, so we developed a structured, competency-based, multidisciplinary curriculum for initiation into ambulatory practice. The curriculum focused on systems-based practice, patient safety, quality improvement, and collaborative work while emphasizing the importance of continuity of care and long-term doctor-patient relationships. Direct observation of patient encounters was done by an attending physician to evaluate communication and physical examination skills. Systems of care commonly used in the clinic were demonstrated. Resources for practice-based learning were used.

CONCLUSION

The immersion of interns in an intensive, hands-on experience using a structured ambulatory care orientation curriculum early in training may prepare the intern to be a successful provider and learner in the primary care ambulatory setting.