Use of an outpatient medical record audit to achieve educational objectives: changes in residents' performances over six years

A Mixed-Methods Program Evaluation of a Self-directed Learning Panel Management Curriculum in an Internal Medicine Residency Clinic

BACKGROUND

Panel management (PM) curricula in internal medicine (IM) residency programs often assign performance measures which may not address the varied interests or needs of resident-learners.

AIM

To evaluate a self-directed learning (SDL)-based PM curriculum.

SETTING

University-based primary care practice in Burlington, Vermont.

PARTICIPANTS

Thirty-five internal medicine residents participated.

PROGRAM DESCRIPTION

Residents completed a PM curriculum that integrated SDL, electronic health record (EHR)-driven performance feedback, mentorship, and autonomy to set learning and patient care goals.

PROGRAM EVALUATION

Pre/post-curricular surveys assessed EHR tool acceptability, weekly curricular surveys and post-curricular focus groups assessed resident perceptions and goals, and an interrupted time series analysis of care gap closure rates was used to compare the pre-intervention and intervention periods. Majority of residents (28-32 or 80-91%) completed the surveys and focus groups. Residents found the EHR tools acceptable and valued protected time, mentorship, and autonomy to set goals. A total of 13,313 patient visits were analyzed. There were no significant differences between rates between the pre-intervention period and the first intervention period (p=0.44).

DISCUSSION

A longitudinal PM curriculum that incorporated SDL and goal setting with EHR-driven performance feedback was well-received by residents, however did not significantly impact the rate of care gap closure.

Continuity Clinic Practice Feedback Curriculum for Residents: A Model for Ambulatory Education

BACKGROUND

There is an unmet need for formal curricula to deliver practice feedback training to residents.

OBJECTIVE

We developed a curriculum to help residents receive and interpret individual practice feedback data and to engage them in quality improvement efforts.

METHODS

We created a framework based on resident attribution, effective metric selection, faculty coaching, peer and site comparisons, and resident-driven goals. The curriculum used electronic health record-generated resident-level data and disease-specific ambulatory didactics to help motivate quality improvement efforts. It was rolled out to 144 internal medicine residents practicing at 1 of 4 primary care clinic sites from July 2016 to June 2017. Resident attitudes and behaviors were tracked with presurveys and postsurveys, completed by 126 (88%) and 85 (59%) residents, respectively. Data log-ins and completion of educational activities were monitored. Group-level performance data were tracked using run charts.

RESULTS

Survey results demonstrated significant improvements on a 5-point Likert scale in residents' self-reported ability to receive (from a mean of 2.0 to 3.3, P < .001) and to interpret and understand (mean of 2.4 to 3.2, P < .001) their practice performance data. There was also an increased likelihood they would report that their practice had seen improvements in patient care (13% versus 35%, P < .001). Run charts demonstrated no change in patient outcome metrics.

CONCLUSIONS

A learner-centered longitudinal curriculum on ambulatory patient panels can help residents develop competency in receiving, interpreting, and effectively applying individualized practice performance data.

Interventions to increase influenza vaccination rates of those 60 years and older in the community

BACKGROUND

The effectiveness of interventions to increase influenza vaccination uptake in people aged 60 years and older varies by country and participant characteristics. This review updates versions published in 2010 and 2014.

OBJECTIVES

To assess access, provider, system, and societal interventions to increase the uptake of influenza vaccination in people aged 60 years and older in the community.

SEARCH METHODS

We searched CENTRAL, which includes the Cochrane Acute Respiratory Infections Group's Specialised Register, MEDLINE, Embase, CINAHL, and ERIC for this update, as well as WHO ICTRP and ClinicalTrials.gov for ongoing studies to 7 December 2017. We also searched the reference lists of included studies.

SELECTION CRITERIA

Randomised controlled trials (RCTs) and cluster-randomised trials of interventions to increase influenza vaccination in people aged 60 years or older in the community.

DATA COLLECTION AND ANALYSIS

We used standard methodological procedures as specified by Cochrane.

MAIN RESULTS

We included three new RCTs for this update (total 61 RCTs; 1,055,337 participants). Trials involved people aged 60 years and older living in the community in high-income countries. Heterogeneity limited some meta-analyses. We assessed studies as at low risk of bias for randomisation (38%), allocation concealment (11%), blinding (44%), and selective reporting (100%). Half (51%) had missing data. We assessed the evidence as low-quality. We identified three levels of intervention intensity: low (e.g. postcards), medium (e.g. personalised phone calls), and high (e.g. home visits, facilitators).Increasing community demand (12 strategies, 41 trials, 53 study arms, 767,460 participants)One successful intervention that could be meta-analysed was client reminders or recalls by letter plus leaflet or postcard compared to reminder (odds ratio (OR) 1.11, 95% confidence interval (CI) 1.07 to 1.15; 3 studies; 64,200 participants). Successful interventions tested by single studies were patient outreach by retired teachers (OR 3.33, 95% CI 1.79 to 6.22); invitations by clinic receptionists (OR 2.72, 95% CI 1.55 to 4.76); nurses or pharmacists educating and nurses vaccinating patients (OR 152.95, 95% CI 9.39 to 2490.67); medical students counselling patients (OR 1.62, 95% CI 1.11 to 2.35); and multiple recall questionnaires (OR 1.13, 95% CI 1.03 to 1.24).Some interventions could not be meta-analysed due to significant heterogeneity: 17 studies tested simple reminders (11 with 95% CI entirely above unity); 16 tested personalised reminders (12 with 95% CI entirely above unity); two investigated customised compared to form letters (both 95% CI above unity); and four studies examined the impact of health risk appraisals (all had 95% CI above unity). One study of a lottery for free groceries was not effective.Enhancing vaccination access (6 strategies, 8 trials, 10 arms, 9353 participants)We meta-analysed results from two studies of home visits (OR 1.30, 95% CI 1.05 to 1.61) and two studies that tested free vaccine compared to patient payment for vaccine (OR 2.36, 95% CI 1.98 to 2.82). We were unable to conduct meta-analyses of two studies of home visits by nurses plus a physician care plan (both with 95% CI above unity) and two studies of free vaccine compared to no intervention (both with 95% CI above unity). One study of group visits (OR 27.2, 95% CI 1.60 to 463.3) was effective, and one study of home visits compared to safety interventions was not.Provider- or system-based interventions (11 strategies, 15 trials, 17 arms, 278,524 participants)One successful intervention that could be meta-analysed focused on payments to physicians (OR 2.22, 95% CI 1.77 to 2.77). Successful interventions tested by individual studies were: reminding physicians to vaccinate all patients (OR 2.47, 95% CI 1.53 to 3.99); posters in clinics presenting vaccination rates and encouraging competition between doctors (OR 2.03, 95% CI 1.86 to 2.22); and chart reviews and benchmarking to the rates achieved by the top 10% of physicians (OR 3.43, 95% CI 2.37 to 4.97).We were unable to meta-analyse four studies that looked at physician reminders (three studies with 95% CI above unity) and three studies of facilitator encouragement of vaccination (two studies with 95% CI above unity). Interventions that were not effective were: comparing letters on discharge from hospital to letters to general practitioners; posters plus postcards versus posters alone; educational reminders, academic detailing, and peer comparisons compared to mailed educational materials; educational outreach plus feedback to teams versus written feedback; and an intervention to increase staff vaccination rates.Interventions at the societal levelNo studies reported on societal-level interventions.Study funding sourcesStudies were funded by government health organisations (n = 33), foundations (n = 9), organisations that provided healthcare services in the studies (n = 3), and a pharmaceutical company offering free vaccines (n = 1). Fifteen studies did not report study funding sources.

AUTHORS' CONCLUSIONS

We identified interventions that demonstrated significant positive effects of low (postcards), medium (personalised phone calls), and high (home visits, facilitators) intensity that increase community demand for vaccination, enhance access, and improve provider/system response. The overall GRADE assessment of the evidence was moderate quality. Conclusions are unchanged from the 2014 review.

Medical record keeping and system performance in orthopaedic trauma patients

Objective The medical record is critical for documentation and communication between healthcare professionals. The aim of the present study was to evaluate important aspects of the orthopaedic medical record and system performance to determine whether any deficiencies exist in these areas. Methods Review of 200 medical records of surgically treated traumatic lower limb injury patients was undertaken. The operative report, discharge summary and first and second outpatient reviews were evaluated. Results In all cases, an operative report was completed by a senior surgeon. Weight-bearing status was adequately documented in 91% of reports. Discharge summaries were completed for 82.5% of admissions, with 87.3% of these having instructions reflective of those in the operative report. Of first and second outpatient reviews, 69% and 73%, respectively, occurred within 1 week of the requested time. Previously documented management plans were changed in 30% of reviews. At 6-months post-operatively, 42% of patients had been reviewed by a member of their operating team. Discussion Orthopaedic medical record documentation remains an area for improvement. In addition, hospital out-patient systems perform suboptimally and may affect patient outcomes. What is known about the topic? Medical records are an essential tool in modern medical practice. Despite the importance of comprehensive documentation in the medical record, numerous examples of poor documentation have been demonstrated, including substandard documentation during consultant ward rounds by junior doctors leading to a breakdown in healthcare professional communication and potential patient mismanagement. Further inadequacies of medical record documentation have been demonstrated in surgical discharge notes, with complete and correct documentation reported to be as low as 65%. What does this paper add? Standards of patient care should be constantly monitored and deficiencies identified in order to implement a remedy and close the quality loop. The present study has highlighted that the standard of orthopaedic trauma medical record keeping at an Australian Level 1 trauma centre is below what is expected and several key areas of documentation require improvement. This paper further evaluates the system performance of the out-patient system, an area where, to the authors knowledge, there is no previous work published. The findings show that the performance was below what is expected for surgical review, with many patients failing to be reviewed by their operating surgeon. What are the implications for practitioners? The present study shows that there is a poor level of documentation and a standard of out-patient review below what is expected. The implications of these findings will be to highlight current deficiencies to practitioners and promote change in current practice to improve the quality of medical record documentation among medical staff. Further, the findings of poor system performance will promote change in the current system of delivering out-patient care to patients.

Interventions to increase influenza vaccination rates of those 60 years and older in the community

BACKGROUND

The effectiveness of interventions to increase the uptake of influenza vaccination in people aged 60 and older is uncertain.

OBJECTIVES

To assess access, provider, system and societal interventions to increase the uptake of influenza vaccination in people aged 60 years and older in the community.

SEARCH METHODS

We searched CENTRAL (2014, Issue 5), MEDLINE (January 1950 to May week 3 2014), EMBASE (1980 to June 2014), AgeLine (1978 to 4 June 2014), ERIC (1965 to June 2014) and CINAHL (1982 to June 2014).

SELECTION CRITERIA

Randomised controlled trials (RCTs) of interventions to increase influenza vaccination uptake in people aged 60 and older.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed study quality and extracted influenza vaccine uptake data.

MAIN RESULTS

This update identified 13 new RCTs; the review now includes a total of 57 RCTs with 896,531 participants. The trials included community-dwelling seniors in high-income countries. Heterogeneity limited meta-analysis. The percentage of trials with low risk of bias for each domain was as follows: randomisation (33%); allocation concealment (11%); blinding (44%); missing data (49%) and selective reporting (100%). Increasing community demand (32 trials, 10 strategies)The interventions with a statistically significant result were: three trials (n = 64,200) of letter plus leaflet/postcard compared to letter (odds ratio (OR) 1.11, 95% confidence interval (CI) 1.07 to 1.15); two trials (n = 614) of nurses/pharmacists educating plus vaccinating patients (OR 3.29, 95% CI 1.91 to 5.66); single trials of a phone call from a senior (n = 193) (OR 3.33, 95% CI 1.79 to 6.22), a telephone invitation versus clinic drop-in (n = 243) (OR 2.72, 95% CI 1.55 to 4.76), a free groceries lottery (n = 291) (OR 1.04, 95% CI 0.62 to 1.76) and nurses educating and vaccinating patients (n = 485) (OR 152.95, 95% CI 9.39 to 2490.67).We did not pool the following trials due to considerable heterogeneity: postcard/letter/pamphlets (16 trials, n = 592,165); tailored communications (16 trials, n = 388,164); customised letter/phone-call (four trials, n = 82,465) and client-based appraisals (three trials, n = 4016), although several trials showed the interventions were effective. Enhancing vaccination access (10 trials, six strategies)The interventions with a statistically significant result were: two trials (n = 2112) of home visits compared to clinic invitation (OR 1.30, 95% CI 1.05 to 1.61); two trials (n = 2251) of free vaccine (OR 2.36, 95% CI 1.98 to 2.82) and one trial (n = 321) of patient group visits (OR 24.85, 95% CI 1.45 to 425.32). One trial (n = 350) of a home visit plus vaccine encouragement compared to a home visit plus safety advice was non-significant.We did not pool the following trials due to considerable heterogeneity: nurse home visits (two trials, n = 2069) and free vaccine compared to no intervention (two trials, n = 2250). Provider- or system-based interventions (17 trials, 11 strategies)The interventions with a statistically significant result were: two trials (n = 2815) of paying physicians (OR 2.22, 95% CI 1.77 to 2.77); one trial (n = 316) of reminding physicians about all their patients (OR 2.47, 95% CI 1.53 to 3.99); one trial (n = 8376) of posters plus postcards (OR 2.03, 95% CI 1.86 to 2.22); one trial (n = 1360) of chart review/feedback (OR 3.43, 95% CI 2.37 to 4.97) and one trial (n = 27,580) of educational outreach/feedback (OR 0.77, 95% CI 0.72 to 0.81).Trials of posters plus postcards versus posters (n = 5753), academic detailing (n = 1400) and increasing staff vaccination rates (n = 26,432) were non-significant.We did not pool the following trials due to considerable heterogeneity: reminding physicians (four trials, n = 202,264) and practice facilitators (three trials, n = 2183), although several trials showed the interventions were effective. Interventions at the societal level We identified no RCTs of interventions at the societal level.

AUTHORS' CONCLUSIONS

There are interventions that are effective for increasing community demand for vaccination, enhancing access and improving provider/system response. Heterogeneity limited pooling of trials.

Effectiveness of interventions to increase screening for breast, cervical, and colorectal cancers: nine updated systematic reviews for the guide to community preventive services

CONTEXT

Screening reduces mortality from breast, cervical, and colorectal cancers. The Guide to Community Preventive Services previously conducted systematic reviews on the effectiveness of 11 interventions to increase screening for these cancers. This article presents results of updated systematic reviews for nine of these interventions.

EVIDENCE ACQUISITION

Five databases were searched for studies published during January 2004-October 2008. Studies had to (1) be a primary investigation of one or more intervention category; (2) be conducted in a country with a high-income economy; (3) provide information on at least one cancer screening outcome of interest; and (4) include screening use prior to intervention implementation or a concurrent group unexposed to the intervention category of interest. Forty-five studies were included in the reviews.

EVIDENCE SYNTHESIS

Recommendations were added for one-on-one education to increase screening with fecal occult blood testing (FOBT) and group education to increase mammography screening. Strength of evidence for client reminder interventions to increase FOBT screening was upgraded from sufficient to strong. Previous findings and recommendations for reducing out-of-pocket costs (breast cancer screening); provider assessment and feedback (breast, cervical, and FOBT screening); one-on-one education and client reminders (breast and cervical cancer screening); and reducing structural barriers (breast cancer and FOBT screening) were reaffirmed or unchanged. Evidence remains insufficient to determine effectiveness for the remaining screening tests and intervention categories.

CONCLUSIONS

Findings indicate new and reaffirmed interventions effective in promoting recommended cancer screening, including colorectal cancer screening. Findings can be used in community and healthcare settings to promote recommended care. Important research gaps also are described.

Factors influencing colorectal cancer screening participation

Colorectal cancer (CRC) is a major health problem worldwide. Although population-based CRC screening is strongly recommended in average-risk population, compliance rates are still far from the desirable rates. High levels of screening uptake are necessary for the success of any screening program. Therefore, the investigation of factors influencing participation is crucial prior to design and launches a population-based organized screening campaign. Several studies have identified screening behaviour factors related to potential participants, providers, or health care system. These influencing factors can also be classified in non-modifiable (i.e., demographic factors, education, health insurance, or income) and modifiable factors (i.e., knowledge about CRC and screening, patient and provider attitudes or structural barriers for screening). Modifiable determinants are of great interest as they are plausible targets for interventions. Interventions at different levels (patient, providers or health care system) have been tested across the studies with different results. This paper analyzes factors related to CRC screening behaviour and potential interventions designed to improve screening uptake.

Academic detailing to increase colorectal cancer screening by primary care practices in Appalachian Pennsylvania

BACKGROUND

In the United States, colorectal cancer (CRC) is the third most frequently diagnosed cancer and second leading cause of cancer death. Screening is a primary method to prevent CRC, yet screening remains low in the U.S. and particularly in Appalachian Pennsylvania, a largely rural area with high rates of poverty, limited health care access, and increased CRC incidence and mortality rates. Receiving a physician recommendation for CRC screening is a primary predictor for patient adherence with screening guidelines. One strategy to disseminate practice-oriented interventions is academic detailing (AD), a method that transfers knowledge or methods to physicians, nurses or office staff through the visit(s) of a trained educator. The objective of this study was to determine acceptability and feasibility of AD among primary care practices in rural Appalachian Pennsylvania to increase CRC screening.

METHODS

A multi-site, practice-based, intervention study with pre- and 6-month post-intervention review of randomly selected medical records, pre- and post-intervention surveys, as well as a post-intervention key informant interview was conducted. The primary outcome was the proportion of patients current with CRC screening recommendations and having received a CRC screening within the past year. Four practices received three separate AD visits to review four different learning modules.

RESULTS

We reviewed 323 records pre-intervention and 301 post-intervention. The prevalence of being current with screening recommendation was 56% in the pre-intervention, and 60% in the post-intervention (p=0.29), while the prevalence of having been screened in the past year increased from 17% to 35% (p<0.001). Colonoscopies were the most frequently performed screening test. Provider knowledge was improved and AD was reported to be an acceptable intervention for CRC performance improvement by the practices.

CONCLUSIONS

AD appears to be acceptable and feasible for primary care providers in rural Appalachia. A ceiling effect for CRC screening may have been a factor in no change in overall screening rates. While the study was not designed to test the efficacy of AD on CRC screening rates, our evidence suggests that AD is acceptable and may be efficacious in increasing recent CRC screening rates in Appalachian practices which could be tested through a randomized controlled study.

Interventions to increase influenza vaccination rates of those 60 years and older in the community

BACKGROUND

Although the evidence to support influenza vaccination is poor, it is promoted by many health authorities. There is uncertainty about the effectiveness of interventions to increase influenza vaccination rates in those 60 years or older.

OBJECTIVES

To assess effects of interventions to increase influenza vaccination rates in those 60 or older.

SEARCH STRATEGY

We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (The Cochrane Library, 2010, issue 3), containing the Cochrane Acute Respiratory Infections Group's Specialized Register, MEDLINE (January 1950 to July 2010), PubMed (January 1950 to July 2010), EMBASE (1980 to 2010 Week 28), AgeLine (1978 to July 2010), ERIC (1965 to July 2010) and CINAHL (1982 to July 2010).

SELECTION CRITERIA

Randomized controlled trials (RCTs) to increase influenza vaccination rates in those aged 60 years and older, recording influenza vaccination status either through clinic records, billing data or local/national vaccination registers.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed study quality and extracted data.

MAIN RESULTS

Forty-four RCTs were included. All included RCTs studied seniors in the community and in high-income countries. No RCTs of society-level interventions were included. Heterogeneity was marked and meta-analysis was limited. Only five RCTs were graded at low and six at moderate risk of bias. They included three of 13 personalized postcard interventions (all three with the 95% confidence interval (CI) above unity), two of the four home visit interventions (both with 95% CI above unity, but one a small study), three of the four reminder to physicians interventions (none with 95% CI above unity) and three of the four facilitator interventions (one with 95% CI above unity, and one P < 0.01). The other 33 RCTs were at high risk of bias and no recommendations for practice can be drawn.

AUTHORS' CONCLUSIONS

Personalized postcards or phone calls are effective, and home visits, and facilitators, may be effective. Reminders to physicians are not. There is insufficient good evidence for other interventions.

Organizational factors and the cancer screening process

Cancer screening is a process of care consisting of several steps and interfaces. This article reviews what is known about the association between organizational factors and cancer screening rates and examines how organizational strategies can address the steps and interfaces of cancer screening in the context of both intraorganizational and interorganizational processes. We reviewed 79 studies assessing the relationship between organizational factors and cancer screening. Screening rates are largely driven by strategies to 1) limit the number of interfaces across organizational boundaries; 2) recruit patients, promote referrals, and facilitate appointment scheduling; and 3) promote continuous patient care. Optimal screening rates can be achieved when health-care organizations tailor strategies to the steps and interfaces in the cancer screening process that are most critical for their organizations, the providers who work within them, and the patients they serve.

The importance of measuring competency-based outcomes: standard evaluation measures are not surrogates for clinical performance of internal medicine residents

BACKGROUND

Despite recent emphasis on educational outcomes, program directors still rely on standard evaluation techniques such as tests of knowledge and subjective ratings.

PURPOSES

To assess the correlation of standard internal medicine (IM) residency evaluation scores (attending global evaluations, In-Training examination, and Mini-Clinical Examination Exercise) with documented performance of preventive measures for continuity clinic patients.

METHODS

Cross-sectional study of 132 IM residents attending an IM teaching clinic, July 2000 to June 2003, comparing standard evaluations with chart audit.

RESULTS

Mean resident performance ranged from 53% (SD = 24) through 89% (SD = 20) across the 6 preventive measures abstracted from 1,102 patient charts. We found weak and mostly not significant correlations between standard measures and performance of preventive services.

CONCLUSIONS

Standard measures are not adequate surrogates for measuring clinical outcomes. This supports the Accreditation Council for Graduate Medical Education's recommendations to incorporate novel Toolbox measures, like chart audit, into residency evaluations.

Resident self-assessment and self-reflection: University of Wisconsin-Madison's Five-Year Study

BACKGROUND

Chart review represents a critical cornerstone for practice-based learning and improvement in our internal medicine residency program.

OBJECTIVE

To document residents' performance monitoring and improvement skills in their continuity clinics, their satisfaction with practice-based learning and improvement, and their ability to self-reflect on their performance.

DESIGN

Retrospective longitudinal design with repeated measures.

PARTICIPANTS

Eighty Internal Medicine residents abstracted data for 3 consecutive years from the medical records of their 4,390 patients in the University of Wisconsin-Madison (UW) Hospital and Clinics and William S. Middleton Veterans Administration (VA) outpatient clinics.

MEASUREMENT

Logistic modeling was used to determine the effect of postgraduate year, resident sex, graduation cohort, and clinic setting on residents' "compliance rate" on 17 nationally recognized health screening and chronic disease management parameters from 2003 to 2007.

RESULTS

Residents' adherence to national preventive and chronic disease standards increased significantly from intern to subsequent years for administering immunizations, screening for diabetes, cholesterol, cancer, and behavioral risks, and for management of diabetes. Of the residents, 92% found the chart review exercise beneficial, with 63% reporting gains in understanding about their medical practices, 26% reflecting on specific gaps in their practices, and 8% taking critical action to improve their patient outcomes.

CONCLUSIONS

This paper provides support for the feasibility and practicality of this limited-cost method of chart review. It also directs our residency program's attention in the continuity clinic to a key area important to internal medicine training programs by highlighting the potential benefit of enhancing residents' self-reflection skills.

Interventions to increase recommendation and delivery of screening for breast, cervical, and colorectal cancers by healthcare providers systematic reviews of provider assessment and feedback and provider incentives

Most major medical organizations recommend routine screening for breast, cervical, and colorectal cancers. Screening can lead to early detection of these cancers, resulting in reduced mortality. Yet not all people who should be screened are screened, either regularly or, in some cases, ever. This report presents results of systematic reviews of effectiveness, applicability, economic efficiency, barriers to implementation, and other harms or benefits of two provider-directed intervention approaches to increase screening for breast, cervical, and colorectal cancers. These approaches, provider assessment and feedback, and provider incentives encourage providers to deliver screening services at appropriate intervals. Evidence in these reviews indicates that provider assessment and feedback interventions can effectively increase screening by mammography, Pap test, and fecal occult blood test. Health plans, healthcare systems, and cancer control coalitions should consider such evidence-based findings when implementing interventions to increase screening use. Evidence was insufficient to determine the effectiveness of provider incentives in increasing use of any of these tests. Specific areas for further research are suggested in this report, including the need for additional research to determine whether provider incentives are effective in increasing use of any of these screening tests, and whether assessment and feedback interventions are effective in increasing other tests for colorectal cancer (i.e., flexible sigmoidoscopy, colonoscopy, or double-contrast barium enema).

Use of a registry-generated audit, feedback, and patient reminder intervention in an internal medicine resident clinic--a randomized trial

BACKGROUND

Disease registries, audit and feedback, and clinical reminders have been reported to improve care processes.

OBJECTIVE

To assess the effects of a registry-generated audit, feedback, and patient reminder intervention on diabetes care.

DESIGN

Randomized controlled trial conducted in a resident continuity clinic during the 2003-2004 academic year.

PARTICIPANTS

Seventy-eight categorical Internal Medicine residents caring for 483 diabetic patients participated. Residents randomized to the intervention (n = 39) received instruction on diabetes registry use; quarterly performance audit, feedback, and written reports identifying patients needing care; and had letters sent quarterly to patients needing hemoglobin A1c or cholesterol testing. Residents randomized to the control group (n = 39) received usual clinic education.

MEASUREMENTS

Hemoglobin A1c and lipid monitoring, and the achievement of intermediate clinical outcomes (hemoglobin A1c <7.0%, LDL cholesterol <100 mg/dL, and blood pressure <130/85 mmHg) were assessed.

RESULTS

Patients cared for by residents in the intervention group had higher adherence to guideline recommendations for hemoglobin A1c testing (61.5% vs 48.1%, p = .01) and LDL testing (75.8% vs 64.1%, p = .02). Intermediate clinical outcomes were not different between groups.

CONCLUSIONS

Use of a registry-generated audit, feedback, and patient reminder intervention in a resident continuity clinic modestly improved diabetes care processes, but did not influence intermediate clinical outcomes.

Peer chart audits: a tool to meet Accreditation Council on Graduate Medical Education (ACGME) competency in practice-based learning and improvement

BACKGROUND

The Accreditation Council on Graduate Medical Education (ACGME) supports chart audit as a method to track competency in Practice-Based Learning and Improvement. We examined whether peer chart audits performed by internal medicine residents were associated with improved documentation of foot care in patients with diabetes mellitus.

METHODS

A retrospective electronic chart review was performed on 347 patients with diabetes mellitus cared for by internal medicine residents in a university-based continuity clinic from May 2003 to September 2004. Residents abstracted information pertaining to documentation of foot examinations (neurological, vascular, and skin) from the charts of patients followed by their physician peers. No formal feedback or education was provided.

RESULTS

Significant improvement in the documentation of foot exams was observed over the course of the study. The percentage of patients receiving neurological, vascular, and skin exams increased by 20% (from 13% to 33%) (p = 0.001), 26% (from 45% to 71%) (p < 0.001), and 18% (51%-72%) (p = 0.005), respectively. Similarly, the proportion of patients receiving a well-documented exam which includes all three components - neurological, vascular and skin foot exam - increased over time (6% to 24%, p < 0.001).

CONCLUSION

Peer chart audits performed by residents in the absence of formal feedback were associated with improved documentation of the foot exam in patients with diabetes mellitus. Although this study suggests that peer chart audits may be an effective tool to improve practice-based learning and documentation of foot care in diabetic patients, evaluating the actual performance of clinical care was beyond the scope of this study and would be better addressed by a randomized controlled trial.

Implementing achievable benchmarks in preventive health: a controlled trial in residency education

PURPOSE

To evaluate the Preventive Health Achievable Benchmarks Curriculum, a multifaceted improvement intervention that included an objective, practice-based performance evaluation of internal medicine and pediatric residents' delivery of preventive services.

METHOD

The authors conducted a nonrandomized experiment of intervention versus control group residents with baseline and follow-up of performance audited for 2001-2004. All 130 internal medicine and 78 pediatric residents at two continuity clinics at the University of Alabama School of Medicine, Birmingham, participated. Performance of preventive care was assessed by structured chart review. The multifaceted feedback curriculum included individualized performance feedback, academic detailing by faculty, and collective didactic sessions. The main outcome was difference in receipt of preventive care for patients seen by intervention and control residents, comparing baseline and follow-up.

RESULTS

Charts were reviewed for 3,958 patients. Receipt of preventive care increased for patients of intervention residents, but not for patients of control residents. For the intervention group, significant increases occurred for five of six indicators in internal medicine: smoking screening, quit smoking advice, colon cancer screening, pneumonia vaccine, and lipid screening; and four of six in pediatrics: parental quit smoking advice, car seats, car restraints, and eye alignment (p < .05 for all). For control residents, no consistent improvements were seen. There was greater improvement for intervention than for control residents for four of six indicators in internal medicine, and two of six in pediatrics.

CONCLUSIONS

Using a multifaceted feedback curriculum, the authors taught residents about the care they provide and improved documented patient care.

Individualized performance feedback to increase prenatal domestic violence screening

OBJECTIVES

Universal domestic violence (DV) screening once per trimester of pregnancy is recommended but rarely accomplished. Clinical leaders in this setting sought to improve adherence with this protocol. This prospective study used medical record audit and individualized performance feedback with peer comparison (IPF) to improve DV screening among first and second year obstetrics and gynecology (ob/gyn) residents.

METHODS

The setting is a northeastern, urban, hospital-based, prenatal clinic serving low-income women. Most patients are Latina (75%); 11% are black and 9% are white. Few begin care in the first trimester (8.5%). We gave all residents DV training. Next we gave IPF-four reports at seven-week intervals. We reviewed medical record notes on patient visits corresponding to the first medical encounter and week 16 and week 28 of pregnancy. We used this data to compare screening immediately before IPF and following each IPF report.

RESULTS

Screening increased steadily over time, from 60% of appropriate visits before IPF to 91% after the fourth report (Chi Square 28.4, p<.001). Adjusting for key factors, the odds of screening after the last IPF report were seven and a half times greater than the odds of screening before IPF (Odds Ratio: 7.6; 95% Confidence Interval: 3.0, 19.0).

CONCLUSIONS

IPF was associated with increased DV screening among first and second year ob/gyn residents in this setting. Increased screening improved compliance with the clinic protocol and increased opportunities for patient disclosure, education, and treatment, critical public health objectives.

Measuring resident physicians' performance of preventive care. Comparing chart review with patient survey

BACKGROUND

The Accreditation Council for Graduate Medical Education has suggested various methods for evaluation of practice-based learning and improvement competency, but data on implementation of these methods are limited.

OBJECTIVE

To compare medical record review and patient surveys on evaluating physician performance in preventive services in an outpatient resident clinic.

DESIGN

Within an ongoing quality improvement project, we collected baseline performance data on preventive services provided for patients at the University of Alabama at Birmingham (UAB) Internal Medicine Residents' ambulatory clinic.

PARTICIPANTS

Seventy internal medicine and medicine-pediatrics residents from the UAB Internal Medicine Residency program.

MEASUREMENTS

Resident- and clinic-level comparisons of aggregated patient survey and chart documentation rates of (1) screening for smoking status, (2) advising smokers to quit, (3) cholesterol screening, (4) mammography screening, and (5) pneumonia vaccination.

RESULTS

Six hundred and fifty-nine patient surveys and 761 charts were abstracted. At the clinic level, rates for screening of smoking status, recommending mammogram, and for cholesterol screening were similar (difference <5%) between the 2 methods. Higher rates for pneumonia vaccination (76% vs 67%) and advice to quit smoking (66% vs 52%) were seen on medical record review versus patient surveys. However, within-resident (N=70) comparison of 2 methods of estimating screening rates contained significant variability. The cost of medical record review was substantially higher ($107 vs $17/physician).

CONCLUSIONS

Medical record review and patient surveys provided similar rates for selected preventive health measures at the clinic level, with the exception of pneumonia vaccination and advising to quit smoking. A large variation among individual resident providers was noted.

Differences in preventive health quality by residency year. Is seniority better?

BACKGROUND

It is assumed that the performance of more senior residents is superior to that of interns, but this has not been assessed objectively.

OBJECTIVE

To determine whether adherence to national guidelines for outpatient preventive health services differs by year of residency training.

DESIGN

Cross-sectional study.

PARTICIPANTS

One hundred twenty Internal Medicine residents, postgraduate year (PGY)- 1 and PGY -2, attending a University Internal Medicine teaching clinic between June 2000 and May 2003.

MEASUREMENTS

We studied 6 preventive health care services offered or received by patients by abstracting data from 1,017 patient records. We examined the differences in performance between PGY-1 and PGY-2 residents.

RESULTS

Postgraduate year-2 residents did not statistically outperform PGY-1 residents on any measure. The overall proportion of patients receiving appropriate preventive health services for pneumococcal vaccination, advising tobacco cessation, breast and colon cancer screening, and lipid screening was similar across levels of training. PGY-1s outperformed PGY-2s for tobacco use screening (58%, 51%, P = .03). These results were consistent after accounting for clustering of patients within provider and adjusting for patient age, gender, race and insurance, resident gender, and number of visits during the measurement year.

CONCLUSIONS

Overall, patients cared for by PGY-2 residents did not receive more outpatient preventive health services than those cared for by PGY-1 residents. Efforts should be made to ensure quality patient care in the outpatient setting for all levels of training.

Objective Assessment of Skills in Intraocular Surgery (OASIS)

OBJECTIVE

To establish an objective ophthalmic surgical evaluation protocol to assess residents' surgical competency and improve residents' surgical outcomes.

PARTICIPANTS

Eight experts in resident education from comprehensive ophthalmology, cornea, glaucoma, and retina services; 2 chief residents (postgraduate year 5 [PGY5]); and resident representatives from PGYs 2, 3, and 4 participated in the development of an objective assessment tool of skills in resident cataract surgery.

METHODS

Analysis of all resident cataract surgeries performed at our service from July 2001 to July 2003 led to the development of a 1-page objective evaluation form to assess residents' skills in cataract surgery. A panel of surgeons at the Massachusetts Eye and Ear Infirmary reviewed the database and the evaluation form and provided constructive feedback.

RESULTS

Development of a unique database of all resident cataract cases and constructive feedback by experts in resident teaching assisted in creating a 1-page evaluation form entitled Objective Assessment of Skills in Intraocular Surgery (OASIS).

CONCLUSIONS

OASIS has face and content validity and can be used to assess, objectively, surgical events and surgical skill. We believe the OASIS evaluation form and database will be a valuable tool for assessing ophthalmology residents' surgical skills at other residency programs as well.

Assessment of Ophthalmology Resident On-Call Performance

PURPOSE

To design and implement a valid tool for assessment of ophthalmology resident on-call performance.

DESIGN

Retrospective chart audit.

SETTING

Tertiary care academic ophthalmology programs.

PARTICIPANTS

Ophthalmology faculty and residents at the University of Cincinnati and the University of Iowa.

METHODS

A 1-page on-call assessment tool (OCAT) and scoring rubric were developed to evaluate ophthalmology resident on-call performance. A retrospective chart audit of consecutive resident on-call charts was performed at the University of Cincinnati and the University of Iowa, and resident performance was scored using the OCAT.

RESULTS

A consensus of faculty comments established the face and content validity of the OCAT. One hundred ninety-one on-call consultations were assessed. Timeliness of consultation was the most common category receiving a borderline or unsatisfactory rating. Borderline ratings in knowledge-based categories (history, examination, assessment and plan, urgency rating) occurred more often for postgraduate year 2 (PGY2) residents than for PGY3 residents (P = 0.05, chi-square test). Incomplete differential diagnosis (n = 6) and lack of follow-up instruction (n = 5) were the most common deficiencies observed.

CONCLUSIONS

The OCAT has face, content, and discriminative validity. It can be used to assess resident competence in patient care, professionalism, and medical knowledge. Interrater and intrarater reliability still need to be determined. The OCAT may prove to be an additional assessment tool for meeting the Accreditation Council for Graduate Medical Education competencies mandate.

Teaching and improving quality of care in a primary care internal medicine residency clinic

Purpose Learning and applying quality of care principles are essential to practice-based learning and improvement. The authors investigated the feasibility and effects of a self-directed curriculum in quality of care for residents. Method In 2001-02, 13 second-year residents at two community-based outpatient clinics in the Yale University primary care internal medicine residency program were asked to participate in a trial of a quality improvement curriculum (intervention group). Thirteen third-year residents in the same residency program served as the comparison group. The curriculum consisted of readings in quality of care, weekly self-reflection with a faculty member, completion of a commitment to change survey, and medical record audits. Study outcome measures were patient level quality of care measures for diabetes, satisfaction with the curriculum, and self-reported behavioral changes. Results In the follow-up, patients of the intervention group were significantly more likely to have received a monofilament foot examination and baseline electrocardiogram than were patients of the comparison group. When comparing the change between baseline and follow-up, patients for the second-year residents showed significantly more improvement in hemoglobin A1c and LDL cholesterol levels and Pneumovax administration than did patients of the comparison group. All residents in the intervention group were highly satisfied with the curriculum. Thirty-five of 54 residents' personal commitments to change were either partially or fully implemented six months after the curriculum. Conclusions A multifaceted curriculum in quality improvement led to modest improvements in the care of diabetic patients and meaningful changes in self-reported practice behaviors. Future research should include more focus on the microsystems of residency outpatient experiences.

The Ophthalmic Clinical Evaluation Exercise (OCEX)

PURPOSE

New concise tools must be developed to assess reliably and validly the core residency competencies identified by the Accreditation Council for Graduate Medical Education.

PARTICIPANTS

Eighteen content experts (residency program directors).

METHODS

A 1-page Ophthalmic Clinical Exercise Examination (OCEX) checklist, for use during observed resident-patient interactions, was developed by an American Board of Ophthalmology taskforce. The OCEX checklist was sent to 18 content experts for their review and constructive comments.

RESULTS

Experts' comments were incorporated, establishing face and content validity.

CONCLUSIONS

The OCEX has face and content validity. It can be used to assess a resident's patient care skills, medical knowledge, and interpersonal skills. Reliability and predictive validity still need to be determined.

Do peer chart audits improve residents' performance in providing preventive care?

PURPOSE

One recommended method to evaluate residents' competence in practice-based learning and improvement is chart audit. This study determined whether residents improved in providing preventive care after a peer chart audit program was initiated.

METHOD

Residents audited 1,005 charts and scored their peers on 12 clinical preventive services. The mean total chart audit scores were compared across five time blocks of the 45-month study.

RESULTS

Residents' performance in providing preventive care initially improved significantly but declined in the last ten months. However, their performance remained significantly higher than at the beginning.

CONCLUSIONS

By auditing their peers' charts, residents improved their own performance in providing preventive care. The diffusion of innovations theory may explain the prolonged implementation phase and problems in maintaining a chart audit program.

Effectiveness of report cards based on chart audits of residents' adherence to practice guidelines on practice performance: a randomized controlled trial

BACKGROUND

Medical record audits have been used to provide physicians with feedback about their compliance with preventive health and disease management recommendations.

PURPOSE

To determine if report cards summarizing medicine residents' preventive health and disease management practices can be used as a feedback tool to improve practice performance.

METHODS

Randomized, blinded, controlled study of 44 internal medicine residents using an individualized 78-item report card based on outpatient record audits.

RESULTS

Four hundred ninety-seven charts were retrospectively audited at baseline and 284 charts in follow-up. There were no significant differences in baseline performance between the residents in the intervention and control group. There were no differences in performance scores between residents receiving report cards and those who had not in immunizations, counseling, total preventive health, diabetes, hypertension, and total disease management.

CONCLUSIONS

Intensive data-based feedback using report cards may not be a successful way to improve ambulatory performance of medical house officers.

Provider feedback improves adherence with AHCPR Smoking Cessation Guideline

BACKGROUND

This study evaluated the effect of primary care providers' adherence with the AHCPR Smoking Cessation Guideline after receiving a multicomponent intervention.

METHODS

A quasi-experimental study with one intervention and one control team was conducted in a southeastern Veterans Affairs Medical Center primary care setting. During phase I, chart reviews were conducted to measure baseline provider adherence and documentation of the four A's (ask, advise, assist, arrange). In phase II, the intervention team received a single educational session on the AHCPR Guideline, four A's, and tobacco dependence treatment. This was followed by chart reviews of patients seen 4 to 8 weeks after the educational intervention to measure provider adherence and documentation of the four A's. During phase III, the intervention team received individual and team feedback from the chart reviews in phases I and II and booster education on the AHCPR Guideline. Chart reviews were conducted from patient visits 4 to 8 weeks after the feedback and booster education to determine provider adherence and documentation of the four A's.

RESULTS

A nested repeated measures two-factor analysis of variance was performed for each of the following outcomes: ask, advise, assist, and arrange. Data analyses revealed that both the control and the intervention teams had 100% compliance in asking the patient about smoking status. There was a prestudy implementation of the vital sign stamp that included smoking status in this setting. Education on tobacco dependence and the AHCPR Guideline had no significant impact on provider performance with the advisement, assistance, and arrangement of follow-up. However, significant improvements occurred in the intervention team in the advisement (P = 0.05), assistance (P = 0.001), and arrangement of follow-up (P = 0.001) phase after individual and team feedback was provided. This research supports the fact that feedback impacts individuals and team performances and facilitated positive system changes to improve provider adherence with the AHCPR recommendations in treating tobacco dependence.

Reviews of evidence regarding interventions to reduce tobacco use and exposure to environmental tobacco smoke

This report presents the results of systematic reviews of effectiveness, applicability, other effects, economic evaluations, and barriers to use of selected population-based interventions intended to reduce tobacco use and exposure to environmental tobacco smoke. The related systematic reviews are linked by a common conceptual approach. These reviews form the basis of recommendations by the Task Force on Community Preventive Services (TFCPS) regarding the use of these selected interventions. The TFCPS recommendations are presented on page 67 of this supplement.

The effect of audit and feedback on immunization delivery: a systematic review

OBJECTIVE

To assess the effective of audit and feedback (A&F) on immunization delivery by health care professionals.

DESIGN

Systematic review of published literature.

MAIN OUTCOME MEASURES

Changes in immunization rates.

METHODS

We searched Medline between 1966 and 1997. We obtained additional studies from back-searching reference lists and the files of study collaborators. We included studies that were written in English, that included audit and feedback in at least one arm of the study, that studied universally recommended childhood or adult vaccines, and that provided immunization coverage data. Two reviewers read studies independently and abstracted using a validated checklist. Study quality was assessed using criteria standardized by the Cochrane Collaboration. Differences between reviewers were resolved by consensus.

RESULTS

The search process resulted in 60 citations; 44 were fully reviewed and 15 met eligibility criteria. Five were randomized trials. Twelve of the fifteen studies found that A&F, alone or in combination with other interventions, were associated with improvements in immunization rates. The magnitude of the effect varied from -17% to +49% change. Study design heterogeneity precluded statistical pooling of study results.

CONCLUSIONS

The evidence available from published studies suggests that A&F alone may be an effective strategy for improving immunization rates. The number of well-conducted studies is small, and the effect is variable. Additional well-designed studies are needed to identify the independent effects of A&F, optimal format and frequency of A&F, and to examine its long-term effect on provider immunization practices and costs.

Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. The Task Force on Community Preventive Services

This paper presents the results of systematic reviews of the effectiveness, applicability, other effects, economic impact, and barriers to use of selected population-based interventions intended to improve vaccination coverage. The related systematic reviews are linked by a common conceptual approach. These reviews form the basis for recommendations by the Task Force on Community Preventive Services (the Task Force) regarding the use of these selected interventions. The Task Force recommendations are presented on pp. 92-96 of this issue.

The impact of feedback to medical housestaff on chart documentation and quality of care in the outpatient setting

OBJECTIVE

To determine whether feedback from attending physicians to residents about outpatient medical records improves chart documentation and quality of care.

DESIGN

Cross-sectional study with repeated measures.

SETTING

Primary care internal medicine clinic at a metropolitan community hospital.

PATIENT/PARTICIPANTS

Fifteen interns and 20 residents.

INTERVENTION

Attending physicians reviewed at least two charts for each resident on three occasions about 4 months apart and then discussed their findings with the residents.

MEASUREMENTS AND MAIN RESULTS

Explicit criteria defined the extent of chart documentation and the comprehensiveness of care delivery. Attending physicians also made a subjective assessment of the overall quality of care. All results were converted to 0-to-1 scales. From the first to the third period, chart documentation increased from 0.60 to 0.86 (p < .001), but there were no significant changes in the delivery of care or in the subjective assessments of the overall quality of care.

CONCLUSIONS

Both review of residents' outpatient medical records and periodic feedback from attending physicians improve how well medical housestaff document care in the chart.

Interobserver variation in the assessment of the sampling quality of cervical smears

In order to assess the interobserver variation among cytological assistants in evaluating the sampling quality of cervical smears, all four laboratory assistants from one cytological department examined the same set of 71 smears. They assessed the presence of endocervical cells, the quality of fixation, the number of cells and the undesirable presence of clumps of cells. The interobserver agreement, uncorrected for chance, varied from 73 to 93% for the presence of endocervical cells and from 86 to 100% for the three other criteria. For the presence of endocervical cells, the kappa values, a measure for the interobserver agreement corrected for chance, varied from 42 to 84%. This is a reasonable outcome. For the three other criteria many kappa values were zero or uncalculable. The interobserver agreement in evaluating the sampling quality of cervical smears is reasonable to good, but not perfect. Prudence in the interpretation of quality data is warranted.