A validated search assessment tool: assessing practice-based learning and improvement in a residency program

OBJECTIVE

The objective of this study was to validate an assessment instrument for MEDLINE search strategies at an academic medical center.

METHOD

Two approaches were used to investigate if the search assessment tool could capture performance differences in search strategy construction. First, data from an evaluation of MEDLINE searches from a pediatric resident's longitudinal assessment were investigated. Second, a cross-section of search strategies from residents in one incoming class was compared with strategies of residents graduating a year later. MEDLINE search strategies formulated by faculty who had been identified as having search expertise were used as a gold standard comparison. Participants were presented with a clinical scenario and asked to identify the search question and conduct a MEDLINE search. Two librarians rated the blinded search strategies.

RESULTS

Search strategy scores were significantly higher for residents who received training than the comparison group with no training. There was no significant difference in search strategy scores between senior residents who received training and faculty experts.

CONCLUSION

The results provide evidence for the validity of the instrument to evaluate MEDLINE search strategies. This assessment tool can measure improvements in information-seeking skills and provide data to fulfill Accreditation Council for Graduate Medical Education competencies.

Teaching primary care in community health centers: addressing the workforce crisis for the underserved

Universal coverage and multiple initiatives to improve health care delivery are crucial components of health care reform. However, the missing link has been a plan to rapidly address the primary care workforce crisis for the underserved. The authors propose a link between primary care graduate medical education and care for the underserved in community health centers, where expansion will be necessary for the anticipated increase in Medicaid and insured patients. This can be achieved by establishing primary care teaching health centers in expanded community health centers, which have established a patient-centered medical home practice environment. Residents would receive their final year of training in these centers, and then have the incentive of National Health Service Corps debt repayment if they subsequently practice in an underserved area. Primary care residents being trained in this setting would immediately increase the clinical capacity of community health centers and ultimately expand the primary care physician workforce. This proposal addresses the primary care physician workforce crisis and the associated key problems of limited access for the underserved and suboptimal primary care graduate medical education.

Choosing a career in combined internal medicine-pediatrics: insights from interns

BACKGROUND

Combined internal medicine-pediatrics (med-peds) programs may be competing for the same students who would have otherwise chosen family medicine. The degree to which this is happening is not known.

METHODS

We sent an eight-item questionnaire to new med-peds interns to assess their career plans at different stages of their decision making. Questionnaires were mailed to the directors of all US med-peds programs in 2002.

RESULTS

A total of 288/333 (87%) responded. The med-peds interns were more likely to be interested in internal medicine or pediatrics than they were in family medicine. If med-peds were not available, only 52/286 (18%) would have chosen family medicine as an alternative. A total of 55/288 (19%) anticipated practicing in rural areas.

CONCLUSIONS

The majority of med-peds interns would have chosen internal medicine or pediatrics if med-peds was not available. A small percentage would have chosen family medicine, thus having a minor impact on recruitment. An even smaller proportion would have chosen a non-primary care specialty. A sizable number anticipate practicing in rural areas.

Development of an evaluation of medical student competence in evidence-based medicine using a computer-based OSCE station

BACKGROUND

Instruction in evidence-based medicine (EBM) has been widely incorporated into medical school curricula with little evidence of its effectiveness. Our goal was to create, implement, and validate a computer-based assessment tool that measured medical students' EBM skills.

DESCRIPTION

As part of a required objective structured clinical examination, we developed a specific case scenario in which students (a) asked a structured clinical question using a standard framework, (b) generated effective MEDLINE search terms to answer a specific question, and (c) elected the most appropriate of 3 abstracts generated from a search justifying which best applies to the patient scenario.

EVALUATION

Between the 3 blinded raters, there was very good interrater reliability with 84, 94, and 96% agreement on the scoring for each component, respectively (k = .64, .82, and .91, respectively). In addition, students found the station appropriately difficult for their level of training.

CONCLUSIONS

This computer-based tool appears to measure several EBM skills independently and combines simple administration and scoring. Its generalizability to other cases and settings requires further study.

Assessing residents' competencies at baseline: identifying the gaps

PURPOSE

Entering residents have variable medical school experiences and differing knowledge and skill levels. To structure curricula, enhance patient safety, and begin to meet accreditation requirements, baseline assessment of individual resident's knowledge and skills is needed. To this end, in 2001 the University of Michigan Health System created the Postgraduate Orientation Assessment (POA), an eight-station, objective structured clinical examination for incoming residents.

METHOD

The POA, administered at orientation, included items addressing critical laboratory values, cross-cultural communication, evidence-based medicine, radio-graphic image interpretation, informed consent, pain assessment and management, aseptic technique, and system compliance such as fire safety. The POA assessed many of the skills needed by interns in their initial months of training when supervision by senior physicians might not be present.

RESULTS

In 2002, 132 interns from 14 different specialties and 59 different schools participated in the POA. The mean score was 74.8% (SD = 5.8). When scores were controlled for U.S. Medical Licensing Examination scores, there were no significant differences in performance across specialties. There were differences between University of Michigan Medical School graduates and those from other institutions (p <.001). Eighty-one percent of the residents would recommend the POA.

CONCLUSIONS

The POA provides a feasible format to measure initial knowledge and skills and identify learning needs. Orientation is an effective time to identify important gaps in learning between medical school and residency. This is the first step in a continuing evaluation of the Accreditation Council for Graduate Medical Education's general competencies.

Internal medicine-pediatrics residency training: current program trends and outcomes

PURPOSE

Combined internal medicine-pediatrics (med-peds) residency programs have existed since 1967. Due to the rapid growth in the number and size of programs during the 1990s, most current med-peds physicians completed their residency in the last ten years, making older studies of med-peds programs obsolete. The authors sought to determine completion rates of med-peds residency programs and describe the initial career plans for five cohorts of graduating residents from combined med-peds training programs.

METHOD

Program directors of all U.S. med-peds residency programs were asked to complete a Web-based survey and base their responses on the records of cohorts of residents completing their programs from 1998 through 2002. To allow sufficient time to complete both the American Board of Pediatrics (ABP) and American Board of Internal Medicine (ABIM) certification examinations, certification status was requested only for the cohort completing training in 1998.

RESULTS

Responses were obtained from 92% (83/90) of the programs, reflecting 1,595 residents entering med-peds programs. Of these residents, 91% graduated from a med-peds program. Among the graduates, 82% were seeing both adults and children, 22% went on to subspecialty residencies, 21% began practice in rural or underserved areas, and 25% entered an academic position. ABIM and ABP pass rates for the 1998 cohort were 97% and 96%, respectively. Overall, 79% of the 1998 graduates are board certified in both specialties.

CONCLUSIONS

Compared with previous studies, a greater proportion of residents who recently entered med-peds programs completed their dual training, and a larger percentage of graduates are seeing both adults and children. The proportion of residents entering subspecialty residencies has increased significantly, but the proportion of graduates in academic careers has remained stable.

Assessing residents' competency in care management: report of a consensus conference

BACKGROUND

Residency programs must prepare physicians to practice in the current health care environment. This mandate is reflected in 3 of the 6 competency domains now required by the Accreditation Council for Graduate Medical Education: systems-based practice, interpersonal skills and communication, and practice-based learning and improvement.

SUMMARY

An invitational conference was convened, with experts in clinical practice, managed care administration, and education to identify and recommend optimal and promising assessment methods for 4 target areas: physician-patient communication, ethics, teamwork and collaboration, and practice management. Working in small groups, participants considered a range of resident assessment methods and identified current or future methods for each area, based on reliability, validity, use of behaviorally oriented outcomes, feasibility, and cost. Preferred methods of assessment varied by domain and include written examinations, computer-based patient management problems, standardized patients, objective structured clinical examinations, portfolios, 360-degree evaluations, and patient satisfaction surveys.

CONCLUSIONS

The use of several practical, scientifically sound, and specific methods for assessing residents' competency in care management are recommended. Assessment instruments will need to be flexible enough to adapt to the rapid changes in the health care delivery system and terminology.

Impact of a veterans affairs continuity clinic on resident competencies in women's health

BACKGROUND

Education in women's health is now considered a core curricular component during residency training in Internal Medicine. There is potential for insufficient training in women's health for residents with a continuity clinic based at a Veterans Affairs (VA) hospital.

OBJECTIVE

To determine the impact of a 3-year continuity clinic based at a VA hospital on residents' self-reported competencies in women's health.

DESIGN

Cross sectional survey using an internal website.

SETTING

University-based residency program in Ann Arbor, Michigan.

MEASUREMENTS AND MAIN RESULTS

Comparison of residents with a VA clinic with residents with non-VA clinics (university and community) in self-reported competencies in knowledge base, counseling, and physical exam skills in the area of women's health. Responses were obtained from 66% (n = 72) of eligible residents. When compared to residents with either a university hospital- or community-based clinic site, VA-based residents reported less confidence in the majority of competencies surveyed. Clinic site had the strongest impact in the knowledge base domain, accounting for between 17% and 33% of the variance in each specific competency. For estimated number of Pap smears and breast exams done in the prior year, VA-based residents reported doing, on average, less than 5 of each per year while non-VA residents reported doing between 11 and 20 of each exam.

CONCLUSIONS

Our data suggest that despite other clinical opportunities in women's health during ambulatory rotations, regular clinical experiences in women's health in the continuity clinic setting are necessary to improve education in this area.

A computer-based OSCE station to measure competence in evidence-based medicine skills in medical students

OBJECTIVE

To create a feasible, valid, and reliable tool to measure third-year medical students' skills in evidence-based medicine (EBM).

DESCRIPTION

EBM skills-asking clinical questions, finding appropriate medical information resources, and appraising and applying them to patients-involve higher-order critical thinking abilities and are essential to being a competent physician. Students at our institution must pass a required OSCE exam at the end of their third year. As part of this exam, we developed a new 20-minute computer-based station to assess students' EBM skills. Using a specific case scenario, we asked the students to (1) ask a question using the population/intervention/comparison/outcome (PICO) framework; (2) generate appropriate search terms, given a specific question; and (3) select an appropriate abstract to answer a given question and state why two other abstracts were not appropriate. Prior to the assessment, we determined grading and passing criteria for each of the three components and for the station overall. Of the 140 students who completed the station, the percentages that passed the components were 71%, 81%, and 49% respectively, with only 29% passing all three parts. Preliminary analysis of psychometric properties of the station shows very good to excellent interrater reliability, with 65%, 67%, and 94% agreement on the scoring for the components, and kappas of.64,.82, and.94, respectively.

DISCUSSION

Although there are many curricula for teaching EBM concepts, there are few tools to measure whether students are competent in applying their EBM skills. Our pilot station appears to be an innovative and promising tool to measure several EBM skills independently. By being computer-based, it is relatively simple to administer, grade, and evaluate. While preliminary data show good inter-rater reliability with our use of a single case, future work will include further testing of reliability and assessment of different types of cases. We will also use the results of this assessment to drive continuous improvement in our EBM curriculum. The students who completed this pilot station had not received an extensive formal EBM curriculum, whereas future groups will. We also will explore whether scores on our station correlate with those on other OSCE stations that also assess critical thinking skills, or if scores correlate with a student's clinical grades or overall class standing. We hope to test these hypotheses: (1) skills used in EBM are useful and valid measures of critical thinking abilities in learners and (2) tools such as ours will help to measure these essential competencies.

Infant sleep placement after the back to sleep campaign

OBJECTIVES

The Back to Sleep campaign has been credited with recent declines in the incidence of sudden infant death syndrome. Using survey data for the 1996-1998 birth cohorts, this epidemiologic study examines infant sleep position in a large, population-based sample.

DATA AND METHODS

Data concerning infant sleep position are drawn from the 1996-1998 Pregnancy Risk Assessment Monitoring System for 15 states. Weighted multiple logistic regression analysis is used to examine correlates of infant sleep position.

RESULTS

The prevalence of prone infant sleeping significantly declined between 1996 and 1998 (adjusted odds ratio [AOR] = 0.70; 95% confidence interval [CI] = [0.63, 0.78]). African Americans were more likely than non-Hispanic whites to sleep prone, (AOR = 1.45; 95% CI = 1.33,1.59), and were less likely to sleep supine (AOR = 0.52; 95% CI = 0.48, 0.57). Hispanic/Latinos were less likely overall than non-Hispanic whites to sleep prone (AOR = 0.81; 95% CI = 0.69, 0.95), but were also less likely to sleep supine (AOR = 0.78; 95% CI = 0.69, 0.87). Adherence to sleep position recommended by the American Academy of Pediatrics increased sharply among Hispanic/Latino infants. Very low birth weight infants and infants in larger families were less likely to sleep in the recommended supine position. Infants born between 1001 and 1500 g (AOR = 0.67; 95% CI = 0.57, 0.79), and extremely low birth weight infants between 500 and 1000 g (AOR = 0.57; 95% CI = 0.45, 0.72) were especially unlikely to sleep supine. Infants in households with more than 3 other children (AOR = 1.72; 95% CI = 1.08, 2.74) were more likely to sleep prone.

CONCLUSIONS

The prevalence of supine infant sleep increased between 1996 and 1998. Low adherence to sleep position recommendations of the American Academy of Pediatrics among African Americans, very low birth weight infants, and infants in large families remain public health concerns.