Using patient encounter logs for mandated clinical encounters in an internal medicine clerkship

Validation of a Novel Statistical Method to Identify Aberrant Patient Logging: A Multi-Institutional Study

INTRODUCTION

Student patient encounter logging informs the quality of supervised clinical practice experiences (SCPEs). Yet, it is unknown whether logs accurately reflect patient encounters, and the faculty resources necessary to review for potential aberrant logging are significant. The purpose of this study was to identify a statistical method to identify aberrant logging.

METHODS

A multi-institutional (n = 6) study examined a statistical method for identifying potentially aberrant logging behavior. An automated statistical Mahalanobis Distance (MD) measurement was used to categorize student logs as aberrant if they were identified as probable multivariate outliers. This approach was validated using a gold standard for aberrant logging behavior with manual review by 4 experienced faculty ("faculty consensus") and then comparing interrater agreement between faculty and MD-based categorization. In secondary analyses, we compared the relative accuracy of MD-based categorization to individual faculty categorizing data from their own program ("own program" categorization).

RESULTS

323 student logging records from 6 physician assistant (PA) programs were included. Compared to "faculty consensus" (the gold standard), MD-based categorization was highly sensitive (0.846, 95% CI: 0.650, 1.000) and specific (0.766, 95% CI: 0.645, 0.887). Additionally, there was no significant difference in sensitivity, specificity, positive predictive value, or negative predictive value between MD-based categorization and "own program" categorization.

DISCUSSION

The MD-based method of identifying aberrant and nonaberrant student logging compared favorably to the more traditional, faculty-intensive approach of reviewing individual student logging records. This supports MD-based screening as a less labor-intensive alternative to individual faculty review to identify aberrant logging. Identification of aberrant logging may facilitate early intervention with students to improve clinical exposure logging during their SCPEs.

Do senior medical students meet recommended emergency medicine curricula requirements?

BACKGROUND

Emergency departments (EDs) offer a variety of learning opportunities for undergraduate medical students. It is however, difficult to evaluate whether they are receiving recommended training during their emergency medicine (EM) clerkship without identifying their clinical activities. We aimed to evaluate the clinical exposure of the final year medical students at our College during their EM clerkship.

METHODS

This is a retrospective analysis of prospectively collected student logbooks. 75 students rotated in a 4-week EM clerkship during 2015-2016. The students rotated in EDs of two hospitals. Each ED treats more than 120,000 cases annually. The students completed 12 eight-hours shifts. Presentations and procedures seen were compared with EM curriculum recommendations.

RESULTS

Five thousand one hundred twenty-two patient presentations and 3246 procedures were recorded in the logbooks, an average (SD) of 68.3 (17.6) patients and 46.1 (14.0) procedures. None of the students encountered all ten recommended presentations. Two students (2.6%) logged all nine procedure categories of the EM curriculum.

CONCLUSION

Recommended presentations and procedures of the EM clerkship were not fully encountered by all our students. Different settings vary in the availability and type of patients and procedures. Each clinical clerkship should tailor their teaching methods based on the available learning opportunities.

Paediatric case mix in a rural clinical school is relevant to future practice

BACKGROUND

Exposure to a representative case mix is essential for clinical learning, with logbooks established as a way of demonstrating patient contacts. Few studies have reported the paediatric case mix available to geographically distributed students within the same medical school. Given international interest in expanding medical teaching locations to rural contexts, equitable case exposure in rural relative to urban settings is topical. The Rural Clinical School of Western Australia locates students up to 3500 km from the urban university for an academic year. There is particular need to examine paediatric case mix as a study reported Australian graduates felt unprepared for paediatric rotations. We asked: Does a rural clinical school provide a paediatric case mix relevant to future practice? How does the paediatric case mix as logged by rural students compare with that by urban students?

METHODS

The 3745 logs of 76 urban and 76 rural consenting medical students were categorised by presenting symptoms and compared to the Australian Institute of Health and Welfare (AIHW) database Major Diagnostic Categories (MDCs).

RESULTS

Rural and urban students logged core paediatric cases, in similar order, despite the striking difference in geographic locations. The pattern of overall presenting problems closely corresponded to Australian paediatric hospital admissions. Rural students logged 91% of cases in secondary healthcare settings; urban students logged 90% of cases in tertiary settings. The top four presenting problems were ENT/respiratory, gastrointestinal/urogenital, neurodevelopmental and musculoskeletal; these made up 60% of all cases. Rural and urban students logged similar proportions of infants, children and adolescents, with a variety of case morbidity.

CONCLUSIONS

Rural clinical school students logged a mix of core paediatric cases relevant to illnesses of Australian children admitted to public hospitals, with similar order and pattern by age group to urban students, despite major differences in clinical settings. Logged cases met the curriculum learning outcomes of graduates. Minor variations were readily addressed via recommendations about logging. This paper provides evidence of the legitimacy of student logs as useful tools in affirming appropriate paediatric case mix. It validates the rural clinical school context as appropriate for medical students to prepare for future clinical paediatric practice.

Tracking Patient Encounters and Clinical Skills to Determine Competency in Ambulatory Care Advanced Pharmacy Practice Experiences

OBJECTIVE

To determine if the amount of exposure to patient encounters and clinical skills correlates to student clinical competency on ambulatory care advanced pharmacy practice experiences (APPEs).

DESIGN

Students in ambulatory care APPEs tracked the number of patients encountered by medical condition and the number of patient care skills performed. At the end of the APPE, preceptors evaluated students' competency for each medical condition and skill, referencing the Dreyfus model for skill acquisition.

ASSESSMENT

Data was collected from September 2012 through August 2014. Forty-six responses from a student tracking tool were matched to preceptor ratings. Students rated as competent saw more patients and performed more skills overall. Preceptors noted minimal impact on workload.

CONCLUSIONS

Increased exposure to patient encounters and skills performed had a positive association with higher Dreyfus stage, which may represent a starting point in the conversation for more thoughtful design of ambulatory care APPEs.

The need for logbooks to evolve in the undergraduate medical setting

None

Rural Versus Suburban/Urban Experiences in a Family Medicine Preceptorship

PURPOSE

Accreditation standards require physician assistant (PA) programs to ensure students receive adequate clinical experiences. During their clinical year of training, PA students complete rotations with multiple clinical preceptors, introducing them to practice and exposing them to a variety of clinical problems. In this article, we examined Typhon Physician Assistant Student Tracking (PAST) system patient encounter logs' value for program evaluation, but also in research to address questions relevant to PA education. Specifically, we explored the MEDEX Northwest Physician Assistant Training Program student experience across rural versus suburban/urban placements in a 4-month family medicine preceptorship.

METHODS

Student experience was analyzed from 2 years of collected Typhon PAST encounter data. Encounter characteristics included duration, number of clinical problems, student level of responsibility, and decision type. Patient characteristics included sex, age, race, and clinical problems recorded as ICD-9 codes.

RESULTS

Individual student experience varied widely across different preceptors. However, these differences were more specific to the preceptor-student relationship than to whether the site was classified as rural or suburban/urban. Across these settings, significant differences were only noted for percentage of female and 65 or older patient encounters. The most common clinical problems reported across rural versus suburban/urban sites were highly correlated.

CONCLUSIONS

Physician Assistant Student Tracking data demonstrated that individual student experience in their family medicine rotation varied widely. However, in general, rural and suburban/urban experiences were more similar than different. This study supports the value of the Typhon PAST logging system for not only tracking student activity but also addressing program evaluation and research questions.

Comparison of students' encountered diseases and available diseases at clerkship sites by exploratory multivariate analysis: Are encountered diseases predictable?

BACKGROUND

Students in clerkship are expected to gain clinical expertise by interacting with real patients in clinical situations. Monitoring and predicting the students' encounter diseases (EDs) is important for providing an optimal experience. EDs should be compared with the available diseases (ADs) at the clerkship site and with the required diseases described in some guidelines for the clinical curriculum.

AIMS

To explore the differences in ADs as learning resources among different types of clerkship sites and to investigate discrepancies between EDs and ADs.

METHOD

A retrospective observational study used secondary data from government statistics to compare ADs of various types of observable clerkship sites by biplot analyses, which allowed multivariate comparisons. EDs collected from logbooks during clerkships at a university hospital were also compared with ADs across sites.

RESULTS

The distributions of ADs differed according to institutional type, and EDs at Kyushu University Hospital were similar to the ADs for the category of hospitals in which it was placed.

CONCLUSION

EDs at a clerkship site may be predictable to some extent by analysing the site's distribution of ADs, but further study is needed. Biplot is useful for visualising these types of statistical similarity.

Could clinical experience during clerkship enhance students' clinical performance?

BACKGROUND

Medical students learn and practice various clinical skills during clinical clerkship. Patient encounters are important for developing clinical thinking, communication skills, and professional attitude. We investigated whether the amount of clinical experience during clerkship correlated with students' clinical competency and students' perception of effectiveness of their clerkship on it.

METHODS

Fourth year medical students undertook the Objective Structured Clinical Examinations (OSCE) in August 2012. Students provided the number of patients for whom they took medical histories or performed physical examinations during clerkship and provided feedback as to whether or not the clinical clerkship was helpful in preparing OSCE. The correlation between the OSCE score and number of patients was analyzed.

RESULTS

One hundred thirty students completed the questionnaire (86.6%). OSCE scores correlated with the total number of patients encountered for physical examinations (correlation coefficient, 0.274; p = 0.0105). Cumulative 3-year GPAs were positively correlated with OSCE scores (correlation coefficient, 0.330; p = 0.0001). Most (92.3%) answered that their clinical clerkship was helpful in preparing them for the OSCE; however, only 20% felt that their clinical clerkship was most helpful. Others felt that role playing (38.46%) or the guide book (33.84%) was most helpful.

CONCLUSIONS

The amount of clinical experience during the students' clerkship had a small but positive relationship with students' clinical performance. Further research to elucidate the influence of clinical experience on clinical competency is needed.

Medical student education in emergency medicine: do students meet the national standards for clinical encounters of selected core conditions?

BACKGROUND

Establishing a core curriculum for undergraduate Emergency Medicine (EM) education is crucial to development of the specialty. The Clerkship Directors in Emergency Medicine (CDEM) National Curriculum Task Force recommended that all students in a 4(th)-year EM clerkship be exposed to 10 emergent clinical conditions.

OBJECTIVES

To evaluate the feasibility of encountering recommended core conditions in a clinical setting during a 4(th)-year EM clerkship.

METHODS

Students from three institutions participated in this ongoing, prospective observation study. Students' patient logs were collected during 4-week EM clerkships between July 2011 and June 2012. De-identified logs were reviewed and the number of patient encounters for each of the CDEM-identified emergent conditions was recorded. The percentage of students who saw each of the core complaints was calculated, as was the average number of core complaints seen by each.

RESULTS

Data from 130 students at three institutions were captured; 15.4% of students saw all 10 conditions during their rotation, and 76.9% saw at least eight. The average number of conditions seen per student was 8.4 (range of 7.0-8.6). The percentage of students who saw each condition varied, ranging from 100% (chest pain and abdominal pain) to 31% (cardiac arrest).

CONCLUSIONS

Most students do not encounter all 10 conditions during patient encounters throughout a 4-week EM rotation, although most have exposure to at least eight. Certain conditions are far less likely than others to be encountered, and may need to be taught in a nonclinical setting.

Mobile technology for the facilitation of direct observation and assessment of student performance

BACKGROUND

We developed, implemented, and assessed a web-based clinical evaluation application (i.e., CEX app) for Internet-enabled mobile devices, including mobile phones. The app displays problem-specific checklists that correspond to training problems created by the Clerkship Directors in Internal Medicine.

PURPOSE

We hypothesized that use of the CEX app for directly observing students' clinical skills would be feasible and acceptable, and would demonstrate adequate reliability and validity.

METHODS

Between July 2010 and February 2012, 266 third-year medical students completed 5 to 10 formative CEXs during their internal medicine clerkship. The observers (attendings and residents), who performed the CEX, used the app to guide and document their observations, record their time observing and giving feedback to the students, and their overall satisfaction with the CEX app. Interrater reliability and validity were assessed with 17 observers who viewed 6 videotaped student-patient encounters, and by measuring the correlation between student CEX scores and their scores on subsequent standardized-patient Objective Structured Clinical Examination (OSCE) exams.

RESULTS

A total of 2,523 CEXs were completed by 411 observers. The average number of evaluations per student was 9.8 (± 1.8 SD), and the average number of CEXs completed per observer was 6 (± 11.8 SD). Observers spent less than 10 min on 45.3% of the CEXs and 68.6% of the feedback sessions. An overwhelming majority of observers (90.6%) reported satisfaction with the CEX. Interrater reliability was measured at 0.69 among the observers viewing the videotapes, and their ratings discriminated between competent and noncompetent performances. Student CEX grades, however, did not correlate with their end of 3rd-year OSCE scores.

CONCLUSIONS

The use of this CEX app is feasible and it captures students' clinical performance data with a high rate of user satisfaction. Our embedded checklists had adequate interrater reliability and concurrent validity. The grades measured on this app, however, were not predictive of subsequent student performance.

The internal medicine clerkship and ambulatory learning experiences: results of the 2010 clerkship directors in internal medicine survey

BACKGROUND

Education in the ambulatory setting should be an integral part of undergraduate medical education. However, previous studies have shown education in this setting has been lacking in medical school. Ambulatory education occurs on some internal medicine clerkships. The extent of this education is unclear.

PURPOSE

The purpose of this survey was to assess the structure, curriculum, assessment methods, and barriers to implementation of ambulatory education on the internal medicine clerkship.

METHODS

An annual survey of institutional members of the Clerkship Directors in Internal Medicine (CDIM) was done in April 2010. The data were anonymous and descriptive statistics were used to summarize responses. Free text results were analyzed using qualitative techniques.

RESULTS

The response rate was 75%. The majority of respondents had a required ambulatory component to the clerkship. Ambulatory experiences distinct from the inpatient internal medicine experience were common (46%). Integration with either the inpatient experiences or other departmental clerkships also occurred. The majority of ambulatory educational experiences were with generalists (74%) and/or subspecialists (45%). The most common assessment tool was the National Board of Medical Examiners (NBME) ambulatory shelf exam. Thematic analysis of the question about how practice based learning was taught elicited four major themes: Not taught; taught in the context of learning evidence based medicine; taught while learning chronic disease management with quality improvement; taught while learning about health care finance. Barriers to implementation included lack of faculty and financial resources.

CONCLUSIONS

There have been significant increases in the amount of time dedicated to ambulatory internal medicine. The numbers of medical schools with ambulatory internal medicine education has increased. Integration of the ambulatory experiences with other clerkships such as family medicine occurs. Curriculum was varied but difficulties with dissemination and assessment in these disparate settings was noted. Overall, the results of this study demonstrate increased implementation and recognition of the importance of ambulatory education in internal medicine.

Relationship between clinical experiences and internal medicine clerkship performance

OBJECTIVES

This study was conducted to assess the associations between several clerkship process measures and students' clinical and examination performance in an internal medicine clerkship.

METHODS

We collected data from the internal medicine clerkship at one institution over a 3-year period (classes of 2010-2012; n = 507) and conducted correlation and multiple regression analyses. We examined the associations between clerkship process measures (student-reported number of patients evaluated, percentage of core problems encountered, total number of core problems encountered, total number of clinics attended) and four clerkship outcomes (clinical points [a weighted summation of a student's clinical grade recommendations], ambulatory clinical points [the out-patient portion of clinical points], examination points [a weighted summation of scores on three clerkship examinations], and National Board of Medical Examiners examination score).

RESULTS

After controlling for pre-clerkship ability and gender, percentage of core problems was significantly associated with ambulatory clinical points (b = 3.84, total model R(2) = 0.14). Further, number of patients evaluated was significantly associated with clinical points (b = 0.19, total model R(2) = 0.22), but only for students who undertook first-quarter clerkships, who reported higher numbers of patients.

CONCLUSIONS

Notwithstanding a few positive (but small) associations, the results from this study suggest that clinical exposure is, at best, weakly associated with internal medicine clerkship performance.

Reaching the limits of mandated self-reporting: clinical logbooks do not predict clerkship performance

BACKGROUND

Logbooks are used by US medical schools to evaluate curricular objectives and meet accreditation requirements, but research supporting their utility is conflicting.

AIMS

The goal of our study was to examine the relationship between volume of clinical rotation experiences as reported in a logbook and clerkship grades within a longitudinal integrated clerkship.

METHODS

We conducted a retrospective cohort study of third-year (clinical) medical students during academic year 2008-2009. We tracked student entries in a pocket-sized logbook (number of clinical encounters per clerkship, total number of exams, and procedures over the academic year). We performed correlation analyses between logbook entries and clerkship grades.

RESULTS

We enrolled 36 students, who reported a total of 2992 encounters, 2262 exams, and 2342 procedures. Correlation coefficients between volume of clinical experience and clerkship grades were less than 0.4, indicating low correlation. We found borderline statistical significance for the Neurology, OB/Gyn, and Surgery clerkships. Sensitivity analyses showed little correlation between low-reporting activity and clerkship grades.

CONCLUSIONS

Even within an integrated longitudinal clerkship framework, our findings are consistent with previous studies showing a negligible relationship between logbooks as an educational process measure and how they relate to educational outcomes.

Associations between medical student log data and clerkship learning outcomes

BACKGROUND

Largely due to regulatory requirements, medical students increasingly document patient encounters in logs. Prior studies demonstrated value in achieving course objectives but not regarding objective learning outcomes.

PURPOSE

The purpose of this article is to assess the associations between students' log data and clerkship outcomes.

METHODS

We assessed ambulatory, internal medicine-pediatrics clerkship logs. Students recorded patients' diagnoses, preceptor's teaching content and performing histories and physicals (H&P). We examined associations between log data and National Board of Medical Examiners Subject Exams (NBME), Objective Structured Clinical Exams (OSCE), and clinical evaluations.

RESULTS

All 272 students completed logs (M diagnoses = 146.5, SD = 24.9). In univariate analysis number of diagnoses, preceptors' teaching patient management and performing H&P independently correlated with OSCE scores (r = .13-.24, p < .05). NBME scores correlated with diagnoses, performing H&P independently (r = .13-.18, p < .05). Teaching patient management, diagnoses, performing H&P independently were associated with clinical evaluations (all ps < .05). Regressions demonstrated that performing physicals independently was associated with NBME and OSCE scores (p < .05, R (2) = .03 and .05, respectively). Teaching disease management and pediatric diagnoses were associated with clinical evaluations (p < .05) in medicine and pediatrics, respectively.

CONCLUSIONS

Aspects of teaching, increased clinical involvement, and student autonomy were associated with clerkship performance.

Investing in medical students: which ones are worth a surgeon's time?

BACKGROUND

Little information exists on which third-year medical students will become worthy protégés.

METHODS

Performance and encounter metrics for 4 classes of medical students were correlated with the students who matched into a surgery residency.

RESULTS

Overall, 68 of 422 (16%) graduating seniors matched into a surgery residency. Male students were twice as likely as female students to match in surgery were, and most (81%) matching students took their general surgery rotation at the university hospital. Students who took certain elective rotations (eg, urology and orthopedics) tended to enter those surgical professions. Grades, test scores, clinical performance, oral presentation skills, and encounter data were not predictive of which students would enter the surgical profession.

CONCLUSION

For surgeons who are willing to mentor medical students, our data suggest that currently, the selection of medical students worthy of surgeon-mentor's time will remain an art.

The implementation of a mobile problem-specific electronic CEX for assessing directly observed student-patient encounters

BACKGROUND

Facilitating direct observation of medical students' clinical competencies is a pressing need.

METHODS

We developed an electronic problem-specific Clinical Evaluation Exercise (eCEX) based on a national curriculum. We assessed its feasibility in monitoring and recording students' competencies and the impact of a grading incentive on the frequency of direct observations in an internal medicine clerkship. Students (n = 56) at three clinical sites used the eCEX and comparison students (n = 56) at three other clinical sites did not. Students in the eCEX group were required to arrange 10 evaluations with faculty preceptors. Students in the second group were required to document a single, faculty observed 'Full History and Physical' encounter with a patient. Students and preceptors were surveyed at the end of each rotation.

RESULTS

eCEX increased students' and evaluators' understanding of direct-observation objectives and had a positive impact on the evaluators' ability to provide feedback and assessments. The grading incentive increased the number of times a student reported direct observation by a resident preceptor.

CONCLUSIONS

eCEX appears to be an effective means of enhancing student evaluation.