A grant-based experiment to train clinical investigators: the AACR/ASCO methods in clinical cancer research workshop

To address the need for clinical investigators in oncology, AACR and ASCO established the Methods in Clinical Cancer Research Workshop (MCCRW). The workshop's objectives were to: (1) provide training in the methods, design, and conduct of clinical trials; (2) ensure that clinical trials met federal and international ethical guidelines; (3) evaluate the effectiveness of the workshop; and (4) create networking opportunities for young investigators with mentoring senior faculty. Educational methods included: (1) didactic lectures; (2) Small Group Discussion Sessions; (3) Protocol Development Groups; (4) one-on-one mentoring. Learning focused on the development of an IRB-ready protocol, which was submitted on the last day of the workshop. Evaluation methods included: (1) pre- and post-workshop tests; (2) students' workshop evaluations; (3) faculty's ratings of protocol development; (4) students' productivity in clinical research after the workshop; (5) an independent assessment of the workshop. From 1996-2014, 1932 students from diverse backgrounds attended the workshop. There was a significant improvement in the students' level of knowledge from the pre- to the post-workshop exams (p < 0.001). Across the classes, student evaluations were very favorable. At the end of the workshop, faculty rated 92-100% of the students' protocols as ready for IRB submission. Intermediate and long-term follow-ups indicated that more than 92% of students were actively involved in patientrelated research, and 66% had implemented five or more protocols. This NCI-sponsored MCCRW has had a major impact on the training of clinicians in their ability to design and implement clinical trials in cancer research.

Effects of a patient's name and image on medical knowledge acquisition

PURPOSE

To assess whether there are differences in medical students' (MS) knowledge acquisition after being provided a virtual patient (VP) case summary with a patient's name and facial picture included compared to no patient's name or image.

METHOD

76 MS from four clerkship blocks participated. Blocks one and three (Treatment group) were provided case materials containing the patient's name and facial picture while blocks two and four (Control group) were provided similar materials without the patient's name or image. Knowledge acquisition was evaluated with a multiple-choice-question examination (CQA_K).

RESULTS

Treatment group CQA_K scores were 64.6% (block one, n = 18) and 76.0% (block three, n = 22). Control group scores were 71.7%, (block two, n = 17) and 68.4% (block four, n = 19). ANOVA F-test among the four block mean scores was not significant; F (3, 72) = 1.68, p = 0.18, η2=0.07. Only 22.2% and 27.3% of the MS from blocks one and three respectively correctly recalled the patient's name while 16.7% and 40.9% recalled the correct final diagnosis of the patient.

CONCLUSIONS

These results suggest that including a patient's name and facial picture on reading materials may not improve MS knowledge acquisition. Corroborating studies should be performed before applying these results to the design of instructional materials.

Reporting school data on the dental licensure examination

Beginning in 2012, candidate performance on the National Board Dental Examination (NBDE) has been reported as pass/fail, and only the failing candidates receive numerical scores for remediation purposes. The Joint Commission on National Dental Examinations recognizes that the numerical scores have been important information to dental schools for curriculum evaluation and that the pass/fail reports do not provide meaningful information to the schools. This article describes the process of evaluating and validating a new model for reporting standardized school-level performance data on the NBDE. Under this new model, schools are able to monitor the overall performance of their students compared to a national cohort.

Constructing licensure exams: a reliability study of case-based questions on the National Board Dental Hygiene Examination

Patient cases with associated questions are a method for increasing the clinical relevance of licensure exams. This study used generalizability theory to assess changes in score reliability when the number of questions per case varied in the National Board Dental Hygiene Examination (NBDHE). The experimental design maintained the same total number of case-based items, while varying the number of cases and items within cases to assess changes in score reliability. Using generalizability theory, the amounts of error variance within cases and between cases on the NBDHE were assessed. Impact on score reliability (generalizability) was computed. The data were from the responses of 4,528 candidates who took the paper-pencil version of the NBDHE in spring 2009. Results showed that the minimum value of generalizability occurred when fourteen cases with ten items each were used in the examination. The maximum value of generalizability occurred when fifty cases with three items each were used in the examination. The research findings support the development of more cases with fewer items per case on the NBDHE in order to enhance test score reliability and validity. Practical constraints should be considered if more cases with fewer items per case are developed for future examinations.

Career transition and dental school faculty development program

Academic dentistry, as a career track, is not attracting sufficient numbers of new recruits to maintain a corps of skilled dental educators. The Faculty Development Program (FDP) at the University of Texas Health Science Center at San Antonio Dental School received federal funds to institute a 7-component program to enhance faculty recruitment and retention and provide training in skills associated with success in academics including:(1) a Teaching Excellence and Academic Skills (TExAS)Fellowship, (2) training in research methodology,evidence-based practice research, and information management, (3) an annual dental hygiene faculty development workshop for dental hygiene faculty, (4) a Teaching Honors Program and Academic Dental Careers Fellowship to cultivate students' interest in educational careers, (5) an Interprofessional Primary Care Rotation,(6) advanced education support toward a master's degree in public health, and (7) a key focus of the entire FDP, an annual Career Transition Workshop to facilitate movement from the practice arena to the educational arm of the profession.The Career Transition Workshop is a cap stone for the FDP; its goal is to build a bridge from practice to academic environment. It will provide guidance for private practice, public health, and military dentists and hygienists considering a career transition into academic dentistry. Topics will be addressed including: academic culture, preparation for the academic environment,academic responsibilities, terms of employment,compensation and benefits, career planning, and job search / interviewing. Instructors for the workshop will include dental school faculty who have transitioned from the practice, military, and public health sectors into dental education.Objectives of the Overall Faculty Development Program:• Provide training in teaching and research skills,career planning, and leadership in order to address faculty shortages in dental schools and under representation of minority faculty.• Provide resident and faculty training in cultural and linguistic competency.• Develop and conduct a collaborative inter professional education project with a Pediatric Medicine department, a nursing school, and other health professions' education programs.• Provide faculty and residents with financial support to pursue a master's degree in public health; and • Provide support and assistance for dental practitioners desiring to explore a transition into the educational environment.

Short-answer questions and formula scoring separately enhance dental student academic performance

In this study, numerical course scores of second-year dental students in four successive classes in an oral and maxillofacial pathology course were compared. While the course content and teaching methods were essentially unchanged throughout the four years, two modest departures from the sole use of multiple-choice format questions were made in the assessment of student achievements. The modifications consisted of creating a more challenging examination procedure through the inclusion of un-cued short-answer format questions and the institution of correction-for-guessing scoring on multiple-choice examinations. Academically, the students in the four classes were comparable, as indicated by their respective numerical course score distributions in a prerequisite general pathology course in which the course content was unchanged, and all multiple-choice format questions were used to assess student academic achievements. This four-year study demonstrated that two qualitative changes in the educational environment-utilization of un-cued short-answer questions and correction for guessing scoring of multiple-choice questions-separately resulted in significant improvements in student course scores. Our results support the notion that, without any changes in curricular content or emphasis, combinations of qualitative changes in the assessment procedures alter student behavior and, as a consequence, appreciably improve their academic achievements.

Validating the standard for the National Board Dental Examination Part II

As part of the overall exam validation process, the Joint Commission on National Dental Examinations periodically reviews and validates the pass/fail standard for the National Board Dental Examination (NBDE), Parts I and II. The most recent standard-setting activities for NBDE Part II used the Objective Standard Setting method. This report describes the process used to set the pass/fail standard for the 2009 exam. The failure rate on the NBDE Part II increased from 5.3 percent in 2008 to 13.7 percent in 2009 and then decreased to 10 percent in 2010. This article describes the Objective Standard Setting method and presents the estimated probabilities of classification errors based on the beta binomial mathematical model. The results show that the probability of correct classifications of candidate performance is very high (0.97) and that probabilities of false negative and false positive errors are very small (.03 and <0.001, respectively). The low probability of classification errors supports the conclusion that the pass/fail score on the NBDE Part II is a valid guide for making decisions about candidates for dental licensure.

Short-answer examinations improve student performance in an oral and maxillofacial pathology course

The effect of examination question format on student performance was assessed by investigating three academically comparable second-year dental school classes in an oral and maxillofacial pathology course. One class was given examinations with all multiple-choice questions, one class was given examinations with all short-answer questions, and one class was given examinations with half multiple-choice questions and half short-answer questions. The class given examinations with half short-answer questions along with half multiple-choice questions had a significantly higher average score and grade category distribution (80-100 percent, 70-79 percent, <70 percent) than the class given examinations with all multiple-choice questions. When students in these two classes were divided into three academic ability groups based on the student's score in a prerequisite general pathology course, the class given examinations with half short-answer questions and half multiple-choice questions in the oral and maxillofacial pathology course had significantly higher scores and grade category distributions in all three ability groups. The average score and grade category distribution in the class given examinations with all short-answer questions in the oral and maxillofacial pathology course were not significantly different from the average score and grade category distribution in the class given examinations with half short-answer and half multiple-choice questions. Our interpretation of these results is that the utilization of examinations containing short-answer questions created a more challenging learning environment that motivated students to adopt more effective study regimens.

Prospective implementation of correction for guessing in oral and maxillofacial pathology multiple-choice examinations: did student performance improve?

A standard correction for random guessing on multiple-choice examinations was implemented prospectively in an Oral and Maxillofacial Pathology course for second-year dental students. The correction was a weighted scoring formula for points awarded for correct answers, incorrect answers, and unanswered questions such that the expected gain in the multiple-choice examination score due to random guessing was zero. An equally weighted combination of four examinations using equal numbers of short-answer questions and multiple-choice questions was used for student evaluation. Scores on both types of examinations, after implementation of the correction for guessing on the multiple-choice component (academic year 2005-06), were compared with the previous year (academic year 2004-05) when correction for guessing was not used for student evaluation but was investigated retrospectively. Academically, the two classes were comparable as indicated by the grade distributions in a General Pathology course taken immediately prior to the Oral and Maxillofacial Pathology course. Agreement between scores on short-answer examinations and multiple-choice examinations was improved in the 2005-06 class compared with the 2004-05 class. Importantly, the test score means were higher on both the short-answer and multiple-choice examinations in the Oral and Maxillofacial Pathology course, and the standard deviations were significantly smaller in 2005-06 compared to 2004-05; these differences reflected an upward shift in the lower part of the grade distributions to higher grades in 2005-06. Furthermore, when students were classified by their grade in the General Pathology course, students receiving a C (numerical grade of 70-79 percent) in General Pathology had significantly improved performance in the Oral and Maxillofacial Pathology course in 2005-06, relative to 2004-05, on both short-answer and multiple-choice examinations representing an aptitude-treatment interaction. We interpret this improved performance as a response to a higher expectation imposed on the 2005-06 students by the prospective implementation of correction for guessing.

Improving resident performance assessment data: numeric precision and narrative specificity

PURPOSE

To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments.

METHOD

Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments.

RESULTS

Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%).

CONCLUSIONS

Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Educating treatment professionals about addiction science research: demographics of knowledge and belief changes

Communication of accurate, objective, and timely scientific information to treatment professionals is important--especially in the "drug abuse" and addiction field where misinformation and a lack of exposure to new information are common. The purpose of this study was to assess knowledge and belief changes that accompanied educational workshops (3 or 6 hr-long) on addiction science targeted to treatment professionals (N=1403) given in the United States and Puerto Rico between July 2000 and August 2001. Each workshop covered three main concepts: (1) terms and definitions; (2) basic neurochemistry of addiction; and (3) how new neurobiological knowledge will affect the treatment of addictions in the future. Analysis of variance was used to compare mean pretest to posttest change scores among levels of four independent variables: gender, age, occupation/position, and race/ethnicity. Workshop participants achieved a significant improvement in knowledge about addiction with younger groups achieving greater gains. Participants' beliefs shifted in the desired direction. Significant differences in belief shifts occurred among occupational and gender groups, but not among race/ethnicity or age groups. There was also a consistent change in the policy belief subscale that related to how strongly the audience members believed research on addiction was important. We conclude that addiction science education provided to treatment professionals can increase their knowledge and change their beliefs about the causes of addictions. In addition, the workshop participants form a base of constituents who are likely to support greater addiction research funding.

Do peer chart audits improve residents' performance in providing preventive care?

PURPOSE

One recommended method to evaluate residents' competence in practice-based learning and improvement is chart audit. This study determined whether residents improved in providing preventive care after a peer chart audit program was initiated.

METHOD

Residents audited 1,005 charts and scored their peers on 12 clinical preventive services. The mean total chart audit scores were compared across five time blocks of the 45-month study.

RESULTS

Residents' performance in providing preventive care initially improved significantly but declined in the last ten months. However, their performance remained significantly higher than at the beginning.

CONCLUSIONS

By auditing their peers' charts, residents improved their own performance in providing preventive care. The diffusion of innovations theory may explain the prolonged implementation phase and problems in maintaining a chart audit program.

A multimedia patient simulation for teaching and assessing endodontic diagnosis

Teaching and assessing diagnostic skills are difficult due to relatively small numbers of total clinical experiences and a shortage of clinical faculty. Patient simulations could help teach and assess diagnosis by displaying a well-defined diagnostic task, then providing informative feedback and opportunities for repetition and correction of errors. This report describes the development and initial evaluation of SimEndo I, a multimedia patient simulation program that could be used for teaching or assessing endodontic diagnosis. Students interact with a graphical interface that has four pull-down menus and related submenus. In response to student requests, the program presents patient information. Scoring is based on diagnosis of each case by endodontists. Pilot testing with seventy-four junior dental students identified numerous needed improvements to the user interface program. A multi-school field test of the interface program using three patient cases addressed three research questions: 1) How did the field test students evaluate SimEndo I? Overall mean evaluation was 8.1 on a 0 to 10 scale; 2) How many cases are needed to generate a reproducible diagnostic proficiency score for an individual student using the Rimoldi scoring procedure? Mean diagnostic proficiency scores by case ranged from .27 to .40 on a 0 to 1 scale; five cases would produce a score with a 0.80 reliability coefficient; and 3) Did students accurately diagnose each case? Mean correct diagnosis scores by case ranged from .54 to .78 on a 0 to 1 scale. We conclude that multimedia patient simulations offer a promising alternative for teaching and assessing student diagnostic skills.

Effectiveness of addiction science presentations to treatment professionals, using a modified Solomon study design

OBJECTIVES

Knowledge of addiction research findings is critical for healthcare professionals who treat addicted patients. However, there is little information available about the instructional effectiveness of lecture-slide presentations in changing knowledge vs. beliefs of such professionals.

DESIGN

A modified Solomon four-group experimental design was used to assess the instructional effectiveness (knowledge gain vs. belief changes) of three-hour addiction science workshops presented to health-care professionals by neurobiologically-trained academic researchers. Effectiveness of the workshops was assessed by a 28-item questionnaire on participant versus control group knowledge/beliefs on addiction. Six-month follow-up questionnaires measured "retention" of knowledge and belief changes.

RESULTS

The workshop participants showed significant knowledge gain and belief changes, whereas the two control groups showed no change in knowledge or beliefs. After six months, knowledge gains decreased, but were still higher than pre-test scores. In contrast, belief changes on three subscales persisted over six months in 40 to 52 percent of the subjects.

CONCLUSIONS

These results illustrate a successful continuing education model by which academic researchers who are skilled teachers present a three-hour lecture-slide workshop with extensive question-and-answer sessions on addictions. We conclude that motivated health-care professionals can experience important knowledge gains and belief changes by participating in such workshops. In contrast to the transient retention of knowledge, belief changes persisted surprisingly well for at least six months in about half the subjects. These results suggest that long-term changes in the professional orientation of these health-care workers are possible.

Third year medical student attitudes toward learning urology

PURPOSE

We studied certain research questions, including the learning environments in which third year medical students perceive that they acquire urological knowledge and skills, and whether medical students interested in urology as a career have different perceived learning needs than those interested in other specialties.

MATERIALS AND METHODS

A survey instrument was pilot tested and revised. The instrument elicited student perceptions of how they best learned urological diagnosis and skills. Student attitudes toward the third year urology rotation and career motivation toward urology were assessed. Consecutive students were surveyed after completing the third year urology rotation.

RESULTS

Most students perceived that they learned to manage most urological problems by seeing patients in outpatient clinics and they learned to perform physical examination and urinalysis interpretation by seeing patients. The overall usefulness of various learning environments was highest for seeing patients in clinic, followed by resident teaching, following inpatients, independent reading, watching open surgery, formal conferences, watching endoscopic surgery and routine menial work. Students interested in urology as a career choice were equally motivated by seeing patients in clinic, the subject matter and seeing surgery.

CONCLUSIONS

Third year medical students perceive that the most important urological learning environment is outpatient evaluation of patients. The urological learning needs of third year medical students are not different in those interested and not interested in urology as a career.

Identification of high-risk residents

BACKGROUND

Identification of high-risk residents allows remediation and support for administrative action when necessary. This study characterizes differences in documentation of marginally performing residents in a general surgery residency.

METHODS

High-risk residents were identified by the former program director. Twenty-four of one hundred fifteen residents over a 10-year period had one to four problematic areas: cognitive, synthetic, family/health, and interpersonal skills. Outcomes included finished (18), voluntary withdrawal (1), and involuntary withdrawal (5). A case-control study matching controls to cases by date of entry into the training program was used. Records were reviewed for demographics, preentry qualifications, American Board of Surgery In-Training Exam (ABSITE) scores, letters of complaint or praise, events of counseling, and monthly ratings. The records of 48 residents were reviewed. Ward evaluations were on eight categories with a 5-point Leikert scale (3-unacceptable to 7-outstanding). The evaluation score assigns points only to low ratings. High scores represent progressively poorer performance. A Wilcoxon signed ranks test was used to compare the cases and controls for continuous variables. The McNemar test was used in comparisons of categorical data with binary outcomes. Exact P values are reported.

RESULTS

Objective data were similar for both groups. Study residents tended to score higher on monthly evaluations at Year 2 and by Year 3 this achieved significance (0.026). Study residents were more likely to have negative faculty letters (0.016) and events of counseling by a faculty member (0.017) and the program director (0.005).

CONCLUSIONS

Identification of residents at risk should begin as early as possible during training. A combination of faculty evaluations and evidence of letters of counseling can detect high-risk residents. Programs may use such indicators to support decisions regarding remedial work or administrative action.

Does a health maintenance curriculum for pediatric residents improve performance?

OBJECTIVE

To assess the knowledge and skills of residents in health maintenance and to measure the effectiveness of a new curriculum.

DESIGN

Longitudinal cohort study of first- and second-year pediatric residents over 2 academic years (AY). Residents in AY 1995-1996 (n = 32) comprised a control group. Residents in AY 1996-1997 (n = 36) served as a study group who completed a structured 12-week program of reading materials, small group discussions, and case scenarios.

METHODS

A needs assessment identified 7 topic areas in health maintenance: health screening, immunizations, nutrition, development, behavior, injury prevention, and dental health. Control and study residents' cognitive knowledge, chart documentation, and clinical performance were assessed using a 50-item multiple choice examination, medical records audit, and standardized patient evaluations.

RESULTS

Mean examination scores, chart documentation, and clinical performance did not differ significantly between first- and second-year control residents. In the study group, mean examination scores of first-year residents were not significantly better than controls. Second-year residents significantly improved their knowledge about health screening, nutrition, and dental health compared with controls but showed no improvement in other areas. Medical record documentation increased significantly among study residents compared with control residents. Clinically, first-year residents significantly improved in the task of health screening (relative risk [RR] 6.33) and markedly improved (36.8% vs 66.7%) their injury prevention counseling compared with controls. Second-year residents showed significant increases in health screening (RR 6.09) and taking a nutritional history (RR 1. 44) compared with controls. Assessment of the curriculum by residents showed a high degree of satisfaction with the program.

CONCLUSIONS

Pediatric residents who had clinical experience and completed the curriculum demonstrated significant improvements in their medical record documentation and selective gains in knowledge and clinical performance of health maintenance compared with residents with clinical experience only. A structured curriculum in health maintenance and a multifaceted assessment system can identify and enhance the skills of pediatric residents.

Evaluation of a Role-Play Learning Exercise in an Ambulatory Clinic Setting

Learning in ambulatory settings can be aided by teaching activities which do not slow the pace of the clinic. In this study, simulated-parent role-play scripts were developed for use with students prior to seeing actual patients. During the learning exercise, a faculty member role-played various parents, a medical student role-played the physician while another student observed. Students were randomly assigned to either the Treatment Group (participated in role-play exercise) or Control Group (did not participate). Sixteen Treatment Group students completed approximately 20 role-play cases on the first day of the Pediatrics clerkship. Both Treatment and Control Group students were then individually assessed one to two days later using two role-play cases by an examiner blinded to the students' experimental status. Student performance was scored on three criteria: history taking, differential diagnosis, and correct diagnosis. The Treatment Group achieved higher scores than the Control Group for history taking, but not for differential diagnosis or correct diagnosis. Role-play learning exercises can be used to improve student history taking in an ambulatory clinic prior to seeing actual patients. Potential use of role-play cases as a screening exam to exempt some students from the role-play learning exercise is discussed.

Education of nonscientists about new alcohol research: results of two types of presentations plus 6-month follow-up

Education of nonscientists by scientists is assumed to be beneficial for enhancing public understanding of the research process and increasing public excitement about science. However, evaluation of audience response to receiving such information has rarely been performed. In particular, the effectiveness of communicating new research on alcohol abuse and alcohol dependence has never been evaluated. Evaluation data in the present study show significant knowledge transfer, belief changes, and participant reports of possible behavioral changes in targeted audiences. These occur when alcohol researchers present basic neuropharmacological concepts and new neurobiological research to audiences consisting primarily of chemical dependency counselors, social workers, criminal justice workers, physicians, nurses, family, clergy, and others interested in alcohol-related problems (defined as "clinicians" and the "reachable public"). Together, these results suggest that it is possible to change the beliefs, knowledge, and behavior of chemical dependency clinicians and the reachable public about alcoholism, its causes, and its treatment.

Resident education outcomes and perception of a seminar format to teach female pelvic anatomy and procedures to correct pelvic prolapse

Urologic residents need to learn basic and applied knowledge of female pelvic anatomy, a subject rarely taught beyond preclinical undergraduate medical education. This study tests the hypothesis that urologic resident knowledge of female pelvic anatomy and prolapse may be enhanced with a seminar. Twenty residents attended a one day seminar combining didactics and a cadaveric dissection related to female pelvic anatomy and female pelvic prolapse conditions. Resident knowledge was measured with a multiple choice test administered in a pretest-posttest experimental design. Resident attitudes toward the seminar were assessed by a 20 item survey using a strongly disagree to strongly agree scale. Pretest and posttest mean scores were 55% and 71% respectively, p=.0007 (Kuder-Richardson 20 coefficients were 0.7). Questionnaire responses indicated positive opinions regarding the educational value of the seminar. Urologic resident knowledge of female pelvic anatomy and pelvic prolapse conditions may be enhanced by conducting a one day teaching seminar.

Using narratives to evaluate psychiatry residents' competence

The authors propose a new way to assess the clinical competence of psychiatric residents. Faculty eualuators would be trained by using narrative descriptions of appropriate and inappropriate practice in various clinical settings. The training has the potential to provide more realistic data about the clinical abilities of residents than current methods.

A cost-effectiveness analysis of three staffing models for the delivery of low-risk prenatal care

BACKGROUND

Health care costs are increasing at more than twice the rate of inflation, thus, public officials are seeking safe and economic methods to deliver quality prenatal care to poor pregnant women. This study was undertaken to determine the relationship between the cost and effectiveness of three prenatal clinic staffing models: physician based, mixed staffing, and clinical nurse specialist with physicians available for consultation.

METHODS

Maternal and neonatal physiological outcome data were obtained from the hospital clinical records of 156 women attending these clinics. The women were then interviewed concerning their satisfaction with their prenatal care clinic. The financial officer from each clinic provided data on the clinic staffing costs and hours of service.

RESULTS

There were no differences in outcomes for the maternal-neonatal physiological variables, although newborn admission to the Neonatal Intensive Care Unit (NICU) approached significance among the clinics. The clinic staffed by clinical nurse specialists had the greatest client satisfaction and the lowest cost per visit.

CONCLUSIONS

The use of clinical nurse specialists might substantially reduce the cost of providing prenatal care while maintaining quality, and might thereby save valuable resources.

Comparing learning logs and note summaries in a dental research methods course

This paper reports a study that examines how two types of writing influenced dental students' performance in critical thinking and recall in a graduate research methods course. Forty-five students in this five-week, lecture-based course were stratified by dental specialty and randomly assigned to one of two groups. The treatment group kept learning logs, expressing personal understanding of lecture material; a second group wrote note summaries, entailing the reorganization of key lecture points. Each student was asked to write for a ten-minute interval during each class meeting. Mean scores for the log writers were higher on a post-course essay measure of critical analysis; note summarizers performed better on recall; while in the direction predicted, differences did not reach statistical significance. Note summarizers' attitude toward future use of their activity in other classrooms was more positive and they complied more highly with their activity than did the log writers. This preference for the note summary method suggests that students may find the structured nature of the summary more useful when being introduced to a dental concept.

Impact of orthognathic surgery on normal and abnormal personality dimensions: a 2-year follow-up study of 61 patients

The psychological adjustments and self-concepts of 61 orthognathic surgery patients were evaluated before surgery and at 1 month, 6 months, 1 year, and 2 years after surgery. The impact of orthognathic surgery on self-concept and personality adjustment was assessed. Significant improvements in group scores were found with the use of a repeated measures ANOVA in the following subscales of personality disturbances: general maladjustment, psychosis, neurosis, personality disorder, and personality integration. A significantly positive effect was also observed in the following subscales of self-concept: self-esteem, self-satisfaction, self-identity, physical self, family self, social self, and total self-conflict. The improved changes in psychological profile two years after orthognathic surgery are encouraging.

Telephone management curriculum for pediatric interns: a controlled trial

The purpose of this study was to evaluate the effect of a role-play telephone management curriculum on history taking and management skills. The study was a nonrandomized controlled trial carried out in the outpatient department of a primary care pediatric residency training program. Six PL-I residents (treatment group) participated in a role play curriculum during their first month of rotation through the outpatient department; seven PL-I residents (control group) received no formal instruction in telephone management. Baseline performance was measured on entry into the study using telephone calls made by a "simulated" mother using standardized scripts. Follow-up calls were made approximately 3 months later. Transcripts of these calls were rated by examiners who were unaware of group assignment using a standardized instrument that measured three aspects of performance: general history taking, specific history taking, and general management. Results were as follows: Mean posttest scores for the combined groups were: 59.4% (general history taking), 75.9% (specific history taking), and 77.7% (general management). By analysis of covariance, treatment group residents had significantly higher posttest scores in general history taking than control group residents (67.1% v 53.6%, P = .004). This difference occurred without a significant increase in the time spent answering the call. These data indicate that a role play curriculum results in improved performance in at least one aspect of telephone management.

Videotape review: a valuable tool for self-assessment of teaching skills

This paper describes the components of an instructor improvement program known as VISIT (Videotaping Instruction for Self-Assessment of Instructional Technique). Through the use of videotape recordings of classroom, laboratory or clinical teaching and observation by trained observers, faculty can pinpoint instructional strengths and problem areas, plan refinements and evaluate the impact of these refinements. Ten years of experience with VISIT indicates that post-lecture consultation sessions between the instructor and observer are vital to the success of this activity.

Improving nutrition components in medical and dental school curriculums

This article has presented an alternative approach to the traditional, freestanding, concentrated course in nutrition. The integration of nutrition into other courses, with continuous reinforcement throughout the 4-year curriculum, may result in better learning and retention by the student. However, the lack of an accurate curriculum representation presents a problem in selecting topics that integrate with other courses. The unobtrusive approach to nutrition curriculum improvement involves three key points: development of the Nutrition Curriculum Guide, which provides detailed information on what is currently being taught, comparison of what is actually being taught with a set of ideal objectives in order to identify redundancies or omissions, and orchestrating the efforts of faculty in courses throughout the 4-year curriculum to help students integrate nutrition-related topics from the various disciplines. When one makes practical application of the unobtrusive approach, the most difficult problem is defining which interdisciplinary topics are currently being taught. An education specialist and the Tracer Method are important resources for one who is seeking to ameliorate the problem.

The reliability and utility of clinical competency assessments for drug monograph preparation by pharmacy students

This educational research project evaluated the utility and reliability of an evaluation system designed to assess the clinical competency of pharmacy students in the preparation and presentation of a comparative drug monograph. The educational program consisted of a mini-course to review drug information resources and skills and a drug information clinical clerkship. The evaluation system for student performance (competency) included 15 criteria (competency areas), a set of descriptors for each criterion, and a scaling mechanism. This competency assessment system for monograph services quantified the overall level of competency and identified specific strong and weak competency areas. The reliability of the system was good in general (standardized alpha-coefficient = 0.701), over time, and among several preceptors. Students indicated that the system was fair and provided them with feedback. Clinical competency assessment should be structured to minimize subjectivity and must be tested for reliability; then, such evaluations can become more consistent and equitable.

Development of core competencies in clinical nutrition

A nationwide survey of core competencies of nutrition knowledge and skills that primary care physicians should know and/or be able to do has been completed. Nutrition competencies were synthesized from a review of clinical nutrition practices as reported in the medical literature and by medical school faculty group discussions. A nutrition competency questionnaire was sent to 445 practicing physicians and to 752 department chairpersons in every US medical school in the disciplines of family practice, medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery. The overall response rate for practitioners and faculty combined was 46.03%. Of the 55 nutrition competencies, 28 items received very strong support (greater than 90% of respondents marked agree or completely agree). Twenty-two competencies received strong support (80 to 89% of respondents marked agree or completely agree), and five items received mixed support (40 to 79% of respondents marked agree or completely agree). Statistical comparisons of the item responses between practitioners and faculty were significantly different (p less than 0.05) on 19 (35%) of the items. It is anticipated that the 50 nutrition competencies that have received strong or very strong agreement among the 551 physicians representing primary care disciplines across the US will serve as guidelines for continued development of medical school curriculum and continuing medical education in clinical nutrition.

Evaluating performance in writing drug information consultations

A form for evaluating written drug information consultations was used to measure the performance of pharmacy students in a drug information service clerkship; the reliability of the evaluation process was also assessed. Drug consultations were written by 74 clerks during a four- or five-week rotation. A total of 147 consultations were written: Nine clerks wrote one consultation, 57 clerks wrote two, and eight clerks wrote three. The consultations were evaluated on 11 criteria by 18 different preceptors using an evaluation form. Performance on each criterion was rated on a four-level scale. Clerk improvement over time, reliability of the instrument, and inter-rater reliability for preceptors were analyzed statistically. The overall mean score for all consultations was 84 7%. Some improvement was observed between the first and second consultations completed by clerks, but the difference was not significant. Reliability of the evaluation process was good (coefficient of alpha = 0.666). There as no significant difference among preceptors. The authors concluded that the written drug information consultation evaluation form was a valid and reliable method for measuring individual and program competence.

A description and four-year analysis of a clinical clerkship evaluation system

Major issues related to evaluating student performance in clinical clerkships are reviewed, and the development of a clerkship evaluation system which attempts to deal with each issue is described. Three evaluation methods are utilized: multiple choice examination, oral examination, and ward ratings. Scores from each evaluation method for the classes of 1977-80 were analyzed to determine dependability and validity. Results indicate that the multiple choice and oral exam scores were highly dependable each year; however, the ward ratings were not dependable during the first year of use. Over the four-year period, the ward rating criteria were kept the same, annual feedback to individual faculty was provided, and residents were added as raters. The cumulative effect appears to have been a pronounced improvement in the dependability of the ward rating scores. As a result, ward rating scores now receive greater emphasis in evaluating student clerkship performance.

Assessing the dependability of faculty observations using the generalizability coefficient

Evaluating performance of dental students is often accomplished by faculty observation. Ratings are assigned and later translated into scores which form the basis for important decisions such as admission to dental school, course grades, or admission to graduate programs. Situational factors such as variability among faculty observers or day-to-day fluctuations in a student's performance may influence observational ratings. These situational factors can introduce error into the ratings and thereby reduce the dependability of decisions based upon the ratings. Generalizability coefficients are a recent development in educational measurement which can aid the dental educator in evaluating the degree of error present in ratings to be used for a specific purpose. In this study, dental school applicant ratings from an admissions interview are used to demonstrate the usefulness of generalizability coefficients in assessing the impact of interviewer disagreements on decisions based on these scores.