A comparison of behaviour on simulated patients and patient management problems

A Comparison of Checklist and Domain-Based Ratings in the Assessment of Objective Structured Clinical Examination (OSCE) Performance

INTRODUCTION

The Objective Structured Clinical Examination (OSCE) is the mainstay of clinical assessment in the final-year undergraduate Family Medicine clerkship at King Faisal Specialist Hospital and Research Centre (KFSHRC). The gold standard for OSCE assessment is the checklist rating, completed by physician examiners. Numerous studies have suggested that global or domain-based OSCE ratings may be a better indicator of competence than checklist ratings. The aim of this study was to examine the utility of domain-based OSCE ratings in the context of final-year, undergraduate, Family Medicine OSCE examinations in Riyadh, Saudi Arabia. This is akin to an exercise in quality improvement, as we continuously look for ways to improve our OSCE assessment processes.

METHODS

This study utilised a quantitative methodology. Three final year OSCE exams were chosen. Physicians rated each student using a checklist score and using a more holistic domain-based score. Physician checklist scores and physician domain-based scores were then compared, and correlation was assessed. We also looked at the internal consistency of the scoring methods.

RESULTS

A significant correlation was found between checklist and domain-based scores by physicians for all exams (r=0.858, p<0.01), with a good internal consistency for these methodologies for all exams.

CONCLUSION

The results demonstrate that both checklist and domain-based scores offer some benefit to the assessment, with a similar internal consistency and strong correlation. Domain-based ratings should be utilised for softer skills that are not easily assessed by checklists. There is clearly a need to rethink our OSCE assessment. The assessment should combine checklist and domain-based physician scores. As trainees become more experienced, checklist OSCE may penalise directness and efficiency, while domain-based ratings would offer a better appraisal of competence, and have been shown to be more sensitive to the level of training and expertise. Changing the assessment methods will lead to necessary changes in the student approach to the OSCE and improve authenticity and validity.

A situated cognition model for clinical reasoning performance assessment: a narrative review

Background Clinical reasoning performance assessment is challenging because it is a complex, multi-dimensional construct. In addition, clinical reasoning performance can be impacted by contextual factors, leading to significant variation in performance. This phenomenon called context specificity has been described by social cognitive theories. Situated cognition theory, one of the social cognitive theories, posits that cognition emerges from the complex interplay of human beings with each other and the environment. It has been used as a valuable conceptual framework to explore context specificity in clinical reasoning and its assessment. We developed a conceptual model of clinical reasoning performance assessment based on situated cognition theory. In this paper, we use situated cognition theory and the conceptual model to explore how this lens alters the interpretation of articles or provides additional insights into the interactions between the assessee, patient, rater, environment, assessment method, and task. Methods We culled 17 articles from a systematic literature search of clinical reasoning performance assessment that explicitly or implicitly demonstrated a situated cognition perspective to provide an "enriched" sample with which to explore how contextual factors impact clinical reasoning performance assessment. Results We found evidence for dyadic, triadic, and quadratic interactions between different contextual factors, some of which led to dramatic changes in the assessment of clinical reasoning performance, even when knowledge requirements were not significantly different. Conclusions The analysis of the selected articles highlighted the value of a situated cognition perspective in understanding variations in clinical reasoning performance assessment. Prospective studies that evaluate the impact of modifying various contextual factors, while holding others constant, can provide deeper insights into the mechanisms by which context impacts clinical reasoning performance assessment.

The key-features approach to assess clinical decisions: validity evidence to date

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

Strategies in Comparison of Methods for Scoring Patient Managementproblems

The study was designed to determine whether alternative scoring strategies result in improved measurement properties of patient management problems (PMP). For each of 16 PMPs used in a certifying examination, thefollowingscoringsystems were developed: proficiency, efficiency, select, omit, data gathering, therapy, absolute (dichotomouspass/fail), goal-oriented, and an empiric expert score. Scores with each system were developedfor 4,590firsttime takers and compared to two external criterion measures, namely, the examinees' "clinical-competence" rating by their program director, and the examinees'membership in a "high" or "low" criterion status group. The results demonstrated high correlations between all scoring systems and high correlations with simultaneous multiple-choice (MCQ) scores. Each showed moderate correlation with the validity measures. Using regression analysis, none of the PMPs coresprovided significant unique incremental validity not provided by MCQ scores. The implications of these data in the use of PMPs in certifying or licensure examinations are discussed.

A Preliminary Study of the Impact of Case Specificity on Computer-Based Assessment of Medical Student Clinical Performance

The implications of case specificity of two computer-based clinical simulation examination cases (CBX) were examined by a classical measurement approach and by a Bayesian analysis of test characteristics. The CBXs (a surgery and an ob/gyn case) were designed by the National Board of Medical Examiners and administered to 163 University of Michigan Medical School students. The results indicate that the students performed differently on the two cases, the surgery case appearing to be more difficult. The ob/gyn case had greater sensitivity (more accuracy in passing competent students), whereas the surgery case had greater specificity (more accuracy in failing noncompetent students). The differences between the cases and evidence of case specificity raise the issue of an exam's objective and the acceptable type of classification error These results suggest that additional studies are required before widespread use of such exams can be implemented in "high stakes" situations for licensure purposes.

The Impact of Clinical Appearance on Pediatric Residents' Assessment of the Febrile Infant

In order to examine physicians' use of clinical appearance versus historical and laboratory information in medical decision making, 29 pediatric residents were shown videotapes of febrile infants and accompanying clinical histories of varying severity. The results showed that contradictory visual and historical information led to higher, but much more variable, diagnostic likelihood estimates. Severity of appearance was directly related to further diagnostic and therapeutic action but was inversely related to confidence in diagnostic likelihood estimates. Clinical appearance plays an important role in medical decision making but when coupled with contradictory historical information may pose difficulties for physicians attempting to assess diagnostic likelihood.

A Criterion-Referenced Approach to Measuring Medical Problem Solving

A criterion-referenced approach was used to examine the validity of a set of 15 Patient Management Problems (PMPs) representing a broad range of medical problems. Results of performance of 175 medical students indicated that a new problem-solving index that includes an open-ended differential diagnosis section had greater validity than the more traditional proficiency index, as the spread between the distributions of masters and nonmasters was greater for the problemsolving index. Implications for medical education and research are discussed.

Performance Assessment in Medical Education

The assessment of clinical competence is becoming increasingly complex, patient cen tered, and student driven. Traditionally, clini cal evaluation methods consisted primarily of faculty observations, oral examinations, and multiple-choice tests. Increased faculty work load, discontent with traditional methods of clinical skill assessment, and developments in the fields of psychology and education have led to the formation of new modalities, namely performance assessments. The literature per taining to the performance assessment with standardized patients is reviewed. Based on this literature, several areas for the future direction of performance assessment are pro posed, including (a) toward evidence-based locally developed assessments, (b) toward an understanding of educational outcomes and noncognitive assessment factors, and (c) toward more student-driven assessments.

An Evaluation of a Computer Simulation in the Assessment of Physician Competence

This study compares physician performance on the Computer-Aided Simulation of the Clinical Encounter (CASE) with peer ratings and performance on multiple choice questions (MCQs) and patient management problems (PMPs). CASEis a simulation of the clinical encounter where the computer plays the role of the patient and the physician elicits information by entering "natural language" questions into a computer terminal. Results indicate that all formats are equally valid, although MCQs are the most reliable methods of assessment per unit of testing time, followed by PMPs and CASE, in that order. All methods measure the same or very highly correlated aspects of competence.

Using checklists in a gross anatomy laboratory improves learning outcomes and dissection quality

Checklists have been widely used in the aviation industry ever since aircraft operations became more complex than any single pilot could reasonably remember. More recently, checklists have found their way into medicine, where cognitive function can be compromised by stress and fatigue. The use of checklists in medical education has rarely been reported, especially in the basic sciences. We explored whether the use of a checklist in the gross anatomy laboratory would improve learning outcomes, dissection quality, and students' satisfaction in the first-year Human Structure didactic block at Mayo Medical School. During the second half of a seven-week anatomy course, dissection teams were each day given a hardcopy checklist of the structures to be identified during that day's dissection. The first half of the course was considered the control, as students did not receive any checklists to utilize during dissection. The measured outcomes were scored on four practice practical examinations and four dissection quality assessments, two each from the first half (control) and second half of the course. A student satisfaction survey was distributed at the end of the course. Examination and dissection scores were analyzed for correlations between practice practical examination score and checklist use. Our data suggest that a daily hardcopy list of anatomical structures for active use in the gross anatomy laboratory increases practice practical examination scores and dissection quality. Students recommend the use of these checklists in future anatomy courses.

Fifty years of medical education research: waves of migration

CONTEXT

Medical education research has been an academic pursuit for over 50 years, tracing its roots back to the Office of Medical Education at the State University of New York at Buffalo, New York, with George Miller. As the field has matured, the nature of the questions posed and the disciplinary bases of its practitioners have evolved.

METHODS

I identify three chronological 'generations' of academics who have contributed to the field, at intervals of roughly 10-15 years.

RESULTS

Members of the first generation came from diverse and unrelated academic backgrounds and essentially learned their craft on the job. A second generation, emerging in the 1980s and 1990s, consisted of individuals with PhD-level training in relevant fields such as psychology, psychometrics and sociology, who actively chose a career in health sciences education, often during graduate work. These individuals brought a strong disciplinary orientation to their research. Finally, the proliferation of graduate programmes in medical education means that we are now seeing the evolution of a new type of academic, often a health professional, whose only discipline is medical education.

CONCLUSIONS

I propose that we should strike a balance between seeking to create a separate specialty of medical education and continuing to actively recruit from other academic disciplines. I believe that the strong disciplinary roots of these individuals are a critical element in the continuing growth and progress of medical education research.

Towards effective evaluation and reform in medical education: a cognitive and learning sciences perspective

Health professions education is dealing with major transformations in light of the changing nature of the health care delivery system, including the use of technology for "just in time" delivery of care, evidence-based practice, personalized medical care and learning, as health professionals strive to integrate biomedical advances and clinical practice. This has forced the medical education community to reassess the current teaching and learning practices and more importantly, the evaluation of the medical education process. There have been recent advances in cognitive and learning sciences theories, some of which can inform medical educators about best teaching and learning practices and their impact on the evaluation process. An understanding of these theories provides a sound rationale for choosing specific instructional strategies and choosing evaluation measures that assess the curricular objectives. The review begins with an overview of evaluation and assessment in education, followed by an overview of major theories from the cognitive and learning sciences. Next, the role of cognitive and learning sciences theories in informing the process of medical education evaluation is discussed, including its impact on student learning, performance and professional competence, as well as recommendations for reform of medical curricula based on such theories. The paper continues with the elaboration of current trends in health sciences education, particularly medical education, and available evidence for the impact on student learning and performance as well as areas where more research is needed.

Obstetricians' choice of cesarean delivery in ambiguous cases: is it influenced by risk attitude or fear of complaints and litigation?

OBJECTIVE

The aim of this study was to test the hypothesis that obstetricians' choice of delivery method is influenced by their risk attitude and perceived risk of complaints and malpractice litigation.

STUDY DESIGN

The choice of delivery method in ambiguous cases was studied in a nationwide survey of Norwegian obstetricians (n = 716; response rate, 71%) using clinical scenarios. The risk attitude was measured by 6 items from the Jackson Personality Inventory-Revised.

RESULTS

The proportion of obstetricians consenting to the cesarean request varied both within and across the scenarios. The perceived risk of complaints and malpractice litigation was a clear determinant of obstetricians' choice of cesarean in all of the clinical scenarios, whereas no impact was observed for risk attitude.

CONCLUSION

Obstetricians' judgments about cesarean request in ambiguous clinical cases vary considerably. Perceived risk of complaints and litigation is associated with compliance with the requested cesarean.

The effect of question format and task difficulty on reasoning strategies and diagnostic performance in Internal Medicine residents

BACKGROUND

Previous studies have suggested an association between reasoning strategies and diagnostic success, but the influence on this relationship of variables such as question format and task difficulty, has not been studied. Our objective was to study the association between question format, task difficulty, reasoning strategies and diagnostic success.

METHODS

Study participants were 13 Internal Medicine residents at the University of Calgary. Each was given eight problem-solving questions in four clinical presentations and were randomized to groups that differed only in the question format, such that a question presented as short answer (SA) to the first group was presented as extended matching (EM) to the second group. There were equal numbers of SA/EM questions and straightforward/difficult tasks. Participants performed think-aloud during diagnostic reasoning. Data were analyzed using multiple logistic regression.

RESULTS

Question format was associated with reasoning strategies; hypothetico-deductive reasoning being used more frequently on EM questions and scheme-inductive reasoning on SA questions. For SA question, non-analytic reasoning alone was used more frequently to answer straightforward cases than difficult cases, whereas for EM questions no such association was observed. EM format and straightforward task increased the odds of diagnostic success, whereas hypothetico-deductive reasoning was associated with reduced odds of success.

CONCLUSIONS

Question format and task difficulty both influence diagnostic reasoning strategies and studies that examine the effect of reasoning strategies on diagnostic success should control for these effects. Further studies are needed to investigate the effect of reasoning strategies on performance of different groups of learners.

Can concept sorting provide a reliable, valid and sensitive measure of medical knowledge structure?

CONTEXT

Evolution from novice to expert is associated with the development of expert-type knowledge structure. The objectives of this study were to examine reliability and validity of concept sorting (ConSort) as a measure of static knowledge structure and to determine the relationship between concepts in static knowledge structure and concepts used during diagnostic reasoning.

METHOD

ConSort was used to identify static knowledge concepts and analysis of think-aloud protocols was used to identify dynamic knowledge concepts (used during diagnostic reasoning). Intra- and inter-rater reliability, and correlation across cases, were evaluated. Construct validity was evaluated by comparing proportions of nephrologists and students with expert-type knowledge structure. Sensitivity and specificity of static knowledge concepts as a predictor of dynamic knowledge concepts were estimated.

RESULTS

Thirteen first-year medical students and 19 nephrologists participated. Intra- and inter-rater agreement for determination of static knowledge concepts were 1.0 and 0.90, respectively. Reliability across cases was 0.45. The proportions of nephrologists and students identified as having expert-type knowledge structure were 82.9% and 55.8%, respectively (p=0.001). Sensitivity and specificity of ConSort((c)) in predicting concepts that were used during diagnostic reasoning were 96.8% and 27.8% for nephrologists and 87.2% and 55.1% for students.

CONCLUSIONS

ConSort is a reliable, valid and sensitive tool for studying static knowledge structure. The applicability of tools that evaluate static knowledge structure should be explored as an addition to existing tools that evaluate dynamic tasks such as diagnostic reasoning.

Expert-type knowledge structure in medical students is associated with increased odds of diagnostic success

BACKGROUND

The relation between knowledge structure and diagnostic performance is unclear. Similarly, variables affecting knowledge structure are poorly understood.

PURPOSE

The 1st objective was to examine the relation between concepts in knowledge structure and diagnostic performance. The 2nd objective was to examine the relation between the use of diagnostic schemes by small-group preceptors and knowledge structure of medical students.

METHODS

This was a cross-sectional study of 1st-year medical students in 4 clinical presentations: hyponatremia, hyperkalemia, metabolic acidosis, and metabolic alkalosis. The 1st dependent variable was diagnostic success with the number of expert-type concepts in knowledge structure (determined by concept sorting), diagnostic scheme use by preceptors, and clinical presentation as independent variables. The 2nd dependent variable was the number of expert-type concepts in knowledge structure with diagnostic scheme use by preceptors and clinical presentation as independent variables. Data were analyzed using multiple logistic and linear regression.

RESULTS

Thirty 1st-year medical students participated. After adjusting for clinical presentation and scheme use by preceptors, the number of expert-type concepts in knowledge structure was associated with increased odds of diagnostic success (odds ratio 1.18 [1.03, 1.35], p = .016). After adjustment for clinical presentation, scheme use by preceptors was associated with increased number of expert-type concepts in knowledge structure (2.22 vs. 1.86, p = .01, d = 0.23).

CONCLUSIONS

The number of expert-type concepts in knowledge structure is associated with increased odds of diagnostic success. Scheme use by small-group preceptors is associated with an increased number of expert-type concepts in knowledge structure.

Clinical simulations: development and validation

Simulations used as an educational strategy can mimic clinical reality bringing real life activity into the learning environment. This paper presents a conceptual approach to simulation development and validation that is applied to develop assessment simulations for both childbirth and triage situations. A process-based method of presenting information to the learner in the assessment phase is incorporated in simulations developed from actual clinical cases. The uncertain conditions of assessment are recognized as mirroring practice reality and validation of this construct and that of expertise are considered. Use of these simulations has the potential to provide learners with the opportunity to experience dimensions of simulated practice reality and educators with an experiential strategy that can be used to prepare students and also clinicians who are unfamiliar with new clinical practice areas.

Threats from patients and their effects on medical decision making: a cross-sectional, randomised trial

BACKGROUND

Negative experiences are not uncommon among doctors in Norway. Our aim was to find out about the various types of negative reactions (eg, complaints, negative exposure to the media, financial claims, and notification to the police) received by physicians from patients or relatives in response to treatment, to identify their cause, and to study their effects on subsequent clinical decisions.

METHODS

We posted questionnaires about negative reactions of patients to a random sample (n=1260) of Norwegian doctors. Each doctor was additionally sent five written case simulations and asked to choose one of several proposed clinical strategies. Half (630) the physicians received cases containing threats from the patient or their relatives.

FINDINGS

988 (78%) physicians returned the questionnaire, 463 (47%) of whom reported negative experiences. Such experiences were reported more frequently by men (357 [51%]) and family physicians (157 [58%]) than by other participants. Negative experiences did not affect choice of strategy for case simulations. For the first case, chest pain, 217 (44%) physicians presented with a threat chose a defensive strategy compared with 145 (30%) of those who were not (difference 14%; 95% CI 8-20). For the second case, a headache case, the corresponding numbers were 278 (57%) and 118 (25%) (32%; 26-38). Physician age, sex, specialty, or experience of negative reactions of patients did not alter the effect of threats received during our study.

INTERPRETATION

Negative experiences do not affect subsequent decision making. However, doctors do comply with wishes from patients or relatives when presented with direct threats.

Dentists' judgment strategies on prophylactic removal of mandibular third molars

The number of molars selected for prophylactic removal varies widely among general dental practitioners and oral surgeons. To understand the basis for such variations, we investigated two hypotheses: (1) Individual judgment strategies will differ concerning the use of cues (items of information), and (2) few dentists will integrate the cues according to evidence in the literature. To analyze 30 general dental practitioners' (GDPs) and 10 oral surgeons' use of cues in the judgment preceding the treatment decision, we used the Brunwik's lens as a conceptual model. The cues were the patient's age, and the angular position and the degree of impaction of the molar. The clinical situation was simulated by written case descriptions. The proportion of variation explained by the cues and their combinations (total model) varied between 61% and 100% and between 4% and 76% as main effects. Two GDPs and one oral surgeon integrated the cues additively, i.e., any of the cues is independent of the other cues in the judgment. In general, the dentists integrated the cues interactively, i.e., the impact of one cue depends on the levels of some other cues. Even though most variations in judgments were accounted for by the cues, the dentists did not integrate the cues according to evidence in the literature and lacked insight into their decision-making thought processes.

The erythrocyte sedimentation rate in general practice: clinical assessment based on case histories

Thirteen case histories were mailed to 273 general practitioners to study the clinical assessment of the ESR. Participants were asked to state their reference limits for the ESR, as well as action values for the ESR in several clinical situations typical of primary care. The action value should represent the minimal ESR change from a given value necessary to initiate some kind of action towards the patient. The response rate was 76%. In most case histories, half the general practitioners reacted on an ESR change of 10 mm h-1 or less, which is usually due to analytical and biological variation, thus underlining the need for good analytical quality. In general the response was of the same magnitude irrespective of type of case history, i.e. whether the ESR was used in case finding, in diagnosis, or in follow-up. Estimation of the reference limit displayed considerable variation, and knowledge of a previous 'normal' ESR was of minor clinical importance. Furthermore, we found substantial variation regarding the change in ESR necessary to take action in different clinical situations. In principle, for many general practitioners the action value increased as the given ESR increased whereas others reacted on a constant change in ESR, or the change necessary to take action depended on the clinical situation. We conclude that both the different assessments as to the clinical significance of the ESR, and the unawareness of the significance of analytical and biological variation indicate that guidelines for rational use are needed.

Emergency thoracotomy: comparison of medical student, resident, and faculty performances on written, computer, and animal-model assessments

OBJECTIVE

In clinical practice, thoracotomy and other critical emergency procedures are rarely required. Consequently, medical students and residents have difficulty acquiring procedural competency in these critical procedures. The authors developed objective written, computer, and animal-model assessments of thoracotomy procedural competency to permit comparison of the reliability and validity of these three procedural assessment modalities.

METHODS

Thoracotomy procedural competency was evaluated for 18 persons at three levels of training (medical student, resident, faculty), using written, computer, and animal-model assessments. A prospective, sequential assessment design was used, with the examinees serving as their own controls. Procedural competency was defined in terms of performance time (animal time scale) and performance accuracy (written accuracy, computer accuracy, and animal accuracy scales) for three thoracotomy procedures (opening the chest, pericardiotomy, and aortic cross-clamping). Level of training was the independent variable, and procedural competency scores were the outcome measures. Confounding variables included previous thoracotomy and computer experience.

RESULTS

Computer and animal-model assessments produced reliable results (Chronbach's alpha > 0.50). The animal time scale and computer accuracy scale best reflected the expected skill differences among levels of physician training, providing support for construct validity. In contrast, written and animal accuracy scale scores did not significantly differ by level of physician training. Moreover, previous thoracotomy experience (i.e., number of procedures previously performed) was not a significant predictor of procedural competency.

CONCLUSIONS

This study demonstrates that critical emergency medicine procedures can be evaluated reliably and validly using computer simulation and animal-model assessments. Neither previous thoracotomy experience nor knowledge of procedure content adequately predicts thoracotomy competency.

A randomized comparison of alternative formats for clinical simulations

Computer-based clinical simulations for medical education vary widely in structure and format, yet few studies have examined which formats are optimal for particular educational settings. This study is a randomized comparison of the same simulated case in three formats: a "pedagogic" format offering explicit educational support, a "high-fidelity" format attempting to model clinical reasoning in the real world, and a "problem-solving" format that requires students to express specific diagnostic hypotheses. Data were collected from rising third-year medical students using a posttest, attitudinal questionnaire, students' write-ups of the case, and log files of students' progress through the simulation. Student performances on all measures differed significantly by format. In general, students using the pedagogic format were more proficient but less efficient. They acquired more information but were able to do proportionately less with it. The results suggest that the format of computer-based simulations is an important educational variable.

Clinical assessment of haemoglobin values by general practitioners related to analytical and biological variation

Twelve case stories were mailed to 273 general practitioners to study the clinical assessment of haemoglobin values. For each case the general practitioner was asked to fill in the haemoglobin value representing the minimal change from a given value considered necessary to take action. haemoglobin change corresponding to the median haemoglobin value stated was transformed to a so-called 'medically useful coefficient of variation' in order to relate clinical demands to analytical and biological variation. The average medically useful coefficient was calculated to 3.9% (range 2.3-7.8%). We found that general practitioners assess haemoglobin values rather uniformly, although their judgement varies substantially with the clinical situation; they are not fully aware of the consequences of analytical and biological variation, and the advantage of knowing a previous haemoglobin value is not recognized. The analytical imprecision of haemometers based on clinical demands should be 2.8%, and the analytical quality should be the same in primary and secondary care.

Analysis of videotaped consultations to certify competence

The assessment of videotaped genuine consultations in order to provide feedback to the learner is now a well accepted part of postgraduate training for general practice. This article reports on a pilot study which investigates the adaptation of videotape assessment of genuine patient consultations as an alternative to the present simulated patient interview segments of the examination for Fellowship of the Royal Australian College of General Practitioners. In particular, the study addresses: the feasibility and cost of collecting and assessing a suitable range of videotaped consultations using portable videocamera equipment and remote examiners; the interobserver reliability of four independent examiners; and the correlation between scores on two different rating scales. The results demonstrate that the assessment of videotaped genuine consultations has the potential to become a means of end-point assessment of clinical competence. The problem of how to collect videotapes of a suitable range and number of consultations to permit a candidate to demonstrate proficiency needs to be addressed and will be followed up in a trial of the method during 1990.

Audit of decision-making regarding female genitourinary infections in outpatient practice

Patient data, diagnosis, work-up measures, and prescriptions were collected from 63 consultations by general practitioners (GPs) to study day-to-day management of women with complaints suggestive of genitourinary (GU) infections. The collected patient data were thereafter presented to a panel of 9 specialist physicians for their individual recommendations. On drug prescriptions, the panel supported 81% of the decisions, did not support 14%, and remained undecided on 5% of the cases. In all but one of the unsupported cases, the GP had prescribed antibiotics. On diagnoses, the panel supported 62% of the decisions, did not support 13%, and was undecided on 25% of the cases. The most frequent discordance concerned diagnosis of urethritis. On collection of medical data by history and physical examination, the panel found 22% of the consultations unsatisfactory. Discordance in both diagnosis and drug prescription were related to remarks about data collection. The results suggest that the body of knowledge available for GPs for management of female GU infections in outpatient practice is incomplete.

Written case simulations: do they predict physicians' behavior?

Written case simulations are often used to investigate physicians' decision making and clinical competence. Their use rests on the assumption that physicians' responses to written simulations closely agree with their responses to actual clinical encounters, yet this assumption of criterion validity has received little attention. To determine the ability of written case simulations to predict actual clinical behavior, we applied methodologic criteria to published articles that used written simulations. Only 11 (15%) of 74 articles included an assessment of the criterion validity of their written case simulations. Only 2 of those 11 studies were designed and executed in such a way that criterion validity could be fully interpreted. No clear consensus emerged from an examination on the 11 studies on how well responses to written case simulations perform as proxy measures of actual behavior. More work is needed before assuming that written case simulations measure actual behavior.

Evaluating interns' performance using simulated patients in a casualty department

An observational study examined the care that was provided by 61 interns for three simulated (standardized) patients in a casualty department within a large teaching hospital. The consultations were recorded on videotape and were scored by reference to explicit criteria that were set by an expert panel. The criteria described standards of clinical competence, communication skills and preventive care for each of the three cases. An additional list of the general opportunities for preventive advice that were suggested by the patients' case histories also was compared with the interns' actual behaviour. Clinical competence was not demonstrated uniformly across the three cases. The problem of urinary-tract infection was managed best; 19% of the interns complied with all criteria, and 93% of the interns complied with 75% of the criteria. For tension headache, only 3% of the interns complied with all the criteria but 67% of interns complied with 75% of the criteria. In the case of bronchitis, only 3% of interns complied with all the criteria but 58% of interns met 75% of the criteria. The quality of preventive care also varied from case to case, being best for the patient with bronchitis and worst for the patient with tension headache. Over all, only three of the 13 criteria were met by more than half the interns. The communication-skills criteria identified three behaviours which the interns were most likely to omit: the interns summarized the patients' problems in only 73 of the 173 consultations; they explained the aetiology of the problems in only 127 of the 173 consultations; and they ensured that the patients understood their plan of management in only 21 of the 173 consultations.

Can regional differences in consumption of tranquillizers and hypnotics be explained by variations in general practitioners' threshold of prescribing? A methodological study

Large differences exist in the consumption of tranquillizers and hypnotics within and between countries. The present study investigates the hypothesis that regional differences in the utilization of tranquillizers and hypnotics are associated with corresponding differences in thresholds of prescribing. A questionnaire that described in a standardized manner 12 patients with a symptomatology of anxiety and insomnia was sent to all general practitioners in two regions (south/north) in Norway. In the south, tranquillizers and hypnotics are used much more frequently than in the north. Based on written simulations of patients, no differences in thresholds of prescribing was found between the general practitioners in the two regions. Neither could the doctors' choice of drugs, nor their recommendations for dosage and duration of treatment, explain the different consumption of tranquillizers and hypnotics in the two regions. The method could, however, be criticized for insufficient validity, since our case stories were less controversial with respect to psychotropic drug prescribing than intended.

Simulated patients in general practice: a different look at the consultation

To develop a better empirical basis for developing quality assessment in general practice three simulated patients made appointments with 48 general practitioners during actual surgery hours and collected facts about their performance. The simulated patients were indistinguishable from real patients and presented a standardised story of a symptomatic urinary tract infection. Two months later the same general practitioners received a written simulation about a patient who had the same urinary tract infection and were asked how they would handle this in real practice. Both results were scored against an existing consensus standard. The overall score for both methods did not show any substantial differences. A more differentiated analysis, however, showed that general practitioners performed significantly better with simulated patients. It also showed that general practitioners answering the written simulation performed significantly more unnecessary and superfluous actions. The results of this study show that the use of simulated patients seems to show the efficient performance of general practitioners in practice.

The diagnostic process in primary care: a comparison of general internists and family physicians

This investigation examined the formulation of diagnostic hypotheses by general internists and family physicians in response to three patient cases (dyspnea, abdominal pain and syncope). The investigation was conducted in the United States. Physician responses to sequentially presented written clinical information were audiotaped. Each transcribed protocol was scored to enumerate and characterize the hypotheses considered by physicians in each specialty. Results of the analyses of variance of hypothesis measures revealed that internists generated more hypotheses than family physicians and that the internist's hypotheses were more specific and were less likely to be generated by other physicians. In addition, internists tended to consider hypotheses more closely related to the final diagnosis sooner in the case presentation than did family physicians. The findings of increased number, specificity, and uniqueness of hypothesis considered by internists are consistent with previously demonstrated differences in the amount and nature of diagnostic information collected by family physicians and internists.

Problem solving in undergraduate medical students

In recent years the systematic development of the skill of clinical reasoning has come to assume a high priority as an explicit aim of medical education. Clinical reasoning, it is contended, is the application of general reasoning and problem-solving skills to the specific knowledge base of medicine. The results presented in this paper constitute a preliminary study designed to investigate the ability of first-year medical undergraduates to solve abstract problems using a simple nonmedical knowledge base. Further studies are being carried out and will continue to extend this into the specific medical knowledge base area. The potential implications for medical education and the development of clinical reasoning are discussed.

Validity of patient management problems re-examined

Newble et al. (1982) examined the construct validity of one Patient Management Problem (PMP) with groups of subjects with varied levels of medical competence, ranging from fourth-year medical students to post-intern medical registrars and consultant physicians. The present study reanalysed their data using analysis of variance followed by Scheffé a postiori comparisons of the mean scores for each group. Results suggest virtually no difference among the various groups' performances on this particular PMP, in contrast to the assertion of Newble et al. (1982) that 'the most competent groups (the post-interns) generally scored less well on the calculated indices than the senior students and interns'. While Newble et al. (1982) maintained that their findings do not show the PMP to be a valid test of competence, the present analysis suggests that their study was not an adequate test of the construct validity of PMPs due to lack of power and insufficient sample size. An alternative interpretation of this data is presented, along with implications for medical education and future research.

How we measure problem-solving ability

A review of some of the more recent literature on problem solving is presented. An attempt is made to identify factors which may lead to discrediting the PMP (Patient Management Problem) as a measure of problem-solving ability. A definition of competence in problem solving is proposed and the question of scoring is discussed. A possible method for future research is suggested.

Patient management problems in general practice using a medical journal for self-assessment exercises

A series of six patient management problems, with clinical case histories from the authors' own practices, was presented in a Norwegian medical journal (Utposten) during 1981 and the spring of 1982. The readers were asked to respond to a number of statements which related to each case history, using a 5-point rating scale. The answers were prepared from return forms mailed anonymously to the authors, leaving the doctors with copies of their own answers. The following issue of the journal then presented the distribution of all incoming answers by giving histograms and mean values of the ratings for each question. In addition, the case history author gave his own ratings, a review of the answers, and more details about the case. In this way the respondents were given a possibility of comparing their own diagnostic skills and management decisions with those of their colleagues. There was a total of 406 incoming answers from 243 different participants, among whom 210 were general practitioners or district physicians. Background characteristics of these doctors are discussed, as well as the results of an evaluation questionnaire. This experiment with problem solving by mail was well received by the readers, mainly as a supplement to their continuing medical education. The method in this study, which permits quantification of clinical decision, may be a useful tool for medical audit activities.

Patient management problems. Issues of validity

Patient management problems (PMP) are being used in medical examinations with increasing frequency despite evidence which throws doubt on their validity as measures of clinical competence. This study investigated the construct validity of a PMP constructed in both written and interview formats. Each test was administered to groups of students of different seniorities and to two groups of doctors, interns and post-interns. The pattern of scores for the different groups was not that expected of a valid test of competence. The most competent groups (the post-interns) generally scored less well on the calculated indices than the senior students and interns. These findings were similar for both formats of the test so cueing was not thought to be the major factor. It appears that the scoring system is at fault. A comparison of performance on the written and interview (uncued) formats showed that many more options were chosen by all groups tested on the written PMP. It was concluded that written PMPs cannot yet be regarded as a valid simulation of clinical performance. Although content validity is high this does not appear to be so for construct validity or concurrent validity.