Patient management problems. Issues of validity

Assessing Clinical Reasoning: Targeting the Higher Levels of the Pyramid

Clinical reasoning is a core component of clinical competency that is used in all patient encounters from simple to complex presentations. It involves synthesis of myriad clinical and investigative data, to generate and prioritize an appropriate differential diagnosis and inform safe and targeted management plans.The literature is rich with proposed methods to teach this critical skill to trainees of all levels. Yet, ensuring that reasoning ability is appropriately assessed across the spectrum of knowledge acquisition to workplace-based clinical performance can be challenging.In this perspective, we first introduce the concepts of illness scripts and dual-process theory that describe the roles of analytic system 1 and non-analytic system 2 reasoning in clinical decision making. Thereafter, we draw upon existing evidence and expert opinion to review a range of methods that allow for effective assessment of clinical reasoning, contextualized within Miller's pyramid of learner assessment. Key assessment strategies that allow teachers to evaluate their learners' clinical reasoning ability are described from the level of knowledge acquisition, through to real-world demonstration in the clinical workplace.

Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance

PURPOSE

An evidence-based approach to assessment is critical for ensuring the development of clinical reasoning (CR) competence. The wide array of CR assessment methods creates challenges for selecting assessments fit for the purpose; thus, a synthesis of the current evidence is needed to guide practice. A scoping review was performed to explore the existing menu of CR assessments.

METHOD

Multiple databases were searched from their inception to 2016 following PRISMA guidelines. Articles of all study design types were included if they studied a CR assessment method. The articles were sorted by assessment methods and reviewed by pairs of authors. Extracted data were used to construct descriptive appendixes, summarizing each method, including common stimuli, response formats, scoring, typical uses, validity considerations, feasibility issues, advantages, and disadvantages.

RESULTS

A total of 377 articles were included in the final synthesis. The articles broadly fell into three categories: non-workplace-based assessments (e.g., multiple-choice questions, extended matching questions, key feature examinations, script concordance tests); assessments in simulated clinical environments (objective structured clinical examinations and technology-enhanced simulation); and workplace-based assessments (e.g., direct observations, global assessments, oral case presentations, written notes). Validity considerations, feasibility issues, advantages, and disadvantages differed by method.

CONCLUSIONS

There are numerous assessment methods that align with different components of the complex construct of CR. Ensuring competency requires the development of programs of assessment that address all components of CR. Such programs are ideally constructed of complementary assessment methods to account for each method's validity and feasibility issues, advantages, and disadvantages.

Plus ça change, plus c'est pareil: Making a continued case for the use of MCQs in medical education

Despite the increased emphasis on the use of workplace-based assessment in competency-based education models, there is still an important role for the use of multiple choice questions (MCQs) in the assessment of health professionals. The challenge, however, is to ensure that MCQs are developed in a way to allow educators to derive meaningful information about examinees' abilities. As educators' needs for high-quality test items have evolved so has our approach to developing MCQs. This evolution has been reflected in a number of ways including: the use of different stimulus formats; the creation of novel response formats; the development of new approaches to problem conceptualization; and the incorporation of technology. The purpose of this narrative review is to provide the reader with an overview of how our understanding of the use of MCQs in the assessment of health professionals has evolved to better measure clinical reasoning and to improve both efficiency and item quality.

On the Meaning of Scores Derived from Patient Management Problems

Patient Management Problem (PMP) test scores have been questioned by many for their interpretability and validity. Despite these measurement problems, PMPs remain popular evaluation and research instruments. A paired-comparisons study was undertaken to investigate the relationship between performance and corresponding scores (proficiency andpathway). The scale values for the performances do not reflect the same relative quality ofperformance as the proficiency and pathway scores. Even more remarkable is the emergence of a different standard when performance is assessed as a whole. The findings of this study suggest that assigning points at the "option level" culminates in scores that cannot easily be interpreted and cannot readily be linked to the behavior they are supposed to represent. These conclusions have implications for construct validity and educational decisions.

A Criterion-Referenced Approach to Measuring Medical Problem Solving

A criterion-referenced approach was used to examine the validity of a set of 15 Patient Management Problems (PMPs) representing a broad range of medical problems. Results of performance of 175 medical students indicated that a new problem-solving index that includes an open-ended differential diagnosis section had greater validity than the more traditional proficiency index, as the spread between the distributions of masters and nonmasters was greater for the problemsolving index. Implications for medical education and research are discussed.

An Evaluation of a Computer Simulation in the Assessment of Physician Competence

This study compares physician performance on the Computer-Aided Simulation of the Clinical Encounter (CASE) with peer ratings and performance on multiple choice questions (MCQs) and patient management problems (PMPs). CASEis a simulation of the clinical encounter where the computer plays the role of the patient and the physician elicits information by entering "natural language" questions into a computer terminal. Results indicate that all formats are equally valid, although MCQs are the most reliable methods of assessment per unit of testing time, followed by PMPs and CASE, in that order. All methods measure the same or very highly correlated aspects of competence.

General overview of the theories used in assessment: AMEE Guide No. 57

There are no scientific theories that are uniquely related to assessment in medical education. There are many theories in adjacent fields, however, that can be informative for assessment in medical education, and in the recent decades they have proven their value. In this AMEE Guide we discuss theories on expertise development and psychometric theories, and the relatively young and emerging framework of assessment for learning. Expertise theories highlight the multistage processes involved. The transition from novice to expert is characterised by an increase in the aggregation of concepts from isolated facts, through semantic networks to illness scripts and instance scripts. The latter two stages enable the expert to recognise the problem quickly and form a quick and accurate representation of the problem in his/her working memory. Striking differences between experts and novices is not per se the possession of more explicit knowledge but the superior organisation of knowledge in his/her brain and pairing it with multiple real experiences, enabling not only better problem solving but also more efficient problem solving. Psychometric theories focus on the validity of the assessment - does it measure what it purports to measure and reliability - are the outcomes of the assessment reproducible. Validity is currently seen as building a train of arguments of how best observations of behaviour (answering a multiple-choice question is also a behaviour) can be translated into scores and how these can be used at the end to make inferences about the construct of interest. Reliability theories can be categorised into classical test theory, generalisability theory and item response theory. All three approaches have specific advantages and disadvantages and different areas of application. Finally in the Guide, we discuss the phenomenon of assessment for learning as opposed to assessment of learning and its implications for current and future development and research.

The validity of performance-based measures of clinical reasoning and alternative approaches

CONTEXT

The development of a valid and reliable measure of clinical reasoning ability is a prerequisite to advancing our understanding of clinically relevant cognitive processes and to improving clinical education. A record of problem-solving performances within standardised and computerised patient simulations is often implicitly assumed to reflect clinical reasoning skills. However, the validity of this measurement method for assessing clinical reasoning is open to question.

OBJECTIVES

Explicitly defining the intended clinical reasoning construct should help researchers critically evaluate current performance score interpretations. Although case-specific measurement outcomes (i.e. low correlations between cases) have led medical educators to endorse performance-based assessments of problem solving as a method of measuring clinical reasoning, the matter of low across-case generalisation is a reliability issue with validity implications and does not necessarily support a performance-based approach. Given this, it is important to critically examine whether our current performance-based testing efforts are correctly focused. To design a valid educational assessment of clinical reasoning requires a coherent argument represented as a chain of inferences supporting a clinical reasoning interpretation.

DISCUSSION

Suggestions are offered for assessing how well an examinee's existing knowledge organisation accommodates the integration of new patient information, and for focusing assessments on an examinee's understanding of how new patient information changes case-related probabilities and base rates.

Veterinary students' attitudes toward the assessment of clinical reasoning using extended matching questions

For the purposes of assessment, clinical expertise has been broken down into three broad components: scientific and clinical knowledge, clinical reasoning, and practical or technical skills. This structure can be used to define the tools used for assessment of clinical students. Knowledge can be assessed through a variety of written formats, and skills through various practical assessments, including the objective structured clinical examination. The assessment of clinical reasoning is more of a challenge, and, partly in order to address this challenge, the Royal Veterinary College recently introduced veterinary clinical-scenario-based extended matching questions. A questionnaire was used to collect students' perceptions of the new format. Surprisingly, this questionnaire also delivered important insights into the students' understanding of the process of clinical reasoning itself that could be crucial in future curriculum design. Despite a theory course that introduced students to the nature of expertise and the importance of pattern recognition to experienced clinicians, some final-year students could not recognize this approach as relevant to them and objected to the way in which some of the questions were driving them to think. This may relate to the variety of methods of case management that students observe during their practical experience and the different attitudes of clinicians to the way students work up cases. Overall, the students perceived this question type as an appropriate way to test clinical reasoning and as relevant to the experience they had gained during their clinical rotations, both within the college and in veterinary practices outside it.

Use of Iliad to Improve Diagnostic Performance of Nurse Practitioner Students

Nurse practitioners (NPs) have dual goals as primary care providers, combining the traditional goals of nursing with extended goals as diagnosticians. Diagnostic reasoning, therefore, is a critical component of NP education. Iliad, a computerized diagnostic reasoning expert system, has been used effectively to teach diagnostic skills to medical students. A pilot study was undertaken to determine the effects of Iliad training on NP students' diagnostic skill performance and to identify technical and instructional issues of implementation. The study found that the use of Iliad improved NP students' diagnostic reasoning, and that the training effects were modified by prior nursing experience. Successful use of Iliad required planning, faculty commitment, and technical support.

A randomized comparison of alternative formats for clinical simulations

Computer-based clinical simulations for medical education vary widely in structure and format, yet few studies have examined which formats are optimal for particular educational settings. This study is a randomized comparison of the same simulated case in three formats: a "pedagogic" format offering explicit educational support, a "high-fidelity" format attempting to model clinical reasoning in the real world, and a "problem-solving" format that requires students to express specific diagnostic hypotheses. Data were collected from rising third-year medical students using a posttest, attitudinal questionnaire, students' write-ups of the case, and log files of students' progress through the simulation. Student performances on all measures differed significantly by format. In general, students using the pedagogic format were more proficient but less efficient. They acquired more information but were able to do proportionately less with it. The results suggest that the format of computer-based simulations is an important educational variable.

Written case simulations: do they predict physicians' behavior?

Written case simulations are often used to investigate physicians' decision making and clinical competence. Their use rests on the assumption that physicians' responses to written simulations closely agree with their responses to actual clinical encounters, yet this assumption of criterion validity has received little attention. To determine the ability of written case simulations to predict actual clinical behavior, we applied methodologic criteria to published articles that used written simulations. Only 11 (15%) of 74 articles included an assessment of the criterion validity of their written case simulations. Only 2 of those 11 studies were designed and executed in such a way that criterion validity could be fully interpreted. No clear consensus emerged from an examination on the 11 studies on how well responses to written case simulations perform as proxy measures of actual behavior. More work is needed before assuming that written case simulations measure actual behavior.

Validity of patient management problems re-examined

Newble et al. (1982) examined the construct validity of one Patient Management Problem (PMP) with groups of subjects with varied levels of medical competence, ranging from fourth-year medical students to post-intern medical registrars and consultant physicians. The present study reanalysed their data using analysis of variance followed by Scheffé a postiori comparisons of the mean scores for each group. Results suggest virtually no difference among the various groups' performances on this particular PMP, in contrast to the assertion of Newble et al. (1982) that 'the most competent groups (the post-interns) generally scored less well on the calculated indices than the senior students and interns'. While Newble et al. (1982) maintained that their findings do not show the PMP to be a valid test of competence, the present analysis suggests that their study was not an adequate test of the construct validity of PMPs due to lack of power and insufficient sample size. An alternative interpretation of this data is presented, along with implications for medical education and future research.

Assessment of undergraduate general practice teaching

Final-year medical students at the University of Bristol were asked to outline a plan of management for an elderly female patient who presented with several problems. These problems were presented in the form of a 'patient management questionnaire'. This questionnaire was completed before and again after a 9-week course that included a 3-week attachment to two general practitioners. The patterns of answers were compared to look for modifications in the plan of management that could be attributed to the teaching. The results indicate a broader plan of management after the course with more students considering the physical, social and family needs of the patient, rather than the clinical problems alone. Nevertheless, further experience of this method of assessment is needed before the technique can be considered a useful tool in the evaluation of general practice teaching.

How we measure problem-solving ability

A review of some of the more recent literature on problem solving is presented. An attempt is made to identify factors which may lead to discrediting the PMP (Patient Management Problem) as a measure of problem-solving ability. A definition of competence in problem solving is proposed and the question of scoring is discussed. A possible method for future research is suggested.

The presence of distinct and stable skills among students assessed for clinical competence in oncology

Several aspects of clinical competence in oncology were assessed among sixty-one fourth-year medical students, and results correlated with success in other areas of the curriculum as well as the extent of previous exposure to cancer patients. The inter-relationships identified by factor analysis among the outcome and predictor measures suggest that several distinct competencies were being assessed. In addition, certain skills, such as the ability to generate appropriate diagnostic hypotheses and to perform well on multiple-choice exams, appear to have been stable across time. These findings imply that there are distinct components to clinical competence, and that various methods of evaluation reflect these competencies differently.