Assessment of patient management skills and clinical skills of practising doctors using computer-based case simulations and standardised patients

The most influence factor of the medical competence achievement regarding patient management ability on medical school graduates

PURPOSE

A doctor's professional behavior and clinical competency reflect a range of personal and interpersonal qualities, attributes, commitments, and values. This study aimed to identify the most influential factor of medical competence regarding patient management ability.

METHODS

We used an analytic observational design with a cross-sectional approach, and gathered the perceptions of medical school graduates of Bandung Islamic University via an online questionnaire scored on a Likert scale. Two hundred and six medical graduates who graduated at least 3 years prior to survey were included in the study. The factors evaluated included humanism, cognitive competence, clinical skill competence, professional behavior, patient management ability, and interpersonal skill. IBM AMOS ver. 26.0 (IBM Corp., Armonk, USA) was used for structural equation modelling of the six variables latent and 35 indicator variables.

RESULTS

We found that graduates have highly positive perceptions of the humanism (95.67%). Followed by interpersonal skills (91.26%), patient management (89.53%), professional behavior (88.47%), and cognitive competence (87.12%). They rated clinical skill competence the lowest (81.7%). Regarding factors that contribute to patient management ability, the aspects of humanism, interpersonal skill, and professional behavior were found to significantly affect patient management ability (p-value=0.035, 0.00, and 0.00, respectively) with a critical rate of 2.11, 4.31, and 4.26 consecutively.

CONCLUSION

Humanism and interpersonal skill are two important factors that medical graduates assessed very positively. According to surveyed medical graduates, their expectations of the institution were met regarding humanism. However, there is a need to strengthen medical students' clinical skills and improve their cognitive abilities through educational programs.

A Literature Study of Medical Simulations for Non-Technical Skills Training in Emergency Medicine: Twenty Years of Progress, an Integrated Research Framework, and Future Research Avenues

Medical simulations have led to extensive developments in emergency medicine. Apart from the growing number of applications and research efforts in patient safety, few studies have focused on modalities, research methods, and professions via a synthesis of simulation studies with a focus on non-technical skills training. Intersections between medical simulation, non-technical skills training, and emergency medicine merit a synthesis of progress over the first two decades of the 21st century. Drawing on research from the Web of Science Core Collection's Science Citation Index Expanded and Social Science Citation Index editions, results showed that medical simulations were found to be effective, practical, and highly motivating. More importantly, simulation-based education should be a teaching approach, and many simulations are utilised to substitute high-risk, rare, and complex circumstances in technical or situational simulations. (1) Publications were grouped by specific categories of non-technical skills, teamwork, communication, diagnosis, resuscitation, airway management, anaesthesia, simulation, and medical education. (2) Although mixed-method and quantitative approaches were prominent during the time period, further exploration of qualitative data would greatly contribute to the interpretation of experience. (3) High-fidelity dummy was the most suitable instrument, but the tendency of simulators without explicitly stating the vendor selection calls for a standardised training process. The literature study concludes with a ring model as the integrated framework of presently known best practices and a broad range of underexplored research areas to be investigated in detail.

Assessment of Diagnostic Competences With Standardized Patients Versus Virtual Patients: Experimental Study in the Context of History Taking

BACKGROUND

Standardized patients (SPs) have been one of the popular assessment methods in clinical teaching for decades, although they are resource intensive. Nowadays, simulated virtual patients (VPs) are increasingly used because they are permanently available and fully scalable to a large audience. However, empirical studies comparing the differential effects of these assessment methods are lacking. Similarly, the relationships between key variables associated with diagnostic competences (ie, diagnostic accuracy and evidence generation) in these assessment methods still require further research.

OBJECTIVE

The aim of this study is to compare perceived authenticity, cognitive load, and diagnostic competences in performance-based assessment using SPs and VPs. This study also aims to examine the relationships of perceived authenticity, cognitive load, and quality of evidence generation with diagnostic accuracy.

METHODS

We conducted an experimental study with 86 medical students (mean 26.03 years, SD 4.71) focusing on history taking in dyspnea cases. Participants solved three cases with SPs and three cases with VPs in this repeated measures study. After each case, students provided a diagnosis and rated perceived authenticity and cognitive load. The provided diagnosis was scored in terms of diagnostic accuracy; the questions asked by the medical students were rated with respect to their quality of evidence generation. In addition to regular null hypothesis testing, this study used equivalence testing to investigate the absence of meaningful effects.

RESULTS

Perceived authenticity (1-tailed t81=11.12; P<.001) was higher for SPs than for VPs. The correlation between diagnostic accuracy and perceived authenticity was very small (r=0.05) and neither equivalent (P=.09) nor statistically significant (P=.32). Cognitive load was equivalent in both assessment methods (t82=2.81; P=.003). Intrinsic cognitive load (1-tailed r=-0.30; P=.003) and extraneous load (1-tailed r=-0.29; P=.003) correlated negatively with the combined score for diagnostic accuracy. The quality of evidence generation was positively related to diagnostic accuracy for VPs (1-tailed r=0.38; P<.001); this finding did not hold for SPs (1-tailed r=0.05; P=.32). Comparing both assessment methods with each other, diagnostic accuracy was higher for SPs than for VPs (2-tailed t85=2.49; P=.01).

CONCLUSIONS

The results on perceived authenticity demonstrate that learners experience SPs as more authentic than VPs. As higher amounts of intrinsic and extraneous cognitive loads are detrimental to performance, both types of cognitive load must be monitored and manipulated systematically in the assessment. Diagnostic accuracy was higher for SPs than for VPs, which could potentially negatively affect students' grades with VPs. We identify and discuss possible reasons for this performance difference between both assessment methods.

Use of Augmented and Virtual Reality in Remote Higher Education: A Systematic Umbrella Review

In this systematic umbrella review we aggregate the current knowledge of how virtual and augmented reality technologies are applicable to and impact remote learning in higher education; specifically, how they impact such learning outcomes as performance and engagement in all stages of higher education from course preparation to student evaluation and grading. This review was done as part of a state wide research effort of Latvia, to mitigate the impact of COVID-19 and specifically to provide a framework for a technological transformation of education in this context. In this work we search the Scopus and Web of Science databases for articles describing the use of virtual and/or augmented reality technologies in remote learning for higher education and their impact on learning outcomes. We identified 68 articles from which, after multiple screening and eligibility phases, nine review articles were left for extraction phase in which 30 structural elements with corresponding interventions and measured effects were extracted. Of these, 24 interventions had a measured effect on student performance (11 positive, seven negative, six no impact) and six interventions had a measured effect on student engagement (all six positive).

Online Simulated Cases Assess Retention of Virtual Neighborhood Tour Curriculum

Background Learning to identify and address social determinants of health (SDH) is a crucial component of pediatric residency training. A virtual tour of an impoverished neighborhood previously demonstrated efficacy in increasing residents' self-assessed knowledge and competence, but its impact on performance has not yet been reported. Online simulated cases are emerging as feasible assessment tools to measure trainees' skills across various healthcare settings. We developed online simulated cases to evaluate residents' retention of the virtual tour's key SDH-related learning objectives 1 month after completing this curriculum. Methods Three online simulated cases with interpolated open-ended questions were created to assess residents' ability to identify SDH, recommend appropriate resources, and display empathy. Scoring rubrics to objectively evaluate responses were developed and borderline scores were decided by a team of educators. Results 19 residents participated. Mean scores for all cases exceeded pre-established borderline scores (statistically significant in two of the three cases). More than 90% of residents identified relevant SDH in the primary care and emergency department cases. Ninety-five percent of residents recommended appropriate resources in all cases, and 89% displayed empathy. Discussion Residents' performance in online simulated cases demonstrated retention and application of the virtual tour's learning objectives, including recognizing SDH, offering appropriate resources, and displaying empathy, which supports the long-term effectiveness of the virtual tour curriculum to train pediatricians about SDH. Online simulated cases provided a standardized and cost-effective way to measure residents' skills related to curricular uptake, suggesting that this assessment approach may be adapted to evaluate other educational interventions.

Individual identification and correction of mistakes in statements of biochemical significance: An effective learning process for graduate medical students

Focused self-directed learning process needs to be promoted at present among graduate medical students in view of the information load and limited time available with them. In this study, comparison of the effect of identifying and correcting the mistakes in a given set of wrong statements with the effect of studying correct statements individually by graduate medical students on their understanding of the topic "Cancer Biology" was carried out. At the end of regular teaching module, students were asked to identify the mistakes in a set of seven incorrect statements and correct them individually without peer consultation. Another set of seven correct statements were also provided to them and were asked to study their significance. The effects on low, medium, and high achievers were evaluated by pre- and post-tests with the same set of Multiple Choice Questions (MCQs). Mean post-test marks were significantly higher among all the three groups compared to pretest marks for the wrong statements given. For the correct statements, the post-test marks were significantly higher than the pretest marks only for the low and medium achievers. The gain from identifying the mistakes was higher than studying the correct statements for all the three groups of students. Individual correction of errors in statements is a superior active learning process in comparison to the study of correct statements. © 2019 International Union of Biochemistry and Molecular Biology, 47(4):476-480, 2019.

Simulation-based assessments in health professional education: a systematic review

INTRODUCTION

The use of simulation in health professional education has increased rapidly over the past 2 decades. While simulation has predominantly been used to train health professionals and students for a variety of clinically related situations, there is an increasing trend to use simulation as an assessment tool, especially for the development of technical-based skills required during clinical practice. However, there is a lack of evidence about the effectiveness of using simulation for the assessment of competency. Therefore, the aim of this systematic review was to examine simulation as an assessment tool of technical skills across health professional education.

METHODS

A systematic review of Cumulative Index to Nursing and Allied Health Literature (CINAHL), Education Resources Information Center (ERIC), Medical Literature Analysis and Retrieval System Online (Medline), and Web of Science databases was used to identify research studies published in English between 2000 and 2015 reporting on measures of validity, reliability, or feasibility of simulation as an assessment tool. The McMasters Critical Review for quantitative studies was used to determine methodological value on all full-text reviewed articles. Simulation techniques using human patient simulators, standardized patients, task trainers, and virtual reality were included.

RESULTS

A total of 1,064 articles were identified using search criteria, and 67 full-text articles were screened for eligibility. Twenty-one articles were included in the final review. The findings indicated that simulation was more robust when used as an assessment in combination with other assessment tools and when more than one simulation scenario was used. Limitations of the research papers included small participant numbers, poor methodological quality, and predominance of studies from medicine, which preclude any definite conclusions.

CONCLUSION

Simulation has now been embedded across a range of health professional education and it appears that simulation-based assessments can be used effectively. However, the effectiveness as a stand-alone assessment tool requires further research.

The virtual-patient pilot: testing a new tool for undergraduate surgical education and assessment

BACKGROUND

The virtual patient (VP) is a web-based tool that allows students to test their clinical decision-making skills using simulated patients.

METHODS

Three VP cases were developed using commercially available software to simulate common surgical scenarios. Surgical clerks volunteered to complete VP cases. Upon case completion, an individual performance score (IPS, 0-100) was generated and a 16-item survey was administered. Surgery shelf exam scores of clerks who completed VP cases were compared with a cohort of students who did not have exposure to VP cases. Descriptive statistics were performed to characterize survey results and mean IPS.

RESULTS

Surgical clerks felt that the VP platform was simple to use, and both the content and images were well presented. They also felt that VPs enhanced learning and were helpful in understanding surgical concepts. Mean IPS at conclusion of the surgery clerkship was 69.2 (SD 26.5). Mean performance on the surgery shelf exam for the student cohort who had exposure to VPs was 86.5 (SD 7.4), whereas mean performance for the unexposed student cohort was 83.5 (SD 9).

DISCUSSION

The VP platform represents a new educational tool that allows surgical clerks to direct case progression and receive feedback regarding clinical-management decisions. Its use as an assessment tool will require further validation.

Assessing patient care: summary of the breakout group on assessment of observable learner performance

There is an established expectation that physicians in training demonstrate competence in all aspects of clinical care prior to entering professional practice. Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs. A thorough assessment of competence in patient care requires a mixture of methods, taking into account each method's costs, benefits, and current level of evidence. At the 2012 Academic Emergency Medicine (AEM) consensus conference on educational research, one breakout group reviewed and discussed the evidence supporting various methods of assessing patient care and defined a research agenda for the continued development of specific assessment methods based on current best practices. In this article, the authors review each method's supporting reliability and validity evidence and make specific recommendations for future educational research.

Formulation of multiple choice questions as a revision exercise at the end of a teaching module in biochemistry

The graduate medical students often get less opportunity for clarifying their doubts and to reinforce their concepts after lecture classes. Assessment of the effect of MCQ preparation by graduate medical students as a revision exercise on the topic "Mineral metabolism." At the end of regular teaching module on the topic "Mineral metabolism," graduate medical students were asked to prepare the stems of 15 MCQs based on the four discriminators given for each. They were told that one of the discriminators could be the answer for the MCQ and the remaining three could be the distracters. They were further guided in their task by providing few key word(s) in the stem of the expected MCQ. In the first phase of the exercise, the students attempted the MCQ preparation individually without peer consultation. In the second phase, the students participated in small group discussion to formulate the best MCQs of the group. The effects on low, medium, and high achievers were evaluated by pre and post-tests with the same set of MCQs. Both the individual endeavor in Phase 1 and small group discussion in Phase 2 for the formulation of MCQs significantly contributed to the gain from the exercise. The gains from the individual task and from small group discussion were equal among the different categories of students. Both phases of the exercise were equally beneficial for the low, medium, and high achievers. The high and medium achievers retained the gain from the exercise even after 1 week of the exercise whereas the low achievers could not retain the gain completely. Formulation of MCQs is an effective and useful unconventional revision exercise in Biochemistry for graduate medical students.

Hunter disease eClinic: interactive, computer-assisted, problem-based approach to independent learning about a rare genetic disease

BACKGROUND

Computer-based teaching (CBT) is a well-known educational device, but it has never been applied systematically to the teaching of a complex, rare, genetic disease, such as Hunter disease (MPS II).

AIM

To develop interactive teaching software functioning as a virtual clinic for the management of MPS II.

IMPLEMENTATION AND RESULTS

The Hunter disease eClinic, a self-training, user-friendly educational software program, available at the Lysosomal Storage Research Group (http://www.lysosomalstorageresearch.ca), was developed using the Adobe Flash multimedia platform. It was designed to function both to provide a realistic, interactive virtual clinic and instantaneous access to supporting literature on Hunter disease. The Hunter disease eClinic consists of an eBook and an eClinic. The eClinic is the interactive virtual clinic component of the software. Within an environment resembling a real clinic, the trainee is instructed to perform a medical history, to examine the patient, and to order appropriate investigation. The program provides clinical data derived from the management of actual patients with Hunter disease. The eBook provides instantaneous, electronic access to a vast collection of reference information to provide detailed background clinical and basic science, including relevant biochemistry, physiology, and genetics. In the eClinic, the trainee is presented with quizzes designed to provide immediate feedback on both trainee effectiveness and efficiency. User feedback on the merits of the program was collected at several seminars and formal clinical rounds at several medical centres, primarily in Canada. In addition, online usage statistics were documented for a 2-year period. Feedback was consistently positive and confirmed the practical benefit of the program. The online English-language version is accessed daily by users from all over the world; a Japanese translation of the program is also available.

CONCLUSIONS

The Hunter disease eClinic employs a CBT model providing the trainee with realistic clinical problems, coupled with comprehensive basic and clinical reference information by instantaneous access to an electronic textbook, the eBook. The program was rated highly by attendees at national and international presentations. It provides a potential model for use as an educational approach to other rare genetic diseases.

Advanced life support cardiac arrest scenario test evaluation

BACKGROUND

The cardiac arrest scenario test (CASTest) is a central component of the assessment strategy on the Advanced Life Support Course. The aim of this study was to establish equivalence between the four different CASTest scenarios and investigate the impact of profession, candidate order and course centre on the pass rate.

MATERIALS AND METHODS

This was a cluster randomised study. CASTest scenarios were randomly allocated to candidates stratified by course centre. Candidate demographics and performance were recorded on the criterion referenced check list along with the final assessment outcome (pass/fail). Differences in pass rates according scenario; profession, course centre and candidate order were examined by Chi-squared and multiple logistic regression.

RESULTS

Two thousand, four hundred and forty-nine assessments from 65 course centres were evaluated. There was no difference in pass rate between scenarios (average pass rate 74.4%). Pass rates according to course centre varied widely (40-93%, P<0.0001) as did professional group (42-100%, P<0.0001). The order that candidates took the test did not influence the pass rate.

CONCLUSION

The CASTest assessment scenarios used during ALS testing appear equivalent in terms of difficulty. In contrast, the professional background of the candidate and centre at which the assessment is performed do significantly influence the likelihood of passing the assessment. Further evaluation of the reasons for differences between course centres is required.

Use of Human Patient Simulation and the Situation Awareness Global Assessment Technique in Practical Trauma Skills Assessment

BACKGROUND

Situation awareness (SA) is defined as the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future. This construct is vital to decision making in intense, dynamic environments. It has been used in aviation as it relates to pilot performance, but has not been applied to medical education. The most widely used objective tool for measuring trainee SA is the Situation Awareness Global Assessment Technique (SAGAT). The purpose of this study was to design and validate SAGAT for assessment of practical trauma skills, and to compare SAGAT results to traditional checklist style scoring.

METHODS

Using the Human Patient Simulator, we designed SAGAT for practical trauma skills assessment based on Advanced Trauma Life Support objectives. Sixteen subjects (four staff surgeons, four senior residents, four junior residents, and four medical students) participated in three scenarios each. They were assessed using SAGAT and traditional checklist assessment. A questionnaire was used to assess possible confounding factors in attaining SA and overall trainee satisfaction.

RESULTS

SAGAT was found to show significant difference (analysis of variance; p < 0.001) in scores based on level of training lending statistical support to construct validity. SAGAT was likewise found to display reliability (Cronbach's alpha 0.767), and significant scoring correlation with traditional checklist performance measures (Pearson's coefficient 0.806). The questionnaire revealed no confounding factors and universal satisfaction with the human patient simulator and SAGAT.

CONCLUSIONS

SAGAT is a valid, reliable assessment tool for trauma trainees in the dynamic clinical environment created by human patient simulation. Information provided by SAGAT could provide specific feedback, direct individualized teaching, and support curriculum change. Introduction of SAGAT could improve the current assessment model for practical trauma education.

Using self-reported data to assess the validity of driving simulation data

In this article, we use self-reported driving behaviors from a written questionnaire to assess the measurement validity of data derived from a driving simulation. The issue of validity concerns the extent to which measures from the experimental context map onto constructs of interest. Following a description of the experimental methods and setting, an argument for the face validity of the data is advanced. Convergent validity was assessed by regressing behaviors observed in thedriving simulatoron self-reported measures of driving behaviors. Significant relationships were found across six measures: accidents, speeding, velocity, passing, weaving between traffic, and behavior at stop signs. Concurrent validity was evaluated with an analysis of simulator accident involvement and attention deficit hyperactivity disorder status. Discriminant validity was assessed using a multitrait-multimethod matrix of simulator and questionnaire data. We concluded that although the relationship between self-reported behaviors and observed responses in the simulator falls short of perfect correspondence, the data collected from the driving simulator are valid measures of the behaviors of interest.

Feasibility of a Web-Based Continuing Medical Education Program in Dermatology: The DermoFAD Experience in Italy

BACKGROUND

Web-based systems are increasingly being considered for medical education. A draft legislation on distance-learning programs was licensed in Italy by the National Commission for Continuous Education in November 2003. A series of pilot studies were developed, among these the DermoFAD project, based on five simulated clinical cases of acne and a systematic appraisal of the evidence for their clinical management. From July 1 to August 27, 2004, a total of 500 medical doctors participated in a free of charge evaluation program of the project.

OBSERVATIONS

Users were distributed all over Italy. Two hundred and eighty-one (56.2%) were primary care physicians, 83 (16.6%) dermatologists, and 136 (27.2%) other medical specialists. A wide range of connecting times was observed. The pass rate of each individual case, at first attempt, ranged from 44 to 77%. When asked to assess the overall distance-learning experience, 98% of the doctors considered it to be enjoyable. A total of 2,152 continuing medical education (CME) credits were awarded. Over 50% of the users stated they would still use the system if they had to pay for it.

CONCLUSIONS

Our experience shows that distance learning is feasible and is well accepted by physicians. The DermoFAD program was an efficient means of delivering CME to the Italian medical community at large.

Validation of learning style measures: implications for medical education practice

BACKGROUND

It is unclear which learners would most benefit from the more individualised, student-structured, interactive approaches characteristic of problem-based and computer-assisted learning. The validity of learning style measures is uncertain, and there is no unifying learning style construct identified to predict such learners.

OBJECTIVE

This study was conducted to validate learning style constructs and to identify the learners most likely to benefit from problem-based and computer-assisted curricula.

METHODS

Using a cross-sectional design, 3 established learning style inventories were administered to 97 post-Year 2 medical students. Cognitive personality was measured by the Group Embedded Figures Test, information processing by the Learning Styles Inventory, and instructional preference by the Learning Preference Inventory. The 11 subscales from the 3 inventories were factor-analysed to identify common learning constructs and to verify construct validity. Concurrent validity was determined by intercorrelations of the 11 subscales.

RESULTS

A total of 94 pre-clinical medical students completed all 3 inventories. Five meaningful learning style constructs were derived from the 11 subscales: student- versus teacher-structured learning; concrete versus abstract learning; passive versus active learning; individual versus group learning, and field-dependence versus field-independence. The concurrent validity of 10 of 11 subscales was supported by correlation analysis. Medical students most likely to thrive in a problem-based or computer-assisted learning environment would be expected to score highly on abstract, active and individual learning constructs and would be more field-independent.

CONCLUSIONS

Learning style measures were validated in a medical student population and learning constructs were established for identifying learners who would most likely benefit from a problem-based or computer-assisted curriculum.

Online clinical reasoning assessment with the Script Concordance test: a feasibility study

Background

The script concordance (SC) test is an assessment tool that measures capacity to solve ill-defined problems, that is, reasoning in context of uncertainty. This tool has been used up to now mainly in medicine. The purpose of this pilot study is to assess the feasibility of the test delivered on the Web to French urologists.

Methods

The principle of SC test construction and the development of the Web site are described. A secure Web site was created with two sequential modules: (a) The first one for the reference panel (n = 26) with two sub-tasks: to validate the content of the test and to elaborate the scoring system; (b) The second for candidates with different levels of experience in Urology: Board certified urologists, residents, medical students (5 or 6^th year). Minimum expected number of participants is 150 for urologists, 100 for residents and 50 for medical students. Each candidate is provided with an individual access code to this Web site. He/she may complete the Script Concordance test several times during his/her curriculum.

Results

The Web site has been operational since April 2004. The reference panel validated the test in June of the same year during the annual seminar of the French Society of Urology. The Web site is available for the candidates since September 2004. In six months, 80% of the target figure for the urologists, 68% of the target figure for the residents and 20% of the target figure for the student passed the test online. During these six months, no technical problem was encountered.

Conclusion

The feasibility of the web-based SC test is successful as two-thirds of the expected number of participants was included within six months. Psychometric properties (validity, reliability) of the test will be evaluated on a large scale (N = 300). If positive, educational impact of this assessment tool will be useful to help urologists during their curriculum for the acquisition of clinical reasoning skills, which is crucial for professional competence.