Medical Student Evaluation With a Serious Game Compared to Multiple Choice Questions Assessment

Training nurses in an international emergency medical team using a serious role-playing game: a retrospective comparative analysis

BACKGROUND

Although game-based applications have been used in disaster medicine education, no serious computer games have been designed specifically for training these nurses in an IEMT setting. To address this need, we developed a serious computer game called the IEMTtraining game. In this game, players assume the roles of IEMT nurses, assess patient injuries in a virtual environment, and provide suitable treatment options.

METHODS

The design of this study is a retrospective comparative analysis. The research was conducted with 209 nurses in a hospital. The data collection process of this study was conducted at the 2019-2020 academic year. A retrospective comparative analysis was conducted on the pre-, post-, and final test scores of nurses in the IEMT. Additionally, a survey questionnaire was distributed to trainees to gather insights into teaching methods that were subsequently analyzed.

RESULTS

There was a significant difference in the overall test scores between the two groups, with the game group demonstrating superior performance compared to the control group (odds ratio = 1.363, p value = 0.010). The survey results indicated that the game group exhibited higher learning motivation scores and lower cognitive load compared with the lecture group.

CONCLUSIONS

The IEMT training game developed by the instructor team is a promising and effective method for training nurses in disaster rescue within IEMTs. The game equips the trainees with the necessary skills and knowledge to respond effectively to emergencies. It is easily comprehended, enhances knowledge retention and motivation to learn, and reduces cognitive load.

A Serious Game About Hematology for Health Care Workers (SUPER HEMO): Development and Validation Study

BACKGROUND

Complete blood count (CBC) and hemostatic screening tests are among the most commonly prescribed blood tests worldwide. All health care workers (nurse practitioners, pharmacists, dentists, midwives, and physicians) are expected to correctly interpret the results in their daily practice. Currently, the undergraduate hematology curriculum consists predominantly of lecture-based teaching. Because hematology combines basic science (blood cells and hemostasis physiology) and clinical skills, students report that they do not easily master hematology with only lecture-based teaching. Having interviewed students at the University of Lorraine, we considered it necessary to develop new teaching approaches and methods.

OBJECTIVE

We aimed to develop and validate a serious game about CBC analysis for health care students. Our primary objective was to help students perceive hematology as being a playful and easy topic and for them to feel truly involved in taking care of their patients by analyzing blood tests. We considered that this game-based approach would be attractive to students as an addition to the classic lecture-based approach and improve their knowledge and skills in hematology.

METHODS

We developed an adventure game called SUPER HEMO, a video game in which the player assumes the role of a protagonist in an interactive story driven by exploration and problem-solving tests. Following validation with beta testing by a panel of volunteer students, we used a novel, integrated teaching approach. We added 1.5 hours of gaming to the standard curriculum for a small group of volunteer students. Physician and pharmacy students in their third year at a single French university were invited to attend this extracurricular course. Pregame and postgame tests and satisfaction surveys were immediately recorded. Final hematology exam results were analyzed.

RESULTS

A total of 86 of 324 physician students (26.5%) and 67 of 115 pharmacy students (58%) opted to participate. Median scores on the pre- and posttests were 6 out of 10 versus 7 out of 10, respectively, for the physician students, (P<.001) and 7.5 out of 10 versus 8 out of 10, respectively, for the pharmacy students (P<.001). At the final hematology evaluation, physician students who played SUPER HEMO had a slightly better median score than those who did not: 13 out of 20 versus 12 out of 20, respectively (P=.002). Pharmacy students who played SUPER HEMO had a median score of 21.75 out of 30; this was not significantly different from pharmacy students who did not play SUPER HEMO (20/30; P=.12). Among the participants who answered the survey (n=143), more than 86% (123/143) believed they had strengthened their knowledge and nearly 80% (114/143) of them had fun.

CONCLUSIONS

Feedback from this game session provided evidence to support the integration of interactive teaching methods in undergraduate hematology teaching. The development of SUPER HEMO is intended to be completed so that it can become a support tool for continuing education.

Performance effects of simulation training for medical students - a systematic review

OBJECTIVE

Simulation based medical education (SBME) is fast becoming embedded into undergraduate medical curricula with many publications now describing its various modes and student self-reported impacts. This systematic review synthesizes the available literature for evidence of performance effects of SBME as an adjunct within traditional teaching programmes.

METHODS

A narrative systematic review was conducted according to PRISMA guidelines using Ovid MEDLINE, EMBASE, and PubMed databases for studies, published in English, reporting on general medical and surgical undergraduate SBME between 2010 to 2020. Two reviewers independently assessed potential studies for inclusion. Methods and topics of simulation with their assessments were evaluated. Descriptive statistics were used to describe pooled student cohorts.

RESULTS

3074 articles were initially identified using the search criteria with 92 full-text articles then screened for eligibility. Nineteen articles, including nine randomised trials, concerning 2459 students (median 79/study), were selected for review. Cardiac scenarios were commonest (n=6) with three studies including surgical topics. Nine studies used mannequin simulators (median time/session 17.5minutes) versus standardised patients in seven (median time/session=82 minutes). Educational impact was measured by written (n=10), checklist (n=5) and OSCEs (n=3) assessment either alone or in combination (n=1, OSCE/written assessment). All articles reported a positive effect of SBME on knowledge including improved retention in three.

CONCLUSION

SBME, as an adjunct to existing curricula, improves knowledge-based performance of medical students at least in the short-term. Future studies should broaden its topics, assess longer term impacts and cost-effectiveness while also considering whether and what areas of traditional undergraduate learning it can replace.

Learning Analytics Applied to Clinical Diagnostic Reasoning Using a Natural Language Processing-Based Virtual Patient Simulator: Case Study

BACKGROUND

Virtual patient simulators (VPSs) log all users' actions, thereby enabling the creation of a multidimensional representation of students' medical knowledge. This representation can be used to create metrics providing teachers with valuable learning information.

OBJECTIVE

The aim of this study is to describe the metrics we developed to analyze the clinical diagnostic reasoning of medical students, provide examples of their application, and preliminarily validate these metrics on a class of undergraduate medical students. The metrics are computed from the data obtained through a novel VPS embedding natural language processing techniques.

METHODS

A total of 2 clinical case simulations (tests) were created to test our metrics. During each simulation, the students' step-by-step actions were logged into the program database for offline analysis. The students' performance was divided into seven dimensions: the identification of relevant information in the given clinical scenario, history taking, physical examination, medical test ordering, diagnostic hypothesis setting, binary analysis fulfillment, and final diagnosis setting. Sensitivity (percentage of relevant information found) and precision (percentage of correct actions performed) metrics were computed for each issue and combined into a harmonic mean (F₁), thereby obtaining a single score evaluating the students' performance. The 7 metrics were further grouped to reflect the students' capability to collect and to analyze information to obtain an overall performance score. A methodological score was computed based on the discordance between the diagnostic pathway followed by students and the reference one previously defined by the teacher. In total, 25 students attending the fifth year of the School of Medicine at Humanitas University underwent test 1, which simulated a patient with dyspnea. Test 2 dealt with abdominal pain and was attended by 36 students on a different day. For validation, we assessed the Spearman rank correlation between the performance on these scores and the score obtained by each student in the hematology curricular examination.

RESULTS

The mean overall scores were consistent between test 1 (mean 0.59, SD 0.05) and test 2 (mean 0.54, SD 0.12). For each student, the overall performance was achieved through a different contribution in collecting and analyzing information. Methodological scores highlighted discordances between the reference diagnostic pattern previously set by the teacher and the one pursued by the student. No significant correlation was found between the VPS scores and hematology examination scores.

CONCLUSIONS

Different components of the students' diagnostic process may be disentangled and quantified by appropriate metrics applied to students' actions recorded while addressing a virtual case. Such an approach may help teachers provide students with individualized feedback aimed at filling competence drawbacks and methodological inconsistencies. There was no correlation between the hematology curricular examination score and any of the proposed scores as these scores address different aspects of students' medical knowledge.

Conceptual Ambiguity Surrounding Gamification and Serious Games in Health Care: Literature Review and Development of Game-Based Intervention Reporting Guidelines (GAMING)

BACKGROUND

In health care, the use of game-based interventions to increase motivation, engagement, and overall sustainability of health behaviors is steadily becoming more common. The most prevalent types of game-based interventions in health care research are gamification and serious games. Various researchers have discussed substantial conceptual differences between these 2 concepts, supported by empirical studies showing differences in the effects on specific health behaviors. However, researchers also frequently report cases in which terms related to these 2 concepts are used ambiguously or even interchangeably. It remains unclear to what extent existing health care research explicitly distinguishes between gamification and serious games and whether it draws on existing conceptual considerations to do so.

OBJECTIVE

This study aims to address this lack of knowledge by capturing the current state of conceptualizations of gamification and serious games in health care research. Furthermore, we aim to provide tools for researchers to disambiguate the reporting of game-based interventions.

METHODS

We used a 2-step research approach. First, we conducted a systematic literature review of 206 studies, published in the Journal of Medical Internet Research and its sister journals, containing terms related to gamification, serious games, or both. We analyzed their conceptualizations of gamification and serious games, as well as the distinctions between the two concepts. Second, based on the literature review findings, we developed a set of guidelines for researchers reporting on game-based interventions and evaluated them with a group of 9 experts from the field.

RESULTS

Our results show that less than half of the concept mentions are accompanied by an explicit definition. To distinguish between the 2 concepts, we identified four common approaches: implicit distinction, synonymous use of terms, serious games as a type of gamified system, and distinction based on the full game dimension. Our Game-Based Intervention Reporting Guidelines (GAMING) consist of 25 items grouped into four topics: conceptual focus, contribution, mindfulness about related concepts, and individual concept definitions.

CONCLUSIONS

Conceptualizations of gamification and serious games in health care literature are strongly heterogeneous, leading to conceptual ambiguity. Following the GAMING can support authors in rigorous reporting on study results of game-based interventions.

Efficacy of Serious Games in Healthcare Professions Education: A Systematic Review and Meta-analysis

SUMMARY STATEMENT

Serious games (SGs) are interactive and entertaining software designed primarily with an educational purpose. This systematic review synthesizes evidence from experimental studies regarding the efficacy of SGs for supporting engagement and improving learning outcomes in healthcare professions education. Randomized controlled trials (RCTs) published between January 2005 and April 2019 were included. Reference selection and data extraction were performed in duplicate, independently. Thirty-seven RCTs were found and 29 were included in random-effect meta-analyses. Compared with other educational interventions, SGs did not lead to more time spent with the intervention {mean difference 23.21 minutes [95% confidence interval (CI) = -1.25 to 47.66]}, higher knowledge acquisition [standardized mean difference (SMD) = 0.16 (95% CI = -0.20 to 0.52)], cognitive [SMD 0.08 (95% CI = -0.73 to 0.89)], and procedural skills development [SMD 0.05 (95% CI = -0.78 to 0.87)], attitude change [SMD = -0.09 (95% CI = -0.38 to 0.20)], nor behavior change [SMD = 0.2 (95% CI = -0.11 to 0.51)]. Only a small SMD of 0.27 (95% CI = 0.01 to 0.53) was found in favor of SGs for improving confidence in skills.

Effect of implementing a mobile game on improving dietary information in diabetic patients

Background: Nowadays, digital games are not just entertainment, but beside routine treatments, they are used in patient care, especially in patients with diabetes. Application of digital games in patient's education can improve self-management of diabetes. The aim of the present study was to evaluate the effect of a mobile game (Amoo) implementation on enhancing dietary information in patients with type 2 diabetes. Methods: A mobile game (called Amoo), which was developed by researchers of this study, was applied to assess the self-education of patients with diabetes. Sixty patients with type 2 diabetes participated in the study. The participants took part in a pre-intervention test to determine their dietary information. The participants were randomly divided into one of two groups, including the intervention group: played the game for 15 minutes daily for 6 weeks, and the control group: did not involve in the game. A post-intervention test was run to show a possible improvement in dietary information. Data were analyzed using paired t test and suitable non-parametric testes including Mann-Whitney and Wilcoxon signed rank tests as well as Spearman and Pearson correlation coefficients via IBM SPSS statistics version 21 (SPSS, v 21.0, IBM, Armonk, NY, USA). A P-value less than 0.05 was considered as a significant level. Results: The results indicated a statistically significant difference between the pre and post test scores in the intervention group (p<0.001). However, there was no significant difference in fasting blood sugar (p=0.125). Conclusion: The mobile game (Amoo) could enhance the knowledge of patients with type 2 diabetes about food calories and glycemic index. This means that mobile games may serve as an educational aid to these patients.

Simulation Game Versus Multiple Choice Questionnaire to Assess the Clinical Competence of Medical Students: Prospective Sequential Trial

BACKGROUND

The use of simulation games (SG) to assess the clinical competence of medical students has been poorly studied.

OBJECTIVE

The objective of this study was to assess whether an SG better reflects the clinical competence of medical students than a multiple choice questionnaire (MCQ).

METHODS

Fifth-year medical students in Paris (France) were included and individually evaluated on a case of pediatric asthma exacerbation using three successive modalities: high-fidelity simulation (HFS), considered the gold standard for the evaluation of clinical competence, the SG Effic'Asthme, and an MCQ designed for the study. The primary endpoint was the median kappa coefficient evaluating the correlation of the actions performed by the students between the SG and HFS modalities and the MCQ and HFS modalities. Student satisfaction was also evaluated.

RESULTS

Forty-two students were included. The actions performed by the students were more reproducible between the SG and HFS modalities than between the MCQ and HFS modalities (P=.04). Students reported significantly higher satisfaction with the SG (P<.01) than with the MCQ modality.

CONCLUSIONS

The SG Effic'Asthme better reflected the actions performed by medical students during an HFS session than an MCQ on the same asthma exacerbation case. Because SGs allow the assessment of more dimensions of clinical competence than MCQs, they are particularly appropriate for the assessment of medical students on situations involving symptom recognition, prioritization of decisions, and technical skills.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03884114; https://clinicaltrials.gov/ct2/show/NCT03884114.

A Web-Based Serious Game on Delirium as an Educational Intervention for Medical Students: Randomized Controlled Trial

Background

Adequate delirium recognition and management are important to reduce the incidence and severity of delirium. To improve delirium recognition and management, training of medical staff and students is needed.

Objective

In this study, we aimed to gain insight into whether the serious game, Delirium Experience, is suited as an educational intervention.

Methods

We conducted a three-arm randomized controlled trial. We enrolled 156 students in the third year of their Bachelor of Medical Sciences degree at the University Medical Centre Groningen. The Game group of this study played Delirium Experience. The Control D group watched a video with explanations on delirium and a patient’s experience of delirious episodes. The Control A group watched a video on healthy aging. To investigate students’ skills, we used a video of a delirious patient for which students had to give care recommendations and complete the Delirium Observations Screening Scale and Delirium Rating Scale R-98. Furthermore, students completed the Delirium Attitude Scale, the Learning Motivation and Engagement Questionnaire, and self-reported knowledge on delirium.

Results

In total, 156 students participated in this study (Game group, n=51; Control D group, n=51; Control A group, n=55). The Game group scored higher with a median (interquartile range) of 6 (4-8) for given recommendations and learning motivation and engagement compared with the Control D (1, 1-4) and A (0, 0-3) groups (P<.001). Furthermore, the Game group scored higher (7, 6-8) on self-reported knowledge compared with the Control A group (6, 5-6; P<.001). We did not find differences between the groups regarding delirium screening (P=.07) and rating (P=.45) skills or attitude toward delirious patients (P=.55).

Conclusions

The serious game, Delirium Experience, is suitable as an educational intervention to teach delirium care to medical students and has added value in addition to a lecture.

Use of Simulator-Based Teaching to Improve Medical Students' Knowledge and Competencies: Randomized Controlled Trial

BACKGROUND

Simulator-based teaching for coronary angiography (CA) is an attractive educational tool for medical students to improve their knowledge and skills. Its pedagogical impact has not been fully evaluated yet.

OBJECTIVE

The aim of this study was to compare traditional face-to-face teaching with a simulator-based teaching for the acquisition of coronary anatomy knowledge and CAs interpretation.

METHODS

A total of 118 medical school students in their fourth to sixth year were prospectively randomized in 2 groups: (1) a control teaching group (n=59, CONT group) and (2) a simulator group (using the Mentice VIST-Lab CA simulator; n=59, SIM group). The CONT group received a PowerPoint-based course, whereas the SIM group received a simulator-based course including the same information. After the course, all students were evaluated by 40 multiple choice questions (maximum of 100 points), including questions on coronary anatomy (part 1), angiographic projections (part 2), and real CAs interpretation (part 3). Satisfaction of the students was also evaluated by a simple questionnaire.

RESULTS

Student characteristics were identical in both the groups: 62/118 (52.5%) were female and age was 22.6 (SD 1.4) years. Moreover, 35.6% (42/118) were in their fourth year, 35.6% (42/118) were in the fifth year, and 28.8% (34/118) in the sixth year. During the evaluation, SIM students had higher global scores compared with CONT students, irrespective of their year of medical school (59.5 [SD 10.8] points vs 43.7 [SD 11.3] points, P<.001). The same observations were noted for each part of the test (36.9 [SD 6.6] points vs 29.6 [SD 6.9] points, P<.001; 5.9 [SD 3.0] points vs 3.1 [SD 2.8] points, P<.001; and 16.8 [SD 6.9] points vs 10.9 [SD 6.5] points, P<.001; for parts 1, 2, and 3, respectively). Student satisfaction was higher in the SIM group compared with the CONT group (98% vs 75%, P<.001).

CONCLUSIONS

This study suggests that simulator-based teaching could potentially improve students' knowledge of coronary anatomy, angiography projections, and interpretation of real clinical cases, suggesting better clinical skills. These results should encourage further evaluation of simulator-based teaching in other medical specialties and how they can translate into clinical practice.