Performance equivalency between computer-based and traditional pen-and-paper assessment: A case study in clinical anatomy

Spot(ters) the difference: Bringing traditional anatomical examinations online

The COVID-19 pandemic caused a shift in anatomy education forcing institutions to find innovative ways to teach and assess online. This study details the development of an online spotter across multiple modules that allowed students to sit the examination at home whilst still maintaining the integrity of the assessment. The online spotter consisted of individual, Zoom calls between students and examiners whereby slides with images and questions were screen shared. To examine the viability of this spotter in non-lockdown scenarios several parameters were considered. Mean marks were compared to traditional versions and Pearson's r correlation coefficients were calculated between online and traditional spotters and between online spotters and overall performance in anatomy modules. A survey was carried out to determine the students' view of the assessment. Pearson's r was between 0.33 and 0.49 when comparing online spotters to the traditional format, and between 0.65 and 0.75 (p < 0.01) when compared to a calculated anatomy score. The survey indicated overall student satisfaction as 82.5% reported that it was a fair way to test their knowledge and 55% reported the same or lower levels of anxiety when compared to traditional spotters. However, there was nothing to indicate that the students preferred this format over laboratory-based spotters. These results indicate that this new exam format would be useful for small cohorts who are undertaking online or hybrid courses, or in circumstances when running a full spotter is too costly, and represents a fair and robust way to assess practical anatomical knowledge online.

Tablet-assisted objective structured spotter practical examination (TOSSPE): Advantages of an innovative anatomy spotter examination method for medical student assessment

The affordances of technology-based assessments, like the objectively structured practical examination, have become an integral part of gross anatomy courses. The Department of Anatomy Faculty of Medicine at the University of Warmia and Mazury developed and introduced an application for tablet devices which has been implemented in student examinations and assessments, called the tablet-assisted objective structured spotter practical examination. It was created to simplify the educational process and to build a rich learning environment, facilitating deep learning for students through examination and feedback data. The method consists of cadaver stations with traditional corresponding pin spotters in an expanded tablet application. It not only provides instant feedback on various observations of teaching-learning skills but has also positively affected the entire process of education. The method provides an unbiased evaluation of knowledge and understanding of the anatomy course, ensuring objectivity and standardization. The current study was performed on a total of 608 first-year medical students in Polish and English divisions and focused on the observed advantages since the new method was introduced. Outcomes indicate that after the implementation of the method for both the Polish and English divisions' first-year medical students, the mean score of examinations significantly increased compared to other teaching-learning methods. The study highlights that students were excited about the implementation of the new method and identified its many benefits. It is recognized that technological development and the digital environment offer a range of opportunities and added value versus traditional assessment activities, methods, and processes.

Comparison of the agreement between WeChat-based self-administered and interviewer-administered data on infant and young child feeding in China: A test-retest study

Background

Measuring infant and young child feeding (IYCF) indicators is important in evaluating child health programs and making evidence-based decisions. With Internet and new media rapidly developing, communication apps such as WeChat (the most popular mobile social media platform in China) are widely used and can potentially be used as an alternative way to collect infant and young child feeding information. This study compares data agreement between a WeChat-based self-administered and an interviewer-administered survey on infant and young child feeding information.

Methods

We recruited 297 mothers of children aged 6-23 months in Fenxi County, Shanxi Province, China. Using the Test-Retest method, we first collected data through a self-administered survey using a WeChat-based electronic questionnaire and asked 36 questions on breastfeeding and complementary feeding knowledge, practices, and information sources. We then conducted an interviewer-administered survey using the same questionnaire and compared the data agreement between the two survey methods during the same day. Cohen's kappa score (κ) and intraclass correlation coefficients (ICC) were used for data agreement analysis for all 36 questions and six key IYCF indicators. The McNemar test was used to identify differences between the two survey methods for the six key indicators.

Results

There was substantial or almost perfect agreement for 33 questions (κ/ICC>0.60), and slight or fair agreement for the other 3 questions (κ/ICC<0.40). Agreement of all six key IYCF indicators was substantial or almost perfect (κ = 0.78-0.94), while two indicators showed statistical differences between the two survey methods (P = 0.03 for “Minimum meal frequency” and P = 0.001 for “Minimum accepted diet”). Analysis of reasons for inconsistencies showed that 43.6% of all the inconsistencies were not caused by the self-administered survey method. The cost of the interviewer-administered survey was much higher than that of the self-administered survey: ¥45.9 (US$6.8) vs ¥19.7 (US$2.9) per questionnaire.

Conclusions

The WeChat-based self-administered method can be used for future data collection of infant and young child feeding information in China. Most of the questions and key indicators showed very good agreement without statistical differences between the two methods.

Low-stakes performance testing in Germany by the VERA assessment: analysis of the mode effects between computer-based testing and paper-pencil testing

The German school system employs centrally organized performance assessments (some of which are called “VERA”) as a way of promoting lesson development. In recent years, several German federal states introduced a computer-based performance testing system which will replace the paper-pencil testing system in the future. Scores from computer-based testing are required to be equivalent to paper-pencil testing scores so that the new testing medium does not lead to disadvantages for students. Therefore, the current study aimed at investigating the size of the mode effect and the moderating impact of students’ gender, academic achievement and mainly spoken language in everyday life. In addition, the variance of the mode effect across tasks was investigated. The study was conducted in four German federal states in 2019 using a field experimental design. The test scores of 5140 eighth-graders from 165 schools in the subject German were analysed. The results of multi-level modelling revealed that students’ test scores in the computerized version of the VERA test were significantly lower than in the paper-pencil version. Students with a lower academic achievement were more disadvantaged by the VERA computerized test. The results were inconsistent regarding the interactions between testing mode and students’ gender and mainly spoken language in everyday life. The variance of the mode effect across tasks was high. Research into different subjects and in other federal states and countries under different testing conditions might yield further evidence about the generalizability of these results.

How Computer-Assisted Learning Influences Medical Students' Performance in Anatomy Courses

Anatomy is an essential subject of the medical curriculum. Despite its relevance, the curricular time and logistical resources devoted to teaching anatomy are in decline, favoring the introduction of new pedagogical approaches based on computer-assisted learning (CAL). This new pedagogical approach provides an insight into students' learning profiles and features, which are correlated with knowledge acquisition. The aim of this study was to understand how training with CAL platforms can influence medical students' anatomy performance. A total of 611 medical students attending Musculoskeletal Anatomy (MA) and Cardiovascular Anatomy (CA) courses were allocated to one of three groups (MA Group, CA Group, and MA + CA Group). An association between the performance in these anatomy courses and the number of CAL training sessions was detected. In the MA Group (r = 0.761, P < 0.001) and the MA + CA Group (r = 0.786, P < 0.001), a large positive correlation was observed between musculoskeletal anatomy performance and the number of CAL training sessions. Similarly, in the CA Group (r = 0.670, P < 0.001) and the MA + CA Group (r = 0.772, P < 0.001), a large positive correlation was observed between cardiovascular anatomy performance and the number of CAL training sessions. Multiple linear regression models were performed, considering either musculoskeletal or cardiovascular anatomy performance as the dependent variable. The results suggest that using CAL platforms to study has a positive dose-dependent effect on anatomy performance. Understanding students' individual features and academic background may contribute to the optimization of the learning process.

The Role of Anatomy Computer-Assisted Learning on Spatial Abilities of Medical Students

Currently, medical education context poses different challenges to anatomy, contributing to the introduction of new pedagogical approaches, such as computer-assisted learning (CAL). This approach provides insight into students' learning profiles and skills that enhance anatomy knowledge acquisition. To understand the influence of anatomy CAL on spatial abilities, a study was conducted. A total of 671 medical students attending Musculoskeletal (MA) and Cardiovascular Anatomy (CA) courses, were allocated to one of three groups (MA Group, CA Group, MA + CA Group). Students' pre-training and post-training spatial abilities were assessed through Mental Rotations Test (MRT), with scores ranging between 0-24. After CAL training sessions, students' spatial abilities performance improved (9.72 ± 4.79 vs. 17.05 ± 4.57, P < 0.001). Although male students in both MA Group and CA Group show better baseline spatial abilities, no sex differences were found after CAL training. The improvement in spatial abilities score between sessions (Delta MRT) was correlated with Musculoskeletal Anatomy training sessions in MA Group (r = 0.333, P < 0.001) and MA + CA Group (r = 0.342, P < 0.001), and with Cardiovascular Anatomy training sessions in CA Group (r = 0.461, P = 0.001) and MA + CA Group (r = 0.324, P = 0.001). Multiple linear regression models were used, considering the Delta MRT as dependent variable. An association of Delta MRT to the amount of CAL training and the baseline spatial abilities was observed. The results suggest that CAL training in anatomy has positive dose-dependent effect on spatial abilities.