Evaluation of performance and perceptions of electronic vs. paper multiple-choice exams

Computerized testing in reading comprehension skill: investigating score interchangeability, item review, age and gender stereotypes, ICT literacy and computer attitudes

Score interchangeability of Computerized Fixed-Length Linear Testing (henceforth CFLT) and Paper-and-Pencil-Based Testing (henceforth PPBT) has become a controversial issue over the last decade when technology has meaningfully restructured methods of the educational assessment. Given this controversy, various testing guidelines published on computerized testing may be used to investigate the interchangeability of CFLT and PPBT mean scores to corroborate if test takers' testing performance is influenced by the effects of testing administration mode; specifically, if validity and reliability of two versions of the same test are affected. This research was conducted to probe not only score interchangeability across testing modes but also to explore the role of age and gender stereotypes, item review, ICT literacy and attitudes towards computer use as moderator variables in test takers' reading achievement in CFLT. Fifty-eight EFL learners homogeneous in both general English and reading skills assigned into one testing group participated in this study. Three different versions of TOEFL reading comprehension test, Computer Attitude Scale (CAS), and ICT literacy Scale of TOEFL Examinees were used in this crossover quasi-controlled empirical study with a common-person and pretest-posttest design to collect data. The findings demonstrated that although the reading scores of test takers were interchangeable in both CFLT and PPBT versions regarding testing administration modes, they were different regarding item review. Furthermore, no significant interaction was found between age, gender, and ICT literacy and CFLT performance. However, attitudes towards the use of computer led to a significant change in testing achievement on CFLT.

Students' perceptions of computer-based testing using ExamSoft

INTRODUCTION

In fall 2017, West Coast University School of Pharmacy implemented ExamSoft for testing. Three courses in each didactic year employed ExamSoft. Prior to this, courses had Scantron-based exams. We surveyed the students to assess their perception of ExamSoft. We hypothesized that students' inherent bias towards technology affected their perception of ExamSoft.

METHODS

To assess this hypothesis, we conducted a survey of all students. The survey contained questions about comfort with technology and nine questions on students' perceptions of ExamSoft and its usefulness.

RESULTS

The survey responses were stratified according to the preference of respondents towards technology and its use in exams. Respondents were stratified into three groups: tech-embracers, tech-skeptics, and neutral. Our results showed that respondents classified as tech-skeptics tended to have a more negative view of ExamSoft and its perceived impact on their grades than students stratified as tech-embracers or neutral.

CONCLUSIONS

Our study suggests that students' inherent bias towards technology plays an important role in their perception of computer-based testing. Assessing incoming students' comfort with technology and student orientation activities to help acquaint students with new technology could help improve their acceptance of educational technology used for testing.

Pathology Resident Evaluation During the Pandemic: Testing and Implementation of a Comprehensive Online Pathology Exam

Despite global digitization, evaluating pathology trainees by paper exams remains the norm. As new social distancing practices require new ways of administering exams, we assessed the viability of an online format for in-house exams from the resident and examiner perspectives. First, pathology residents participated in a practice exam, while staff who were experienced in creating exams were given an online exam-creation demonstration. Subsequently, residents completed a formal 3-hour online exam comprised of multiple-choice, matching, short answer, and whole slide images in place of the paper exam regularly used to evaluate trainees. The experience of the participants was evaluated by surveys. Eighteen residents completed the practice exam; 67% were receptive to the new format and 94% were in favor of moving to digital exams. Seven staff evaluated the digital format and 6 were in favor of it. For the formal online in-house exam, 20 residents participated and 14 completed the survey. Feedback was generally positive with the most common issue being slow-loading digital slides. Exam scores stratified by postgraduate training years in a statistically significant manner, showing positive correlation with resident training level. The online exam format was preferred over paper exams by trainees, with support from both staff and trainees for a permanent transition. Online exams have clear advantages, but technical issues should be addressed before widespread implementation. Our study demonstrates that online exams are a feasible alternative for trainee assessment, especially in socially distanced environments.

Medical assessment in the age of digitalisation

BACKGROUND

Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed.

METHODS

In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified.

RESULTS

The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions.

CONCLUSIONS

It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Medical assessment in the age of digitalisation

BACKGROUND

Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed.

METHODS

In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified.

RESULTS

The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions.

CONCLUSIONS

It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.

Impact of an electronic-based assessment on student pharmacist performance in a required therapeutics course

INTRODUCTION

The use of technology in the classroom has continued to grow, and with the advancement of classroom management systems and online exam software, there are opportunities to administer exams electronically. This study assessed the impact of electronic-based assessments on examination scores in a required therapeutics course.

METHODS

This was a retrospective, single-centered, observational study including second professional year pharmacy students enrolled in a required, one semester therapeutics course. Four assessments were administered each semester. Lecture content and exam format, a mixture of multiple-choice questions and free response written cases, did not differ significantly between years. Assessments administered during the first two years were printed on paper, while assessments administered during the third and fourth year of the study were all electronic, submitted through a classroom management system. Following institutional review board approval, the change in mean overall examination scores between paper and electronic-based assessments were analyzed.

RESULTS

Of the 948 students enrolled in this study, there was no difference in overall mean scores between paper and electronic-based assessments (74.8% vs. 73.8%). In addition, there was no difference in mean examination scores between overall individual paper and electronic Exam 1 through 4 or overall multiple-choice or free response scores between paper and electronic-based assessments.

CONCLUSIONS

Scores did not differ between paper and electronic-based assessments. From this study, test method does not appear to impact exam results.

Interfacing virtual and face-to-face teaching methods in an undergraduate human physiology course for health professions students

Human physiology is a core physical sciences course for health professions students, such as nurses and exercise science majors. The concepts of human physiology lay the foundation for health professions courses, such as pathophysiology. The National Council Licensing Exam for registered nurses (a timed nursing licensure exam) and the American College of Sports Medicine timed licensure exams for exercise sciences students have a framework consisting of human physiology concepts and are computer adaptive testing (CAT) assessments. This provides a case for electronic testing (in the undergraduate class setting) as a preparatory measure for CAT licensing exams. Case studies have illustrated a high information retention rate, with students completing online homework vs. paper, as well. Additionally, in recent years, virtual laboratories for non-physical science majors have been described as safer and effective for the purposes of educating students in laboratory techniques and experimental measures. Lastly, a successful learning approach utilized by museums has been found to be effective in younger students as well: "touch learning" (tactile learning). It also is important to note that student discussions and the face-to-face teaching dynamic play a critical role in the undergraduate education process. As such, the teaching methodology discussed here combines e-learning, virtual laboratories, tactile learning, and face-to-face didactic instruction of human physiology in developing a course to engage undergraduate health professions students, increase retention of human physiology course materials, and simultaneously prepare students for the CAT assessments that are licensing exams.