Using multimedia in large-scale computer-based testing programs

Concept and Implementation of a Two-Stage Coding Scheme for the Development of Computer-Based Testing (CBT)-Items in Traditional Test Software

Computer-based testing (CBT) is gaining importance for studies addressing the diagnosis of competencies, because it is possible to simulate authentic action situations and may reduce the effort of analyzing the data. This benefit is most important for the phase of item design. In this phase of assessment construction, the pattern of answers of a sample is used to draw conclusions on the functionality of the items. Currently, there are no standards for the encodement of items which consider the specifications of CBT-instruments. These specifications are, for example, the a posteriori non-variability of the coding, a lack of information when using conventional test scores and the need of standardization of different formats of items. Taking these specifications into consideration, this paper proposes and discusses a two-stage coding systematization for CBT-items. For this, a distinction between item-coding and answer-coding was done. The coding is discussed for single-section and multi-section formats as well as dichotomous and polytomous answer modes. Therefore, this paper is for users of CBT-instruments who want to achieve the optimal information value of their test results with efficient coding.

The promise and challenge of including multimedia items in medical licensure examinations: some insights from an empirical trial

BACKGROUND

The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items.

METHOD

Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively.

RESULTS

Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept.

CONCLUSIONS

Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.

Computer-based testing: initial report of extensive use in a medical school curriculum

BACKGROUND

Computer-based digital technology offers many opportunities in medical education. One type of digital technology, computer-based testing (CBT), has potentially wide application in undergraduate, graduate, and continuing medical education.

DESCRIPTION

This articles describes the development of CBT in an undergraduate medical curriculum.

EVALUATION

An early step in evaluating CBT is to be sure that the exam format is measuring the examinees' knowledge and not their comfort level or confidence with the technology. It is, therefore, important that the CBT reproduce or accommodate traditional test-taking behavior. CBT also provides for a number of enhancements not easily achieved with traditional paper-and-pencil exams. These include easier control and editing of exam items, better incorporation of testing into the learning environment using specific feedback, and enhancing the questions by incorporating multimedia. CBT does present some unique challenges in testing security, and this article discusses the approach to the security issues. In addition, before initiating CBT into a medical curriculum, the institution must understand the technical and infrastructure requirements for CBT.

CONCLUSIONS

By providing a 5-year experience with CBT in the medical curriculum, this article hopes to facilitate discussions among medical educators in its appropriate application and evaluation.