Experiences from Two Ways of Integrating Pre- and Post-course Multiple-choice Assessment Questions in Educational Events for Surgeons

To examine how to optimise the integration of multiple-choice questions (MCQs) for learning in continuing professional development (CPD) events in surgery, we implemented and evaluated two methods in two subspecialities over multiple years. The same 12 MCQs were administered pre- and post-event in 66 facial trauma courses. Two different sets of 10 MCQs were administered pre- and post-event in 21 small animal fracture courses. We performed standard psychometric tests on responses from participants who completed both the pre- and post-event assessment. The average difficulty index pre-course was 57% with a discrimination index of 0.20 for small animal fractures and 53% with a discrimination index of 0.15 for facial trauma. For the majority of the individual MCQs, the scores were between 30%-70% and the discrimination index was >0.10. The difficulty index post-course increased in both groups (to 75% and 62%). The pre-course MCQs resulted in an average score in the expected range for both formats suggesting they were appropriate for the intended level of difficulty and an appropriate pre-course learning activity. Post-course completion resulted in increased scores with both formats. Both delivery methods worked well in all regions and overall quality depends on applying a solid item development and validation process.

Flaws of Multiple Choice Questions in Teacher-Constructed Nursing Examinations: A Pilot Descriptive Study

BACKGROUND

In many middle eastern universities, English is the medium of instruction and testing. As nurse educators construct multiple choice questions (MCQs), it is essential that items are developed to be valid and reliable to assess student learning.

METHOD

This study examined the structure of 98 MCQs included in nursing examinations at three middle eastern universities using a checklist composed of 22 literature-based principles.

RESULTS

Ninety MCQs (91.8%) experienced one or more item-writing flaws. Examples of these included linguistic errors, various problems with the stem, and answer options. Of importance, most faculty did not use item analysis to assess the integrity of the examinations.

CONCLUSION

Results confirm concerns about the standards faculty use for test construction and item analysis. Universities must ensure that the faculty they hired are fluent in English. Faculty would also benefit from workshops that focus on test construction and the use of item analysis. [J Nurs Educ. 2017;56(8):490-496.].

Does Educator Training or Experience Affect the Quality of Multiple-Choice Questions?

RATIONALE AND OBJECTIVES

Physicians receive little training on proper multiple-choice question (MCQ) writing methods. Well-constructed MCQs follow rules, which ensure that a question tests what it is intended to test. Questions that break these are described as "flawed." We examined whether the prevalence of flawed questions differed significantly between those with or without prior training in question writing and between those with different levels of educator experience.

MATERIALS AND METHODS

We assessed 200 unedited MCQs from a question bank for our senior medical student radiology elective: an equal number of questions (50) were written by faculty with previous training in MCQ writing, other faculty, residents, and medical students. Questions were scored independently by two readers for the presence of 11 distinct flaws described in the literature.

RESULTS

Questions written by faculty with MCQ writing training had significantly fewer errors: mean 0.4 errors per question compared to a mean of 1.5-1.7 errors per question for the other groups (P < .001). There were no significant differences in the total number of errors between the untrained faculty, residents, and students (P values .35-.91). Among trained faculty 17/50 questions (34%) were flawed, whereas other faculty wrote 38/50 (76%) flawed questions, residents 37/50 (74%), and students 44/50 (88%). Trained question writers' higher performance was mainly manifest in the reduced frequency of five specific errors.

CONCLUSIONS

Faculty with training in effective MCQ writing made fewer errors in MCQ construction. Educator experience alone had no effect on the frequency of flaws; faculty without dedicated training, residents, and students performed similarly.