Identification of technical item flaws leads to improvement of the quality of single best Multiple Choice Questions

Dissecting knowledge, guessing, and blunder in multiple choice assessments

Multiple choice results are inherently probabilistic outcomes, as correct responses reflect a combination of knowledge and guessing, while incorrect responses additionally reflect blunder, a confidently committed mistake. To objectively resolve knowledge from responses in an MC test structure, we evaluated probabilistic models that explicitly account for guessing, knowledge and blunder using eight assessments (>9,000 responses) from an undergraduate biotechnology curriculum. A Bayesian implementation of the models, aimed at assessing their robustness to prior beliefs in examinee knowledge, showed that explicit estimators of knowledge are markedly sensitive to prior beliefs with scores as sole input. To overcome this limitation, we examined self-ranked confidence as a proxy knowledge indicator. For our test set, three levels of confidence resolved test performance. Responses rated as least confident were correct more frequently than expected from random selection, reflecting partial knowledge, but were balanced by blunder among the most confident responses. By translating evidence-based guessing and blunder rates to pass marks that statistically qualify a desired level of examinee knowledge, our approach finds practical utility in test analysis and design.

Improving Summative Assessment Through a Resource-Efficient Faculty Review Process

Committee reviews improve the quality of multiple-choice question (MCQ) exams; however, such review processes are typically highly resource-intensive and time-consuming. We report a review process that requires limited faculty time and administrative resources. A small committee reviewed selected items (14-20%) of the final exams of six independent block courses taken by first- and second-year medical students. This process resulted in a significant increase in the item discrimination of reviewed questions in all exams. Our findings support the utility of a review process and may offer health profession educators a more practical and efficient approach for improving the quality of in-house MCQ exams.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-022-01631-9.

Multiple-Choice Tests: A-Z in Best Writing Practices

Multiple-choice tests are the most used method of assessment in medical education. However, there is limited literature in medical education and psychiatry to inform the best practices in writing good-quality multiple-choice questions. Moreover, few physicians and psychiatrists have received training and have experience in writing them. This article highlights the strategies in writing high-quality multiple-choice items and discusses some common flaws that can impact validity and reliability of the assessment examinations.

Assessing the educational impact of cognitive level of MCQ and SEQ on learning approaches of dental students

Objectives:

MCQ’s and SEQ’s are the most widely used assessment tool in dental colleges across Pakistan. This study explores the impact of assessment tool: MCQ’s and SEQ’s on learning approach of dental students and also identifies correlation between these assessment tools and deep & surface learning approaches in integrated and discipline based curriculum.

Methods:

A quantitative co-relational study was conducted in 2018 on 2nd and 4th year dental students. A pre-validated “Revised Study Process Questionnaire” was used. Spearman’s rho correlation coefficient and Wilcoxon signed ranks test were applied to determine the relationship between learning approaches and assessment tools. Internal consistency was calculated by Cronbach’s alpha.

Results:

Ninety six students out of one hundred and fifty completed the questionnaire. Correlation test showed that surface approach correlates significantly with MCQ’s (0.73) while no co-relation exists with SEQ’s (-0.14) in our study. Deep approach has a strong and significant correlation with SEQ’s (0.80) as compared to MCQ’s (0.056).

Conclusion:

Assessment tool has an impact on learning approaches used by the students. It was concluded that students used to prefer deep learning approach while preparing for SEQ’s as they were designed at higher cognitive level, whereas, they preferred surface approach while preparing for MCQ’s as they were developed at low cognitive order.

Are Multiple Choice Questions for Post Graduate Dental Entrance Examinations Spot On?-Item Analysis of MCQs in Prosthodontics in India

BACKGROUND

Construction of appropriate test items is a challenge in preparing quality multiple choice questions. Item analysis provides valuable feedback data on validity of multiple choice questions. The present study was conducted to evaluate the difficulty index, discrimination index and distracter efficiency of the items present in the multiple choice questions of post graduate dental entrance examinations.

METHODS

A list consisting of 20 MCQs was taken from the entrance exam books of MCQs on an introductory topic and administered to 104 undergraduate students.

RESULTS

In the present study 15% of the MCQs related to impression making procedure were difficult with difficulty index (p) less than 30%, 15% were poor discriminators and 55% had at least one non-functional distracter.

CONCLUSION

Item analysis of MCQs in post graduate entrance examinations demonstrated low difficulty index, discrimination index and distracter efficiency. Hence, we propose a strong need for faculty training in test constructors and their post validation.

Psychometric analysis of Anatomy MCQs in Modular examination

Objective

To analyze the psychometric indices of Anatomy question items in modular system assessment.

Methods

A quantitative study was done to determine the quality of MCQs and to analyze the performance of 1^st year 100 MBBS students. Each module covers different subjects of MBBS curriculum but psychometric analysis was done on the subject of Anatomy only. The assessment results of 3 modules were taken and checked by item analysis to see the mean differences between the modules using ANOVA. Post hoc analysis was determined by using Tukey HSD test.

Results

A total of 140 one best (OB) Anatomy MCQ items were calculated for difficulty index, discriminatory index and reliability. Difficulty index was found to be higher in module I when compared with module II and III. Discriminatory index comparatively showed higher results in module II whereas reliability of module III was significantly higher than the other modules. Results were considered to be significant with p value≤ 0.05.

Conclusions

The psychometric analysis of Anatomy MCQs showed average difficulty, good discrimination and reliability.

How-to-guide for writing multiple choice questions for the pharmacy instructor

BACKGROUND

Writing multiple choice questions (MCQ) takes a lot of practice. Often, pharmacy practitioners lack the training to write effective MCQ. Sources for instruction in effective MCQ writing can be overwhelming with numerous suggestions of what should and should not be done.

PURPOSE

The following guide is prepared to serve as a succinct reference for creation and revision of MCQ by both novice and seasoned pharmacy faculty practitioners.

METHODS

The literature is summarized into 12 best practices for writing effective MCQ. Pharmacy specific examples that demonstrate violations of best practices and how they can be corrected are provided.

IMPLICATIONS

The guide can serve as a primer to write new MCQ, as a reference to revise previously created questions, or as a guide to peer review of MCQ.

Faculty development programs improve the quality of Multiple Choice Questions items' writing

The aim of this study was to assess the utility of long term faculty development programs (FDPs) in order to improve the quality of multiple choice questions (MCQs) items' writing. This was a quasi-experimental study, conducted with newly joined faculty members. The MCQ items were analyzed for difficulty index, discriminating index, reliability, Bloom's cognitive levels, item writing flaws (IWFs) and MCQs' nonfunctioning distractors (NFDs) based test courses of respiratory, cardiovascular and renal blocks. Significant improvement was found in the difficulty index values of pre- to post-training (p = 0.003). MCQs with moderate difficulty and higher discrimination were found to be more in the post-training tests in all three courses. Easy questions were decreased from 36.7 to 22.5%. Significant improvement was also reported in the discriminating indices from 92.1 to 95.4% after training (p = 0.132). More number of higher cognitive level of Bloom's taxonomy was reported in the post-training test items (p<0.0001). Also, NFDs and IWFs were reported less in the post-training items (p<0.02). The MCQs written by the faculties without participating in FDPs are usually of low quality. This study suggests that newly joined faculties need active participation in FDPs as these programs are supportive in improving the quality of MCQs' items writing.