Item and Test Analysis to Identify Quality Multiple Choice Questions (MCQs) from an Assessment of Medical Students of Ahmedabad, Gujarat

Item Analysis of Multiple-Choice Question (MCQ)-Based Exam Efficiency Among Postgraduate Pediatric Medical Students: An Observational, Cross-Sectional Study From Saudi Arabia

BACKGROUND

Several modalities of written examination have been employed in medical education, with multiple-choice questions (MCQs) being the most frequently used and preferred format. This underlines the need to regularly assess and monitor the quality of MCQs in medical exams. Such assessment of MCQs helps to ensure that these exams are well‑designed and adequately powered to evaluate students' performance. Hence, the current study assessed the efficiency of an MCQ-based examination, in a cohort of pediatric post-graduate students in Saudi Arabia.

METHODS

This observational, cross-sectional study examined the efficiency of MCQs in terms of their validity, reliability, difficulty index (DFI), discrimination index (DI), and distractor efficiency (DE). The exam consisted of a total of 48 MCQs, 144 distractors, a total score of 48, and no negative marking.

RESULTS

The reliability index of 0.76 showed the consistency and reproducibility of the exam results. The exam had a DFI of 69.77%, indicating an overall moderate level of difficulty. The exam had a balanced mix of 23 easy (47.9%), 20 (41.7%) moderately difficult, and five (10.4%) tough questions. Twenty (41.6%) items had a DI of ≥0.3, indicating good discrimination of high and low performers, while the remaining 28 MCQs (58.3%) had a lower DI of ≤0.19, implying poor discriminative ability. The DE was 81.25%, indicating that the majority of distractors in the exam were functional.

CONCLUSION

To the best of the author's knowledge, this is the first study among post-graduate pediatric students from Saudi Arabia, to present the results of item analysis of an MCQ-based exam. The study highlights the importance of optimizing the quality of MCQs by following established guidelines, to make MCQ-based clinical assessments more effective. It reiterates the importance of a reasonable DFI well aligned with students' knowledge levels, maximum distractor functionality, and an impactful DI, in developing high-quality MCQs.

Comparing students' performance in self-directed and directed self-learning in College of Medicine, University of Bisha

Background

Student-centered learning strategy increases the likelihood of graduation of competent, self-dependent, and problem-solving physicians. The University of Bisha, College of Medicine (UBCOM) adopted self-directed learning (SDL) represented by problem-based learning (PBL), and directed self-learning (DSL) represented by team-based learning (TBL).

Aim

To compare the students' performance in SDL and DSL among UBCOM students.

Methodology

A total of 502 multiple choice questions (MCQs) from the mid-course and final exams were collected by the relevant subject experts from nine courses during the period from September 2020 till June 2023 that adopted PBL and TBL; 247 MCQs related to PBL and 255 related to TBL. Psychometric analysis was used to determine difficult, easy, and optimum questions (≤25%, ≥90%, and 26-89%, respectively). Point biserial as <0.19, 0.20-0.29, 0.30-0.39, and >0.40 which indicate poor, marginal, good, and excellent point biserial, respectively. Finally, the number of functional distractors was attempted by >5% of the candidates.

Results

No significant differences were noted for the students' performance in MCQs related to PBL (representing self-directed, small group learning tool), and TBL (representing directed-self, large group learning tool) regarding difficulty index (DI), point biserial, and distractors functionality.

Conclusion

It has been observed that there is no difference in students' performance whether PBL or TBL is used for learning Basic Medical Science courses. Small group learning such as PBL needs more resources in comparison to large group learning as in TBL, therefore any institute can decide on the adopted learning strategy depending on its resources and the number of students.

An assessment of preclinical removable prosthodontics based on multiple-choice questions: Stakeholders' perceptions

PURPOSE

Item analysis of multiple-choice questions (MCQs) is an essential tool for identifying items that can be stored, revised, or discarded to build a quality MCQ bank. This study analyzed MCQs based on item analysis to develop a pool of valid and reliable items and investigate stakeholders' perceptions regarding MCQs in a written summative assessment (WSA) based on this item analysis.

METHODS

In this descriptive study, 55 questions each from 2016 to 2019 of WSA in preclinical removable prosthodontics for fourth-year undergraduate dentistry students were analyzed for item analysis. Items were categorized according to their difficulty index (DIF I) and discrimination index (DI). Students (2021-2022) were assessed using this question bank. Students' perceptions of and feedback from faculty members concerning this assessment were collected using a questionnaire with a five-point Likert scale.

RESULTS

Of 220 items when both indices (DIF I and DI) were combined, 144 (65.5%) were retained in the question bank, 66 (30%) required revision before incorporation into the question bank, and only 10 (4.5%) were discarded. The mean DIF I and DI values were 69% (standard deviation [Std.Dev] = 19) and 0.22 (Std.Dev = 0.16), respectively, for 220 MCQs. The mean scores from the questionnaire for students and feedback from faculty members ranged from 3.50 to 4.04 and from 4 to 5, respectively, indicating that stakeholders tended to agree and strongly agree, respectively, with the proposed statements.

CONCLUSION

This study assisted the prosthodontics department in creating a set of prevalidated questions with known difficulty and discrimination capacity.

AI in medical education: uses of AI in construction type A MCQs

BACKGROUND

The introduction of competency-based education models, student centers, and the increased use of formative assessments have led to demands for high-quality test items to be used in assessments. This study aimed to assess the use of an AI tool to generate MCQs type A and evaluate its quality.

METHODS

The study design was cross-sectional analytics conducted from June 2023 to August 2023. This study utilized formative TBL. The AI tool (ChatPdf.com) was selected to generate MCQs type A. The generated items were evaluated using a questionnaire for subject experts and an item (psychometric) analysis. The questionnaire to the subject experts about items was formed based on item quality and rating of item difficulty.

RESULTS

The total number of recurrent staff members as experts was 25, and the questionnaire response rate was 68%. The quality of the items ranged from good to excellent. None of the items had scenarios or vignettes and were direct. According to the expert's rating, easy items represented 80%, and only two had moderate difficulty (20%). Only one item out of the two moderate difficulties had the same difficulty index. The total number of students participating in TBL was 48. The mean mark was 4.8 ± 1.7 out of 10. The KR20 is 0.68. Most items were of moderately difficult (90%) and only one was difficult (10%). The discrimination index of the items ranged from 0.77 to 0.15. Items with excellent discrimination represented 50% (5), items with good discrimination were 3 (30%), and only one time was poor (10%), and one was none discriminating. The non-functional distractors were 26 (86.7%), and the number of non-functional distractors was four (13.3%). According to distractor analysis, 60% of the items were excellent, and 40% were good. A significant correlation (p = 0.4, r = 0.30) was found between the difficulty and discrimination indices.

CONCLUSION

Items constructed using AI had good psychometric properties and quality, measuring higher-order domains. AI allows the construction of many items within a short time. We hope this paper brings the use of AI in item generation and the associated challenges into a multi-layered discussion that will eventually lead to improvements in item generation and assessment in general.

Assessment of the Quality of Multiple-Choice Questions in the Surgery Course for an Integrated Curriculum, University of Bisha College of Medicine, Saudi Arabia

INTRODUCTION

Multiple-choice questions (MCQs) have been recognized as reliable assessment tools, and incorporating clinical scenarios in MCQ stems has enhanced their effectiveness in evaluating knowledge and understanding. Item analysis is used to assess the reliability and consistency of MCQs, indicating their suitability as an assessment tool. This study aims to ensure the competence of graduates in serving the community and establish an examination bank for the surgery course.

OBJECTIVE

This study aims to assess the quality and acceptability of MCQs in the surgery course at the University of Bisha College of Medicine (UBCOM).

METHODS

A psychometric study evaluated the quality of MCQs used in surgery examinations from 2019 to 2023 at UBCOM in Saudi Arabia. The MCQs/items were analyzed and categorized for their difficulty index (DIF), discrimination index (DI), and distracter efficiency (DE) Fifth-year MBBS students undergo a rotation in the department and are assessed at the end of 12 weeks. The assessment includes 60 MCQs/items and written items. Data was collected and analyzed using SPSS version 24.

RESULTS

A total of 189 students were examined across five test sessions, with 300 MCQ items. Student scores ranged from 28.33% to 90.0%, with an average score of 64.6%±4.35. The 300 MCQ items had a total of 900 distractors. The DIF was 75.3% for the items, and 63.3% of the items showed good discrimination. No items had negative points in terms of biserial correlation. The mean number of functional distractors per test item was 2.19±1.007, with 34% of the items having three functional distractors.

CONCLUSION

The psychometric indices used to evaluate the MCQs in this study were encouraging, with acceptable DIF, distractor efficiencies, and item reliability. Providing robust faculty training and capacity-building is recommended to enhance item development skills.

The Effect of a One-Day Workshop on the Quality of Framing Multiple Choice Questions in Physiology in a Medical College in India

Background Multiple choice questions (MCQs) are commonly used in medical exams for more objectivity in assessment. However, the quality of the questions should be optimum for a proper assessment of the students. A faculty development program (FDP) may improve the quality of MCQs. The effect of a one-day workshop on framing MCQ as a part of a FDP has not been explored in our institution. Aim This study aimed to evaluate the quality of MCQ in the subject of physiology before and after a one-day workshop on framing MCQ as a part of a FDP. Methods This was a retrospective study conducted in the Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. A one-day workshop on framing MCQ as a part of a FDP was conducted in March 2022. We took 100 MCQs and responses from the students from examinations conducted before the workshop and 100 MCQs and responses from the students after the workshop. In pre-workshop and post-workshop, the same five faculties framed the questions. Post-validation item analysis including difficulty index (DIFI), discrimination index (DI), distractor effectiveness (DE), and Kuder-Richardson Formula 20 (KR-20) for internal consistency was calculated. Results Pre-workshop and post-workshop quality of the MCQ remain equal in terms of DIFI (chi-square {3} = 2.42, P = 0.29), DI (chi-square {3} = 2.44, P = 0.49), and DE (chi-square {3} = 4.97, P = 0.17). The KR-20 in pre-workshop and post-workshop was 0.65 and 0.87, respectively. Both had acceptable internal consistency. Conclusion The one-day workshop on framing MCQs as a part of a FDP did not have a significant impact on the quality of the MCQs as measured by the three indices of item quality but did improve the internal consistency of the MCQs. Further educational programs and research are required to find out what measures can improve the quality of MCQs.

Relations of the Number of Functioning Distractors With the Item Difficulty Index and the Item Discrimination Power in the Multiple Choice Questions

Background Multiple choice questions (MCQ) are used nowadays in summative assessments for certification of courses, in a competitive examination, and for recruitment. A single-best-answer stem-type MCQ is one of the formats most commonly used for this purpose; it contains a question and options from which the examinee needs to select the correct answer. Item analysis is used for the determination of the quality, validity, and reliability of the MCQ. Item difficulty index, item discrimination power, and distractor effectiveness are critical for the quality of the MCQs. Aim This study was conducted to estimate the effect of distractor effectiveness on the item difficulty index and item discriminating power. Method A test paper consisting of 200 single-best-answer stem-type MCQs with four options responded to by 400 medical undergraduates was analyzed for item difficulty index, item discriminating power, and distractor effectiveness with established item analysis formulas. The effect of distractor effectiveness on item difficulty index and item discriminating power was statistically analyzed using Epi-Info 7TM software. Result The mean item difficulty index, item discriminating power, and distractor effectiveness were 57.75% ± 28.65%, 0.4388 ± 0.36, and 84.17%, respectively (p<0.05). The item difficulty index was higher for items with single-functioning distractors in comparison to items with three-functioning distractors. Item discriminating power was higher in items with three functioning distractors in comparison to items with one and two functioning distractors. Conclusion Two or three functioning distractors show an appropriate item difficulty index and item discriminating power. In addition, item discriminating power is lower for the easy MCQs. Two to three distractors are appropriate to construct a quality MCQ.

Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods

BACKGROUND

Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees' responses have to be analyzed and scored to derive information about examinees' true knowledge.

OBJECTIVE

The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees' true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived.

METHODS

Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees' true knowledge using fictitious examinations with 100 single-choice items were calculated.

RESULTS

A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between -3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between -1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between -2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees' true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between -60.0% (95% CI -60% to -60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5.

CONCLUSIONS

In examinations with single-choice items, the scoring result is not always equivalent to examinees' true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.

Development and validation of the Adult cirrhosiS Knowledge Questionnaire

BACKGROUND

Assessing a patient's knowledge regarding liver cirrhosis is important to improve patient outcomes. This study aimed to develop and validate the Adult cirrhosiS Knowledge Questionnaire (ASK-Q) to assess patients' knowledge regarding liver cirrhosis from multiple aspects.

METHODS

A 24-item ASK-Q with four domains: self-understanding (5 items), aetiology (5 items), complications (5 items) and management (9 items) of liver cirrhosis was developed based on literature review and expert panel input. It was then piloted in five English-speaking patients with liver cirrhosis. These patients commented that the font size was too small. Hence, the font was enlarged and the final version of the ASK-Q was administered to English-speaking patients with liver cirrhosis, aged ≥18 years, with or without decompensation, at a tertiary hospital, from September 2020 to November 2021, at baseline and fortnight later. Patients with encephalopathy were excluded.

RESULTS

120/135 patients agreed to participate (response rate = 88.9%). The overall median score was 59.1 (45.6-68.2). A total of 7/22 (31.8%) items were "easy", 14/22 (63.6%) items were "moderately easy" and 1/22 (4.5%) items were "difficult". Exploratory factor analysis extracted nine factors, and two items were omitted. The ASK-Q was able to discriminate the knowledge level of patients with and without tertiary education [59.1 (50.0-72.7) vs. 54.5 (36.4-63.6); P < 0.05]. The overall Kuder-Richardson coefficient was 0.760, indicating adequate internal consistency. At retest, 77/120 patients participated (response rate = 64.2%) and 15/22 items were not statistically significant, indicating adequate reliability.

CONCLUSIONS

The ASK-Q was found to be a valid and reliable questionnaire for evaluating the knowledge of liver cirrhosis among English-speaking adult patients.

Evaluation of Multiple-Choice Questions by Item Analysis, from an Online Internal Assessment of 6th Semester Medical Students in a Rural Medical College, West Bengal

Background:

Properly constructed single best-answer multiple choice questions (MCQs) or items assess higher-order cognitive processing of Bloom's taxonomy and accurately discriminate between high and low achievers. However, guidelines for writing good test items are rarely followed, leading to generation and application of faulty MCQs.

Materials and Methods:

During lockdown period in 2020, internal assessment was taken through online mode using Google Forms. There were 60 'single response type' MCQs, each consisting of single stem and four options including one correct answer and three distractors. Each item was analyzed for difficulty index (Dif I), discrimination index (DI), and distractor efficiency (DE).

Results:

The mean of achieved marks was 42.92± (standard deviation [SD], 5.07). Dif I, DI, and DE were 47.95± (SD 16.39) in percentage, 0.12± (SD 0.10), and 18.42± (SD 15.35), respectively. 46.67% of the items were easy and 21.66% were of acceptable discrimination. Very weak negative correlation was found between Dif I and DI. Out of total 180 distractors, 51.66% were nonfunctional one.

Conclusion:

Item analysis and storage of MCQs with their indices provides opportunity for an examiner to select MCQs of appropriate difficulty level as per the need of assessment and decide their placement in the question paper.

Post-validation item analysis to assess the validity and reliability of multiple-choice questions at a medical college with an innovative curriculum

Background In medical education, the need to obtain reliable and valid assessments is critical for the learning process. This study implemented a post-validation item analysis to create a supply of valid questions for incorporation into the question bank. Methods A cross-sectional study was performed in the College of Medicine, University of Bisha, Saudi Arabia. The study was targeting 250 items and 750 distractors from 2017 to 2020. The post-validation item analysis was done to evaluate the quality of the items using test-scoring and reporting software. Data were analysed by SPSS Version 25. Quantitative variables were expressed as mean (SD), while qualitative variables were expressed as number and percentage. An independent t-test was done to reveal the association between the item analysis parameters. A value of p<0.05 was considered statistically significant. Results The mean difficulty index (DIF I), discrimination index (DI) and distractors efficacy (DE) were 73.8, 0.26 and 73.5%, respectively. Of 250 items, 38.8% had an acceptable DIF I (30%-70%) and 66.4% had 'good to excellent' DI (>0.2). Of 750 distractors, 33.6%, 37%, 20% and 9.2% had zero, one, two and three non-functional distractors, respectively. The mean Kuder-Richardson was 0.76. The DIF I was significantly associated with DE (p=0.048). The post-validation item analysis of this study showed that a considerable proportion of questions had acceptable parameters and were recommended for item banking. However, some questions needed to be rephrased and reassessed or discarded. Conclusion Three-option multiple-choice questions should be considered for future examinations to improve the assessment process.

Auditing RadExam: Employing Psychometrics to Improve Exam Quality

INTRODUCTION

RadExam is a question item and exam database jointly developed by the Association of Program Directors in Radiology and the American College of Radiology to provide formative resident assessment, offering performance metrics benchmarked against institutional and national resident performance. Beyond resident performance, data is available on question and exam performance. Despite considerable investment in the education and training of its question writers and editors and meticulous attention to current psychometrically validated methods, it was anticipated a minority of exam questions would still perform poorly. Audits were performed to identify these questions, identify reasons for poor performance, and modify or replace so-affected questions. Exam performance was also assessed.

METHODS

Two audits were performed, the first after the February-May 2018 RadExam pilot phase, and the second nearly 1 year after the full implementation of RadExam. In each audit, RadExam subspecialty editors evaluated all exam questions and exams using statistical data: question and test number of administrations, question p value, question Discrimination Index (DI), question Bloom's taxonomy learning level, exam P-value, and the number of image-based questions in each exam. Identified questions were modified or removed and replaced.

RESULTS

Audit 1 was performed after the administration of 3114 exams comprised of 2520 questions administered across 100 residency programs. Audit 1 identified 617 questions with DI <0.1 and 565 questions with unacceptable P-values, all of which were modified or replaced. Audit 2 was performed after the administration of 16,416 exams, comprised of 2,507 questions. Audit 2 identified 229 questions with DI <0.1 and 290 questions with unacceptable P-values, representing a 49.1% decrease in total flagged questions compared to Audit 1. Statistically significant decreases were seen in questions with both DI and P-values outside of the desired range across nearly all subspecialties.

CONCLUSION

The positive impact of our audit system on question and exam performance was reflected in a significant decrease in the number of questions flagged and improved overall exam performance in Audit 2. This illustrates the positive impact of Audit 1.

Multiple-Choice Tests: A-Z in Best Writing Practices

Multiple-choice tests are the most used method of assessment in medical education. However, there is limited literature in medical education and psychiatry to inform the best practices in writing good-quality multiple-choice questions. Moreover, few physicians and psychiatrists have received training and have experience in writing them. This article highlights the strategies in writing high-quality multiple-choice items and discusses some common flaws that can impact validity and reliability of the assessment examinations.

Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-sectional study

Purpose

Multiple-choice questions (MCQs) are useful in assessing student performance, covering a wide range of topics in an objective way. Its reliability and validity depend upon how well it is constructed. Defective Item detected by item analysis must be looked for item writing flaws and optimized. The aim of this study was to evaluate the MCQs for difficulty levels, discriminating power with functional distractors by item analysis, analyze poor items for writing flaws, and optimize.

Methods

This was a prospective cross-sectional study involving 120 MBBS students writing formative assessment in Ophthalmology. It comprised 40 single response MCQs as a part of 3-h paper for 20 marks. Items were categorized according to their difficulty index, discrimination index, and distractor efficiency with simple proportions, mean, standard deviation, and correlation. The defective items were analyzed for proper construction and optimized.

Results

The mean score of the study group was 13.525 ± 2.617. Mean difficulty index, discrimination index, and distractor efficiency were 53.22, 0.26, and 78.32, respectively. Among 40 MCQs, twenty-five MCQs did not have non-functioning distractor; 7 had one, 5 had two, and 3 had three. Of the 20 defective items, 17 were optimized and added to the question bank, two were added without modification, and one was dropped.

Conclusion

Item analysis is a valuable tool in detecting poor MCQs, and optimizing them is a critical step. The defective items identified should be optimized and not dropped so that the content area covered by the defective item is not kept of the assessment.

All India AYUSH post graduate entrance exam 2019 - AYURVEDA MCQ item analysis

Background

AIAPGET 2019, an all India ranking entrance Test for MD/MS courses of Ayurveda, Unani, Siddha and Homeopathy stream was conducted by joint collaboration of National Testing Agency (NTA) and All India Institute of Ayurveda (AIIA). In this article, we present the item analysis of AIAPGET 2019 Ayurveda stream MCQs.

Objectives

The aim of this article was to analyse the MCQs of AIAPGET 2019 of Ayurveda stream.

Materials and methods

This exam was computer based conducted all over 25 centers across India. The question paper had 100 MCQs with 1 correct answer and 3 distractors for each item (Problem statement).

Results

AIAPGET 2019 question paper of Ayurveda stream had a Difficulty index of 37.32 ± 16.11 Discriminatory Index of 0.46 ± 0.27 and Distractor Index of 89 ± 17.8.

Conclusion

Our analysis showed that though ideal, the question paper trended towards difficulty side.

Item Analysis of Multiple-Choice Questions in Pharmacology in an Indian Medical School

Introduction Student assessment by multiple-choice questions (MCQs) is an integral part of student evaluation in medicine. The medical teacher should be trained to construct an item with proper stem and valid options. Periodic item analyses will make the process of assessment more meaningful. Hence, we conducted the study to analyze MCQs (item analysis) tested on a batch of MBBS students in pharmacology in their three internal assessment examinations.

Methods The study was conducted in the Department of Pharmacology of a medical college in Mangaluru on 150 students. The MCQs of the three internal assessment examinations (20 each) respectively were analyzed. We analyzed each question for difficulty index (DI), discrimination index (DsI), and distracter efficacy or functionality and expressed the percentage results.

Results The DI was in an acceptable range of 60, 75, and 90%, respectively, in the three internal assessments. The percentage of “too difficult” questions was 10, 20, and 10% and the average DsI was 0.32 ± 0.04, 0.28 ± 0.02, and 0.26 ± 0.02, respectively. In the second and third internal assessments, 95% of questions had functional distracters, while in the first internal assessment, only 60% of questions had functional distracters.

Conclusion We conclude from our study that even though the items (MCQs) framed for the internal assessments were in the acceptable range of quality in terms of the parameters assessed, we must improve MCQ’s construction in selecting distracters in some topics.

How important is distractor efficiency for grading Best Answer Questions?

BACKGROUND

Distractor efficiency and the optimum number of functional distractors per item in One Best Answer Questions have been debated. The prevalence of non-functional distractors has led to a reduction in the number of distractors per item with the advantage of adding more items in the test. The existing literature eludes a definite answer to the question of what distractor efficiency best matches excellent psychometric indices. We examined the relationship between distractor efficiency and the psychometric indices of One Best Answer Questions in search of an answer.

METHODS

We analysed 350 items used in 7 professional examinations and determined their distractor efficiency and the number of functional distractors per item. The items were sorted into five groups - excellent, good, fair, remediable and discarded based on their discrimination index. We studied how the distractor efficiency and functional distractors per item correlated with these five groups.

RESULTS

Correlation of distractor efficiency with psychometric indices was significant but far from perfect. The excellent group topped in distractor efficiency in 3 tests, the good group in one test, the remediable group equalled excellent group in one test, and the discarded group topped in 2 tests.

CONCLUSIONS

The distractor efficiency did not correlate in a consistent pattern with the discrimination index. Fifty per cent or higher distractor efficiency, not hundred percent, was found to be the optimum.

Psychometric analysis of multiple-choice questions in an innovative curriculum in Kingdom of Saudi Arabia

Background and Aims:

Worldwide, medical education and assessment of medical students are evolving. Psychometric analysis of the adopted assessment methods is thus, necessary for an efficient, reliable, valid and evidence based approach to the assessment of the students. The objective of this study was to determine the pattern of psychometric analysis of our courses conducted in the academic year 2018-2019, in an innovative curriculum.

Methods:

It was a cross-sectional-design study involving review of examination items over one academic session -2018/2019. All exam item analysis of courses completed within the three phases of the year were analyzed using SPSS V20 statistical software.

Results:

There were 24 courses conducted during the academic year 2018-2019, across the three academic phases. The total examination items were 1073 with 3219 distractors in one of four best option multiple choice questions (MCQs). The item analysis showed that the mean difficulty index (DIF I) was 79.1 ± 3.3. Items with good discrimination have a mean of 65 ± 11.2 and a distractor efficiency of 80.9%. Reliability Index (Kr20) across all exams in the three phases was 0.75. There was a significant difference within the examination items block (F = 12.31, F critical = 3.33, P < 0.05) across all the phases of the courses taken by the students. Similarly, significant differences existed among the three phases of the courses taken (F ratio = 12.44, F critical 4.10, P < 0.05).

Conclusion:

The psychometric analysis showed that the quality of examination questions was valid and reliable. Though differences were observed in items quality between different phases of study as well as within courses of study, it has generally remained consistent throughout the session. More efforts need to be channeled towards improving the quality in the future is recommended.

The Role of Faculty Development in Improving the Quality of Multiple-Choice Questions in Dental Education

Valid and reliable assessment of students' knowledge and skills is integral to dental education. However, most faculty members receive no formal training on student assessment techniques. The aim of this study was to quantify the value of a professional development program designed to improve the test item-writing skills of dental faculty members. A quasi-experimental (pretest, intervention, posttest) study was conducted with faculty members in the dental school of Majmaah University, Saudi Arabia. Data assessed were 450 multiple-choice questions (MCQs) from final exams in 15 courses in 2017 (prior to the intervention; pretest) and the same number in 2018 (after the intervention; posttest). The intervention was a faculty development program implemented in 2018 to improve the writing of MCQs. This training highlighted construct-irrelevant variance-the abnormal increase or decrease in test scores due to factors extraneous to constructs of interest-and provided expert advice to rectify flaws. Item analysis of pre- and post-intervention MCQs determined the difficulty index, discrimination index, and proportion of non-functional distractors for each question. MCQs on 2017 and 2018 exams were compared on each of these parameters. The results showed statistically significant improvements in MCQs from 2017 to 2018 on all parameters. MCQs with low discrimination decreased, those with high discrimination increased, and the proportion of questions with more than two non-functional distractors were reduced. These results provide evidence of improved test item quality following implementation of a long-term faculty development program. Additionally, the findings underscore the need for an active dental education department and demonstrate its value for dental schools.

Development and Validity of a Questionnaire on Dyslipidemia Dietary Knowledge

Objective

Dyslipidemia is one of the most important modifiable risk factors for cardiovascular disease. Diet and lifestyle interventions positively contribute to the management of dyslipidemia. Adequate knowledge of the correct diet for dyslipidemia is an effective preventive strategy against cardiovascular disease.

Method

This study developed a questionnaire to test dyslipidemia dietary knowledge. The initial, multiple-choice dyslipidemia dietary knowledge questionnaire (DDKQ) was formed based on a literature review of previous studies. Two Delphi rounds were performed to ensure content validity and face validity, items were pretested and filtered through item analysis, and the reliability and validity of the final questionnaire were checked.

Results

The DDKQ consisted of 5 parts, with 25 items in total. It demonstrated adequate content validity (0.94), face validity, internal consistency (KR20 = 0.67), and test-retest reliability (0.74). The mean item difficulty (P) was 0.55 and ranged from 0.15 to 0.83, and the mean item discrimination index (D) was 0.36, ranging from 0.21 to 0.58. The questionnaire was also able to distinguish between participants with/without a medical background; medical workers produced significantly higher total scores (16.70±2.84 vs. 14.57±4.26, p< 0.05).

Conclusion

The DDKQ is a reliable and valid measure of dyslipidemia dietary knowledge. It is suitable for providing scientific assessments for targeted health education interventions.

Quantitative analysis of single best answer multiple choice questions in pharmaceutics

INTRODUCTION

The purpose of this study was to: (1) analyze the quality of single best answer multiple choice questions (MCQs) used in pharmaceutics exams, (2) identify the correlation between difficulty index (DIF I), discriminating index (DI), and distractor efficiency (DE), and (3) understand the relationship between DIF I, DI, and DE and the number of MCQ answer options and their cognitive level.

METHODS

429 MCQs used in pharmaceutics exams were analyzed. The quality of the MCQs was evaluated using DIF I, DI, and DE. The number of answer options and the cognitive level tested by each item were evaluated. Relationships between DIF I, DI, and DE were measured using Pearson's correlations and t-tests.

RESULTS

DIF I showed a significant negative correlation with DI within questions that measured information recall. A significant negative correlation between DIF I and DI was observed in questions with four and five answer options regardless of the cognitive level measured. The highest DI values were found in moderate difficulty questions, while the worst DE was observed for the easiest questions. Questions that measured analytical and problem-solving abilities were more difficult than those measuring information recall. Questions with four and five answer options had excellent discrimination.

CONCLUSIONS

Single best answer MCQs are a valuable assessment tool capable of evaluating higher cognitive skills. Significant correlation between DIF I and DI can indicate the examination quality. Higher quality MCQs are constructed using four and five answer options.

Distractor Efficiency in an Item Pool for a Statistics Classroom Exam: Assessing Its Relation With Item Cognitive Level Classified According to Bloom's Taxonomy

Multiple-choice items are one of the most commonly used tools for evaluating students' knowledge and skills. A key aspect of this type of assessment is the presence of functioning distractors, i.e., incorrect alternatives intended to be plausible for students with lower achievement. To our knowledge, no work has investigated the relationship between distractor performance and the complexity of the cognitive task required to give the correct answer. The aim of this study was to investigate this relation, employing the first three levels of Bloom's taxonomy (Knowledge, Comprehension, and Application). Specifically, it was hypothesized that items classified into a higher level of Bloom's classification would show a greater number of functioning distractors. The study involved 174 items administered to a sample of 848 undergraduate psychology students during their statistics exam. Each student received 30 items randomly selected from the 174-item pool. The bivariate results mainly supported the authors' hypothesis: the highest percentage of functioning distractors was observed among the items classified into the Application category (η² = 0.024 and Phi = 0.25 for the dichotomized measure). When the analysis controlled for other item features, it lost statistical significance, partly because of the confounding effect of item difficulty.

Item Analysis of Multiple Choice Questions at the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain

Objectives

The current study aimed to carry out a post-validation item analysis of multiple choice questions (MCQs) in medical examinations in order to evaluate correlations between item difficulty, item discrimination and distraction effectiveness so as to determine whether questions should be included, modified or discarded. In addition, the optimal number of options per MCQ was analysed.

Methods

This cross-sectional study was performed in the Department of Paediatrics, Arabian Gulf University, Manama, Bahrain. A total of 800 MCQs and 4,000 distractors were analysed between November 2013 and June 2016.

Results

The mean difficulty index ranged from 36.70-73.14%. The mean discrimination index ranged from 0.20-0.34. The mean distractor efficiency ranged from 66.50-90.00%. Of the items, 48.4%, 35.3%, 11.4%, 3.9% and 1.1% had zero, one, two, three and four nonfunctional distractors (NFDs), respectively. Using three or four rather than five options in each MCQ resulted in 95% or 83.6% of items having zero NFDs, respectively. The distractor efficiency was 91.87%, 85.83% and 64.13% for difficult, acceptable and easy items, respectively (P <0.005). Distractor efficiency was 83.33%, 83.24% and 77.56% for items with excellent, acceptable and poor discrimination, respectively (P <0.005). The average Kuder-Richardson formula 20 reliability coefficient was 0.76.

Conclusion

A considerable number of the MCQ items were within acceptable ranges. However, some items needed to be discarded or revised. Using three or four rather than five options in MCQs is recommended to reduce the number of NFDs and improve the overall quality of the examination.

An interactive contouring module improves engagement and interest in radiation oncology among preclinical medical students: Results of a randomized trial

PURPOSE

Studies have shown significant gaps in knowledge of radiation therapy among medical students and primary care providers. The goal of this study was to evaluate the effect of an interactive contouring module on knowledge and interest in radiation oncology among preclinical medical students.

METHODS AND MATERIALS

Second-year medical students at the University of California, San Diego were randomized to participate in an interactive contouring exercise or watch a traditional didactic lecture on radiation oncology. Participants completed knowledge tests and surveys at baseline, immediately following the exercise, and 3 months later. Statistical analysis included Wilcoxon signed-rank test for pre- and posttest comparisons and Wilcoxon rank sum test for comparison between groups.

RESULTS

Forty-three medical students participated in the trial (21 in the didactic group; 22 in the contouring group). Students completing the contouring module demonstrated similar overall knowledge improvement compared with the traditional didactic group (+8.6% vs +6.6%, not significant) but endorsed greater engagement on a 5-point Likert-type scale (3.10 vs 3.76, P = .02). At 3-month follow-up, there was a nonsignificant trend toward improved overall knowledge in the contouring group (43% vs 51%, P = .10), with a significance difference in a subset of questions on knowledge of the process of radiation therapy as well as side effects (51% vs 75%, P = .002). Students in the contouring group demonstrated more interest in pursuing a clinical radiation oncology rotation (2.52 vs 3.27, P = .01).

CONCLUSIONS

Use of an interactive contouring module was an effective method to teach preclinical medical students about radiation oncology, with no significant difference in knowledge gained compared with a traditional didactic lecture; however, higher engagement among students completing the contouring module led to improved retention of knowledge of radiation side effects and greater interest in radiation oncology. These data suggest a potential benefit of integrating an interactive radiation oncology module into the preclinical medical school curriculum.

Are Multiple Choice Questions for Post Graduate Dental Entrance Examinations Spot On?-Item Analysis of MCQs in Prosthodontics in India

BACKGROUND

Construction of appropriate test items is a challenge in preparing quality multiple choice questions. Item analysis provides valuable feedback data on validity of multiple choice questions. The present study was conducted to evaluate the difficulty index, discrimination index and distracter efficiency of the items present in the multiple choice questions of post graduate dental entrance examinations.

METHODS

A list consisting of 20 MCQs was taken from the entrance exam books of MCQs on an introductory topic and administered to 104 undergraduate students.

RESULTS

In the present study 15% of the MCQs related to impression making procedure were difficult with difficulty index (p) less than 30%, 15% were poor discriminators and 55% had at least one non-functional distracter.

CONCLUSION

Item analysis of MCQs in post graduate entrance examinations demonstrated low difficulty index, discrimination index and distracter efficiency. Hence, we propose a strong need for faculty training in test constructors and their post validation.

Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study

This study examines the long-term impact of the faculty development programs on the multiple choice question (MCQ) items’ quality leading to study its effect on the students’ overall competency level during their yearly academic assessment. A series of longitudinal highly constructed faculty development workshops were conducted to improve the quality of the MCQs items writing skills. A total of 2207 MCQs were constructed by 58 participants for the assessment of 882 students’ cognitive competency level during the academic years 2012–2015. The MCQs were analyzed for the difficulty index (P-value), discriminating index (DI), presence/absence of item writing flaws (IWFs), and non-functioning distractors (NFDs), Bloom’s taxonomy cognitive levels, test reliability, and the rate of students’ scoring. Significant improvement in the difficulty index and DI were noticed during each successive academic year. Easy and poor discriminating questions, NFDs and IWFs were decreased significantly, whereas distractor efficiency (DE) mean score and high cognitive level (K2) questions were increased substantially during the each successive academic year. Improved MCQs’ quality leaded to increased competency level of the borderline students. Overall, the longitudinal faculty development workshops help in improving the quality of the MCQs items writing skills of the faculty that leads to students’ high competency levels.

Psychometric analysis of Anatomy MCQs in Modular examination

Objective

To analyze the psychometric indices of Anatomy question items in modular system assessment.

Methods

A quantitative study was done to determine the quality of MCQs and to analyze the performance of 1^st year 100 MBBS students. Each module covers different subjects of MBBS curriculum but psychometric analysis was done on the subject of Anatomy only. The assessment results of 3 modules were taken and checked by item analysis to see the mean differences between the modules using ANOVA. Post hoc analysis was determined by using Tukey HSD test.

Results

A total of 140 one best (OB) Anatomy MCQ items were calculated for difficulty index, discriminatory index and reliability. Difficulty index was found to be higher in module I when compared with module II and III. Discriminatory index comparatively showed higher results in module II whereas reliability of module III was significantly higher than the other modules. Results were considered to be significant with p value≤ 0.05.

Conclusions

The psychometric analysis of Anatomy MCQs showed average difficulty, good discrimination and reliability.

Evaluation of Cognitive levels and Item writing flaws in Medical Pharmacology Internal Assessment Examinations

OBJECTIVES

This study aimed to evaluate the cognitive levels of Multiple Choice Questions (MCQs) & Short Answer Questions (SAQs) and types of Item Writing Flaws (IWFs) in MCQs in Medical Pharmacology internal assessment exams.

METHODS

This descriptive, study was conducted over a period of six months, from December 2015 to May 2016 and evaluated six internal assessment examinations comprising SAQs and MCQs. A total of 150 MCQs and 43 SAQs were analyzed. These questions were administered to third-year medical students in the year of 2015. All SAQs were reviewed for their cognitive levels and MCQs were reviewed for cognitive levels as well as for IWFs. Items were classified as flawed if they contained one or more than one flaw. The cognitive level of the questions was determined by the modified Bloom's taxonomy.

RESULTS

The proportion of flawed items out of 150 items in six exams ranged from 16% to 52%. While the percentage of total flawed items was 28%. Most common types of flaws were implausible distractors 19.69% (26), extra detail in correct option 18.18% (24), vague terms 9.85% (13), unfocused stem 9.09% (12) and absolute terms 9.09% (12). The two-third of MCQs 97(64.67%) were assessing the recall of information, while 29 (19.33%) and 24 (16%) were assessing the interpretation of data and problem-solving skills respectively. The majority of the SAQs (90.7%) were assessing recall of the information and only 9.3% were assessing interpretation of data while none of the questions was assessing the problem-solving skills.

CONCLUSIONS

The cognitive level of assessment tools (SAQs & MCQs) is low, and IWFS are common in the MCQs. Therefore, faculty should be urged and groomed to design problem-solving questions which are devoid of any flaws.

Flaws of Multiple Choice Questions in Teacher-Constructed Nursing Examinations: A Pilot Descriptive Study

BACKGROUND

In many middle eastern universities, English is the medium of instruction and testing. As nurse educators construct multiple choice questions (MCQs), it is essential that items are developed to be valid and reliable to assess student learning.

METHOD

This study examined the structure of 98 MCQs included in nursing examinations at three middle eastern universities using a checklist composed of 22 literature-based principles.

RESULTS

Ninety MCQs (91.8%) experienced one or more item-writing flaws. Examples of these included linguistic errors, various problems with the stem, and answer options. Of importance, most faculty did not use item analysis to assess the integrity of the examinations.

CONCLUSION

Results confirm concerns about the standards faculty use for test construction and item analysis. Universities must ensure that the faculty they hired are fluent in English. Faculty would also benefit from workshops that focus on test construction and the use of item analysis. [J Nurs Educ. 2017;56(8):490-496.].

A comparison of clinical-scenario (case cluster) versus stand-alone multiple choice questions in a problem-based learning environment in undergraduate medicine

OBJECTIVES

To compare stand-alone multiple choice questions (MCQs) and integrated clinical-scenario (case cluster) multiple choice questions (CS-MCQs) in a problem-based learning (PBL) environment.

METHODS

A retrospective descriptive analysis of MCQ examinations was conducted in a course that integrates the subspecialties of anatomical pathology, chemical pathology, hematology, immunology, microbiology and pharmacology. The MCQ items were analyzed for their reliability (Kuder-Richardson-20, KR-20), level of difficulty (Pi), discrimination index (Di), item distractors and student performances. The statistical analysis of the results was extracted from the integrity online item-analysis programme. The results of the standard stand-alone and CS multiple choice questions were compared.

RESULTS

KR-20 for the CS-MCQs and stand-alone MCQs was consistently high. KR-20 and Pi were higher for the CS-MCQs. There was no significant difference between the CS-MCQs and stand-alone MCQs in Pi and Di. A range of difficulty levels was found based on Bloom's taxonomy. The mean scores for the class were higher for the CS-MCQ examination. The compilation of the CS-MCQ examination was more challenging.

CONCLUSIONS

CS-MCQs compare favorably to stand-alone MCQs and provide opportunities for the integration of sub-specialties and assessment in keeping with PBL. They assess students' cognitive skills and are reliable and practical. Different levels of item difficulty promote multi-logical and critical thinking. Students' scores were higher for the CS-MCQ examination, which may suggest better understanding of the material and/or better question clarity. The scenarios have to flow logically. Increasing the number of scenarios ensures the examination of more course content.

The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value

BACKGROUND

Failure to adhere to standard item-writing guidelines may render examination questions easier or more difficult than intended. Item complexity describes the cognitive skill level required to obtain a correct answer. Higher cognitive examination items promote critical thinking and are recommended to prepare students for clinical training. This study evaluated faculty-authored examinations to determine the impact of item-writing flaws and item complexity on the difficulty and discrimination value of examination items used to assess third year veterinary students.

METHODS

The impact of item-writing flaws and item complexity (cognitive level I-V) on examination item difficulty and discrimination value was evaluated on 1925 examination items prepared by clinical faculty for third year veterinary students.

RESULTS

The mean (± SE) percent correct (83.3 % ± 17.5) was consistent with target values in professional education, and the mean discrimination index (0.18 ± 0.17) was slightly lower than recommended (0.20). More than one item-writing flaw was identified in 37.3 % of questions. The most common item-writing flaws were awkward stem structure, implausible distractors, longest response is correct, and responses are series of true-false statements. Higher cognitive skills (complexity level III-IV) were required to correctly answer 38.4 % of examination items. As item complexity increased, item difficulty and discrimination values increased. The probability of writing discriminating, difficult examination items decreased when implausible distractors and all of the above were used, and increased if the distractors were comprised of a series of true/false statements. Items with four distractors were not more difficult or discriminating than items with three distractors.

CONCLUSION

Preparation of examination questions targeting higher cognitive levels will increase the likelihood of constructing discriminating items. Use of implausible distractors to complete a five-option multiple choice question does not strengthen the discrimination value.

Item analysis of in use multiple choice questions in pharmacology

Background:

Multiple choice questions (MCQs) are a common method of assessment of medical students. The quality of MCQs is determined by three parameters such as difficulty index (DIF I), discrimination index (DI), and distracter efficiency (DE).

Objectives:

The objective of this study is to assess the quality of MCQs currently in use in pharmacology and discard the MCQs which are not found useful.

Materials and Methods:

A class test of central nervous system unit was conducted in the Department of Pharmacology. This test comprised 50 MCQs/items and 150 distracters. A correct response to an item was awarded one mark with no negative marking for incorrect response. Each item was analyzed for three parameters such as DIF I, DI, and DE.

Results:

DIF of 38 (76%) items was in the acceptable range (P = 30–70%), 11 (22%) items were too easy (P > 70%), and 1 (2%) item was too difficult (P < 30%). DI of 31 (62%) items was excellent (d > 0.35), of 12 (24%) items was good (d = 0.20–0.34), and of 7 (14%) items was poor (d < 0.20). A total of 50 items had 150 distracters. Among these, 27 (18%) were nonfunctional distracters (NFDs) and 123 (82%) were functional distracters. Items with one NFD were 11 and with two NFDs were 8. Based on these parameters, 6 items were discarded, 17 were revised, and 27 were kept for subsequent use.

Conclusion:

Item analysis is a valuable tool as it helps us to retain the valuable MCQs and discard the items which are not useful. It also helps in increasing our skills in test construction and identifies the specific areas of course content which need greater emphasis or clarity.

Stepping up of MCQs' quality through a multi-stage reviewing process

BACKGROUND

Assessment is a crucial and complex process. Thus quality assurance of assessment Methods is essential when assessment is used for the purposes of certification.

AIM

To evaluate the effect of continuous well-structured process of the question bank revision, enlightened by item analysis, in improving the quality of Multiple-Choice Questions (MCQs).

SETTING

The Family Medicine (FM) training certification exam for the Egyptian Board accredited for International Membership of the Royal College of General Practitioners (MRCGP[INT]).

METHOD

The results of the item analysis of Applied Knowledge Tests (AKTs) of two academic years (2009, 2013) were included in the study. The AKT test consisted of two papers, each of 100 MCQS, and blue printed against the FM training curriculum. A total of 226 candidates sat for the two exams; 102 in 2009 and 124 in 2013.

RESULTS

There were more MCQs with moderate difficulty and higher discrimination in 2013. Significant improvement was found in the discrimination index (DI) values in 2013 (p < 0.001). and questions with a high facility and classified as easy decreased from 40.5 to 28.7%. The average number of functioning distractors per item increased from 1.99 in 2009 to 2.19 in 2013 (p = 0.015).

CONCLUSION

Revision of well-constructed MCQs on a regular basis, and in a structured manner, improved the quality of the MCQs and consequently improved the validity of the examination.

Case-based learning and multiple choice questioning methods favored by students

BACKGROUND

Investigating and understanding how students learn on their own is essential to effective teaching, but studies are rarely conducted in this context. A major aim within medical education is to foster procedural knowledge. It is known that case-based questioning exercises drive the learning process, but the way students deal with these exercises is explored little.

METHODS

This study examined how medical students deal with case-based questioning by evaluating 426 case-related questions created by 79 fourth-year medical students. The subjects covered by the questions, the level of the questions (equivalent to United States Medical Licensing Examination Steps 1 and 2), and the proportion of positively and negatively formulated questions were examined, as well as the number of right and wrong answer choices, in correlation to the formulation of the question.

RESULTS

The evaluated case-based questions' level matched the United States Medical Licensing Examination Step 1 level. The students were more confident with items aiming on diagnosis, did not reject negatively formulated questions and tended to prefer handling with right content, while keeping wrong content to a minimum.

CONCLUSION

These results should be taken into consideration for the formulation of case-based questioning exercises in the future and encourage the development of bedside teaching in order to foster the acquisition of associative and procedural knowledge, especially clinical reasoning and therapy-oriented thinking.