Evaluation of the effect of items' format and type on psychometric properties of sixth year pharmacy students clinical clerkship assessment items

An assessment of preclinical removable prosthodontics based on multiple-choice questions: Stakeholders' perceptions

PURPOSE

Item analysis of multiple-choice questions (MCQs) is an essential tool for identifying items that can be stored, revised, or discarded to build a quality MCQ bank. This study analyzed MCQs based on item analysis to develop a pool of valid and reliable items and investigate stakeholders' perceptions regarding MCQs in a written summative assessment (WSA) based on this item analysis.

METHODS

In this descriptive study, 55 questions each from 2016 to 2019 of WSA in preclinical removable prosthodontics for fourth-year undergraduate dentistry students were analyzed for item analysis. Items were categorized according to their difficulty index (DIF I) and discrimination index (DI). Students (2021-2022) were assessed using this question bank. Students' perceptions of and feedback from faculty members concerning this assessment were collected using a questionnaire with a five-point Likert scale.

RESULTS

Of 220 items when both indices (DIF I and DI) were combined, 144 (65.5%) were retained in the question bank, 66 (30%) required revision before incorporation into the question bank, and only 10 (4.5%) were discarded. The mean DIF I and DI values were 69% (standard deviation [Std.Dev] = 19) and 0.22 (Std.Dev = 0.16), respectively, for 220 MCQs. The mean scores from the questionnaire for students and feedback from faculty members ranged from 3.50 to 4.04 and from 4 to 5, respectively, indicating that stakeholders tended to agree and strongly agree, respectively, with the proposed statements.

CONCLUSION

This study assisted the prosthodontics department in creating a set of prevalidated questions with known difficulty and discrimination capacity.

Functional near-infrared spectroscopy of medical students answering various item types

Background

Traditionally, the effect of assessment item types including true/false questions (TFQs), multiple-choice questions (MCQs), short answer questions (SAQs), and case scenario questions (CSQs) is examined through psychometric qualities or student interviews. However, brain activity while answering such questions or items remains unknown. Functional near-infrared spectroscopy (fNIRS) can be used to safely measure cerebral cortex hemodynamic response during various tasks. Hence, this fNIRS study aimed to determine differences in frontotemporal cortex activity as medical students answered TFQs, MCQs, SAQs, and CSQs.

Methods

In total, 24 medical students (13 males and 11 females) were recruited in this study during their mid-psychiatry posting. Oxy-hemoglobin and deoxy-hemoglobin levels in the frontal and temporal regions were measured with a 52-channel fNIRS system. Participants answered 9-18 trials under each of the four types of tasks that were based on their psychiatry curriculum during fNIRS measurements. The area under the oxy-hemoglobin curve (AUC) for each participant and each item type was derived. Repeated measures ANOVA with post-hoc Bonferroni-corrected pairwise comparisons were used to determine differences in oxy-hemoglobin AUC between TFQs, MCQs, SAQs, and CSQs.

Results

Oxy-hemoglobin AUC was highest during the CSQs, followed by SAQs, MCQs, and TFQs in both the frontal and temporal regions. Statistically significant differences between different types of items were observed in oxy-hemoglobin AUC of the frontal region (p ≤ 0.001). Oxy-hemoglobin AUC in the frontal region was significantly higher during the CSQs than TFQ (p = 0.005) and during the SAQ than TFQ (p = 0.025). Although the percentage of correct responses was significantly lower in MCQ than in the other item types, there was no correlation between the percentage of correct response and oxy-hemoglobin AUC in both regions for all four item types (p > 0.05).

Conclusion

CSQs and SAQs elicited greater hemodynamic response than MCQs and TFQs in the prefrontal cortex of medical students. This suggests that more cognitive skills may be required to answer CSQs and SAQs.

Implications of introducing case based radiological images in anatomy on teaching, learning and assessment of medical students: a mixed-methods study

BACKGROUND

Introducing radiological anatomy in the preclinical curriculum can increase the understanding of Anatomy. Regardless of the integration when teaching anatomy, it is essential to maintain oversight as to what and how much is being taught. In addition, the knowledge requirements for preclinical students should be considered. The purpose of this kind of integration is that the student should be able to apply the knowledge which can help them better understand anatomy and not to make the course more challenging. This study aimed to understand whether adding radiological images would increase the difficulty level of the questions.

METHODS

We introduced radiological images, including X Rays, CT scans and MRIs, when teaching anatomy in the preclinical curriculum. A class of 99 students were tested using A-type MCQs (n = 84). All 84 questions were categorized on whether they were case-based with or without a radiological image. The item analysis of both groups of test questions was then compared based on their difficulty and discrimination index. A qualitative student perception regarding the inclusion of radiological images in anatomy was also measured using a questionnaire with a 5-point Likert scale.

RESULTS

The results showed that the performance level of the students was similar when comparing the test questions in both groups. The item analysis of the MCQs in the two groups revealed that by integrating radiological images when teaching anatomy, the various parameters in both groups of test questions were in the same range. More than 80% of the students felt that radiological images facilitate the achievement of learning outcomes and help to apply their knowledge in clinical contexts. The study's findings reported that the rate of satisfaction by including radiological images when teaching anatomy is high.

CONCLUSION

Recognition and interpretation of images are essential in an undergraduate medical program. Students found it helpful when radiological images were introduced to them when teaching anatomy. Since the students' performance in summative exams in both groups of questions was in the same range, the findings also point out that adding radiological images when teaching anatomy does not increase the difficulty of the subject.

Average and individual differences between the 12-item MOS Short-form Health Survey version 2 (SF-12 V.2) and the veterans RAND 12-item Health Survey (VR-12) in the Chinese population

BACKGROUND

The 12-item MOS Short-form Health Survey version 2 (SF-12v2) and the Veterans RAND 12-item Health Survey (VR-12) are generic health-related quality of life measures. They are fairly similar, but their differences in scores have not been assessed. Therefore, this study aimed to assess the differences between the SF-12v2 and the VR-12 in a Chinese population.

METHODS

We conducted a household survey of 500 Chinese adults in Hong Kong. Both the SF-12v2 and the VR-12 were self-administered. The physical component summary score (PCS) and the mental component summary score (MCS) of each instrument were computed using well established algorithms. Their mean differences were assessed using 95% confidence interval (CI), and their individual differences were assessed by Bland-Altman analysis.

RESULTS

The participants had a mean age of 38 years (range: 18-80 years). The mean PCS and MCS scores of the SF-12v2 were 50.3 (SD = 6.5) and 49.0 (SD = 9.0), while those of the VR-12 were 49.6 (SD = 6.2) and 49.7 (SD = 8.8), respectively. The corresponding paired differences (SF-12v2-VR-12) of the PCS and MCS were 0.8, 95% CI (0.4-1.1) and - 0.7, 95% CI (- 1.2 to - 0.2), respectively. All confidence limits fell within the minimal clinical important difference (MCID) of 3. The 95% limits of agreement were - 7.0, 8.5 for PCS and - 11.2, 9.9 for MCS, which fell outside the corresponding MCID for individual responses.

CONCLUSION

The SF-12v2 and the VR-12 reached mean equivalence at the group sample level, but there was a range of individual differences.