Utility of a multimodal computer-based assessment format for assessment with a higher degree of reliability and validity

Multiple choice questions (MCQs) suffer from cueing, item quality and factual knowledge testing. This study presents a novel multimodal test containing alternative item types in a computer-based assessment (CBA) format, designated as Proxy-CBA. The Proxy-CBA was compared to a standard MCQ-CBA, regarding validity, reliability, standard error of measurement, and cognitive load, using a quasi-experimental crossover design. Biomedical students were randomized into two groups to sit a 65-item formative exam starting with the MCQ-CBA followed by the Proxy-CBA (group 1, n = 38), or the reverse (group 2, n = 35). Subsequently, a questionnaire on perceived cognitive load was taken, answered by 71 participants. Both CBA formats were analyzed according to parameters of the Classical Test Theory and the Rasch model. Compared to the MCQ-CBA, the Proxy-CBA had lower raw scores (p < 0.001, η² = 0.276), higher reliability estimates (p < 0.001, η² =  0.498), lower SEM estimates (p < 0.001, η² = 0.807), and lower theta ability scores (p < 0.001, η² = 0.288). The questionnaire revealed no significant differences between both CBA tests regarding perceived cognitive load. Compared to the MCQ-CBA, the Proxy-CBA showed increased reliability and a higher degree of validity with similar cognitive load, suggesting its utility as an alternative assessment format.

The Utilization of ExamSoft®-iPad® Technology in Administering and Grading Anatomy Practical Examinations

This article provides a detailed, step-by-step description of the introduction of technology (iPad® use of ExamSoft® web-based assessment software) for administering and grading cadaver-oriented anatomy practical examinations. We explain the advantages of computer-based testing in the delivery and grading of practical examinations, including minimizing human error inherent in hand grading, improving the efficiency of reporting, and mapping the examination to course objectives. Furthermore, item analysis (including item difficulty, discrimination indices, and point biserial correlations) allows evaluation of laboratory teaching and testing and monitoring of the curriculum. We also address how the advantages of this format override any limitation, such as cost. In conclusion, we describe our future directions and how others could adopt this approach.

Comparison of long-menu and single-best-answer multiple choice questions in computer-based summative assessments: a randomised controlled trial

BACKGROUND

Little is known regarding the psychometric properties of computerized long-menu formats in comparison to classic formats. We compared single-best-answer (Type A) and long-menu formats using identical question stems during the computer-based, summative, intermediate clinical-clerkship exams for nine disciplines.

METHODS

In this randomised sequential trial, we assigned the examinees for every summative exam to either the Type A or long-menu format (four different experimental questions, otherwise identical). The primary outcome was the power of discrimination. The study was carried out at the Faculty of Medicine, University of Geneva, Switzerland, and included all the students enrolled for the exams that were part of the study. Examinees were surveyed about the long-menu format at the end of the trial.

RESULTS

The trial was stopped for futility (p = 0.7948) after 22 exams including 88 experimental items. The long-menu format had a similar discriminatory power but was more difficult than the Type A format (71.45% vs 77.80%; p = 0.0001). Over half of the options (54.4%) chosen by the examinees in long-menu formats were not proposed as distractors in the Type A formats. Most examinees agreed that their reasoning strategy was different.

CONCLUSIONS

In a non-selected population of examinees taking summative exams, long-menu questions have the same discriminatory power as classic Type A questions, but they are slightly more difficult. They are perceived to be closer to real practice, which could have a positive educational impact. We would recommend their use in the final years of the curriculum, within realistic key-feature problems, to assess clinical reasoning and patient management skills.

Development of a student-driven information technology support service

BACKGROUND AND PURPOSE

To share our experience of a partnership created amongst students, staff, and faculty in order to address a gap in campus information technology (IT) customer services provided to students.

EDUCATIONAL ACTIVITY AND SETTING

Student reliance on a complex educational technology ecosystem requires a robust IT infrastructure; however, campus IT services are often stretched in terms of their capacity to deliver immediate customer support. Compounding this problem is the inability of campus IT services to address issues arising from pharmacy education specific hardware or software. A student help desk (SHD), a student-initiated technology user group was developed. The support provided by the SHD covers student devices to the level of ensuring access to required curriculum technological resources.

FINDINGS

Over 24 months, a total of 259 cases were addressed by the SHD. When examining the type of submissions, the top five requested categories included computer-based assessment, e-mail synchronization, curricular management software synchronization, wireless printing and encryption. These results suggest the perceived value and confidence by students and faculty in the service provided by the SHD.

SUMMARY

The use of a SHD helped to resolve technology issues faced by students for curriculum engagement. Regardless of the challenges institutions may face in delivering their curriculum, students have the desire to be engaged in the governance of their curriculum. By creating a collaborative triad, this represents one example of how student motivation can be leveraged to conquer not just gaps in IT customer service, but potentially other programmatic issues within an institution.

Early motor repertoire in very low birth weight infants in India is associated with motor development at one year

BACKGROUND

Most studies on Prechtl's method of assessing General Movements (GMA) in young infants originate in Europe.

AIM

To determine if motor behavior at an age of 3 months post term is associated with motor development at 12 months post age in VLBW infants in India.

METHODS

243 VLBW infants (135 boys, 108 girls; median gestational age 31wks, range 26-39wks) were video-recorded at a median age of 11wks post term (range 9-16wks). Certified and experienced observers assessed the videos by the "Assessment of Motor Repertoire - 2-5 Months". Fidgety movements (FMs) were classified as abnormal if absent, sporadic or exaggerated, and as normal if intermittently or continually present. The motor behaviour was evaluated by repertoire of co-existent other movements (age-adequacy) and concurrent motor repertoire. In addition, videos of 215 infants were analyzed by computer and the variability of the spatial center of motion (C_SD) was calculated. The Peabody Developmental Motor Scales was used to assess motor development at 12 months.

RESULTS

Abnormal FMs, reduced age adequacy, and an abnormal concurrent motor repertoire were significantly associated with lower Gross Motor and Total Motor Quotient (GMQ, TMQ) scores (p < 0.05). The C_SD was higher in children with TMQ scores <90 (-1SD) than in children with higher TMQ scores (p = 0.002).

CONCLUSION

Normal FMs (assessed by Gestalt perception) and a low variability of the spatial center of motion (assessed by computer-based video analysis) predicted higher Peabody scores in 12-month-old infants born in India with a very low birth weight.

Development and validation of a computer-based situational judgement test to assess medical students' communication skills in the field of shared decision making

OBJECTIVE

To develop a computer-based test (CBT) measuring medical students' communication skills in the field of shared decision making (SDM) and to evaluate its construct validity.

METHODS

The CBT was developed in the context of an experimental study comparing three different trainings for SDM (including e-learning and/or role-play) and a control group. Assessment included a CBT (Part A: seven context-poor questions, Part B: 15 context-rich questions) and interviews with two simulated patients (SP-assessment). Cronbach's α was used to test the internal consistency. Correlations between CBT and SP-assessment were used to further evaluate construct validity of the CBT.

RESULTS

Seventy-two students took part in the study. Mean value for the CBT score was 72% of the total score. Cronbach's α was 0.582. After eliminating three items, Cronbach's α increased to 0.625. Correlations between the CBT and SP-assessment were low to moderate. The control group scored significantly lower than the training settings (p<0.001).

CONCLUSION

The CBT was reliable enough to test for group differences. For summative assessment purposes, considerably more questions would be needed.

PRACTICE IMPLICATIONS

We encourage teachers who particularly work with large student numbers to consider CBT as a feasible assessment method for cognitive aspects of communication skills.

Assessing clinical reasoning (ASCLIRE): Instrument development and validation

Clinical reasoning is an essential competency in medical education. This study aimed at developing and validating a test to assess diagnostic accuracy, collected information, and diagnostic decision time in clinical reasoning. A norm-referenced computer-based test for the assessment of clinical reasoning (ASCLIRE) was developed, integrating the entire clinical decision process. In a cross-sectional study participants were asked to choose as many diagnostic measures as they deemed necessary to diagnose the underlying disease of six different cases with acute or sub-acute dyspnea and provide a diagnosis. 283 students and 20 content experts participated. In addition to diagnostic accuracy, respective decision time and number of used relevant diagnostic measures were documented as distinct performance indicators. The empirical structure of the test was investigated using a structural equation modeling approach. Experts showed higher accuracy rates and lower decision times than students. In a cross-sectional comparison, the diagnostic accuracy of students improved with the year of study. Wrong diagnoses provided by our sample were comparable to wrong diagnoses in practice. We found an excellent fit for a model with three latent factors-diagnostic accuracy, decision time, and choice of relevant diagnostic information-with diagnostic accuracy showing no significant correlation with decision time. ASCLIRE considers decision time as an important performance indicator beneath diagnostic accuracy and provides evidence that clinical reasoning is a complex ability comprising diagnostic accuracy, decision time, and choice of relevant diagnostic information as three partly correlated but still distinct aspects.