Measuring assessment standards in undergraduate medical programs: Development and validation of AIM tool

Impact of COVID Pandemic and Hybrid teaching on Final year MBBS students' End of clerkship Exam performance

BACKGROUND & OBJECTIVES

The novel coronavirus (SAR-CoV-2) pandemic has revolutionized medical education worldwide. Most medical schools have adopted the online teaching and assessments. Students attending modified clerkships and assessments under the stress of the pandemic, perform and score differently when compared to normal clerkships. We aimed to identify the impact of COVID-19 on final year MBBS students' EOC (End of Clerkship) examination by comparing them with their scores prior to the COVID and with scores of the previous final year.

METHODS

This cross-sectional study was conducted at Shifa College of Medicine. Final year MBBS students' scores of years 2019 and 2020 were included. Students' EOC MCQ and OSCE scores were compared in pre-COVID and COVID affected rotations of the same year and with the previous year (2019). Data were analyzed in SPSS version 21, means scores were calculated, and one-way ANOVA was applied. Pearson correlation was calculated for correlation assessment of MCQ and OSCE scores.

RESULTS

There were 118 students. The mean EOC, OSCE, and MCQ scores in rotations one to four were 72.8±6.4, 73.3± 8.1, 71.6± 7.4, 72.7± 6.7 and 44.4± 8, 47.2± 8.4, 46.1± 8.2, 48.8± 8.1, respectively. One-way ANOVA results before and after COVID lockdown were statistically insignificant (p=0.3) for OSCE and significant for MCQ in the final year class of 2020 (p=0.001). The Pearson correlation assessment between MCQ and OSCE scores (n=416) had a significant positive correlation (r=0.42, p=0.000). The overall comparison between scores of the final year class of 2019 and 2020 showed significant improvement in Surgery and Obstetrics/Gynae scores in 2020.

CONCLUSION

During the COVID pandemic, the final year students' performance in EOC MCQ and OSCE over all remained unaffected.

Development and validation of teacher and student questionnaires measuring inhibitors of curriculum viability

BACKGROUND

Curriculum viability is determined by the degree to which quality standards have or have not been met, and by the inhibitors that affect attainment of those standards. The literature reports many ways to evaluate whether a curriculum reaches its quality standards, but less attention is paid to the identification of viability inhibitors in different areas of the curriculum that hamper the attainment of quality. The purpose of this study is to develop and establish the reliability and validity of questionnaires that measure the presence of inhibitors in an undergraduate medical curriculum.

METHODS

Teacher and student questionnaires developed by the authors were sent to medical educationalists for qualitative expert validation and to establish their content validity. To establish the response process validity, cognitive interviews were held with teachers and students to clarify any confusion about the meaning of items in the questionnaires. Reliability and construct validity of the questionnaires were established by responses from 575 teachers and 247 final-year medical students.

RESULTS

Qualitative expert validation was provided by 21 experts. The initial teacher and student questionnaires containing respectively 62 items to measure 12 theoretical constructs, and 28 items to measure 7 constructs, were modified to improve their clarity and relevance. The overall scale validity index for the questionnaires was, in order, .95 and .94. Following the cognitive interviews, the resultant teacher and student questionnaires were reduced to respectively 52 and 23 items. Furthermore, after the confirmatory analysis, the final version of the teacher questionnaire was reduced to 25 items to measure 6 constructs and the student questionnaire was reduced to 14 items to measure 3 constructs. Good-for-fit indices were established for the final model and Cronbach alphas of, in order, .89 and .81 were found for the teacher and student questionnaire.

CONCLUSION

The valid and reliable curriculum viability inhibitor questionnaires for teachers and students developed in this study can be used by medical schools to identify inhibitors to achieve standards in different areas of the curriculum.

Curriculum Viability Indicators: A Delphi Study to Determine Standards and Inhibitors of a Curriculum

Curriculum evaluation is typically done by using quality standards defined by accrediting bodies. This does not include inhibitors that hinder the achievement of standards. Hence, to address both standards and inhibitors, we have coined the new concept of "curriculum viability." This study establishes consensus among experts on curriculum viability indicators, i.e. standards and inhibitors, and aims to provide a framework for evaluating the curriculum viability. The study was done in two phases. In the first phase, a consensus was established on the curriculum viability indicators using the Modified Delphi Technique using two rounds. In the first round of the Delphi process, 25 experts participated, which were reduced to 19 in the second round. After two rounds, experts developed a consensus on 40 out of 44 items. These included 27 standards and 13 inhibitors. In the second phase, 18 experts rank-ordered the indicators according to their relative importance in the areas of educational content and strategies, faculty, leadership, assessment, students, educational/working environment, communication, and technology. The list of indicators provides a framework for evaluating the curriculum viability, and their ordering enables curriculum managers to prioritize them during curriculum evaluation.

Technology Enhanced Assessment (TEA) in COVID 19 Pandemic

Online teaching and learning is not a new phenomenon. For the last many years, it has been mainly used as a part of face to face teaching. Assessment is an essential part of teaching and learning, as it establishes the achievement of course learning outcomes by the students. Computer-based assessment is in place for a long time now, however, online assessments have been less practiced. This is because of the issues of validity, reliability and dishonesty. During the COVID 19 pandemic, the educational environment has taken a paradigm shift in many medical schools, both nationally and internationally. This situation demands a method of assessment that is safe, valid, reliable, acceptable, feasible and fair. This paper describes the different formats of online assessment and their application in formative and summative assessments during and after the COVID 19 pandemic.