Student assessment: Moving over to programmatic assessment

Changing Assessment Scenarios: Lessons for Changing Practice

Assessment is a process that includes ascertainment of improvement in the performance of students over time, motivation of students to study, evaluation of teaching methods, and ranking of student capabilities. It is an important component of the educational process influencing student learning. Although we have embarked on a new curricular model, assessment has remained largely ignored despite being the hallmark of competency-based education. During the earlier stages, the assessment was considered akin to "measurement," believing that competence is "generic, fixed and transferable across content," could be measured quantitatively and can be expressed as a single score. The objective assessment was the norm and subjective tools were considered unreliable and biased. It was soon realized that "competence is specific and nontransferable," mandating the use of multiple assessment tools across multiple content areas using multiple assessors. A paradigm change through "programmatic assessment" only occurred with the understanding that competence is "dynamic, incremental and contextual." Here, information about the students' competence and progress is gathered continually over time, analysed and supplemented with purposefully collected additional information when needed, using carefully selected combination of tools and assessor expertise, leading to an authentic, observation-driven, institutional assessment system. In the conduct of any performance assessment, the assessor remains an important part of the process, therefore making assessor training indispensable. In this paper, we look at the changing paradigms of our understanding of clinical competence, corresponding global changes in assessment and then try to make out a case for adopting the prevailing trends in the assessment of clinical competence.

Blended programmatic assessment for competency based curricula

The uncertainty in all spheres of higher education due to the COVID-19 pandemic has had an unprecedented impact on teaching-learning and assessments in medical colleges across the globe. The conventional ways of assessment are now neither possible nor practical for certifying medical graduates. This has necessitated thoughtful considerations in making adjustments to the assessment system, with most institutions transitioning to online assessments that so far have remained underutilized. Programmatic assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment and at present can be utilized optimally as it ensures the collection of multiple low-stake assessment data which can be aggregated for high-stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance. Though efforts have been made to introduce programmatic assessment in the competency-based undergraduate curriculum, transitioning to online assessment can be a potential opportunity if the basic tenets of programmatic assessment, choice of online assessment tools, strategies, good practices of online assessments and challenges are understood and explored explicitly for designing and implementing online assessments. This paper explores the possibility of introducing online assessment with face-to-face assessment and structuring a blended programmatic assessment in competency-based medical education.

Item analysis of multiple choice questions: A quality assurance test for an assessment tool

Background

The item analysis of multiple choice questions (MCQs) is an essential tool that can provide input on validity and reliability of items. It helps to identify items which can be revised or discarded, thus building a quality MCQ bank.

Methods

The study focussed on item analysis of 90 MCQs of three tests conducted for 150 first year Bachelor of Medicine and Bachelor of Surgery (MBBS) physiology students. The item analysis explored the difficulty index (DIF I) and discrimination index (DI) with distractor effectiveness (DE). Statistical analysis was performed by using MS Excel 2010 and SPSS, version 20.0.

Results

Of total 90 MCQs, the majority, that is, 74 (82%) MCQs had a good/acceptable level of difficulty with a mean DIF I of 55.32 ± 7.4 (mean ± SD), whereas seven (8%) were too difficult and nine (10%) were too easy. A total of 72 (80%) items had an excellent to acceptable DI and 18 (20%) had a poor DI with an overall mean DI of 0.31 ± 0.12. There was significant weak correlation between DIF I and DI (r = 0.140, p < .0001). The mean DE was 32.35 ± 31.3 with 73% functional distractors in all. The reliability measure of test items by Cronbach alpha was 0.85 and Kuder-Richardson Formula 20 was 0.71, which is good. The standard error of measurement was 1.22.

Conclusion

Our study helped teachers identify good and ideal MCQs which can be part of the question bank for future and those MCQs which needed revision. We recommend that item analysis must be performed for all MCQ-based assessments to determine validity and reliability of the assessment.

The power of subjectivity in competency-based assessment

With the introduction of competency-based undergraduate curriculum in India, a paradigm shift in the assessment methods and tools will be the need of the hour. Competencies are complex combinations of various attributes, many of which being not assessable by objective methods. Assessment of affective and communication domains has always been neglected for want of objective methods. Areas like professionalism, ethics, altruism, and communication—so vital for being an Indian Medical Graduate, can be assessed longitudinally applying subjective means only. Though subjectivity has often been questioned as being biased, it has been proven time and again that a subjective assessment in expert hands gives comparable results as that of any objective assessment. By insisting on objectivity, we may compromise the validity of the assessment and deprive the students of enriched subjective feedback and judgement also. This review highlights the importance of subjective assessment in competency-based assessment and ways and means of improving the rigor of subjective assessment, with particular emphasis on the development and use of rubrics.