Artificial Intelligence for Student Assessment: A Systematic Review

Programmatic assessment for learning: A programmatically designed assessment for the purpose of learning: AMEE Guide No. 174

Programmatic assessment for learning (PAL) involves programmatically structured collection of assessment data for the purpose of learning. In this guide, we examine and provide recommendations on several aspects: First, we review the evolution that has led to the development of programmatic assessment, providing clarification of some of its terminology. Second, we outline the learning processes that guide the design of PAL, including distributed learning, interleaving, overlearning, and test-enhanced learning. Third, we review the evolving nature of validity and provide insights into validity from a program perspective. Finally, we examine opportunities, challenges, and future directions of assessment in the context of artificial intelligence.

The promises and challenges of AI-based chatbots in language education through the lens of learner emotions

The integration of AI-based chatbots in language education has garnered significant attention, yet the interplay between chatbots and positive psychology remains underexplored. Filling this gap through a critical analysis of existing theories, measurement scales, and empirical evidence, this paper evaluates the potential benefits and drawbacks of incorporating AI chatbots in language learning environments and how AI chatbots may positively or negatively impact emotional dimensions of language acquisition. The findings unravel that the primary advantages of the AI chatbots are personalized instruction with rapid feedback, a decrease in anxiety levels and a surge in motivation, greater learner independence and self-directed learning, and the fostering of metacognitive abilities. Conversely, the identified obstacles encompass restricted emotional awareness, a deficiency in genuine human interaction, ethical dilemmas and privacy issues, as well as the potential reinforcement of biases and stereotypes. By highlighting the importance of learner emotions in the language learning process, this conceptual analysis review underscores the need for a nuanced understanding of how AI chatbots can support or hinder emotional engagement and motivation. The paper discusses the impacting factors of AI-based chatbots in language education, and strategies for addressing challenges and optimizing chatbot-learner interactions, such as incorporating affective computing techniques and designing culturally-sensitive chatbots. Finally, the article outlines future research directions, emphasizing the need for validated emotion scales in chatbot assisted language learning contexts, longitudinal studies, mixed-methods research, comparative analyses, and investigations into the role of chatbots in fostering emotional intelligence and intercultural competence.

Intelligent Tutoring Systems, Generative Artificial Intelligence (AI), and Healthcare Agents: A Proof of Concept and Dual-Layer Approach

This study introduces a novel methodology for enhancing intelligent tutoring systems (ITS) through the integration of generative artificial intelligence (GenAI) and specialized AI agents. We present a proof of concept (PoC) demo that implements a dual-layer GenAI validation approach that utilizes multiple large language models to ensure the reliability and pedagogical integrity of the AI-generated content. The system features role-specific AI agents, a GenAI-powered scoring mechanism, and an AI mentor that provides periodic guidance. This approach demonstrates capabilities in dynamic scenario generation and real-time adaptability while addressing key challenges in AI-driven education, such as personalization, scalability, and domain-specific knowledge integration. Although exemplified here through a case study in healthcare root cause analysis, the methodology is designed for broad applicability across various fields. Our findings suggest that this approach has significant potential for advancing adaptive learning and personalized instruction while raising important considerations regarding ethical AI application in education. This work provides a foundation for further research into the efficacy and impact of GenAI-enhanced ITS on learning outcomes and instructional design across diverse educational domains.

AI in medical education: uses of AI in construction type A MCQs

BACKGROUND

The introduction of competency-based education models, student centers, and the increased use of formative assessments have led to demands for high-quality test items to be used in assessments. This study aimed to assess the use of an AI tool to generate MCQs type A and evaluate its quality.

METHODS

The study design was cross-sectional analytics conducted from June 2023 to August 2023. This study utilized formative TBL. The AI tool (ChatPdf.com) was selected to generate MCQs type A. The generated items were evaluated using a questionnaire for subject experts and an item (psychometric) analysis. The questionnaire to the subject experts about items was formed based on item quality and rating of item difficulty.

RESULTS

The total number of recurrent staff members as experts was 25, and the questionnaire response rate was 68%. The quality of the items ranged from good to excellent. None of the items had scenarios or vignettes and were direct. According to the expert's rating, easy items represented 80%, and only two had moderate difficulty (20%). Only one item out of the two moderate difficulties had the same difficulty index. The total number of students participating in TBL was 48. The mean mark was 4.8 ± 1.7 out of 10. The KR20 is 0.68. Most items were of moderately difficult (90%) and only one was difficult (10%). The discrimination index of the items ranged from 0.77 to 0.15. Items with excellent discrimination represented 50% (5), items with good discrimination were 3 (30%), and only one time was poor (10%), and one was none discriminating. The non-functional distractors were 26 (86.7%), and the number of non-functional distractors was four (13.3%). According to distractor analysis, 60% of the items were excellent, and 40% were good. A significant correlation (p = 0.4, r = 0.30) was found between the difficulty and discrimination indices.

CONCLUSION

Items constructed using AI had good psychometric properties and quality, measuring higher-order domains. AI allows the construction of many items within a short time. We hope this paper brings the use of AI in item generation and the associated challenges into a multi-layered discussion that will eventually lead to improvements in item generation and assessment in general.

An Automated Virtual Reality Training System for Teacher-Student Interaction: A Randomized Controlled Trial

BACKGROUND

Shortages in qualified supervision and other resources prevent education personnel from rehearsing effective practices. Interactive simulations, although increasingly used in education, frequently require instructor management. Automated simulations rarely engage trainees in skills related to practice (eg, speech).

OBJECTIVE

We evaluated the capability of delivering behavioral skills training through an automated virtual reality (VR) simulation using artificial intelligence to improve the implementation of a nondirective mathematical questioning strategy.

METHODS

We recruited and randomly assigned 30 college-aged participants to equivalent treatment (ie, lecture, modeling, and VR; 15/30, 50%) and control groups (ie, lecture and modeling only; 15/30, 50%). The participants were blind to treatment conditions. Sessions and assessments were conducted face to face and involved the use of VR for assessment regardless of the condition. Lessons concerned the use of a nondirective mathematical questioning strategy in instances where a simulated student provided correct or incorrect answers to word problems. The measures included observed and automated assessments of participant performance and subjective assessments of participant confidence. The participants completed the pretest, posttest, and maintenance probes each week over the course of 3 weeks.

RESULTS

A mixed ANOVA revealed significant main effects of time (F_2,27=124.154; P<.001; η_p²=0.816) and treatment (F_1,28=19.281; P<.001; η_p²=0.408) as well as an interaction effect (F_2,28=8.429; P<.001; η_p²=0.231) for the average percentage of steps in the questioning procedure. Posttest scores for the intervention group (mean 88%, SD 22.62%) exceeded those of the control group (mean 63.33%, SD 22.64%), with t₂₈=3.653, P<.001, and Cohen d=1.334. Maintenance scores indicated a positive effect of the intervention (mean 83.33%, SD 24.40%) relative to the control (mean 54.67%, SD 15.98%), t₂₈=3.807, P<.001, Cohen d=1.39. A Mann-Whitney U test indicated that the treatment groups' self-ratings of confidence (mean 2.41, SD 0.51) were higher than those of the control group (mean 2.04, SD 0.52), U=64, P=.04, r=0.137.

CONCLUSIONS

The results demonstrate the potential of artificial intelligence-augmented VR to deliver effective, evidence-based training with limited instructor management. Additional work is needed to demonstrate the cascading effect of training on authentic practice and to encompass a wider range of skills.

Use of Latent Dirichlet Allocation and Structural Equation Modeling in Determining the Factors for Continuance Intention of Knowledge Payment Platform

Knowledge payment is a new type of E-learning that has developed in the era of social media. With the influence of the COVID-19 epidemic, the knowledge payment market is developing rapidly. Exploring the influencing factors of users’ continuance intention is beneficial for the sustainable development of knowledge payment platforms. Our study took “Himalayan FM” as an example and included two studies: Study 1 used latent dirichlet allocation (LDA) to explore the main factors affecting the users’ willingness to continue use, through mining user comment data on the knowledge payment platform; Study 2 constructed the conceptual model by integrating the technology acceptance model (TAM) and IS success model (IS) and carried out empirical analysis by SPSS and AMOS using the data that were collected through the questionnaire. The results show that: (1) perceived usefulness, user satisfaction, and spokesperson identity have a direct positive impact on users’ willingness to continuous use, while perceived cost has a direct negative impact on users’ willingness to continue use; (2) perceived ease of use, content quality, and system quality of knowledge payment platforms impacted user satisfaction directly, then affected users’ willingness to continue use indirectly; (3) users’ perceived enjoyment, membership experience, auditory experience, and other factors also directly impacted user satisfaction, affecting users’ willingness to continue use indirectly. This study effectively expands the factors influencing knowledge payment users’ willingness to continue use and provides a useful reference for the sustainable development of knowledge payment platforms.

The Use of Mixed, Augmented and Virtual Reality in History of Art Teaching: A Case Study

The incorporation of mixed, virtual, and augmented reality into the educational context takes place in this study through the development of a 3D object shaped by the artistic expressions of the Church of the Annunciation in Seville. In a study of an experimental and exploratory nature with a single group, we worked with a total of 20 students enrolled in a Master’s degree, taught at Seville University, under the title of “Arte: idea y producción” (Art: idea and production). A questionnaire based on the “Technology Acceptance Model” (TAM) was used to ascertain the degree of acceptance that the utilized technology had created among students after their participation in the experience. Likewise, the questionnaire permitted understanding of the assessment made by students concerning the presented objects. Among the results obtained, it is noticeable that the participating students show a high level of acceptance of augmented and virtual reality technologies, alongside favorable attitudes towards their utilization and the intention to use them. It is worth highlighting as a significant conclusion that the exploratory study was performed within a real classroom situation, suggesting that both technologies can be applied in formal training environments.

Artificial intelligence (AI) library services innovative conceptual framework for the digital transformation of university education

Purpose

Artificial intelligence (AI) is one of the latest digital transformation (DT) technological trends the university library can use to provide library users with alternative educational services. AI can foster intelligent decisions for retrieving and sharing information for learning and research. However, extant literature confirms a low adoption rate by the university libraries in using AI to provide innovative alternative services, as this is missing in their strategic plan. The research develops (AI-LSICF) an artificial intelligence library services innovative conceptual framework to provide new insight into how AI technology can be used to deliver value-added innovative library services to achieve digital transformation. It will also encourage library and information professionals to adopt AI to complement effective service delivery.

Design/methodology/approach

This study adopts a qualitative content analysis to investigate extant literature on how AI adoption fosters innovative services in various organisations. The study also used content analysis to generate possible solutions to aid AI service innovation and delivery in university libraries.

Findings

This study uses its findings to develop an Artificial Intelligence Library Services Innovative Conceptual Framework (AI-LSICF) by integrating AI applications and functions into the digital transformation framework elements and discussed using a service innovation framework.

Research limitations/implications

In research, AI-LSICF helps increase an understanding of AI by presenting new insights into how the university library can leverage technology to actualise innovation in service provision to foster DT. This trail will be valuable to scholars and academics interested in addressing the application pathways of AI library service innovation, which is still under-explored in digital transformation.

Practical implications

In practice, AI-LSICF could reform the information industry from its traditional brands into a more applied and resolutely customer-driven organisation. This reformation will awaken awareness of how librarians and information professionals can leverage technology to catch up with digital transformation in this age of the fourth industrial revolution.

Social implications

The enlightenment of AI-LSICF will motivate library professionals to take advantage of AI's potential to enhance their current business model and achieve a unique competitive advantage within their community.

Originality/value

AI-LSICF development serves as a revelation, motivating university libraries and information professionals to consider AI in their strategic plan to enable technology to support university education. This act will enable alternative service delivery in the face of unforeseen circumstances like technological disruption and the present global COVID-19 pandemic that requires non-physical interaction.

Mixed, Augmented and Virtual, Reality Applied to the Teaching of Mathematics for Architects

This paper examines the possibilities of Mixed Reality, the combination of two emerging technologies—Augmented Reality and Virtual Reality—in university education. For this purpose, an object was elaborated in Mixed Reality that underwent the evaluation of 44 first-year students from the degree in architecture who were enrolled in the subject “Mathematical Foundations for Architecture.” The instrument utilized was based on the TAM model, which analyzes the degree of acceptance of the technology used. The analysis of the responses provided by students supported the 23 hypotheses formulated in this study. It was found that MR significantly influences the perceived usefulness and ease of use. The results imply that MR utilization has positive effects on the mathematical teaching-learning processes in architecture from the students’ perception of their mastery of technology. It becomes necessary to offer support to those university teachers who promote the use of active MR-based methodologies in classrooms.