Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes

Joint modeling of action sequences and action time in computer-based interactive tasks

Process data refers to data recorded in computer-based assessments that reflect the problem-solving processes of participants and provide greater insight into how they solve problems. Action time, namely the amount of time required to complete a state transition, is also included in such data along with actions. In this study, an action-level joint model of action sequences and action time is proposed, in which the sequential response model (SRM) is used as the measurement model for action sequences, and a new log-normal action time model is proposed as the measurement model for action time. The proposed model can be regarded as an extension of the SRM by incorporating action time within the joint-hierarchical modeling framework and as an extension of the conventional item-level joint models in process data analysis. Results of the empirical and simulation studies demonstrated that the model setup was justified, model parameters could be interpreted, parameter estimates were accurate, and taking into account participants' action time further was beneficial for obtaining a deep understanding of participants' behavioral patterns. Overall, the proposed action-level joint model provides an innovative modeling framework for analyzing process data in computer-based assessments from the latent variable modeling perspective.

Biclustering of Log Data: Insights from a Computer-Based Complex Problem Solving Assessment

Computer-based assessments provide the opportunity to collect a new source of behavioral data related to the problem-solving process, known as log file data. To understand the behavioral patterns that can be uncovered from these process data, many studies have employed clustering methods. In contrast to one-mode clustering algorithms, this study utilized biclustering methods, enabling simultaneous classification of test takers and features extracted from log files. By applying the biclustering algorithms to the "Ticket" task in the PISA 2012 CPS assessment, we evaluated the potential of biclustering algorithms in identifying and interpreting homogeneous biclusters from the process data. Compared with one-mode clustering algorithms, the biclustering methods could uncover clusters of individuals who are homogeneous on a subset of feature variables, holding promise for gaining fine-grained insights into students' problem-solving behavior patterns. Empirical results revealed that specific subsets of features played a crucial role in identifying biclusters. Additionally, the study explored the utilization of biclustering on both the action sequence data and timing data, and the inclusion of time-based features enhanced the understanding of students' action sequences and scores in the context of the analysis.

Investigating Planning and Non-Targeted Exploration in PIAAC 2012: Validating Their Measures Based on Process Data and Investigating Their Relationships with Problem-Solving Competency

Problem-solving is a critical aspect of intelligence that has become increasingly important in modern society. Mapping out the determinants of success in problem-solving helps understand the underlying cognitive processes involved. This article focuses on two key cognitive processes in problem-solving: non-targeted exploration and planning. We generalize previously defined indicators of planning and non-targeted exploration across tasks in the 2012 Programme for the International Assessment of Adult Competencies and examine the internal construct validity of the indicators using confirmatory factor analysis. We also investigate the relationships between problem-solving competency, planning, and non-targeted exploration, along with the specific dependence between indicators from the same task. The results suggest that (a) the planning indicator across tasks provides evidence of internal construct validity; (b) the non-targeted exploration indicator provides weaker evidence of internal construct validity; (c) overall, non-targeted exploration is strongly related to problem-solving competency, whereas planning and problem-solving competencies are weakly negatively related; and (d) such relationships vary substantially across tasks, emphasizing the importance of accounting for the dependency of measures from the same task. Our findings deepen our understanding of problem-solving processes and can support the use of digital tools in educational practice and validate task design by comparing the task-specific relationships with the desired design.

A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks

Early detection of risk of failure on interactive tasks comes with great potential for better understanding how examinees differ in their initial behavior as well as for adaptively tailoring interactive tasks to examinees' competence levels. Drawing on procedures originating in shopper intent prediction on e-commerce platforms, we introduce and showcase a machine learning-based procedure that leverages early-window clickstream data for systematically investigating early predictability of behavioral outcomes on interactive tasks. We derive features related to the occurrence, frequency, sequentiality, and timing of performed actions from early-window clickstreams and use extreme gradient boosting for classification. Multiple measures are suggested to evaluate the quality and utility of early predictions. The procedure is outlined by investigating early predictability of failure on two PIAAC 2012 Problem Solving in Technology Rich Environments (PSTRE) tasks. We investigated early windows of varying size in terms of time and in terms of actions. We achieved good prediction performance at stages where examinees had, on average, at least two thirds of their solution process ahead of them, and the vast majority of examinees who failed could potentially be detected to be at risk before completing the task. In-depth analyses revealed different features to be indicative of success and failure at different stages of the solution process, thereby highlighting the potential of the applied procedure for gaining a finer-grained understanding of the trajectories of behavioral patterns on interactive tasks.

Understanding students' problem-solving patterns: Evidence from an allotted response time in a PISA 2012 item

Understanding students' learning characteristics is central to successfully designing student-centered learning. Particularly in the problem-solving area, it is vital to know that students can possess their styles to solve problems, which should be considered central to addressing adaptive learning. To date, analyzing students' learning characteristics has been mainly based on their final answers. However, there is a limit to understanding the thinking process of students with the correct answer, because their responses are, de facto, singular and identical. With this background, we propose an approach for investigating students' cognitive behavior in problem-solving using response time in the process data. In this paper, we analyzed an item in Programme for International Student Assessment 2012 Creative Problem Solving (CP038q1). We analyzed log data from the PISA CPS item Ticket encompassing 30,453 students (7,196 students with the correct answer and 23,257 students with incorrect answers) from 42 countries. We found that students with the correct answer are categorized into four clusters, and the problem-solving patterns of each cluster are distinguishable. We also showed the internal validity of this approach by confirming that students with incorrect answers can also be similarly classified. Our results indicate that allotted response time in an item can shed light on several distinguished problem-solving patterns, which implies that adaptive learning and feedback are vital for them.