Assessing Critical Thinking in Higher Education: Current State and Directions for Next-Generation Assessment

Linking essay-writing tests using many-facet models and neural automated essay scoring

For essay-writing tests, challenges arise when scores assigned to essays are influenced by the characteristics of raters, such as rater severity and consistency. Item response theory (IRT) models incorporating rater parameters have been developed to tackle this issue, exemplified by the many-facet Rasch models. These IRT models enable the estimation of examinees' abilities while accounting for the impact of rater characteristics, thereby enhancing the accuracy of ability measurement. However, difficulties can arise when different groups of examinees are evaluated by different sets of raters. In such cases, test linking is essential for unifying the scale of model parameters estimated for individual examinee-rater groups. Traditional test-linking methods typically require administrators to design groups in which either examinees or raters are partially shared. However, this is often impractical in real-world testing scenarios. To address this, we introduce a novel method for linking the parameters of IRT models with rater parameters that uses neural automated essay scoring technology. Our experimental results indicate that our method successfully accomplishes test linking with accuracy comparable to that of linear linking using few common examinees.

Assessment of the psychometric properties of the Vietnamese version of the Critical Thinking Disposition Scale

BACKGROUND

The Critical Thinking Disposition Scale is a valid and reliable tool for assessing an individual's attitudes towards critical thinking. However, it is unclear if the scale can be used with Vietnamese nursing students.

OBJECTIVES

This study aimed to translate the scale into Vietnamese and examine the psychometric properties of the Vietnamese version of the Critical Thinking Disposition Scale (CTDS-V).

SETTINGS

Data were collected from senior nursing students at four universities in southern Vietnam in October and December 2022.

PARTICIPANTS

In total, 398 senior nursing students participated in this study.

METHODS

A cross-cultural adaptation of the CTDS-V was carried out with participants obtained through convenience sampling. Data were collected through an online survey. The study evaluated the validity of the CTDS-V by examining the content validity, concurrent validity, and construct validity, employing both Exploratory and Confirmatory Factor Analyses. The study also reported the internal consistency reliability using Cronbach's alpha and test-retest stability of the CTDS-V.

RESULTS

The CTDS-V had a good content validity index with a score of at least 0.85 for each item and an excellent overall content validity index of 0.96. Exploratory Factor Analysis identified two factors, critical openness, and reflective scepticism, with 11 items. Confirmatory Factor Analysis provided an acceptable fit model and supported the factor structure of the original scale. Cronbach's alpha coefficient of the scale was 0.88, and those of the two subscales were each 0.84. The test-retest stability of the CTDS-V was good with an intraclass correlation coefficient of 0.98 (95 % confidence interval = 0.96, 0.99).

CONCLUSIONS

This study indicated that the CTDS-V is a valid and reliable tool for assessing the critical thinking disposition of nursing students. The scale is short, rendering it feasible for regular use in education and research.

Computer-based performance approach for critical thinking assessment in children

BACKGROUND

Critical thinking (CT) has been identified as one of the key skills in the modern world. Despite a variety of CT assessment systems for secondary school and higher education, there is a shortage of CT assessments for children.

AIMS

To fill the gap, we developed a computer-based performance assessment (CPBA) of CT with automatic scoring. In the study, we analysed: (1) To what extent is the internal structure of the critical thinking assessment tool confirmed? (2) What are the patterns of students' behaviour within the CPBA? (3) To what extent is the convergent validity of the CPBA supported?

SAMPLE AND METHODS

The sample consists of 1689 students, aged 9-11 years. To answer the research questions, the following quantitative data analysis methods were selected. The Bayesian Network was applied to investigate the internal structure. Latent Class Analysis was used to investigate the patterns of students' behaviour. To verify the convergent validity, the students were asked to solve the standardized math and language literacy tests.

RESULTS AND CONCLUSIONS

The theoretically expected internal structure of the CPBA was confirmed: the two-factor structure (the Analysis of information and the Making inferences subskills of CT) describes the data appropriately. Moreover, CT was described not as a list but as a sequence of cognitive operations. Four different strategies that lead to different results of CT were revealed. A weak positive correlation between the results of the CPBA and the achievement tests confirmed the test validity.

Constructing a critical thinking evaluation framework for college students majoring in the humanities

Introduction

Education for sustainable development (ESD) has focused on the promotion of sustainable thinking skills, capacities, or abilities for learners of different educational stages. Critical thinking (CT) plays an important role in the lifelong development of college students, which is also one of the key competencies in ESD. The development of a valuable framework for assessing college students' CT is important for understanding their level of CT. Therefore, this study aimed to construct a reliable self-evaluation CT framework for college students majoring in the humanities.

Methods

Exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and Item analysis were conducted to explore the reliability and validity of the CT evaluation framework. Six hundred and forty-two college students majoring in the humanities were collected. The sample was randomly divided into two subsamples (n1 = 321, n2 = 321).

Results

The Cronbach's alpha coefficient for the whole scale was 0.909, and the values of the Cronbach's alpha coefficients for individual factors of the scale ranged from 0.724 to 0.878. Then CFA was conducted within the scope of the validity study of the scale. In this way, the structure of the 7-factor scale was confirmed. Results indicated that the constructed evaluation framework performed consistently with the collected data. CFA also confirmed a good model fitting of the relevant 22 factors of the college students' CT framework (χ²/df = 3.110, RMSEA = 0.056, GFI = 0.927, AGFI = 0.902, NFI = 0.923, and CFI = 0.946).

Discussion

These findings revealed that the CT abilities self-evaluation scale was a valid and reliable instrument for measuring the CT abilities of college students in the humanities. Therefore, the college students' CT self-evaluation framework included three dimensions: discipline cognition (DC), CT disposition, and CT skills. Among them, CT disposition consisted of motivation (MO), attention (AT), and open-mindedness (OM), while CT skills included clarification skills (CS), organization skills (OS), and reflection (RE). Therefore, this framework can be an effective instrument to support college students' CT measurement. Consequently, some suggestions are also put forward regarding how to apply the instrument in future studies.

Clothing motivation, online critical thinking, and the behavioural intention of clothing collocation: Mediation analysis on Chinese youth

Recent years have witnessed a boom of fashion blogging sharing information about clothing and cosmetics on diverse social media platforms. Constant exposure to fashion-related digital information heavily impacts the conception and behaviours of Chinese youth. Compared to the substantial studies on the impact of social media, scarce research has been conducted on how youth’s cognitive processing of fashion-related digital information interacts with motivational factors to determine the subsequent behaviours. This study made an initial attempt to address this issue by exploring the successive associations between clothing motivation (amotivation, controlled, and autonomous motivation), online critical thinking (for information credibility, objectivity, and relevance), and the subsequent behavioural intention. A total of 1997 Chinese youth with diverse educational backgrounds voluntarily participated in the study. Results confirmed the direct links between clothing motivation and the behavioural intention, and these links were mediated by different online critical thinking practices. This study provides new insights for both practitioners and scholars in the fields of education, psychology, social media, and marketing.

Exploring the role of content and language integrated learning approach in developing transversal skills in university students with respect to the mediating role of emotional intelligence

Although a range of studies has explored the effects of the content and language integrated learning (CLIL) approach on students’ achievement, its impact on improving university students’ transversal skills (TSs) has remained largely unexplored in Iran. Thus, the present study has a two-fold purpose. First, it aims to investigate the role of the CLIL approach in improving Iranian university students’ TSs. Second, it purports to explore the mediating role of emotional intelligence (EI) in the relationship between the CLIL approach and TSs. For this purpose, a survey design was used wherein a research-made questionnaire was distributed among 123 university students to gather the required data. Findings disclosed that the CLIL approach had a positive role in developing the university students’ TSs. Further, the results indicated that EI positively affected the development of TSs in university students. Moreover, the findings evidenced that EI significantly mediated the relationship between the CLIL approach and the development of the university students’ TSs. The study concludes that applying the CLIL approach involves an entirely new paradigm, offering valuable implications for relevant stakeholders.

SBL Effectiveness in Teaching Entrepreneurship Skills to Young Immigrant Mothers Head of Household in Colombia: An Experimental Study

This article analyses the perception of the effectiveness of the Simulation-Based Learning method in enhancing entrepreneurship skills and academic performance in 78 young immigrant mothers head of household in Colombia in an entrepreneurship course titled Youth and Entrepreneurship. A Wilcoxon signed rank test, and a Kendall Tau correlation coefficient were used to analyze the perceptions and academic performance from the sample studied during a five-month period. A pre-test, post-test, and achievement of academic objectives questionnaire were applied, as well as a verification test of previous knowledge and a focus group at the end of the entrepreneurship course. It was found that SBL enhance the Planning and Management Skills and the Communication Skill and does not fulfill all the course objectives. This study contributes to the analysis of the effectiveness of unconventional teaching methods in different socioeconomic contexts.

The effects of example-free instruction and worked examples on problem-solving

The use of worked examples has been shown to be an effective instructional method for reducing cognitive load and successfully engaging in problem-solving. Extant research often views worked examples as an integrated part of direct instruction. Studies have examined the problem-solving effects of worked examples used in tandem with instructional explanations. However, a gap exists in research focusing on the individual problem-solving effects of example-free instructional explanations and worked examples containing no instructional explanation. This study uses a method in which worked examples are separated from direct instruction to examine the problem-solving effects of individual parts of such instruction, namely example-free instruction and worked examples containing no instructional explanation. Considering the importance of critical thinking skills in the current educational environment, the current study was conducted on a group of university students (n = 32) studying critical thinking in South Korea. Results showed that example-free instruction was more effective for problem-solving than worked examples containing no instructional explanation. Additionally, participants reported more efficient cognitive processing ability when critical thinking problems were presented through instructional explanation rather than worked examples. These results allow for a granular look at the different aspects of direct instruction and their effects on cognitive load and problem-solving.

Skill levels and gains in university STEM education in China, India, Russia and the United States

Universities contribute to economic growth and national competitiveness by equipping students with higher-order thinking and academic skills. Despite large investments in university science, technology, engineering and mathematics (STEM) education, little is known about how the skills of STEM undergraduates compare across countries and by institutional selectivity. Here, we provide direct evidence on these issues by collecting and analysing longitudinal data on tens of thousands of computer science and electrical engineering students in China, India, Russia and the United States. We find stark differences in skill levels and gains among countries and by institutional selectivity. Compared with the United States, students in China, India and Russia do not gain critical thinking skills over four years. Furthermore, while students in India and Russia gain academic skills during the first two years, students in China do not. These gaps in skill levels and gains provide insights into the global competitiveness of STEM university students across nations and institutional types.

Fostering Sustainability and Critical Thinking through Debate—A Case Study

Transversal competences such as sustainability or critical thinking have become more important in the last decades in University teaching. The objective of this article is to assess the effectiveness of debate as a teaching method capable of fostering such competences in engineering students. To do this, a debate activity has been held facing two reasonable positions: Sustainable Development versus Degrowth. The research methodology consisted of performing this activity in the classroom (with 13 students in this particular case study), and conducting some tests before and after the debate that served as feedback to assess the effectiveness of the debate on the learning process. This feedback is composed of different documents, including a pre-post test, a specific survey about the debate, a pre-post knowledge survey and the Student Evaluation of Educational Quality (SEEQ) survey. A methodology for the organization of the debate is proposed and the answers of the students to these feedback documents are analyzed. As it can be inferred from the different surveys, the debate has been an efficient learning tool to acquire knowledge and to develop sustainability and critical thinking competences.

Accuracy of performance-test linking based on a many-facet Rasch model

Performance assessments, in which human raters assess examinee performance in practical tasks, have attracted much attention in various assessment contexts involving measurement of higher-order abilities. However, difficulty persists in that ability measurement accuracy strongly depends on rater and task characteristics such as rater severity and task difficulty. To resolve this problem, various item response theory (IRT) models incorporating rater and task parameters, including many-facet Rasch models (MFRMs), have been proposed. When applying such IRT models to datasets comprising results of multiple performance tests administered to different examinees, test linking is needed to unify the scale for model parameters estimated from individual test results. In test linking, test administrators generally need to design multiple tests such that raters and tasks partially overlap. The accuracy of linking under this design is highly reliant on the numbers of common raters and tasks. However, the numbers of common raters and tasks required to ensure high accuracy in test linking remain unclear, making it difficult to determine appropriate test designs. We therefore empirically evaluate the accuracy of IRT-based performance-test linking under common rater and task designs. Concretely, we conduct evaluations through simulation experiments that examine linking accuracy based on a MFRM while changing numbers of common raters and tasks with various factors that possibly affect linking accuracy.

The Role of Students' Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

Critical reasoning (CR) when confronted with contradictory information from multiple sources is a crucial ability in a knowledge-based society and digital world. Using information without critically reflecting on the content and its quality may lead to the acceptance of information based on unwarranted claims. Previous personal beliefs are assumed to play a decisive role when it comes to critically differentiating between assertions and claims and warranted knowledge and facts. The role of generic epistemic beliefs on critical stance and attitude in reflectively dealing with information is well researched. Relatively few studies however, have been conducted on the influence of domain-specific beliefs, i.e., beliefs in relation to specific content encountered in a piece of information or task, on the reasoning process, and on how these beliefs may affect decision-making processes. This study focuses on students’ task- and topic-related beliefs that may influence their reasoning when dealing with multiple and partly contradictory sources of information. To validly assess CR among university students, we used a newly developed computer-based performance assessment in which the students were confronted with an authentic task which contains theoretically defined psychological stimuli for measuring CR. To investigate the particular role of task- and topic-related beliefs on CR, a purposeful sample of 30 university students took part in a performance assessment and then were interviewed immediately afterward. In the semi-structured cognitive interviews, the participants’ task-related beliefs were assessed. Based on qualitative analyses of the interview transcripts, three distinct profiles of decision-making among students have been identified. More specifically, the different types of students’ beliefs and attitudes derived from the cognitive interview data suggest their influence on information processing, reasoning approaches and decision-making. The results indicated that the students’ beliefs had an influence on their selection, critical evaluation and use of information as well as on their reasoning processes and final decisions.

Psychometric properties of an instrument to assess critical thinking disposition and metacognition in dental students

OBJECTIVE

The purpose of this study was to determine the psychometric properties of instruments designed to assess the critical thinking domains of disposition and metacognition in a dental student population.

METHODS

Two instruments originally developed to assess disposition and metacognition domains in undergraduate student populations were administered to dental students from 3 classes in the Spring of 2019. The overall response rate was 73%. Sampling adequacy was established for both domains. Exploratory factor analysis was used to determine underlying dimensions. Criterion validity was established by correlation with traditional measures of academic achievement.

RESULTS

Kaiser-Meyer-Olkin index (>0.80) and Bartlett's test of sphericity (P < 0.005) indicated sampling adequacy for both domains. Factor analysis supported a 2-factor solution for Disposition (Level of Open Mindedness and Tolerance for Cognitive Complexity) and a 2-factor solution for Metacognition (Metacognitive Thinking and Metacognitive Strategies). Items loading at 0.4 or higher were retained. Cronbach's alpha was above 0.8 for all but the Metacognitive Thinking subscale, which nonetheless showed adequate reliability (0.65). Reliability for Disposition and Metacognition scales was also above 0.8.

CONCLUSION

These results indicate that the instrument has sufficient validity and reliability to support further use in dental education.

Using Online Images to Teach Quantitative Skills via Comparative Neuroanatomy: Applying the Directives of Vision and Change

Vision and Change calls for increasing the quantitative skills of biology majors, which includes neuroscience majors. Accordingly, we have devised a module to give students practice at regression analyses, covariance, and ANOVA. This module consists of a quantitative comparative neuroanatomy lab in which students explore the size of the hippocampus relative to the brain in 62 different mammalian species-from an anteater to a zebu. We utilize a digital image library (with appropriate metadata) allowing students to quantify the size of the hippocampus as well as obtain an index of the size of the brain in these various species. Students then answer the following questions: (1) Do brains scale with body size? (2) Does the hippocampus scale with brain size? (3) If we control for body size, does the hippocampus still scale with brain size? (4) How does the hippocampus change as a proportion of brain size? (5) Is the proportional scaling of the hippocampus different among primates, carnivores, and other mammals? (6) Do the data provide evidence for mosaic or concerted evolution? Measures of the pedagogical efficacy showed clear and significant gains on a PreTest vs PostTest assessment of material related to the module. An open ended qualitative measure revealed students' perception of the purposes of the module, which were consistent with the learning goals. This module utilizes open access digital resources and can be performed at any institution. All the materials or links to online resources can be found at https://mdcune.psych.ucla.edu/modules/cna.