Essay content and style are strongly related to household income and SAT scores: Evidence from 60,000 undergraduate applications

Lingualyzer: A computational linguistic tool for multilingual and multidimensional text analysis

Most natural language models and tools are restricted to one language, typically English. For researchers in the behavioral sciences investigating languages other than English, and for those researchers who would like to make cross-linguistic comparisons, hardly any computational linguistic tools exist, particularly none for those researchers who lack deep computational linguistic knowledge or programming skills. Yet, for interdisciplinary researchers in a variety of fields, ranging from psycholinguistics, social psychology, cognitive psychology, education, to literary studies, there certainly is a need for such a cross-linguistic tool. In the current paper, we present Lingualyzer ( https://lingualyzer.com ), an easily accessible tool that analyzes text at three different text levels (sentence, paragraph, document), which includes 351 multidimensional linguistic measures that are available in 41 different languages. This paper gives an overview of Lingualyzer, categorizes its hundreds of measures, demonstrates how it distinguishes itself from other text quantification tools, explains how it can be used, and provides validations. Lingualyzer is freely accessible for scientific purposes using an intuitive and easy-to-use interface.

Income and campus application disparities among European and non-European heritage Hispanic undergraduate applicants

Leveraging every undergraduate application submitted by self-identified Hispanic applicants to the University of California system in the 2016 and 2017 application cycles, we show that a significant number of applicants claim Hispanic identity by virtue of European heritage. We subsequently demonstrate that Hispanic-identifying students of European descent are significantly more affluent and more likely to apply to selective University of California campuses than their non-European Hispanic peers. We comment on the practical implications of these disparities, as well as their relevance for studies of inequality in the social sciences and education.

Dear admission committee…: Which moves in application essays predict student master grades?

Application essays are a commonly used admission instrument for students entering higher education. The quality of the essay is usually scored, but this score is often subjective and has poor interrater reliability due to the unstructured format of the essays. This results in mixed findings on the validity of application essays as an admission instrument. We propose a more objective method of using application essays, using Latent Dirichlet Allocation (LDA), a text mining method, to distinguish seven moves occurring in application essays written by students who apply to a master degree program. We use the probability that these moves occur in the essay to predict study success in the master. Thereby we answer the following research question: What is the effect of discussing different moves in students' application essays on the student grades in a master program? From the seven different moves (functional unit of text) we distinguished, five of which have a significant effect on student grades. The moves we labeled as 'master specific' and 'interest to learn' have a positive effect on student grades, and the moves we labeled as 'research skills', 'societal impact' and 'city and university' have a negative effect. Our interpretation of this finding is that topics related to intrinsic motivation and specific knowledge, as opposed to generic knowledge, are positively related with study success. We thereby demonstrate that application essays can be a valid predictor of study success. This contributes to justifying their use as admission instruments.

A prosocial value intervention in gateway STEM courses

Many college students, especially first-generation and underrepresented racial/ethnic minority students, desire courses and careers that emphasize helping people and society. Can instructors of introductory science, technology, engineering, and math (STEM) courses promote motivation, performance, and equity in STEM fields by emphasizing the prosocial relevance of course material? We developed, implemented, and evaluated a prosocial utility-value intervention (UVI): A course assignment in which students were asked to reflect on the prosocial value of biology or chemistry course content; our focus was on reducing performance gaps between first-generation and continuing generation college students. In Studies 1a and 1b, we piloted two versions of a prosocial UVI in introductory biology (N = 282) and chemistry classes (N = 1,705) to test whether we could encourage students to write about the prosocial value of course content. In Study 2, we tested a version of the UVI that combines personal and prosocial values, relative to a standard UVI, which emphasizes personal values, using a randomized controlled trial in an introductory chemistry course (N = 2,505), and examined effects on performance and motivation in the course. In Study 3, we tested the prosocial UVI against a standard UVI in an introductory biology course (N = 712). Results suggest that the prosocial UVI may be particularly effective in promoting motivation and performance for first-generation college students, especially those who are more confident that they can perform well in the class, reflecting a classic expectancy-value interaction. Mediation analyses suggest that this intervention worked by promoting interest in chemistry. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Using artificial intelligence to assess personal qualities in college admissions

Personal qualities like prosocial purpose and leadership predict important life outcomes, including college success. Unfortunately, the holistic assessment of personal qualities in college admissions is opaque and resource intensive. Can artificial intelligence (AI) advance the goals of holistic admissions? While cost-effective, AI has been criticized as a "black box" that may inadvertently penalize already disadvantaged subgroups when used in high-stakes settings. Here, we consider an AI approach to assessing personal qualities that aims to overcome these limitations. Research assistants and admissions officers first identified the presence/absence of seven personal qualities in n = 3131 applicant essays describing extracurricular and work experiences. Next, we fine-tuned pretrained language models with these ratings, which successfully reproduced human codes across demographic subgroups. Last, in a national sample (N = 309,594), computer-generated scores collectively demonstrated incremental validity for predicting 6-year college graduation. We discuss challenges and opportunities of AI for assessing personal qualities.

Predicting Success: An Examination of the Predictive Validity of a Measure of Motivational-Developmental Dimensions in College Admissions

Amid the COVID-19 pandemic, an unprecedented number of higher education institutions adopted test-optional admissions policies. The proliferation of these policies and the criticism of standardized admissions tests as unreliable predictors of applicants' postsecondary educational promise have prompted the reimagining of evaluative methodologies in college admissions. However, few institutions have designed and implemented new measures of applicants' potential for success, rather opting to redistribute the weight given to other variables such as high school course grades and high school GPA. We use multiple regression to investigate the predictive validity of a measure of non-cognitive, motivational-developmental dimensions implemented as part of a test-optional admissions policy at a large urban research university in the United States. The measure, composed of four short-answer essay questions, was developed based on the social-cognitive motivational and developmental-constructivist perspectives. Our findings suggest that scores derived from the measure make a statistically significant but small contribution to the prediction of undergraduate GPA and 4-year bachelor's degree completion. We also find that the measure does not make a statistically significant nor practical contribution to the prediction of 5-year graduation.

Three families of automated text analysis

Since the beginning of this millennium, data in the form of human-generated text in a machine-readable format has become increasingly available to social scientists, presenting a unique window into social life. However, harnessing vast quantities of this highly unstructured data in a systematic way presents a unique combination of analytical and methodological challenges. Luckily, our understanding of how to overcome these challenges has also developed greatly over this same period. In this article, I present a novel typology of the methods social scientists have used to analyze text data at scale in the interest of testing and developing social theory. I describe three "families" of methods: analyses of (1) term frequency, (2) document structure, and (3) semantic similarity. For each family of methods, I discuss their logical and statistical foundations, analytical strengths and weaknesses, as well as prominent variants and applications.

Linguistic, cultural, and narrative capital: computational and human readings of transfer admissions essays

Variation in college application materials related to social stratification is a contentious topic in social science and national discourse in the United States. This line of research has also started to use computational methods to consider qualitative materials, such as personal statements and letters of recommendation. Despite the prominence of this topic, fewer studies have considered a fairly common academic pathway: transferring. Approximately 40% of all college students in the US transfer schools at least once. One quirk of the system is that students from community colleges are applying for the same spots for students already enrolled in four year schools and trying to transfer. How might different aspects the transfer application itself correlate with institutional stratification and make students more or less distinguishable? We use a dataset of 20,532 transfer admissions essays submitted to the University of California system to describe how transfer applicants vary linguistically, culturally, and narratively with respect to academic pathways and essay prompts. Using a variety of methods for computational text analysis and qualitative coding, we find that essays written by community college students tend to be distinct from those written by university students. However, the strength and character of these results changed with the writing prompt provided to applicants. These results show how some forms of stratification, such as the type of school students attend, inform educational processes intended to equalize opportunity and how combining computational and human reading might illuminate these patterns.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s42001-022-00185-5.

Goal language is associated with attrition and weight loss on a digital program: Observational study

Behavioral weight loss reduces risk of weight-related health complications. Outcomes of behavioral weight loss programs include attrition and weight loss. There is reason to believe that individuals' written language on a weight management program may be associated with outcomes. Exploring associations between written language and these outcomes could potentially inform future efforts towards real-time automated identification of moments or individuals at high risk of suboptimal outcomes. Thus, in the first study of its kind, we explored whether individuals' written language in actual use of a program (i.e., outside of a controlled trial) is associated with attrition and weight loss. We examined two types of language: goal setting (i.e., language used in setting a goal at the start of the program) and goal striving (i.e., language used in conversations with a coach about the process of striving for goals) and whether they are associated with attrition and weight loss on a mobile weight management program. We used the most established automated text analysis program, Linguistic Inquiry Word Count (LIWC), to retrospectively analyze transcripts extracted from the program database. The strongest effects emerged for goal striving language. In striving for goals, psychologically distanced language was associated with more weight loss and less attrition, while psychologically immediate language was associated with less weight loss and higher attrition. Our results highlight the potential importance of distanced and immediate language in understanding outcomes like attrition and weight loss. These results, generated from real-world language, attrition, and weight loss (i.e., from individuals' natural usage of the program), have important implications for how future work can better understand outcomes, especially in real-world settings.