1
|
Skirzyński J, Jain YR, Lieder F. Automatic discovery and description of human planning strategies. Behav Res Methods 2024; 56:1065-1103. [PMID: 37253960 DOI: 10.3758/s13428-023-02062-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/06/2023] [Indexed: 06/01/2023]
Abstract
Scientific discovery concerns finding patterns in data and creating insightful hypotheses that explain these patterns. Traditionally, each step of this process required human ingenuity. But the galloping development of computer chips and advances in artificial intelligence (AI) make it increasingly more feasible to automate some parts of scientific discovery. Understanding human planning is one of the fields in which AI has not yet been utilized. State-of-the-art methods for discovering new planning strategies still rely on manual data analysis. Data about the process of human planning is often used to group similar behaviors together. Researchers then use this data to formulate verbal descriptions of the strategies which might underlie those groups of behaviors. In this work, we leverage AI to automate these two steps of scientific discovery. We introduce a method for automatic discovery and description of human planning strategies from process-tracing data collected with the Mouselab-MDP paradigm. Our method utilizes a new algorithm, called Human-Interpret, that performs imitation learning to describe sequences of planning operations in terms of a procedural formula and then translates that formula to natural language. We test our method on a benchmark data set that researchers have previously scrutinized manually. We find that the descriptions of human planning strategies that we obtain automatically are about as understandable as human-generated descriptions. They also cover a substantial proportion of relevant types of human planning strategies that had been discovered manually. Our method saves scientists' time and effort, as all the reasoning about human planning is done automatically. This might make it feasible to more rapidly scale up the search for yet undiscovered cognitive strategies that people use for planning and decision-making to many new decision environments, populations, tasks, and domains. Given these results, we believe that the presented work may accelerate scientific discovery in psychology, and due to its generality, extend to problems from other fields.
Collapse
Affiliation(s)
- Julian Skirzyński
- Max Planck Institute for Intelligent Systems, Tübingen, Germany.
- University of California, San Diego, CA, 92093, USA.
| | - Yash Raj Jain
- Max Planck Institute for Intelligent Systems, Tübingen, Germany
| | - Falk Lieder
- Max Planck Institute for Intelligent Systems, Tübingen, Germany
| |
Collapse
|
2
|
Srivastava N, Sifar A, Srinivasan N. Statistical prediction alone cannot identify good models of behavior. Behav Brain Sci 2023; 46:e408. [PMID: 38054355 DOI: 10.1017/s0140525x23001784] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
The dissociation between statistical prediction and scientific explanation advanced by Bowers et al. for studies of vision using deep neural networks is also observed in several other domains of behavior research, and is in fact unavoidable when fitting large models such as deep nets and other supervised learners, with weak theoretical commitments, to restricted samples of highly stochastic behavioral phenomena.
Collapse
Affiliation(s)
- Nisheeth Srivastava
- Department of Cognitive Science, Indian Institute of Technology Kanpur, Kalyanpur, UP, India https://www.cse.iitk.ac.in/users/nsrivast/ https://sites.google.com/site/ammuns68/
| | - Anjali Sifar
- Department of Cognitive Science, Indian Institute of Technology Kanpur, Kalyanpur, UP, India https://www.cse.iitk.ac.in/users/nsrivast/ https://sites.google.com/site/ammuns68/
| | - Narayanan Srinivasan
- Department of Cognitive Science, Indian Institute of Technology Kanpur, Kalyanpur, UP, India https://www.cse.iitk.ac.in/users/nsrivast/ https://sites.google.com/site/ammuns68/
| |
Collapse
|
3
|
Kuperwajs I, Schütt HH, Ma WJ. Using deep neural networks as a guide for modeling human planning. Sci Rep 2023; 13:20269. [PMID: 37985896 PMCID: PMC10662256 DOI: 10.1038/s41598-023-46850-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 11/06/2023] [Indexed: 11/22/2023] Open
Abstract
When developing models in cognitive science, researchers typically start with their own intuitions about human behavior in a given task and then build in mechanisms that explain additional aspects of the data. This refinement step is often hindered by how difficult it is to distinguish the unpredictable randomness of people's decisions from meaningful deviations between those decisions and the model. One solution for this problem is to compare the model against deep neural networks trained on behavioral data, which can detect almost any pattern given sufficient data. Here, we apply this method to the domain of planning with a heuristic search model for human play in 4-in-a-row, a combinatorial game where participants think multiple steps into the future. Using a data set consisting of 10,874,547 games, we train deep neural networks to predict human moves and find that they accurately do so while capturing meaningful patterns in the data. Thus, deviations between the model and the best network allow us to identify opportunities for model improvement despite starting with a model that has undergone substantial testing in previous work. Based on this analysis, we add three extensions to the model that range from a simple opening bias to specific adjustments regarding endgame planning. Overall, our work demonstrates the advantages of model comparison with a high-performance deep neural network as well as the feasibility of scaling cognitive models to massive data sets for systematically investigating the processes underlying human sequential decision-making.
Collapse
Affiliation(s)
| | - Heiko H Schütt
- Center for Neural Science, New York University, New York, NY, USA
| | - Wei Ji Ma
- Center for Neural Science, New York University, New York, NY, USA
- Department of Psychology, New York University, New York, NY, USA
| |
Collapse
|
4
|
Aka A, Bhatia S, McCoy J. Semantic determinants of memorability. Cognition 2023; 239:105497. [PMID: 37442022 DOI: 10.1016/j.cognition.2023.105497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 03/26/2023] [Accepted: 05/11/2023] [Indexed: 07/15/2023]
Abstract
We examine why some words are more memorable than others by using predictive machine learning models applied to word recognition and recall datasets. Our approach provides more accurate out-of-sample predictions for recognition and recall than previous psychological models, and outperforms human participants in new studies of memorability prediction. Our approach's predictive power stems from its ability to capture the semantic determinants of memorability in a data-driven manner. We identify which semantic categories are important for memorability and show that, unlike features such as word frequency that influence recognition and recall differently, the memorability of semantic categories is consistent across recognition and recall. Our paper sheds light on the complex psychological drivers of memorability, and in doing so illustrates the power of machine learning methods for psychological theory development.
Collapse
Affiliation(s)
- Ada Aka
- Stanford University, United States of America.
| | - Sudeep Bhatia
- University of Pennsylvania, United States of America.
| | - John McCoy
- University of Pennsylvania, United States of America.
| |
Collapse
|
5
|
Huang L. A quasi-comprehensive exploration of the mechanisms of spatial working memory. Nat Hum Behav 2023; 7:729-739. [PMID: 36959326 DOI: 10.1038/s41562-023-01559-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Accepted: 02/16/2023] [Indexed: 03/25/2023]
Abstract
Why are some spatial patterns remembered more easily than others? There are many possible mechanisms underlying spatial working memory function. Here, the author explores different mechanisms simultaneously in a single conceptual model. He conducts a large-scale experiment (35.4 million responses used to measure human observers' spatial working memory across 80,000 patterns) and builds a convolutional neural network as a benchmark for what is expected to be explainable. The author then creates a quasi-comprehensive exploration model of spatial working memory based on classic concepts, as well as new notions, including spatial uncertainty, Bayesian integration, out-of-range responses, averaging, grouping, categorical memory, line detection, gap detection, blurring, lateral inhibition, chunking, multiple spatial-frequency channels, redundancy, response bias and random guess. This model provides a tentative overarching framework for the mechanisms of spatial working memory.
Collapse
Affiliation(s)
- Liqiang Huang
- Department of Psychology, The Chinese University of Hong Kong, Hong Kong, China.
| |
Collapse
|
6
|
Suchow JW. Scaling up behavioural studies of visual memory. Nat Hum Behav 2023; 7:672-673. [PMID: 36959328 DOI: 10.1038/s41562-023-01565-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/25/2023]
|
7
|
Paulino D, Guimarães D, Correia A, Ribeiro J, Barroso J, Paredes H. A Model for Cognitive Personalization of Microtask Design. SENSORS (BASEL, SWITZERLAND) 2023; 23:3571. [PMID: 37050630 PMCID: PMC10098703 DOI: 10.3390/s23073571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/08/2023] [Revised: 03/21/2023] [Accepted: 03/27/2023] [Indexed: 06/19/2023]
Abstract
The study of data quality in crowdsourcing campaigns is currently a prominent research topic, given the diverse range of participants involved. A potential solution to enhancing data quality processes in crowdsourcing is cognitive personalization, which involves appropriately adapting or assigning tasks based on a crowd worker's cognitive profile. There are two common methods for assessing a crowd worker's cognitive profile: administering online cognitive tests, and inferring behavior from task fingerprinting based on user interaction log events. This article presents the findings of a study that investigated the complementarity of both approaches in a microtask scenario, focusing on personalizing task design. The study involved 134 unique crowd workers recruited from a crowdsourcing marketplace. The main objective was to examine how the administration of cognitive ability tests can be used to allocate crowd workers to microtasks with varying levels of difficulty, including the development of a deep learning model. Another goal was to investigate if task fingerprinting can be used to allocate crowd workers to different microtasks in a personalized manner. The results indicated that both objectives were accomplished, validating the usage of cognitive tests and task fingerprinting as effective mechanisms for microtask personalization, including the development of a deep learning model with 95% accuracy in predicting the accuracy of the microtasks. While we achieved an accuracy of 95%, it is important to note that the small dataset size may have limited the model's performance.
Collapse
Affiliation(s)
- Dennis Paulino
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
- INESC TEC, 4200-465 Porto, Portugal
| | - Diogo Guimarães
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
- INESC TEC, 4200-465 Porto, Portugal
| | - António Correia
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
- INESC TEC, 4200-465 Porto, Portugal
| | - José Ribeiro
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
| | - João Barroso
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
- INESC TEC, 4200-465 Porto, Portugal
| | - Hugo Paredes
- School of Science and Technology, University of Trás-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal; (D.P.)
- INESC TEC, 4200-465 Porto, Portugal
| |
Collapse
|
8
|
Bartlett LK, Pirrone A, Javed N, Gobet F. Computational Scientific Discovery in Psychology. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2023; 18:178-189. [PMID: 35943820 PMCID: PMC9902966 DOI: 10.1177/17456916221091833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/31/2023]
Abstract
Scientific discovery is a driving force for progress involving creative problem-solving processes to further our understanding of the world. The process of scientific discovery has historically been intensive and time-consuming; however, advances in computational power and algorithms have provided an efficient route to make new discoveries. Complex tools using artificial intelligence (AI) can efficiently analyze data as well as generate new hypotheses and theories. Along with AI becoming increasingly prevalent in our daily lives and the services we access, its application to different scientific domains is becoming more widespread. For example, AI has been used for the early detection of medical conditions, identifying treatments and vaccines (e.g., against COVID-19), and predicting protein structure. The application of AI in psychological science has started to become popular. AI can assist in new discoveries both as a tool that allows more freedom to scientists to generate new theories and by making creative discoveries autonomously. Conversely, psychological concepts such as heuristics have refined and improved artificial systems. With such powerful systems, however, there are key ethical and practical issues to consider. This article addresses the current and future directions of computational scientific discovery generally and its applications in psychological science more specifically.
Collapse
Affiliation(s)
- Laura K. Bartlett
- Laura K. Bartlett, Centre for Philosophy of Natural and Social Science, London School of Economics and Political Science
| | | | | | | |
Collapse
|
9
|
Lieder F, Prentice M, Corwin‐Renner ER. An interdisciplinary synthesis of research on understanding and promoting well‐doing. SOCIAL AND PERSONALITY PSYCHOLOGY COMPASS 2022. [DOI: 10.1111/spc3.12704] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Affiliation(s)
- Falk Lieder
- Max Planck Institute for Intelligent Systems Tübingen Germany
| | - Mike Prentice
- Max Planck Institute for Intelligent Systems Tübingen Germany
| | - Emily R. Corwin‐Renner
- Hector Research Institute of Education Sciences and Psychology University of Tübingen Tübingen Germany
| |
Collapse
|
10
|
Almaatouq A, Becker J, Houghton JP, Paton N, Watts DJ, Whiting ME. Empirica: a virtual lab for high-throughput macro-level experiments. Behav Res Methods 2021; 53:2158-2171. [PMID: 33782900 PMCID: PMC8516782 DOI: 10.3758/s13428-020-01535-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/30/2020] [Indexed: 12/27/2022]
Abstract
Virtual labs allow researchers to design high-throughput and macro-level experiments that are not feasible in traditional in-person physical lab settings. Despite the increasing popularity of online research, researchers still face many technical and logistical barriers when designing and deploying virtual lab experiments. While several platforms exist to facilitate the development of virtual lab experiments, they typically present researchers with a stark trade-off between usability and functionality. We introduce Empirica: a modular virtual lab that offers a solution to the usability-functionality trade-off by employing a "flexible defaults" design strategy. This strategy enables us to maintain complete "build anything" flexibility while offering a development platform that is accessible to novice programmers. Empirica's architecture is designed to allow for parameterizable experimental designs, reusable protocols, and rapid development. These features will increase the accessibility of virtual lab experiments, remove barriers to innovation in experiment design, and enable rapid progress in the understanding of human behavior.
Collapse
Affiliation(s)
| | | | | | - Nicolas Paton
- Massachusetts Institute of Technology, Cambridge, MA, USA.
| | | | | |
Collapse
|
11
|
Plonsky O, Erev I. To predict human choice, consider the context. Trends Cogn Sci 2021; 25:819-820. [PMID: 34330661 DOI: 10.1016/j.tics.2021.07.007] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Accepted: 07/16/2021] [Indexed: 11/28/2022]
Abstract
Choice prediction competitions suggest that popular models of choice, including prospect theory, have low predictive accuracy. Peterson et al. show the key problem lies in assuming each alternative is evaluated in isolation, independently of the context. This observation demonstrates how a focus on predictions can promote understanding of cognitive processes.
Collapse
Affiliation(s)
- Ori Plonsky
- Faculty of Industrial Engineering and Management, Technion - Israel Institute of Technology, Haifa, Israel.
| | - Ido Erev
- Faculty of Industrial Engineering and Management, Technion - Israel Institute of Technology, Haifa, Israel
| |
Collapse
|
12
|
Hofman JM, Watts DJ, Athey S, Garip F, Griffiths TL, Kleinberg J, Margetts H, Mullainathan S, Salganik MJ, Vazire S, Vespignani A, Yarkoni T. Integrating explanation and prediction in computational social science. Nature 2021; 595:181-188. [PMID: 34194044 DOI: 10.1038/s41586-021-03659-0] [Citation(s) in RCA: 51] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Accepted: 05/20/2021] [Indexed: 12/30/2022]
Abstract
Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.
Collapse
Affiliation(s)
| | - Duncan J Watts
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA. .,The Annenberg School of Communication, University of Pennsylvania, Philadelphia, PA, USA. .,Operations, Information, and Decisions Department, University of Pennsylvania, Philadelphia, PA, USA.
| | - Susan Athey
- Graduate School of Business, Stanford University, Stanford, CA, USA
| | - Filiz Garip
- Department of Sociology, Princeton University, Princeton, NJ, USA
| | - Thomas L Griffiths
- Department of Psychology, Princeton University, Princeton, NJ, USA.,Department of Computer Science, Princeton University, Princeton, NJ, USA
| | - Jon Kleinberg
- Department of Computer Science, Cornell University, Ithaca, NY, USA.,Department of Information Science, Cornell University, Ithaca, NY, USA
| | - Helen Margetts
- Oxford Internet Institute, University of Oxford, Oxford, UK.,Public Policy Programme, The Alan Turing Institute, London, UK
| | | | | | - Simine Vazire
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Victoria, Australia
| | - Alessandro Vespignani
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA
| | - Tal Yarkoni
- Department of Psychology, University of Texas at Austin, Austin, TX, USA
| |
Collapse
|
13
|
Affiliation(s)
- Sudeep Bhatia
- Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA.
| | - Lisheng He
- School of Business and Management, Shanghai International Studies University, Shanghai, China
| |
Collapse
|
14
|
Peterson JC, Bourgin DD, Agrawal M, Reichman D, Griffiths TL. Using large-scale experiments and machine learning to discover theories of human decision-making. Science 2021; 372:1209-1214. [DOI: 10.1126/science.abe2629] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Accepted: 05/06/2021] [Indexed: 01/13/2023]
Abstract
Predicting and understanding how people make decisions has been a long-standing goal in many fields, with quantitative models of human decision-making informing research in both the social sciences and engineering. We show how progress toward this goal can be accelerated by using large datasets to power machine-learning algorithms that are constrained to produce interpretable psychological theories. Conducting the largest experiment on risky choice to date and analyzing the results using gradient-based optimization of differentiable decision theories implemented through artificial neural networks, we were able to recapitulate historical discoveries, establish that there is room to improve on existing theories, and discover a new, more accurate model of human decision-making in a form that preserves the insights from centuries of research.
Collapse
Affiliation(s)
- Joshua C. Peterson
- Department of Computer Science, Princeton University, Princeton, NJ 08540, USA
| | - David D. Bourgin
- Department of Computer Science, Princeton University, Princeton, NJ 08540, USA
| | - Mayank Agrawal
- Department of Psychology, Princeton University, Princeton, NJ 08540, USA
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Daniel Reichman
- Department of Computer Science, Worcester Polytechnic Institute, Worcester, MA 01609, USA
| | - Thomas L. Griffiths
- Department of Computer Science, Princeton University, Princeton, NJ 08540, USA
- Department of Psychology, Princeton University, Princeton, NJ 08540, USA
| |
Collapse
|