1
|
PsycAssist: A Web-Based Artificial Intelligence System Designed for Adaptive Neuropsychological Assessment and Training. Brain Sci 2024; 14:122. [PMID: 38391697 PMCID: PMC10886469 DOI: 10.3390/brainsci14020122] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 01/15/2024] [Accepted: 01/19/2024] [Indexed: 02/24/2024] Open
Abstract
Assessing executive functions in individuals with disorders or clinical conditions can be challenging, as they may lack the abilities needed for conventional test formats. The use of more personalized test versions, such as adaptive assessments, might be helpful in evaluating individuals with specific needs. This paper introduces PsycAssist, a web-based artificial intelligence system designed for neuropsychological adaptive assessment and training. PsycAssist is a highly flexible and scalable system based on procedural knowledge space theory and may be used potentially with many types of tests. We present the architecture and adaptive assessment engine of PsycAssist and the two currently available tests: Adap-ToL, an adaptive version of the Tower of London-like test to assess planning skills, and MatriKS, a Raven-like test to evaluate fluid intelligence. Finally, we describe the results of an investigation of the usability of Adap-ToL and MatriKS: the evaluators perceived these tools as appropriate and well-suited for their intended purposes, and the test-takers perceived the assessment as a positive experience. To sum up, PsycAssist represents an innovative and promising tool to tailor evaluation and training to the specific characteristics of the individual, useful for clinical practice.
Collapse
|
2
|
Algorithms for the adaptive assessment of procedural knowledge and skills. Behav Res Methods 2023; 55:3929-3951. [PMID: 36526887 PMCID: PMC10616228 DOI: 10.3758/s13428-022-01998-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/01/2022] [Indexed: 12/23/2022]
Abstract
Procedural knowledge space theory (PKST) was recently proposed by Stefanutti (British Journal of Mathematical and Statistical Psychology, 72(2) 185-218, 2019) for the assessment of human problem-solving skills. In PKST, the problem space formally represents how a family of problems can be solved and the knowledge space represents the skills required for solving those problems. The Markov solution process model (MSPM) by Stefanutti et al. (Journal of Mathematical Psychology, 103, 102552, 2021) provides a probabilistic framework for modeling the solution process of a task, via PKST. In this article, three adaptive procedures for the assessment of problem-solving skills are proposed that are based on the MSPM. Beside execution correctness, they also consider the sequence of moves observed in the solution of a problem with the aim of increasing efficiency and accuracy of assessments. The three procedures differ from one another in the assumption underlying the solution process, named pre-planning, interim-planning, and mixed-planning. In two simulation studies, the three adaptive procedures have been compared to one another and to the continuous Markov procedure (CMP) by Doignon and Falmagne (1988a). The last one accounts for dichotomous correct/wrong answers only. Results show that all the MSP-based adaptive procedures outperform the CMP in both accuracy and efficiency. These results have been obtained in the framework of the Tower of London test but the procedures can also be applied to all psychological and neuropsychological tests that have a problem space. Thus, the adaptive procedures presented in this paper pave the way to the adaptive assessment in the area of neuropsychological tests.
Collapse
|
3
|
Empirical indistinguishability: From the knowledge structure to the skills. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2023; 76:312-326. [PMID: 36366819 DOI: 10.1111/bmsp.12291] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 08/30/2022] [Accepted: 09/13/2022] [Indexed: 06/16/2023]
Abstract
Recent literature has pointed out that the basic local independence model (BLIM) when applied to some specific instances of knowledge structures presents identifiability issues. Furthermore, it has been shown that for such instances the model presents a stronger form of unidentifiability named empirical indistinguishability, which leads to the fact that the existence of certain knowledge states in such structures cannot be empirically tested. In this article the notion of indistinguishability is extended to skill maps and, more generally, to the competence-based knowledge space theory. Theoretical results are provided showing that skill maps can be empirically indistinguishable from one another. The most relevant consequence of this is that for some skills there is no empirical evidence to establish their existence. This result is strictly related to the type of probabilistic model investigated, which is essentially the BLIM. Alternative models may exist or can be developed in knowledge space theory for which this indistinguishability problem disappears.
Collapse
|
4
|
On the empirical indistinguishability of knowledge structures. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2021; 74:465-486. [PMID: 33782939 DOI: 10.1111/bmsp.12235] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/22/2020] [Revised: 01/07/2021] [Indexed: 06/12/2023]
Abstract
In recent years a number of articles have focused on the identifiability of the basic local independence model. The identifiability issue usually concerns two model parameter sets predicting an identical probability distribution on the response patterns. Both parameter sets are applied to the same knowledge structure. However, nothing is known about cases where different knowledge structures predict the same probability distribution. This situation is referred to as 'empirical indistinguishability' between two structures and is the main subject of the present paper. Empirical indistinguishability is a stronger form of unidentifiability, which involves not only the parameters, but also the structural and combinatorial properties of the model. In particular, as far as knowledge structures are concerned, a consequence of empirical indistinguishability is that the existence of certain knowledge states cannot be empirically established. Most importantly, it is shown that model identifiability cannot guarantee that a certain knowledge structure is empirically distinguishable from others. The theoretical findings are exemplified in a number of different empirical scenarios.
Collapse
|
5
|
Stat-Knowlab. Assessment and Learning of Statistics with Competence-based Knowledge Space Theory. INTERNATIONAL JOURNAL OF ARTIFICIAL INTELLIGENCE IN EDUCATION 2020. [DOI: 10.1007/s40593-020-00223-1] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
6
|
Extending the Basic Local Independence Model to Polytomous Data. PSYCHOMETRIKA 2020; 85:684-715. [PMID: 32959202 PMCID: PMC7599199 DOI: 10.1007/s11336-020-09722-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2019] [Revised: 06/19/2020] [Indexed: 05/28/2023]
Abstract
A probabilistic framework for the polytomous extension of knowledge space theory (KST) is proposed. It consists in a probabilistic model, called polytomous local independence model, that is developed as a generalization of the basic local independence model. The algorithms for computing "maximum likelihood" (ML) and "minimum discrepancy" (MD) estimates of the model parameters have been derived and tested in a simulation study. Results show that the algorithms differ in their capability of recovering the true parameter values. The ML algorithm correctly recovers the true values, regardless of the manipulated variables. This is not totally true for the MD algorithm. Finally, the model has been applied to a real polytomous data set collected in the area of psychological assessment. Results show that it can be successfully applied in practice, paving the way to a number of applications of KST outside the area of knowledge and learning assessment.
Collapse
|
7
|
On the assessment of procedural knowledge: From problem spaces to knowledge spaces. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2019; 72:185-218. [PMID: 30035297 DOI: 10.1111/bmsp.12139] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/27/2017] [Revised: 04/17/2018] [Indexed: 06/08/2023]
Abstract
By generalizing and completing the work initiated by Stefanutti and Albert (2003, Journal of Universal Computer Science, 9, 1455), this article provides the mathematical foundations of a theoretical approach whose primary goal is to construct a bridge between problem solving, as initially conceived by Newell and Simon (1972, Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.), and knowledge assessment (Doignon and Falmagne, 1985, International Journal of Man-Machine Studies, 23, 175; Doignon and Falmagne, 1999, Knowledge spaces. Berlin, Germany: Springer-Verlag.; Falmagne et al., 2013, Knowledge spaces: Applications in education. New York, NY: Springer-Verlag; Falmagne and Doignon, 2011, Learning spaces: Interdisciplinary applied mathematics. Berlin, Germany: Springer-Verlag.). It is shown that the collection of all possible knowledge states for a given problem space is a learning space. An algorithm for deriving a learning space from a problem space is illustrated. As an example, the algorithm is used to derive the learning space of a neuropsychological test whose problem space is well known: the Tower of London (TOL; Shallice, 1982, Philosophical Transactions of the Royal Society of London B: Biological Sciences, 298, 199). The derived learning space could then be used for adaptively assessing individual planning skills with the TOL.
Collapse
|
8
|
Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2018. [DOI: 10.1177/2515245917747646] [Citation(s) in RCA: 267] [Impact Index Per Article: 44.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Collapse
|
9
|
The assessment of knowledge and learning in competence spaces: The gain-loss model for dependent skills. THE BRITISH JOURNAL OF MATHEMATICAL AND STATISTICAL PSYCHOLOGY 2017; 70:457-479. [PMID: 28211048 DOI: 10.1111/bmsp.12095] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/07/2016] [Revised: 12/15/2016] [Indexed: 06/06/2023]
Abstract
The gain-loss model (GaLoM) is a formal model for assessing knowledge and learning. In its original formulation, the GaLoM assumes independence among the skills. Such an assumption is not reasonable in several domains, in which some preliminary knowledge is the foundation for other knowledge. This paper presents an extension of the GaLoM to the case in which the skills are not independent, and the dependence relation among them is described by a well-graded competence space. The probability of mastering skill s at the pretest is conditional on the presence of all skills on which s depends. The probabilities of gaining or losing skill s when moving from pretest to posttest are conditional on the mastery of s at the pretest, and on the presence at the posttest of all skills on which s depends. Two formulations of the model are presented, in which the learning path is allowed to change from pretest to posttest or not. A simulation study shows that models based on the true competence space obtain a better fit than models based on false competence spaces, and are also characterized by a higher assessment accuracy. An empirical application shows that models based on pedagogically sound assumptions about the dependencies among the skills obtain a better fit than models assuming independence among the skills.
Collapse
|
10
|
Abstract
In knowledge space theory, existing adaptive assessment procedures can only be applied when suitable estimates of their parameters are available. In this paper, an iterative procedure is proposed, which upgrades its parameters with the increasing number of assessments. The first assessments are run using parameter values that favor accuracy over efficiency. Subsequent assessments are run using new parameter values estimated on the incomplete response patterns from previous assessments. Parameter estimation is carried out through a new probabilistic model for missing-at-random data. Two simulation studies show that, with the increasing number of assessments, the performance of the proposed procedure approaches that of gold standards.
Collapse
|
11
|
Erratum to: On the Link between Cognitive Diagnostic Models and Knowledge Space Theory. PSYCHOMETRIKA 2016; 81:250-251. [PMID: 26676448 DOI: 10.1007/s11336-015-9494-5] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
|
12
|
On the Link between Cognitive Diagnostic Models and Knowledge Space Theory. PSYCHOMETRIKA 2015; 80:995-1019. [PMID: 25838246 DOI: 10.1007/s11336-015-9457-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/15/2012] [Indexed: 06/04/2023]
Abstract
The present work explores the connections between cognitive diagnostic models (CDM) and knowledge space theory (KST) and shows that these two quite distinct approaches overlap. It is proved that in fact the Multiple Strategy DINA (Deterministic Input Noisy AND-gate) model and the CBLIM, a competence-based extension of the basic local independence model (BLIM), are equivalent. To demonstrate the benefits that arise from integrating the two theoretical perspectives, it is shown that a fairly complete picture on the identifiability of these models emerges by combining results from both camps. The impact of the results is illustrated by an empirical example, and topics for further research are pointed out.
Collapse
|
13
|
|
14
|
An analysis of item response theory and Rasch models based on the most probable distribution method. PSYCHOMETRIKA 2014; 79:377-402. [PMID: 25205004 DOI: 10.1007/s11336-013-9348-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/01/2012] [Revised: 02/01/2013] [Indexed: 06/03/2023]
Abstract
The most probable distribution method is applied to derive the logistic model as the distribution accounting for the maximum number of possible outcomes in a dichotomous test while introducing latent traits and item characteristics as constraints to the system. The item response theory logistic models, with a particular focus on the one-parameter logistic model, or Rasch model, and their properties and assumptions, are discussed for both infinite and finite populations.
Collapse
|
15
|
Assessing parameter invariance in the BLIM: bipartition models. PSYCHOMETRIKA 2013; 78:710-724. [PMID: 24092485 DOI: 10.1007/s11336-013-9325-5] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/21/2012] [Revised: 10/23/2012] [Indexed: 06/02/2023]
Abstract
In knowledge space theory, the knowledge state of a student is the set of all problems he is capable of solving in a specific knowledge domain and a knowledge structure is the collection of knowledge states. The basic local independence model (BLIM) is a probabilistic model for knowledge structures. The BLIM assumes a probability distribution on the knowledge states and a lucky guess and a careless error probability for each problem. A key assumption of the BLIM is that the lucky guess and careless error probabilities do not depend on knowledge states (invariance assumption). This article proposes a method for testing the violations of this specific assumption. The proposed method was assessed in a simulation study and in an empirical application. The results show that (1) the invariance assumption might be violated by the empirical data even when the model's fit is very good, and (2) the proposed method may prove to be a promising tool to detect invariance violations of the BLIM.
Collapse
|
16
|
|
17
|
Skill map based knowledge structures: some considerations about their identifiability. ACTA ACUST UNITED AC 2013. [DOI: 10.1016/j.endm.2013.05.148] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
18
|
|
19
|
The Gain-Loss Model: A Probabilistic Skill Multimap Model for Assessing Learning Processes. JOURNAL OF EDUCATIONAL MEASUREMENT 2010. [DOI: 10.1111/j.1745-3984.2010.00119.x] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
20
|
Tropopause and hygropause variability over the equatorial Indian Ocean during February and March 1999. ACTA ACUST UNITED AC 2006. [DOI: 10.1029/2005jd006639] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
21
|
|
22
|
Ozone measurements during the Airborne Polar Experiment: Aircraft instrument validation, isentropic trends, and hemispheric fields prior to the 1997 Arctic ozone depletion. ACTA ACUST UNITED AC 2000. [DOI: 10.1029/2000jd900038] [Citation(s) in RCA: 12] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
23
|
Optical classification, existence temperatures, and coexistence of different polar stratospheric cloud types. ACTA ACUST UNITED AC 1999. [DOI: 10.1029/1999jd900064] [Citation(s) in RCA: 38] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
24
|
Airborne Polar Experiment-Polar Ozone, Leewaves, Chemistry, and Transport (APE-POLECAT): Rationale, road map and summary of measurements. ACTA ACUST UNITED AC 1999. [DOI: 10.1029/1998jd100078] [Citation(s) in RCA: 17] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
25
|
Aircraft lidar observations of an enhanced type Ia polar stratospheric clouds during APE-POLECAT. ACTA ACUST UNITED AC 1999. [DOI: 10.1029/1998jd100055] [Citation(s) in RCA: 53] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
26
|
Parameterization of cloud lidar backscattering profiles by means of asymmetrical Gaussians. APPLIED OPTICS 1995; 34:3449-3456. [PMID: 21052159 DOI: 10.1364/ao.34.003449] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
A fitting procedure for cloud lidar data processing is shown that is based on the computation of the first three moments of the vertical-backscattering (or -extinction) profile. Single-peak clouds or single cloud layers are approximated to asymmetrical Gaussians. The algorithm is particularly stable with respect to noise and processing errors, and it is much faster than the equivalent least-squares approach. Multilayer clouds can easily be treated as a sum of single asymmetrical Gaussian peaks. The method is suitable for cloud-shape parametrization in noisy lidar signatures (like those expected from satellite lidars). It also permits an improvement of cloud radiative-property computations that are based on huge lidar data sets for which storage and careful examination of single lidar profiles can't be carried out.
Collapse
|
27
|
Derivation of Mount Pinatubo stratospheric aerosol mean size distribution by means of a multiwavelength lidar. APPLIED OPTICS 1994; 33:5690-5697. [PMID: 20935970 DOI: 10.1364/ao.33.005690] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/30/2023]
Abstract
A multiwavelength lidar operated in Sodankyla, Finland, during the European Arctic Stratospheric Ozone Experiment (December 1991-March 1992). It produced vertical profiles of stratospheric aerosols at four wavelengths. The determination of aerosol mean size distribution has been performed by use of extinction/backscattering ratios as obtained from lidar data processing at 355, 352, and 750 nm. Lognormal distributions of sulfuric particles with mode radius of r(m) = 0.12-0.25 µm and corresponding widths of s = 2-1.6 have been retrieved as best fits of experimental data, in good agreement with in situ measurements. A successful attempt to derive bimodal log-normal distributions is also described, together with the experimental and theoretical problems involved.
Collapse
|
28
|
Polar stratospheric cloud observations over the Antarctic continent at Dumont d'Urville. ACTA ACUST UNITED AC 1991. [DOI: 10.1029/91jd00776] [Citation(s) in RCA: 16] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
29
|
Criterion for the choice of exposure time in atmospheric turbulence investigation with an optical wave. APPLIED OPTICS 1973; 12:136-138. [PMID: 20125248 DOI: 10.1364/ao.12.000136] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
|
30
|
Investigation of atmospheric turbulence by narrow laser beams. APPLIED OPTICS 1970; 9:2543-2547. [PMID: 20094303 DOI: 10.1364/ao.9.002543] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
In order to determine the characteristic parameters of the atmospheric turbulence, the mutual dancing of two parallel narrow laser beams propagating through a turbulent atmosphere has been investigated theoretically and experimentally. The experiments have been carried out with a He-Ne laser on a 130-m path, one meter from the ground. The method allowed the determination of the variance of the refractive index fluctuations, as well as the values of scale of the turbulence in the vertical and horizontal planes.
Collapse
|