Automated verbal credibility assessment of intentions: The model statement technique and predictive modeling

Verbal lie detection using Large Language Models

Human accuracy in detecting deception with intuitive judgments has been proven to not go above the chance level. Therefore, several automatized verbal lie detection techniques employing Machine Learning and Transformer models have been developed to reach higher levels of accuracy. This study is the first to explore the performance of a Large Language Model, FLAN-T5 (small and base sizes), in a lie-detection classification task in three English-language datasets encompassing personal opinions, autobiographical memories, and future intentions. After performing stylometric analysis to describe linguistic differences in the three datasets, we tested the small- and base-sized FLAN-T5 in three Scenarios using 10-fold cross-validation: one with train and test set coming from the same single dataset, one with train set coming from two datasets and the test set coming from the third remaining dataset, one with train and test set coming from all the three datasets. We reached state-of-the-art results in Scenarios 1 and 3, outperforming previous benchmarks. The results revealed also that model performance depended on model size, with larger models exhibiting higher performance. Furthermore, stylometric analysis was performed to carry out explainability analysis, finding that linguistic features associated with the Cognitive Load framework may influence the model's predictions.

The use-the-best heuristic facilitates deception detection

Decades of research have shown that people are poor at detecting deception. Understandably, people struggle with integrating the many putative cues to deception into an accurate veracity judgement. Heuristics simplify difficult decisions by ignoring most of the information and relying instead only on the most diagnostic cues. Here we conducted nine studies in which people evaluated honest and deceptive handwritten statements, video transcripts, videotaped interviews or live interviews. Participants performed at the chance level when they made intuitive judgements, free to use any possible cue. But when instructed to rely only on the best available cue (detailedness), they were consistently able to discriminate lies from truths. Our findings challenge the notion that people lack the potential to detect deception. The simplicity and accuracy of the use-the-best heuristic provides a promising new avenue for deception research.

Deception detection with machine learning: A systematic review and statistical analysis

Several studies applying Machine Learning to deception detection have been published in the last decade. A rich and complex set of settings, approaches, theories, and results is now available. Therefore, one may find it difficult to identify trends, successful paths, gaps, and opportunities for contribution. The present literature review aims to provide the state of research regarding deception detection with Machine Learning. We followed the PRISMA protocol and retrieved 648 articles from ACM Digital Library, IEEE Xplore, Scopus, and Web of Science. 540 of them were screened (108 were duplicates). A final corpus of 81 documents has been summarized as mind maps. Metadata was extracted and has been encoded as Python dictionaries to support a statistical analysis scripted in Python programming language, and available as a collection of Jupyter Lab Notebooks in a GitHub repository. All are available as Jupyter Lab Notebooks. Neural Networks, Support Vector Machines, Random Forest, Decision Tree and K-nearest Neighbor are the five most explored techniques. The studies report a detection performance ranging from 51% to 100%, with 19 works reaching accuracy rate above 0.9. Monomodal, Bimodal, and Multimodal approaches were exploited and achieved various accuracy levels for detection. Bimodal and Multimodal approaches have become a trend over Monomodal ones, although there are high-performance examples of the latter. Studies that exploit language and linguistic features, 75% are dedicated to English. The findings include observations of the following: language and culture, emotional features, psychological traits, cognitive load, facial cues, complexity, performance, and Machine Learning topics. We also present a dataset benchmark. Main conclusions are that labeled datasets from real-life data are scarce. Also, there is still room for new approaches for deception detection with Machine Learning, especially if focused on languages and cultures other than English-based. Further research would greatly contribute by providing new labeled and multimodal datasets for deception detection, both for English and other languages.

The Science of Lie Detection by Verbal Cues: What Are the Prospects for Its Practical Applicability?

There is agreement among researchers that no simple verbal cues to deception detectable by humans have been demonstrated. This paper examines the evidence for the most prominent current methods, critically considers the prevailing research strategy, proposes a taxonomy of lie detection methods and concludes that two common types of approach are unlikely to succeed. An approach to lie detection is advocated that derives both from psychological science and common sense: When an interviewee produces a statement that contradicts either a previous statement by the same person or other information the authorities have, it will in many cases be obvious to interviewer and interviewee that at least one of the statements is a lie and at the very least the credibility of the witness is reduced. The literature on Strategic Use of Evidence shows that features of interviews that foster such revelatory and self-trapping situations have been established to be a free account and the introduction of independent information late and gradually into the proceedings, and tactics based on these characteristics constitute the best current general advice for practitioners. If any other approach 1 day challenges this status quo, it is likely to be highly efficient automated systems.

A Personal Model of Trumpery: Linguistic Deception Detection in a Real-World High-Stakes Setting

Language use differs between truthful and deceptive statements, but not all differences are consistent across people and contexts, complicating the identification of deceit in individuals. By relying on fact-checked tweets, we showed in three studies (Study 1: 469 tweets; Study 2: 484 tweets; Study 3: 24 models) how well personalized linguistic deception detection performs by developing the first deception model tailored to an individual: the 45th U.S. president. First, we found substantial linguistic differences between factually correct and factually incorrect tweets. We developed a quantitative model and achieved 73% overall accuracy. Second, we tested out-of-sample prediction and achieved 74% overall accuracy. Third, we compared our personalized model with linguistic models previously reported in the literature. Our model outperformed existing models by 5 percentage points, demonstrating the added value of personalized linguistic analysis in real-world settings. Our results indicate that factually incorrect tweets by the U.S. president are not random mistakes of the sender.

How humans impair automated deception detection performance

BACKGROUND

Deception detection is a prevalent problem for security practitioners. With a need for more large-scale approaches, automated methods using machine learning have gained traction. However, detection performance still implies considerable error rates. Findings from different domains suggest that hybrid human-machine integrations could offer a viable path in detection tasks.

METHOD

We collected a corpus of truthful and deceptive answers about participants' autobiographical intentions (n = 1640) and tested whether a combination of supervised machine learning and human judgment could improve deception detection accuracy. Human judges were presented with the outcome of the automated credibility judgment of truthful or deceptive statements. They could either fully overrule it (hybrid-overrule condition) or adjust it within a given boundary (hybrid-adjust condition).

RESULTS

The data suggest that in neither of the hybrid conditions did the human judgment add a meaningful contribution. Machine learning in isolation identified truth-tellers and liars with an overall accuracy of 69%. Human involvement through hybrid-overrule decisions brought the accuracy back to chance level. The hybrid-adjust condition did not improve deception detection performance. The decision-making strategies of humans suggest that the truth bias - the tendency to assume the other is telling the truth - could explain the detrimental effect.

CONCLUSIONS

The current study does not support the notion that humans can meaningfully add the deception detection performance of a machine learning system. All data are available at https://osf.io/45z7e/.

Detecting false intentions using unanticipated questions

The present study investigated whether measurable verbal differences occur when people vocalize their true and false intentions. To test potential differences, we used an experimental set-up where liars planned a criminal act (i.e., installing a virus on a network computer) and truth-tellers a non-criminal act (i.e., installing a new presentation program "SlideDog" on a network computer). Before they could carry out these acts, a confederate intercepted the participant and interviewed them about their intentions and the planning phase by using both anticipated and unanticipated questions. Liars used a cover story to mask their criminal intentions while truth-tellers told the entire truth. In contrast to our hypotheses, both human and automated coding did not show any evidence that liars and truth-tellers differed in plausibility or detailedness. Furthermore, results showed that asking unanticipated questions resulted in lengthier answers than anticipated questions. These results are in line with the mixed findings in the intention literature and suggest that plausibility and detailedness are less diagnostic cues for deception about intentions.

Being accurate about accuracy in verbal deception detection

PURPOSE

Verbal credibility assessments examine language differences to tell truthful from deceptive statements (e.g., of allegations of child sexual abuse). The dominant approach in psycholegal deception research to date (used in 81% of recent studies that report on accuracy) to estimate the accuracy of a method is to find the optimal statistical separation between lies and truths in a single dataset. However, this method lacks safeguards against accuracy overestimation.

METHOD & RESULTS

A simulation study and empirical data show that this procedure produces overoptimistic accuracy rates that, especially for small sample size studies typical of this field, yield misleading conclusions up to the point that a non-diagnostic tool can be shown to be a valid one. Cross-validation is an easy remedy to this problem.

CONCLUSIONS

We caution psycholegal researchers to be more accurate about accuracy and propose guidelines for calculating and reporting accuracy rates.

Verbal Deception and the Model Statement as a Lie Detection Tool

We have been reliably informed by practitioners that police officers and intelligence officers across the world have started to use the Model Statement lie detection technique. In this article we introduce this technique. We describe why it works, report the empirical evidence that it works, and outline how to use it. Research examining the Model Statement only started recently and more research is required. We give suggestions for future research with the technique. The Model Statement technique is one of many recently developed verbal lie detection methods. We start this article with a short overview of the-in our view- most promising recent developments in verbal lie detection before turning our attention to the Model Statement technique.

The first direct replication on using verbal credibility assessment for the detection of deceptive intentions

Verbal deception detection has gained momentum as a technique to tell truth-tellers from liars. At the same time, researchers' degrees of freedom make it hard to assess the robustness of effects. Replication research can help evaluate how reproducible an effect is. We present the first replication in verbal deception research whereby ferry passengers were instructed to tell the truth or lie about their travel plans. The original study found truth-tellers to include more specific time references in their answers. The replication study that closely mimicked the setting, procedure, materials, coding, and analyses found no lie-truth difference for specific time references. Although the power of our replication study was suboptimal (0.77), Bayesian statistics showed evidence in favor of the null hypothesis. Given the great applied consequences of verbal credibility tests, we hope this first replication attempt ignites much needed preregistered, high-powered, multilab replication efforts.