Cognitive and Human Factors in Expert Decision Making: Six Fallacies and the Eight Sources of Bias

Challenges to reasoning in forensic science decisions

The success of forensic science depends heavily on human reasoning abilities. Although we typically navigate our lives well using those abilities, decades of psychological science research shows that human reasoning is not always rational. In addition, forensic science often demands that its practitioners reason in non-natural ways. This article addresses how characteristics of human reasoning (either specific to an individual or in general) and characteristics of situations (either specific to a case or in general in a lab) can contribute to errors before, during, or after forensic analyses. In feature comparison judgments, such as fingerprints or firearms, a main challenge is to avoid biases from extraneous knowledge or arising from the comparison method itself. In causal and process judgments, for example fire scenes or pathology, a main challenge is to keep multiple potential hypotheses open as the investigation continues. Considering the contributions to forensic science judgments by persons, situations, and their interaction, reveals ways to develop procedures to decrease errors and improve accuracy.

Metric forensic anthropology decisions: Reliability and biasability of sectioning-point-based sex estimates

Subjective decisions make human cognitive processes more susceptible to bias and error. Specifically, research indicates that additional context biases forensic anthropologists' morphological analyses. To address whether metric analyses are also subject to bias, we conducted a pilot study in which 52 experienced osteologists measured a difficult-to-classify human femur, with or without additional contextual information. Using a metric sectioning-point sex-estimation method, participants provided a sex estimate for individual skeletal element(s) and, when given multiple elements, the combined skeletal assemblage. Control group participants (n = 24) measured only the femur. In addition to the femur, bias group participants (n = 28) either measured a female humerus and viewed a female-biasing photograph (n = 14) or measured a male humerus and viewed a male-biasing photograph (n = 14). We explored whether the experts in the different groups would differ in: (1) femoral measurements; (2) femoral sex-estimation conclusions; and (3) final sex-estimation conclusions for the skeletal assemblage. Although the femoral measurements and femoral sex estimates were comparable across groups, the overall sex estimates in the female-biased group were impacted by contextual information-differing from both the control and male-biased groups (p < 0.001). Our results demonstrate that cognitive bias can occur even in metric sex-estimation conclusions. Specifically, this occurred when the metric data and single-element sex estimates were synthesized into an overall estimate. Thus, our results suggest that metric methods are most vulnerable to bias when data are synthesized into an overall conclusion, highlighting the need for bias countermeasures and comprehensive statistical frameworks for synthesizing metric data to mitigate the effects of cognitive bias.

Objectivity is a myth that harms the practice and diversity of forensic science

•

Forensic science data are theory laden; pure scientific objectivity is a myth.

•

Upholding this myth marginalizes forensic scientists with subjective positionalities

•

Objectivity rhetoric is exclusive; ethical forensic science needs diverse perspectives.

•

Espousing objectivity prevents us from supporting the communities we serve.

•

Mitigated objectivity acknowledges implicit bias, constraining it via quality control.

Interpretation of DNA data within the context of UK forensic science - evaluation

Forensic DNA provides a striking contribution to the provision of justice worldwide. It has proven to be crucial in the investigative phase of an unsolved crime where a suspect needs to be identified, e.g. from a DNA database search both nationally and internationally. It is also a powerful tool in the assignment of evidential weight to the comparison of a profile of a person of interest and a crime scene profile. The focus of this document is the evaluation of autosomal profiles for criminal trials in the UK. A separate review covers investigation and evaluation of Y-STR profiles, investigation using autosomal profiles, kinship analysis, body identification and Forensic Genetic Genealogy investigations. In less than 40 years, forensic DNA profiling has developed from a specialist technique to everyday use. Borrowing on advances in genome typing technology, forensic DNA profiling has experienced a substantial increase in its sensitivity and informativeness. Alongside this development, novel interpretation methodologies have also been introduced. This document describes the state of the art and future advances in the interpretation of forensic DNA data.

The biasing impact of irrelevant contextual information on forensic odontology radiograph matching decisions

The potential biasing effect of irrelevant context information on the forensic odontology method of radiograph-based identification has never been empirically investigated despite being a recognized problem in other forensic science disciplines. This study examines the effect of irrelevant context information on the probability judgment of match (JOM) of practicing forensic odontologist and dentist participants who were asked to match pairs of dental radiographs supplemented with irrelevant case information. The irrelevant case information contained domain task-irrelevant context information which varied in strength (strong or weak). It suggested either supportive or contradictory bias relative to the actual match status of the radiograph pairs. The dental radiographs consisted of verified match and non-match radiographs pairs sampled and de-identified from actual forensic cases. Changes in accuracy and JOM between supportive and contradictory contexts conditions revealed a contextual bias. Mixed model analysis showed that strong supportive context increased the odds ratio of correct decisions by a factor of 2.4 [1.23, 4.46]; p = 0.0097. Consistent with the biasing effect, the JOM score differences between strong supportive and contradictory irrelevant context information were 1.03 and 0.43 respectively for the non-match and match decisions. The direction of context suggestion (p = 0.0067), the radiograph match status (p = 0.014), and their interactions (p = 0.0061), were all found to impact the participants' decision. The weak context information was not strong enough to have a significant effect on accuracy or JOM scores. This study demonstrates that radiograph match judgment is affected and can be biased by strong irrelevant contextual information.

'Anyone who commits such a cruel crime, must be criminally irresponsible': context effects in forensic psychological assessment

In recent years, it has become clear that expert opinion can be biased. It has been argued that forensic psychologists may also be susceptible to bias. In the present study, the vulnerability of forensic psychological evaluation of the suspect's mental health to the context effect (i.e. the influencing of the expert opinion by irrelevant information) was tested. Master students in forensic psychology were asked to interpret test scores of a suspect in a fictitious double murder case. Some participants received a version of the case in which the description of the murders was neutral. Others received a more explicit version. Whereas the explicitness should not affect the forensic psychological evaluation, it was found that participants in the latter condition seemed more concerned about the suspect's mental health than those in the former. It is concluded that training programmes in forensic psychological assessment should devote attention to bias.

A hierarchy of expert performance as applied to forensic anthropology

Due to their medicolegal repercussions, forensic anthropology conclusions must be reliable, consistent, and minimally compromised by bias. Yet, a synthetic analysis of the reliability and biasability of the discipline's methods has not yet been conducted. To do so, this study utilized Dror's (2016) hierarchy of expert performance (HEP), an eight-level model aimed at examining intra- and inter-expert reliability and biasability (the potential for cognitive bias) within the literature of forensic science disciplines. A systematic review of the forensic anthropology literature was conducted (1972-present), including papers published in the American Journal of Physical Anthropology, Forensic Anthropology, Forensic Science International, and the Journal of Forensic Sciences and Anthropology Section abstracts published in the Proceedings of the Annual Meeting of the AAFS which matched keywords such as "forensic anthropology," "bias," "reliability," "cognition," "cognitive," or "error." The resulting forensic anthropology HEP showcases areas that have ample research and areas where more research can be conducted. Specifically, statistically significant increases in reliability (p < 0.001) and biasability (p < 0.001) publications were found since 2009 (publication of the NAS report). Extensive research examined the reliability of forensic anthropological observations and conclusions (n = 744 publications). However, minimal research investigated the biasability of forensic anthropological observations and conclusions (n = 20 publications). Notably, while several studies demonstrated the biasing effect of extraneous information on anthropological morphological assessments, there was no research into these effects on anthropological metric assessments. The findings revealed by the forensic anthropology HEP can help to guide future research, ultimately informing the development and refinement of best-practice standards for the discipline.

A scenario for writing creative scenarios

This paper examines the elements of a pathway to writing the bold, innovative scenarios necessary to promote sustainable socio-ecological integration. Innovative scenarios incorporate three virtues essential to making knowledge systemically useful: creativity, collaboration, and communication. The main features of such a scenario-writing process include: (1) the effective integration of participants from a diverse array of disciplinary perspectives; (2) an integrated approach to defining problems as a system of interrelated structures, functions, and processes; (3) the collaborative learning required to synthesize a comprehensive understanding of the interrelated components and processes of the system containing the problem(s); (4) a process of co-writing that guides the alteration of socio-ecological systems to eliminate the problem(s); and (5) communication that clearly and persuasively conveys the message required to actualize resolution.

The effect of contextual information on professional judgment: Reliability and biasability of expert workplace safety inspectors

INTRODUCTION

A critical aspect of occupational safety is workplace inspections by experts, in which hazards are identified. Scientific research demonstrates that expectation generated by context (i.e., prior knowledge and experience) can bias the judgments of professionals and that individuals are largely unaware when their judgments are affected by bias.

METHOD

The current research tested the reliability and biasability of expert safety inspectors' judgments. We used a two-study design (Study 1, N = 83; Study 2, N = 70) to explore the potential of contextual, task-irrelevant, information to bias professionals' judgments. We examined three main issues: (1) the effect that biasing background information (safe and unsafe company history) had on professional regulatory safety inspectors' judgments of a worksite; (2) the reliability of those judgments amongst safety inspectors and (3) inspectors' awareness of bias in their judgments and confidence in their performance.

RESULTS

Our findings establish that: (i) inspectors' judgments were biased by historical contextual information, (ii) they were not only biased, but the impact was implicit: they reported being unaware that it affected their judgments, and (iii) independent of our manipulations, inspectors were inconsistent with one another and the variations were not a product of experience.

CONCLUSION

Our results are a replication of findings from a host of other professional domains, where honest, hardworking professionals underappreciate the biasing effect of context on their decision making. The current paper situates these findings within the relevant research on safety inspection, cognitive bias and decision making, as well as provides suggestions for bias mitigation in workplace safety inspection. Practical Application: Our results have implications for occupational health and safety given that inspection is an integral aspect of an effective safety system. In addition to our findings, this study contributes to the literature by providing recommendations regarding how to mitigate the effect of bias in inspection.

The distinction between discriminability and reliability in forensic science

Forensic science plays an increasingly important role in the criminal justice system; yet, many forensic procedures have not been subject to the empirical scrutiny that is expected in other scientific disciplines. Over the past two decades, the scientific community has done well to bridge the gap, but have likely only scratched the tip of the iceberg. We offer the discriminability-reliability distinction as a critical framework to guide future research on diagnostic-testing procedures in the forensic science domain. We argue that the primary concern of the scientist ought to be maximizing discriminability and that the primary concern of the criminal justice system ought to be assessing the reliability of evidence. We argue that Receiver Operating Characteristic (ROC) analysis is uniquely equipped for determining which of two procedures or conditions has better discriminability and we also demonstrate how estimates of reliability can be extracted from this Signal Detection framework.

Cognitive bias in forensic pathology decisions

Forensic pathologists’ decisions are critical in police investigations and court proceedings as they determine whether an unnatural death of a young child was an accident or homicide. Does cognitive bias affect forensic pathologists’ decision‐making? To address this question, we examined all death certificates issued during a 10‐year period in the State of Nevada in the United States for children under the age of six. We also conducted an experiment with 133 forensic pathologists in which we tested whether knowledge of irrelevant non‐medical information that should have no bearing on forensic pathologists’ decisions influenced their manner of death determinations. The dataset of death certificates indicated that forensic pathologists were more likely to rule "homicide" rather than "accident" for deaths of Black children relative to White children. This may arise because the base‐rate expectation creates an a priori cognitive bias to rule that Black children died as a result of homicide, which then perpetuates itself. Corroborating this explanation, the experimental data with the 133 forensic pathologists exhibited biased decisions when given identical medical information but different irrelevant non‐medical information about the race of the child and who was the caregiver who brought them to the hospital. These findings together demonstrate how extraneous information can result in cognitive bias in forensic pathology decision‐making.

The tendency to stop collecting information is linked to illusions of causality

Previous research proposed that cognitive biases contribute to produce and maintain the symptoms exhibited by deluded patients. Specifically, the tendency to jump to conclusions (i.e., to stop collecting evidence soon before making a decision) has been claimed to contribute to delusion formation. Additionally, deluded patients show an abnormal understanding of cause-effect relationships, often leading to causal illusions (i.e., the belief that two events are causally connected, when they are not). Both types of bias appear in psychotic disorders, but also in healthy individuals. In two studies, we test the hypothesis that the two biases (jumping to conclusions and causal illusions) appear in the general population and correlate with each other. The rationale is based on current theories of associative learning that explain causal illusions as the result of a learning bias that tends to wear off as additional information is incorporated. We propose that participants with higher tendency to jump to conclusions will stop collecting information sooner in a causal learning study than those participants with lower tendency to jump to conclusions, which means that the former will not reach the learning asymptote, leading to biased judgments. The studies provide evidence in favour that the two biases are correlated but suggest that the proposed mechanism is not responsible for this association.

Translational Application of a Neuro-Scientific Multi-Modal Approach Into Forensic Psychiatric Evaluation: Why and How?

A prominent body of literature indicates that insanity evaluations, which are intended to provide influential expert reports for judges to reach a decision "beyond any reasonable doubt," suffer from a low inter-rater reliability. This paper reviews the limitations of the classical approach to insanity evaluation and the criticisms to the introduction of neuro-scientific approach in court. Here, we explain why in our opinion these criticisms, that seriously hamper the translational implementation of neuroscience into the forensic setting, do not survive scientific scrutiny. Moreover, we discuss how the neuro-scientific multimodal approach may improve the inter-rater reliability in insanity evaluation. Critically, neuroscience does not aim to introduce a brain-based concept of insanity. Indeed, criteria for responsibility and insanity are and should remain clinical. Rather, following the falsificationist approach and the convergence of evidence principle, the neuro-scientific multimodal approach is being proposed as a way to improve reliability of insanity evaluation and to mitigate the influence of cognitive biases on the formulation of insanity opinions, with the final aim to reduce errors and controversies.

Contemporary issues in forensic science-Worldwide survey results

Forensic science continues to be openly challenged and criticized. The aim of this study was to gain an understanding of forensic workplaces and the perceived current and potential future issues from forensic scientists via a detailed survey. An online survey was designed and disseminated to forensic practitioners and researchers worldwide. 544 participants from more than 20 countries took part in the survey. Participants personally rated ten forensic disciplines with subjective methodologies, responded on a five-point Likert scale to 22 statements that addressed subjectivity and objectivity, validation and proficiency testing and error and bias and answered demographic questions relating to their workplace type, level of experience and qualifications. Participants also commented freely on forensic issues specifically important to them. The purpose of this paper is to provide the survey results and consensuses captured on several key issues. Overall participants believed forensic science must be valid and reliable and supported development of objective methodologies, validation and further investigation into the application statistics, use of error rates and implications of cognitive bias. Participants raised consensus concerns with the provision of expert evidence and other broader issues. This information and understanding from the forensic front line are essential for forensic science moving forward to best address current challenges and criticisms not only of forensic evidence for the court but for applications of forensic science beyond the courtroom.

Law and order effects: on cognitive dissonance and belief perseverance

Order of evidence presentation affects the evaluation and the integration of evidence in mock criminal cases. In this study, we aimed to determine whether the order in which incriminating and exonerating evidence is presented influences cognitive dissonance and subsequent display of confirmation bias. Law students (N = 407) were presented with a murder case vignette, followed by incriminating and exonerating evidence in various orders. Contrary to a predicted primacy effect (i.e. early evidence being most influential), a recency effect (i.e. late evidence being most influential) was observed in ratings of likelihood of the suspect's guilt. The cognitive dissonance ratings and conviction rates were not affected by the order of evidence presentation. The effects of evidence presentation order may be limited to specific aspects of legal decisions. However, there is a need to replicate the results using procedures and samples that are more representative of real-life criminal law trials.

Clinical Reasoning in Forensic Psychiatry: Concepts, Processes, and Pitfalls

Forensic psychiatrists are often sought by the court of law to provide professional opinion on specific legal matters that have a major impact on the evaluee and possibly society at large. The quality of that opinion and recommendations rely on the quality of the analysis from the assessment results conducted by the psychiatrist. However, the definition and scope of a forensic psychiatric analysis is not clear. While existing literature on forensic psychiatric analysis generally includes organizing information, identifying relevant details, and formulating a set of forensic psychiatric opinions as components, there is no explicit and unified definition of these terms and process. This lack of clarity and guidelines may hinder forensic psychiatry from achieving its goal of providing objective information to the court or other relevant parties. Forensic psychiatric analysis exhibits numerous parallels to clinical reasoning in other fields of medicine. Therefore, this review aims to elaborate forensic psychiatric analysis through the lens of clinical reasoning, which has been developed by incorporating advances in cognitive sciences. We describe forensic psychiatric analysis through three prominent clinical reasoning theories: hypothetico-deductive model, illness script theory, and dual process theory. We expand those theories to elucidate how forensic psychiatrists use clinical reasoning not only to diagnose mental disorders, but also to determine mental capacities as requested by law. Cognitive biases are also described as potential threat to the accuracy of the assessment and analysis. Additionally, situated cognition theory helps elucidate how contextual factors influence risk of errors. Understanding the processes involved in forensic psychiatric analysis and their pitfalls can assist forensic psychiatrists to be aware of and try to mitigate their bias. Debiasing strategies that have been implemented in other fields of medicine to mitigate errors in clinical reasoning can be adapted for forensic psychiatry. This may also shape the training program of general psychiatrists and forensic psychiatrists alike.

Biasability and reliability of expert forensic document examiners

The performance of experts can be characterized in terms of biasability and reliability of their judgments. The current research is the first to explore the judgments of practicing forensic document experts, professionals who examine and compare disputed handwritten evidence to handwriting exemplars of individuals involved in criminal or civil litigation. Forensic handwriting experts determine if questioned and known handwritten items are of common authorship or written by different individuals, and present their findings in legal proceedings. The expert participants in our study (N=25) were not aware that they were part of a research study. Thirteen participants were led to believe that they were working on a case commissioned from the prosecution and the other twelve that it was for the defense. We did not find evidence in this study that this information biased their judgments, which may make sense since document examiners (in contrast to many other forensic domains) do not primarily work within an organizational forensic laboratory culture. The lack of bias in our findings may have been also due to the stimuli we used or/and the great variability in the judgments within each group, reflecting a lack of consistency in conclusions among examiners. A detailed discussion of our findings is presented along with the limitations that may have affected our results.

A multidisciplinary approach to insanity assessment as a way to reduce cognitive biases

Insanity assessment requires the evaluation of the psychopathological condition that underlies the mens rea. Psychopathological evaluation may be quite challenging due to (i) absence of biomarkers; (ii) low inter-rater reliability; (iii) presence of cognitive bias. This intrinsic low reliability of forensic psychiatric diagnosis does impact on insanity assessment, leading to arbitrary and unjust legal outcomes for the examinee. Thus, strategies to improve the reliability of insanity evaluation are strongly needed. A multidisciplinary approach has been proposed as a way to enrich clinical diagnosis with reliable and biologically founded data, thus minimizing subjectivity, reducing controversies and increasing inter-subject concordance in insanity assessment. By discussing a real case, here we show how the convergence of multiple indices can produce evidence that cannot be denied without introducing logical fallacies. Applying this approach, the forensic discussion will move from the presence/absence of psychopathology to the impact of psychopathology on insanity. This article illustrates how a multidisciplinary evaluation, which integrates neuroscientific methods with the classical insanity assessment, may lead to a more accurate approach in insanity evaluation. Critically, this approach will minimize the impact of cognitive bias on insanity opinion and thus result in an improvement of the whole criminal justice process.

Firearm examination: Examiner judgments and computer-based comparisons

Forensic firearm examination provides the court of law with information about the source of fired cartridge cases. We assessed the validity of source decisions of a computer-based method and of 73 firearm examiners who compared breechface and firing pin impressions of 48 comparison sets. We also compared the computer-based method's comparison scores with the examiners' degree-of-support judgments and assessed the validity of the latter. The true-positive rate (sensitivity) and true-negative rate (specificity) of the computer-based method (for the comparison of both the breechface and firing pin impressions) were 94.4% and at least 91.7%, respectively. For the examiners, the true-positive rate was at least 95.3% and the true-negative rate was at least 86.2%. The validity of the source decisions improved when the evaluations of breechface and firing pin impressions were combined and for the examiners also when the perceived difficulty of the comparison decreased. The examiners were reluctant to provide source decisions for "difficult" comparisons even though their source decisions were mostly correct. The correlation between the computer-based method's comparison scores and the examiners' degree-of-support judgments was low for the same-source comparisons to negligible for the different-source comparisons. Combining the outcomes of computer-based methods with the judgments of examiners could increase the validity of firearm examinations. The examiners' numerical degree-of-support judgments for their source decisions were not well-calibrated and showed clear signs of overconfidence. We suggest studying the merits of performance feedback to calibrate these judgments.

Organizational and Human Factors Affecting Forensic Decision-Making: Workplace Stress and Feedback

Although forensic examiners operate in a stressful environment, there is a lack of understanding about workplace stress and feedback. These organizational and human factors can potentially impact forensic science judgments. In this study, 150 practicing forensic examiners from one laboratory were surveyed about their experiences of workplace stress, and the explicit and implicit feedback they receive. Forensic examiners reported that their high stress levels originated more from workplace-related factors (management and/or supervision, backlogs, and the pressure to do many cases) than from personal related factors (family, medical, and/or financial). The findings showed that a few (8%) of the forensic examiners sometimes felt strong implicit feedback about what conclusions were expected from them and that some (14%) also strongly felt that they were more appreciated when they helped to solve a case (e.g., by reaching a "match" as opposed to an "inconclusive" conclusion). Differences were found when comparing workplace stress and feedback levels across three core forensic science fields (forensic biology, chemistry, and latent prints) and across career stages (early, mid, and late). Gaining insights into the stress factors within a workplace and explicit and implicit feedback has implications for developing policies to improve the well-being, motivation, and performance of forensic examiners.

The effect of contextual information on decision-making in forensic toxicology

The impact of cognitive bias on decisions in forensic science has been demonstrated in numerous disciplines such as DNA and fingerprints, but has not been empirically investigated in the more objective domains, such as forensic toxicology. In the first experiment, participants (n = 58) were affected by irrelevant case information when analysing data from an immunoassay test for opiate-type drugs. In the second experiment, participants (n = 53) were biased in their choice of tests, for example, the age of the deceased impacted testing strategy: for older people, medicinal drugs were commonly chosen, whereas for younger people drugs of abuse were selected. Based on the results that examiners analyzing case data may have biases if they are given access to case context, we propose that examiners analysing presumptive test data are blind to irrelevant contextual information. Furthermore, that forensic toxicology laboratories use a consistent protocol for selecting tests, and that any deviations are documented and justified.

(Mis)use of scientific measurements in forensic science

Forensic science error rate studies have not given sufficient attention or weight to inconclusive evidence and inconclusive decisions. Inconclusive decisions can be correct decisions, but they can also be incorrect decisions. Errors can occur when inconclusive evidence is determined as an identification or exclusion, or conversely, when same- or different-source evidence is incorrectly determined as inconclusive. We present four common flaws in error rate studies: 1. Not including test items which are more prone to error; 2. Excluding inconclusive decisions from error rate calculations; 3. Counting inconclusive decisions as correct in error rate calculations; and 4. Examiners resorting to more inconclusive decisions during error rate studies than they do in casework. These flaws seriously undermine the credibility and accuracy of error rates reported in studies. To remedy these shortcomings, we present the problems and show the way forward by providing a corrected experimental design that quantifies error rates more accurately.