Careful calculation or a leap of faith? A field study of the translation of CBCA ratings to final credibility judgements

How Information on a Motive to Lie Influences CBCA-Based Ratings and Veracity Judgments

We investigated how information on a motive to lie impacts on the perceived content quality of a statement and its subsequent veracity rating. In an online study, 300 participants rated a statement about an alleged sexual harassment on a scale based on Criteria-based Content Analysis (CBCA) and judged its veracity. In a 3 × 3 between-subjects design, we varied prior information (motive to lie, no motive to lie, and no information on a motive), and presented three different statement versions of varying content quality (high, medium, and low). In addition to anticipating main effects of both independent variables (motive information and statement version), we predicted that the impact of motive information on both ratings would be highest for medium quality statements, because their assessment is especially ambiguous (interaction effect). Contrary to our hypotheses, results showed that participants were unaffected by motive information and accurately reproduced the manipulated quality differences between statement versions in their CBCA-based judgments. In line with the expected interaction effect, veracity ratings decreased in the motive-to-lie group compared to controls, but only when the medium- and the low-quality statements were rated (truth ratings dropped from approximately 80 to 50%). Veracity ratings in both the no-motive-to-lie group and controls did not differ across statement versions (≥82% truth ratings). In sum, information on a motive to lie thus encouraged participants to consider content quality in their veracity judgments by being critical only of statements of medium and low quality. Otherwise, participants judged statements to be true irrespective of content quality.

A call to improve the validity of criterion-based content analysis (CBCA): Results from a field-based study including 60 children's statements of sexual abuse

The growing awareness of sexually-abused children has led to a major shift: previously considered untrustworthy, children are now regarded as competent in providing medico-legal evidence. Professionals undertaking the challenging task of assessing the child's credibility need to rely upon approved evaluation methods. The Criteria-Based Content Analysis is a tool developed to assess the truthfulness of a child's verbal statement. This field-based study explores its validity and its limitations. Three independent experts rated the verbatim statements of 60 real-life alleged victims of sexual abuse. The CBCA scoring and final assessment of credibility were linked to the outcomes: confirmed or unconfirmed allegation of sexual abuse. Inter-rater reliability coefficient was 0.74. The average overall accuracy rate corresponding to confirmed and unconfirmed cases was 75%. Among the confirmed allegations, the accuracy rate reached 90%, whereas the probability of discriminating the true negative cases within the unconfirmed cases was lower than chance level. Of all the 19 criteria, items 6 "Reproduction of conversation" and 12 "Accounts of subjective mental state" were the strongest predictors of genuine accounts. A significant association between age and CBCA scores was noted, the effect of age on CBCA scores was strongest in the unconfirmed cases. Although some may argue that the validity of the CBCA is reasonably acceptable, results from this field study are less convincing. Increasing the diagnostic accuracy of the CBCA by adding new criteria, so as to raise the percentage of correct classifications in the confirmed accounts as well as in the unconfirmed accounts, would represent a major improvement.

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W=0.84) and for overall evaluation (Kendall's W=0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.