Validity of a computerized on-line MMPI interpretive system

Relative user ratings of MMPI-2 computer-based test interpretations

There are eight commercially available computer-based test interpretations (CBTIs) for the Minnesota Multiphasic Personality Inventory-2 (MMPI-2), of which few have been empirically evaluated. Prospective users of these programs have little scientific data to guide choice of a program. This study compared ratings of these eight CBTIs. Test users were randomly assigned to rate either a single authentic CBTI report on one of their clients or a single CBTI report generated from a modal MMPI-2 profile for their clinical setting. In all, 257 authentic and modal CBTI reports were rated by 41 clinicians on 10 dimensions. Each of the authentic reports received substantially higher ratings than the modal reports, with ratings of perceived accuracy and opinion confirmation best differentiating between authentic and modal reports. Automated Assessment Associates' report received the highest overall ratings; reports published by Western Psychological Services, Pearson Assessments, and the Caldwell Report were also distinguished on one or more ratings dimensions.

The Barnum effect in a computerized Rorschach interpretation system

Twelve psychiatric outpatients were administered the Rorschach test, and results were interpreted using the Exner (1983, 1986) Report for the Comprehensive System computer-based test interpretation (CBTI) program. Four psychiatrists made accuracy ratings for both real and bogus reports for each of their patients. Data were analyzed using two-way analyses of covariance (ANCOVAs), where report type was a repeated main effect, psychiatrist was a random main effect, and the number of statements in the report was the covariate. Results indicated that this CBTI provided only 5% discriminating power for any one patient, with 60% of the interpretive statements merely describing typical characteristics of the outpatient population. No significant psychiatrist, interaction, or covariate effects were encountered.

Evaluating medical expert systems

Approximately 90% of all computerized medical expert systems have not been evaluated in clinical environments. This paper: identifies the principal methods used to assess the performance of medical expert systems in both laboratory and clinical settings, describes the different research strategies used in the evaluation of medical expert systems at different development stages, and discusses past evaluation efforts in relationship to future applications of different decision support technologies and expert systems in health care.