Relationship of psychiatric experience and interrater reliability in assessment of negative symptoms

Problemi della previsione in psichiatria

Sono esaminati vari problemi relativi alla previsione in psichiatria. I dati disponibili mostrano, in modo simile alle scienze sociali, ampi limiti nella capacità previsionale, specie per quanto riguarda il suicidio, la violenza e altri aspetti comportamentali. Vengono esaminate le difficoltà che nascono dal cercare di derivare il futuro della persona dal suo passato, la mancata coerenza fra aspetti di personalità e possibili comportamenti e il privilegio dato a strumenti psicopatologici incentrati sul singolo caso, rispetto a quelli attuariali con valutazioni testistiche e statistiche. Vengono anche evidenziati i numerosi bias cognitivi che distorcono le previsioni, in particolare l'errore fondamentale di attribuzione, che privilegia aspetti personologici rispetto a quelli situazionali. Ma altri bias hanno una importante azione distorsiva, da quelli della rappresentatività a quelli della disponibilità, da quelli statistici, al framing o al priming. Emerge una psichiatria molto legata nelle pratiche ancora al senso comune e alla folk psychology, con la ricchezza ma anche i molti errori che la caratterizzano. Di fatto esiste una modesta capacità previsionale riconosciuta alla psicologia popolare e alla psichiatria, ma è legata più a vincoli situazionali che a modelli personologici e psicopatologici e in ogni caso scarsamente affidabile per la previsione clinica in psichiatria.

Constructive Biases in Clinical Judgment

With a pair of oppositely valenced stimuli, rating the first one sometimes leads to a more extreme evaluation for the second (e.g., if the second is negatively valenced, rating the first stimulus would lead to a more negative rating for the second). We considered an evaluation bias in the case of clinical diagnosis relating to eating disorders. A population sample which included experienced clinical psychologists and psychiatrists showed partial evidence of an evaluation bias, when judging descriptions of individuals designed to be consistent with eating disorders or not. Quantum probability theory, the probability rules from quantum mechanics without any of the physics, is particularly well-suited to modeling the evaluation bias (and constructive influences generally), because a measurement (or judgment) can change the state of the system. We applied a previous quantum model to the present result, an extension of the model embodying noisy processes, and belief adjustment model. We discuss how model fits inform an examination of rationality in the observed behavior.

Can delusions be self-assessed? Concordance between self- and observer-rated delusions in schizophrenia

Several multi-dimensional self-report scales have been developed to assess delusional ideation in the general population. However, self-ratings of positive symptoms in patients with psychosis are often considered unreliable due to neuro-cognitive disturbance and lack of insight. This study tested associations of self- and observer-rated delusions as well as factors associated with discrepancies. Observer-rated delusions were assessed in 80 in- and outpatients with schizophrenia spectrum disorders by trained raters with the Positive and Negative Syndrome Scale. Self-rated delusions were assessed with the Peters et al. Delusions Inventory and the Paranoia Checklist. Correlations between self- and observer-rated overall delusions ranged from 0.49 to 0.57. Associations between specific delusions of persecution and grandiosity were moderate but unique. Good concordance of ratings was not restricted to outpatients or patients with fewer positive symptoms. Patients with lower self- than observer ratings of delusions were characterised by fewer years of education, lower functioning, more negative symptoms and less insight. The results indicate that patients can reliably provide information with regard to the presence and type of delusional beliefs. Thus, patient ratings are a valid additional source of diagnostic information.

Standardized rater training for the Hamilton Depression Rating Scale (HAMD-17) in psychiatric novices

BACKGROUND

Despite the long and widespread use of the Hamilton Depression Rating Scale (HAMD), standardized reliability studies in inexperienced raters are not available.

METHODS

Rater training was carried using three videotaped interviews with depressed patients in 21 psychiatric novices who had negligible previous experience with the HAMD. Chance-corrected coefficients of rating agreement with expert standards (weighted kappa, ICC) were computed for single items and the total score of the HAMD.

RESULTS

The results demonstrate sufficiently high interrater reliability (kappa>0.60) for most of the HAMD items and the total score (ICC=0.57-0.73). Three standardized HAMD training sessions seem adequate to establish satisfactory agreement among psychiatric novices.

LIMITATIONS

The sample of video-taped interviews and, hence, the generalizability of the results, was restricted.

CONCLUSIONS

High inter-rater reliability of the HAMD justifies the use by clinically inexperienced researchers after standardized training.

Effects of interrater reliability of psychopathologic assessment on power and sample size calculations in clinical trials

Although rater training is increasingly used to improve the quality of the investigated outcome parameters, the reliability of assessments is not perfect. Thus, empirical reliability estimates should be used instead of theoretically assumed perfect reliability. Implications of the reliability of psychiatric assessments for sample size and power calculations in clinical trials are presented. The theoretical basis of sample size and power calculations using empirical reliability scores is delineated. Examples from contemporary research on schizophrenia and depression are used to illustrate several implications for study design and interpretation of results. The tremendous impact of the lack of reliability of psychopathologic assessments on sample size, power, and detectable true score differences in clinical trials is shown. The problem of multiple outcome variables with different reliabilities is addressed. Studies lacking power because of unreliable assessments carry the risk of false-negative findings and raise ethical questions. Rater training is strongly recommended to assess and improve interrater reliability whenever necessary and possible before trials are started. Sample size calculations and power analysis should be based on empirical reliability values of outcome parameters as part of quality assurance and cost savings.

The Calgary Depression Rating Scale for Schizophrenia: development and interrater reliability of a German version (CDSS-G)

A German version of the Calgary Depression Rating Scale for Schizophrenia (CDSS-G) approved by the author of the original scale is presented comprising a semi-structured interview for 9 items to sensitively and specifically assess depression in schizophrenia and related disorders. The process of translation is outlined and the finally derived CDSS-G was investigated with respect to interrater reliability in three studies. To keep comparability with the CDSS source version a standard procedure was used. Two trained raters jointly assessed ten schizophrenic patients (study I). In a second study, videotapes with the CDSS-G were presented to clinically inexperienced raters (study II, N = 14/15) to test the agreement on the CDSS-G in this sample. Finally, in a third study clinically experienced researchers participated in a rater training (study III, N = 34). They carried out CDSS ratings on three patients with mild depressive symptoms. The dependence of interrater reliability on depression severity was investigated for all studied patients. Both intraclass correlation coefficients (ICC) and weighted kappa coefficients (kappa(w)) were calculated. The results revealed a high ICC = 0.97 in study I for the total CDSS-G score. Single item ICC values were all above 0.70. The results of study II revealed somewhat lower agreement on CDSS-G items and total scores in psychiatric novices with however acceptable values of kappa(w)>0.50 for the total scores. Study III yielded satisfactory results (0.66<kappa(w)<0.76) for clinically experienced psychiatrists in schizophrenic patients with mild depression. The results demonstrate adequate to excellent interrater reliability of the CDSS-G corroborating the results of the original version in different settings.