Variation and Interactional Non-Standardization in Neuropsychological Tests: The Case of the Addenbrooke's Cognitive Examination

Inter-rater variability in scoring of Addenbrooke's Cognitive Examination-Third Edition (ACE-III) protocols

BACKGROUND

Despite its wide use in dementia diagnosis on the basis of cut-off points, the inter-rater variability of the Addenbrooke's Cognitive Examination-Third Edition (ACE-III) has been poorly studied.

METHODS

Thirty-one healthcare professionals from an older adults' mental health team scored two ACE-III protocols based on mock patients in a computerised form. Scoring accuracy, as well as total and domain-specific scoring variability, were calculated; factors relevant to participants were obtained, including their level of experience and self-rated confidence administering the ACE-III.

RESULTS

There was considerable inter-rater variability (up to 18 points for one of the cases), and one case's mean score was significantly higher (by nearly four points) than the true score. The Fluency, Visuospatial and Attention domains had greater levels of variability than Language and Memory. Higher scoring accuracy was not associated with either greater levels of experience or higher self-confidence in administering the ACE-III.

CONCLUSIONS

The results suggest that the ACE-III is susceptible to scoring error and considerable inter-rater variability, which highlights the critical importance of initial, and continued, administration and scoring training.

Speech patterns in responses to questions asked by an intelligent virtual agent can help to distinguish between people with early stage neurodegenerative disorders and healthy controls

Previous research has provided strong evidence that speech patterns can help to distinguish between people with early stage neurodegenerative disorders (ND) and healthy controls. This study examined speech patterns in responses to questions asked by an intelligent virtual agent (IVA): a talking head on a computer which asks pre-recorded questions. The study investigated whether measures of response length, speech rate and pausing in responses to questions asked by an IVA help to distinguish between healthy control participants and people diagnosed with Mild Cognitive Impairment (MCI) or Alzheimer's disease (AD). The study also considered whether those measures can further help to distinguish between people with MCI, people with AD, and healthy control participants (HC). There were 38 people with ND (31 people with MCI, 7 people with AD) and 26 HC. All interactions took place in English. People with MCI spoke fewer words compared to HC, and people with AD and people with MCI spoke for less time than HC. People with AD spoke at a slower rate than people with MCI and HC. There were significant differences across all three groups for the proportion of time spent pausing and the average pause duration: silent pauses make up the greatest proportion of responses from people with AD, who also have the longest average silent pause duration, followed by people with MCI then HC. Therefore, the study demonstrates the potential of an IVA as a method for collecting data showing patterns which can help to distinguish between diagnostic groups.

How do quantitative studies involving people with dementia report experiences of standardised data collection? A narrative synthesis of NIHR published studies

BACKGROUND

People with dementia are routinely included as research participants in trials and other quantitative studies in which they are invited to respond to standardised measures. This paper reviews the reporting of standardised data collection from people with dementia in reports published in the National Institute for Health and Care Research (NIHR) Journals Library. The aim was to understand how the administration of standardised, self-report measures with people with dementia is reported in NIHR monographs and what could be learnt from this about the feasibility and acceptability of data collection approaches for future studies.

METHODS

This was a systematic review with narrative synthesis. Broad search terms (Dementia OR Alzheimer*) were used to search the NIHR Journals Library website in December 2021. All studies that used (or intended to use) standardised measures to collect research data directly from people with dementia were eligible for inclusion. Information was extracted (where reported) on the process of data collection, dementia severity, levels of missing data and the experiences and reflections of those involved.

RESULTS

Searches returned 42 records, from which 17 reports were assessed as eligible for inclusion, containing 22 studies. Response rates from participants with dementia in these studies varied considerably and appeared to be related to dementia severity and place of residence. Little information was reported on the process of data collection or the reasons for missing data, and most studies did not report the experiences of participants or those administering the measures. However, there was an indication from two studies that standardised data collection could provoke emotional distress in some participants with dementia.

CONCLUSIONS

Through this review we identified both variation in levels of missing data and gaps in reporting which make it difficult to ascertain the reasons for this variation. We also identified potential risks to the well-being of participants with dementia which may be associated with the content of standardised measures and the context of data collection. Open reporting of and reflection upon data collection processes and the experiences of people involved is essential to ensure both the success of future data collection and the wellbeing of study participants.

TRIAL REGISTRATION

Registered with Research on Research https://ror-hub.org/study/2905/ .

Standardised data collection from people with dementia over the telephone: A qualitative study of the experience of DETERMIND programme researchers in a pandemic

There is a notable lack of evidence on what constitutes good practice in remote quantitative data collection from research participants with dementia. During the COVID-19 pandemic face-to-face research became problematic, especially where participants were older and more at risk of infection. The DETERMIND-C19 study, a large cohort study of people with dementia, switched to telephone data collection over this period. This paper explores the experiences of researchers who collected quantitative data over the telephone from people with dementia during the first COVID-19 lockdowns in England. The aim was to learn from these experiences, share insights and inform future research practice across disciplines. Seven DETERMIND researchers were interviewed about the processes and challenges of collecting quantitative data from people with dementia over the telephone compared to face-to-face. Data were analysed using reflexive thematic analysis. Two themes were developed: first the telephone adds an extra layer of confusion to an already cognitively complex interaction. Second, researchers found it difficult to recognise subtle cues that signalled participants' rising emotion over the telephone in time to prevent distress. The researchers employed strategies to support participants which may not have conformed to the strict conventions of structured interviewing, but which were informed by person-oriented principles. Whilst in practice this may be a common approach to balancing the needs of participants and the requirements of quantitative research, it is rare for studies to openly discuss such trade-offs in the literature. Honest, reflective reporting is required if the practice of remote data collection from people with dementia is to progress ethically and with integrity.

A social practice theory approach to exploring the ubiquity of quizzes in dementia care settings

Quizzes are a ubiquitous part of the dementia social care landscape. This article explores why. Using an ethnographic approach which draws on close analysis of communication, we examine dementia quizzes as a ‘social practice’, and what such a lens can tell us about their popularity in social care settings. Vignettes of real interactions drawn from ten different quizzes recorded in four different group settings attended by 28 people living with dementia and 15 staff members are presented to highlight particular issues. We show that the conditions of post-diagnosis dementia social care are uniquely well suited to an activity such as quizzes which are malleable, requiring little preparation or materials, and impose a communication framework which can help to organise the interactional space. Quizzes also draw on previously forged interactional competences, such as turn-taking and question–answer sequences, a skill that has been shown to persist even as dementia progresses. Finally, we argue that the meaning of quizzes with people with dementia feeds into wider societal values and associations attached to memory, dementia and personhood. The extent to which quizzes are akin to a ‘test’ or a fun and enjoyable social activity rests in how they are enacted. We suggest that practice can be adapted, developed and made more inclusive through input from people living with dementia themselves.

A direct observation of pain scale use in five video-recorded palliative care consultations: Using conversation analysis to show how practitioners support patients to describe pain

OBJECTIVE

Assessing pain intensity is an important palliative care task. Self-report pain intensity scales are frequently used within assessment. In contrast to formal studies of validity and reliability, we examine administration of, and responses to these scales in everyday palliative care.

METHODS

We searched episodes of pain scale use in a dataset of (video/audio-recorded) UK palliative care consultations involving five doctors, 37 terminally ill patients and their companions. We found five, and applied the techniques and tools of conversation analysis to characterise scales' administration and functioning.

RESULTS

Generally, the patients responded to scales by reporting multiple aspects of pain; the doctors supported and encouraged this. In two episodes, the scales generated misunderstandings. The doctors worked to resolve these in ways that avoided implying the patient was at fault.

CONCLUSION

Pain intensity scales can yield richer information than just intensity. They can also generate misunderstandings and social friction which take skill and effort to resolve.

PRACTICE IMPLICATIONS

Patients tend to respond to pain intensity scales by reporting on multiple aspects of pain, professionals should support them in this. These scales sometimes generate misunderstandings. To preserve the therapeutic relationship, professionals should work to resolve these without implying the patient is to blame.

Use of the Patient Health Questionnaire (PHQ-9) in Practice: Interactions between patients and physicians

We analyze the use of nine-item Patient Health Questionnaire (PHQ-9), an instrument that is widely used in diagnosing and determining the severity of depression. Using conversation analysis, we show how the doctor deploys the PHQ-9 in response to the patient's doubts about whether she is depressed. Rather than relaying the PHQ-9 verbatim, the doctor deviates from the wording so that the response options are selectively offered to upgrade the severity of the patient's symptoms. This works in favor of a positive diagnosis and is used to justify a treatment recommendation that the patient previously resisted. This contrasted with the rest of the data set, where diagnosis was either not delivered (as patients are presenting with ongoing problems) or delivered without using the PHQ-9. When clinician-administered, the PHQ-9 can be influenced by how response items are presented. This can lead to either downgrading or upgrading the severity of depression.