Design Issues for Socially Intelligent User Interfaces

Objectives: This study aims to demonstrate the usability of discourse analyses as a means of evaluating medical informatics systems by examining one particular computer-based data-to-text system for delivering neonatal health care information.

Methods: Six textual summaries of clinical information, three produced by human clinicians and three by the data-to-text system, were subjected to fine-grain discourse analysis. Analysis was performed ‘blind’ on all six textual summaries. Analysis focused on the identification of lexical items and on the potential effects of these items on users of these clinical information summaries.

Results: Results showed that there were clear differences between human- and system-generated clinical summaries, with human clinicians providing better narrative flow and textual detail. The data-to-text system successfully produced textual summaries although it fell short of human abilities.

Conclusions: These results indicate potential future improvements to the system. Discourse analysis as used here may offer significant advantages in evaluating and developing similar medical informatics systems.

Comparison of the accuracy of patients' recall of the content of telephone and face-to-face consultations: an exploratory study

BACKGROUND To comply with an action plan patients need to recall information accurately. Little is known about how well patients recall consultations, particularly telephone consultations increasingly used to triage acute problems. PURPOSE OF STUDY This was an exploratory study to measure how accurately patients recall the content of face-to-face and telephone consultations and what factors may be associated with accurate recall. STUDY DESIGN In Scotland in 2008, the advice (diagnoses; management plan(s); and safety-netting arrangements) given in audio recorded face-to-face and telephone consultations was compared with the advice recalled by patients at interview approximately 13&emsp14;days later. Patients also performed a memory test. Interactions were sought between accurate recall, consultation type, and factors postulated to influence recall. RESULTS Ten general practitioners (GPs) and 175 patients participated; 144 (82%) patients were interviewed. Patients recalled most important components of telephone and face-to-face consultations equally accurately or with only minor errors. Overall, patients presenting multiple problems (p<0.001), with brain injury (p<0.01) or low memory score (p<0.01) had reduced recall. GPs rarely used strategies to improve recall; however, these were not associated with improved recall. CONCLUSIONS Contrary to previous hospital based research, patients tended to remember important components of both face-to-face and telephone consultations-perhaps reflecting the familiar, less anxiety provoking environment of primary care. The unsuccessful use of strategies to improve recall may reflect selective use in cognitively impaired patients. Clinicians should compensate for situations where recall is poorer such as patients presenting multiple problems or with brain injury. Patients might be advised to restrict the number of problems they present in any one consultation.

Short-term memory binding deficits in Alzheimer's disease

Alzheimer's disease impairs long term memories for related events (e.g. faces with names) more than for single events (e.g. list of faces or names). Whether or not this associative or 'binding' deficit is also found in short-term memory has not yet been explored. In two experiments we investigated binding deficits in verbal short-term memory in Alzheimer's disease. Experiment 1: 23 patients with Alzheimer's disease and 23 age and education matched healthy elderly were recruited. Participants studied visual arrays of objects (six for healthy elderly and four for Alzheimer's disease patients), colours (six for healthy elderly and four for Alzheimer's disease patients), unbound objects and colours (three for healthy elderly and two for Alzheimer's disease patients in each of the two categories), or objects bound with colours (three for healthy elderly and two for Alzheimer's disease patients). They were then asked to recall the items verbally. The memory of patients with Alzheimer's disease for objects bound with colours was significantly worse than for single or unbound features whereas healthy elderly's memory for bound and unbound features did not differ. Experiment 2: 21 Alzheimer's disease patients and 20 matched healthy elderly were recruited. Memory load was increased for the healthy elderly group to eight items in the conditions assessing memory for single or unbound features and to four items in the condition assessing memory for the binding of these features. For Alzheimer's disease patients the task remained the same. This manipulation permitted the performance to be equated across groups in the conditions assessing memory for single or unbound features. The impairment in Alzheimer's disease patients in recalling bound objects reported in Experiment 1 was replicated. The binding cost was greater than that observed in the healthy elderly group, who did not differ in their performance for bound and unbound features. Alzheimer's disease grossly impairs the mechanisms responsible for holding integrated objects in verbal short-term memory.

Summarising complex ICU data in natural language: demonstration of the BT-45 system

As ICUs generate increasing amounts of information, writing medical reports involves complex time-consuming reasoning to build a coherent text which will be meaningful to those who will use it for decision making (e.g.: for nurse handover). Moreover, it has been shown that summarizing complex multi-channel physiological and discrete data in natural language (text) can lead to better decision-making in the intensive care unit (ICU). To facilitate this summarisation, as part of the BabyTalk project, we have developed a system called BT-45 that automatically generates textual summaries from periods of continuous and discrete data in a neonatal ICU. The demonstration will show the system running on real data and will detail the steps in the construction of the final text. Although these summaries are not yet as good as those generated by human experts, we have demonstrated experimentally that they lead to as good decision-making as can be achieved through presenting the same data graphically.

Summarising complex ICU data in natural language

It has been shown that summarizing complex multi-channel physiological and discrete data in natural language (text) can lead to better decision-making in the intensive care unit (ICU). As part of the BabyTalk project, we describe a prototype system (BT-45) which can generate such textual summaries automatically. Although these summaries are not yet as good as those generated by human experts, we have demonstrated experimentally that they lead to as good decision-making as can be achieved through presenting the same data graphically.

Mismatched concepts in a neonatal intensive care unit (NICU): further issues for computer decision support?

BACKGROUND

Common concepts and definitions are important for the effective practice of medicine. In an intensive care unit clear understanding of terminology and communication between different staff groups may be critical for optimal care. If computerised decision support tools are to be successfully deployed in these high intensity environments, all staff must understand the concepts and information that is to be portrayed.

OBJECTIVE

To examine the similarity of language and concepts related to newborn infants and their care in staff groups with different experience.

METHODS

An experimental study in a tertiary regional neonatal intensive care unit involved 32 staff in 4 groups with varying experience (junior and senior / nurses and doctors). A psychologist developed a lexicon of clinical actions and a second lexicon of possible patient descriptors applicable to newborn infants receiving all degrees of neonatal care by conducting interviews with staff on the unit. Card sorting experiments were performed on the terms in the action and descriptor lexicons, the staff being asked to group words that were related to the same or similar concepts. The card sort data were analysed using conventional cluster analysis to produce tree-diagrams or dendrograms and then by distance matrix analysis to give cumulative probability plots.

RESULTS

Differences were shown in the way various classes of staff and staff with different experience mentally map clinical concepts. Clinical actions were grouped more randomly by nurses and by those with less experience with a polarisation between senior doctors and junior nurses. Descriptors were classed more definitively and similarly by junior and senior nurses and senior doctors but more randomly and quite differently by junior doctors. Thus there were differences seen between nurses and doctors with different roles within the unit and differences related to experience.

CONCLUSIONS

Concepts are used differently by various staff groups in a neonatal unit: this may have an impact on the effectiveness of computerised decision aids unless it is taken into account during their development.

Role and experience determine decision support interface requirements in a neonatal intensive care environment

The aim of this paper is to describe a novel approach to the analysis of data obtained from card-sorting experiments. These experiments were performed as a part of the initial phase of a project, called NEONATE. One of the aims of the project is to develop decision support tools for the neonatal intensive care environment. Physical card-sorts were performed using clinical "action" and patient "descriptor" words. Thirty-two staff (eight junior nurses, eight senior nurses, eight junior doctors, and eight senior doctors) participated in the actions card-sorts and the same number of staff participated in separate descriptors card-sorting experiments. To check for consistency, the card-sorts were replicated for nurses during the action card-sorts. The card-sort data were analysed using hierarchical cluster analysis to produce tree-diagrams or dendrograms. Differences were shown in the way various classes of staff with different levels of experience mentally map clinical concepts. Clinical actions were grouped more loosely by nurses and by those with less experience, with a polarisation between senior doctors and junior nurses. Descriptors were classed more definitively and similarly by nurses and senior doctors but in a less structured way and quite differently by junior doctors. This paper presents a summary of the differences in the card-sort data for the various staff categories. It is shown that concepts are used differently by various staff groups in a neonatal unit and that this may diminish the effectiveness of computerised decision aids unless it is explored during their development.

Computerisation and decision making in neonatal intensive care: a cognitive engineering investigation

This paper reports results from a cognitive engineering study that looked at the role of computerised monitoring in neonatal intensive care. A range of methodologies was used: interviews with neonatal staff, ward observations, and experimental techniques. The purpose was to investigate the sources of information used by clinicians when making decisions in the neonatal ICU. It was found that, although it was welcomed by staff, computerised monitoring played a secondary role in the clinicians' decision making (especially for junior and nursing staff) and that staff used the computer less often than indicated by self-reports. Factors that seemed to affect staff use of the computer were the lack (or shortage) of training on the system, the specific clinical conditions involved, and the availability of alternative sources of information. These findings have relevant repercussions for the design of computerised decision support in intensive care and suggest ways in which computerised monitoring can be enhanced, namely: by systematic staff training, by making available online certain types of clinical information, by adapting the user interface, and by developing intelligent algorithms.

INFORM: European survey of computers in intensive care units

The aims of this study were (a) to survey and evaluate the impact of information technology applications in High Dependency Environments (HDEs) on organizational, psychological and cost-effectiveness factors, (b) to contribute information and design requirements to the other workpackages in the INFORM Project, and (c) to develop useful evaluation methodologies. The evaluation methodologies used were: questionnaires, case studies, objective findings (keystroke) and literature search and review. Six questionnaires were devised covering organizational impact, cost-benefit impact and perceived advantages and disadvantages of computerized systems in HDE (psychological impact). The general conclusion was that while existing systems have been generally well received, they are not yet designed in such a developed and integrated way as to yield their full potential. Greater user involvement in design and implementation and more emphasis on training emerged as strong requirements. Lack of reliability leading to parallel charting was a major problem with the existing systems. It proved difficult to assess cost effectiveness due to a lack of detailed accounting costs; however, it appeared that in the short term, computerisation in HDEs tended to increase costs. It is felt that through a better stock control and better decision making, costs may be reduced in the longer run and effectiveness increased; more detailed longitudinal studies appear to be needed on this subject.

The decline of working memory in Alzheimer's disease. A longitudinal study

A previous study (Baddeley et al., 1986) explored the hypothesis that patients suffering from dementia of the Alzheimer type (AD) are particularly impaired in the functioning of the central executive component of working memory. It showed that, when patients are required to perform 2 concurrent tasks simultaneously, the AD patients are particularly impaired, even when level of performance on the individual tasks is equated with that of age-matched controls. Although the results were clear, interpretation was still complicated by 2 issues: first, the question of comparability of performance on the separate tests between AD and control patients; secondly, the question of whether our results could be interpreted simply in terms of a limited general processing capacity being more taxed by more difficult dual tasks than by the individual tasks performed alone. The present study followed up the AD and control patients after 6 and 12 mths. We were able to allow for the problem of comparability of performance by using patients as their own control. Under these conditions, there is a very clear tendency for dual task performance to deteriorate while single task performance is maintained. A second experiment varied difficulty within a single task in which patients and controls were required to categorize words as belonging to 1, 2 or 4 semantic categories. There was a clear effect of number of categories on performance and a systematic decline in performance over time. There was, however, no interaction between task difficulty as measured by number of alternatives and rate of deterioration, suggesting that the progressive deterioration in performance shown by AD patients is a function of whether single or dual task performance is required, and is not dependent on simple level of task difficulty. Implications for the analysis of the central executive component of working memory are discussed.

Human factors and computerisation in intensive care units: a review

Current literature on the computerisation of Intensive Care Units indicates that many human factor considerations are relevant to the design and introduction of computer systems and to the evaluation of such systems within this environment. This paper aims to review and summarise essential points from the literature.

Users' perceptions of a computerised information system in intensive care (ABICUS) on introduction and after 2 months use

The aim of the present study was to assess the perceived utility of a computerised information system in an intensive care unit (ICU). Questionnaires were devised in which ICU staff indicated the ease or difficulty of obtaining and recording information (a) under the previous manual system, (b) soon after the introduction of the computerised system and (c) two months after computerisation. Results indicated the system was well received immediately and this favourable attitude persisted unchanged after two months experience. The questionnaire method also served to pinpoint some particular interface problems which are to be remedied in future versions of the system.

Dementia and working memory

This study explored the hypothesis that patients suffering from dementia of the Alzheimer type (DAT) are particularly impaired in the functioning of the Central Executive component of working memory, and that this will be reflected in the capacity of patients to perform simultaneously two concurrent tasks. DAT patients, age-matched controls and young controls were required to combine performance on a tracking task with each of three concurrent tasks, articulatory suppression, simple reaction time to a tone and auditory digit span. The difficulty of the tracking task and length of digit sequence were both adjusted so as to equate performance across the three groups when the tasks were performed alone. When digit span or concurrent RT were combined with tracking, the deterioration in performance shown by the DAT patients was particularly marked.