Framework for design and evaluation of complex interventions to improve health

Instruments to measure acceptability of information and acquisition of knowledge in patients with heart failure

BACKGROUND

Patients with heart failure suffer from poor health outcomes and require combinations of medications to treat their disease. Providing patients with knowledge through education is one mechanism to help them improve compliance with complicated treatment regimens.

METHODS

We developed and tested two instruments. The first instrument, which we call the measure of educational material acceptability (EMA), was designed to help us differentiate between written educational materials according to patients' subjective responses. The second instrument, the knowledge acquisition questionnaire (KAQ), which measures knowledge gained, was designed to determine whether patients understand the rationale and mechanics of their heart failure management. We explored the measurement properties of both instruments.

RESULTS

The internal consistency of the EMA was 0.79 (Cronbach's alpha). The internal consistency of the KAQ was 0.61 and its responsiveness, measured using change scores of knowledge before and after an educational intervention, was 0.75.

CONCLUSIONS

We have developed instruments that measure acceptability and knowledge acquisition, and that clinicians and investigators involved in heart failure programs may find useful in developing educational material and measuring the impact of their interventions on patients' knowledge.

Using socioeconomic evidence in clinical practice guidelines

The effects of socioeconomic position on health have been largely ignored in clinical guidelines. Australia's National Health and Medical Research Council has produced a framework to ensure that they are taken into account

Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students

OBJECTIVE

To assess the educational effectiveness on learning evidence based medicine of a handheld computer clinical decision support tool compared with a pocket card containing guidelines and a control.

DESIGN

Randomised controlled trial.

SETTING

University of Hong Kong, 2001.

PARTICIPANTS

169 fourth year medical students.

MAIN OUTCOME MEASURES

Factor and individual item scores from a validated questionnaire on five key self reported measures: personal application and current use of evidence based medicine; future use of evidence based medicine; use of evidence during and after clerking patients; frequency of discussing the role of evidence during teaching rounds; and self perceived confidence in clinical decision making.

RESULTS

The handheld computer improved participants' educational experience with evidence based medicine the most, with significant improvements in all outcome scores. More modest improvements were found with the pocket card, whereas the control group showed no appreciable changes in any of the key outcomes. No significant deterioration was observed in the improvements even after withdrawal of the handheld computer during an eight week washout period, suggesting at least short term sustainability of effects.

CONCLUSIONS

Rapid and convenient access to valid and relevant evidence on a portable computing device can improve learning in evidence based medicine, increase current and future use of evidence, and boost students' confidence in clinical decision making.

Assessing the value of assertive outreach. Qualitative study of process and outcome generation in the UK700 trial

BACKGROUND

It is unclear why intensive case management (ICM) failed to reduce hospitalisation in the UK700 trial.

AIMS

To investigate outcome generation in the UK700 trial.

METHOD

A qualitative investigation was undertaken in one UK700 centre.

RESULTS

Both intensive and standard case management practised individual casework, employed assertive outreach with comparable frequency, and performed similarly in the out-patient management of emergencies and in-patient discharge. However, ICM was advantaged in managing some non-compliance and undertaking casework that prevented psychiatric emergencies. Absence of team-based management and bureaucratised access to social care limited the impact of these differences on outcomes and the effective practice of assertive outreach, although this was relevant to only a sub-population of patients.

CONCLUSIONS

The impact of ICM was undermined by organisational factors. Sensitive anticipatory casework, which prevents psychiatric emergencies, may make ICM more effective than an exclusive focus on assertive outreach. Our findings demonstrate the value of qualitative research in evaluating complex interventions.

The effectiveness and costs of comprehensive geriatric evaluation and management

Comprehensive geriatric assessment (CGA) is a multidimensional interdisciplinary diagnostic process focused on determining a frail elderly person's medical, psychological, and functional capabilities in order to develop a coordinated and integrated plan for treatment and long-term follow-up. Geriatrics interventions building on CGA are defined from their historical emergence to the present day in a discussion of their complexity, goals and normative components. Through literature review, questions of the effectiveness and costs of these interventions are addressed. Evidence of effectiveness is derived from individual trials and, particularly, recent systematic reviews. While the trial evidence lends support to the proposition that geriatric interventions can be effective, the results have not been uniform. Review of meta-regression studies suggests that much of this outcome variability is related to identifiable program design parameters. In particular, targeting the frail, an interdisciplinary team structure with clinical control of care, and long-term follow-up, tend to be associated with effective programs. Answers to cost-effectiveness questions also vary and are more rare. With some exceptions, existing evidence as exists suggest that geriatrics interventions can be effective without raising total costs of care. Despite the attention given to these questions in recent years, there is still much room for clinical and scientific advance as we move to better understand what CGA interventions do well and in whom.

A conceptual framework for evaluating outpatient electronic prescribing systems based on their functional capabilities

OBJECTIVE

Electronic prescribing (e-prescribing) may substantially improve health care quality and efficiency, but the available systems are complex and their heterogeneity makes comparing and evaluating them a challenge. The authors aimed to develop a conceptual framework for anticipating the effects of alternative designs for outpatient e-prescribing systems.

DESIGN

Based on a literature review and on telephone interviews with e-prescribing vendors, the authors identified distinct e-prescribing functional capabilities and developed a conceptual framework for evaluating e-prescribing systems' potential effects based on their capabilities. Analyses of two commercial e-prescribing systems are presented as examples of applying the conceptual framework.

MEASUREMENTS

Major e-prescribing functional capabilities identified and the availability of evidence to support their specific effects.

RESULTS

The proposed framework for evaluating e-prescribing systems is organized using a process model of medication management. Fourteen e-prescribing functional capabilities are identified within the model. Evidence is identified to support eight specific effects for six of the functional capabilities. The evidence also shows that a functional capability with generally positive effects can be implemented in a way that creates unintended hazards. Applying the framework involves identifying an e-prescribing system's functional capabilities within the process model and then assessing the effects that could be expected from each capability in the proposed clinical environment.

CONCLUSION

The proposed conceptual framework supports the integration of available evidence in considering the full range of effects from e-prescribing design alternatives. More research is needed into the effects of specific e-prescribing functional alternatives. Until more is known, e-prescribing initiatives should include provisions to monitor for unintended hazards.

Acupuncture for People with Chronic Illness: Combining Qualitative and Quantitative Outcome Assessment

OBJECTIVES

Many people with chronic disease seek acupuncture treatment, despite it being largely excluded from Western state-funded health care systems. What benefits and problems do such people perceive and experience as important? To what extent do three subjective health questionnaires encompass and measure these treatment effects?

DESIGN AND OUTCOME MEASURES

A longitudinal qualitative study, using a constant comparative method, informed by grounded theory. Each person was interviewed three times over 6 months and before each interview participants completed three health status questionnaires, EuroQol, COOP-WONCA charts, and MYMOP2. Semistructured interviews used open-ended questions to explore peoples' experiences of illness and treatment, and then used the cognitive interview technique to discuss their questionnaire responses.

SUBJECTS AND SETTINGS

Eight professional acupuncturists in seven different settings publicized the study to their patients. We interviewed a maximum variation sample of 23 people with chronic illness, who were having acupuncture for the first time.

RESULTS

In addition to changes in their presenting symptoms people experienced whole-person effects that were characterized by changes in strength and energy, and changes in personal and social identity. These effects were distinct but not divisible because of their interdependence and their complex and individual development over time. The health status questionnaires varied in their ability to reflect and measure these changes, and the whole person effects were often missed. After 6 months some people had changed their treatment goals and some individuals were still seeing health improvement. One person found that acupuncture exacerbated her symptoms.

CONCLUSIONS

Knowledge of the range, individuality, and varied temporal patterns of treatment outcomes can guide potential patients, clinicians, health care providers, and researchers. Qualitative methods have highlighted the strengths and weaknesses of different outcome questionnaires, and the importance of long-term follow-up. We plan to do further work on appropriate outcome questionnaire selection and development.

Using combined research methods for exploring diabetes patient education

This paper explores a trial of an educational intervention designed for people who have Type 2 diabetes. The aim of the trial was to understand how the intervention had influenced outcomes in the context of participants' everyday lives. A randomised-controlled wait-list trial design was used. The study was also informed by a qualitative approach which explored the meanings held by participants for informing their behaviours. Outcomes were measured using diabetes-specific questionnaires and clinical measures of blood glucose control, weight and drug treatment. Alongside these, focus group discussions were held to explore how outcome effects had transpired. Using these different methods resulted in two separate data sets which required diverse methods for analysis. This paper uses examples of compatibilities and contradictions between the data sets to look at how they were combined to produce valid results. Conclusions drawn showed that a combined methods approach was essential to expand the scope and improve the analytic power of trials of patient education. It produced illuminating results which provided guidelines for practice and suggested further areas for research.

A service-academic partnership in primary care research: one practice's experience

Further development of a strong research base for general practice is important if the profession is to respond appropriately to its central role in service provision. It can be difficult for general practitioners (GPs) who have not pursued an academic career path to make a significant contribution to research. The development of a service-academic partnership is described, together with an honest account of the difficulties encountered.

Research methods: beyond the clinical trial

Emergency medicine differs from other specialties in that it has few restrictions in terms of pathology or patient group but is instead defined by time, with patients being selected on the basis of urgency. The important research questions generated by emergency medicine are therefore often complex and relate to organizational, economic, or social issues. Clinical trials have a limited role to play in these circumstances, and concepts such as the hierarchy of evidence might be unhelpful if the best methodology is not appropriate to the research question. Emergency medicine researchers therefore need to be prepared to use a wide range of methods, often in combination and often drawing on the social sciences. This article will introduce readers to methods from clinical epidemiology, operational research, health economics, and qualitative research, discussing the value of these approaches and identifying potential pitfalls.

The value of therapeutic communities in the treatment of personality disorder: a suitable place for treatment?

In the United Kingdom, a government program investigating the links between offending and personality disorder has stimulated renewed interest in the treatment of personality disorders. One psychosocial treatment option for patients with personality disorders is the therapeutic community (TC). In 2000, the authors conducted a small qualitative study with a sample of psychiatrists which suggested that TCs were not well understood and that the status of evidence on efficacy might be partly responsible for low referral numbers. This article reviews the evidence for the efficacy and cost-effectiveness of TCs as a treatment for personality disorders and considers which types of disorders are amenable to TC treatment. We conclude that there is a strong case for more rigorous evaluation and that some of the difficulties anticipated in applying randomized clinical trial (RCT) methodology to the study of TCs could be overcome.

Designing a quality improvement intervention: a systematic approach

Most quality improvement or change management interventions are currently designed intuitively and their results are often disappointing. While improving the effectiveness of interventions requires systematic development, no specific methodology for composing intervention strategies and programmes is available. This paper describes the methodology of systematically designing quality of care improvement interventions, including problem analysis, intervention design and pretests. Several theories on quality improvement and change management are integrated and valuable materials from health promotion are added. One method of health promotion-intervention mapping-is introduced and applied. It describes the translation of knowledge about barriers to and facilitators of change into a concrete intervention programme. Systematic development of interventions, although time consuming, appears to be worthwhile. Decisions that have to be made during the design process of a quality improvement intervention are visualised, allowing them to serve as a starting point for a systematic evaluation of the intervention.

Improving the detection and management of depression in primary care

The effectiveness of screening and organisational strategies to improve the recognition and management of depression in primary care published in a recent issue of Effective Health Care is reviewed.

Factors affecting general practitioners' decisions about plain radiography for back pain: implications for classification of guideline barriers--a qualitative study

BACKGROUND

General practitioners often diverge from clinical guidelines regarding spine radiography. This study aimed to identify and describe A) factors general practitioners consider may affect their decisions about ordering plain radiography for back pain and B) barriers to guideline adherence suggested by such factors.

METHODS

Focus group interviews regarding factors affecting ordering decisions were carried out on a diverse sample of Norwegian general practitioners and were analysed qualitatively. Results of this study and two qualitative studies from the Netherlands and USA on use of spine radiography were interpreted for barriers to guideline adherence. These were compared with an existing barrier classification system described by Dr Cabana's group.

RESULTS

The factors which Norwegian general practitioners considered might affect their decisions about ordering plain radiography for back pain concerned the following broader issues: clinical ordering criteria, patients' wishes for radiography and the general practitioner's response, uncertainty, professional dignity, access to radiology services, perception of whether the patient really was ill, sense of pressure from other health care providers/social security, and expectations about the consequences of ordering radiography. The three studies suggested several attitude-related and external barriers as classified in a previously reported system described by Dr Cabana in another study. Identified barriers not listed in this system were: lack of expectancy that guideline adherence will lead to desired health care process, emotional difficulty with adherence, improper access to actual/alternative health care services, and pressure from health care providers/organisations.

CONCLUSIONS

Our findings may help implement spine radiography guidelines. They also indicate that Cabana et al.'s barrier classification system needs extending. A revised system is proposed.

Randomised controlled trial of a theoretically grounded tailored intervention to diffuse evidence-based public health practice [ISRCTN23257060]

BACKGROUND

Previous studies have shown that Norwegian public health physicians do not systematically and explicitly use scientific evidence in their practice. They work in an environment that does not encourage the integration of this information in decision-making. In this study we investigate whether a theoretically grounded tailored intervention to diffuse evidence-based public health practice increases the physicians' use of research information.

METHODS

148 self-selected public health physicians were randomised to an intervention group (n = 73) and a control group (n = 75). The intervention group received a multifaceted intervention while the control group received a letter declaring that they had access to library services. Baseline assessments before the intervention and post-testing immediately at the end of a 1.5-year intervention period were conducted. The intervention was theoretically based and consisted of a workshop in evidence-based public health, a newsletter, access to a specially designed information service, to relevant databases, and to an electronic discussion list. The main outcome measure was behaviour as measured by the use of research in different documents.

RESULTS

The intervention did not demonstrate any evidence of effects on the objective behaviour outcomes. We found, however, a statistical significant difference between the two groups for both knowledge scores: Mean difference of 0.4 (95% CI: 0.2-0.6) in the score for knowledge about EBM-resources and mean difference of 0.2 (95% CI: 0.0-0.3) in the score for conceptual knowledge of importance for critical appraisal. There were no statistical significant differences in attitude-, self-efficacy-, decision-to-adopt- or job-satisfaction scales. There were no significant differences in Cochrane library searching after controlling for baseline values and characteristics.

CONCLUSION

Though demonstrating effect on knowledge the study failed to provide support for the hypothesis that a theory-based multifaceted intervention targeted at identified barriers will change professional behaviour.

Intensive, time-series measurement of upper limb recovery in the subacute phase following stroke

OBJECTIVES

To discover if intensive monitoring of wrist extension would produce consistent recovery curves during the subacute period, and whether any impact of additional physiotherapy could be detected. We also investigated improved approaches to statistical analysis in single-case experiments.

DESIGN

A randomized multiple-baseline experiment with very frequent assessment.

SETTING

Stroke rehabilitation unit.

SUBJECTS

Four patients with some active wrist movement less than seven weeks after stroke.

INTERVENTIONS

Wrist extension was measured twice daily with an electrogoniometer for 3-4 weeks. Additional upper limb physiotherapy 115 minutes, twice per day) commenced after a randomly determined period.

MAIN OUTCOME MEASURES

Speed and range of wrist movement.

RESULTS

A logarithmic function was fitted to the data to produce recovery curves. In all cases, active range and maximum velocity of wrist extension rose gradually over time. Mean variability in range was <5%, but with occasional outliers. Range of passive movement decreased in two cases in association with pain and increased tone. There were no large improvements coinciding with additional physiotherapy but autoregression analysis indicated statistically significant changes in three cases. A randomization test confirmed an increase in active range associated with additional physiotherapy.

CONCLUSIONS

Intensive electrogoniometry provided a detailed recovery pattern for each of these patients. The data were surprisingly consistent over time, showing that it is feasible to use a time-series approach to investigate subacute recovery. Changes associated with additional physiotherapy were observed on some measures, demonstrating the potential of this approach for exploratory evaluation of interventions.

Research designs for studies evaluating the effectiveness of change and improvement strategies

The methods of evaluating change and improvement strategies are not well described. The design and conduct of a range of experimental and non-experimental quantitative designs are considered. Such study designs should usually be used in a context where they build on appropriate theoretical, qualitative and modelling work, particularly in the development of appropriate interventions. A range of experimental designs are discussed including single and multiple arm randomised controlled trials and the use of more complex factorial and block designs. The impact of randomisation at both group and individual levels and three non-experimental designs (uncontrolled before and after, controlled before and after, and time series analysis) are also considered. The design chosen will reflect both the needs (and resources) in any particular circumstances and also the purpose of the evaluation. The general principle underlying the choice of evaluative design is, however, simple-those conducting such evaluations should use the most robust design possible to minimise bias and maximise generalisability.

External incentives, information technology, and organized processes to improve health care quality for patients with chronic diseases

CONTEXT

Organized care management processes (CMPs) can improve health care quality for patients with chronic diseases. The Institute of Medicine of the National Academy of Sciences has called for public and private purchasers of health care to create incentives for physician organizations (POs) to use CMPs and for the government to assist POs in implementing information technology (IT) to facilitate CMP use. Research is lacking about the extent to which POs use CMPs or about the degree to which incentives, IT, or other factors are associated with their use.

OBJECTIVES

To determine the extent to which POs with 20 or more physicians use CMPs and to identify key factors associated with CMP use for 4 chronic diseases (asthma, congestive heart failure, depression, and diabetes).

DESIGN, SETTING, AND PARTICIPANTS

One thousand five hundred eighty-seven US POs (medical groups and independent practice associations) with 20 or more physicians were identified using 5 large databases. One thousand one hundred four of these POs (70%) agreed to participate in a telephone survey conducted between September 2000 and September 2001. Sixty-four responding POs were excluded because they did not treat any of the 4 diseases, leaving 1040 POs.

MAIN OUTCOME MEASURES

Extent of use of CMPs as calculated on the basis of a summary measure, a PO care management index (POCMI; range, 0-6) and factors associated with CMP use.

RESULTS

Physician organizations' mean use of CMPs was 5.1 of a possible 16; 50% used 4 or fewer. External incentives and clinical IT were most strongly associated with CMP use. Controlling for other factors, use of the 2 most strongly associated incentives-public recognition and better contracts for health care quality-was associated with use of 1.3 and 0.7 additional CMPs, respectively (P<.001 and P =.007). Each additional IT capability was associated with 0.37 additional CMPs (P<.001). However, 33% of POs reported no external incentives and 50% reported no clinical IT capability.

CONCLUSIONS

The use of CMPs varies greatly among POs, but it is low on average. Government and private purchasers of health care may increase CMP use by providing external incentives for improvement of health care quality to POs and by assisting them in improving their clinical IT capability.

Effect on hip fractures of increased use of hip protectors in nursing homes: cluster randomised controlled trial

OBJECTIVE

To assess the effects of an intervention programme designed to increase use of hip protectors in elderly people in nursing homes.

DESIGN

Cluster randomised controlled trial with 18 months of follow up.

SETTING

Nursing homes in Hamburg (25 clusters in intervention group; 24 in control group).

PARTICIPANTS

Residents with a high risk of falling (459 in intervention group; 483 in control group).

INTERVENTION

Single education session for nursing staff, who then educated residents; provision of three hip protectors per resident in intervention group. Usual care optimised by brief information to nursing staff about hip protectors and provision of two hip protectors per cluster for demonstration purposes.

MAIN OUTCOME MEASURE

Incidence of hip fractures.

RESULTS

Mean follow up was 15 months for the intervention group and 14 months for the control group. In total 167 residents in the intervention group and 207 in the control group died or moved away. There were 21 hip fractures in 21 (4.6%) residents in the intervention group and 42 hip fractures in 39 (8.1%) residents in the control group (relative risk 0.57, absolute risk difference -3.5%, 95% confidence interval -7.3% to 0.3%, P=0.072). After adjustment for the cluster randomisation the proportions of fallers who used a hip protector were 68% and 15% respectively (mean difference 53%, 38% to 67%, P=0.0001). There were 39 other fractures in the intervention group and 38 in the control group.

CONCLUSION

The introduction of a structured education programme and the provision of free hip protectors in nursing homes increases the use of protectors and may reduce the number of hip fractures.

Evaluation of inherent performance of intelligent medical decision support systems: utilising neural networks as an example

Researchers who design intelligent systems for medical decision support, are aware of the need for response to real clinical issues, in particular the need to address the specific ethical problems that the medical domain has in using black boxes. This means such intelligent systems have to be thoroughly evaluated, for acceptability. Attempts at compliance, however, are hampered by lack of guidelines. This paper addresses the issue of inherent performance evaluation, which researchers have addressed in part, but a Medline search, using neural networks as an example of intelligent systems, indicated that only about 12.5% evaluated inherent performance adequately. This paper aims to address this issue by concentrating on the possible evaluation methodology, giving a framework and specific suggestions for each type of classification problem. This should allow the developers of intelligent systems to produce evidence of a sufficiency of output performance evaluation.

The benefits of a qualitative approach to telemedicine research

We used qualitative research to evaluate the experience of the participants in teleconsultations between primary and secondary care. Semistructured interviews were conducted with 15 hospital specialists, 24 general practitioners and 30 patients. Focus groups were also held with hospital specialists (two groups), general practitioners (six groups) and administrative staff (five groups). Sixty teleconsultations in six different specialties were video-recorded. Early findings show that the participants (hospital specialists, general practitioners and patients) had different perceptions of the same teleconsultations. Furthermore, the participants perceptions of consultations differed from those of the researchers.

Factors affecting uptake of childhood immunisation: a Bayesian synthesis of qualitative and quantitative evidence

Falls in levels of measles, mumps, and rubella (MMR) immunisation in the UK and the continuing debate on how to respond to this situation emphasise the importance of identifying and understanding the factors that affect the uptake of recommended childhood immunisations. Both qualitative and quantitative evidence could be useful in this process. We aimed to explore the feasibility and value of an approach to formal synthesis of qualitative and quantitative evidence in the context of factors affecting the uptake of childhood immunisation in developed countries. We used a Bayesian approach to meta-analysis. Evidence from 11 qualitative and 32 quantitative studies of factors affecting uptake of childhood immunisation was combined and assessed. We conclude that use of either qualitative or quantitative research alone might not identify all relevant factors, or might result in inappropriate judgments about their importance, and could thus lead to inappropriate formulation of evidence-based policy. Further development of our methods might enable rigorous synthesis of qualitative and quantitative evidence in this and other contexts.

Gender differences in living with schizophrenia. A cross-sectional European multi-site study

The EPSILON project (European Psychistric Services: Inputs Linked to Outcomes and Needs) is a cross-sectional study of the clinical and social characteristics, needs, satisfaction with services, quality of life, and service utilisation and costs for people with schizophrenia in five European sites (Amsterdam, Copenhagen, London, Santander, and Verona). This study examined five hypotheses: (1) Men will have more total needs and more unmet needs for: 'accommodation', 'substance misuse', 'psychotic symptoms', 'harm to others', and 'sexual expression', whereas women will have more total needs and more unmet needs in the domains of 'childcare' and 'harm to self'. (2) Caregivers of male patients will show higher rates of psychological distress, and higher scores for 'supervision' and 'urging' than caregivers of female patients. (3) Male and female patients will show similar levels of satisfaction with services, both in total scores and subscores. (4) Male patients will show lower objective quality of life, but similar subjective quality of life compared with women. (5) Service utilization for men and women will differ, and patterns will vary by site. The results confirmed hypotheses 1 (in part) and 3, but failed to support hypotheses 2, 4 and 5. Graphical models were used to generate hypotheses for future research. The implications for planning separate services for male and female schizophrenic patients are discussed.

Preparing for change in the secondary prevention of coronary heart disease: a qualitative evaluation of cardiac rehabilitation within a region of Scotland

BACKGROUND

Secondary prevention of Coronary Heart Disease (CHD) is often poorly managed and its benefits attained in only a minority of those with CHD. Guidelines developed in the United Kingdom and North America suggest that in future cardiac rehabilitation programmes should provide services through individualized programmes that cater for a wide range of conditions associated with CHD. This will involve substantial and costly changes to current programmes that are mostly standardized and for postmyocardial infarction patients. Based on change theory, this study examined the dynamics, strengths and weaknesses of an existing programme in a Scottish region which was due to undergo the changes suggested by guidelines.

AIM

To examine the perceived provision of secondary prevention services for CHD from the perspectives of health professionals within one region in the West of Scotland.

METHODS

A purposive sample of 14 health professionals (eight primary and six secondary care health professionals) was selected to cover a range of professional roles including both specialists and generalists. Separate focus group discussions (2) were held with primary care and secondary care professionals.

FINDINGS

Whilst the health professionals were enthusiastic about CHD prevention and their involvement, they perceived barriers to the success of the existing service as being complex and multifactorial, including patient, social and service-related factors. Although both groups identified motivation as the most influential personal factor, secondary care staff tended to focus on the importance of patient factors in influencing motivation to change, whereas the primary care staff referred more to the cumulative effects of social and cultural factors. Professionals highlighted weaknesses in the transition between hospital and community-based services with regard to the information flow between primary and secondary care.

CONCLUSIONS

Although the study has immediate relevance for the local area, it highlighted issues of more general relevance to cardiac rehabilitation programme development and intersectoral working, such as communications and role perceptions in multi-professional working and the need to adapt services to local socio-economic conditions.

Integrating qualitative research into evidence based practice

This article attempts to provide an overview of qualitative tools and methods using mainly examples from diabetes research. The other articles in this issue of the Endocrinology and Metabolism Clinics of North America have demonstrated the enormous contribution made in the past 15 years or so by rigorous quantitative studies of prevalence, diagnosis, prognosis, and therapy to clinical decision-making in endocrinology. In the early 21st century, the state of qualitative research into such topics as the illness experience of diabetes; the barriers to effective self care and positive health choices; the design of complex educational interventions; the design of appropriate, acceptable and responsive health services; and the decision-making behavior of health professionals, is such that there remain many more questions than answers. But qualitative research is increasingly recognized as an important, legitimate and expanding dimension of evidence-based health care (18;19). It is highly likely that the major landmark studies in diabetes care over the next decade will build on an exploratory qualitative study or incorporate an explanatory or evaluative dimension based on qualitative methods.

Body image and weight change in middle age: a qualitative study

AIMS

To explore men's and women's experiences of weight change in adulthood, body image preferences and beliefs about the health consequences of overweight and to inform the development of a primary care intervention to prevent obesity.

SAMPLE

Seventy-two men and women aged 35-55, with a range of BMIs from 22 to 29.9, were identified from two UK general practice registers and invited to participate in an interview about experiences of weight change in adulthood.

METHODS

Audio tape recorded, semi-structured interviews were conducted in respondents' homes by trained researchers. Open-ended questions were used to collect experiences of weight change since early adulthood and views about weight change in middle age. Illustrations of a range of men's and women's body shapes were used to prompt discussion of respondents' preferences for male and female body shapes and their perspectives of the health, social and practical problems associated with underweight and overweight. The data were analysed using both quantitative and qualitative methods.

RESULTS

Some 87% (33/38) of the women and 59% (20/34) of the men said that they had ever tried to lose weight. At least one instance of successful weight loss was reported by 58% of the women and 47% of the men, although many of these attempts were relatively short-lived and often motivated by specific goals such as a holiday or a wedding. Respondents were sceptical of the possibility of controlling weight without considerable personal sacrifice. Explanations for middle-age weight gain included a sedentary lifestyle, as well as several gender-specific reasons, including hormonal changes and comfort eating for women and beer drinking for men. Nearly all (97%) respondents associated heart disease with overweight, while diabetes was mentioned by only 22% and none mentioned cancer.

CONCLUSION

People who have gained weight in middle age may be deterred from trying to prevent further gain by pessimism about the effort required. The efficacy of interventions to encourage relatively small substitutions and changes to diet and physical activity need to be tested. Interventions to help prevent weight gain in middle age could include information about the less widely known health risks such as diabetes and cancer.

Application of the development stages of a cluster randomized trial to a framework for valuating complex health interventions

INTRODUCTION

Trials of complex health interventions often pose difficult methodologic challenges. The objective of this paper is to assess the extent to which the various development steps of a cluster randomized trial to optimize antibiotic use in nursing homes are represented in a recently published framework for the design and evaluation of complex health interventions. In so doing, the utility of the framework for health services researchers is evaluated.

METHODS

Using the five phases of the framework (theoretical, identification of components of the intervention, definition of trial and intervention design, methodological issues for main trial, promoting effective implementation), corresponding stages in the development of the cluster randomized trial using diagnostic and treatment algorithms to optimize the use of antibiotics in nursing homes are identified and described.

RESULTS

Synthesis of evidence needed to construct the algorithms, survey and qualitative research used to define components of the algorithms, a pilot study to assess the feasibility of delivering the algorithms, methodological issues in the main trial including choice of design, allocation concealment, outcomes, sample size calculation, and analysis are adequately represented using the stages of the framework.

CONCLUSIONS

The framework is a useful resource for researchers planning a randomized clinical trial of a complex intervention.

Difficulties in conducting a randomized controlled trial of health service interventions in intellectual disability: implications for evidence-based practice

BACKGROUND

In an era of evidence-based medicine, practice is constantly monitored for quality in accordance with the needs of clinical governance (Oyebode et al. 1999). This is likely to lead to a dramatic change in the treatment of those with intellectual disability (ID), in which evidence for effective intervention is limited for much that happens in ordinary practice. As Fraser (2000, p. 10) has commented, the word that best explains "the transformation of learning disability practice in the past 30 years is 'enlightenment'." This is not enough to satisfy the demands of evidence, and Fraser exhorted us to embrace more research-based practice in a subject that has previously escaped randomized controlled trials (RCTs) of treatment because of ethical concerns over capacity and consent, which constitute a denial of opportunity which "is now at last regarded as disenfranchising".

CONCLUSIONS

The present paper describes the difficulties encountered in setting up a RCT of a common intervention, i.e. assertive community treatment, and concludes that a fundamental change in attitudes to health service research in ID is needed if proper evaluation is to prosper.

The changing nature of consultation-liaison in primary care: bridging the gap between research and practice

Consultation-liaison (C-L) psychiatry is hypothesized to be a model of interface between primary care and specialist mental health services with significant advantages over other models of organizing mental health care. However, there are significant complexities in the definition and evaluation of this model. As well as discussing the definition of C-L in primary care, this paper highlights the gap between models of traditional C-L that are popular in practice and the increasingly complex models (based on chronic disease management) evaluated in research studies. It is hypothesized that traditional C-L approaches and newer models use different mechanisms of change to achieve their goals. The former focus on the relationships between primary care and specialist professionals, while the latter highlight the importance of the development of effective systems of delivering care. Although the latter may be crucial in enhancing the "efficacy" and "effectiveness" of these models in terms of clinician behavior change and patient outcome, the former may be crucial in terms of "dissemination" and "implementation" of these models from research contexts to routine care settings.

Criteria for evaluating evidence on public health interventions

Public health interventions tend to be complex, programmatic, and context dependent. The evidence for their effectiveness must be sufficiently comprehensive to encompass that complexity. This paper asks whether and to what extent evaluative research on public health interventions can be adequately appraised by applying well established criteria for judging the quality of evidence in clinical practice. It is adduced that these criteria are useful in evaluating some aspects of evidence. However, there are other important aspects of evidence on public health interventions that are not covered by the established criteria. The evaluation of evidence must distinguish between the fidelity of the evaluation process in detecting the success or failure of an intervention, and the success or failure of the intervention itself. Moreover, if an intervention is unsuccessful, the evidence should help to determine whether the intervention was inherently faulty (that is, failure of intervention concept or theory), or just badly delivered (failure of implementation). Furthermore, proper interpretation of the evidence depends upon the availability of descriptive information on the intervention and its context, so that the transferability of the evidence can be determined. Study design alone is an inadequate marker of evidence quality in public health intervention evaluation.

A review of evidence of health benefit from artificial neural networks in medical intervention

The purpose of this review is to assess the evidence of healthcare benefits involving the application of artificial neural networks to the clinical functions of diagnosis, prognosis and survival analysis, in the medical domains of oncology, critical care and cardiovascular medicine. The primary source of publications is PUBMED listings under Randomised Controlled Trials and Clinical Trials. The rĵle of neural networks is introduced within the context of advances in medical decision support arising from parallel developments in statistics and artificial intelligence. This is followed by a survey of published Randomised Controlled Trials and Clinical Trials, leading to recommendations for good practice in the design and evaluation of neural networks for use in medical intervention.

Improving the delivery and organisation of mental health services: beyond the conventional randomised controlled trial

BACKGROUND

There is an ethical imperative to evaluate service and policy initiatives, such as those highlighted in the recent National Service Framework, just as there is to evaluate individual treatments.

AIMS

To outline the best methods available for evaluating the delivery and organisation of mental health services.

METHOD

We present a narrative methodological overview, using salient examples from mental health services research.

RESULTS

Cluster randomised studies involve the random allocation of groups of clinicians, clinical teams or hospitals rather than individual patients, and produce the least biased evaluation of mental health policy, organisation or service delivery. Where randomisation is impossible or impractical (often when services or policies are already implemented), then quasi-experimental designs can be used. Such designs have both strengths and many potential flaws.

CONCLUSIONS

The gold standard remains the randomised trial, but with due consideration to the unit of randomisation. Use of quasi-experimental designs can be justified in certain circumstances but should be attempted and interpreted with caution.

Patient and carer satisfaction with 'hospital at home': quantitative and qualitative results from a randomised controlled trial

BACKGROUND

'Hospital At Home' schemes are set to increase in the United Kingdom (UK) in response to the NHS Plan. To date, little detailed work has been done on the acceptability of these schemes to patients and their carers.

AIM

To compare Hospital at Home patient and carer satisfaction with hospital care.

DESIGN OF STUDY

Pragmatic randomised controlled trial.

SETTING

Consecutive patients assessed as suitablefor the Leicester Hospital at Home scheme were randomised to Hospital at Home or one of three acute hospitals in the city.

METHOD

Patient satisfaction was assessed two weeks after randomisation, or at discharge if later using a six-item questionnaire. Patients' and carers' views of the services were assessed by semistructured interviews.

RESULTS

One hundred and two patients were randomised to Hospital at Home and 97 to hospital. Forty-eight (47%) patients in the Hospital at Home arm and 35 (36%) in the hospital arm completed the satisfaction questionnaire, representing 96% and 85% of those eligible, respectively. Total scores were significantly higher in the Hospital at Home (median = 15) than in the hospital group (median = 12). (P<0.001, Mann-Whitney U-test.) Responses to all six questions favoured Hospital at Home, with all but one of these differences being statistically significant. In the Hospital at Homegroup, 24 patients and 18 of their carers were interviewed; in the hospital group 18 patients and seven of their carers were interviewed. Themes emerging from these interviews were that patients appreciated the more personal care and better communication offered by Hospital at Home and placed great value on staying at home, which was seen to be therapeutic. Patients largely felt safe in Hospital at Home, although some would have felt safer in hospital. Some patients and carers felt that better medical care would have been provided in hospital. Carers felt that the workload imposed by Hospital at Home was no greater than by hospital admission and that the relief from care duties at home would be counterbalanced by the added strain of hospital visiting.

CONCLUSIONS

Patient satisfaction was greater with Hospital at Home than with hospital. Reasons included a more personal style of care and a feeling that staying at home was therapeutic. Carers did not feel that Hospital at Home imposed an extra workload.

Is the randomized clinical trial the gold standard of research?

All research has flaws. Some flaws are so trivial that the research can still stand as the definitive study. Other flaws prevent a study from being definitive, but the study still provides useful guidance in the context of other research. Some flaws are so serious that the research provides no useful information at all. The tricky part is not finding flaws in the research but in deciding to what extent the flaws erode the credibility of the research. In general, the use of RCTs can add substantial credibility to a research study. There are calls for greater use of RCTs in many areas, such as surgery (Baum, 1999) and psychiatry (Andrews, 1999). Of course, nonrandomized trials are an important complement to RCTs when the latter are ethically inappropriate or logistically impossible (Black, 1996). Failure to use randomization or blinding, however, is not a fatal flaw. Furthermore, the artificial nature of RCTs will often restrict their applicability to overly simple interventions. When RCTs focus on narrow patient groups or exclude important segments of the population, there may be difficulty in generalizing their results. So it would be a mistake to label the RCT as a gold standard for all research. A silver standard may be a more appropriate label.

Physical, functional and other non-pharmacological interventions for osteoarthritis

Non-pharmacological interventions are frequently and widely used in the management of patients with osteoarthritis (OA). However, there is little evidence that most of these interventions are efficacious (i.e. that they work under ideal circumstances) owing to the paucity of research studies on these interventions and the fundamental methodological flaws in published studies. Moreover, the clinical effectiveness of non-pharmacological interventions (i.e. their efficacy under the conditions prevailing in clinical practice) is unknown, and cost-analysis of these interventions has not been carried out. If evidence-based management guidelines were to be constructed solely from firm research studies few could be recommended. Established and developing research methodologies and techniques should be employed to construct and conduct research programmes that appropriately evaluate the clinical and cost effectiveness of these complex non-pharmacological healthcare interventions. Establishing clinical effectiveness will ensure that we have practice-based evidence and are administering relevant, effective and optimal healthcare for patients with OA, maximizing the efficient use of healthcare resources.

Randomised controlled trial of training health visitors to identify and help couples with relationship problems following a birth

BACKGROUND

Stresses imposed by parenthood can provoke or intensify relationship problems between parents. These problems, which are often associated with postnatal depression, can have serious consequences for family well-being but are often not revealed to primary health care personnel.

AIM

To evaluate a means of extending the primary health care team's ability to identify and respond to relationship problems of mothers and their partners in the postnatal period.

DESIGN OF STUDY

Cluster randomised controlled trial.

SETTING

Specially trained health visitors in nine 'intervention' clinics--each matched with a 'control' clinic' in an outer London borough.

METHOD

Health visitors in intervention clinics invited mothers attending for the six-to-eight-week developmental check to complete a screening scale for relationship problems, and offered help (supportive listening, advice, or referral) if needed. When visiting the clinic for the 12-week immunizations, mothers from all clinics were asked to complete a follow-up self-report questionnaire. After the completion of the trial, 25 women who had attended the intervention clinics and had been offered support with a relationship problem were interviewed to elicit their views on the acceptability and value of the intervention. All 25 of the health visitors engaged in the intervention were asked to complete a questionnaire on their experience.

RESULTS

Screening led to striking differences between intervention and control clinics in the percentage of women identified at the six-to-eight-week check as potentially in need of help with a relationship problem (21% versus 5%, P = 0.007) and in the percentage actually offered help (18% versus 3%, P = 0.014). About one-half of the mothers so identified were also identified as having postnatal depression. At the 12-week visit for immunizations, the intervention group was twice as likely (P = 0.006) as the control group to report having discussed relationship problems with the health visitor and 75% more likely (P = 0.046) to report having received help with a problem.

CONCLUSION

The intervention offers a useful way of extending the primary health care team's ability to respond to problems that often have serious consequences for family well-being.