Assessing quality in cross-country comparisons of health systems and policies: towards a set of generic quality criteria

There is a growing body of cross-country comparisons in health systems and policy research. However, there is little consensus as to how to assess its quality. This is partly due to the fact that cross-country comparison constitutes a diverse inter-disciplinary field of study, with much variation in the motives for research, foci and levels of analyses, and methodological approaches. Inspired by the views of subject area experts and using the distinction between variable-based and case-based research, we briefly review the main different types of cross-country comparisons in health systems and policy research to identify pertinent quality issues. From this, we identify the following generic quality criteria for cross-country comparisons: (1) appropriate use of theory, (2) explicit selection of comparator countries, (3) rigour of the comparative design, (4) attention to the complexity of cross-national comparison, (5) rigour of the research methods, and (6) contribution to knowledge. This list may not be exclusive though publication and discussion of the list of criteria should help raise awareness in this field of what constitutes high quality research. In turn, this should be helpful for those planning, undertaking, or commissioning cross-country comparative research.

Studying policy implementation using a macro, meso and micro frame analysis: the case of the Collaboration for Leadership in Applied Health Research & Care (CLAHRC) programme nationally and in North West London

BACKGROUND

The publication of Best research for best health in 2006 and the "ring-fencing" of health research funding in England marked the start of a period of change for health research governance and the structure of research funding in England. One response to bridging the 'second translational gap' between research knowledge and clinical practice was the establishment of nine Collaborations for Leadership in Applied Health Research and Care (CLAHRCs). The goal of this paper is to assess how national-level understanding of the aims and objectives of the CLAHRCs translated into local implementation and practice in North West London.

METHODS

This study uses a variation of Goffman's frame analysis to trace the development of the initial national CLAHRC policy to its implementation at three levels. Data collection and analysis were qualitative through interviews, document analysis and embedded research.

RESULTS

Analysis at the macro (national policy), meso (national programme) and micro (North West London) levels shows a significant common understanding of the aims and objectives of the policy and programme. Local level implementation in North West London was also consistent with these.

CONCLUSIONS

The macro-meso-micro frame analysis is a useful way of studying the transition of a policy from high-level idea to programme in action. It could be used to identify differences at a local (micro) level in the implementation of multi-site programmes that would help understand differences in programme effectiveness.

Interpretations of referral appropriateness by senior health managers in five PCT areas in England: a qualitative investigation

Aim

To explore interpretations of “appropriate” and “inappropriate” elective referral from primary to secondary surgical care among senior clinical and non-clinical managers in five purposively sampled primary care trusts (PCTs) and their main associated acute hospitals in the English National Health Service (NHS).

Methods

Semi-structured face-to-face interviews were undertaken with senior managerial staff from clinical and non-clinical backgrounds. Interviews were tape-recorded, transcribed and analysed according to the Framework approach developed at the National Centre for Social Research using N6 (NUD*IST6) qualitative data analysis software.

Results

Twenty-two people of 23 approached were interviewed (between three and five respondents per PCT and associated acute hospital). Three attributes relating to appropriateness of referral were identified: necessity: whether a patient with given characteristics was believed suitable for referral; destination or level: where or to whom a patient should be referred; and quality (or process): how a referral was carried out, including (eg, investigations undertaken before referral, information contained in the referral and extent of patient involvement in the referral decision. Attributes were hierarchical. “Necessity” was viewed as the most fundamental attribute, followed by “destination” and, finally, “quality”. In general, but not always, all three attributes were perceived as necessary for a referral to be defined as appropriate.

Conclusions

For senior clinical and non-clinical managers at the local level in the English NHS,, three hierarchical attributes (necessity, appropriateness of destination and quality of referral process) contributed to the overall concept of appropriateness of referral from primary to secondary surgical care.

Evaluating Labour's market reforms, 2002-10

Starting in 2002, the UK Labour government of 1997-2010 introduced a series of changes to the National Health Service (NHS) in England designed to increase individual NHS patient choice of place of elective hospital care and competition among public and private providers of elective hospital services for NHS-funded patients. In 2006, the Department of Health initiated the Health Reform Evaluation Programme (HREP) to assess the impact of the changes. The changes broadly had the effects that proponents had predicted but the effects were mostly modest. Most of the undesirable impacts feared by critics appeared not to have materialized to any discernible extent, at least by early in 2010. Labour's market appeared to have generated stronger incentives for quality and efficiency than its 1990 s predecessor with no obvious detriment to equity of access. However, this high level conclusion conceals a far more nuanced and complex picture of both the process of implementation and the impact of the changes, as the papers in this supplement drawn from the HREP show.

Assessing health care planning--a framework-led comparison of Germany and New Zealand

OBJECTIVES

With markets and competition dominating much of the debate on health care reform, health care planning has received little scholarly attention in recent years. Yet in many high-income countries, governments have continued to plan some elements of their health care systems. We use a new framework for analysing health care planning organised around the dimensions of 'vision', 'governance' and 'intelligence' to assess the approach in two deliberately contrasting countries, Germany and New Zealand.

METHODS

A review of the literature on health care planning in general and specifically in Germany and New Zealand, supported by key participant interviews.

RESULTS

Planning in both countries largely reflects the different institutional arrangements of their wider health systems. Planning in Germany is fragmented, in part due to federalism and corporatism, with separate approaches in different health care sectors and regions. In contrast, New Zealand's NHS-style health system favours a more hierarchical, integrated approach, with clear lines of accountability, and central government capacity to define objectives and monitor developments. Both countries find it difficult to use planning to align demand for and supply of health care though New Zealand makes some use of population needs assessments to support this process while these are currently absent in Germany.

CONCLUSIONS

While it remains challenging to compare health care systems that are institutionally very different, this new framework for analysing their approaches to planning draws attention to their advantages and disadvantages. It also generates an agenda for future research to improve our understanding of the role and effectiveness of different forms of planning versus, and in combination with, other policy tools to relating health care supply and demand.

Health services research in Europe and its use for informing policy

OBJECTIVE

As an applied field of research, health services research (HSR) is undertaken primarily to inform health care policy. However, the relationship between policy and HSR is not straightforward. Many policies are developed without taking account of available evidence. Our objective was to provide an overview of the infrastructure and capacity of HSR in Europe, and to explore the links between HSR and policy in European countries.

METHODS

A survey of health policy experts in 34 European countries undertaken between September 2009 and June 2010, covering: funding and prioritizing HSR; production and producers of HSR; health policy users of HSR; activities to promote the use of HSR; and barriers and facilitators to using HSR in policy-making. Experts were selected based on their professional expertise and reputation in HSR and/or health policy.

RESULTS

Information was returned from 30 of the 34 countries. Infrastructure and capacity for HSR varied widely across countries as did the mechanisms likely to promote the use of research for national policy-making. In many cases, HSR was not separately identified from other health-related research. Government funding for HSR varied between zero and over €170 million per year (in 2004). Few countries had mechanisms to match HSR priorities with those of policy-makers. Although most countries provided some relevant training, multidisciplinary training was available in few countries. Few mechanisms supporting the use of research in policy were reported. One exception was coverage decisions for new drugs and technologies where formal accountability for the use of health technology assessments appeared to be most developed. Overall, respondents struggled to locate information on many aspects of HSR, particularly its use in decision-making.

CONCLUSION

HSR is unevenly developed across Europe. There is considerable scope to build the infrastructure and to take steps to improve the use of HSR in policy-making. There is also a need for research, as opposed to expert opinion, on how HSR is undertaken and used.

Pharmacy practice research in the United Kingdom: which issues should shape future developments?

Pharmacy practice research (PPR) is a developing area of research in the United Kingdom. To date, most studies have been carried out in secondary care, although efforts have been made recently to encourage studies in primary care. The quality of PPR is variable and there have been few well-performed studies. It is likely that this has resulted from the relative novelty of PPR and the inexperience of the researchers. Studies have usually been performed on services at a single site and have been carried out in isolation from fellow researchers or university departments with appropriate expertise. Initiatives have been taken within pharmacy to improve the quality of PPR but the resources that are available, and hence the potential for improvement, are limited. In contrast, health services research (HSR) is a relatively well-established area of research in which related topics in the broader areas of health care are considered. Comparable methods could be employed to examine PPR and HSR topics. It is suggested that substantial improvements may be made in the quality of PPR by fostering long-term collaborative ventures between PPR and HSR.

Who needs what from a national health research system: lessons from reforms to the English Department of Health's R&D system

Health research systems consist of diverse groups who have some role in health research, but the boundaries around such a system are not clear-cut. To explore what various stakeholders need we reviewed the literature including that on the history of English health R&D reforms, and we also applied some relevant conceptual frameworks.We first describe the needs and capabilities of the main groups of stakeholders in health research systems, and explain key features of policymaking systems within which these stakeholders operate in the UK. The five groups are policymakers (and health care managers), health professionals, patients and the general public, industry, and researchers. As individuals and as organisations they have a range of needs from the health research system, but should also develop specific capabilities in order to contribute effectively to the system and benefit from it.Second, we discuss key phases of reform in the development of the English health research system over four decades - especially that of the English Department of Health's R&D system - and identify how far legitimate demands of key stakeholder interests were addressed.Third, in drawing lessons we highlight points emerging from contemporary reports, but also attempt to identify issues through application of relevant conceptual frameworks. The main lessons are: the importance of comprehensively addressing the diverse needs of various interacting institutions and stakeholders; the desirability of developing facilitating mechanisms at interfaces between the health research system and its various stakeholders; and the importance of additional money in being able to expand the scope of the health research system whilst maintaining support for basic science.We conclude that the latest health R&D strategy in England builds on recent progress and tackles acknowledged weaknesses. The strategy goes a considerable way to identifying and more effectively meeting the needs of key groups such as medical academics, patients and industry, and has been remarkably successful in increasing the funding for health research. There are still areas that might benefit from further recognition and resourcing, but the lessons identified, and progress made by the reforms are relevant for the design and coordination of national health research systems beyond England.