Key steps for a mental health and addictions performance measurement framework for Canada

Prioritizing measures to assess performance of drug treatment services: a Delphi process with funders, treatment providers and service-users

BACKGROUND AND AIM

While many studies have examined outcome measurement as part of clinical trials and routine outcome collection at the person-level in alcohol and other drug (AOD) treatment services, there has been limited attention to measures required to assess performance at the service-level. In Australia, non-government services are primarily funded by government using public funds; however, there is no standardized approach to performance measurement. This study sought to establish a finite list of performance measures that represented consensus between funders, treatment providers and service-users.

METHOD

A three-round Delphi process was undertaken with (i) funders of treatment (n = 10), (ii) treatment providers (n = 10) and (iii) treatment service-users (n = 10). Participants were asked to rate a range of measures on a 10-point Likert scale on how important they were to be included in contracts with funders. Measures with a median score > 7 and agreement among participants above 70% were the criteria for inclusion in the final set of measures. Qualitative data in the form of text responses provided by participants for their ratings in rounds 1 and 2 were also analysed.

RESULTS

Participants rated 93 measures in round 1, which reduced to 78 measures in round 2 and 32 measures in round 3. Fifteen service-level measures and two system-level measures met criteria for inclusion in the final set of performance measures. The final set of measures cover a range of measurement types: outcomes (n = 5), access (n = 3), structural (n = 3), experience (n = 2), input (n = 2), process (n = 1) and output (n = 1).

CONCLUSION

In Australia, performance measures for alcohol and other drug treatment services that represent a consensus among service-users, providers and funders focus upon demonstrating accountability for public funds, improving services and communicating key measures of success to future service-users and the broader community.

Lessons from mandated implementation of a performance management system

PURPOSE

Lean-inspired approaches and performance management systems are being implemented in public healthcare organisations internationally. However, the literature is inconclusive regarding the benefits of these management tools and there is a lack of knowledge regarding processes for large-scale implementation of these tools. This article aims to describe the implementation process and to better understand how this process influences the mandated performance management system.

DESIGN/METHODOLOGY/APPROACH

This research is based on a comparative case study of three healthcare organisations in Canada. Data consist documents, non-participant observation and semi-structured interviews with key actors (n = 30). Analysis is based on a sociotechnical approach to management tools that considers organisational context, and the tool's technical substrate, theory of action and managerial philosophy.

FINDINGS

Results show that despite a standardised national mandate, the tool as implemented varied between organisations in terms of technical substrate and managerial philosophy. These variations are explained by the flexibility of the technical substrate, the lack of clarity of the managerial philosophy, and some contextual elements. Successful implementation may rest upon high hybridization of the tool on these different dimensions. A precise and prescribed technical substrate is not sufficient to guarantee implementation of a managerial philosophy.

PRACTICAL IMPLICATIONS

Mandated implementation of management tools may be more successful if it is explicit on the managerial philosophy, the technical substrate and the link between the two, and if it provides some leeway to adapt both to the organisational context.

ORIGINALITY/VALUE

This is one of the few studies to describe and analyse the process involved in mandated large-scale implementation of performance management systems in public healthcare organisations.

Contracting treatment services in Australia: Do measures adhere to best practice?

BACKGROUND

Contracting non-government services to provide alcohol and other drug (AOD) treatment requires the specification of performance measures to ensure accountability for public funds. There is currently no standardised approach by funders to the measurement of performance of AOD treatment in Australia. Funding to non-government organisations (NGOs) is complex, with NGOs managing funding agreements from a range of sources that contain a large number of differing performance measures. This study aimed to assess performance measures used in contracts for NGO AOD treatment providers and how they align with best practice.

METHODS

Performance measures contained in funding agreements for treatment providers were collected from a diverse sample of both funders (n=8) and treatment providers (n=20) resulting in more than 1,100 measures. The list of measures was synthesized to a finite, non-duplicative list (n=537). Measures were assessed by three raters against 11 criteria (for example, measurable and timely) documented as best practice elements of performance measures (Council of Australian Governments, 2011a). Measures were also coded into different measurement types (such as outcome and process).

RESULTS

None of the 537 unique performance measures used in funding agreements for treatment providers fully met the criteria for best practice in performance measurement. Whilst the literature and government policy is being directed towards outcomes-based funding and reporting, only 7.6% of measures were classified as outcome measures. The majority of measures were classified at output (41.3%) and process (23.6%) measures.

CONCLUSION

Current measures in contracts applied by funders to treatment services do not adhere to best practice. The development and implementation of new performance measures is required to inform AOD policy and accountability of public funds. Further, identifying more robust performance measures has the potential to lead to reduced reporting burden on service providers and better monitoring of service quality and outcomes.