Reliability and validity of the Intercollegiate Stroke Audit Package

Allied health clinicians using translational research in action to develop a reliable stroke audit tool

OBJECTIVE

To design and establish reliability of a local stroke audit tool by engaging allied health clinicians within a privately funded hospital.

METHODS

Design: Two-stage study involving a modified Delphi process to inform stroke audit tool development and inter-tester reliability.

PARTICIPANTS

Allied health clinicians.

INTERVENTIONS

A modified Delphi process to select stroke guideline recommendations for inclusion in the audit tool. Reliability study: 1 allied health representative from each discipline audited 10 clinical records with sequential admissions to acute and rehabilitation services.

MAIN OUTCOME MEASURES

Recommendations were admitted to the audit tool when 70% agreement was reached, with 50% set as the reserve agreement. Inter-tester reliability was determined using intra-class correlation coefficients (ICCs) across 10 clinical records.

RESULTS

Twenty-two participants (92% female, 50% physiotherapists, 17% occupational therapists) completed the modified Delphi process. Across 6 voting rounds, 8 recommendations reached 70% agreement and 2 reached 50% agreement. Two recommendations (nutrition/hydration; goal setting) were added to ensure representation for all disciplines. Substantial consistency across raters was established for the audit tool applied in acute stroke (ICC .71; range .48 to .90) and rehabilitation (ICC.78; range .60 to .93) services.

CONCLUSIONS

Allied health clinicians within a privately funded hospital generally agreed in an audit process to develop a reliable stroke audit tool. Allied health clinicians agreed on stroke guideline recommendations to inform a stroke audit tool. The stroke audit tool demonstrated substantial consistency supporting future use for service development. This process, which engages local clinicians, could be adopted by other facilities to design reliable audit tools to identify local service gaps to inform changes to clinical practice.

National Audit of Seizure management in Hospitals (NASH): results of the national audit of adult epilepsy in the UK

OBJECTIVES

About 100,000 people present to hospitals each year in England with an epileptic seizure. How they are managed is unknown; thus, the National Audit of Seizure management in Hospitals (NASH) set out to assess prior care, management of the acute event and follow-up of these patients. This paper describes the data from the second audit conducted in 2013.

SETTING

154 emergency departments (EDs) across the UK.

PARTICIPANTS

Data from 4544 attendances (median age of 45 years, 57% men) showed that 61% had a prior diagnosis of epilepsy, 12% other neurological problems and 22% were first seizure cases. Each ED identified 30 consecutive adult cases presenting due to a seizure.

PRIMARY AND SECONDARY OUTCOME MEASURES

Details were recorded of the patient's prior care, management at hospital and onward referral to neurological specialists onto an online database. Descriptive results are reported at national level.

RESULTS

Of those with epilepsy, 498 (18%) were on no antiepileptic drug therapy and 1330 (48%) were on monotherapy. Assessments were often incomplete and witness histories were sought in only 759 (75%) of first seizure patients, 58% were seen by a senior doctor and 57% were admitted. For first seizure patients, advice on further seizure management was given to 264 (27%) and only 55% were referred to a neurologist or epilepsy specialist. For each variable, there was wide variability among sites that was not explicable. For the sites who partook in both audits, there was a trend towards better care in 2013, but this was small and dwarfed by the intersite variability.

CONCLUSIONS

These results have parallels with the Sentinel Audit of Stroke performed a decade earlier. There is wide intersite variability in care covering the entire care pathway, and a need for better organised and accessible care for these patients.

Telephone audit for monitoring stroke unit facilities: a post hoc analysis from PROSIT study

BACKGROUND

Although several valid approaches exist to measure the number and the quality of acute stroke units, only few studies tested their reliability. This study is aimed at establishing whether the telephone administration of the PROject of Stroke unIt ITaly (PROSIT) audit questionnaire is reliable compared with direct face-to-face interview.

METHODS

Forty-three medical leaders in charge of in-hospital stroke services were interviewed twice using the same PROSIT questionnaire with 2 different modalities. First, the interviewers approached the medical leaders by telephone. Thereafter, they went to the hospital site and performed a direct face-to-face interview. Six independent couples of trained researchers conducted the audit interviews. The degree of intermodality agreement was measured with kappa statistic.

RESULTS

We found a perfect agreement for stroke units identification between the 2 different audit modalities (K = 1.00; standard error [SE], 1.525). The agreement was also very good for stroke dedicated beds (K = 1.00; SE, 1.525) and dedicated personnel (K = 1.00; SE, 1.525), which are the 2 components of stroke unit definition. The agreement was lower for declared in use process of care and availability of diagnostic investigations.

CONCLUSIONS

The telephone audit can be used for monitoring stroke unit structures. It is more rapid, less expensive, and can repeatedly be used at appropriate intervals. However, a reliable description of the process of care and diagnostic investigations indicators should be obtained by either local site audit visit or prospective stroke register based on individual patient data.

Measuring how well the NHS looks after its own staff: methodology of the first national clinical audits of occupational health services in the NHS

RATIONALE, AIMS AND OBJECTIVES

Little is known about the quality of occupational health care provided to National Health Service (NHS) staff. We designed the first national clinical audits of occupational health care in England. We chose to audit depression and back pain as health care workers have high levels of both conditions compared with other employment sectors. The aim of the audits was to drive up quality of care for staff with these conditions. The object of this paper is to describe how we developed an audit methodology and overcame challenges presented by the organization and delivery of occupational health care for NHS staff.

METHODS

We designed two retrospective case note audits which ran simultaneously. Sites submitted up to 40 cases for each audit. We used duplicate case entry to test inter-rater reliability and performed selection bias checks. Participants received their site's audit results, benchmarked against the national average, within 4 months of the end of the data entry period. We used electronic voting at a results dissemination conference to inform implementation activities.

RESULTS

Occupational Health departments providing services to 278 (83%) trusts in England participated in one or both audits. Median kappa scores were above 0.7 for both pilot and full audits, indicating 'good' levels of inter-rater reliability. In total, 79% of participants at a dissemination conference said that they had changed their clinical practice either during data collection (52%) or following receipt of their audit results (27%).

CONCLUSIONS

Clinical audit can be conducted successfully in the occupational health setting. We obtained meaningful data that have stimulated local and national quality improvement activities. Our methodology would be transferable to occupational health settings outside the NHS and in other countries.

Measuring the Quality of Dysphagia Management Practices following Stroke: A Systematic Review

Adherence to recommended clinical practices improves stroke outcomes. As a result, stroke clinicians are increasingly expected to evaluate the quality of the care they provide so that areas for improvement can be targeted. Finding the best method to evaluate the quality of dysphagia management can be challenging.

Aim

To systematically review process indicators used to assess the quality of care provided to patients with dysphagia following acute stroke and examine the level of evidence underpinning these indicators.

Methods

Databases were systematically searched to identify publications (January 2006–April 2009) that describe process indicators relating to the clinical management of acute stroke-related dysphagia. Relevant process indicators were extracted from the reviewed publications for detailed post hoc analysis including supporting evidence and alignment to the current Australian and English stroke guidelines.

Results

Title and abstract review found 150 potential studies. Full-text review resulted in 25 publications that met the study's inclusion criteria. Thirteen process indicators were identified in the literature that related to the initial assessment, clinical management, rehabilitation and discharge planning for patients with acute stroke-related dysphagia. These processes were supported by levels of evidence ranging from high ‘level 1’ (8%) down to ‘expert opinion’ evidence (46%). Two process indicators did not align to recommendations in the clinical guidelines. This systematic review underpins informed selection of process indicators for evaluating the quality of dysphagia management following stroke. The selection of quality indicators is complicated by equivocal supporting evidence; however, indicators should reflect expected local practices, align with national stroke guidelines and be feasible for clinical auditing.

Systematic Review of Process Indicators: Including Early Rehabilitation Interventions Used to Measure Quality of Acute Stroke Care

Background

Stroke is a leading cause of disease burden. The quality of care provided in hospitals can affect outcome. Therefore, examining adherence to clinically important processes of care can help improve care delivery and patient outcomes. However, knowing which process indicators to measure is essential.

Aim

Systematically review process indicators used to evaluate acute stroke services, including early rehabilitation interventions, and assess whether published indicators conform to clinical guidelines.

Methods

Publications (1985-2006) were identified by systematically searching databases (e.g. Medline and Cochrane Library), and the internet using free text terms: ‘stroke unit’, ‘process’, ‘quality’, ‘mobilisation’, ‘acute’, and ‘early rehabilitation’. Publications describing process indicators relating to the first 2 weeks of in-patient stroke care were included. Process indicators were categorised according to six clinical process domains covering the acute stroke admission. Commonly cited indicators (≥6 publications) were then mapped to the 2003 Australian clinical guidelines.

Results

Sixty potential studies were found from title and abstract. Following full text review, 32 publications were retained. Of the 161 process indicators identified, 43 were commonly cited. Seventy-nine per cent of commonly cited indicators were found in the guidelines. The level of evidence underpinning each indicator ranged from low ‘expert opinion’ (59%), to high, ‘level 1’ (12%) evidence. Indicators related to rehabilitation were rare.

Conclusion

Many acute stroke process indicators have been published. However, a quarter did not align with current clinical guidelines. Developing an ‘ideal set’ of process indicators to reflect the evidence base seems sensible and should include rehabilitation interventions.

Quality of stroke care at an Irish Regional General Hospital and Stroke Rehabilitation Unit

BACKGROUND

Robust international data support the effectiveness of stroke unit (SU) care. Despite this, most stroke care in Ireland are provided outside of this setting. Limited data currently exist on the quality of care provided.

AIM

The aim of this study is to examine the quality of care for patients with stroke in two care settings-Regional General Hospital (RGH) and Stroke Rehabilitation Unit (SRU).

METHODS

A retrospective analysis of the stroke records of consecutive patients admitted to the SRU between May-November 2002 and April-November 2004 was performed applying the UK National Sentinel Audit of Stroke (NSAS) tool.

RESULTS

The results of the study reveal that while SRU processes of care was 74% compliant with standards; compliance with stroke service organisational standards was only 15 and 43% in the RGH and SRU, respectively.

CONCLUSION

The quality of stroke care in our area is deficient. Comprehensive reorganisation of stroke services is imperative.

Quality performance measurement using the text of electronic medical records

BACKGROUND

Annual foot examinations (FE) constitute a critical component of care for diabetes. Documented evidence of FE is central to quality-of-care reporting; however, manual abstraction of electronic medical records (EMR) is slow, expensive, and subject to error. The objective of this study was to test the hypothesis that text mining of the EMR results in ascertaining FE evidence with accuracy comparable to manual abstraction.

METHODS

The text of inpatient and outpatient clinical reports was searched with natural-language (NL) queries for evidence of neurological, vascular, and structural components of FE. A manual medical records audit was used for validation. The reference standard consisted of 3 independent sets used for development (n=200 ), validation (n=118), and reliability (n=80).

RESULTS

The reliability of manual auditing was 91% (95% confidence interval [CI]= 85-97) and was determined by comparing the results of an additional audit to the original audit using the records in the reliability set. The accuracy of the NL query requiring 1 of 3 FE components was 89% (95% CI=83-95). The accuracy of the query requiring any 2 of 3 components was 88% (95% CI=82-94). The accuracy of the query requiring all 3 components was 75% (95% CI= 68- 83).

CONCLUSIONS

The free text of the EMR is a viable source of information necessary for quality of health care reporting on the evidence of FE for patients with diabetes. The low-cost methodology is scalable to monitoring large numbers of patients and can be used to streamline quality-of-care reporting.

Automatic classification of foot examination findings using clinical notes and machine learning

We examine the feasibility of a machine learning approach to identification of foot examination (FE) findings from the unstructured text of clinical reports. A Support Vector Machine (SVM) based system was constructed to process the text of physical examination sections of in- and out-patient clinical notes to identify if the findings of structural, neurological, and vascular components of a FE revealed normal or abnormal findings or were not assessed. The system was tested on 145 randomly selected patients for each FE component using 10-fold cross validation. The accuracy was 80%, 87% and 88% for structural, neurological, and vascular component classifiers, respectively. Our results indicate that using machine learning to identify FE findings from clinical reports is a viable alternative to manual review and warrants further investigation. This application may improve quality and safety by providing inexpensive and scalable methodology for quality and risk factor assessments at the point of care.

Improving clinical practice in stroke through audit: results of three rounds of National Stroke Audit

BACKGROUND

The results of three rounds of National Stroke Audit in England, Wales and Northern Ireland are compared.

METHODS

Audit of the organization of stroke services and retrospective case-note audit of up to 40 consecutive cases admitted per hospital over a 3-month period was conducted in each of 1998, 1999 and 2001/02. The changes in the organizational, case-mix and process results of the hospitals that had participated in all three rounds were analysed.

RESULTS

60% of all eligible trusts from England, Wales and Northern Ireland took part in all three audits in 1998, 1999 and 2001/02. Total numbers of cases were 4996, 4841 and 5152, respectively. Case-mix variables were similar over the three rounds. Mortality at 7 and 30 days fell by 3% and 5%, respectively. The proportion of hospitals with a stroke unit rose from 48% to 77%. The proportion of patients spending most of their stay in a stroke unit rose from 17% in 1998 to 26% in 1999 and 29% in 2001/02. Improvements achieved in process standards of care between 1998 and 1999 (median change was a gain of 9%) failed to improve further by 2001/02 (median change was 0%). In all three rounds process standards of care tended to be better in stroke units.

CONCLUSIONS

Three rounds of national audit of stroke care have shown standards of care on stroke units were notably higher than on general wards. Slowing in the rise of the proportion managed on stroke units mirrors the slow down in improvement to overall national standards of care. To further improve outcomes and national standards of stroke care a much higher proportion of patients needs to be managed in stroke units.

Lessons from the management of acute myocardial infarction

The National Service Framework for coronary heart disease set a number of challenging targets for the care of patients following an acute myocardial infarction. The Myocardial Infarction National Audit Project (MINAP) was devised to monitor progress and has been notably successful in winning professional support and participation and helping trusts to meet these targets. The new challenge is in translating this success to other areas of medicine. Heart failure is one such area, although it poses a number of difficulties relating primarily to disease definition and the definition of a successful outcome. MINAP was overseen by a multidisciplinary group of stakeholders, including patient organisations, and was project managed by a professionally led team at the Royal College of Physicians. Successful projects must retain confidence of all stakeholders and in part this depends on ensuring that timelines are met. Central monitoring of returns and anticipation of problems has been an important component of data completeness and quality. Next day updates to those collecting the data and more detailed quarterly reports for clinicians and chief executives within days of quarter end have been vital. Change depends on clinicians and managers working together. But most importantly, the attention to detail outlined above means the data have been believed and the resulting change for patients has been remarkable.

Psychological services for people with stroke: compliance with the U.K. National Clinical Guidelines

BACKGROUND

The UK National Clinical Guidelines for Stroke (2000) include recommendations on psychological services. The third National Sentinel Audit of Stroke was completed in 2001-2002.

OBJECTIVES

To examine the extent to which UK stroke services complied with the national guidelines.

DESIGN

Use of three retrospective case note audits of hospital admissions, covering the period from admission to six months after discharge, and audits of how stroke services were organized.

SETTING

Hospitals within England, Wales, Northern Ireland, the Channel Islands and the Isle of Man.

SUBJECTS

Stroke patients admitted consecutively within a three-month time frame.

MAIN MEASURES

Compliance with the guidelines on mood disorders and cognitive impairments, and changes between audits.

RESULTS

The 2001-2002 audit provided data on 60% of possible participants, from 145 hospitals and 5152 patients. Compliance with the guideline to screen for mood disturbance was poor; the median patient compliance rate of hospitals was 50%. More hospitals (88%) had a locally agreed cognitive assessment protocol in 2001-2002 than in 1998 (68%) and in 1999 (82%). However, actual rates of screening for cognitive difficulties were lower than implied by the existence of a local protocol. There were no strong case-mix associates of mood and cognitive screening. Access to clinical psychologists was poor. Mood and cognitive assessment rates were not much better for stroke units with access to clinical psychologists than for units without access (mood: p = 0.6, cognition: p = 0.09).

CONCLUSIONS

Although compliance with some of the guidelines has improved, many areas in current psychological services for stroke urgently need attention.

Stroke units: research and reality. Results from the National Sentinel Audit of Stroke

OBJECTIVES

To use data from the 2001-2 National Stroke Audit to describe the organisation of stroke units in England, Wales and Northern Ireland, and to see if key characteristics deemed effective from the research literature were present.

DESIGN

Data were collected as part of the National Sentinel Audit of Stroke in 2001, both on the organisation and structure of inpatient stroke care and the process of care to hospitals managing stroke patients.

SETTING

240 hospitals from England, Wales and Northern Ireland took part in the 2001-2 National Stroke Audit, a response rate of over 95%. These sites audited a total of 8200 patients. AUDIT TOOL: Royal College of Physicians Intercollegiate Working Party Stroke Audit Tool.

RESULTS

73% of hospitals participating in the audit had a stroke unit but only 36% of stroke admissions spent any time on one. Only 46% of all units describing themselves as stroke units had all five organisational characteristics that previous research literature had identified as being key features, while 26% had four and 28% had three or less. Better organisation was associated with better process of care for patients, with patients managed on stroke units receiving better care than those managed in other settings.

CONCLUSION

The National Service Framework for Older People set a target for all hospitals treating stroke patients to have a stroke unit by April 2004. This study suggests that in many hospitals this is being achieved without adequate resource and expertise.

Relationship between process and outcome in stroke care

BACKGROUND AND PURPOSE

Better processes in stroke care are assumed to lead to better stroke outcomes. We sought to test whether current measures of stroke process are related to measures of stroke outcome.

METHODS

This was a prospective study of consecutive patients with acute stroke admitted to each of the 3 general hospitals in 1 region who were followed up for 12 months after hospital discharge. Process was measured by use of the Royal College of Physicians Stroke Audit Package, and outcomes were measured with a range of disability, health status, handicap, and independence measures, as well as mortality.

RESULTS

One hundred eighty-one patients were recruited. There was evidence for a relationship between some process variables and outcomes at hospital discharge, but the relationships were generally weak. None of the process variables remained in regression models of functional outcomes at 12 months. The hospital with the best process scores had the worst case mix-adjusted outcomes.

CONCLUSIONS

The link between stroke process and outcome, through the use of currently available measures of process such as the Royal College of Physicians Stroke Audit Package, is not straightforward. Ongoing work may clarify some of these issues and provide guidance to stroke clinicians on how best to improve existing services.

Obtaining evidence for use by healthcare payers on the success of chronic obstructive pulmonary disease management

Healthcare payers make decisions on funding for treatments for diseases, such as chronic obstructive pulmonary disease (COPD), on a population level, so require evidence of treatment success in appropriate populations, using usual routine care as the comparison for alternative management approaches. Such health outcomes evidence can be obtained from a number of sources. The 'gold standard' method for obtaining evidence of treatment success is usually taken as the randomized controlled prospective clinical trial. Yet the value of such studies in providing evidence for decision-makers can be questioned due to the restricted entry criteria limiting the ability to generalize to real life populations, narrow focus on individual parameters, use of placebo for comparison rather than usual therapy and unrealistic intense monitoring of patients. Evidence obtained from retrospective and observational studies can supplement that from randomized clinical trials, providing that care is taken to guard against bias and confounders. However, very large numbers of patients must be investigated if small differences between drugs and treatment approaches are to be detected. Administrative databases from healthcare systems provide an opportunity to obtain observational data on large numbers of patients. Such databases have shown that high healthcare costs in patients with COPD are associated with co-morbid conditions and current smoking status. Analysis of an administrative database has also shown that elderly patients with COPD who received inhaled corticosteroids within 90 days of discharge from hospital had 24% fewer repeat hospitalizations for COPD and were 29% less likely to die during the 1-year follow-up period. In conclusion, there are a number of sources of meaningful evidence of the health outcomes arising from different therapeutic approaches that should be of value to healthcare payers making decisions on resource allocation.

Stroke care: how do we measure quality?

The purpose of this paper is to review the methods used to measure quality of stroke care. Relevant articles were searched for on Medline using the following key words: stroke, quality, outcome of care, process of care, structure of care. Articles that examined how to measure the quality of stroke care and that examined difficulties in the measurement of care outcomes, processes, and structures were selected. Selected articles were reviewed to summarise methods used to measure quality of stroke care and the primary outcome measures of the studies were extracted. Conclusions were drawn about the best ways to measure the quality of stroke care. Practical problems in using outcome measures to monitor quality of care include the consequences of case mix and difficulties in risk adjustment. Clinicians may use process measures to understand differences in outcome. Once a process of care has been linked to an outcome measure, this care process should be measured. The national sentinel audit for stroke is an audit tool used to examine the quality of the processes of stroke care.

National stroke audit: a tool for change?

OBJECTIVES

To describe the standards of care for stroke patients in England, Wales and Northern Ireland and to determine the power of national audit, coupled with an active dissemination strategy to effect change.

DESIGN

A national audit of organisational structure and retrospective case note audit, repeated within 18 months. Separate postal questionnaires were used to identify the types of change made between the first and second round and to compare the representativeness of the samples.

SETTING

157 trusts (64% of eligible trusts in England, Wales, and Northern Ireland) participated in both rounds.

PARTICIPANTS

5589 consecutive patients admitted with stroke between 1 January 1998 and 31 March 1998 (up to 40 per trust) and 5375 patients admitted between 1 August 1999 and 31 October 1999 (up to 40 per trust). Audit tool-Royal College of Physicians Intercollegiate Working Party stroke audit.

RESULTS

The proportion of patients managed on stroke units rose between the two audits from 19% to 26% with the proportion managed on general wards falling from 60% to 55% and those managed on general rehabilitation wards falling from 14% to 11%. Standards of assessment, rehabilitation, and discharge planning improved equally on stroke units and general wards, but in many aspects remained poor (41% formal cognitive assessment, 46% weighed once during admission, 67% physiotherapy assessment within 72 hours, 24% plan documented for mood disturbance, 36% carers' needs assessed separately).

CONCLUSIONS

Nationally conducted audit linked to a comprehensive dissemination programme was effective in stimulating improvements in the quality of care for patients with stroke. More patients are being managed on stroke units and multidisciplinary care is becoming more widespread. There remain, however, many areas where standards of care are low, indicating a need for investment of skills and resources to achieve acceptable levels.

Regional variations in stroke care in England, Wales and Northern Ireland: results from the National Sentinel Audit of Stroke. Royal College of Physicians Intercollegiate Stroke Working Party

STUDY OBJECTIVE

To identify the variations between regions in England, Wales and Northern Ireland in the case-mix, organization and process of care for stroke.

DESIGN

Retrospective audit of case notes and service organization.

SETTING

Two hundred and ten Trust sites from 197 Trusts in 10 Health Regions in England, Wales and Northern Ireland.

PATIENTS

The 6894 consecutive stroke patients admitted between 1 January and 31 March 1998 (up to 40 per Trust). Audit tool: The Intercollegiate Stroke Audit.

RESULTS

There are significant differences in stroke care between regions that cannot be explained by known case-mix or clinical variables. The proportion of patients spending more than half their hospital stay in stroke unit care varied between regions from 10% to 27%. Thirty-day mortality in different regions ranged between 21% and 33%. Institutionalization rates for those admitted from home varied between 6% and 19%. Similar variations existed in discharge disability and length of stay.

CONCLUSIONS

There were widespread variations in specialist service provision for stroke in different regions. Regional variation in 30-day mortality and in institutionalization after stroke is not explained by clinical factors and therefore may represent different local health care policies and expectations.