Development and validation of an audit instrument: the Prostate Outcomes Questionnaire

OBJECTIVE

To develop a short, patient-based questionnaire for auditing the outcomes of treatment for benign prostatic hyperplasia (BPH) and to evaluate the acceptability, reliability and validity of the measure.

PATIENTS AND METHODS

Data from a long research questionnaire used in the Oxford/North West Thames Prostatectomy Study were analysed to identify the subset of items which contained the most scientifically sound indicators of outcome. Items were selected on the basis of standard psychometric analyses to develop a short questionnaire, the Prostate Outcomes Questionnaire (POQ), a 27-item instrument covering urinary symptoms, complications after surgery, quality of life and patient satisfaction with outcome. The POQ was field tested for acceptability, reliability and validity in a postal survey of 125 men undergoing transurethral resection of the prostate for BPH from seven hospitals in south-east England.

RESULTS

The POQ was highly acceptable to patients, as indicated by a 95% response rate, the absence of 'floor' and 'ceiling' effects and a low proportion of missing data. It showed excellent internal consistency reliability (Cronbach's alpha of 0.82-0.93 for the five summary scales and a mean item-total correlation of 0.56) and test-retest reliability (test-retest correlations for items were 0.40-1.00, with a mean test-retest correlation of 0.85, and for summary scales of 0.91-0.95). Construct validity was confirmed by: high intercorrelations between the five summary scales and the total score (0.88-0.95), with a pattern of intercorrelations among specific subscales that showed item-convergent and discriminant validity; higher scores for patients who reported an improvement after surgery than in those who were not improved (P < 0.001); the expected gradient of scores for patients reporting different levels of symptom distress (P < 0.001); high correlations with the longer parent questionnaire (0.75-0.88); and low to moderate correlations with the Nottingham Health Profile and Activities of Daily Living scores (0.15-0.67), with the pattern of correlations providing evidence of scale convergent and discriminant validity. Correlations with age and social class were low, suggesting that responses were not biased by sociodemographic factors.

CONCLUSION

The POQ is a practical and scientifically sound patient-based measure of outcome after treatment for BPH which can be used for routine audit. It takes < 5 min to complete, is feasible for routine monitoring of large numbers of patients by postal survey, and is accompanied by a Users' Manual which provides practical help in conducting a local patient survey, and a computer program for scoring data. Most importantly, the POQ has been shown scientifically to perform well, having met standard psychometric criteria for reliability and validity.

The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

OBJECTIVE

To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity.

DESIGN

A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non-randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden.

MAIN RESULTS

The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes).

CONCLUSIONS

This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.

Does comorbidity affect the outcome of surgery? Total hip replacement in the UK and Japan

OBJECTIVES

To assess the impact of comorbidity on the outcome of surgery in the UK and in Japan; to determine the predictive ability of a new measure of comorbidity, the Index of Co-Existent Diseases (ICED); and to see if its predictive power could be improved.

DESIGN

Logistic regression using data from two retrospective cohorts with prospective outcome data collection.

SETTING

Six hospitals (three teaching, three non-teaching) in the UK and 15 (12 teaching, three non-teaching) hospitals in Japan.

STUDY PARTICIPANTS

Patients undergoing total hip replacement (THR) surgery in the UK (n = 268) and in Japan (n = 249).

MAIN OUTCOME MEASURES

Serious complications before hospital discharge and change in three measures of general health status [basic activities of daily living (ADL); instrumental ADL; social activities].

RESULTS

The distribution of levels of comorbidity differed between the UK and Japan: none (26.1 versus 42.2%); mild (30.6 versus 43.0%); moderate (23.5 versus 12.0%); and severe (19.8 versus 2.8%). In the UK, the incidence of serious complications was higher in patients with moderate (27.0%) or severe (26.4%) comorbidity than in those with no (14.3%) or mild (13.4%) comorbidity (P<0.001). In contrast, no significant association was found in Japan. The relationship between comorbidity and change in health status was weak in the UK and non-significant in Japan. Logistic regression confirmed that comorbidity was a significant predictor of serious complications in the UK. The only other significant factor was surgical approach (anterior/antero-lateral; odds ratio 2.16, P<0.05). Attempts to improve the predictive power of the ICED by modifying its structure and by reclassifying complications was successful in achieving a linear (rather than dichotomous) relationship. The predictive power, however, was poor.

CONCLUSIONS

Comorbidity is a significant determinant of serious complications following THR but not of changes in functional or health status. Comparisons of clinical performance using post-operative complications must take levels of comorbidity into account if they are to be meaningful. The ICED is of less validity in the UK and Japan than in its country of origin, the USA. Further work to develop better instruments for the UK and Japan is needed.

Impact of surgery for stress incontinence on morbidity: cohort study

OBJECTIVES

To describe the impact of surgery for stress incontinence on the severity of symptoms, other mental and physical symptoms, and overall health. To describe the incidence of postoperative complications.

DESIGN

Prospective cohort study; questionnaires completed by patients before and 3, 6, and 12 months after surgery. Questionnaires completed by surgeons both before and after surgery.

SETTING

18 hospitals in the North Thames region.

SUBJECTS

442 women treated surgically for stress incontinence between January 1993 and June 1994. 367 women returned the 3 month questionnaire; 364 returned the 6 month questionnaire; and 359 returned the 12 month questionnaire. 49 surgeons provided perioperative information on 285 of the 442 women and postoperative information on 278.

MAIN OUTCOME MEASURES

Stress incontinence symptom severity index, other urinary symptoms, bowel function, mental health, complications, global measures.

RESULTS

Most women (288; 87%) reported an improvement in the severity of their stress incontinence, though only 92 (28%) were cured (continent). These improvements persisted for at least 12 months. The likelihood of improvement was similar regardless of whether urodynamic pressure studies had been conducted before surgery. Following surgery, women were less likely to suffer from urinary frequency, nocturia, postvoid fullness, dysuria, and urgency. While mental health improved for 194 (71%), a quarter of women reported deterioration. Only 37 (10%) were satisfied with postoperative pain control. A third experienced one or more complications while in hospital, most commonly difficulty urinating. This problem affected 1 in 11 women after discharge. A year after surgery two thirds of women reported feeling better (251; 72%), that the outcome met or exceeded their expectations (230; 66%), and that they would recommend the operation to a friend in a similar situation (239; 68%), and that they would recommend the operation to a friend in a similar situation (239; 68%). Surgeons tended to be more optimistic about the effects of surgery; they were satisfied with the outcome in 176 (85%) cases and would again treat 245 (94%) of the women as they had done previously.

CONCLUSIONS

Although surgery reduces the severity of stress incontinence it is not as effective as current textbooks suggest. Women considering surgery should be provided with more accurate information on the likelihood of an improvement in symptoms and the occurrence of complications, including postoperative pain. Urgency and urge incontinence should not be considered contraindications to surgery. The need for urodynamic assessment before surgery should be reappraised.

Efficacy of combination therapy in non-insulin dependent diabetes mellitus

Secondary failure of oral hypoglycaemic agents raises the dilemma of whether to institute therapy with insulin alone, or in combination. We reviewed our experience of combination therapy following secondary failure of oral hypoglycaemic therapy. Seventeen subjects were receiving combination therapy for 6 months or more. Such treatment was associated with a significant fall in HbA1C--from 10.7 +/- 0.38 per cent to 8.3 +/- 0.35 per cent (p < 0.01) after 6 months and remained significantly reduced at 12 months (8.7 +/- 0.34 per cent (p < 0.01)). Mean body weight, systolic and diastolic blood pressure were unchanged during treatment with adjuvant insulin therapy. Insulin therapy is a useful adjunct in the daily management of subjects with NIDDM who experience secondary failure of oral hypoglycaemic agents.

Reliability of a comorbidity measure: the Index of Co-Existent Disease (ICED)

The reliability of an established comorbidity index (the Index of Co-Existent Disease) was tested using retrospective data from the case notes of elderly patients who had undergone total hip replacement. Inter-rater reliability was examined twice, first with two raters (n = 39) and then with three (n = 49). Intra-rater reliability was assessed using one rater (n = 45). Reasons for any lack of reliability were explored. The inter-rater reliability of the ICED was moderate (kappa 0.5-0.6). While the Functional Severity index performed well (kappa 0.6-1.0), the Index of Disease Severity subindex was less reliable (kappa 0.4-0.5). Differences between raters had an impact on the observed association between comorbidity and serious post-operative complications. Intra-rater reliability was excellent (kappa 0.9). Several reasons why inter-rater reliability was only moderate were identified, mostly related to uncertainties in applying the ICED. The reliability of the ICED needs to be improved before it is used more widely with retrospective data. This might be achieved by further clarification of the instructions for its use.

Regional variation in intervention rates: what are the implications for patient selection?

BACKGROUND

Whereas geographical variations in intervention rates are well recognized, little is known about their implications for patient selection. This study looks at how the relative probability of being treated in different regions within England vary with a person's need for treatment, and whether higher intervention rates are associated with a greater probability of treatment at all levels of need or confined to only certain levels.

METHODS

The method was modelling of retrospective data from population surveys, patient cohort studies and population intervention rates. Two southern regions (SW Thames and Wessex) and two northern regions (Northern and Mersey) were compared. Subjects were men aged 55 years and above in the population with urinary symptoms suggestive of benign prostatic hyperplasia and men undergoing surgical treatment. The ratio of probability of surgery in the southern regions to that in the northern regions by level of symptom severity was determined.

RESULTS

The rate of surgery in the southern regions was 26.5 per cent higher than in the north. A higher proportion of patients in the north had severe symptoms before surgery (58 per cent vs 52 per cent; p = 0.002). The probabilities of being operated on in a given year varied by symptom severity in both the north and the south. The probability was higher in the south at all levels of symptom severity: none/mild (ratio = 1.44; p > 0.01), low-moderate (ratio = 1.35; p = 0.003), high-moderate (ratio = 1.53; p < 0.0001), and severe (ratio = 1.15; p > 0.01). On testing the sensitivity of the key assumptions by assuming a more severe distribution of symptoms in the south, the differences at none/mild and low-moderate symptom levels were enhanced but differences at high-moderate and severe symptom levels were reversed.

CONCLUSIONS

As few men with mild symptoms qualify for surgery and most men with severe symptoms are operated on, any difference in patient selection between high and low rate regions is inevitably confined to the intermediate group of men with moderate symptoms. Surgeons appear to be rationing their resources in a sensible way, though perhaps not as stringently as could be achieved.

A national strategy for research and development: lessons from England

Increasing concern about the state of basic and clinical research in England during the 1980s led to an influential parliamentary review. Surprisingly, the review recommended the strengthening of public health and health services research through the establishment of a research and development (R&D) program for the National Health Service (NHS). The program that started in 1991 was unique in that it was fully integrated into the management structure of the NHS. No country had ever attempted such an ambitious approach. While a review of the first five years of the program reveals many achievements, it also raises several concerns: Debate about the philosophy and aims of the program continues; the need to maintain political support requires constant attention; policy changes in other areas need to be accommodated; central control of a national, coordinated R&D program has to be guarded; methods of priority setting need to be enhanced; insufficient human resources to run the program have to be contended with; and the program needs to be rigorously evaluated. Other countries with a unified health system could learn much from the English experience. Countries with pluralist systems might benefit from specific parts of the experience.

Impact of the 1991 NHS reforms on the availability and use of coronary revascularisation in the UK (1987-1995)

OBJECTIVE

To describe changes in the availability, utilisation, and waiting times for coronary artery bypass grafting (CABG) and percutaneous transluminal coronary angioplasty (PTCA) between 1987/88 and 1994/95 and to review commissioning of these services.

DESIGN

A series of cross sectional surveys and interviews with purchasers and providers.

SETTING

Four health regions in the United Kingdom.

PATIENTS

All residents aged 25 years or more who underwent coronary revascularisation.

RESULTS

There has been little change in the availability of consultants in cardiology in specialist centres, while the number of non-consultant cardiologists has risen significantly. The availability of consultant surgeons more than doubled in some regions, while non-consultant surgical staff increased by 40-90%. The NHS rate of use of both CABG and PTCA has increased steadily since 1987/88. In 1994/95, only two districts had CABG rates of less than 300 per million population. The additional contribution of privately funded cases varied between 14-23% for CABG and 7-30% for PTCA. Regional rates varied 1.3-fold for CABG and threefold for PTCA in 1994/95, while district rates of CABG varied 3.6-fold and PTCA 18-fold. Revascularisation rates were higher in districts with least need in 1991/92 and this persisted over the following three years. The overall waiting time for CABG (214 days) was largely unchanged from 1992/93 (234 days). The overall waiting time for PTCA (138 days) was 25% shorter than in 1992/93 (185 days). Prioritisation of patients waiting over a year had not yet adversely affected the waiting time of more urgent patients. Commissioning has faced a complex web of interconnected problems which, in general, caused more problems for purchasers than providers initially but which appear to be of increasing concern to providers.

CONCLUSIONS

The 1991 NHS reforms had had no observable impact on the availability and use of coronary revascularisation by 1995. Continued monitoring is necessary to detect any delayed effect.

Does a shorter length of hospital stay affect the outcome and costs of hysterectomy in southern England?

STUDY OBJECTIVE

To see whether a shorter postoperative length of stay (LOS) for a major procedure, abdominal hysterectomy for benign conditions, was associated with health outcome, the use of formal and lay care after discharge, cost, and satisfaction.

DESIGN

Prospective cohort study.

SETTING

Three hospitals in London and three in Hertfordshire and Bedfordshire.

PATIENTS

A total of 363 women undergoing total abdominal hysterectomy with or without oophorectomy: 112 with a short postoperative LOS (five days or less) and 251 with a standard LOS (six days or more).

MAIN OUTCOME MEASURES

Wound infection within 10 days and six weeks; change in general health status (Nottingham health profile) after six weeks; general health and change in social activity (lifestyle index) three months after surgery. Mean cost difference for hospitals, use of formal and lay care after discharge, and patient satisfaction.

RESULTS

Short LOS was associated with benefits: a lower risk of wound infection in the first 10 days (odds ratio 0.44; p = 0.03) and no deterioration in physical mobility (measured using the NHP) after six weeks- and with adverse outcomes: constipation six weeks later (OR 0.48; p < 0.001) and moderate or severe urinary symptoms six weeks (OR 0.69; p < 0.004) and three months (OR 0.65; p < 0.008) later. On multivariate analysis, the only outcome to remain significantly associated with LOS was physical mobility after six weeks (p = 0.024). There was no significant difference between short and standard stay women as regards their use of formal or lay care after discharge from hospital. The mean cost of hospital care was Pounds251 (in 1992) less for short than for standard stay patients. Most women (73% at six weeks) felt their LOS was appropriate. Short stay women were more likely to feel it was too short, though the difference was not statistically significant.

CONCLUSIONS

Short postoperative stays do not seem to be associated with any adverse outcomes and result in modest financial saving to the health service. There is potential for greater use of early discharge.

Why we need observational studies to evaluate the effectiveness of health care

The view is widely held that experimental methods (randomised controlled trials) are the "gold standard" for evaluation and that observational methods (cohort and case control studies) have little or no value. This ignores the limitations of randomised trials, which may prove unnecessary, inappropriate, impossible, or inadequate. Many of the problems of conducting randomised trials could often, in theory, be overcome, but the practical implications for researchers and funding bodies mean that this is often not possible. The false conflict between those who advocate randomised trials in all situations and those who believe observational data provide sufficient evidence needs to be replaced with mutual recognition of the complementary roles of the two approaches. Researchers should be united in their quest for scientific rigour in evaluation, regardless of the method used.

Why we need observational studies to evaluate the effectiveness of health care

The view is widely held that experimental methods (randomised controlled trials) are the "gold standard" for evaluation and that observational methods (cohort and case control studies) have little or no value. This ignores the limitations of randomised trials, which may prove unnecessary, inappropriate, impossible, or inadequate. Many of the problems of conducting randomised trials could often, in theory, be overcome, but the practical implications for researchers and funding bodies mean that this is often not possible. The false conflict between those who advocate randomised trials in all situations and those who believe observational data provide sufficient evidence needs to be replaced with mutual recognition of the complementary roles of the two approaches. Researchers should be united in their quest for scientific rigour in evaluation, regardless of the method used.

The effect of prostatectomy on symptom severity and quality of life

OBJECTIVE

To determine the effectiveness of prostatectomy in reducing symptom severity and bother and in improving disease-specific and general quality of life.

PATIENTS AND METHODS

A prospective, cohort study was performed in National Health Service and private hospitals in the Northern, Wessex, Mersey, and South-West Thames Health Regions which comprised 5276 men undergoing prostatectomy recruited by 101 of the 106 (96%) surgeons (specialist and non-specialist) performing prostatectomy during a 6-month period. Patients were assessed using the American Urological Association (AUA) Symptom Index Score, the AUA symptom bother score, disease-specific and generic quality-of-life scores, the occurrence of adverse events (urinary incontinence, erectile impotence and retrograde ejaculation) and three global (general) questions on the results of their treatment. The outcome was assessed 3 months after surgery.

RESULTS

Prostatectomy was effective in reducing both symptoms (initial mean score 20.1 reduced to 7.4, P < 0.001) and symptom bother (initial mean score 14.4 reduced to 4.3, P < 0.001). Not all men experienced a good reduction in symptoms; 121 (3.9%) were worse, 301 (9.6%) were the same, and 721 (23%) experienced only slight improvement. The type of operation, grade of principal operator and use of pre-operative investigations were not associated with the extent of symptomatic improvement. Changes in symptom severity were highly correlated with changes in bothersomeness++ and disease-specific quality of life but not with generic quality of life. A third of men who were continent before surgery reported some incontinence 3 months later, although only 6% found it a problem. Two-thirds of men experienced retrograde ejaculation and 31% experienced some erectile impotence following surgery.

CONCLUSIONS

Prostatectomy is effective in reducing symptoms in most men. Men who experience a substantial reduction in symptoms were more likely to report a favourable outcome. The study confirmed that approximately one-third of men reported an unfavourable result 3 months after their operation.

Development of a symptom severity index and a symptom impact index for stress incontinence in women

Stress incontinence is a common problem among women, yet there is no adequately validated instrument for measuring women's views of its severity (disease-specific health status). The only instrument for measuring the impact or bothersomeness of symptoms (disease-specific quality of life) has poor internal consistency. This paper describes the development and psychometric assessment of two new indexes, a Symptom Severity Index and a Symptom Impact Index. Following several qualitative enquiries, a questionnaire was developed and administered to 442 women undergoing stress incontinence surgery. The face and content validity of the items comprising the indexes was good. The Severity Index (0-20) showed good variability (median 14, interquartile range 6) and adequate internal consistency (alpha 0.76). The Impact Index (0-12) also had good variability (median 5, interquartile range 3.5) and internal consistency (alpha 0.80). Convergent and discriminant validity were demonstrated for both indexes. Test-retest reliability was high. While responsiveness is still to be tested, the two indexes are psychometrically strong and can be used to measure the severity and impact of stress incontinence in women.

Appropriateness of total hip replacement in the United Kingdom

The paper discusses the use of consensus models in determining the application of Total Hip Replacement surgery. The paper notes the wide national and international variation in use of this procedure but considers it in the broader context of reaching consensus on other surgical interventions.