Reproducibility and responsiveness of evaluative outcome measures. Theoretical considerations illustrated by an empirical example

OBJECTIVE

This article outlines basic methods for quantifying reproducibility and responsiveness of evaluative outcome measures.

METHODS

The background noise in stable patients provides the desired information to quantify the reproducibility. From this, the smallest real difference (SRD) for longitudinal differences can be derived. We propose to use the SRD to define responsiveness: from all patients who change according to an external criterion, we take the percentage that changes at least SRD on the outcome measure. A more complete picture of the responsiveness of the outcome measure arises when the receiver operating characteristic (ROC) is constructed, corresponding to all possible cut-off change scores. The proposed methods are illustrated with an empirical example.

RESULTS

In the illustration the methods appeared to be very useful and complemented each other. We could evaluate whether the observed change score was larger than that expected due to chance. With the methods it was possible to evaluate both the ability of an instrument to detect change if there is a real change in health status (sensitivity to change) and the ability to detect absence of change if there is no real change (specificity to change).

CONCLUSION

We presented the use of SRDs and ROC curves for quantifying reproducibility and responsiveness. We started with the basic notions and arrived at methods that are both understandable and useful.

Measurement of prolonged fatigue in the working population: determination of a cutoff point for the checklist individual strength

In the Netherlands, a large-scale prospective cohort study was started on prolonged fatigue in the working population. The 1st issue that had to be addressed was the determination of a cutoff point for fatigue for use in the working population. Fatigue is measured with the Checklist Individual Strength (CIS), a 20-item self-report questionnaire. This article demonstrates the process of decision making in the determination of the cutoff point. Total CIS scores were calculated, sensitivity and specificity were compared for potential cutoff points, and a receiver operating characteristics analysis was conducted. A CIS total cutoff point for fatigue of >76 was determined, with a specificity of 90% and a sensitivity of 73%. Limitations regarding the use of cutoff points are discussed. It is concluded that the defined cutoff point seems to be appropriate for use in the working population.

Behavioral-graded activity compared with usual care after first-time disk surgery: considerations of the design of a randomized clinical trial

OBJECTIVE

To present the design of a trial on the effectiveness of a behavioral-graded activity model.

DESIGN

Randomized clinical trial.

PATIENTS

Patients undergoing first-time lumbar disk surgery who still have low-back pain at the 6-week neurosurgical consultation.

INTERVENTIONS

A patient-tailored behavioral-graded activity program that is based on operant therapy. The key elements of this program are baseline measurements, goal-setting, and time-contingency. This program is compared with usual care in physiotherapy, which is pain-contingent.

OUTCOME MEASURES

Primary measures are the patient's global impression of the effect and their functional status. Secondary measures are kinesiophobia, catastrophizing, pain, main complaint, range of motion, and relapses. The direct and indirect costs will also be assessed. The effect measures are rated before randomization and 3, 6, and 12 months later.

DISCUSSION

Several trials have been conducted on the effectiveness of behavioral treatments. Subjects were always patients with chronic low-back pain. In this trial, we apply such a treatment in patients after first-time disk surgery in a primary care setting.

Fatigue among working people: validity of a questionnaire measure

OBJECTIVES

To evaluate the validity of the checklist individual strength questionnaire (CIS) in the working population. This 20 item self reported questionnaire has often been used in patients with chronic fatigue. To date, no research has focused on the validity of the CIS in occupational groups.

METHODS

To evaluate the discriminant validity the CIS was filled out by five groups of employees with expected differences in fatigue. The convergent validity was evaluated by comparing the results of the CIS with the results of three related measures: measured unidimensional fatigue, burnout, and need for recovery.

RESULTS

The CIS was able to discriminate between fatigued and non-fatigued employees in occupational groups. The expected agreement between the results of the CIS and related measures was confirmed.

CONCLUSIONS

The CIS seems to be an appropriate instrument for measuring fatigue in the working population.

A patient-specific approach for measuring functional status in low back pain

BACKGROUND

Activities and their importance for daily living vary widely between patients. Patient-specific measurement of functional status means that the evaluation is focused on activities that an individual patient selected as main complaints.

OBJECTIVE

To develop and to evaluate a patient-specific approach for measuring functional status in low back pain.

STUDY DESIGN

A cohort of 150 patients was measured at baseline and 12 weeks later.

METHODS

The feasibility of the patient-specific approach was evaluated in patients with nonspecific low back pain. We used effect size statistics to evaluate responsiveness in terms of sensitivity to change and specificity to change.

RESULTS

The selection procedure for the main complaint was feasible but labor intensive. The patient-specific approach was able to detect changes in complaints that were highly relevant for the patients. The patient-specific approach appeared to be more sensitive to change but less specific to change compared with other instruments.

CONCLUSIONS

On the basis of this study it would be valuable to apply the patient-specific approach in future studies, also with the aim of further evaluation. In the meantime a number of practical problems of the method need to be resolved.

[Roaming through the methodology. VIII. Pilot studies: sense and nonsense]

A pilot study is often performed before the start of a large study. Its aim is improving the methodological quality and evaluating the feasibility. The results of a pilot study are also often used to gain an impression of the efficacy of an intervention. For this purpose a pilot is absolutely unsuited. However, the estimate of the effect of an intervention in small studies such as pilot studies is determined to a large extent by chance. When conducting small studies the chance of publication bias is large. Small studies of adequate methodological quality should be published, whether the results are positive or negative, significant or non-significant, because combination in a later cumulative meta-analysis may lead to sufficient power to assess the efficacy of an experimental intervention.

Outcome measures for low back pain research. A proposal for standardized use

STUDY DESIGN

An international group of back pain researchers considered recommendations for standardized measures in clinical outcomes research in patients with back pain.

OBJECTIVES

To promote more standardization of outcome measurement in clinical trials and other types of outcomes research, including meta-analyses, cost-effectiveness analyses, and multicenter studies.

SUMMARY OF BACKGROUND DATA

Better standardization of outcome measurement would facilitate comparison of results among studies, and more complete reporting of relevant outcomes. Because back pain is rarely fatal or completely cured, outcome assessment is complex and involves multiple dimensions. These include symptoms, function, general well-being, work disability, and satisfaction with care.

METHODS

The panel considered several factors in recommending a standard battery of outcome measures. These included reliability, validity, responsiveness, and practicality of the measures. In addition, compatibility with widely used and promoted batteries such, as the American Academy of Orthopaedic Surgeons Lumbar Cluster were considered to minimize the need for changes when these instruments are used.

RESULTS

First, a six-item set was proposed, which is sufficiently brief that it could be used in routine care settings for quality improvement and for research purposes. An expanded outcome set, which would provide more precise measurement for research purposes, includes measures of severity and frequency of symptoms, either the Roland or the Oswestry Disability Scale, either the SF-12 or the EuroQol measure of general health status, a question about satisfaction with symptoms, three types of "disability days," and an optional single item on overall satisfaction with medical care.

CONCLUSION

Standardized measurement of outcomes would facilitate scientific advances in clinical care. A short, 6-item questionnaire and a somewhat expanded, more precise battery of questionnaires can be recommended. Although many considerations support such recommendations, more data on responsiveness and the minimally important change in scores are needed for most of the instruments.

[Roaming through methodology. VII. Reproducibility of measurements]

Reproducibility measurements are important for a proper interpretation of medical data. Kappa is the most adequate measure for categorical variables. Kappa adjusts the observed agreement for chance agreement. The interpretation of kappa is rather difficult. The kappa value is influenced by the number of categories used for classification and the prevalence of scores of the observers. For continuous variables the Pearson correlation coefficient can be used, keeping in mind its ignoring systematic errors and its dependence on the heterogeneity of the data. Another method to assess reproducibility for continuous variables is the method of limits of agreement. This method distinguishes systematic and random errors, and quantifies the differences in the dimension of the measurements. In general, the interpretation of the various measures of agreement is helped by a visual presentation of the data in a table or figure.

Efficacy of traction for nonspecific low back pain. 12-week and 6-month results of a randomized clinical trial

STUDY DESIGN

A randomized clinical trial.

OBJECTIVES

To assess the efficacy of motorized continuous traction for low back pain.

SUMMARY OF BACKGROUND DATA

The available studies on the efficacy of lumbar traction do not allow clear conclusions because of severe methodologic flaws. The current trial aimed to overcome these shortcomings.

METHODS

Patients with at least 6 weeks of nonspecific low back pain were selected. High-dose traction was compared with sham (or low-dose) traction. Sham traction was given with a specially developed brace that becomes tighter in the back during traction. This was experienced as if real traction were exerted. The patients and the outcome assessor were unaware of treatment allocation. Outcome measures were: patient's global perceived effect, severity of main complaints, functional status, pain, range of motion, work absence, and medical treatment. Results for the outcome measures at 12 weeks and 6 months after randomization are presented.

RESULTS

One hundred and fifty-one patients were randomly allocated to one of the two treatment methods. Intention-to-treat analysis of the 12-week and 6-month results showed no statistically significant differences between the groups on all outcome measures: all 95% confidence intervals included the value zero. The number of patients lost to follow-up study was very low. Other analyses showed the same results.

CONCLUSIONS

Most common flaws of earlier studies on traction therapy could be overcome. This trial did not support the claim that traction is efficacious for patients with low back pain.

Efficacy of traction for non-specific low back pain: a randomised clinical trial

Previous trials to assess the efficacy of lumbar traction for back pain have been methodologically flawed. To avoid these shortcomings, we conducted a randomised controlled trial in which high-dose traction was compared with sham traction. The sham traction was given with a specially developed brace that tightens in the back during traction. To the patient, the experience is that of traction. The patients and outcome assessor were blinded for the assigned treatment. 151 patients with at least six weeks of non-specific low back pain were randomised. Intention to treat analysis showed no differences between the groups on all outcome measures (patients' global perceived effect, severity of main complaints, functional status and pain); all 95% confidence intervals included the value zero. The number of withdrawals from treatment, loss to follow-up, and protocol deviations was low. Consequently, the per-protocol analysis showed results similar to the intention to treat analysis. Subgroup analyses did not show any group for which traction might seem promising. Our data do not support the claim that traction is effective for patients with low back pain.

Measuring the functional status of patients with low back pain. Assessment of the quality of four disease-specific questionnaires

STUDY DESIGN

This study was a literature review of the quality of four disease-specific functional status questionnaires for patients with low back pain: Oswestry; Million; Roland; and Waddell disability questionnaire.

OBJECTIVES

The questionnaires were evaluated in terms of general description, scale structure, reliability, validity, responsiveness, and clinical research applications.

SUMMARY OF BACKGROUND DATA

Functional status is an outcome of great interest for clinical trials of low back pain.

METHODS

A computer-aided search was conducted of articles published between 1981 and 1993 and references given in selected relevant publications. Articles were selected if at least one of the four functional status questionnaires was used or if the authors gave relevant information about the methodology of these questionnaires.

RESULTS

There was not enough information available about the criteria of item selection used for the development of the questionnaires. The test-retest reproducibility of the questionnaires seemed satisfactory. The Oswestry and Roland disability questionnaires have been used and evaluated more frequently than the Million and Waddell. Therefore, we can be more certain about the validity and responsiveness of the former pair of questionnaires.

CONCLUSION

In the absence of a gold standard, direct comparisons of evaluative functional status questionnaires in a single patient group are needed. Through direct comparisons, comparative validity and responsiveness can be assessed. Functional status measures are not currently used in many settings in which they would be valuable. It is important to encourage their wider use in clinical trials. Additional research is needed to compare and improve the existing questionnaires.

The efficacy of traction for lumbar back pain: design of a randomized clinical trial

OBJECTIVE

To present the design of a trial on the efficacy of lumbar traction.

DESIGN

Randomized clinical trial.

PATIENTS

Patients with a minimum of 6 wk, nonspecific low back pain.

INTERVENTION

High-dose, motorized, continuous traction with a force between 35% and 50% of the total body weight was compared with sham or low-dose traction with a force between 0 and 20% of body weight. The sham traction was given with a specially developed brace that becomes tighter in the back during traction. This is experienced by patients as if traction were exerted.

OUTCOME MEASURES

Primary measures were the patient's global impression of the effect and the severity of three main complaints. Secondary effect measures were functional status, pain, range of motion, work absence and recurrences. The effect measures were rated before randomization and 4 wk, 12 wk and 6 months later.

CONCLUSION

There have been a number of earlier trials on the efficacy of lumbar traction; they suffer, however, from severe methodological flaws. This trial aimed to avoid these shortcomings.

The efficacy of traction for back and neck pain: a systematic, blinded review of randomized clinical trial methods

BACKGROUND AND PURPOSE

The purpose of this study was to conduct a systematic analysis of the literature to assess the efficacy of traction for patients with neck or back pain.

SUBJECTS

For this purpose, randomized clinical trials comparing traction with other treatments were selected.

METHODS

A computer-aided search of the literature was conducted for relevant articles, followed by blinded assessment of the methods of the studies. The main outcome measures were (1) scoring for quality of the designated conduct of studies (based on a methodological checklist with four main categories: study population, interventions, measurement of effect, and data presentation) and the main conclusions of author(s) with regard to traction and (2) calculation of confidence intervals and power of the studies.

RESULTS

Only three studies scored more than 50 points (maximum score = 100 points), suggesting that most of the selected studies were of poor quality. None of these three studies showed favorable results for traction. Only four studies, of which one scored more than 50 points, had an acceptable power (1- beta > 80%).

CONCLUSION AND DISCUSSION

The available reports of studies on the efficacy of traction for back and neck pain do not allow clear conclusions due to the methodological flaws in their design and conduct. Most studies lacked power (1-beta) due to small sample sizes. To date, no conclusions can be drawn about whether a specific traction modality for back or neck pain is effective, or more efficacious than other treatments. There are no clear indications, however, that traction is an ineffective therapy for back and neck pain. Further trials are needed in which much more attention should be paid to proper design and conduct, as well as to clear descriptions of crucial methodological features and results.