Intra- and interrater reliability of the Ergo-Kit functional capacity evaluation method in adults without musculoskeletal complaints

Updating the Evidence on Functional Capacity Evaluation Methods: A Systematic Review

Objectives To synthesize the evidence on the psychometrics functional capacity evaluation (FCE) methods. Methods A systematic literature search in nine databases. The resulting articles were screened based on predefined in- and exclusion criteria. Two reviewers independently performed this screening. Included studies were appraised based on their methodological quality. Results The search resulted in 20 eligible studies about nine different FCE methods. The Baltimore Therapeutic Equipment work simulator showed a moderate predictive validity. The Ergo-Kit (EK) showed moderate variability and high inter- and intra-rater reliability. Low discriminative abilities and high convergent validity were found for the EK. Concurrent validity of the EK and the ERGOS Work Simulator was low to moderate. Moderate to high test-retest, inter- and intra-reliability was found in the Isernhagen Work-Systems (IWS) FCE. The predictive validity of the IWS was low. The physical work performance evaluation (PWPE) showed moderate test-retest reliability and moderate to high inter-rater reliability. Low internal and external responsiveness were found for the PWPE, predictive validity was high. The predictive validity of the short-form FCE was also high but need to be further examined on several psychometric properties. Low discriminative and convergent validity were found for the work disability functional assessment battery. The WorkHab showed moderate to high test-retest, inter- and intra-rater reliability. Conclusion Well-known FCE methods have been rigorously studied, but some of the research indicates weaknesses in their reliability and validity. Future research should address how these weaknesses can be overcome.

An evaluation of the inter-rater and intra-rater reliability of OccuPro's functional capacity evaluation

BACKGROUND

OccuPro's functional capacity evaluation (FCE) is used for assessing the client's readiness to return to work and three separate intra-rater and inter-rater reliability studies are explored here.

OBJECTIVE

Three separate studies were conducted on injured and un-injured adults to evaluate the inter-rater and intra-rater reliability of the OccuPro FCE (upper extremity and material handling subtests). All three studies are summarized in this publication.

METHODS

In study one, twenty participants completed firm grasp, simple grasp, pinch, fine motor, and gross motor testing. The participants included subjects with an orthopedic or musculoskeletal disorder affecting the upper extremities. In study two and three, 62 participants completed occasional squat lifts, occasional power lifts, occasional carrying, frequent squat lifts, frequent power lifts, and frequent carrying. The participants in all three studies were adult subjects between 20 and 70 years of age. Study one subjects had a previous illness or injury while subjects in study two and three had no history of injury.

RESULTS

Results from study one showed that the OccuPro FCE's four upper extremity subtests have moderate to excellent inter-rater reliability. In study two and three, results showed that the intra-rater reliability of these subtests were excellent and the inter-reliability of these subtests were moderate to good.

CONCLUSIONS

These three studies establish inter-rater and intra-rater reliability for the four upper extremity subtests and material handling testing within the OccuPro FCE system. This allows for multiple therapists to use OccuPro's FCE system with the same patient or multiple patients while having the confidence they will achieve consistent results and make sound return-to work or residual functional capacity decisions.

An investigation of the inter-rater reliability of the Valpar Joule functional capacity evaluation in healthy adults

BACKGROUND

A functional capacity evaluation (FCE) can provide a comprehensive, objective measure of a worker's ability to meet work demands to support return to work decision making. Research evidence of a FCE's reliability and validity, involving more than one study, and covering all test components with a diverse range of populations, is essential to ensure confidence in any FCE system.

OBJECTIVE

This study aimed to establish the inter-rater reliability of the Valpar Joule FCE functional capacity evaluation (FCE) for which there is currently limited published literature regarding its reliability.

METHODS

Twelve healthy subjects were digitally recorded completing the initial protocol of the Valpar Joule. Assessments were rated separately by 3 raters and the results then compared.

RESULTS

Using Intraclass Correlation Coefficients (ICC), with percentages of agreement and t-tests to determine bias, inter-rater reliability was high for determining last safe weight lifted for forceful tasks with ICC>0.90. Agreement ranged from 97.2% -100% for determining reasons for terminating tests; 97.2% -98.6% for identifying maximum safe capacity, but was only between 8.3% -50% for full agreement for identification of last weight safely lifted in forceful tasks. Differences were identified between raters with different training and experience for identifying poor body mechanics in lifting.

CONCLUSION

Results demonstrated high inter-rater reliability for the Valpar Joule functional capacity evaluation in healthy adults. Further development of criteria identifying poor body mechanics and training in its use is recommended to increase evaluator objectivity.

Interest of the Ergo-Kit(®) for the clinical practice of the occupational physician. A study of 149 patients recruited in a rehabilitation program

OBJECTIVE

Functional capacity evaluation is commonly used to assess the abilities of patients to perform some tasks. Ergo-Kit(®) is a validated tool assessing both functional capacities of patients and workplace demands. The objective of this study was to evaluate the relevance of the Ergo-Kit(®) data for occupational physicians during the return-to-work process.

METHODS

A retrospective and monocenter study was conducted on all patients included in a rehabilitation program and assessed with the Ergo-Kit(®) tool between 2005 and 2014. Workplace demands and patients' functional capacities were evaluated and confronted. Self-beliefs and perceived disability were also assessed and compared to the functional capacity evaluation.

RESULTS

One hundred and forty-nine working-age patients (85 men, 64 women; 39±12 years) suffering from musculoskeletal disorders or other diseases were included. Main causes of mismatch between workplace demands and functional capacities were manual handling of loads, postures with arms away from the body and repetitive motions at work; sitting posture was correlated with a lesser physical workload; and Oswestry score was correlated with functional capacities evaluated by the Ergo-Kit(®).

CONCLUSION

Ergo-Kit(®) is a relevant tool to assess the multidimensional aspects of workplace demands and functional capacities. It could be very helpful for occupational physicians to manage return-to-work.

Reliability of clinician rated physical effort determination during functional capacity evaluation in patients with chronic musculoskeletal pain

INTRODUCTION

Functional capacity evaluation (FCE) can be used to make clinical decisions regarding fitness-for-work. During FCE the evaluator attempts to assess the amount of physical effort of the patient. The aim of this study is to analyze the reliability of physical effort determination using observational criteria during FCE.

METHODS

Twenty-one raters assessed physical effort in 18 video-recorded FCE tests independently on two occasions, 10 months apart. Physical effort was rated on a categorical four-point physical effort determination scale (PED) based on the Isernhagen criteria, and a dichotomous submaximal effort determination scale (SED). Cohen's Kappa, squared weighted Kappa and % agreement were calculated.

RESULTS

Kappa values for intra-rater reliability of PED and SED for all FCE tests were 0.49 and 0.68 respectively. Kappa values for inter-rater reliability of PED for all FCE tests in the first and the second session were 0.51, and 0.72, and for SED Kappa values were 0.68 and 0.77 respectively. The inter-rater reliability of PED ranged from κ = 0.02 to κ = 0.99 between FCE tests. Acceptable reliability scores (κ > 0.60, agreement ≥80 %) for each FCE test were observed in 38 % of scores for PED and 67 % for SED. On average material handling tests had a higher reliability than postural tolerance and ambulatory tests.

CONCLUSION

Dichotomous ratings of submaximal effort are more reliable than categorical criteria to determine physical effort in FCE tests. Regular education and training may improve the reliability of observational criteria for effort determination.

Oxytocin in pregnancy and the postpartum: relations to labor and its management

The purpose of this study was to examine variations in endogenous oxytocin levels in pregnancy and postpartum state. We also explored the associations between delivery variables and oxytocin levels. A final sample of 272 mothers in their first trimester of pregnancy was included for the study. Blood samples were drawn during the first trimester and third trimester of pregnancy and at 8 weeks postpartum. Socio-demographic data were collected at each time point and medical files were consulted for delivery details. In most women, levels of circulating oxytocin increased from the first to third trimester of pregnancy followed by a decrease in the postpartum period. Oxytocin levels varied considerably between individuals, ranging from 50 pg/mL to over 2000 pg/mL. Parity was the main predictor of oxytocin levels in the third trimester of pregnancy and of oxytocin level changes from the first to the third trimester of pregnancy. Oxytocin levels in the third trimester of pregnancy predicted a self-reported negative labor experience and increased the chances of having an epidural. Intrapartum exogenous oxytocin was positively associated with levels of oxytocin during the postpartum period. Our exploratory results suggest that circulating oxytocin levels during the third trimester of pregnancy may predict the type of labor a woman will experience. More importantly, the quantity of intrapartum exogenous oxytocin administered during labor predicted plasma oxytocin levels 2 months postpartum, suggesting a possible long-term effect of this routine intervention, the consequences of which are largely unknown.

Reliability and Validity of Functional Capacity Evaluations: An Update

Functional capacity evaluations (FCEs) and other work-related assessments continue to be used in occupational rehabilitation to determine the capacity of injured and disabled workers to either return to their preinjury jobs, or to determine what they are capable of doing in a work context. New instruments have been developed and others refined. There is a continuing call however for these systems to demonstrate acceptable reliability and validity. Comprehensive reviews of reliability and validity of work-related assessments were published and included information up to the end of 1997. This study updates that information by examining research conducted on five FCEs over the subsequent 8-year period (January 1998–March 2006). The Isernhagen Work Systems (IWS) FCE had the most comprehensive coverage of all aspects of reliability and validity, while the Progressive Isoinertial Lifting Evaluation (PILE) was also extensively researched. Newer FCEs show promise and should continue to be investigated. Clinicians and others are encouraged to be informed consumers of the evidence that exists for the reliability and validity of FCEs and other work-related assessments.

Maximum acceptable weight of lift reflects peak lumbosacral extension moments in a functional capacity evaluation test using free style, stoop and squat lifting

It is unclear whether the maximum acceptable weight of lift (MAWL), a common psychophysical method, reflects joint kinetics when different lifting techniques are employed. In a within-participants study (n = 12), participants performed three lifting techniques--free style, stoop and squat lifting from knee to waist level--using the same dynamic functional capacity evaluation lifting test to assess MAWL and to calculate low back and knee kinetics. We assessed which knee and back kinetic parameters increased with the load mass lifted, and whether the magnitudes of the kinetic parameters were consistent across techniques when lifting MAWL. MAWL was significantly different between techniques (p = 0.03). The peak lumbosacral extension moment met both criteria: it had the highest association with the load masses lifted (r > 0.9) and was most consistent between the three techniques when lifting MAWL (ICC = 0.87). In conclusion, MAWL reflects the lumbosacral extension moment across free style, stoop and squat lifting in healthy young males, but the relation between the load mass lifted and lumbosacral extension moment is different between techniques.

PRACTITIONER SUMMARY

Tests of maximum acceptable weight of lift (MAWL) from knee to waist height are used to assess work capacity of individuals with low-back disorders. This article shows that the MAWL reflects the lumbosacral extension moment across free style, stoop and squat lifting in healthy young males, but the relation between the load mass lifted and lumbosacral extension moment is different between techniques. This suggests that standardisation of lifting technique used in tests of the MAWL would be indicated if the aim is to assess the capacity of the low back.

Quantitative assessment of the cervical spinal cord damage in neuromyelitis optica using diffusion tensor imaging at 3 Tesla

PURPOSE

To investigate whether quantitative MRI measures of cervical spinal cord white matter (WM) using diffusion tensor imaging (DTI) in neuromyelitis optica (NMO) differed from controls and correlated with clinical disability.

MATERIALS AND METHODS

Ten referred patients and 12 healthy volunteers were imaged on a 3 Tesla scanner and patients were clinically assessed on the Expanded Disability Status Scale (EDSS). Two raters quantified DTI-derived indices from all participants, including fractional anisotropy (FA), mean diffusivity (MD), parallel diffusivity (lambda[parallel]) and perpendicular diffusivity (lambda[perpendicular]) at C1-C6 for lateral and dorsal columns. After the inter-rater reliability test, univariate correlations between DTI measures and disability were assessed using the Spearman's rho correlation coefficient. Multiple regression analysis was performed to investigate which DTI measures independently correlated with the clinical score.

RESULTS

Statistical test results indicated high reliability of all DTI measurements between two raters. NMO patients showed reduced FA, increased MD and lambda[perpendicular] compared with controls while lambda[parallel] did not show any significant difference. The former three DTI metrics also showed significant correlations with disability scores, and especially FA was found to be sensitive to mild NMO (EDSS ≤ 3)

CONCLUSION

FA is a potentially useful quantitative biomarker of otherwise normal appearing WM damage in NMO. Such damage is associated with clinical disability.

Inter- and intra-rater reliability of the manual handling component of the WorkHab Functional Capacity Evaluation

The WorkHab Functional Capacity Evaluation (FCE) is widely used in Australian workplace injury management and occupational rehabilitation arenas; however, there is a lack of published literature regarding its reliability and validity.

PURPOSE

 This study investigated the intra- and inter-rater reliability of the manual handling component of this FCE.

METHOD

 A DVD was produced containing footage of the manual handling components of the WorkHab conducted with four injured workers. Therapist raters (n = 17) who were trained and accredited in use of the WorkHab FCE scored these components and 14 raters re-evaluated them after approximately 2 weeks. Ratings were compared using intraclass correlation coefficients (ICCs), paired sample t-tests (intra-rater), chi-squared (inter-rater) and percentage agreement.

RESULTS

Intra-rater agreement was high with ICCs for the manual handling components and manual handling score showing excellent reliability (0.94-0.98) and good reliability for identification of the safe maximal lift (ICC: 0.81). Overall inter-rater agreement ranged from good to excellent for the manual handling components and safe maximal lift determination (ICC > 0.9). Agreement for safe maximal lift identification was good.

CONCLUSIONS

 Ratings demonstrated substantial levels of intra-rater and inter-rater reliability for the lifting components of the WorkHab FCEs.

Reliability and reproducibility of perfusion MRI in cognitively normal subjects

Arterial spin labeling (ASL) magnetic resonance imaging (MRI) is becoming a popular method for measuring perfusion due to its ability of generating perfusion maps noninvasively. This allows for frequent repeat scanning, which is especially useful for follow-up studies. However, limited information is available regarding the reliability and reproducibility of ASL perfusion measurements. Here, the reliability and reproducibility of pulsed ASL was investigated in an elderly population to determine the variation in perfusion among cognitively normal individuals in different brain structures. Intraclass correlation coefficients (ICC) and within-subject variation coefficients (wsCV) were used to estimate reliability and reproducibility over a period of 1 year. Twelve cognitively normal subjects (75.5 ± 5.3 years old, six male and six female) were scanned four times (at 0, 3, 6 and 12 months). No significant difference in cerebral blood flow (CBF) was found over this period. CBF values ranged from 46 to 53 ml/100 g per minute in the medial frontal gyrus (MFG) and from 40 to 44 ml/100 g per minute over all gray matter regions in the superior part of the brain. Data obtained from the first two scans were processed by two readers and showed high reliability (ICC >0.97) and reproducibility (wsCV <6%). However, over the total period of 1 year, reliability reduced to a moderate level (ICC=0.63-0.74) with wsCVs of gray matter, left MFG, right MFG of 13.5%, 12.3%, and 15.4%, respectively. In conclusion, measurement of CBF with pulsed ASL provided good agreement between inter-raters. A moderate level of reliability was obtained over a 1-year period, which was attributed to variance in slice positioning and coregistration. As such pulsed ASL has the potential to be used for CBF comparison in longitudinal studies.

Test-Retest Reliability of the GAPP Functional Capacity Evaluation in Healthy Adults

Background. Functional capacity evaluations are commonly used in work rehabilitation practice to assess a person's capacity to perform work-related activities. Purpose. This study examined the test-retest reliability of participants' performance and administrator ratings using the Gibson Approach to Functional Capacity Evaluation (GAPP FCE). Methods. Forty-eight healthy adults were evaluated twice on 12 recommended core items of the GAPP FCE and rated for overall performance. Findings. The ICCs and 95% CIs for the Physical Level of Work and Alternative Physical Level of Work Ratings were 0.93 (0.87-0.96) and 0.86 (0.72-0.93) respectively. The ICCs for the core item-level ratings ranged from 0.15 to 0.94, and the ICCs for the actual loads handled in the manual handling items ranged from 0.88 to 0.95. Implications. The stability of an overall physical level of work rating shows potential for use in functional capacity evaluation practice and research. Further research is needed to investigate other measurement properties of the GAPP FCE using populations with injury or disability.

Effect of Functional Capacity Evaluation information on the judgment of physicians about physical work ability in the context of disability claims

Purpose

To test whether Functional Capacity Evaluation (FCE) information lead insurance physicians (IPs) to change their judgment about the physical work ability of claimants with musculoskeletal disorders (MSDs).

Methods

Twenty-seven IPs scored twice the physical work ability of two claimants for 12 specified activities, using a visual analogue scale. One claimant performed an FCE, the other served as a control. Outcome measure was the difference between experimental and control group in number of shifts in the physical work ability for the total of 12 specified activities.

Results

The IPs changed their judgment about the work ability 141 times when using FCE information compared to 102 times when not using this information (P-value = 0.001), both in the direction of more and less ability.

Conclusions

The IPs change their judgment of the physical work ability of claimants with MSDs in the context of disability claim procedures more often when FCE information is provided.

Construct validity of functional capacity evaluation lifting tests in construction workers on sick leave as a result of musculoskeletal disorders

OBJECTIVES

To assess the construct (discriminative and convergent) validity of 5 Ergo-Kit (EK) functional capacity evaluation (FCE) lifting tests in construction workers on sick leave as a result of musculoskeletal disorders (MSDs).

DESIGN

Cross sectional within-subject design.

SETTING

Occupational health service for the construction industry.

PARTICIPANTS

Male construction workers (N=72) on 6-week sick leave as a result of MSDs.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE

After being assessed on 5 EK FCE lifting tests, participants were asked to complete the Von Korff questionnaire on pain intensity and disability as a result of MSDs and the instrument for disability risk assessing the risk for work disability. Discriminative validity was evaluated by comparing the results of the EK FCE lifting test scores between the 2 groups of participants based on the instrument for disability risk scores (high risk for work disability compared with low risk for work disability). Convergent validity was evaluated by assessing the associations between the results of the EK FCE lifting tests and Von Korff questionnaire self-reported pain intensity and disability as a result of MSDs.

RESULTS

The hypothesized differences between both instrument for disability risk groups on the 5 EK FCE lifting tests were found in the expected direction but were not statistically significant (1 test exhibited a trend). Pearson correlation coefficients showed a poor convergent validity between the scores of the Von Korff questionnaire and the EK FCE lifting tests (-.29< or =r< or =.05).

CONCLUSIONS

Poor construct validity of the 5 EK lifting tests was found: discriminative validity was not statistically established, and convergent validity with self-reported pain intensity and disability was poor.

Reliability and agreement of 5 Ergo-Kit functional capacity evaluation lifting tests in subjects with low back pain

OBJECTIVE

To assess interrater reliability and agreement of 5 Ergo-Kit functional capacity evaluation lifting tests in subjects with low back pain (LBP).

DESIGN

Within-subjects design, with 2 repeated measurements.

SETTING

Academic medical center in The Netherlands.

PARTICIPANTS

Twenty-four subjects (10 men, 14 women) with LBP.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURE

Five Ergo-Kit lifting tests (2 isometric, 3 dynamic) were assessed on 2 occasions (t1, t2), by 2 different raters (R1, R2). The interval between the test sessions was 3 days. Interrater reliability level was expressed with the intraclass correlation coefficient (ICC), and the level of agreement between raters with the standard error (SE) of measurement.

RESULTS

ICCs means (reliability) of isometric and dynamic Ergo-Kit lifting tests ranged from .94 to .97, and SE of measurement values (agreement) ranged from 1.9 to 8.6 kg.

CONCLUSIONS

There was good reliability and agreement between raters of the isometric and dynamic Ergo-Kit lifting tests in subjects with LBP, which supports the use of these tests to assess functional lifting capacity.

The utility of functional capacity evaluation: the opinion of physicians and other experts in the field of return to work and disability claims

OBJECTIVES

This qualitative study explored how Dutch experts perceive the utility of functional capacity evaluation (FCE) for return to work (RTW) and disability claim (DC) assessment purposes.

METHODS

Twenty-one RTW case managers and 29 DC experts were interviewed by telephone using a semi-structured interview schedule.

RESULTS

The RTW case managers valued the utility of FCE on a scale of 0-10. Their mean valuation was 6.5 (SD 1.5). The average valuation for DC experts was 4.8 (SD 2.2). Arguments in favor of FCE were (1) its ability to confirm own opinions and (2) the objectivity of its measurement method. Arguments against FCE were (1) the redundancy of the information it provides and (2) the lack of objectivity. Indications for FCE were musculoskeletal disorders, a positive patient self-perception of ability to work, and the presence of an actual job. Contraindications for FCE were medically unexplained disorders, a negative patient self-perception of ability to work, and the existence of disputes and legal procedures.

CONCLUSIONS

The responding RTW case managers perceived FCE to be more useful than the responding DC experts. The question of whether the arguments presented for and against the utility of FCE are valid is one that should be addressed in a future study.