Use of a structured functional evaluation process for independent medical evaluations of claimants presenting with disabling mental illness: rationale and design for a multi-center reliability study

Patient-centeredness in psychiatric work disability evaluations and the reproducibility of work capacity estimates

OBJECTIVES

To evaluate the extent of patient-centeredness in psychiatric work disability evaluations and its association with the reproducibility of work capacity (WC) estimates.

METHODS

In our mixed methods study, 29 video-taped interviews conducted in psychiatric work disability evaluations were coded with the Roter Interaction Analysis System (RIAS) and different measures of patient-centeredness were derived from these codings, including a summary patient-centred communication ratio. Four experts each estimated a claimant's WC on a scale from 0% to 100%.

RESULTS

Patient-centred communication ratios were always >1, suggesting a preponderance of psychosocial information exchange. In contrast, utterances reflecting empathy were rare e.g., the expert did not address the claimant's emotions in 25 of 29 interviews. None of the derived patient-centeredness measures showed a significant association with WC reproducibility.

CONCLUSIONS

Many of the experts' questions addressed the claimant's lifestyle and psychosocial situation. However, this likely reflected factual requirements for the expert opinion, rather than patient-centeredness. Indeed, the experts rarely showed empathy, which is a hallmark characteristic of patient-centeredness. The reproducibility of work capacity estimates was not modulated by patient-centeredness, irrespective of its quantification.

PRACTICE IMPLICATIONS

Patient-centeredness in work disability evaluations should find its entry in continuing education of experts.

The assessment of capacity limitations in psychiatric work disability evaluations by the social functioning scale Mini-ICF-APP

OBJECTIVE

Insurers frequently commission medical experts to estimate the degree of the remaining work capacity (RWC) in claimants for disability benefits. The social functioning scale Mini-ICF-APP allows for a rating of activity and participation limitations in thirteen capacity domains, considered as particularly relevant for work ability. The current study sought to evaluate the role of the Mini-ICF-APP ratings in psychiatric work disability evaluations, by examining how the capacity limitation ratings varied with the claimants' primary psychiatric diagnoses and how the ratings were related to RWC estimates.

METHODS

Medical experts estimated the RWC of 946 claimants with mental disorders and rated their activity and participation limitations using the Mini-ICF-APP, with higher ratings reflecting more severe limitations. The ratings were compared between claimants with different psychiatric diagnoses by analyses of variance. The mean Mini-ICF-APP rating across all capacity domains as well as all capacity-specific ratings were entered in simple or multiple regression models to predict the RWC in an alternative job.

RESULTS

The Mini-ICF-APP capacity limitation ratings in all domains but mobility were higher for claimants with personality and behavior disorders as compared to those with mood disorders or with neurotic, stress-related and somatoform disorders. The largest differences were observed in social capacities (e.g. group integration: F _{2, 847} = 78.300, P < 0.001). In claimants with depression, all ratings increased with the severity of the diagnosis (all Fs _{2, 203} > 16.393, all Ps < 0.001). In the overall sample, the mean Mini-ICF-APP rating showed a strong negative correlation with the estimated RWC (r = -.720, P < 0.001). Adding the capacity-specific ratings to the prediction model improved this prediction only marginally.

DISCUSSION

The Mini-ICF-APP allows for documenting claimants' activity and participation limitations, which is likely to increase the transparency of medical experts' RWC estimates and enables them to check the plausibility of such estimates. However, our study showed that despite the strong association between RWC and Mini-ICF-APP ratings, half of the RWC variance was unrelated to the capacity limitations documented in the Mini-ICF-APP.

Perceived fairness of claimants undergoing a work disability evaluation: Development and validation of the Basel Fairness Questionnaire

Background

There are currently no tools for assessing claimants’ perceived fairness in work disability evaluations. In our study, we describe the development and validation of a questionnaire for this purpose.

Method

In cooperation with subject-matter experts of Swiss insurance medicine, we developed the 30-item Basel Fairness Questionnaire (BFQ). Claimants anonymously answered the questionnaire immediately after their disability evaluation, still unaware about its outcome. For each item, there were four response options, ranging from “strongly disagree” to “strongly agree”. The construct validity of the BFQ was assessed by running a principal component analysis (PCA).

Results

In 4% of the questionnaires, the claimants’ perception on the disability evaluation was negative (below the median of the scale). The PCA of the items responses followed by an orthogonal rotation revealed four factors, namely (1) Interviewing Skills, (2) Rapport, (3) Transparency, and (4) Case Familiarity, explaining 63.5% of the total variance.

Discussion

The ratings presumably have some positive bias by sample selection and response bias. The PCA factors corresponded to dimensions that subject-matter experts had beforehand identified as relevant. However, all item ratings were highly intercorrelated, which suggests that the presumed underlying dimensions are not independent.

Conclusion

The BFQ represents the first self-administered instrument for measuring claimants’ perceived fairness of work disability evaluations, allowing the assessment of informational, procedural, and interactive justice from the perspective of claimants. In cooperation with Swiss assessment centres, we plan to implement a refined version of the BFQ as feedback instrument in work disability evaluations.

Functional Interviewing Was Associated With Improved Agreement Among Expert Psychiatrists in Estimating Claimant Work Capacity: A Secondary Data Analysis of Real-Life Work Disability Evaluations

OBJECTIVE

Functional evaluations establish functional and work (in-)capacities in the context of disability assessments and are increasingly recommended as a modern technique for work disability assessments. The RELY (Reliable disability EvaLuation in psychiatrY)-studies introduced semi-structured functional interviews in real-life assessments of claimants with mental disorders for evaluating their self-perceived health-related limitations and for investigating the reproducibility of work capacity (WC) estimates. Functional interviews elicit claimants' self-perceptions about their work-related limitations and capacities in the labour market. This secondary data analysis explored the coverage of work-related key topics in these interviews and investigated whether interviews with high coverage (versus low coverage) of work-related topics resulted in better reproducibility of WC estimates among experts.

METHODS

Thirty video-taped RELY-assessments underwent a content analysis along a predefined framework for functional interviewing, including the claimant's self-perceived work limitations and work-related health complaints as centrepieces of functional interviewing. Following transcription, interviews were segmented into coding units. Coding units were allocated to the five steps with 19 key topics of the framework. Enquiry into key topics was ascertained by summing the functional coding units per key topic. Median split grouped the interviews into high and low coverage of functional topics and compared them for inter-rater reliability (intraclass correlation coefficient, ICC) and inter-rater agreement (standard error of measurement, SEM).

RESULTS

Interviews were broken down in 40,010 coding units, 31% of which addressed functional topics. Enquiries in self-perceived work limitations and work-related health complaints were sparse (coding units median_psychiatrist between 0 and 1.5, median_patients between 0 and 9.5). High coverage interviews enquired on more functional topics (68% vs. 42%, chi²(1, N = 38) = 5.32, p = 0.021) and in more depth (36% vs. 16% of functional coding units, chi²(1, N = 1,314) = 141.15, p < 0.001). Interviews with higher functional coverage reached significantly higher inter-rater agreement in WC ratings among experts (mean difference in SEM, low-high coverage, 7.5% WC, 95% CI 0.2 to 15.1%WC). Inter-rater reliability was low in both groups (ICC, 0.38 versus 0.40).

CONCLUSIONS

Content analysis showed little enquiry by experts on claimants' self-perceived activity limitations and work-related capacity. The association between interviews with higher functional coverage and better expert agreement on the claimants' remaining WC requires confirmation in prospective studies.

Current challenges in disability evaluation and the need for a goal-oriented approach based on the ICF: a qualitative stakeholder analysis in the context of the Swiss accident insurance

PURPOSE

By using the eligibility determination process of the Swiss accident insurance (Suva) as a case in point, we aimed to examine current challenges and the need for a goal-oriented ICF-based approach in disability evaluation as experienced by different stakeholder groups.

MATERIALS AND METHODS

Descriptive qualitative design involving semi-structured expert interviews with five main stakeholder groups of the eligibility determination process (i.e., Suva insurance officers, Suva insurance physicians, treating physicians, lawyers and judges). Interviews were thematically analyzed and their results reflected upon in focus groups with selected interview participants.

RESULTS

Forty-three interviews and three focus groups were conducted. Participants pointed to challenges related to standardization, transparency, objectivity, efficiency and contextual factors. An ICF-based standard documenting claimant-job mismatches and their determinants was deemed promising for ensuring comprehensible and valid eligibility decisions, systematic and uniform reporting and a goal-oriented eligibility determination process. Concerns primarily addressed a potential pseudo-accuracy when using the standard for eligibility decisions.

CONCLUSIONS

Implementing a goal-oriented ICF-based standard may help to approach current challenges in disability evaluation. Our findings support the development of such a standard for Suva's eligibility determination, but also point at the importance of involving key stakeholders to ensure its applicability and cross-disciplinary acceptance.IMPLICATIONS FOR REHABILITATIONPoor standardization and transparency as well as objectivity and efficiency issues represent key challenges in disability evaluation, as experienced by the different stakeholder groups involved in an eligibility determination process.A standardized ICF-based approach focusing on the documentation of claimant-job mismatches and their determinants is promising for improving transparency, validity and comparability of eligibility decisions.With its focus on claimants' ability to participate at work such an ICF-based standard has the potential to ensure a goal-oriented planning of assessments and interventions in the eligibility determination process.Our study provides specific targets for optimizing eligibility determination in disability evaluation but also for ensuring a customized and effective return to work after sustaining an injury.

The reproducibility of psychiatric evaluations of work disability: two reliability and agreement studies

BACKGROUND

Expert psychiatrists conducting work disability evaluations often disagree on work capacity (WC) when assessing the same patient. More structured and standardised evaluations focusing on function could improve agreement. The RELY studies aimed to establish the inter-rater reproducibility (reliability and agreement) of 'functional evaluations' in patients with mental disorders applying for disability benefits and to compare the effect of limited versus intensive expert training on reproducibility.

METHODS

We performed two multi-centre reproducibility studies on standardised functional WC evaluation (RELY 1 and 2). Trained psychiatrists interviewed 30 and 40 patients respectively and determined WC using the Instrument for Functional Assessment in Psychiatry (IFAP). Three psychiatrists per patient estimated WC from videotaped evaluations. We analysed reliability (intraclass correlation coefficients [ICC]) and agreement ('standard error of measurement' [SEM] and proportions of comparisons within prespecified limits) between expert evaluations of WC. Our primary outcome was WC in alternative work (WCalternative.work), 100-0%. Secondary outcomes were WC in last job (WClast.job), 100-0%; patients' perceived fairness of the evaluation, 10-0, higher is better; usefulness to psychiatrists.

RESULTS

Inter-rater reliability for WCalternative.work was fair in RELY 1 (ICC 0.43; 95%CI 0.22-0.60) and RELY 2 (ICC 0.44; 0.25-0.59). Agreement was low in both studies, the 'standard error of measurement' for WCalternative.work was 24.6 percentage points (20.9-28.4) and 19.4 (16.9-22.0) respectively. Using a 'maximum acceptable difference' of 25 percentage points WCalternative.work between two experts, 61.6% of comparisons in RELY 1, and 73.6% of comparisons in RELY 2 fell within these limits. Post-hoc secondary analysis for RELY 2 versus RELY 1 showed a significant change in SEMalternative.work (- 5.2 percentage points WCalternative.work [95%CI - 9.7 to - 0.6]), and in the proportions on the differences ≤ 25 percentage points WCalternative.work between two experts (p = 0.008). Patients perceived the functional evaluation as fair (RELY 1: mean 8.0; RELY 2: 9.4), psychiatrists as useful.

CONCLUSIONS

Evidence from non-randomised studies suggests that intensive training in functional evaluation may increase agreement on WC between experts, but fell short to reach stakeholders' expectations. It did not alter reliability. Isolated efforts in training psychiatrists may not suffice to reach the expected level of agreement. A societal discussion about achievable goals and readiness to consider procedural changes in WC evaluations may deserve considerations.

Sick-listed workers' expectations about and experiences with independent medical evaluation: a qualitative interview study from Norway

PURPOSE

To reduce the country's sick leave rate, Norwegian politicians have suggested independent medical evaluations (IMEs) for sick-listed workers. IME was tested in a large, randomized controlled trial in one Norwegian county (Evaluation of IME in Norway, or 'the NIME trial'). The current study´s aim was to explore sick-listed workers' expectations about and experiences with participating in an IME.

MATERIAL AND METHODS

Nine individual semi-structured telephone interviews were conducted. Our convenience sample included six women and three men, aged 35-59 years, who had diverse medical reasons for being on sick leave. Systematic text condensation was used for analysis.

RESULTS

The participants questioned both the IME purpose and timing, but felt a moral obligation to participate. Inadequate information provided by their general practitioner (GP) to the IME doctor was considered burdensome by several participants. However, most participants appreciated the IME as a positive discussion, even if they did not feel it had any impact on their follow-up or return-to-work process.

CONCLUSIONS

According to the sick-listed workers the IMEs were administered too late and disturbed already initiated treatment processes and return to work efforts. Still, the consultation with the IME doctor was rated as a positive encounter, contrary to their expectations. Our results diverge from findings in other countries where experiences with IME consultations have been reported as predominantly negative. These findings, along with additional, upcoming evaluations, will serve as a basis for the Norwegian government's decision about whether to implement IMEs on a regular basis. Key points   Independent medical evaluations for sick-listed workers has been tested out in a large Norwegian RCT and will be evaluated through qualitative interviews with participating stakeholders and by assessing the effects on RTW and costs/benefits. In this study, we explored sick-listed workers' expectations about and experiences with participating in an IME.   • Participants questioned both the IME purpose and timing, but felt a moral obligation to participate.   • Inadequate information provided by their general practitioner (GP) to the IME doctor was considered burdensome by several participants   • Sick-listed workers appreciated the IME as a positive discussion, even if they did not feel it had any impact on their follow-up or return-to-work process.

Online Survey of Medical and Psychological Professionals on Structured Instruments for the Assessment of Work Ability in Psychiatric Patients

Objective: To evaluate perceived needs and difficulties related to instruments for assessing work ability in individuals with mental disorders. Method: We conducted an online survey of 104 German-speaking medico-legal experts (forensic psychiatric and psychology experts, insurance physicians) and therapists. Results: The large majority of respondents reported they would welcome a standardized, structured instrument for the assessment of work ability. High predictiveness, inter-rater agreement, comprehensibility for laymen, and symptom validity were desired in roughly equal measure as the main characteristic of such an instrument. More women than men, and more medico-legal experts than therapists, considered symptom validation as always necessary. Pain, personality, and affective disorders were perceived to be the most difficult disorders in the context of work ability assessments. Conclusion: Our survey documents professionals' wish for a structured assessment of work ability in both medico-legal and therapeutic settings.

Inter-rater agreement in evaluation of disability: systematic review of reproducibility studies

OBJECTIVES

 To explore agreement among healthcare professionals assessing eligibility for work disability benefits.

DESIGN

 Systematic review and narrative synthesis of reproducibility studies.

DATA SOURCES

 Medline, Embase, and PsycINFO searched up to 16 March 2016, without language restrictions, and review of bibliographies of included studies.

ELIGIBILITY CRITERIA

 Observational studies investigating reproducibility among healthcare professionals performing disability evaluations using a global rating of working capacity and reporting inter-rater reliability by a statistical measure or descriptively. Studies could be conducted in insurance settings, where decisions on ability to work include normative judgments based on legal considerations, or in research settings, where decisions on ability to work disregard normative considerations. : Teams of paired reviewers identified eligible studies, appraised their methodological quality and generalisability, and abstracted results with pretested forms. As heterogeneity of research designs and findings impeded a quantitative analysis, a descriptive synthesis stratified by setting (insurance or research) was performed.

RESULTS

 From 4562 references, 101 full text articles were reviewed. Of these, 16 studies conducted in an insurance setting and seven in a research setting, performed in 12 countries, met the inclusion criteria. Studies in the insurance setting were conducted with medical experts assessing claimants who were actual disability claimants or played by actors, hypothetical cases, or short written scenarios. Conditions were mental (n=6, 38%), musculoskeletal (n=4, 25%), or mixed (n=6, 38%). Applicability of findings from studies conducted in an insurance setting to real life evaluations ranged from generalisable (n=7, 44%) and probably generalisable (n=3, 19%) to probably not generalisable (n=6, 37%). Median inter-rater reliability among experts was 0.45 (range intraclass correlation coefficient 0.86 to κ-0.10). Inter-rater reliability was poor in six studies (37%) and excellent in only two (13%). This contrasts with studies conducted in the research setting, where the median inter-rater reliability was 0.76 (range 0.91-0.53), and 71% (5/7) studies achieved excellent inter-rater reliability. Reliability between assessing professionals was higher when the evaluation was guided by a standardised instrument (23 studies, P=0.006). No such association was detected for subjective or chronic health conditions or the studies' generalisability to real world evaluation of disability (P=0.46, 0.45, and 0.65, respectively).

CONCLUSIONS

 Despite their common use and far reaching consequences for workers claiming disabling injury or illness, research on the reliability of medical evaluations of disability for work is limited and indicates high variation in judgments among assessing professionals. Standardising the evaluation process could improve reliability. Development and testing of instruments and structured approaches to improve reliability in evaluation of disability are urgently needed.