[Quality assurance in sociomedical evaluation]

Berentung wegen voller Erwerbsminderung: Erfüllen psychiatrische Gutachten die Qualitätskriterien für die sozialmedizinische Begutachtung?

Ziel Die Qualität von sozialmedizinischen Gutachten, die im Rahmen von Anträgen auf Erwerbsminderungsrente wegen psychischer Störung für die Deutsche Rentenversicherung (DRV) angefertigt werden, ist bislang kaum untersucht worden. In der vorliegenden Studie sollte untersucht werden, inwieweit fachärztliche Gutachten zur Posttraumatischen Belastungsstörung (PTBS) der Leitlinie der DRV zu sozialmedizinischen Gutachten bei psychischen und Verhaltensstörungen entsprechen.

Methodik Zur Überprüfung der Gutachtenqualität wurden 52 sozialmedizinische Gutachten, in denen die Diagnose einer Posttraumatischen Belastungsstörung (PTBS) vergeben und ein Leistungsvermögen von unter drei Stunden festgestellt wurde, analysiert. Die Gutachtenqualität wurde sowohl quantitativ mit der Berechnung von Qualitätspunkten, als auch qualitativ durch inhaltliche Analysen erfasst. Zur Überprüfung der Beurteiler_innenübereinstimmung wurde die prozentuale Übereinstimmung und Cohens Kappa für zwei Rater_innen anhand einer Teilstichprobe (n=11) berechnet.

Ergebnisse Die analysierten Gutachten erreichten durchschnittlich 36,1 (SD=7,4) von 92 möglichen Qualitätspunkten (Range 22/56). In den meisten Gutachten waren die Beschreibungen der Teilhabeeinschränkungen, der Epikrise und der sozialmedizinischen Schlussbildung unbefriedigend.

Schlussfolgerung  Die unzureichende Darstellung der Funktions- und Teilhabeeinschränkungen in den Gutachten wird möglicherweise dem tatsächlichen Sachverhalt nicht gerecht, so dass der Beweis einer Gesundheitsstörung mit den entsprechenden Leistungseinschränkungen nicht zweifelsfrei erbracht werden kann. Dementsprechend sind Maßnahmen zur Verbesserung der Gutachtenqualität wie Schulungen und regelmäßige Qualitätsüberprüfungen notwendig.

An instrument for quality assurance in work capacity evaluation: development, evaluation, and inter-rater reliability

Background

Employees insured in pension insurance, who are incapable of working due to ill health, are entitled to a disability pension. To assess whether an individual meets the medical requirements to be considered as disabled, a work capacity evaluation is conducted. However, there are no official guidelines on how to perform an external quality assurance for this evaluation process. Furthermore, the quality of medical reports in the field of insurance medicine can vary substantially, and systematic evaluations are scarce. Reliability studies using peer review have repeatedly shown insufficient ability to distinguish between high, moderate and low quality. Considering literature recommendations, we developed an instrument to examine the quality of medical experts’ reports.

Methods

The peer review manual developed contains six quality domains (formal structure, clarity, transparency, completeness, medical-scientific principles, and efficiency) comprising 22 items. In addition, a superordinate criterion (survey confirmability) rank the overall quality and usefulness of a report. This criterion evaluates problems of inner logic and reasoning. Development of the manual was assisted by experienced physicians in a pre-test. We examined the observable variance in peer judgements and reliability as the most important outcome criteria. To evaluate inter-rater reliability, 20 anonymous experts’ reports detailing the work capacity evaluation were reviewed by 19 trained raters (peers). Percentage agreement and Kendall’s W, a reliability measure of concordance between two or more peers, were calculated. A total of 325 reviews were conducted.

Results

Agreement of peer judgements with respect to the superordinate criterion ranged from 29.2 to 87.5%. Kendall’s W for the quality domain items varied greatly, ranging from 0.09 to 0.88. With respect to the superordinate criterion, Kendall’s W was 0.39, which indicates fair agreement. The results of the percentage agreement revealed systemic peer preferences for certain deficit scale categories.

Conclusion

The superordinate criterion was not sufficiently reliable. However, in comparison to other reliability studies, this criterion showed an equivalent reliability value. This report aims to encourage further efforts to improve evaluation instruments. To reduce disagreement between peer judgments, we propose the revision of the peer review instrument and the development and implementation of a standardized rater training to improve reliability.

Electronic supplementary material

The online version of this article (10.1186/s12913-019-4387-4) contains supplementary material, which is available to authorized users.

Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation

BACKGROUND

The German quality assurance programme for evaluating work capacity is based on peer review that evaluates the quality of medical experts' reports. Low reliability is thought to be due to systematic differences among peers. For this purpose, we developed a curriculum for a standardized peer-training (SPT). This study investigates, whether the SPT increases the inter-rater reliability of social medical physicians participating in a cross-institutional peer review.

METHODS

Forty physicians from 16 regional German Pension Insurances were subjected to SPT. The three-day training course consist of nine educational objectives recorded in a training manual. The SPT is split into a basic module providing basic information about the peer review and an advanced module for small groups of up to 12 peers training peer review using medical reports. Feasibility was tested by assessing selection, comprehensibility and subjective use of contents delivered, the trainers' delivery and design of training materials. The effectiveness of SPT was determined by evaluating peer concordance using three anonymised medical reports assessed by each peer. Percentage agreement and Fleiss' kappa (κ_m) were calculated. Concordance was compared with review results from a previous unstructured, non-standardized peer-training programme (control condition) performed by 19 peers from 12 German Pension Insurances departments. The control condition focused exclusively on the application of peer review in small groups. No specifically training materials, methods and trainer instructions were used.

RESULTS

Peer-training was shown to be feasible. The level of subjective confidence in handling the peer review instrument varied between 70 and 90%. Average percentage agreement for the main outcome criterion was 60.2%, resulting in a κ_m of 0.39. By comparison, the average percentage concordance was 40.2% and the κ_m was 0.12 for the control condition.

CONCLUSION

Concordance with the main criterion was relevant but not significant (p = 0.2) higher for SPT than for the control condition. Fleiss' kappa coefficient showed that peer concordance was higher for SPT than randomly expected. Nevertheless, a score of 0.39 for the main criterion indicated only fair inter-rater reliability, considerably lower than the conventional standard of 0.7 for adequate reliability.

Problems with sickness certification tasks: experiences from physicians in different clinical settings. A cross-sectional nationwide study in Sweden

BACKGROUND

Many physicians find sickness certification of patients problematic. The aims were to explore problems that physicians in different clinical settings experience with sickness certification tasks in general and with assessment of function, work capacity, and need for sick leave, as well as handling of sick-leave spells of different durations.

METHODS

Data from a questionnaire sent to 33,144 physicians aged <68 years, living and working in Sweden in 2012 were analysed. The response rate was 57.6%. The study group comprised the 12,933 responders who had sickness certification tasks. Frequencies and odds ratios with 95% confidence intervals were calculated for questions concerning how problematic the physicians experienced different assessments related to patients' function, work capacity, and need for sick leave, as well as handling sick-leave spells of different durations.

RESULTS

There were large differences between clinical settings regarding how often and to what extent sickness certification consultations were perceived as problematic. Physicians working in primary health care (PHC) had the highest proportions experiencing sickness certification consultations as problematic at least once a week (49.5%) and as very or fairly problematic (56.6%), followed by physicians working in psychiatry, pain management, or orthopaedics. More than half of the responders found it very or fairly problematic to assess patients' work capacity (57.8%), to make a long-term prognosis about patients' future work capacity (55.7%), and to handle long-term or very long-term sickness certifications (51.9% and 51.8%). The proportions were highest among physicians working in PHC, rheumatology, neurology, or psychiatry.

CONCLUSIONS

The rates of physicians finding sickness certification task problematic varied much with clinical setting, and were highest among physicians in PHC. More knowledge is needed about the work conditions and prerequisites for optimal handling of sickness certification in different clinical settings.

Interviews for the assessment of long-term incapacity for work: a study on adherence to protocols and principles

Background

Assessments for long-term incapacity for work are performed by Social Insurance Physicians (SIPs) who rely on interviews with claimants as an important part of the process. These interviews are susceptible to bias. In the Netherlands three protocols have been developed to conduct these interviews. These protocols are expert- and practice-based. We studied to what extent these protocols are adhered to by practitioners.

Methods

We compared the protocols with one another and with the ICF and the biopsychosocial approach. The protocols describe semi-structured interviews with comparable but not identical topics. All protocols prescribe that the client's opinion on his capacity for work, and his arguments, need to be determined and assessed. We developed a questionnaire to elicit the adherence SIPs have to the protocols, their underlying principles and topics. We conducted a survey among one hundred fifty-five experienced SIPs in the Netherlands.

Results

Ninety-eight SIPs responded (64%). All respondents used some form of protocol, either one of the published protocols or their own mix. We found no significant relation between training and the use of a particular protocol. Ninety percent use a semi-structured interview. Ninety-five percent recognise having to verify what the claimant says and eighty-three percent feel the need to establish a good relation (p = 0.019). Twelve topics are basically always addressed by over eighty percent of the respondents. The claimant's opinion of being fit for his own work or other work, and his claim of incapacity and his health arguments for that claim, reach a hundred percent. Description of claimants' previous work reaches ninety-nine percent.

Conclusion

Our study shows professional consensus among experienced Dutch SIPs about the principle of assessment on arguments, the principle of conducting a semi-structured interview and the most crucial interview topics. This consensus can be used to further develop a protocol for interviewing in the assessment of incapacity for work in social insurance. Such a protocol can improve the quality of the assessments in terms of transparency and reproducibility, as well as by enabling clients to better prepare themselves for the assessments.