Two-dimensional structure of the MAAS-Global rating list for consultation skills of doctors

Development of an assessment tool to measure communication skills among family medicine residents in the context of electronic medical record use

BACKGROUND

The introduction of the electronic medical record (EMR) has led to new communication skills that need to be taught and assessed. There is scarce literature on validated instruments measuring electronic-specific communication skills. The aim is to develop an assessment checklist that assesses the general and EMR-specific communication skills and evaluates their content validity and reliability.

METHODS

Using the SEGUE theoretical framework for communication skills, the assessment checklist items were developed by the Communication Skills Working Group (CSWG) at the family medicine department using a literature review about the positive and negative aspects of EMR use on physician-patient communication. A group of faculty members rated real resident-patient encounters on two occasions, three weeks apart. Patients were asked to fill out the Communication Assessment Tool (CAT) at the end of the encounter.

RESULTS

A total of 8 residents agreed to participate in the research, with 21 clinical encounters recorded. The average total score was 65.2 ± 6.9 and 48.1 ± 9.5 for the developed scale and the CAT scale, respectively. The scale reliability was good, with a Cronbach alpha of 0.694. The test-retest reliability was 0.873, p < 0.0001. For the total score on the developed checklist, the intraclass correlation coefficient between raters (ICC) was 0.429 [0.030,0.665], p-value of 0.019. The level of agreement between any two raters on the cumulative score of the 5 subsections ranged from 0.506 (interpersonal skills) to 0.969 (end encounter).

CONCLUSION

This checklist is a reliable and valid instrument that combines basic and EMR-related communication skills.

Optimising planned medical education strategies to develop learners' person-centredness: A realist review

CONTEXT

Person-centeredness is a stated aim for medical education; however, studies suggest this is not being achieved. There is a gap in our understanding of how, why and in what circumstances medical education interventions that aim to develop person-centredness are successful.

METHODS

A realist review was conducted with a search of Medline, Embase, HMIC and ERIC databases and the grey literature using the terms 'medical education' and 'person-centred' and related synonyms. Studies that involved a planned educational intervention in medical education with data on outcomes related to person-centredness were included. The analysis focused on how and why different educational strategies interact with biomedical learner perspectives to trigger mechanisms that may or may not lead to a change in perspective towards person-centredness.

RESULTS

Sixty-one papers representing fifty-three interventions were included in the final synthesis. Nine context-intervention-mechanism-outcome configuration (CIMOc) statements generated from the data synthesis make up our refined programme theory. Where educational interventions focused on communication skills learning or experiences without person-centred theory, learners experienced dissonance with their biomedical perspective which they resolved by minimising the importance of the learning, resulting in perspective endurance. Where educational interventions applied person-centred theory to meaningful experiences and included support for sense making, learners understood the relevance of person-centeredness and felt able to process their responses to learning, resulting in perspective transformation towards person-centredness.

CONCLUSION

Our findings offer explanations as to why communication skills-based interventions may be insufficient to develop learners' person-centredness. Integrating experiential person-centred learning with theory on why person-centredness matters to clinical practice and enabling learners to make sense of their responses to learning, may support perspective transformation towards person-centredness. Our findings offer programme and policymakers testable theory to inform the development of medical education strategies that aim to support person-centredness.

Assessing communication skills in student pharmacists-Psychometric validation of Global Communication Rubric

ASSESSING COMMUNICATION SKILLS IN STUDENT PHARMACISTS

Psychometric validation of the Global Communication Rubric.

OBJECTIVE

The objective of this study was to perform psychometric validation of the Global Communication (GC) rubric.

METHODS

The GC rubric was developed and used for the past 10 years at the institution to assess level of communication skills in Objective Structured Clinical Examinations (OSCE) in the PharmD program. The rubric consisted of 6 questions with the scoring criteria of 0 (failure), 1 (needs improvement), 2 (satisfactory), and 3 (excellent). Data from GC rubric scores for Classes 2010 to 2020 (1,100 students) was used to perform psychometric validation by assessing the following properties: scale reliability; face, content, construct, and predictive validity; and responsiveness.

RESULTS

Internal consistency reliability was acceptable at Cronbach's alpha of 0.78. Construct validity showed that all six items loaded highly onto one component. Responsiveness was demonstrated using uptrend of the score improvement as the students advance in the curriculum from P1 to P3 levels.

CONCLUSION

The result of this study confirmed reliability and validity of the GC rubric in evaluating communication skills of student pharmacists.

PRACTICE IMPLICATIONS

The GC rubric is valuable in evaluating short encounters in an easy and expedited manner.

Assessment of motivational interviewing: Psychometric characteristics of the MITS 2.1 in general practice

OBJECTIVE

Motivational Interviewing (MI) is increasingly used in healthcare. The Motivational Interviewing Target Scheme 2.1 (MITS) can be used to assess MI in short consultations. This quantitative validation study is a sequel to a qualitative study, which showed that the MITS is suitable for low-stakes assessment. We collected validity evidence to determine whether its suitability for high-stakes assessment in the GP-setting.

METHODS

Consultations of GPs and GP-trainees were assessed using the MITS. The internal structure was studied using generalizability theory; intra class correlation (ICC), convergent and divergent validity was determined.

RESULTS

Two coders and seven consultations were found to be necessary for high stakes assessment. We found higher ICCs as coders were more experienced. Convergent validity was found; results for divergent validity were mixed.

CONCLUSION

The MITS is a suitable instrument for high-stakes MI assessments in GP-setting. The number of consultations and coders that are needed for assessment are comparable to other instruments for assessing communication skills.

PRACTICE IMPLICATIONS

The MITS can be used to assess conversations for their MI consistency in GP-setting where most consultations are relatively short and are only partially dedicated to behaviour change. As the MITS assesses complex communication skills, experienced coders are needed.

Development of a self-assessment tool for resident doctors’ communication skills in India

PURPOSE

Effective communication skills are essential for resident doctors to provide optimum patient care. This study was conducted to develop and validate a questionnaire for the self-assessment of resident doctors’ communication skills in India.

METHODS

This was a mixed-methods study conducted in 2 phases. The first phase consisted of questionnaire development, including the identification of relevant literature, focus group discussions with residents and experts from clinical specialties, and pre-testing of the questionnaire. The second phase involved administering the questionnaire survey to 95 residents from the Departments of Medicine, Emergency Medicine, Pediatrics, and Surgery at the All India Institute of Medical Sciences, New Delhi, India in April 2019. Internal consistency was tested and the factor structure was analyzed to test construct validity.

RESULTS

The questionnaire consisted of 3 sections: (A) 4 items on doctor-patient conflicts and the role of communication skills in avoiding these conflicts, (B) 29 items on self-assessment of communication skills in different settings, and (C) 8 items on barriers to practicing good communication skills. Sections B and C had good internal consistency (Cronbach α: 0.885 and 0.771, respectively). Section C had a 2-factor solution, and the barriers were classified as ‘training’ and ‘infrastructure’ factors.

CONCLUSION

This appears to be a valid assessment tool of resident doctors’ communication skills, with potential utility for identifying gaps in communication skills and developing communication skills modules.

Training residents in patient-centred communication and empathy: evaluation from patients, observers and residents

BACKGROUND

Patient-centred communication and empathy are key enablers for patient-centred care. However, several studies suggest a downward trend regarding the empathic communication skills of physicians during medical residency. It is known that communication training can have a positive effect on patient-centred communication, empathy and relational skills. Training residents in patient-centred communication and empathy can be an opportunity to improve the patient-centred care. To evaluate the training a tri-focal perspective will be used.

METHODS

A 3-day training was developed to improve residents' patient-centred communication and empathy skills at an academic medical health centre, in the Netherlands. The training included: (1) the basics of patient-centred communication and empathy (through presentations, scientific literature), (2) practicing with actors, and (3) reflecting on residents' video recorded consultations (by themselves and communication experts). A pilot study with a pre-post design was conducted to evaluate the training from patient and observer perspectives. Semi-structured interviews were used to get insight into residents' perspective. Nine residents from different specialities followed the training and enrolled in the pilot study. During two random days consultations between residents and patients were video recorded. Patients were asked to fill in two questionnaires, indicating their perspective on residents' empathy and communication skills before as well as after the consultation. All video recorded consultations were coded to rate residents' communication skills, empathy, computer use and agenda-setting. Statistical analysis were performed using multilevel analysis.

RESULTS

A total of 137 eligible patients took part in the pilot study. Trained residents showed significant improvement in patient-rated empathy scores. According to observers, residents' computer use improved significantly after the training. The communication skills of trained residents did not improve significantly. Agenda setting by residents showed a downward trend. Almost all residents were satisfied with the training, especially with the video-feedback.

CONCLUSIONS

A brief training significantly increased residents' empathy scores according to patients and significantly decreased residents' computer use according to observers. These findings indicate that the quality of patient-centred care can be improved by integrating patient-centred communication into residency programs, at an academic medical health centre. The ultimate goal is to structurally embed the training in residents' education program.

True communication skills assessment in interdepartmental OSCE stations: Standard setting using the MAAS-Global and EduG

BACKGROUND

Comparing outcome of clinical skills assessment is challenging. This study proposes reliable and valid comparison of communication skills (1) assessment as practiced in Objective Structured Clinical Examinations (2). The aim of the present study is to compare CS assessment, as standardized according to the MAAS Global, between stations in a single undergraduate medical year.

METHODS

An OSCE delivered in an Irish undergraduate curriculum was studied. We chose the MAAS-Global as an internationally recognized and validated instrument to calibrate the OSCE station items. The MAAS-Global proportion is the percentage of station checklist items that can be considered as 'true' CS. The reliability of the OSCE was calculated with G-Theory analysis and nested ANOVA was used to compare mean scores of all years.

RESULTS

MAAS-Global scores in psychiatry stations were significantly higher than those in other disciplines (p<0.03) and above the initial pass mark of 50%. The higher students' scores in psychiatry stations were related to higher MAAS-Global proportions when compared to the general practice stations.

CONCLUSION

Comparison of outcome measurements, using the MAAS Global as a standardization instrument, between interdisciplinary station checklists was valid and reliable.

PRACTICE IMPLICATIONS

The MAAS-Global was used as a single validated instrument and is suggested as gold standard.

Assessment of motivational interviewing: a qualitative study of response process validity, content validity and feasibility of the motivational interviewing target scheme (MITS) in general practice

BACKGROUND

The Motivational Interviewing target Scheme (MITS) is an instrument to assess competency in Motivational Interviewing (MI) and can be used to assess MI in long and brief consultations. In this qualitative study we examined two sources of the Unified Model of Validity, the current standard of assessment validation, in the context of General Practice. We collected evidence concerning response process validity and content validity of the MITS in general practice. Furthermore, we investigated its feasibility.

METHODS

Assessing consultations of General Practitioners and GP-trainees (GPs), the assessors systematically noted down their considerations concerning the scoring process and the content of the MITS in a semi-structured questionnaire. Sampling of the GPs was based on maximum variation and data saturation was used as a stopping criterion. An inductive approach was used to analyse the data. In response to scoring problems the score options were adjusted and all consultations were assessed using the original and the adjusted score options.

RESULTS

Twenty seven assessments were needed to reach data saturation. In most consultations, the health behaviour was not the reason for encounter but was discussed on top of discussing physical problems. The topic that was most discussed in the consultations was smoking cigarettes. The adjusted score options increased the response process validity; they were more in agreement with theoretical constructs and the observed quality of MI in the consultations. Concerning content validity, we found that the MITS represents the broad spectrum and the current understanding of MI. Furthermore, the MITS proved to be feasible to assess MI in brief consultations in general practice.

CONCLUSIONS

Based on the collected evidence the MITS seems a promising instrument to measure MI interviewing in brief consultations. The evidence gathered in this study lays the foundation for research into other aspects of validation.

The effect on reliability and sensitivity to level of training of combining analytic and holistic rating scales for assessing communication skills in an internal medicine resident OSCE

OBJECTIVE

Although previous research has compared checklists to rating scales for assessing communication, the purpose of this study was to compare the effect on reliability and sensitivity to level of training of an analytic, a holistic, and a combined analytic-holistic rating scale in assessing communication skills.

METHODS

The University of Alberta Internal Medicine Residency runs OSCEs for postgraduate year (PGY) 1 and 2 residents and another for PGY-4 residents. Communication stations were scored with an analytic scale (empathy, non-verbal skills, verbal skills, and coherence subscales) and a holistic scale. Authors analyzed reliability of individual and combined scales using generalizability theory and evaluated each scale's sensitivity to level of training.

RESULTS

For analytic, holistic, and combined scales, 12, 12, and 11 stations respectively yielded a Phi of 0.8 for the PGY-1,2 cohort, and 16, 16, and 14 stations yielded a Phi of 0.8 for the PGY-4 cohort. PGY-4 residents scored higher on the combined scale, the analytic rating scale, and the non-verbal and coherence subscales.

CONCLUSION

A combined analytic-holistic rating scale increased score reliability and was sensitive to level of training.

PRACTICE IMPLICATIONS

Given increased validity evidence, OSCE developers should consider combining analytic and holistic scales when assessing communication skills.

[Consultation skills training as an element of general practice training in Germany - a qualitative survey]

INTRODUCTION

The consultation is at the heart of general practice. It is the central setting through which primary care is delivered. The competency requirements are laid down internationally by competency-based curricula for undergraduate and postgraduate education. So far, there is no competency-based vocational training to develop consultation skills in general practice in Germany. The study describes experiences with consultation skills training as an element of general practice training as reported by trainees and trainers in Germany.

METHODS

A qualitative and exploring approach was chosen because there is little experience with the German situation. We conducted structured focus group interviews with trainees and trainers, respectively. We recruited all participants by e-mail via the mail distributor "Junge Allgemeinmedizin Deutschland" (JADE, a trainee and junior GP organization) and the academic teachers of the Friedrich-Alexander University Erlangen-Nürnberg. Altogether, four focus group interviews with three to five participants were conducted, varying in length from 25 to 65minutes. All interviews were recorded digitally and transcribed verbatim. Then a qualitative content analysis was performed.

RESULTS

The statements of the ten trainees and five trainers mapped a system of four main categories: (a) association with the term consultation, (b) parts of a consultation, (c) competencies required for professional practice, (d) consultation skills training as an element of vocational training. Overall, all participants regarded the consultation as the most important element in general practice. Important content of consultations is to build a relationship with the patient, gather information, conduct physical examinations and achieve informed consent on further proceedings. All participants agreed that physicians need different sets of competencies: medical expertise, communication skills, examination skills and professionalism. Finally, there was a broad consensus that a competency-based general practice training to support the development of consultation skills is lacking in Germany. The majority acknowledged the need for change.

CONCLUSION

So far, the consultation skills development within the general practice training in Germany is regarded as deficient. Both trainees and trainers have stressed the importance of change in vocational training. With the new competency-based curriculum for general practice in Germany and the associated development of supporting tools important prerequisites have been provided.

[Validation of a questionnaire for standardized-patient assessment of clinical skills]

BACKGROUND

This study was designed in the context of a new educational program based on standardized-patients (SPs). The study objectives were (1) to evaluate the reliability of the assessment form used by SPs and (2) to compare global ratings to checklists reproducibility.

METHOD

History taking, physical examination and communication skills were assessed by SPs at the end of clinical encounters with year 3 medical students, using generic global rating scales and case-specific checklists. The validation process followed four steps: (1) correlation analysis between global rating and their relative checklist of specific items, (2) internal consistency estimation, (3) validation of the questionnaire dimensions, (4) estimation of the reliability of SPs' ratings compared to medical teachers' ratings.

RESULTS

A total of 3322 consultations have been performed by 444 year 3 medical students. Statistical analysis showed a good internal reliability (Cronbach α was greater than 0.7) and an acceptable inter-judge reproducibility except for communication skills. Case-specific checklists did not prove to be more reliable than global ratings. Reproducibility was lower with SPs' than with medical teachers' assessment.

CONCLUSION

Global rating-based assessment should be preferred to checklists because they are faster and easier to use (shorter duration of SPs training). As SPs proved to be acceptable examiners, no third person seems to be required as external observer.

Health information exchange for patients with intellectual disabilities: a general practice perspective

BACKGROUND

Inadequate health information exchange (HIE) between patients with intellectual disabilities (ID), their carers, and GPs may lead to ineffective treatment and poor treatment compliance. Factors influencing HIE are largely unexplored in previous research.

AIM

To provide insight into the perceived HIE facilitators of GPs and general practice assistants, and the barriers in GP consultations for patients with ID.

DESIGN AND SETTING

An interview-based study with GPs (n = 19) and general practice assistants (n = 11) in the Netherlands.

METHOD

Semi-structured interviews were conducted on topics relating to stages during and around GP consultation. Transcripts were coded and analysed using framework analysis.

RESULTS

The main themes were impaired medical history taking and clinical decision making, and fragile patient follow-up. Factors negatively influencing HIE related to patient communication skills and professional carers' actions in preparing the consultation and in collecting, recording, and sharing information. HIE barriers resulted in risk of delay in diagnosis and treatment, misdiagnosis, unnecessary tests, and ineffective treatment regimens. HIE facilitators were described in terms of GP adjustments in communication, planning of consultations, and efforts to compensate for fragile follow-up situations.

CONCLUSION

Inadequate HIE should be seen as a chain of events leading to less effective consultations, substandard treatment, and insufficient patient follow-up. The results indicate a mismatch between GPs' expectations about professional carers' competencies, responsibilities, and roles in HIE and the setting in which professional carers operate. Further research should focus on how daily GP practice can be attuned to the practicalities of HIE with patients with ID and their professional carers.

Effects of communication training with the MAAS-Global-D instrument on the antibiotic prescribing for respiratory infections in primary care: study protocol of a randomised controlled trial

BACKGROUND

Primary care physicians account for the majority of antibiotic prescribing in ambulatory care in Germany. Respiratory diseases are, regardless of effectiveness, often treated with antibiotics. Research has found this use without indication to be caused largely by communication problems (e.g. expectations on the patient's part or false assumptions about them by the physician). The present randomised controlled trial (RCT) study evaluates whether communication training for primary care physicians can reduce the antibiotic prescribing rate for respiratory tract infections.

METHODS/DESIGN

The study consists of three groups: group A will receive communication training; group B will be given the same, plus additional, access to an evidence-based point-of-care tool; and group C will function as the control group. The primary endpoint is the difference between intervention and control groups regarding the antibiotic prescribing rate before and after the intervention assessed through routine data. The communication skills are captured with the help of the communication instrument MAAS-Global-D, as well as individual videos of physician-patient consultations recorded by the primary care physicians. These skills will also be regarded with respect to the antibiotic prescribing rate. A process evaluation using qualitative as well as quantitative methods should provide information about barriers and enablers to implementing the communication training.

DISCUSSION

The trial contributes to an insight into the effectiveness of the different components to reduce antibiotic prescribing, which will also be supported by an extensive evaluation. Communication training could be an effective method of reducing antibiotic prescribing in primary care.

TRIAL REGISTRATION

DRKS00009566 DATE REGISTRATION: 5 November 2015.

Calibration of communication skills items in OSCE checklists according to the MAAS-Global

BACKGROUND

Communication skills (CS) are commonly assessed using 'communication items' in Objective Structured Clinical Examination (OSCE) station checklists. Our aim is to calibrate the communication component of OSCE station checklists according to the MAAS-Global which is a valid and reliable standard to assess CS in undergraduate medical education.

METHOD

Three raters independently compared 280 checklists from 4 disciplines contributing to the undergraduate year 4 OSCE against the 17 items of the MAAS-Global standard. G-theory was used to analyze the reliability of this calibration procedure.

RESULTS

G-Kappa was 0.8. For two raters G-Kappa is 0.72 and it fell to 0.57 for one rater. 46% of the checklist items corresponded to section three of the MAAS-Global (i.e. medical content of the consultation), whilst 12% corresponded to section two (i.e. general CS), and 8.2% to section one (i.e. CS for each separate phase of the consultation). 34% of the items were not considered to be CS.

CONCLUSION

A G-Kappa of 0.8 confirms a reliable and valid procedure for calibrating OSCE CS checklist items using the MAAS-Global. We strongly suggest that such a procedure is more widely employed to arrive at a stable (valid and reliable) judgment of the communication component in existing checklists for medical students' communication behaviours.

PRACTICE IMPLICATIONS

It is possible to measure the 'true' caliber of CS in OSCE stations. Students' results are thereby comparable between and across stations, students and institutions. A reliable calibration procedure requires only two raters.

Reliability and validity of OSCE checklists used to assess the communication skills of undergraduate medical students: A systematic review

OBJECTIVES

To explore inter-rater agreement between reviewers comparing reliability and validity of checklist forms that claim to assess the communication skills of undergraduate medical students in Objective Structured Clinical Examinations (OSCEs).

METHODS

Papers explaining rubrics of OSCE checklist forms were identified from Pubmed, Embase, PsycINFO, and the ProQuest Education Databases up to 2013. Included were those studies that report empirical validity or reliability values for the communication skills assessment checklists used. Excluded were those papers that did not report reliability or validity.

RESULTS

Papers focusing on generic communication skills, history taking, physician-patient communication, interviewing, negotiating treatment, information giving, empathy and 18 other domains (ICC -0.12-1) were identified. Regarding the validity and reliability of the communication skills checklists, agreement between reviewers was 0.45.

CONCLUSIONS

Heterogeneity in the rubrics used in the assessment of communication skills and a lack of agreement between reviewers makes comparison of student competences within and across institutions difficult.

PRACTICE IMPLICATIONS

Consideration should be afforded to the adoption of a standardized measurement instrument to assess communication skills in undergraduate medical education. Future research will focus upon evaluating the potential impact of adoption of a standardized measurement instrument.

How do trained raters take context factors into account when assessing GP trainee communication performance? An exploratory, qualitative study

Communication assessment in real-life consultations is a complex task. Generic assessment instruments help but may also have disadvantages. The generic nature of the skills being assessed does not provide indications for context-specific behaviour required in practice situations; context influences are mostly taken into account implicitly. Our research questions are: 1. What factors do trained raters observe when rating workplace communication? 2. How do they take context factors into account when rating communication performance with a generic rating instrument? Nineteen general practitioners (GPs), trained in communication assessment with a generic rating instrument (the MAAS-Global), participated in a think-aloud protocol reflecting concurrent thought processes while assessing videotaped real-life consultations. They were subsequently interviewed to answer questions explicitly asking them to comment on the influence of predefined contextual factors on the assessment process. Results from both data sources were analysed. We used a grounded theory approach to untangle the influence of context factors on GP communication and on communication assessment. Both from the think-aloud procedure and from the interviews we identified various context factors influencing communication, which were categorised into doctor-related (17), patient-related (13), consultation-related (18), and education-related factors (18). Participants had different views and practices on how to incorporate context factors into the GP(-trainee) communication assessment. Raters acknowledge that context factors may affect communication in GP consultations, but struggle with how to take contextual influences into account when assessing communication performance in an educational context. To assess practice situations, raters need extra guidance on how to handle specific contextual factors.

Differential growth in doctor-patient communications skills

CONTEXT

Although doctor-patient communication is considered a core competency for medical doctors, the effect of training has not been unequivocally established. Moreover, knowledge about the variance in the growth of different skills and whether certain patterns in growth can be detected could help us to develop more efficient programmes. We therefore investigated the growth in general practitioner (GP) trainees' doctor-patient communication skills in their first year and whether the growth was different for distinct categories of skills.

METHODS

Seventy-one first-year GP trainees were invited to participate in a study aimed at measuring their consultation skills at the beginning (baseline) and at the end of their first year (follow-up). Consultation skills were assessed with the MAAS-Global rating list for consultation skills.

RESULTS

Data on 29 general practitioner trainees were collected. MAAS-Global scores showed a significant growth on all items but one. Patient-oriented skills showed significantly more growth than task-oriented skills. Empathy as a separate skill seems to be mastered predominantly before the start of training.

CONCLUSIONS

Three patterns in the growth in skills were distinguished: (i) low baseline, relatively high follow-up, (ii) moderate baseline, moderate growth and (iii) high baseline, hardly any growth. Patient-oriented skills follow either pattern (i) or (iii), whereas task-oriented skills follow pattern (ii). These findings may help to define where the focus should lie in the training of doctor-patient communication skills.