Reliability of the OSCE for Physical and Occupational Therapists

Development of an Objective Structured Clinical Examination Station for Pediatric Occupational Therapy and an Evaluation of Its Quality

IMPORTANCE

The Objective Structured Clinical Examination (OSCE) is a highly valued measure of students' clinical competencies in medical education. However, few studies have reported on the administration of the OSCE in pediatric occupational therapy education.

OBJECTIVE

To describe the development of a pediatric occupational therapy OSCE station to evaluate students' use of a standardized assessment and examine its standard setting, failure rates, and psychometric properties.

DESIGN

Prospective, cross-sectional, observational study design.

SETTING

Three OSCE stations in a university clinical skills center.

PARTICIPANTS

Five experienced occupational therapists, 60 examinees, 44 child standardized patients, 44 chaperones, and 15 examiners.

MEASURES

The sum of the rating scale and the global performance scores were used. The rating scale measured the examinee's clinical competences in administering a standardized assessment. The 5-point global performance score was used to evaluate the examinee's whole performance.

RESULTS

The OCSE station's expert validity was acceptable (item-level content validity index [CVI] = 0.8-1.0; scale-level CVI = 0.98). Passing scores according to the Angoff method (passing score = 14) and the contrasting-groups M-SD method (passing score = 13) were similar. Failure rates were high (61.7%-73.3%). Internal consistency was acceptable (Cronbach's α = .78). No significant examiner effect was found (p = .554), and interexaminer reliability was acceptable (item score = 0.58-1.00; sum of the rating scale score = 0.97; global performance score = 0.79).

CONCLUSIONS AND RELEVANCE

The OSCE station for using a standardized assessment is a reliable and valid measure of students' interpersonal communication skills and assessment skills. What This Article Adds: The OSCE for education in pediatric occupational therapy is both effective and rigorous.

Using a station within an objective structured clinical examination to assess interprofessional competence performance among undergraduate nursing students

AIM/OBJECTIVE

To describe and analyse the use of a station within an OSCE to assess interprofessional competence performance in undergraduate nursing students. The specific objectives were: - To measure the students' level of competence performance in relation to the interprofessional competences Roles and Responsibilities, Communication and Teamwork. - To determine inter-observer concordance in the assessment of the interprofessional competences.

BACKGROUND

Teamwork competencies are key to improving patient safety and avoiding medical errors. Today, healthcare professionals work in interdisciplinary teams. To foster a culture of safety, some of the measures that can be taken at the individual, team and organisational levels include fostering clear communication among team members, knowledge of respective roles and functions, and deepening team functioning through respect and trust in judgement and capabilities. The World Health Organization recommends starting to develop these competencies in university studies, through interprofessional education. There are numerous programmes in universities all over the world, but more research is needed on the assessment of interprofessional education activities, preferably through objective methods. Competency performance can be assessed by an external evaluator, in a simulated environment, with the Objective Structured Clinical Examination, which is widely used in nursing.

DESIGN

Cross-sectional study.

METHODS

63 second-year nursing undergraduate students completed an interprofessional competencies station within an 8-station OSCE. Communication, Roles and Responsibility and Teamwork competences were assessed. The Interprofessional Collaborator Assessment Rubric (ICAR) was used as a model to assess the performance of students. Inter-observer concordance analysis was performed using the kappa coefficient and the concordance rate.

RESULTS

92.1% of students reached a good level in communication competence, 88.9% in roles and responsibility competence, and 55.6% in teamwork competence. The global concordance rate was 83.8%, and the kappa coefficient was 0.67.

CONCLUSIONS

Most students have demonstrated interprofessional competence performance at a good level. However, the inter-observer concordance obtained for some of the items was not as expected. The assessment of interprofessional competencies, as it deals mainly with relational and communicative aspects, requires greater preparation both in terms of the specification of assessment items and in agreement between examiners.

Using children as standardized patients in OSCE in pediatric occupational therapy

BACKGROUND

Objective structured clinical examinations (OSCEs) have been administered in physical and psychiatric occupational therapy (OT) education, but not in pediatric OT education. The objectives were to examine the satisfaction and the influences of OSCE in pediatric OT on all participants.

METHODS

The OSCE contained evaluation, intervention, and parent education stations. Sixty examinees, 44 child standardized patients (SPs), 44 chaperones, three playroom managers, 14 OSCE assistants, and 15 examiners participated in the OSCE. An OSCE video and three playrooms were prepared for child SPs.

RESULTS

Ninety percent of the child SPs liked taking part in the OSCE and 75-85% expressed interest in participating in an OSCE the following year. Their parents appreciated the chaperones accompanying their children and giving them a memorable day. 88.3% of the examinees thought that the OSCE was helpful for their upcoming clinical training. 73.3% preferred the OSCE over the written exam. 60-93.4% considered the implementation appropriate. Most of the examiners thought that the content (80-100%) and the implementation (93.3-100%) were appropriate. Many chaperones reported having valuable experiences.

CONCLUSIONS

It is practical using child SPs in OSCE in pediatric OT. The OSCE was beneficial to all participants. It is recommended that OSCEs be included in pediatric OT education.

Continuing competence assessment and maintenance in occupational therapy: Scoping review with stakeholder consultation

BACKGROUND/AIM

Continuing competence activities hold the promise of improving health-care service quality, especially given concerns about system inefficiencies and fragmentation. The aim of this scoping review was to describe the assessment and maintenance of occupational therapists' continuing competence and identify knowledge gaps for future research. A secondary aim was comparing scoping review findings with those from other health-care fields and considering possible impact of varying international regulations.

METHODS

A series of database searches retrieved peer-reviewed and grey literature on assessment and maintenance of occupational therapy continuing competence from 1995 to 2015. Themes were developed and findings shared with stakeholders, whose comments drove a second phase: searching for reviews related to continuing competence from allied health, medicine, and nursing, and reviewing websites and documents concerning regulatory requirements for occupational therapy continuing competence in seven English-speaking nations.

RESULTS

Twenty-seven sources from the scoping review search met inclusion criteria. Stakeholder consultation validated the themes and preliminary knowledge gaps. Research into other health-care specialties corresponded to findings from the scoping review. The website/document review of occupational therapy regulatory requirements revealed wide variation on both the state/province and national levels.

CONCLUSIONS

This scoping review highlights gaps in research on effective methods and assessment of occupational therapy continuing competence. Findings suggest a need for research on approaches to continuing competence that incorporate the translation of evidence to practice and address the influence of external factors. Regulatory agencies may address the quality of occupational therapy services by incorporating a variety of professional development options into requirements.

Development of clinical competence assessment tool for novice physical and occupational therapists-a mixed Delphi study

[Purpose] The aim of this study was to clarify essential abilities of novice physical and occupational therapists for independent execution of their duties and to develop a clinical competence assessment tool. [Subjects] Forty-five experienced therapists participated in this study. [Methods] A two-phase mixed-methods design was used. First, semi structured interviews were conducted on 15 experienced therapists to create a comprehensive list of essential abilities that novice therapists need. Second, 30 experienced therapists participated in a two-round Delphi study to select items for the assessment tool being developed. [Results] Fifty-five items were extracted and classified into three categories: basic attitudes, therapeutic skills, and clinical practice-related thoughts. [Conclusion] Present results suggest that not only knowledge of execution of therapy-related duties and therapeutic skills is essential in novice therapist, but also appropriate abilities in social adjustment, self-management, and self-education. The newly developed tool might be useful for postgraduate education in clinical practice.

Contents of supervision needed by physical and occupational therapists for ability development: focusing on their clinical experience

[Purpose] This study examined the contents of supervision needed by novice therapists to develop clinical abilities, focusing on their clinical experience and using an original evaluation table. [Subjects and Methods] An evaluation of clinical abilities basic attitudes, therapeutic skills, and clinical practice-related thoughts was conducted in 29, 21, and 9 therapists with clinical experience of 0–1 (1 year group), 1–2 (2 years group), and 2–3 (3 years group) years, respectively. [Results] There were no significant differences among the 3 groups in basic attitudes. Therapeutic skills markedly varied between the 1 and 3 years groups. In clinical practice-related thoughts, significant differences were observed between the 1 and 3 years groups and between the 2 and 3 years groups. [Conclusion] It may be appropriate for educators to provide technical education regarding skills that are achievable for students in the early stages in consideration of applied movements. Also, education for novices should be provided with importance attached to abilities influenced by clinical experience.

Reliability of clinical competency evaluation list for novice physical and occupational therapists requiring assistance

[Purpose] This study examined the reliability of a clinical ability evaluation table developed in a previous study with the aim of clarifying the abilities necessary for therapists to independently implement their duties. [Subjects and Methods] Forty-eight physical therapists with less than 2 years of clinical experience were targeted for evaluation, 48 main supervisors, and 48 sub-supervisors, 144 in total, were studied. [Results] The total score was lower when the evaluation was conducted by the target therapists themselves than when it was conducted by the main or sub-supervisors. Regarding the reliability of the total scores for the entire scale and each category, values representing the intra-rater reliability were higher when the evaluation was conducted by the target therapists or main supervisors, while there were marked differences between high and low values for each item. Regarding the inter-rater reliability, both the total scores for the entire scale and each category, as well as values for each item, were low. [Conclusion] Values representing the intra-rater reliability of the study table were low, indicating the necessity of further improvement.

Effects of clinical practice focusing on level-3 OSCE items

[Purpose] This study examined the effects of clinical training focusing on level-3 OSCE (analytical and therapeutic skills) items, and compared the achievement levels of physical (PT) and occupational (OT) therapist students. [Subjects] A total of 282 (165 PT and 117 OT) students enrolled at our university between 2008 and 2010 were studied. [Methods] OSCE scores were compared between before and after clinical training focusing on level-3 OSCE items, and between PT and OT students. [Results] Scores for 5 out of the 6 level-3a items were significantly higher after than before clinical training. Increases in scores of 2 or 3 level-3b and −3c items were also observed after clinical training. There were no marked differences between PT and OT students in scores for level-3a, −3b, and −3c items before clinical training. In contrast, after clinical training, OT students’ scores for 3a and 3c items related to dressing were higher than those of PT students, and the latter’s scores for 3b items related to transfer were higher than those of the former. [Conclusion] The results suggest level-3 OSCE items are effectively taught during clinical training.

Difficulty levels of OSCE items related to examination and measurement skills

[Purpose] The difficulty levels of level-2 OSCE (examination and measurement skills) items were examined, with a view to providing reference data for the determination of students’ skills. [Subjects] A total of 284 graduates of physical (PT) and occupational (OT) therapy classes of 2011 (59 and 40), 2012 (46 and 36), and 2013 (61 and 42, respectively) were studied, with PT or OT faculty members as OSCE examiners and a simulated patient. [Methods] Scores for 11 level-2 OSCE items were compared between before and after clinical training. [Results] Scores markedly increased after clinical training. On comparison among the items, scores for sensory examination were the highest, and those for interviews were the lowest. [Conclusion] The results of this study indicate the necessity of considering an appropriate combination of different difficulty levels when adopting OSCE-based educational approaches.

OSCE-based Clinical Skill Education for Physical and Occupational Therapists

[Purpose] The aim of this study was to examine the applicability of the Objective Structured Clinical Examination (OSCE) to postgraduate education systems for novice and mid-career therapists in workplaces. [Subjects] Physical and occupational therapists with 1 to 5 years of clinical experience took the OSCE to assess their learning, with a physical or occupational therapy faculty member and a clinical supervisor as examiners. Another clinical supervisor acted as a simulated patient. [Methods] A Wilcoxon signed-rank test was performed to compare skills between before and after OSCE-based learning, and a Mann-Whitney U test was used to compare them between therapists with 1 to 2 years (novice) and 3 to 5 years (mid-career) of clinical experience. [Results] While no experience-related differences were observed in behavioral aspects, mid-career therapists exhibited markedly higher scores compared with novices in technical aspects, such as skills to guide patients for standing up, transfer, and dressing. [Conclusion] The OSCE may be sufficiently applicable to postgraduate education systems in workplaces.