Development of a tool to measure clinical competence in occupational therapy: a pilot study?

Assessing and Building Clinical Competence in Occupational Therapists Treating Patients with Neurodegenerative Disease: A Community of Practice Study

Communities of practice (CoPs) can be an effective means by which to efficiently build skills, knowledge, and competence for occupational therapists. The objective of this study was to assess changes in clinical competence for occupational therapists treating patients with neurodegenerative diseases (NDD) after participating in a CoP. A cohort of home health occupational therapists was recruited to participate in a seven-week CoP focused on treating patients with NDD. A single group pretest posttest mixed methods design was used to measure changes in clinical competence of the participants through a validated self-report assessment tool and a qualitative analysis of treatment summaries. Thirteen occupational therapists participated in the study. There were significant changes in pretest and post-test knowledge, beliefs, and action scores, indicating a positive change in the participants' competence to treat patients with NDD. Qualitative findings support positive changes in clinical competence through increased knowledge, confidence, and use of interventions aimed at optimizing occupational performance for this population. In summary, the CoP was an effective method for building clinical competence for treating patients with NDD with this cohort of occupational therapists.

Education leads to increased adherence to the 'Australian Competency Standards for Occupational Therapy Driver Assessors'

BACKGROUND

Occupational therapy competency standards provide an evidence-base to inform clinical best practice, however it is not known whether education about competency standards will increase occupational therapists' adherence to their use.

AIMS/OBJECTIVES

To investigate if education about the 'Australian Competency Standards for Occupational Therapy Driver Assessors' leads to increased adherence to the competency standards in the clinical practice of occupational therapy driver assessors.

MATERIALS AND METHODS

A mixed methods multiple case study design was used to evaluate 5 occupational therapy driver assessors' adherence to the competency standards. An audit of 25 client files and interviews were conducted to evaluate practice against the competency standards prior to an education session, followed by an audit of a further 25 files, interview and feedback after education.

RESULTS

Qualitative and quantitative analyses suggest that education about the 'Australian Competency Standards for Occupational Therapy Driver Assessors' was associated with increased adherence to the competency standards in clinical practice. The results also support the use of competency standards in clinical practice.

CONCLUSIONS AND SIGNIFICANCE

Competency standards can be used to inform and guide clinical practice, and individualised education and feedback of practice against the competency standards can increase occupational therapists' adherence to these standards.

Does training in a top-down approach influence recorded goals and treatment plans?

BACKGROUND.

Five stroke rehabilitation teams were supported in the implementation of Cognitive Orientation to daily Occupational Performance (CO-OP) as part of a knowledge translation (KT) project called CO-OP KT. Medical record auditors noted there was occasionally a disconnect between client goals and treatment plans, revealing a need to better understand the characteristics of each and their relationship to each other.

PURPOSE.

This study aimed to examine the characteristics of goals and treatment plans in occupational therapy before and after CO-OP KT.

METHOD.

A descriptive secondary analysis of medical record data was employed.

FINDINGS.

Post intervention, there was a change in goal specificity (p = .04) and therapist-client goal alignment (p = .05). Occupation-based goals were often paired with a bottom-up, impairment-based treatment. Top-down treatments, when present, lacked the same detail given to bottom-up plans.

IMPLICATIONS.

CO-OP KT seemed to lead to more specific goals, but matching top-down treatment plans were not found.

Continuing competence assessment and maintenance in occupational therapy: Scoping review with stakeholder consultation

BACKGROUND/AIM

Continuing competence activities hold the promise of improving health-care service quality, especially given concerns about system inefficiencies and fragmentation. The aim of this scoping review was to describe the assessment and maintenance of occupational therapists' continuing competence and identify knowledge gaps for future research. A secondary aim was comparing scoping review findings with those from other health-care fields and considering possible impact of varying international regulations.

METHODS

A series of database searches retrieved peer-reviewed and grey literature on assessment and maintenance of occupational therapy continuing competence from 1995 to 2015. Themes were developed and findings shared with stakeholders, whose comments drove a second phase: searching for reviews related to continuing competence from allied health, medicine, and nursing, and reviewing websites and documents concerning regulatory requirements for occupational therapy continuing competence in seven English-speaking nations.

RESULTS

Twenty-seven sources from the scoping review search met inclusion criteria. Stakeholder consultation validated the themes and preliminary knowledge gaps. Research into other health-care specialties corresponded to findings from the scoping review. The website/document review of occupational therapy regulatory requirements revealed wide variation on both the state/province and national levels.

CONCLUSIONS

This scoping review highlights gaps in research on effective methods and assessment of occupational therapy continuing competence. Findings suggest a need for research on approaches to continuing competence that incorporate the translation of evidence to practice and address the influence of external factors. Regulatory agencies may address the quality of occupational therapy services by incorporating a variety of professional development options into requirements.

Promoting Therapists' Use of Motor Learning Strategies within Virtual Reality-Based Stroke Rehabilitation

PURPOSE

Therapists use motor learning strategies (MLSs) to structure practice conditions within stroke rehabilitation. Virtual reality (VR)-based rehabilitation is an MLS-oriented stroke intervention, yet little support exists to assist therapists in integrating MLSs with VR system use.

METHOD

A pre-post design evaluated a knowledge translation (KT) intervention incorporating interactive e-learning and practice, in which 11 therapists learned how to integrate MLSs within VR-based therapy. Self-report and observer-rated outcome measures evaluated therapists' confidence, clinical reasoning and behaviour with respect to MLS use. A focus group captured therapists' perspectives on MLS use during VR-based therapy provision.

RESULTS

The intervention improved self-reported confidence about MLS use as measured by confidence ratings (p <0.001). Chart-Stimulated Recall indicated a moderate level of competency in therapists' clinical reasoning about MLSs following the intervention, with no changes following additional opportunities to use VR (p = .944). On the Motor Learning Strategy Rating Instrument, no behaviour change with respect to MLS use was noted (p = 0.092). Therapists favoured the strategy of transferring skills from VR to real-life tasks over employing a more comprehensive MLS approach.

CONCLUSION

The KT intervention improved therapists' confidence but did not have an effect on clinical reasoning or behaviour with regard to MLS use during VR-based therapy.

Developing a Preceptorship Programme for Newly Qualified Occupational Therapists: Action Research

This action research study used three focus groups and one interview to explore the experiences of five recently qualified occupational therapists, four supervisors and five occupational therapy managers. The participants were asked to identify the perceived development needs of new practitioners and the mechanisms that supported or hindered these being met.

The findings showed that the recently qualified occupational therapists experienced satisfaction at making a difference in their first posts. However, for some, their first post had fallen short of their expectations because they faced challenges when moving from student to practitioner, sometimes with limited support. The participants identified development needs that they felt were common to other new practitioners and also the factors that constrained or enabled the meeting of these needs.

These findings informed the second phase of the action research study to design a preceptorship programme that was compliant with the revised National Health Service employment contract (Department of Health 2005) and would improve the transitional experience of occupational therapists. This paper presents the focus group results and the rationale for the preceptorship programme, which was launched as a pilot evaluation study with occupational therapy staff from over 20 organisations in Autumn 2005.

Using patients' charts to assess medical trainees in the workplace: a systematic review

OBJECTIVES

The objective of this review is to summarize and critically appraise existing evidence on the use of chart stimulated recall (CSR) and case-based discussion (CBD) as an assessment tool for medical trainees.

METHODS

Medline, Embase, CINAHL, PsycINFO, Educational Resources Information Centre (ERIC), Web of Science, and the Cochrane Central Register of Controlled Trials were searched for original articles on the use of CSR or CBD as an assessment method for trainees in all medical specialties.

RESULTS

Four qualitative and three observational non-comparative studies were eligible for this review. The number of patient-chart encounters needed to achieve sufficient reliability varied across studies. None of the included studies evaluated the content validity of the tool. Both trainees and assessors expressed high level of satisfaction with the tool; however, inadequate training, different interpretation of the scoring scales and skills needed to give feedback were addressed as limitations for conducting the assessment.

CONCLUSION

There is still no compelling evidence for the use of patient's chart to evaluate medical trainees in the workplace. A body of evidence that is valid, reliable, and documents the educational effect in support of the use of patients' charts to assess medical trainees is needed.

Behavioral Health Competence: An Exploration of Army Reserve Occupational Therapists

The behavioral health competence of Army Reserve Occupational Therapists (OT) was examined by electronic survey to determine current levels of competence and highlight pre-deployment training needs. Results indicated that while Army Reserve OTs report high levels of behavioral health competence, many questions regarding diagnosis, assessment, evaluation, treatment planning, intervention, and progress arose throughout deployment. OT's often relied on skills from Level II fieldwork education and entry-level didactic education for competency. Perceived competencies may be compromised by curriculum changes in entry-level education, available fieldwork settings, and a lack of adequate training currently available prior to deployment.

"Push" versus "Pull" for mobilizing pain evidence into practice across different health professions: a protocol for a randomized trial

BACKGROUND

Optimizing pain care requires ready access and use of best evidence within and across different disciplines and settings. The purpose of this randomized trial is to determine whether a technology-based "push" of new, high-quality pain research to physicians, nurses, and rehabilitation and psychology professionals results in better knowledge and clinical decision making around pain, when offered in addition to traditional "pull" evidence technology. A secondary objective is to identify disciplinary variations in response to evidence and differences in the patterns of accessing research evidence.

METHODS

Physicians, nurses, occupational/physical therapists, and psychologists (n = 670) will be randomly allocated in a crossover design to receive a pain evidence resource in one of two different ways. Evidence is extracted from medical, nursing, psychology, and rehabilitation journals; appraised for quality/relevance; and sent out (PUSHed) to clinicians by email alerts or available for searches of the accumulated database (PULL). Participants are allocated to either PULL or PUSH + PULL in a randomized crossover design. The PULL intervention has a similar interface but does not send alerts; clinicians can only go to the site and enter search terms to retrieve evidence from the cumulative and continuously updated online database. Upon entry to the trial, there is three months of access to PULL, then random allocation. After six months, crossover takes place. The study ends with a final three months of access to PUSH + PULL. The primary outcomes are uptake and application of evidence. Uptake will be determined by embedded tracking of what research is accessed during use of the intervention. A random subset of 30 participants/ discipline will undergo chart-stimulated recall to assess the nature and depth of evidence utilization in actual case management at baseline and 9 months. A different random subset of 30 participants/ discipline will be tested for their skills in accessing evidence using a standardized simulation test (final 3 months). Secondary outcomes include usage and self-reported evidence-based practice attitudes and behaviors measured at baseline, 3, 9, 15 and 18 months.

DISCUSSION

The trial will inform our understanding of information preferences and behaviors across disciplines/practice settings. If this intervention is effective, sustained support will be sought from professional/health system initiatives with an interest in optimizing pain management.

TRIAL REGISTRATION

Registered as NCT01348802 on clinicaltrials.gov.

Administration of the Canadian Occupational Performance Measure: effect on practice

BACKGROUND

The Canadian Occupational Performance Measure (COPM) is recommended as a systematic approach to identify issues and determine client progress in occupational therapy, yet little empirical evidence is available that supports this practice.

PURPOSE

To determine if COPM administration was associated with changes in eight dimensions of occupational therapy practice.

METHODS

Twenty-four occupational therapists on eight geriatric rehabilitation sites completed a before-and-after study with a repeated baseline. The eight practice dimensions were assessed after three months of usual care (no COPM use) and after three months of intervention (COPM use) using chart stimulated recall (CSR) interviews and chart audit.

FINDINGS

Mean practice scores for CSR interviews indicated a statistically significant practice improvement (p < .0001) across the eight dimensions, including knowledge of client perspective, clinical decision making, clinician ability to articulate outcomes, and documentation. Chart audit indicated that COPM use resulted in identifying more occupation-focused issues.

IMPLICATIONS

COPM administration could improve occupational therapy practice.

The feasibility and acceptability of using a portfolio to assess professional competence

PURPOSE

Little is known about physical therapists' views on the use of portfolios to evaluate professional competence. The purpose of this study was to gather the opinions of physical therapists on the feasibility and acceptability of a portfolio prepared to demonstrate evidence of clinical specialization through reported activities and accomplishments related to professional development, leadership, and research.

METHODS

Twenty-nine Canadian physical therapists practising in the neurosciences area were given 8 weeks to prepare a professional portfolio. Participants submitted the portfolio along with a survey addressing the preparation of the portfolio and its role as an assessment tool. Qualitative content analysis was used to interpret the participants' comments.

RESULTS

Participants reported that maintaining organized records facilitated the preparation of their portfolio. They experienced pride when reviewing their completed portfolios, which summarized their professional activities and highlighted their achievements. Concerns were noted about the veracity of self-reported records and the ability of the documentation to provide a comprehensive view of the full scope of the professional competencies required for clinical specialization (e.g., clinical skills).

CONCLUSION

The study's findings support the feasibility and acceptability of a portfolio review to assess professional competence and clinical specialization in physical therapy and have implications for both physical therapists and professional agencies.

Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy

BACKGROUND

The College of Physiotherapists of Ontario implemented an Onsite Assessment to evaluate the continuing competence of physical therapists.

OBJECTIVE

This study was undertaken to examine the reliability of the various tools used in the Onsite Assessment and to consider the relationship between the final decision and demographic factors.

DESIGN

This was a psychometric study.

METHODS

Trained peer assessors (n=63) visited randomly selected physical therapists (n=106) in their workplace. Fifty-three physical therapists were examined by 2 assessors simultaneously. The assessment included a review of practice issues, record keeping, billing practices, the physical therapist's professional portfolio, and a chart-stimulated recall process. The Quality Management Committee made the final decision regarding the physical therapist's performance using the assessor's summary report. Generalizability theory was used to examine the interrater reliability of the tools. Correlation coefficients and regression analyses were used to examine the relationships between demographic factors and performance.

RESULTS

The majority of the physical therapists (88%) completed the program successfully, 11% required remediation, and 1% required further assessment. The interrater reliability of the components was above .70 for 2 raters' evaluations, with the exception of billing practices. There was no relationship between the final decision and age or years since graduation (r<.05). Limitations Limitations include a small sample and a lack of data on system-related factors that might influence performance.

CONCLUSIONS

The vast majority of the physical therapists met the College of Physiotherapists of Ontario's professional standards. Reliability analysis indicated that the number of charts reviewed could be reduced. Strategies to improve the reliability of the various components must take into account feasibility issues related to financial and human resources. Further research to examine factors associated with failure to adhere to professional standards should be considered. These results can provide valuable information to regulatory agencies or managers considering similar continuing competence assessment programs.

Assessment of family physicians' performance using patient charts: interrater reliability and concordance with chart-stimulated recall interview

Peer-assessment processes with chart review have been used for many years to assess the clinical performance of physicians. The Quebec medical licensing authority has been required by provincial law to assess the practicing Quebec physicians on a nonvoluntary basis. During the period from January 2001 to November 2004, 25 family physicians in active practice were randomly selected from a pool of about 300. For each physician, 25 to 40 patients' medical charts were randomly selected to evaluate the interrater reliability of peer-review assessment of medical charts and to compare ratings based on chart review with a chart-stimulated recall interview to those based on chart review alone. The concordance between chart review alone and that of chart review with chart-stimulated recall interview was 75% for chart keeping, 69% for clinical investigation, 81% for diagnostic accuracy, and 74% for treatment plan. Ratings based on chart review alone achieve moderate levels of reliability (Kappa = 0.44 to 0.56). It appears that some important information about quality of care is missed when only chart review is used.

Workplace-based assessment for general practitioners: using stakeholder perception to aid blueprinting of an assessment battery

CONTEXT

The implementation of an assessment system may be facilitated by stakeholder agreement that appropriate qualities are being tested. This study investigated the extent to which stakeholders perceived 8 assessment formats (multiple-choice questions, objective structured clinical examination, video, significant event analysis, criterion audit, multi-source feedback, case analysis and patient satisfaction questionnaire) as able to assess varying qualities of doctors training in UK general practice.

METHODS

Educationalists, general practice trainers and registrars completed a blueprinting style of exercise to rate the extent to which each evaluation format was perceived to assess each of 8 competencies derived primarily from the General Medical Council document 'Good Medical Practice'.

RESULTS

There were high levels of agreement among stakeholders regarding the perceived qualities tested by the proposed formats (G = 0.82-0.93). Differences were found in participants' perceptions of how well qualities were able to be assessed and in the ability of the respective formats to test each quality. Multi-source feedback (MSF) was expected to assess a wide range of qualities, whereas Probity, Health and Ability to work with colleagues were limited in terms of how well they could be tested by the proposed formats.

DISCUSSION

Awareness of the perceptions of stakeholders should facilitate the development and implementation of workplace-based assessment (WPBA) systems. These data shed light on the acceptability of various formats in a way that will inform further investigation of WPBA formats' validity and feasibility, while also providing evidence on which to base educational efforts regarding the value of each format.

Defining the effect and mediators of two knowledge translation strategies designed to alter knowledge, intent and clinical utilization of rehabilitation outcome measures: a study protocol [NCT00298727]

BACKGROUND

A substantial number of valid outcome measures have been developed to measure health in adult musculoskeletal and childhood disability. Regrettably, national initiatives have merely resulted in changes in attitude, while utilization remains unacceptably low. This study will compare the effectiveness and mediators of two different knowledge transfer (KT) interventions in terms of their impact on changing knowledge and behavior (utilization and clinical reasoning) related to health outcome measures.

METHOD/DESIGN

Physical and occupational therapists (n = 144) will be recruited in partnership with the national professional associations to evaluate two different KT interventions with the same curriculum: 1) Stakeholder-Hosted Interactive Problem-Based Seminar (SHIPS), and 2) Online Problem-Based course (e-PBL). SHIPS will consist of face-to-face problem-based learning (PBL) for 2 1/2 days with outcome measure developers as facilitators, using six problems generated in consultation with participants. The e-PBL will consist of a 6-week web-based course with six generic problems developed by content experts. SHIPS will be conducted in three urban centers in Canada. Participants will be block-allocated by a minimization procedure to either of the two interventions to minimize any prognostic differences. Trained evaluators at each site will conduct chart audits and chart-stimulated recall. Trained interviewers will conduct semi-structured interviews focused on identifying critical elements in KT and implementing practice changes. Interviews will be transcribed verbatim. Baseline predictors including demographics, knowledge, attitudes/barriers regarding outcome measures, and Readiness to Change will be assessed by self-report. Immediately post-intervention and 6 months later, these will be re-administered. Primary qualitative and quantitative evaluations will be conducted 6-months post-intervention to assess the relative effectiveness of KT interventions and to identify elements that contribute to changing clinical behavior. Chart audits will determine the utilization of outcome measures (counts). Incorporation of outcome measures into clinical reasoning will be assessed using an innovative technique: chart-stimulated recall.

DISCUSSION

A strategy for optimal transfer of health outcome measures into practice will be developed and shared with multiple disciplines involved in primary and specialty management of musculoskeletal and childhood disability.

Assessment of clinical practice in intensive care: a review of the literature

Nursing is a practice-based discipline, with clinical practice forming the heart of any programme of study. The major goal of all programmes should be on facilitating students to develop the skills necessary for competent and compassionate practice. Assessment of clinical competence is a fundamental aspect of programme development and as such deserves attention during the curriculum design process. The literature on assessment of students in clinical practice suggests that the tools used have evolved through various stages, ranging from the use of simple check list and rating scale to the development of competence assessment tools. Each stage has been fraught with difficulties and has taxed nurse educators, clinical assessors and students alike. The literature also indicates that there is a limited range of research available on clinical assessment tools, especially with an intensive care context. In part one of this two-part paper, the literature on assessment of clinical practice is explored, with specific emphasis on the assessment of competence with the Intensive Care environment. Part two of the paper reports on the findings of a study on students perceptions on a clinical competence assessment tool, used in an Intensive Care environment.

Body weight-supported treadmill training in chronic incomplete spinal cord injury: a pilot study evaluating functional health status and quality of life

STUDY DESIGN

A controlled single-case design: A1 (baseline: 6 weeks), B (intervention: 12 weeks of treadmill training (TT), maximally five times a week/30 min a day), A2 (wash-out: 6 weeks), follow-up measurement: 6 months.

OBJECTIVE

To investigate the effects of TT on functional health status (FHS) and quality of life (QoL) in subjects with a chronic incomplete spinal cord injury (ISCI).

SETTING

Rehabilitation Department, University Medical Centre Utrecht, The Netherlands.

METHODS

Three male subjects with a stable (>48 months postinjury) ISCI, American Spinal Injury Association (ASIA) class C (n=2) and D (n=1). Performance-based walking, subject's perception concerning quality of life (SEIQoL) and activities of daily living Canadian Occupational Performance Measure (COPM).

RESULTS

The results of the three subjects were variable. Changes in QoL were relatively small and diverse. After 6 months' follow-up, QoL was unchanged in subjects 1 and 2, and improved in subject 3. In subject 2, performance of activities of daily living (ADL) was significantly improved, consistent with his perception of improvement (P<0.05), and this improvement was sustained throughout the follow-up period. Walking ability improved in subject 3 (P<0.05) but performance of other activities remained stable. Performance of ADL decreased slightly in subject 1 whereas his walking speed and Get up and Go performance improved (P<0.05).

CONCLUSIONS

This study demonstrates positive effects of TT on FHS. A randomised clinical trial should be executed before definite conclusions about the effect of TT on FHS and QoL can be drawn.

SPONSORSHIP

KF Hein Foundation and Rehabilitation Centre De Hoogstraat Scientific Foundation.