A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique

Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliability, and in particular test-retest reliability, estimates in policy-capturing studies. We found that only 164 of 955 policy-capturing studies (i.e., 17.17%) reported a test-retest reliability estimate. We then conducted a reliability generalization meta-analysis on policy-capturing studies that did report test-retest reliability estimates—and we obtained an average reliability estimate of .78. We additionally examined 16 potential methodological and substantive antecedents to test-retest reliability (equivalent to moderators in validity generalization studies). We found that test-retest reliability was robust to variation in 14 of the 16 factors examined but that reliability was higher in paper-and-pencil studies than in web-based studies and was higher for behavioral intention judgments than for other (e.g., attitudinal and perceptual) judgments. We provide an agenda for future research. Finally, we provide several best-practice recommendations for researchers (and journal reviewers) with regard to (a) reporting test-retest reliability, (b) designing policy-capturing studies for appropriate reportage, and (c) properly interpreting test-retest reliability in policy-capturing studies.

Referral Prioritization in Home Care Occupational Therapy: A Matter of Perspective

Background.

Prioritizing referrals for home care occupational therapy is somewhat subjective, and public and patient perspectives on waiting list priorities are unknown.

Purpose.

To explore the views of home care occupational therapists (OTs), older persons (OPs) and adults with disabilities on waiting list priorities, as well as issues and challenges underlying these priorities.

Method.

We conducted in-depth interviews with 11 OTs, 10 OPs and 9 adults with disabilities. Participants were asked to prioritize referral scenarios while explaining their choices. Directed and conventional content analysis allowed the identification of themes for each group of participants.

Findings.

OTs experienced conflicts of values but mainly prioritized referrals based on client safety. OPs sought to maximize client’s independence, and persons with disabilities aimed to improve clients’ social participation.

Implications.

OTs should seek the perspectives of their target clientele on referral prioritization criteria and strive to adjust prioritization practices accordingly.

Research priorities in mental health occupational therapy: A study of clinician perspectives

BACKGROUND/AIM

The evidence to support mental health occupational therapy has proliferated in the early years of this century, but this growth has tended to be organic rather than targeted. Previous efforts to identify research priorities in this area of practice are either out dated, or encompass discrete areas of practice. The aim of this study was to identify priority areas for research in mental health occupational therapy from clinician's perspectives.

METHODS

A Policy Delphi method was used to enable occupational therapists to define and differentiate their perspectives on research priorities. Forty-two occupational therapists took part in the first two rounds of this method, with 69% (n = 29) going on to complete the third and final round of data collection. A Likert scale was used to rate the importance of each priority, and descriptive quantitative analysis undertaken to identify those most consistently identified as being highly important.

RESULTS

Four research priorities were identified as being highly important in this study: (i) working in an occupationally focussed way; (ii) consumer experience of therapy groups; (iii) identifying factors which increase consumer engagement in occupation; and (iv) engaging patients on the inpatient unit in meaningful and positive occupation.

CONCLUSIONS

Two of the priority areas are already the subject of substantial evidence bases, but there has been far less research into consumer experiences of groups and occupational engagement in acute settings. Collaboration between research teams and greater consumer inclusion are recommended for the future.

SIGNIFICANCE OF THE STUDY

This study provides an updated indication of research priorities for mental health occupational therapy in Australia.

Introducing Prioritisation Protocols to Promote Efficient and Effective Allocation of Mental Health Occupational Therapy Resources

At a time of unprecedented financial constraints in the National Health Service, occupational therapy services are required to develop mechanisms to ensure the efficient allocation of occupational therapy resources. This paper describes the development of prioritisation protocols within a mental health trust. A review of their implementation found that occupational therapists were finding the protocols helpful in prioritising referrals and clarifying the added value of their interventions. These findings and the follow-up action plan suggest how protocols such as these can be used effectively to ensure that the occupational needs of service users are met, resulting in enhanced outcomes.

Teaching young dogs new tricks: improving occupational therapists' referral prioritization capacity with a web-based decision-training aid

OBJECTIVE

A web-based clinical decision aid was designed to improve novices' capacity to prioritize occupational therapy referrals in community mental health teams. The aim of this research was to test the effectiveness of the decision aid.

DESIGN

A trial was conducted using a consensus expert policy as the educational intervention; a one-group pre-test, post-test design was chosen to measure any effect of training.

PARTICIPANTS

A total of 100 pre-registration occupational therapy students accessed the decision aid; they prioritized a set of referrals before and after being trained with graphical and descriptive representations of the expert policy.

SETTING

Data were collected online via the World Wide Web from students at 12 UK universities.

RESULTS

The effect of training was demonstrated through more accurate prioritization ratings when matched with expert ratings on the same referrals and more accurate judgement policies when matched with the consensus standard judgement policy.

CONCLUSION

The tool was found to be effective as it improved referral prioritization capacity.

Effectiveness of a decision-training aid on referral prioritization capacity: a randomized controlled trial

BACKGROUND

In the community mental health field, occupational therapy students lack the capacity to prioritize referrals effectively.

OBJECTIVE

The purpose of this study was to test the effectiveness of a clinical decision-training aid on referral prioritization capacity.

DESIGN

A double-blind, parallel-group, randomized controlled trial was conducted using a judgment analysis approach.

SETTING

Each participant used the World Wide Web to prioritize referral sets at baseline, immediate posttest, and 2-wk follow-up. The intervention group was provided with training after baseline testing; control group was purely given instructions to continue with the task.

PARTICIPANTS

One hundred sixty-five students were randomly allocated to intervention (n = 87) or control (n = 81). Intervention. Written and graphical descriptions were given of an expert consensus standard explaining how referral information should be used to prioritize referrals.

MEASUREMENTS

Participants' prioritization ratings were correlated with the experts' ratings of the same referrals at each stage of testing, as well as to examine the effect on mean group scores, regression weights, and the lens model indices.

RESULTS

At baseline, no differences were found between control and intervention on rating capacity or demographic characteristics. Comparison of the difference in mean correlation baseline scores of the control and intervention group compared with immediate posttest showed a statistically significant result that was maintained at 2-wk follow-up. The effect size was classified as large. At immediate posttest and follow-up, the intervention group improved rating capacity, whereas the control group's capacity remained poor. The results of this study indicate that the decision-training aid has a positive effect on referral prioritization capacity.

CONCLUSIONS

This freely available, Web-based decision-training aid will be a valuable adjunct to the education of these novice health professionals internationally.

Experiences of Occupational Therapists and Occupational Therapy Students in Using the Assessment of Communication and Interaction Skills in Mental Health Settings in Norway

Introduction:

This study investigated the utility of the Assessment of Communication and Interaction Skills (ACIS) in mental health settings, as experienced by Norwegian occupational therapists and occupational therapy students.

Method:

An informal draft translation was used to investigate the appropriateness of the assessment for mental health practice. Focus group interviews were conducted with three occupational therapy practice educators working in mental health settings and six occupational therapy students on practice placement, who used the ACIS during an 8-week mental health practice placement in 2009. The focus groups aimed to explore the experiences of using the translated ACIS in working with clients.

Findings:

Three main themes emerged: observing with the ACIS, scoring the ACIS and using the ACIS in clinical practice.

Conclusion:

The ACIS has the potential for making a positive contribution, both for the individual client and for the larger team of multi professional staff, by giving a structure and terminology that supports precise observation and reporting. Questions were raised as to how the therapist's level of engagement during the observation may have an impact upon the client's performance. The process of scoring and giving feedback concerning the client's skills may be emotionally challenging for therapists, students and clients, and issues about how to give feedback to clients require consideration.