Using standardised patients to measure physicians' practice: validation study using audio recordings

A systematic review of the use of unannounced standardized patients (USPs) in clinical settings: A call for more detailed quality and fidelity descriptions and expansion to new areas

BACKGROUND

Unannounced standardized patients (USPs) have long been used to measure clinical performance in situ. These incognito actors capture data on clinician skills during an encounter, as well as patient experience more broadly. A robust USP program requires extensive preparation and standardization efforts. Given the widespread expansion of USPs for education, research, and improvement efforts, we conducted a systematic review with the goal of capturing the breadth of uses of USPs across settings, along with the standardization measures employed across studies.

METHODS

In collaboration with a medical librarian, we conducted systematic searches across six databases. Two independent researchers screened each report for inclusion. Three coders extracted and reviewed study characteristics and data from the studies deemed eligible for inclusion. We extracted data on: target population, setting, and assessed skills. We also captured the reliability and fidelity measures described in each study, including USP detection, USP training methods, and assessment measures.

RESULTS

128 articles were included. Individual clinicians were the most frequently targeted population (n = 114, 89 %). Common clinician roles included physicians (n = 92, 72 %) and pharmacists (n = 12, 9 %). The collective care team was the target in two studies (2 %), and systems and larger healthcare facilities were targeted in only 1 (1 %) and 13 (10 %) studies, respectively. Studies were primarily conducted in ambulatory settings (n = 118, 92 %). History gathering (n = 76, 59 %), communication (n = 55, 43 %), counseling (n = 51, 40 %), and patient education (n = 49, 38 %) were commonly assessed, as were correct diagnosis (n = 34, 27 %), appropriate ordering of labs/tests (n = 30, 23 %), referrals (n = 35, 27 %), and prescriptions (n = 36, 28 %). USP detection reporting was variable across studies; however, no detection information was provided for 48 studies. 62 % of articles reported incorporating a measure of reliability or fidelity into their study, while the remainder either failed to provide adequate information on use of these measures.

CONCLUSIONS

We explored USP use across settings and describe the scope and limitations of the literature. USPs capture a range of data domains but a lack uniform report of reliability measures can potentially undermine findings. Future studies should incorporate and uniformly report out on detection, training, and assessment.

Using Unannounced Standardized Patients to Assess Clinician Telehealth and Communication Skills at an Urban Student Health Center

PURPOSE

As the COVID-19 pandemic forced most colleges and universities to go online, student health centers rapidly shifted to telehealth platforms without frameworks for virtual care provision. An urban student health center implemented a needs assessment involving unannounced standardized patients (USPs) to evaluate the integration of a new telehealth workflow and clinicians' virtual communication skills.

METHODS

From April to May 2021, USPs conducted two video visits with 12 primary care and four women's health clinicians (N = 16 clinicians; 32 visits). Cases included (1) a 21-year-old female presenting for birth control with a positive Patient Health Questionaire-9 and (2) a 21-year-old male, who vapes regularly, with questions regarding safe sex with men. Clinicians were evaluated using a checklist completed by the USP immediately following the visit and a systematic chart review of the electronic health record.

RESULTS

USP feedback indicates most clinicians received high ratings for general communication skills but may benefit from educational intervention in several key telemedicine skills. Clinicians struggled with using nonverbal signals to enrich communication (47% well done), acknowledging emotions (34% well done), and using video for information gathering (34% well done). Low rates of standard screenings (e.g., 63% administered the PHQ-2, <50% asked about alcohol use) suggested protocols for in-person care were not easily incorporated into telehealth practices, and clinicians may benefit from enhanced care team support. Performance reports were shared with clinicians and leadership postvisit.

DISCUSSION

Results suggest project design and implementation is scalable and feasible for use at other institutions, offering a structured methodology that can improve general student health care.

Comparison of patient exit interviews with unannounced standardised patients for assessing HIV service delivery in Zambia: a study nested within a cluster randomised trial

OBJECTIVES

To compare unannounced standardised patient approach (eg, mystery clients) with typical exit interviews for assessing patient experiences in HIV care (eg, unfriendly providers, long waiting times). We hypothesise standardised patients would report more negative experiences than typical exit interviews affected by social desirability bias.

SETTING

Cross-sectional surveys in 16 government-operated HIV primary care clinics in Lusaka, Zambia providing antiretroviral therapy (ART).

PARTICIPANTS

3526 participants aged ≥18 years receiving ART participated in the exit surveys between August 2019 and November 2021.

INTERVENTION

Systematic sample (every nth file) of patients in clinic waiting area willing to be trained received pre-visit training and post-visit interviews. Providers were unaware of trained patients.

OUTCOME MEASURES

We compared patient experience among patients who received brief training prior to their care visit (explaining each patient experience construct in the exit survey, being anonymous, without manipulating behaviour) with those who did not undergo training on the survey prior to their visit.

RESULTS

Among 3526 participants who participated in exit surveys, 2415 were untrained (56% female, median age 40 (IQR: 32-47)) and 1111 were trained (50% female, median age 37 (IQR: 31-45)). Compared with untrained, trained patients were more likely to report a negative care experience overall (adjusted prevalence ratio (aPR) for aggregate sum score: 1.64 (95% CI: 1.39 to 1.94)), with a greater proportion reporting feeling unwelcome by providers (aPR: 1.71 (95% CI: 1.20 to 2.44)) and witnessing providers behaving rude (aPR: 2.28 (95% CI: 1.63 to 3.19)).

CONCLUSION

Trained patients were more likely to identify suboptimal care. They may have understood the items solicited better or felt empowered to be more critical. We trained existing patients, unlike studies that use 'standardised patients' drawn from outside the patient population. This low-cost strategy could improve patient-centred service delivery elsewhere.

TRIAL REGISTRATION NUMBER

Assessment was nested within a parent study; www.pactr.org registered the parent study (PACTR202101847907585).

Improving eye care quality through brief verbal intervention on optometry service provider by using unannounced standardized patient with refractive error: study protocol for a randomized controlled trial

BACKGROUND

Improper refractive correction can be harmful to eye health, aggravating the burden of vision impairment. During most optometry clinical consultations, practitioner-patient interactions play a key role. Maybe it is feasible for patients themselves to do something to get high-quality optometry. But the present empirical research on the quality improvement of eye care needs to be strengthened. The study aims to test the effect of the brief verbal intervention (BVI) through patients on the quality of optometry service.

METHODS

This study will take unannounced standardized patient (USP) with refractive error as the core research tool, both in measurement and intervention. The USP case and the checklist will be developed through a standard protocol and assessed for validity and reliability before its full use. USP will be trained to provide standardized responses during optical visits and receive baseline refraction by the skilled study optometrist who will be recruited within each site. A multi-arm parallel-group randomized trial will be used, with one common control and three intervention groups. The study will be performed in four cities, Guangzhou and three cities in Inner Mongolia, China. A total of 480 optometry service providers (OSPs) will be stratified and randomly selected and divided into four groups. The common control group will receive USP usual visits (without intervention), and three intervention groups will separately receive USP visits with three kinds of BVI on the patient side. A detailed outcome evaluation will include the optometry accuracy, optometry process, patient satisfaction, cost information and service time. Descriptive analysis will be performed for the survey results, and the difference in outcomes between interventions and control providers will be compared and statistically tested using generalized linear models (GLMs).

DISCUSSION

This research will help policymakers understand the current situation and influencing factors of refractive error care quality, and then implement precise policies; at the same time, explore short and easy interventions for patients to improve the quality of optometry service.

TRIAL REGISTRATION

Chinese Clinical Trial Registry ChiCTR2200062819. Registered on August 19, 2022.

A survey of Indian physiotherapists' clinical practice patterns and adherence to clinical guidelines in the management of patients with acute low back pain

INTRODUCTION

Clinical practice guidelines (CPG) exist for the management of non-specific low back pain (LBP). The objective of this study is to evaluate if Indian physiotherapists' follow CPGs when treating patients with acute LBP.

METHODS

A cross-sectional survey using an online questionnaire was used to collect demographic information, views, and opinion about acute LBP and CPGs, and management strategies of a clinical vignette presenting a patient with acute LBP.

RESULTS

Responses from 328 physiotherapists were included in this study. Eighty-one percent of respondents indicated familiarity with CPGs for LBP and 75.3% (n = 328) respondents indicated that their intervention choices aligned with guidelines to at least some extent. Participants with post-graduate and doctoral degrees were more accustomed to CPGs than those with bachelor's degrees (p < 0.01). There were significant differences in clinical practice (p < 0.01) between therapists who expressed familiarity with guidelines and those who were not, as well as those with and without post-graduate and doctoral degrees.

CONCLUSION

In general, the study showed adherence to guidelines; however, there were areas that did not align with established evidence, especially referral for radiology and use of electrical modalities.

Validity of patients' online reviews at direct-to-consumer teleconsultation platforms: a protocol for a cross-sectional study using unannounced standardised patients

INTRODUCTION

As direct-to-consumer teleconsultation (hereafter referred to as 'teleconsultation') has gained popularity, an increasing number of patients have been leaving online reviews of their teleconsultation experiences. These reviews can help guide patients in identifying doctors for teleconsultation. However, few studies have examined the validity of online reviews in assessing the quality of teleconsultation against a gold standard. Therefore, we aim to use unannounced standardised patients (USPs) to validate online reviews in assessing both the technical and patient-centred quality of teleconsultations. We hypothesise that online review results will be more consistent with the patient-centred quality, rather than the technical quality, as assessed by the USPs.

METHODS AND ANALYSIS

In this cross-sectional study, USPs representing 11 common primary care conditions will randomly visit 253 physicians via the three largest teleconsultation platforms in China. Each physician will receive a text-based and a voice/video-based USP visit, resulting in a total of 506 USP visits. The USP will complete a quality checklist to assess the proportion of clinical practice guideline-recommended items during teleconsultation. After each visit, the USP will also complete the Patient Perception of Patient-Centeredness Rating. The USP-assessed results will be compared with online review results using the intraclass correlation coefficient (ICC). If ICC >0.4 (p<0.05), we will assume reasonable concordance between the USP-assessed quality and online reviews. Furthermore, we will use correlation analysis, Lin's Coordinated Correlation Coefficient and Kappa as supplementary analyses.

ETHICS AND DISSEMINATION

This study has received approval from the Institutional Review Board of Southern Medical University (#Southern Medical Audit (2022) No. 013). Results will be actively disseminated through print and social media, and USP tools will be made available for other researchers.

TRIAL REGISTRATION

The study has been registered at the China Clinical Trials Registry (ChiCTR2200062975).

Does high workload reduce the quality of healthcare? Evidence from rural Senegal

There is a widely held perception that staff shortages in low and middle-income countries (LMICs) lead to excessive workloads, which in turn worsen the quality of healthcare. Yet there is little evidence supporting these claims. We use data from standardised patient visits in Senegal and determine the effect of workload on the quality of primary care by exploiting quasi-random variation in workload. We find that despite a lack of staff, average levels of workload are low. Even at times when workload is high, there is no evidence that provider effort or quality of care are significantly reduced. Our data indicate that providers operate below their production possibility frontier and have sufficient capacity to attend more patients without compromising quality. This contradicts the prevailing discourse that staff shortages are a key reason for poor quality primary care in LMICs and suggests that the origins likely lie elsewhere.

Direct observation of depression screening: identifying diagnostic error and improving accuracy through unannounced standardized patients

Background Depression is substantially underdiagnosed in primary care, despite recommendations for screening at every visit. We report a secondary analysis focused on depression of a recently completed study using unannounced standardized patients (USPs) to measure and improve provider behaviors, documentation, and subsequent claims for real patients. Methods Unannounced standardized patients presented incognito in 217 visits to 59 primary care providers in 22 New Jersey practices. We collected USP checklists, visit audio recordings, and provider notes after visits; provided feedback to practices and providers based on the first two visits per provider; and compared care and documentation behaviors in the visits before and after feedback. We obtained real patient claims from the study practices and a matched comparison group and compared the likelihood of visits including International Classification of Diseases, 10th Revision (ICD-10) codes for depression before and after feedback between the study and comparison groups. Results Providers significantly improved in their rate of depression screening following feedback [adjusted odds ratio (AOR), 3.41; 95% confidence interval (CI), 1.52-7.65; p = 0.003]. Sometimes expected behaviors were documented when not performed. The proportion of claims by actual patients with depression-related ICD-10 codes increased significantly more from prefeedback to postfeedback in the study group than in matched control group (interaction AOR, 1.41; 95% CI, 1.32-1.50; p < 0.001). Conclusions Using USPs, we found significant performance issues in diagnosis of depression, as well as discrepancies in documentation that may reduce future diagnostic accuracy. Providing feedback based on a small number of USP encounters led to some improvements in clinical performance observed both directly and indirectly via claims.

Impact of an unannounced standardized veteran program on access to community-based services for veterans experiencing homelessness

Background

The United States Department of Veterans Affairs established a program in which actors incognito portray veterans experiencing homelessness with pre-determined needs to identify barriers to access and services at community-based organizations.

Methods

From 2017 to 2019, actors who varied in gender, skin color and age portrayed one of three scripts at all VA Community-Based Resource and Referral Centers (CRRCs) serving veterans experiencing homelessness in 30 cities and completed an evaluative survey. They carried authentic VA identification and were registered in a VA patient database for each identity. CRRCs were provided with reports annually and asked to implement corrective plans. Data from the survey were analysed for change over time.

Results

Access to food, counselling, PTSD treatment, and hypertension/prediabetes care services increased significantly from 68–77% in year 2 to 83–97% in year 3 (each P < 0.05 adjusted for script present). A significant disparity in access for African American actors resolved following more uniform adherence to pre-existing policies.

Conclusions

The ‘unannounced standardized veteran’ (USV) can identify previously unrecognized barriers to needed services and care. Audit and feedback programs based on direct covert observation with systematic data collection and rapid feedback may be an effective strategy for improving services to highly vulnerable populations.

The simulated patient method: Design and application in health services research

The simulated patient method is becoming increasingly popular in health services research to observe the behaviour of healthcare practitioners in a naturalistic setting. This method involves sending a trained individual (simulated patient among other names), who is indistinguishable from a regular consumer, into a healthcare setting with a standardised scripted request. This paper provides an overview of the method, a brief history of its use in health services research, comparisons with other methods, ethical considerations, and considerations for the development of studies using the simulated patient method in health services research, with examples from pharmacy and other fields. Methods of analysis, mixed-methods, and the use of simulated patients with feedback are also discussed.

How accurate is the medical record? A comparison of the physician's note with a concealed audio recording in unannounced standardized patient encounters

OBJECTIVES

Accurate documentation in the medical record is essential for quality care; extensive documentation is required for reimbursement. At times, these 2 imperatives conflict. We explored the concordance of information documented in the medical record with a gold standard measure.

MATERIALS AND METHODS

We compared 105 encounter notes to audio recordings covertly collected by unannounced standardized patients from 36 physicians, to identify discrepancies and estimate the reimbursement implications of billing the visit based on the note vs the care actually delivered.

RESULTS

There were 636 documentation errors, including 181 charted findings that did not take place, and 455 findings that were not charted. Ninety percent of notes contained at least 1 error. In 21 instances, the note justified a higher billing level than the gold standard audio recording, and in 4, it underrepresented the level of service (P = .005), resulting in 40 level 4 notes instead of the 23 justified based on the audio, a 74% inflated misrepresentation.

DISCUSSION

While one cannot generalize about specific error rates based on a relatively small sample of physicians exclusively within the Department of Veterans Affairs Health System, the magnitude of the findings raise fundamental concerns about the integrity of the current medical record documentation process as an actual representation of care, with implications for determining both quality and resource utilization.

CONCLUSION

The medical record should not be assumed to reflect care delivered. Furthermore, errors of commission-documentation of services not actually provided-may inflate estimates of resource utilization.

Indicators for Assessing the Quality of Refractive Error Care

SIGNIFICANCE

Quality refractive error care is essential for reducing vision impairment. Quality indicators and standardized approaches for assessing the quality of refractive error care need to be established.

PURPOSE

This study aimed to develop a set of indicators for assessing the quality of refractive error care and test their applicability in a real-world setting using unannounced standardized patients (USPs).

METHODS

Patient outcomes and three quality of refractive error care (Q.REC) indicators (1, optimally prescribed spectacles; 2, adequately prescribed spectacles; 3, vector dioptric distance) were developed using existing literature, refraction training standards, and consulting educators. Twenty-one USPs with various refractive errors were trained to visit optical stores across Vietnam to have a refraction, observe techniques, and order spectacles. Spectacles were assessed against each Q.REC indicator and tested for associations with vision and comfort.

RESULTS

Overall, 44.1% (184/417) of spectacles provided good vision and comfort. Of the spectacles that met Q.REC indicators 1 and 2, 62.5 and 54.9%, respectively, provided both good vision and comfort. Optimally prescribed spectacles (indicator 1) were significantly more likely to provide good vision and comfort independently compared with spectacles that did not meet any indicator (good vision: 94.6 vs. 85.0%, P = .01; comfortable: 66.1 vs. 36.3%, P < .01). Adequately prescribed spectacles (indicator 2) were more likely to provide good comfort compared with spectacles not meeting any indicator (57.7 vs. 36.3%, P < .01); however, vision outcomes were not significantly different (85.9 vs. 85.0%, P = .90). Good vision was associated with a lower mean vector dioptric distance (P < .01) but not with comfort (P = .52).

CONCLUSIONS

The optimally prescribed spectacles indicator is a promising approach for assessing the quality of refractive error care without additional assessments of vision and comfort. Using USPs is a practical approach and could be used as a standardized method for evaluating the quality of refractive error care.

Gasping for air: measuring patient education and activation skillsets in two clinical assessment contexts

Objective structured clinical examinations (OSCEs) provide a controlled, simulated setting for competency assessments, while unannounced simulated patients (USPs) measure competency in situ or real-world settings. This exploratory study describes differences in primary care residents’ skills when caring for the same simulated patient case in OSCEs versus in a USP encounter. Data reported describe a group of residents (n=20) who were assessed following interaction with the same simulated patient case in two distinct settings: an OSCE and a USP visit at our safety-net clinic from 2009 to 2010. In both scenarios, the simulated patient presented as an asthmatic woman with limited understanding of illness management. Residents were rated through a behaviourally anchored checklist on visit completion. Summary scores (mean % well done) were calculated by domain and compared using paired sample t-tests. Residents performed significantly better with USPs on 7 of 10 items and in two of three aggregate assessment domains (p<0.05). OSCE structure may impede assessment of activation and treatment planning skills, which are better assessed in real-world settings. This exploration of outcomes from our two assessments using the same clinical case lays a foundation for future research on variation in situated performance. Using both assessments during residency will provide a more thorough understanding of learner competency.

Assessment of adequacy of respiratory infection prevention in hospitals of Inner Mongolia, China: a cross-sectional study using unannounced standardized patients

INTRODUCTION

Recent respiratory infectious disease (RID) outbreaks of influenza and the novel coronavirus have resulted in global pandemics. RIDs can trigger nosocomial infections if not adequately prevented.

OBJECTIVE

The objective of this study was to rate the adequacy of healthcare workers (HCWs) and hospital settings on RID prevention using unannounced standardized patients (USP) in clinical settings of hospital gateways.

METHODS

Trained USPs visited 5 clinical settings: information desks, registration desks, two outpatient departments and the emergency departments in 10 hospitals across 3 cities of Inner Mongolia, China. USPs observed the hospital air ventilation and distance from the nearest hand-washing facilities to each clinical setting, then mimicked symptoms of either tuberculosis or influenza before observing the HCW's behavior. A total of 480 clinical-setting assessments were made by 19 USPs.

RESULTS

The overall adequacy of triage services was 86.7% and for prevention of the spread of airborne droplets was 83.5%. Almost all hospitals offered adequate air ventilation. Compared to the information desk, adequacy of triage and preventing the spread of airborne droplets by physicians in the three clinical departments was less likely to be adequate. Triage services for USPs simulating symptoms of influenza were 2.6 times more likely to be adequate than for those simulating symptoms of tuberculosis but there was no significant difference in the prevention of the spread of airborne droplets.

CONCLUSIONS

There is a need to improve respiratory infectious disease procedures in our study hospitals, especially in outpatient and emergency departments.

Assessing the quality of primary healthcare in seven Chinese provinces with unannounced standardised patients: protocol of a cross-sectional survey

INTRODUCTION

Primary healthcare (PHC) serves as the cornerstone for the attainment of universal health coverage (UHC). Efforts to promote UHC should focus on the expansion of access and on healthcare quality. However, robust quality evidence has remained scarce in China. Common quality assessment methods such as chart abstraction, patient rating and clinical vignette use indirect information that may not represent real practice. This study will send standardised patients (SP or healthy person trained to consistently simulate the medical history, physical symptoms and emotional characteristics of a real patient) unannounced to PHC providers to collect quality information and represent real practice.

METHODS AND ANALYSIS

1981 SP-clinician visits will be made to a random sample of PHC providers across seven provinces in China. SP cases will be developed for 10 tracer conditions in PHC. Each case will include a standard script for the SP to use and a quality checklist that the SP will complete after the clinical visit to indicate diagnostic and treatment activities performed by the clinician. Patient-centredness will be assessed according to the Patient Perception of Patient-Centeredness Rating Scale by the SP. SP cases and the checklist will be developed through a standard protocol and assessed for content, face and criterion validity, and test-retest and inter-rater reliability before its full use. Various descriptive analyses will be performed for the survey results, such as a tabulation of quality scores across geographies and provider types.

ETHICS AND DISSEMINATION

This study has been reviewed and approved by the Institutional Review Board of the School of Public Health of Sun Yat-sen University (#SYSU 2017-011). Results will be actively disseminated through print and social media, and SP tools will be made available for other researchers.

Skills Training to Support Patients considering place of End-Of-Life Care: A Randomized Control Trial

The effect of a program to train clinicians to support patients making decisions about place of end-of-life care was evaluated. In all, 88 oncology and/or palliative care nursing and allied health providers from three Ontario health networks were randomly assigned to an education or control condition. Quality of decision support provided to standardized patients was measured before and after training, as were participants’ perceptions about the acceptability of the training program and their intentions to engage in patient decision support. Compared to controls, intervention group members improved the quality of decision support provided and were more likely to address a wider range of decision-making needs. Intervention group members scored higher on a knowledge test of decision support than controls and rated the components as acceptable. Improvements in the quality of decision support can be made by providing training and practical tools such as a patient decision aid.

Simulated patients in undergraduate education in psychiatry

This paper describes the use of simulated patients in medical education and how actors have been deployed with medical students in Aberdeen. The advantages and disadvantages of using actors for student education are summarised and we conclude with some possible future developments. At the outset, it may be helpful to outline some definitions, as in the review by Barrows (1993). A ‘standardised patient’ is an umbrella term for both an actual patient who is trained to present his or her own illness in a standardised way and also for a simulated patient who is a well person trained to portray an illness in a standardised way. This paper will use these terms but will relate mainly to the use of professional actors (not volunteers from the general public, who are often deployed by medical teachers) as simulated psychiatric patients.

Use of standardised patients to assess quality of healthcare in Nairobi, Kenya: a pilot, cross-sectional study with international comparisons

INTRODUCTION

The quality of clinical care can be reliably measured in multiple settings using standardised patients (SPs), but this methodology has not been extensively used in Sub-Saharan Africa. This study validates the use of SPs for a variety of tracer conditions in Nairobi, Kenya, and provides new results on the quality of care in sampled primary care clinics.

METHODS

We deployed 14 SPs in private and public clinics presenting either asthma, child diarrhoea, tuberculosis or unstable angina. Case management guidelines and checklists were jointly developed with the Ministry of Health. We validated the SP method based on the ability of SPs to avoid detection or dangerous situations, without imposing a substantial time burden on providers. We also evaluated the sensitivity of quality measures to SP characteristics. We assessed quality of practice through adherence to guidelines and checklists for the entire sample, stratified by case and stratified by sector, and in comparison with previously published results from urban India, rural India and rural China.

RESULTS

Across 166 interactions in 42 facilities, detection rates and exposure to unsafe conditions were both zero. There were no detected outcome correlations with SP characteristics that would bias the results. Across all four conditions, 53% of SPs were correctly managed with wide variation across tracer conditions. SPs paid 76% less in public clinics, but proportions of correct management were similar to private clinics for three conditions and higher for the fourth. Kenyan outcomes compared favourably with India and China in all but the angina case.

CONCLUSIONS

The SP method is safe and effective in the urban Kenyan setting for the assessment of clinical practice. The pilot results suggest that public providers in this setting provide similar rates of correct management to private providers at significantly lower out-of-pocket costs for patients. However, comparisons across countries are sensitive to the tracer condition considered.

What factors contribute to the continued low rates of Indigenous status identification in urban general practice? - A mixed-methods multiple site case study

Background

Indigenous peoples experience worse health and die at younger ages than their non-indigenous counterparts. Ethnicity data enables health services to identify inequalities experienced by minority populations and to implement and monitor services specifically targeting them. Despite significant Government intervention, Australia’s Indigenous peoples, the Aboriginal and Torres Strait Islander peoples, continue to be under identified in data sets. We explored the barriers to Indigenous status identification in urban general practice in two areas in Sydney.

Methods

A mixed-methods multiple-site case study was used, set in urban general practice. Data collection included semi-structured interviews and self-complete questionnaires with 31 general practice staff and practitioners, interviews with three Medicare Local staff, and focus groups with the two local Aboriginal and Torres Strait Islander communities in the study areas. These data were combined with clinical record audit data and Aboriginal unannounced standardised patient visits to participating practices to determine the current barriers to Indigenous status identification in urban general practice.

Results

Findings can be broadly grouped into three themes: a lack of practitioner/staff understanding on the need to identify Indigenous status or that a problem with identification exists; suboptimal practice systems to identify and/or record patients’ Indigenous status; and practice environments that do not promote Indigenous status identification.

Conclusion

Aboriginal and Torres Strait Islander peoples remain under-identified in general practice. There is a need to address the lack of practitioner and staff recognition that a problem with Indigenous status identification exists, along with entrenched attitudes and beliefs and limitations to practice software capabilities. Guidelines recommending Indigenous status identification and Aboriginal and Torres Strait Islander-specific Practice Incentive Payments have had limited impact on Indigenous status identification rates. It is likely that policy change mandating Indigenous status identification and recording in general practice will also be required.

Using a Secret Shopper to Evaluate Pharmacist Provision of Emergency Contraception

Introduction:

Commonly known as “the morning-after pill,” hormonal emergency contraception or the emergency contraceptive pill (ECP) is most effective if used within 72 hours of unprotected sex. In this paper we review a pilot project in Toronto, Ontario, that aimed to increase accessibility by allowing pharmacists to dispense ECP through a pharmacist-physician collaborative agreement, using a predefined protocol. Participating pharmacists received training in the pilot protocol and about ECP. To evaluate whether participating pharmacists were following the project protocol (i.e., giving accurate information to women requesting ECP and dispensing ECP appropriately), and to determine the quality of the pharmacist-patient encounter, we sent “secret shoppers” to participating pharmacies.

Method:

Five trained “secret shoppers,” using one of two predefined scripts (Script One, where ECP was clearly appropriate, and Script Two, where ECP was not appropriate) visited 34 participating pharmacies to request ECP. At the end of the visit, the secret shopper filled in a questionnaire about the encounter and provided comments. Percentages were calculated for all variables.

Results:

For Script One encounters (n=17), most pharmacists followed the protocol correctly to dispense ECP. For Script Two encounters (n=17), 71.4% provided alternatives to ECP, but three pharmacists did not follow protocol and provided ECP. For Script One, 52.9% of pharmacists and for Script Two, 71.4%, provided a community referral for follow-up care. The majority of pharmacists (97%) treated the shopper with respect and 85% communicated clearly. Most of the shoppers' comments were positive and the main negative comment about the encounter was lack of privacy.

Conclusions:

The use of a secret shopper allowed the examination of the pharmacist-customer interaction to delineate issues of importance to the pilot project and provide feedback on areas that may require improvement in the pharmacy provision of emergency contraception, namely, dealing with women who do not qualify for ECP and providing women with community referrals for sexually transmitted infections or ongoing contraception.

Variation in Long-Term Care Needs Assessors’ Willingness to Support Clients’ Requests for Admission to a Residential Home

Objectives: The purpose of this study is to determine what client, needs assessor, and agency factors explain variation in decision making by long-term care needs assessors concerning clients requesting admission to a residential home.

Method: Hypothetical case vignettes were sent to needs assessors allocating services for the elderly. Multilevel logistic regression analysis provided random and fixed effects.

Results: The authors found random effects of the level of needs assessors are negligible, of the level of agencies small though statistically significant, clients receiving largest relative share of the variance. The amount of care already present appeared most important in the decision. Needs assessors were willing to support their clients’ wishes only when they were clearly motivated. Policy implications considering the tension between clients’ preferences and equity are discussed.

Dutch Pain Specialists' Adherence to the Multidisciplinary Guideline on Treating Pain in Patients with Cancer: A Case Vignette Study

BACKGROUND

Many patients with cancer suffer from pain, which is often not optimally treated. In 2008, the evidence-based, multidisciplinary Dutch guideline on the diagnosis and treatment of pain in this patient group was published. We assessed knowledge about and adherence to the guideline by pain specialists.

METHODS

A cross-sectional case vignette survey describing a palliative patient with intractable pancreatic cancer and pain was sent to all 350 Dutch anesthesiologists registered as pain specialists at the Netherlands Association of Anesthesiology. Descriptive statistics were conducted.

RESULTS

Ninety-three pain specialists completed the questionnaire (27%). The majority appeared to follow the guideline recommendations on pharmacological (99%) and invasive treatment (95%) in the diverse stages of the disease. However, the recommendation to use a one-dimensional pain scale to evaluate the effect of pain treatment and the recommendation to perform a multidimensional pain assessment if the patient in pain is in a deteriorating stage were only followed by a minority of the respondents (23% and 15%, respectively).

CONCLUSIONS

Regarding most recommendations, Dutch pain specialists know and intend to follow the national multidisciplinary cancer pain guideline. Yet, only a minority of them perform structural pain assessment of the patient with cancer pain. However, as the response rate was low (27%), the results should be interpreted with caution and cannot be generalized to the entire population of pain specialists in the Netherlands. We recommend that, in the guideline update and implementation programs, more attention be given to thorough assessment of the patient with pain and cancer.

Assessing the appropriateness of prevention and management of venous thromboembolism in Australia: a cross-sectional study

OBJECTIVES

The prevention and management of venous thromboembolism (VTE) is often at variance with guidelines. The CareTrack Australia (CTA) study reported that appropriate care (in line with evidence-based or consensus-based guidelines) is being provided for VTE at just over half of eligible encounters. The aim of this paper is to present and discuss the detailed CTA findings for VTE as a baseline for compliance with guidelines at a population level.

SETTING

The setting was 27 hospitals in 2 states of Australia.

PARTICIPANTS

A sample of participants designed to be representative of the Australian population was recruited. Participants who had been admitted overnight during 2009 and/or 2010 were eligible. Of the 1154 CTA participants, 481(42%) were admitted overnight to hospital at least once, comprising 751 admissions. There were 279 females (58%), and the mean age was 64 years.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary measure was compliance with indicators of appropriate care for VTE. The indicators were extracted from Australian VTE clinical practice guidelines and ratified by experts. Participants' medical records from 2009 to 2010 were analysed for compliance with 38 VTE indicators.

RESULTS

Of the 35,145 CTA encounters, 1078 (3%) were eligible for scoring against VTE indicators. There were 2-84 eligible encounters per indicator at 27 hospitals. Overall compliance with indicators for VTE was 51%, and ranged from 34% to 64% for aggregated sets of indicators.

CONCLUSIONS

The prevention and management of VTE was appropriate for only half of the at-risk patients in our sample; this provides a baseline for tracking progress nationally. There is a need for national and, ideally, international agreement on clinical standards, indicators and tools to guide, document and monitor care for VTE, and for measures to increase their uptake, particularly where deficiencies have been identified.

Protocol: simulation training to improve 9-1-1 dispatcher identification of cardiac arrest

Background

9-1-1 dispatchers are often the first contact for bystanders witnessing an out-of-hospital cardiac arrest. In the time before Emergency Medical Services arrives, dispatcher identification of the need for, and provision of Telephone-CPR (T-CPR) can improve survival. Our study aims to evaluate the use of phone-based standardized patient simulation training to improve identification of the need for T-CPR and shorten time to start of T-CPR instructions.

Methods/Design

The STAT-911 study is a randomized controlled trial. We will recruit 160 dispatchers from 9-1-1 call-centers in the Pacific Northwest; they are randomized to an intervention or control group. Intervention participants complete four telephone simulation training sessions over 6–8 months. Training sessions consist of three mock 9-1-1 calls, with a standardized patient playing a caller witnessing a medical emergency. After the mock calls, an instructor who has been listening in and scoring the dispatcher’s call management, connects to the dispatcher and provides feedback on select call processing skills. After the last training session, all participants complete the simulation test: a call session that includes two mock 9-1-1 calls of medium complexity. During the study, audio from all actual cardiac arrest calls handled by the dispatchers will be collected. All dispatchers complete a baseline survey, and after the intervention, a follow-up survey to measure confidence. Primary outcomes are proportion of calls where dispatchers identify the need for T-CPR, and time to start of T-CPR, assessed by comparing performance on two calls in the simulation test. Secondary outcomes are proportion of actual cardiac arrest calls in which dispatchers identify the need for T-CPR and time to start of T-CPR; performance on call-taking skills during the simulation test; self-reported confidence in the baseline and follow-up surveys; and calculated costs of the intervention training sessions and projected costs for field implementation of training sessions.

Discussion

The STAT-911 study will evaluate if over-the-phone simulation training with standardized patients can improve 9-1-1 dispatchers’ ability identify the need for, and promptly begin T-CPR. Furthermore, it will advance knowledge on the effectiveness of simulation training for health services phone-operators interacting with clients, patients, or bystanders in diagnosis, triage, and treatment decisions.

Trial registration

ClinicalTrials.gov Registration Number: NCT01972087. Registered 23 October 2013.

Use of standardised patients to assess quality of tuberculosis care: a pilot, cross-sectional study

BACKGROUND

Existing studies of the quality of tuberculosis care have relied on recall-based patient surveys, questionnaire surveys of knowledge, and prescription or medical record analysis, and the results mostly show the health-care provider's knowledge rather than actual practice. No study has used standardised patients to assess clinical practice. Therefore we aimed to assess quality of care for tuberculosis using such patients.

METHODS

We did a pilot, cross-sectional validation study of a convenience sample of consenting private health-care providers in low-income and middle-income areas of Delhi, India. We recruited standardised patients in apparently good health from the local community to present four cases (two of presumed tuberculosis and one each of confirmed tuberculosis and suspected multidrug-resistant tuberculosis) to a randomly allocated health-care provider. The key objective was to validate the standardised-patient method using three criteria: negligible risk and ability to avoid adverse events for providers and standardised patients, low detection rates of standardised patients by providers, and data accuracy across standardised patients and audio verification of standardised-patient recall. We also used medical vignettes to assess providers' knowledge of presumed tuberculosis. Correct case management was benchmarked using Standards for Tuberculosis Care in India (STCI).

FINDINGS

Between Feb 2, and March 28, 2014, we recruited and trained 17 standardised patients who had 250 interactions with 100 health-care providers, 29 of whom were qualified in allopathic medicine (ie, they had a Bachelor of Medicine & Surgery [MBBS] degree), 40 of whom practised alternative medicine, and 31 of whom were informal health-care providers with few or no qualifications. The interactions took place between April 1, and April 23, 2014. The proportion of detected standardised patients was low (11 [5%] detected out of 232 interactions among providers who completed the follow-up survey), and standardised patients' recall correlated highly with audio recordings (r=0·63 [95% CI 0·53-0·79]), with no safety concerns reported. The mean consultation length was 6 min (95% CI 5·5-6·6) with a mean of 6·18 (5·72-6·64) questions or examinations completed, representing 35% (33-38) of essential checklist items. Across all cases, only 52 (21% [16-26]) of 250 were correctly managed. Correct management was higher among MBBS-qualified doctors than other types of health-care provider (adjusted odds ratio 2·41 [95% CI 1·17-4·93]; p=0·0166). Of the 69 providers who completed the vignette, knowledge in the vignettes was more consistent with STCI than their actual clinical practice-eg, 50 (73%) ordered a chest radiograph or sputum test during the vignette compared with seven (10%) during the standardised-patient interaction; OR 0·04 (95% CI 0·02-0·11); p<0·0001.

INTERPRETATION

Standardised patients can be successfully implemented to assess tuberculosis care. Our data suggest a big gap between private provider knowledge and practice. Additional work is needed to substantiate our pilot data, understand the know-do gap in provider behaviour, and to identify the best approach to measure and improve the quality of tuberculosis care in India.

FUNDING

Grand Challenges Canada, the Bill & Melinda Gates Foundation, Knowledge for Change Program, and the World Bank Development Research Group.

A trial of e-simulation of sudden patient deterioration (FIRST2ACT WEB) on student learning

BACKGROUND

High-fidelity simulation pedagogy is of increasing importance in health professional education; however, face-to-face simulation programs are resource intensive and impractical to implement across large numbers of students.

OBJECTIVES

To investigate undergraduate nursing students' theoretical and applied learning in response to the e-simulation program-FIRST2ACT WEBTM, and explore predictors of virtual clinical performance.

DESIGN AND SETTING

Multi-center trial of FIRST2ACT WEBTM accessible to students in five Australian universities and colleges, across 8 campuses.

PARTICIPANTS

A population of 489 final-year nursing students in programs of study leading to license to practice.

METHODS

Participants proceeded through three phases: (i) pre-simulation-briefing and assessment of clinical knowledge and experience; (ii) e-simulation-three interactive e-simulation clinical scenarios which included video recordings of patients with deteriorating conditions, interactive clinical tasks, pop up responses to tasks, and timed performance; and (iii) post-simulation feedback and evaluation. Descriptive statistics were followed by bivariate analysis to detect any associations, which were further tested using standard regression analysis.

RESULTS

Of 409 students who commenced the program (83% response rate), 367 undergraduate nursing students completed the web-based program in its entirety, yielding a completion rate of 89.7%; 38.1% of students achieved passing clinical performance across three scenarios, and the proportion achieving passing clinical knowledge increased from 78.15% pre-simulation to 91.6% post-simulation. Knowledge was the main independent predictor of clinical performance in responding to a virtual deteriorating patient R(2)=0.090, F(7, 352)=4.962, p<0.001.

DISCUSSION

The use of web-based technology allows simulation activities to be accessible to a large number of participants and completion rates indicate that 'Net Generation' nursing students were highly engaged with this mode of learning.

CONCLUSION

The web-based e-simulation program FIRST2ACTTM effectively enhanced knowledge, virtual clinical performance, and self-assessed knowledge, skills, confidence, and competence in final-year nursing students.

Three methods of delivering clinic-based training on syndromic management of sexually transmitted diseases in South Africa: a pilot study

Introduction

The South African National Department of Health sought to improve syndromic management of sexually transmitted infections (STIs). Continuing medical education on STIs was delivered at primary healthcare (PHC) clinics using one of three training methods: (1) lecture, (2) computer and (3) paper-based. Clinics with training were compared with control clinics.

Methods

Ten PHC clinics were randomly assigned to control and 10 to each training method arm. Clinicians participated in on-site training on six modules; two per week for three weeks. Each clinic was visited by three or four unannounced standardised patient (SP) actors pre-training and post-training. Male SPs reported symptoms of male urethritis syndrome and female SPs reported symptoms of vaginal discharge syndrome. Quality of healthcare was measured by whether or not clinicians completed five tasks: HIV test, genital exam, correct medications, condoms and partner notification.

Results

An average of 31% of clinicians from each PHC attended each module. Quality of STI care was low. Pre-training (n=128) clinicians completed an average of 1.63 tasks. Post-training (n=114) they completed 1.73. There was no change in the number of STI tasks completed in the control arm and an 11% increase overall in the training arms relative to the control (ratio of relative risk (RRR)=1.11, 95% CI 0.67 to 1.84). Across training arms, there was a 26% increase (RRR=1.26, 95% CI 0.77 to 2.06) associated with lecture, 17% increase (RRR=1.17, 95% CI 0.59 to 2.28) with paper-based and 13% decrease (RRR=0.87, 95% CI 0.40 to 1.90) with computer arm relative to the control.

Conclusions

Future interventions should address increasing training attendance and computer-based training effectiveness.

Trial registration number

AEARCTR-0000668.

Methods of Observing Variations in Physicians' Decisions: The Opportunities of Clinical Vignettes

To support their efforts to promote high quality and efficient care, policymakers need to better understand the key factors associated with variations in physicians' decisions, and in particular, physician deviations from evidence-based care. Clinical vignette survey instruments hold potential for research in this area as an approach that both allows for practical, large-scale study and overcomes the data quality challenges posed by analysis of clinical data. These surveys present respondents with a narrative description of a hypothetical patient case and solicit responses to one or more questions regarding the care of the patient. In this review, we describe various methods for measuring variations in physicians' decisions and highlight a range of design features researchers should consider when developing a clinical vignette survey. We conclude by identifying areas for future research.

Assessing medical student empathy in a family medicine clinical test: validity of the CARE measure

INTRODUCTION

The Consultation and Relational Empathy (CARE) measure developed and validated in primary care settings and used for general practitioner appraisal is a 10-item instrument used by patients to assess doctors' empathy. The aim of this study is to investigate the validity of the CARE measure in assessing medical students' empathy during a formative family medicine clinical test.

METHOD

All 158 final-year medical students were assessed by trained simulated patients (SPs) - who completed the CARE measure, the Jefferson Scale of Patient Perceptions of Physician Empathy (JSPPPE), and a global rating score to assess students' empathy and history-taking ability.

RESULTS

Exploratory and confirmatory factor analysis identified a unidimensional structure. The CARE measure strongly correlated with both convergent measures: global rating (ρ=0.79 and <0.001) and JSPPPE (ρ=0.77 and <0.001) and weakly correlated with the divergent measure: history-taking score (ρ=0.28 and <0.001). Internal consistency was excellent (Cronbach's α=0.94).

CONCLUSION

The CARE measure had strong construct and internal reliability in a formative, undergraduate family medicine examination. Its role in higher stakes examinations and other educational settings should be explored.

Team training for safer birth

Effective and coordinated teamworking is key to achieving safe birth for mothers and babies. Confidential enquiries have repeatedly identified deficiencies in teamwork as factors contributing to poor maternal and neonatal outcomes. The ingredients of a successful multi-professional team are varied, but research has identified some fundamental teamwork behaviours, with good communication, proficient leadership and situational awareness at the heart. Simple, evidence-based methods in teamwork training can be seamlessly integrated into a core, mandatory obstetric emergency training. Training should be an enjoyable, inclusive and beneficial experience for members of staff. Training in teamwork can lead to improved clinical outcomes and better birth experience for women.

Assessing treatment-as-usual provided to control groups in adherence trials: Exploring the use of an open-ended questionnaire for identifying behaviour change techniques

OBJECTIVE

Reporting guidelines call for descriptions of control group support in equal detail as for interventions. However, how to assess the active content (behaviour change techniques (BCTs)) of treatment-as-usual (TAU) delivered to control groups in trials remains unclear. The objective of this study is to pre-test a method of assessing TAU in a multicentre cost-effectiveness trial of an HIV-treatment adherence intervention.

DESIGN

HIV-nurses (N = 21) completed a semi-structured open-ended questionnaire enquiring about TAU adherence counselling. Two coders independently coded BCTs.

MAIN OUTCOME MEASURES

Completeness and clarity of nurse responses, inter-coder reliabilities and the type of BCTs reported were examined.

RESULTS

The clarity and completeness of nurse responses were adequate. Twenty-three of the 26 identified BCTs could be reliably coded (mean κ = .79; mean agreement rate = 96%) and three BCTs scored below κ = .60. Total number of BCTs reported per nurse ranged between 7 and 19 (M = 13.86, SD = 3.35).

CONCLUSIONS

This study suggests that the TAU open-ended questionnaire is a feasible and reliable tool to capture active content of support provided to control participants in a multicentre adherence intervention trial. Considerable variability in the number of BCTs provided to control patients was observed, illustrating the importance of reliably collecting and accurately reporting control group support.

Effectiveness of peer assessment for implementing a Dutch physical therapy low back pain guideline: cluster randomized controlled trial

BACKGROUND

Clinical practice guidelines are considered important instruments to improve quality of care. However, success is dependent on adherence, which may be improved using peer assessment, a strategy in which professionals assess performance of their peers in a simulated setting.

OBJECTIVE

The aim of this study was to determine whether peer assessment is more effective than case-based discussions to improve knowledge and guideline-consistent clinical reasoning in the Dutch physical therapy guideline for low back pain (LBP).

DESIGN

A cluster randomized controlled trial was conducted.

SETTING AND PARTICIPANTS

Ten communities of practice (CoPs) of physical therapists were cluster randomized (N=90): 6 CoPs in the peer-assessment group (n=49) and 4 CoPs in the case-based discussion group (control group) (n=41).

INTERVENTION

Both groups participated in 4 educational sessions and used clinical patient cases. The peer-assessment group reflected on performed LBP management in different roles. The control group used structured discussions.

MEASUREMENTS

Outcomes were assessed at baseline and at 6 months. The primary outcome measure was knowledge and guideline-consistent reasoning, measured with 12 performance indicators using 4 vignettes with specific guideline-related patient profiles. For each participant, the total score was calculated by adding up the percentage scores (0-100) per vignette, divided by 4. The secondary outcome measure was reflective practice, as measured by the Self-Reflection and Insight Scale (20-100).

RESULTS

Vignettes were completed by 78 participants (87%). Multilevel analysis showed an increase in guideline-consistent clinical reasoning of 8.4% in the peer-assessment group, whereas the control group showed a decline of 0.1% (estimated group difference=8.7%, 95% confidence interval=3.9 to 13.4). No group differences were found on self-reflection.

LIMITATIONS

The small sample size, a short-term follow-up, and the use of vignettes as a proxy for behavior were limitations of the study.

CONCLUSIONS

Peer assessment leads to an increase in knowledge and guideline-consistent clinical reasoning.

Directly observed care: can unannounced standardized patients address a gap in performance measurement?

There are three potential sources of information for evaluating a clinician's performance: documentation, patient report, and directly observed care. Current measures draw on just two of these: data recorded in the medical record and surveys of patients. Neither captures an array of performance characteristics, including clinician attention to symptoms and signs while taking a history or conducting a physical exam, accurate recording in the medical record of information obtained during the encounter, evidence based communication strategies for preventive care counseling, and effective communication behavior. Unannounced Standardized Patients (USPs) have been widely deployed as a research strategy for systematically uncovering significant performance deficits in each of these areas, but have not been adopted for quality improvement. Likely obstacles include concerns about the ethics of sending health professionals sham patients, the technical challenges of the subterfuge, and concerns about the relatively small sample sizes and substantial costs involved. However, the high frequency of significant and remediable performance deficits unmasked by USPs, and the potential to adapt registration and record keeping systems to accommodate their visits, suggest that their selective and purposeful deployment could be a cost effective and powerful strategy for addressing a gap in performance measurement.

Comparative effectiveness of audit-feedback versus additional physician communication training to improve cancer screening for patients with limited health literacy

BACKGROUND

We designed a continuing medical education (CME) program to teach primary care physicians (PCP) how to engage in cancer risk communication and shared decision making with patients who have limited health literacy (HL).

OBJECTIVE

We evaluated whether training PCPs, in addition to audit-feedback, improves their communication behaviors and increases cancer screening among patients with limited HL to a greater extent than only providing clinical performance feedback.

DESIGN

Four-year cluster randomized controlled trial.

PARTICIPANTS

Eighteen PCPs and 168 patients with limited HL who were overdue for colorectal/breast/cervical cancer screening.

INTERVENTIONS

Communication intervention PCPs received skills training that included standardized patient (SP) feedback on counseling behaviors. All PCPs underwent chart audits of patients' screening status semiannually up to 24 months and received two annual performance feedback reports.

MAIN MEASURES

PCPs experienced three unannounced SP encounters during which SPs rated PCP communication behaviors. We examined between-group differences in changes in SP ratings and patient knowledge of cancer screening guidelines over 12 months; and changes in patient cancer screening rates over 24 months.

KEY RESULTS

There were no group differences in SP ratings of physician communication at baseline. At follow-up, communication intervention PCPs were rated higher in general communication about cancer risks and shared decision making related to colorectal cancer screening compared to PCPs who only received performance feedback. Screening rates increased among patients of PCPs in both groups; however, there were no between-group differences in screening rates except for mammography. The communication intervention did not improve patient cancer screening knowledge.

CONCLUSION

Compared to audit and feedback alone, including PCP communication training increases PCP patient-centered counseling behaviors, but not cancer screening among patients with limited HL. Larger studies must be conducted to determine whether lack of changes in cancer screening were due to clinic/patient sample size versus ineffectiveness of communication training to change outcomes.

The contribution of lifestyle coaching of overweight patients in primary care to more autonomous motivation for physical activity and healthy dietary behaviour: results of a longitudinal study

Background

Combined lifestyle interventions (CLIs) have been advocated as an effective instrument in efforts to reduce overweight and obesity. The odds of maintaining higher levels of physical activity (PA) and healthier dietary behaviour improve when people are more intrinsically motivated to change their behaviour. To promote the shift towards more autonomous types of motivation, facilitator led CLIs have been developed including lifestyle coaching as key element. The present study examined the shift in types of motivation to increase PA and healthy dieting among participants of a primary care CLI, and the contribution of lifestyle coaching to potential changes in motivational quality.

Methods

This prospective cohort study included participants of 29 general practices in the Netherlands that implemented a CLI named ‘BeweegKuur’. Questionnaires including items on demographics, lifestyle coaching and motivation were sent at baseline and after 4 months. Aspects of motivation were assessed with the Behavioural Regulation and Exercise Questionnaire (BREQ-2) and the Regulation of Eating Behaviour Questionnaire (REBS). We performed a drop out analysis to identify selective drop-out. Changes in motivation were analysed with t-tests and effect size interpretations (Cohen’s d), and multivariate regression analysis was used to identify predictors of motivational change.

Results

For physical activity, changes in motivational regulation were fully in line with the tenets of Self Determination Theory and Motivational Interviewing: participants made a shift towards a more autonomous type of motivation (i.e. controlled types of motivation decreased and autonomous types increased). Moreover, an autonomy supportive coaching style was generally found to predict a larger shift in autonomous types of motivation. For healthy dietary behaviour, however, except for a small decrease in external motivation, no favourable changes in different types of motivation were observed. The relation between coaching and motivation appeared to be influenced by the presence of physical activity guidance in the programme.

Conclusions

Motivation of participants of a real life primary care CLI had changed towards a more autonomous motivation after 4 months of intervention. Autonomy-supportive lifestyle coaching contributed to this change with respect to physical activity. Lifestyle coaching for healthy diet requires thorough knowledge about the problem of unhealthy dieting and solid coaching skills.

Assessors for communication skills: SPs or healthcare professionals?

INTRODUCTION

The complexity of modern medicine creates more challenges for teaching and assessment of communication skills in undergraduate medical programme. This research was conducted to study the level of communication skills among undergraduate medical students and to determine the difference between simulated patients and clinical instructors' assessment of communication skills.

METHODS

This comparative study was conducted for three months at the Clinical Skills and Simulation Centre of the International Medical University in Malaysia. The modified Calgary-Cambridge checklist was used to assess the communication skills of 50 first year and 50 second year medical students (five-minutes pre-recorded interview videos on the scenario of sore throat). These videos were reviewed and scored by simulated patients (SPs), communication skills instructors (CSIs) and non-communication skills instructors (non-CSIs).

RESULTS

Better performance was observed among the undergraduate medical students, who had formal training in communication skills with a significant difference in overall scores detected among the first and second year medical students (p = 0.0008). A non-significant difference existed between the scores of SPs and CSIs for Year 1 (p = 0.151).

CONCLUSIONS

The SPs could be trained and involved in assessment of communication skills. Formal training in communication skills is necessary in the undergraduate medical programme.

Unannounced standardized patients: a promising method of assessing patient-centered care in your health care system

BACKGROUND

While unannounced standardized patients (USPs) have been used to assess physicians' clinical skills in the ambulatory setting, they can also provide valuable information on patients' experience of the health care setting beyond the physician encounter. This paper explores the use of USPs as a methodology for evaluating patient-centered care in the health care system.

METHODS

USPs were trained to complete a behaviorally-anchored assessment of core dimensions of patient-centered care delivered within the clinical microsystem, including: 1) Medical assistants' safe practices, quality of care, and responsiveness to patients; 2) ease of clinic navigation; and 3) the patient-centeredness of care provided by the physician. Descriptive data is provided on these three levels of patient-centeredness within the targeted clinical microsystem. Chi-square analyses were used to signal whether variations by teams within the clinical microsystem were likely to be due to chance or might reflect true differences in patient-centeredness of specific teams.

RESULTS

Sixty USP visits to 11 Primary Care teams were performed over an eight-month period (mean 5 visits/team; range 2-8). No medical assistants reported detecting an USP during the study period. USPs found the clinic easy to navigate and that teams were functioning well in 60% of visits. In 30% to 47% of visits, the physicians could have been more patient-centered. Medical assistants' patient safety measures were poor: patient identity was confirmed in only 5% of visits and no USPs observed medical assistants wash their hands. Quality of care was relatively high for vital signs (e.g. blood pressure, weight and height), but low for depression screening, occurring in only 15% of visits. In most visits, medical assistants greeted the patient in a timely fashion but took time to fully explain matters in less than half of the visits and rarely introduced themselves. Physicians tried to help patients navigate the system in 62% of visits.

CONCLUSIONS

USP assessment captured actionable, critical, behaviorally-specific information on team and system performance in an urban community clinic. This methodology provides unique insight into the patient-centeredness and quality of care in medical settings.

Evaluation of a theory-informed implementation intervention for the management of acute low back pain in general medical practice: the IMPLEMENT cluster randomised trial

INTRODUCTION

This cluster randomised trial evaluated an intervention to decrease x-ray referrals and increase giving advice to stay active for people with acute low back pain (LBP) in general practice.

METHODS

General practices were randomised to either access to a guideline for acute LBP (control) or facilitated interactive workshops (intervention). We measured behavioural predictors (e.g. knowledge, attitudes and intentions) and fear avoidance beliefs. We were unable to recruit sufficient patients to measure our original primary outcomes so we introduced other outcomes measured at the general practitioner (GP) level: behavioural simulation (clinical decision about vignettes) and rates of x-ray and CT-scan (medical administrative data). All those not involved in the delivery of the intervention were blinded to allocation.

RESULTS

47 practices (53 GPs) were randomised to the control and 45 practices (59 GPs) to the intervention. The number of GPs available for analysis at 12 months varied by outcome due to missing confounder information; a minimum of 38 GPs were available from the intervention group, and a minimum of 40 GPs from the control group. For the behavioural constructs, although effect estimates were small, the intervention group GPs had greater intention of practising consistent with the guideline for the clinical behaviour of x-ray referral. For behavioural simulation, intervention group GPs were more likely to adhere to guideline recommendations about x-ray (OR 1.76, 95%CI 1.01, 3.05) and more likely to give advice to stay active (OR 4.49, 95%CI 1.90 to 10.60). Imaging referral was not statistically significantly different between groups and the potential importance of effects was unclear; rate ratio 0.87 (95%CI 0.68, 1.10) for x-ray or CT-scan.

CONCLUSIONS

The intervention led to small changes in GP intention to practice in a manner that is consistent with an evidence-based guideline, but it did not result in statistically significant changes in actual behaviour.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry ACTRN012606000098538.

Evaluation of the theory-based Quality Improvement in Physical Therapy (QUIP) programme: a one-group, pre-test post-test pilot study

Background

Guideline adherence in physical therapy is far from optimal, which has consequences for the effectiveness and efficiency of physical therapy care. Programmes to enhance guideline adherence have, so far, been relatively ineffective. We systematically developed a theory-based Quality Improvement in Physical Therapy (QUIP) programme aimed at the individual performance level (practicing physiotherapists; PTs) and the practice organization level (practice quality manager; PQM). The aim of the study was to pilot test the multilevel QUIP programme’s effectiveness and the fidelity, acceptability and feasibility of its implementation.

Methods

A one-group, pre-test, post-test pilot study (N = 8 practices; N = 32 PTs, 8 of whom were also PQMs) done between September and December 2009. Guideline adherence was measured using clinical vignettes that addressed 12 quality indicators reflecting the guidelines’ main recommendations. Determinants of adherence were measured using quantitative methods (questionnaires). Delivery of the programme and management changes were assessed using qualitative methods (observations, group interviews, and document analyses). Changes in adherence and determinants were tested in the paired samples T-tests and expressed in effect sizes (Cohen’s d).

Results

Overall adherence did not change (3.1%; p = .138). Adherence to three quality indicators improved (8%, 24%, 43%; .000 ≤ p ≤ .023). Adherence to one quality indicator decreased (−15.7%; p = .004). Scores on various determinants of individual performance improved and favourable changes at practice organizational level were observed. Improvements were associated with the programme’s multilevel approach, collective goal setting, and the application of self-regulation; unfavourable findings with programme deficits. The one-group pre-test post-test design limits the internal validity of the study, the self-selected sample its external validity.

Conclusions

The QUIP programme has the potential to change physical therapy practice but needs considerable revision to induce the ongoing quality improvement process that is required to optimize overall guideline adherence. To assess its value, the programme needs to be tested in a randomized controlled trial.

Reducing risk in maternity by optimising teamwork and leadership: an evidence-based approach to save mothers and babies

Poor teamwork results in preventable morbidity and mortality for mothers and babies. Suboptimal communication and lack of leadership cost not only lives but also money that is diverted from clinical care to insurance and litigation. Avoidable harm is usually not the result of staff failing their duty of care, it is the result of poor training failing hard-worked staff. A few simple teamwork and leadership behaviours can make a huge difference to outcome and experience for women and their companions, yet they are often missing from maternity care. Recent research has identified the problems and solutions, including the best way to train maternity teams to make a palpable difference. We describe simple yet evidence-based methods to improve teams and leaders.

Comparing announced with unannounced standardized patients in performance assessment

BACKGROUND

Accurately assessing how physicians perform in practice remains an unresolved psychometric challenge. Neither chart reviews nor patient surveys indicate when physicians overlook important information, which can result in a missed opportunity for a correct diagnosis and appropriate plan of care. Standardized patient (SP) assessments provide an opportunity for direct observation of clinical behavior and are increasingly used in licensure examinations. (SPs who are sent incognito are termed unannounced standardized patients [USPs].) One study showed that physicians had particular difficulty adapting care to individual patient context ("contextual error"). In a subsequent study with the same actors, SP cases, and outcomes, an intervention was deployed to reduce contextual error among medical students. In an exploratory reanalysis of data from the two studies, clinicians' assessments of SPs and USPs were compared.

METHODS

Participants in the first study were 65 board-certified internists visited by USPs; the 59 participants in the second were fourth-year medical students examining SPs in a clinical performance center.

RESULTS

Attending physicians measured with USPs significantly underperformed medical students measured with SPs in the probing of biomedical red flags (odds ratio [OR] = 0.45 [0.30 to 0.67]) and contextual red flags (OR = 0.66 [0.45 to 0.99]) and in planning appropriate care (OR = 0.43 [0.27 to 0.67]).

CONCLUSIONS

Across these two studies, attending physicians underperformed medical students on the same outcomes, measured with the same patient cases presented by the same actors. Studies that seek to assess elicitation and incorporation of patient information by physicians as measures of individualization of care should weigh the benefits and costs of direct observation by USPs.

Unannounced standardized patient assessment of the roter interaction analysis system: the challenge of measuring patient-centered communication

BACKGROUND

Despite wide-spread endorsement of patient-centered communication (PCC) in health care, there has been little evidence that it leads to positive change in health outcomes. The lack of correlation may be due either to an overestimation of the value of PCC or to a measurement problem. If PCC measures do not capture elements of the interaction that determine whether the resulting care plan is patient-centered, they will confound efforts to link PCC to outcomes.

OBJECTIVE

To evaluate whether one widely used measure of PCC, the Roter Interaction Analysis System (RIAS), captures patient-centered care planning.

DESIGN

RIAS was employed in the coding of unannounced standardized patient (USP) encounters that were scripted so that the failure to address patient contextual factors would result in an ineffective plan of care. The design enabled an assessment of whether RIAS can differentiate between communication behavior that does and does not result in a care plan that takes into account a patient's circumstances and needs.

PARTICIPANTS

Eight actors role playing four scripted cases (one African American and one Caucasian for each case) in 399 visits to 111 internal medicine attending physicians.

MAIN MEASURES

RIAS measures included composites for physician utterance types and (in separate models) two different previously applied RIAS patient-centeredness summary composites. The gold standard comparison measure was whether the physician's treatment plan, as abstracted from the visit note, successfully addressed the patient's problem. Mixed effects regression models were used to evaluate the relationship between RIAS measures and USP measured performance, controlling for a variety of design features.

KEY RESULTS

None of the RIAS measures of PCC differentiated encounters in which care planning was patient-centered from care planning in which it was not.

CONCLUSIONS

RIAS, which codes each utterance during a visit into mutually exclusive and exhaustive categories, does not differentiate between conversations leading to and not leading to care plans that accommodate patients' circumstances and needs.