Using standardized patients to measure quality: evidence from the literature and a prospective study

Tuberculosis care provided by private practitioners in an urban setting in Indonesia: Findings from a standardized patient study

In Indonesia, government-owned Community Health Centers (CHCs) spearhead tuberculosis (TB) care at the primary level, but a substantial proportion of individuals with pulmonary TB also seek care from Private Practitioners (PPs). However, little is known about PPs' practice in managing patients with TB-associated symptoms. To avoid bias associated with self-administered surveys, we used standardized patients (SPs) to evaluate PPs' adherence to the national TB guidelines. Four clinical scenarios of individuals presenting complaints suggestive of TB, accompanied by different sputum smear results or TB treatment histories were developed. We assigned 12 trained SPs to PPs practicing in 30 CHC catchment areas in Bandung city, Indonesia. For comparison, two scenarios were also presented to the CHCs. A total of 341 successful SP visits were made to 225 private general practitioners (GPs), 29 private specialists, and 30 CHCs. When laboratory results were not available, adherence to the recommended course of action, i.e., sputum examination, was low among private GPs (31%) and private specialists (20%), while it was requested in 87% of visits to the CHCs. PPs preferred chest X-ray (CXR) in all scenarios, with requests made in 66% of visits to private GPs and 84% of visits to private specialists (vs. 8% CHCs). Prescriptions of incorrect TB drug regimens were reported from 7% and 13% of visits to private GPs and specialists, respectively, versus none of the CHCs. Indonesian PPs have a clear preference for CXR over microbiological testing for triaging presumptive TB patients, and inappropriate prescription of TB drugs is not uncommon. These findings warrant actions to increase awareness among PPs about the importance of microbiological testing and of administering appropriate TB drug regimens. SP studies can be used to assess the impact of these interventions on providers' adherence to guidelines.

Use of standardised patients to assess tuberculosis case management by private pharmacies in Patna, India: A repeat cross-sectional study

As the first point of care for many healthcare seekers, private pharmacies play an important role in tuberculosis (TB) care. However, previous studies in India have showed that private pharmacies commonly dispense symptomatic treatments and broad-spectrum antibiotics over-the-counter (OTC), rather than referring patients for TB testing. Such inappropriate management by pharmacies can delaye TB diagnosis. We assessed medical advice and OTC drug dispensing practices of pharmacists for standardized patients presenting with classic symptoms of pulmonary TB (case 1) and for those with sputum smear positive pulmonary TB (case 2), and examined how practices have changed over time in an urban Indian site. We examined how and whether private pharmacies improved practices for TB in 2019 compared to a baseline study conducted in 2015 in the city of Patna, using the same survey sampling techniques and study staff. The proportion of patient-pharmacist interactions that resulted in correct or ideal management, as well as the proportion of interactions resulting in antibiotic, quinolone, and corticosteroid are presented, with standard errors clustered at the provider level. To assess the difference in case management and the use of drugs across the two cases by round, a difference in difference (DiD) model was employed. A total of 936 SP interactions were completed over both rounds of survey. Our results indicate that across both rounds of data collection, 331 of 936 (35%; 95% CI: 32-38%) of interactions were correctly managed. At baseline, 215 of 500 (43%; 95% CI: 39-47%) of interactions were correctly managed whereas 116 of 436 (27%; 95% CI: 23-31%) were correctly managed in the second round of data collection. Ideal management, where in addition to a referral, patients were not prescribed any potentially harmful medications, was seen in 275 of 936 (29%; 95% CI: 27-32%) of interactions overall, with 194 of 500 (39%; 95% CI: 35-43%) of interactions at baseline and 81 of 436 (19%; 95% CI: 15-22%) in round 2. No private pharmacy dispensed anti-TB medications without a prescription. On average, the difference in correct case management between case 1 vs. case 2 dropped by 20 percent points from baseline to the second round of data collection. Similarly, ideal case management decreased by 26 percentage points between rounds. This is in contrast with the dispensation of medicines, which had the opposite effect between rounds; the difference in dispensation of quinolones between case 1 and case 2 increased by 14 percentage points, as did corticosteroids by 9 percentage points, antibiotics by 25 percentage points and medicines generally by 30 percentage points. Our standardised patient study provides valuable insights into how private pharmacies in an Indian city changed their management of patients with TB symptoms or with confirmed TB over a 5-year period. We saw that overall, private pharmacy performance has weakened over time. However, no OTC dispensation of anti-TB medications occurred in either survey round. As the first point of contact for many care seekers, continued and sustained efforts to engage with Indian private pharmacies should be prioritized.

Direct observation of depression screening: identifying diagnostic error and improving accuracy through unannounced standardized patients

Background Depression is substantially underdiagnosed in primary care, despite recommendations for screening at every visit. We report a secondary analysis focused on depression of a recently completed study using unannounced standardized patients (USPs) to measure and improve provider behaviors, documentation, and subsequent claims for real patients. Methods Unannounced standardized patients presented incognito in 217 visits to 59 primary care providers in 22 New Jersey practices. We collected USP checklists, visit audio recordings, and provider notes after visits; provided feedback to practices and providers based on the first two visits per provider; and compared care and documentation behaviors in the visits before and after feedback. We obtained real patient claims from the study practices and a matched comparison group and compared the likelihood of visits including International Classification of Diseases, 10th Revision (ICD-10) codes for depression before and after feedback between the study and comparison groups. Results Providers significantly improved in their rate of depression screening following feedback [adjusted odds ratio (AOR), 3.41; 95% confidence interval (CI), 1.52-7.65; p = 0.003]. Sometimes expected behaviors were documented when not performed. The proportion of claims by actual patients with depression-related ICD-10 codes increased significantly more from prefeedback to postfeedback in the study group than in matched control group (interaction AOR, 1.41; 95% CI, 1.32-1.50; p < 0.001). Conclusions Using USPs, we found significant performance issues in diagnosis of depression, as well as discrepancies in documentation that may reduce future diagnostic accuracy. Providing feedback based on a small number of USP encounters led to some improvements in clinical performance observed both directly and indirectly via claims.

Disrespectful care in family planning services among youth and adult simulated clients in public sector facilities in Malawi

Background

Provision of high-quality family planning (FP) services improves access to contraceptives. Negative experiences in maternal health have been documented worldwide and likely occur in other services including FP. This study aims to quantify disrespectful care for adult and adolescent women accessing FP in Malawi.

Methods

We used simulated clients (SCs) to measure disrespectful care in a census of public facilities in six districts of Malawi in 2018. SCs visited one provider in each of the 112 facilities: two SCs visits (one adult and one adolescent case scenario) or 224 SC visits total. We measured disrespectful care using a quantitative tool and field notes and report the prevalence and 95% confidence intervals for the indicators and by SC case scenarios contextualized with quotes from the field notes.

Results

Some SCs (12%) were refused care mostly because they did not agree to receive a HIV test or vaccination, or less commonly because the clinic was closed during operating hours. Over half (59%) of the visits did not have privacy. The SCs were not asked their contraceptive preference in 57% of the visits, 28% reported they were not greeted respectfully, and 20% reported interruptions. In 18% of the visits the SCs reported humiliation such as verbal abuse. Adults SCs received poorer counseling compared to the adolescent SCs with no other differences found.

Conclusions

We documented instances of refusal of care, lack of privacy, poor client centered care and humiliating treatment by providers. We recommend continued effort to improve quality of care with an emphasis on client treatment, regular quality assessments that include measurement of disrespectful care, and more research on practices to reduce it.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06353-z.

Comparing quality of primary healthcare between public and private providers in China: study protocol of a cross-sectional study using unannounced standardised patients in seven provinces of China

INTRODUCTION

The Chinese government has encouraged the development of private sector in delivering healthcare, including primary healthcare (PHC) in the new round of national health reform since 2009. However, the debate about the role of the private sector in achieving universal health coverage continues with poor support from theories and empirical evidence. This study intends to compare the quality of PHC services between the private and public providers in seven provinces in China, using unannounced standardised patients (USPs).

METHODS

We are developing and validating 13 USP cases most commonly observed in the PHC setting. Six domains of quality will be assessed by the USP: effectiveness, safety, patient centredness, efficiency, timeliness and equity. The USP will make 2200 visits to 705 public and 521 private PHC institutions across seven provinces, following a multistage clustered sample design. Using each USP-provider encounter as the analytical unit, we will first descriptively compare the raw differences in quality between the private and public providers and then analyse the association of ownership types and quality, using propensity score weighting.

ETHICS AND DISSEMINATION

The study was primarily funded by the National Natural Science Foundation of China (#71974211, #71874116 and # 72074163) and was also supported by the China Medical Board (#16-260, #18-300 and #18-301), and have received ethical approval from Sun Yat-sen University (#2019-024). The validated USP tool and the data collected in this study will be freely available for the public after the primary analysis of the study.

TRIAL REGISTRATION NUMBER

Chinese Clinical Trial Registry: #ChiCTR2000032773.

Lessons on the quality of tuberculosis diagnosis from standardized patients in China, India, Kenya, and South Africa

Standardized patients (SPs) are people who are recruited locally, trained to make identical scripted clinical presentations, deployed incognito to multiple different health care providers, and debriefed using a structured reporting instrument. The use of SPs has increased dramatically as a method for assessing quality of TB care since it was first validated and used for tuberculosis in 2015. This paper summarizes common findings using 3,086 SP-provider interactions involving tuberculosis across various sampling strata in published studies from India, China, South Africa and Kenya. It then discusses the lessons learned from implementing standardized patients in these diverse settings. First, quality is low: relatively few SPs presenting to a health care provider for the first time were given an appropriate diagnostic test, and most were given unnecessary or inappropriate medication. Second, care takes a wide variety of forms – SPs did not generally receive “wait and see” or “symptomatic” care from providers, but they received a medley of care patterns that included broad-spectrum antibiotics as well as contraindicated quinolone antibiotics and steroids. Third, there is a wide range of estimated quality in each observed sampling stratum: more-qualified providers and higher-level facilities performed better than others in all settings, but in every stratum there were both high- and low-quality providers. Evidence from SP studies paired with medical vignettes has shown that providers of all knowledge levels significantly underperform their demonstrated ability with real patients. Finally, providers showed little response to differences in patient identity, but showed strong responses to differences in case presentation that give some clues as to the reasons for these behaviors.

Use of standardised patients to assess gender differences in quality of tuberculosis care in urban India: a two-city, cross-sectional study

BACKGROUND

In India, men are more likely than women to have active tuberculosis but are less likely to be diagnosed and notified to national tuberculosis programmes. We used data from standardised patient visits to assess whether these gender differences occur because of provider practice.

METHODS

We sent standardised patients (people recruited from local populations and trained to portray a scripted medical condition to health-care providers) to present four tuberculosis case scenarios to private health-care providers in the cities of Mumbai and Patna. Sampling and weighting allowed for city representative interpretation. Because standardised patients were assigned to providers by a field team blinded to this study, we did balance and placebo regression tests to confirm standardised patients were assigned by gender as good as randomly. Then, by use of linear and logistic regression, we assessed correct case management, our primary outcome, and other dimensions of care by standardised patient gender.

FINDINGS

Between Nov 21, 2014, and Aug 21, 2015, 2602 clinical interactions at 1203 private facilities were completed by 24 standardised patients (16 men, eight women). We found standardised patients were assigned to providers as good as randomly. We found no differences in correct management by patient gender (odds ratio 1·05; 95% CI 0·76-1·45; p=0·77) and no differences across gender within any case scenario, setting, provider gender, or provider qualification.

INTERPRETATION

Systematic differences in quality of care are unlikely to be a cause of the observed under-representation of men in tuberculosis notifications in the private sector in urban India.

FUNDING

Grand Challenges Canada, Bill & Melinda Gates Foundation, World Bank Knowledge for Change Program.

Assessing the quality of primary healthcare in seven Chinese provinces with unannounced standardised patients: protocol of a cross-sectional survey

INTRODUCTION

Primary healthcare (PHC) serves as the cornerstone for the attainment of universal health coverage (UHC). Efforts to promote UHC should focus on the expansion of access and on healthcare quality. However, robust quality evidence has remained scarce in China. Common quality assessment methods such as chart abstraction, patient rating and clinical vignette use indirect information that may not represent real practice. This study will send standardised patients (SP or healthy person trained to consistently simulate the medical history, physical symptoms and emotional characteristics of a real patient) unannounced to PHC providers to collect quality information and represent real practice.

METHODS AND ANALYSIS

1981 SP-clinician visits will be made to a random sample of PHC providers across seven provinces in China. SP cases will be developed for 10 tracer conditions in PHC. Each case will include a standard script for the SP to use and a quality checklist that the SP will complete after the clinical visit to indicate diagnostic and treatment activities performed by the clinician. Patient-centredness will be assessed according to the Patient Perception of Patient-Centeredness Rating Scale by the SP. SP cases and the checklist will be developed through a standard protocol and assessed for content, face and criterion validity, and test-retest and inter-rater reliability before its full use. Various descriptive analyses will be performed for the survey results, such as a tabulation of quality scores across geographies and provider types.

ETHICS AND DISSEMINATION

This study has been reviewed and approved by the Institutional Review Board of the School of Public Health of Sun Yat-sen University (#SYSU 2017-011). Results will be actively disseminated through print and social media, and SP tools will be made available for other researchers.

The Quality of Tuberculosis Care in Urban Migrant Clinics in China

Large and increasing numbers of rural-to-urban migrants provided new challenges for tuberculosis control in large cities in China and increased the need for high quality tuberculosis care delivered by clinics in urban migrant communities. Based on a household survey in migrant communities, we selected and separated clinics into those that mainly serve migrants and those that mainly serve local residents. Using standardized patients, this study provided an objective comparison of the quality of tuberculosis care delivered by both types of clinics and examined factors related to quality care. Only 27% (95% confidence interval (CI) 14⁻46) of cases were correctly managed in migrant clinics, which is significantly worse than it in local clinics (50%, 95% CI 28⁻72). Clinicians with a base salary were 41 percentage points more likely to demonstrate better case management. Furthermore, clinicians with upper secondary or higher education level charged 20 RMB lower out of pocket fees than less-educated clinicians. In conclusion, the quality of tuberculosis care accessed by migrants was very poor and policies to improve the quality should be prioritized in current health reforms. Providing a base salary was a possible way to improve quality of care and increasing the education attainment of urban community clinicians might reduce the heavy barrier of medical expenses for migrants.

Using smartphone-based virtual patients to assess the quality of primary healthcare in rural China: protocol for a prospective multicentre study

INTRODUCTION

Valid and low-cost quality assessment tools examining care quality are not readily available. The unannounced standardised patient (USP), the gold standard for assessing quality, is costly to implement while the validity of clinical vignettes, as a low-cost alternative, has been challenged. Computerised virtual patients (VPs) create high-fidelity and interactive simulations of doctor-patient encounters which can be easily implemented via smartphone at low marginal cost. Our study aims to develop and validate smartphone-based VP as a quality assessment tool for primary care, compared with USP.

METHODS AND ANALYSIS

The study will be implemented in primary health centres (PHCs) in rural areas of seven Chinese provinces, and physicians practicing at township health centres and village clinics will be our study population. The development of VPs involves three steps: (1) identifying 10 VP cases that can best represent rural PHCs' work, (2) designing each case by a case-specific development team and (3) developing corresponding quality scoring criteria. After being externally reviewed for content validity, these VP cases will be implemented on a smartphone-based platform and will be tested for feasibility and face validity. This smartphone-based VP tool will then be validated for its criterion validity against USP and its reliability (ie, internal consistency and stability), with 1260 VP/USP-clinician encounters across the seven study provinces for all 10 VP cases.

ETHICS AND DISSEMINATION

Sun Yat-sen University: No. 2017-007. Study findings will be published and tools developed will be freely available to low-income and middle-income countries for research purposes.

The Effect of Screen-to-Screen Versus Face-to-Face Consultation on Doctor-Patient Communication: An Experimental Study with Simulated Patients

Background

Despite the emergence of Web-based patient-provider contact, it is still unclear how the quality of Web-based doctor-patient interactions differs from face-to-face interactions.

Objective

This study aimed to examine (1) the impact of a consultation medium on doctors’ and patients’ communicative behavior in terms of information exchange, interpersonal relationship building, and shared decision making and (2) the mediating role of doctors’ and patients’ communicative behavior on satisfaction with both types of consultation medium.

Methods

Doctor-patient consultations on pelvic organ prolapse were simulated, both in a face-to-face and in a screen-to-screen (video) setting. Twelve medical interns and 6 simulated patients prepared 4 different written scenarios and were randomized to perform a total of 48 consultations. Effects of the consultations were measured by questionnaires that participants filled out directly after the consultation.

Results

With respect to patient-related outcomes, satisfaction, perceived information exchange, interpersonal relationship building, and perceived shared decision making showed no significant differences between face-to-face and screen-to-screen consultations. Patients’ attitude toward Web-based communication (b=−.249, P=.02 and patients’ perceived time and attention (b=.271, P=.03) significantly predicted patients’ perceived interpersonal relationship building. Patients’ perceived shared decision making was positively related to their satisfaction with the consultation (b=.254, P=.005). Overall, patients experienced significantly greater shared decision making with a female doctor (mean 4.21, SD 0.49) than with a male doctor (mean 3.66 [SD 0.73]; b=.401, P=.009). Doctor-related outcomes showed no significant differences in satisfaction, perceived information exchange, interpersonal relationship building, and perceived shared decision making between the conditions. There was a positive relationship between perceived information exchange and doctors’ satisfaction with the consultation (b=.533, P<.001). Furthermore, doctors’ perceived interpersonal relationship building was positively related to doctors’ satisfaction with the consultation (b=.331, P=.003).

Conclusions

In this study, the quality of doctor-patient communication, as indicated by information exchange, interpersonal relationship building, and shared decision making, did not differ significantly between Web-based and face-to-face consultations. Doctors and simulated patients were equally satisfied with both types of consultation medium, and no differences were found in the manner in which participants perceived communicative behavior during these consultations. The findings suggest that worries about a negative impact of Web-based video consultation on the quality of patient-provider consultations seem unwarranted as they offer the same interaction quality and satisfaction level as regular face-to-face consultations.

Assessment of the effect of an Interactive Dynamic Referral Interface (IDRI) on the quality of referral letters from general practitioners to gastroenterologists: a randomised cross-over vignette trial

OBJECTIVES

We evaluated whether interactive, electronic, dynamic, diagnose-specific checklists improve the quality of referral letters in gastroenterology and assessed the general practitioners' (GPs') acceptance of the checklists.

DESIGN

Randomised cross-over vignette trial.

SETTING

Primary care in Norway.

PARTICIPANTS

25 GPs.

INTERVENTION

The GPs participated in the trial and were asked to refer eight clinical vignettes in an internet-based electronic health record simulator. A referral support, consisting of dynamic diagnose-specific checklists, was created for the generation of referral letters to gastroenterologists. The GPs were randomised to refer the eight vignettes with or without the checklists. After a minimum of 3 months, they repeated the referral process with the alternative method.

MAIN OUTCOME MEASURES

Difference in quality of the referral letters between referrals with and without checklists, measured with an objective Thirty Point Score (TPS).Difference in variance in the quality of the referral letters and GPs' acceptance of the electronic dynamic user interface.

RESULTS

The mean TPS was 15.2 (95% CI 13.2 to 16.3) and 22.0 (95% CI 20.6 to 22.8) comparing referrals without and with checklist assistance (p<0.001), respectively. The coefficient of variance was 23.3% for the checklist group and 39.6% for the non-checklist group. Two-thirds (16/24) of the GPs thought they had included more relevant information in the referrals with checklists, and considered implementing this type of checklists in their clinical practices, if available.

CONCLUSIONS

Dynamic, diagnose-specific checklists improved the quality of referral letters significantly and reduced the variance of the TPS, indicating a more uniform quality when checklists were used. The GPs were generally positive to the checklists.

Providing Support to Families considering the Option of Organ Donation: An Innovative Training Method

Families must make decisions about organ donation for a loved one during intensely emotional circumstances in the hospital, where support from others is crucial to their coping. Research about families' experiences during the decision-making process regarding donating their loved ones' organs has shown that quality of hospital care and receiving psychosocial support are important factors influencing their decision. Typically, a donation coordinator from the local organ procurement organization approaches the family about the option of donation, whereas the role of medical and nursing staff is to convey diagnostic and prognostic information to the family. Currently, no requirement is in place for training of professional staff in communication skills for approaching and interacting with families about organ donation. This article discusses a simulated training method in empathic communication used for supporting families who are approached about organ donation. This innovative method can be adapted to and should be tested with professional audiences.

Do Health Reforms to Improve Quality Have Long-Term Effects? Results of a Follow-Up on a Randomized Policy Experiment in the Philippines

We tracked doctors who had previously participated in a randomized policy experiment in the Philippines. The original experiment involved 30 district hospitals divided equally into one control site and two intervention sites that increased insurance payments (full insurance support for children under 5 years old) or made bonus payments to hospital staff. During the 3 years of the intervention, quality-as measured by clinical performance and value vignettes-improved and was sustained in both intervention sites compared with controls. Five years after the interventions were discontinued, we remeasured the quality of care of the doctors. We found that the intervention sites continued to have significantly higher quality compared with the control sites. The previously documented quality improvement in intervention sites appears to be sustained; moreover, it was subject to a very low (less than 1% per year) rate of decay in quality scores.

Use of standardised patients to assess quality of tuberculosis care: a pilot, cross-sectional study

BACKGROUND

Existing studies of the quality of tuberculosis care have relied on recall-based patient surveys, questionnaire surveys of knowledge, and prescription or medical record analysis, and the results mostly show the health-care provider's knowledge rather than actual practice. No study has used standardised patients to assess clinical practice. Therefore we aimed to assess quality of care for tuberculosis using such patients.

METHODS

We did a pilot, cross-sectional validation study of a convenience sample of consenting private health-care providers in low-income and middle-income areas of Delhi, India. We recruited standardised patients in apparently good health from the local community to present four cases (two of presumed tuberculosis and one each of confirmed tuberculosis and suspected multidrug-resistant tuberculosis) to a randomly allocated health-care provider. The key objective was to validate the standardised-patient method using three criteria: negligible risk and ability to avoid adverse events for providers and standardised patients, low detection rates of standardised patients by providers, and data accuracy across standardised patients and audio verification of standardised-patient recall. We also used medical vignettes to assess providers' knowledge of presumed tuberculosis. Correct case management was benchmarked using Standards for Tuberculosis Care in India (STCI).

FINDINGS

Between Feb 2, and March 28, 2014, we recruited and trained 17 standardised patients who had 250 interactions with 100 health-care providers, 29 of whom were qualified in allopathic medicine (ie, they had a Bachelor of Medicine & Surgery [MBBS] degree), 40 of whom practised alternative medicine, and 31 of whom were informal health-care providers with few or no qualifications. The interactions took place between April 1, and April 23, 2014. The proportion of detected standardised patients was low (11 [5%] detected out of 232 interactions among providers who completed the follow-up survey), and standardised patients' recall correlated highly with audio recordings (r=0·63 [95% CI 0·53-0·79]), with no safety concerns reported. The mean consultation length was 6 min (95% CI 5·5-6·6) with a mean of 6·18 (5·72-6·64) questions or examinations completed, representing 35% (33-38) of essential checklist items. Across all cases, only 52 (21% [16-26]) of 250 were correctly managed. Correct management was higher among MBBS-qualified doctors than other types of health-care provider (adjusted odds ratio 2·41 [95% CI 1·17-4·93]; p=0·0166). Of the 69 providers who completed the vignette, knowledge in the vignettes was more consistent with STCI than their actual clinical practice-eg, 50 (73%) ordered a chest radiograph or sputum test during the vignette compared with seven (10%) during the standardised-patient interaction; OR 0·04 (95% CI 0·02-0·11); p<0·0001.

INTERPRETATION

Standardised patients can be successfully implemented to assess tuberculosis care. Our data suggest a big gap between private provider knowledge and practice. Additional work is needed to substantiate our pilot data, understand the know-do gap in provider behaviour, and to identify the best approach to measure and improve the quality of tuberculosis care in India.

FUNDING

Grand Challenges Canada, the Bill & Melinda Gates Foundation, Knowledge for Change Program, and the World Bank Development Research Group.

The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India

IMPORTANCE

In rural India, as in many developing countries, childhood mortality remains high and the quality of health care available is low. Improving care in such settings, where most health care practitioners do not have formal training, requires an assessment of the practitioners' knowledge of appropriate care and the actual care delivered (the know-do gap).

OBJECTIVE

To assess the knowledge of local health care practitioners and the quality of care provided by them for childhood diarrhea and pneumonia in rural Bihar, India.

DESIGN, SETTING, AND PARTICIPANTS

We conducted an observational, cross-sectional study of the knowledge and practice of 340 health care practitioners concerning the diagnosis and treatment of childhood diarrhea and pneumonia in Bihar, India, from June 29 through September 8, 2012. We used data from vignette interviews and unannounced standardized patients (SPs).

MAIN OUTCOMES AND MEASURES

For SPs and vignettes, practitioner performance was measured using the numbers of key diagnostic questions asked and examinations conducted. The know-do gap was calculated by comparing fractions of practitioners asking key diagnostic questions on each method. Multivariable regressions examined the relation among diagnostic performance, prescription of potentially harmful treatments, and the practitioners' characteristics. We also examined correct treatment recommended by practitioners with both methods.

RESULTS

Practitioners asked a mean of 2.9 diagnostic questions and suggested a mean of 0.3 examinations in the diarrhea vignette; mean numbers were 1.4 and 0.8, respectively, for the pneumonia vignette. Although oral rehydration salts, the correct treatment for diarrhea, are commonly available, only 3.5% of practitioners offered them in the diarrhea vignette. With SPs, no practitioner offered the correct treatment for diarrhea, and 13.0% of practitioners offered the correct treatment for pneumonia. Diarrhea treatment has a large know-do gap; practitioners asked diagnostic questions more frequently in vignettes than for SPs. Although only 20.9% of practitioners prescribed treatments that were potentially harmful in the diarrhea vignettes, 71.9% offered them to SPs (P < .001). Unqualified practitioners were more likely to prescribe potentially harmful treatments for diarrhea (adjusted odds ratio, 5.11 [95% CI, 1.24-21.13]). Higher knowledge scores were associated with better performance for treating diarrhea but not pneumonia.

CONCLUSIONS AND RELEVANCE

Practitioners performed poorly with vignettes and SPs, with large know-do gaps, especially for childhood diarrhea. Efforts to improve health care for major causes of childhood mortality should emphasize strategies that encourage pediatric health care practitioners to diagnose and manage these conditions correctly through better monitoring and incentives in addition to practitioner training initiatives.

Directly observed care: can unannounced standardized patients address a gap in performance measurement?

There are three potential sources of information for evaluating a clinician's performance: documentation, patient report, and directly observed care. Current measures draw on just two of these: data recorded in the medical record and surveys of patients. Neither captures an array of performance characteristics, including clinician attention to symptoms and signs while taking a history or conducting a physical exam, accurate recording in the medical record of information obtained during the encounter, evidence based communication strategies for preventive care counseling, and effective communication behavior. Unannounced Standardized Patients (USPs) have been widely deployed as a research strategy for systematically uncovering significant performance deficits in each of these areas, but have not been adopted for quality improvement. Likely obstacles include concerns about the ethics of sending health professionals sham patients, the technical challenges of the subterfuge, and concerns about the relatively small sample sizes and substantial costs involved. However, the high frequency of significant and remediable performance deficits unmasked by USPs, and the potential to adapt registration and record keeping systems to accommodate their visits, suggest that their selective and purposeful deployment could be a cost effective and powerful strategy for addressing a gap in performance measurement.

A comparison of the scorings of real and standardized patients on physician communication skills

Objectives: To compare the scorings of real and standardized patients on physician communication skills.

Methods: Patient scoring (n=183) on physicians’ communication skills was determined by 93 real and 90 standardized patients. Eighty physicians (42 specialists and 38 general physicians) in private practice were enrolled. Data were analyzed using self administered questionnaires and checklists including 16 close ended questions.

Results: Twelve percent of patients were not satisfied with the physician communication skills. Poor communication skills were more reported by male patients and those with a higher educational level. The physician communication skill received a higher score with increase of age of patients. A good physician’s communication skill was reported more by married patients. A good physician’s communication skill was significantly more in female doctors, in general physicians and in doctors wearing a White Coat. Real patients scored physician’s communication skills higher than standardized patients.

Conclusion: It is important that physicians try to learn the principles of a good physician-patient communication skill. Therefore, providing medical educational programs on the role of a good doctor and patient relationship at all levels for the doctors and applying them in their clinical practice seem necessary to improve the physician communication skills.

Survey using incognito standardized patients shows poor quality care in China's rural clinics

Over the past decade, China has implemented reforms designed to expand access to health care in rural areas. Little objective evidence exists, however, on the quality of that care. This study reports results from a standardized patient study designed to assess the quality of care delivered by village clinicians in rural China. To measure quality, we recruited individuals from the local community to serve as undercover patients and trained them to present consistent symptoms of two common illnesses (dysentery and angina). Based on 82 covert interactions between the standardized patients and local clinicians, we find that the quality of care is low as measured by adherence to clinical checklists and the rates of correct diagnoses and treatments. Further analysis suggests that quality is most strongly correlated with provider qualifications. Our results highlight the need for policy action to address the low quality of care delivered by grassroots providers.

Training and validation of standardized patients for evaluation of general practitioners' performance in management of obesity and overweight

BACKGROUND

Standardized patient (SP) can serve as a valuable tool to measure the physician performance in actual clinical settings, but it has not been validated for obesity/overweight disorders. This study has been conducted to describe the process of creating reliable and valid SPs for evaluation of general-practitioners' management of obesity/overweight in Iran.

MATERIALS AND METHODS

A total of 6 obese/overweight volunteers (potential SPs) took part in training. Three scenarios, along with corresponding checklists including 102 items representing different aspects of diagnosis and treatment of obesity/overweight, were developed by an expert group. The SPs were trained using role playing method. During this part, one of the SPs failed. The SPs' portrayal of their respective scenario was online watched in another room and the checklist filled independently by the physician, research assistant and other SPs. The reliability of the checklist to be used by the SPs was assessed by Cronbach's alpha. The overall inter-rater agreement was calculated by the intraclass correlation coefficient statistic for total scores.

RESULTS

The 5 eligible SPs were all women between 20 years and 39 years of age. Inter-rater agreement between the SPs' total scores was 0.899, value (95% confidence intervals) were 11.8 (0.68-0.98) and P value was <0.001. The Cronbach's alpha for reliability of completed checklists was 0.91.

CONCLUSION

SP could be a powerful instrument for evaluating medical performance of general practitioners in the field of obesity/overweight management. Further research is needed to find the more aspects of training and validation of unannounced SPs in this field.

In urban and rural India, a standardized patient study showed low levels of provider training and huge quality gaps

This article reports on the quality of care delivered by private and public providers of primary health care services in rural and urban India. To measure quality, the study used standardized patients recruited from the local community and trained to present consistent cases of illness to providers. We found low overall levels of medical training among health care providers; in rural Madhya Pradesh, for example, 67 percent of health care providers who were sampled reported no medical qualifications at all. What's more, we found only small differences between trained and untrained doctors in such areas as adherence to clinical checklists. Correct diagnoses were rare, incorrect treatments were widely prescribed, and adherence to clinical checklists was higher in private than in public clinics. Our results suggest an urgent need to measure the quality of health care services systematically and to improve the quality of medical education and continuing education programs, among other policy changes.

Unannounced standardized patient assessment of the roter interaction analysis system: the challenge of measuring patient-centered communication

BACKGROUND

Despite wide-spread endorsement of patient-centered communication (PCC) in health care, there has been little evidence that it leads to positive change in health outcomes. The lack of correlation may be due either to an overestimation of the value of PCC or to a measurement problem. If PCC measures do not capture elements of the interaction that determine whether the resulting care plan is patient-centered, they will confound efforts to link PCC to outcomes.

OBJECTIVE

To evaluate whether one widely used measure of PCC, the Roter Interaction Analysis System (RIAS), captures patient-centered care planning.

DESIGN

RIAS was employed in the coding of unannounced standardized patient (USP) encounters that were scripted so that the failure to address patient contextual factors would result in an ineffective plan of care. The design enabled an assessment of whether RIAS can differentiate between communication behavior that does and does not result in a care plan that takes into account a patient's circumstances and needs.

PARTICIPANTS

Eight actors role playing four scripted cases (one African American and one Caucasian for each case) in 399 visits to 111 internal medicine attending physicians.

MAIN MEASURES

RIAS measures included composites for physician utterance types and (in separate models) two different previously applied RIAS patient-centeredness summary composites. The gold standard comparison measure was whether the physician's treatment plan, as abstracted from the visit note, successfully addressed the patient's problem. Mixed effects regression models were used to evaluate the relationship between RIAS measures and USP measured performance, controlling for a variety of design features.

KEY RESULTS

None of the RIAS measures of PCC differentiated encounters in which care planning was patient-centered from care planning in which it was not.

CONCLUSIONS

RIAS, which codes each utterance during a visit into mutually exclusive and exhaustive categories, does not differentiate between conversations leading to and not leading to care plans that accommodate patients' circumstances and needs.

Evaluating health worker performance in Benin using the simulated client method with real children

BACKGROUND

The simulated client (SC) method for evaluating health worker performance utilizes surveyors who pose as patients to make surreptitious observations during consultations. Compared to conspicuous observation (CO) by surveyors, which is commonly done in developing countries, SC data better reflect usual health worker practices. This information is important because CO can cause performance to be better than usual. Despite this advantage of SCs, the method's full potential has not been realized for evaluating performance for pediatric illnesses because real children have not been utilized as SCs. Previous SC studies used scenarios of ill children that were not actually brought to health workers. During a trial that evaluated a quality improvement intervention in Benin (the Integrated Management of Childhood Illness [IMCI] strategy), we conducted an SC survey with adult caretakers as surveyors and real children to evaluate the feasibility of this approach and used the results to assess the validity of CO.

METHODS

We conducted an SC survey and a CO survey (one right after the other) of health workers in the same 55 health facilities. A detailed description of the SC survey process was produced. Results of the two surveys were compared for 27 performance indicators using logistic regression modeling.

RESULTS

SC and CO surveyors observed 54 and 185 consultations, respectively. No serious problems occurred during the SC survey. Performance levels measured by CO were moderately higher than those measured by SCs (median CO - SC difference = 16.4 percentage-points). Survey differences were sometimes much greater for IMCI-trained health workers (median difference = 29.7 percentage-points) than for workers without IMCI training (median difference = 3.1 percentage-points).

CONCLUSION

SC surveys can be done safely with real children if appropriate precautions are taken. CO can introduce moderately large positive biases, and these biases might be greater for health workers exposed to quality improvement interventions.

TRIAL NUMBER

http://clinicaltrials.gov Identifier NCT00510679.

Development and preliminary reliability testing of an assessment of patient independence in performing a treatment program: standardized scenarios

BACKGROUND

Physical therapists often assess patient independence through observation; however, it is not known if therapists make these judgments reliably. We have developed a standardized method to assess a patient's ability to perform his or her treatment program independently.

OBJECTIVES

To develop a standardized assessment of patient independence in performance of a treatment program and examine the intra- and inter-rater reliability decisions made by two physical therapists.

DESIGN

Test-retest.

METHODS

An assessment of patient independence in performance was developed. Standardized patient scenarios were used to assess the intra- and inter-tester reliability of two physical therapists. Percentage of agreement (%) and kappa's coefficient (k and k(w)) indexed rater reliability.

RESULTS

Intra-rater reliability of therapist 1 was as follows: knowledge: % = 95, k = 0.90; performance: % = 95, k(w) = 0.82. Intra-rater reliability of therapist 2 was as follows: knowledge: % = 85, k = 0.68; performance: % = 94, k(w) = 0.80. Inter-rater reliability for knowledge was % = 91 and k = 0.79 and for performance was % = 91 and k(w) = 0.72.

CONCLUSION

Trained therapists displayed substantial to excellent intra-rater reliability and substantial inter-rater reliability in assessing a patient's independence in a treatment program. :

Enhancing objectivity in pastoral education: use of standardized patients in video simulation

Measurement of student learning outcomes in clinical pastoral education has historically been assessed through student self-report instruments and subjective evaluation by CPE supervisors. Valid quantitative measurement is limited. This study utilizes standardized patients and a behavioral interview checklist to measure the impact of an initial summer unit of CPE training. Results demonstrate measurable change in student interview style behavior suggesting that simulation utilizing standardized patients is a valid objective measure of students' behavioral communication styles.

Virtual standardized patients: an interactive method to examine variation in depression care among primary care physicians

BACKGROUND: Some primary care physicians provide less than optimal care for depression (Kessler et al., Journal of the American Medical Association 291, 2581-90, 2004). However, the literature is not unanimous on the best method to use in order to investigate this variation in care. To capture variations in physician behaviour and decision making in primary care settings, 32 interactive CD-ROM vignettes were constructed and tested. AIM AND METHOD: The primary aim of this methods-focused paper was to review the extent to which our study method - an interactive CD-ROM patient vignette methodology - was effective in capturing variation in physician behaviour. Specifically, we examined the following questions: (a) Did the interactive CD-ROM technology work? (b) Did we create believable virtual patients? (c) Did the research protocol enable interviews (data collection) to be completed as planned? (d) To what extent was the targeted study sample size achieved? and (e) Did the study interview protocol generate valid and reliable quantitative data and rich, credible qualitative data? FINDINGS: Among a sample of 404 randomly selected primary care physicians, our voice-activated interactive methodology appeared to be effective. Specifically, our methodology - combining interactive virtual patient vignette technology, experimental design, and expansive open-ended interview protocol - generated valid explanations for variations in primary care physician practice patterns related to depression care.

"Could this be something serious?" Reassurance, uncertainty, and empathy in response to patients' expressions of worry

BACKGROUND

Previous work suggests that exploration and validation of patients' concerns is associated with greater patient trust, lower health care costs, improved counseling, and more guideline-concordant care.

OBJECTIVE

To describe physicians' responses to patients' worries, how their responses varied according to clinical context (straightforward versus medically unexplained symptoms [MUS]) and associations between their responses and patients' ratings of interpersonal aspects of care.

DESIGN

Multimethod study. For each physician, we surveyed 50 current patients and covertly audiorecorded 2 unannounced standardized patient (SP) visits. SPs expressed worry about "something serious" in 2 scenarios: straightforward gastroesophageal reflux or poorly characterized chest pain with MUS.

PARTICIPANTS

One hundred primary care physicians and 4,746 patients.

MEASUREMENTS

Patient surveys measuring interpersonal aspects of care (trust, physician knowledge of the patient, satisfaction, and patient activation). Qualitative coding of 189 transcripts followed by descriptive, multivariate, and lag-sequential analyses.

RESULTS

Physicians offered a mean of 3.1 responses to each of 613 SP prompts. Biomedical inquiry and explanations, action, nonspecific acknowledgment, and reassurance were common, whereas empathy, expressions of uncertainty, and exploration of psychosocial factors and emotions were uncommon. Empathy expressed during SP visits was associated with higher patient ratings of interpersonal aspects of care. After adjusting for demographics and comorbidities, the association was only statistically significant for the MUS role. Empathy was most likely to occur if expressed at the beginning of the conversational sequence.

CONCLUSIONS

Empathy is associated with higher patient ratings of interpersonal care, especially when expressed in situations involving ambiguity. Empathy should be expressed early after patient expressions of worry.

Ratings of physician communication by real and standardized patients

PURPOSE

Patient ratings of physician's patient-centered communication are used by various specialty credentialing organizations and managed care organizations as a measure of physician communication skills. We wanted to compare ratings by real patients with ratings by standardized patients of physician communication.

METHODS

We assessed physician communication using a modified version of the Health Care Climate Questionnaire (HCCQ) among a sample of 100 community physicians. The HCCQ measures physician autonomy support, a key dimension in patient-centered communication. For each physician, the questionnaire was completed by roughly 49 real patients and 2 unannounced standardized patients. Standardized patients portrayed 2 roles: gastroesophageal disorder reflux symptoms and poorly characterized chest pain with multiple unexplained symptoms. We compared the distribution, reliability, and physician rank derived from using real and standardized patients after adjusting for patient, physician, and standardized patient effects.

RESULTS

There were real and standardized patient ratings for 96 of the 100 physicians. Compared with standardized patient scores, real-patient-derived HCCQ scores were higher (mean 22.0 vs 17.2), standard deviations were lower (3.1 vs 4.9), and ranges were similar (both 5-25). Calculated real patient reliability, given 49 ratings per physician, was 0.78 (95% confidence interval [CI], 0.71-0.84) compared with the standardized patient reliability of 0.57 (95% CI, 0.39-0.73), given 2 ratings per physician. Spearman rank correlation between mean real patient and standardized patient scores was positive but small to moderate in magnitude, 0.28.

CONCLUSION

Real patient and standardized patient ratings of physician communication style differ substantially and appear to provide different information about physicians' communication style.

Caught in the act? Prevalence, predictors, and consequences of physician detection of unannounced standardized patients

Objective. To examine the prevalence, predictors, and consequences of physician detection of unannounced standardized patients (SPs) in a study of the impact of direct-to-consumer advertising on treatment for depression. Data Sources. Eighteen trained SPs were randomly assigned to conduct 298 unannounced audio-recorded visits with 152 primary care physicians in three U.S. cities between May 2003 and May 2004. Study Design. Randomized controlled trial using SPs. SPs portrayed six roles, created by crossing two clinical conditions (major depression or adjustment disorder) with three medication request scripts (brand-specific request, general request for an antidepressant, or no request). Data Collection. Within 2 weeks following the visit, physicians completed a form asking whether they "suspected" conducting an office visit with an SP during the past 2 weeks; 296 (99 percent) detection forms were returned. Physicians provided contextual data, a Clinician Background Questionnaire. SPs filled in a Standardized Patient Reporting Form for each visit and returned all written prescriptions and medication samples to the laboratory. Principal Findings. Depending on the definition, detection rates ranged from 5 percent (unambiguous detection) to 23.6 percent (any degree of suspicion) of SP visits. In 12.8 percent of encounters, physicians accurately detected the SP before or during the visit but they only rarely believed their suspicions affected their clinical behavior. In random effects logistic regression analyses controlling for role, actor, physician, and practice factors, suspected visits occurred less frequently in HMO settings than in solo practice settings (p<.05). Physicians more frequently referred SPs to mental health professionals when visits aroused high suspicion (p<.05). Conclusions. Trained actors portrayed patient roles conveying mood disorders at low levels of detection. There was some evidence for differential treatment of detected standardized patients by physicians with regard to referrals but not antidepressant prescribing or follow-up recommendations. Systematic assessment of detection is recommended when SPs are used in studies of clinical process and quality of care.

Connoisseurs of care? Unannounced standardized patients' ratings of physicians

BACKGROUND

Patient satisfaction surveys can be informative, but bias and poor response rates may limit their utility as stable measures of physician performance. Using unannounced standardized patients (SPs) may overcome some of these limitations because their experience and training make them able judges of physician behavior.

OBJECTIVES

We sought to understand the reliability of unannounced SPs in rating primary care physicians when covertly presenting as real patients.

STUDY DESIGN

Data from 2 studies (Patient Centered Communication [PCC]; Social Influences in Practice [SIP]) were included. For the PCC study, 5 SPs made 192 visits to 96 physicians; for the SIP study, 18 SPs made 292 visits to 146 physicians. SPs visits to physicians were randomized, thus avoiding mutual selection bias. Each SP rated 16 to 38 physicians on interpersonal skills (autonomy support: PCC, SIP), technical skills (information gathering: SIP-only), and overall satisfaction (SIP-only). We evaluated SP evaluation consistency (physician vs. total variance rho), and SPs' overall satisfaction with specific dimensions of physician performance.

RESULTS

Scale reliability varied from 0.71 to 0.92. Physician rhos (95% confidence intervals) for autonomy support were 0.40 (0.22-0.58; PCC) and 0.30 (0.14-0.45; SIP); information gathering rho was 0.46 (0.33-0.59; SIP). Overall SP satisfaction rho was 0.47 (0.34-0.60; SIP). SPs varied significantly in adjusted overall satisfaction levels, but not other dimensions.

CONCLUSIONS

These analyses provide some evidence that medical connoisseurship can be learned. When adequately sampled by trained SPs, some physician skills can be reliably measured in community practice settings.

Computerized condition-specific templates for improving care of geriatric syndromes in a primary care setting

INTRODUCTION

As the U.S. population ages, primary care clinicians (PCCs) will encounter more patients with geriatric syndromes, such as urinary incontinence (UI) and falls. Yet, current evidence suggests that care of these conditions does not meet expected standards and that PCCs would benefit from tools to improve care of these conditions. Little is known about the role of computerized condition-specific templates for improving care of geriatric syndromes.

AIM

We sought to develop and assess the usefulness of condition-specific computerized templates in a primary care setting.

SETTING

A large academic Veterans Affairs medical center.

PROGRAM DESCRIPTION

We developed and tested the usefulness of 2 condition-specific computerized templates (UI and falls) that could be added on to an existing electronic health record system.

PROGRAM EVALUATION

Semistructured interviews were used to identify barriers to use of computerized templates. Usefulness and usability were assessed through a randomized-controlled trial involving standardized patients.

DISCUSSION

Use of condition-specific templates resulted in improved history and physical exam assessment for both UI and falls (P < .05). Our computerized, condition-specific templates are a promising method for improving care of geriatric conditions in a primary care setting, but require improvement in usability before widespread implementation.

The Validity and Reliability of the Affective Competency Score to Evaluate Death Disclosure Using Standardized Patients

OBJECTIVE

To explore the validity and reliability of the affective competency score (ACS), compared to a global rating measure to predict overall competency to perform a death disclosure in a standardized patient exercise and to investigate useful thresholds of the ACS.

METHODS

Thirty-seven fourth-year students underwent standardized patient training in death disclosure during a fourth-year emergency medicine clerkship. Students were evaluated using a checklist, an ACS, and a global rating assessment. ACS interrater reliability, interitem reliability, item-total reliability, and split-half reliability were calculated. Area under the curve (AUC) measurements were used to establish criterion validity.

RESULTS

For the ACS, item-total correlations ranged from 0.76 to 0.85, 0.76 to 0.93, and 0.42 to 0.87; the split-half reliability was 0.82 (p = 0.0001), 0.86 (p = 0.0001) and 0.55 (p = 0.0007) for the standardized patient (SP), the faculty and the medical students, respectively. Interitem correlations were adequate. A moderate interrater correlation of the ACS was observed between the faculty observer and the SP (r = 0.47; p = 0.04); however, the medical students' self evaluation did not correlate significantly with either the SP (r = -0.04; p = 0.79), or the faculty observer (r = 0.00; p = 0.99). The AUC for was 0.98 (95% confidence interval [CI] 0.94 to 1.00), 0.87 (95% CI 0.73 to 0.99), and 0.74 (95% CI 0.53 to 0.95) for the faculty, SP, and medical student, respectively.

CONCLUSIONS

The ACS may be a valid, reliable, and useful measure to assess communication skills by faculty or SPs in this setting. At an ACS score of 16, 19, and 21 points for faculty, SPs, and medical students, respectively, there is 100% specificity for the detection of competency assessed on a global rating. However, the ACS appears to have limited reliability and validity when used by medical students.

Web-based learning versus standardized patients for teaching clinical diagnosis: a randomized, controlled, crossover trial

BACKGROUND

Little evidence exists to guide the selection of methods for teaching clinical diagnosis.

PURPOSE

To compare the efficacy, student preference, and cost of a Web-based (WB) program versus a standardized patient (SP) encounter for teaching clinical diagnosis skills to 2nd-year medical students.

METHODS

Randomized, controlled, crossover study comparing WB versus SP-based teaching for the clinical diagnosis of abdominal pain and headache. Outcome measures were performance on a 2-case SP examination (scored on the basis of a checklist completed by a faculty observer and an objective score on a postencounter subjective-objective assessment plan [SOAP] note), format preferences as assessed by end-of-course evaluations, and cost.

RESULTS

Thirty students consented to participate. WB and SP training produced similar scores on both the Abdominal Pain checklist (66% vs. 62%; p = .17) and Headache checklist (56% vs. 63%; p = .07). WB training produced a higher score on the Abdominal Pain SOAP note (69% vs. 47%; p = .006), but not the Headache SOAP note (69% vs. 67%; p = .85). Students rated the SP format higher than the WB format on all 7 preference measures. Start-up costs were estimated at 2,190 dollars for the SP format and 2,250 dollars for the WB format. Ongoing costs per case per student were estimated to be 45 dollars for the SP format and 30 dollars for the WB format.

CONCLUSIONS

WB and SP learning outcomes were comparable, but students preferred the SP format. Start-up costs were comparable, but the ongoing costs of the WB format were less expensive, suggesting that WB teaching may be a viable strategy.

Utilizing theatrical tools in consultation training. A way to facilitate students' reflection on action?

The aim was to give the individual student a group-based opportunity to reflect on possible consultation strategies as the consultation was evolving. An actress acted as patient in the consultation training for a group of 30 students. The consultation was stopped at each critical incidence, and time-out given to allow the students to reflect on possible continuation strategies, and then to carry out one of them. The project was evaluated adopting a pragmatic version of the reflective practitioner research strategy as developed by Taylor. The evaluation was based on tutor and actress's field notes, students' written free text evaluation and students' evaluation through two focus groups. The qualitative analysis resulted in the three categories: the fiction created, temporality manipulated, and students' learning through reflection. Implications for students' learning process are discussed. We conclude that our way of creating fiction and manipulating temporality in the consultation training was paralleled by most students' report on substantial learning feed-forward abilities from reflection on action.

Measuring patient-centered communication in Patient–Physician consultations: Theoretical and practical issues

The goal of patient-centered communication (PCC) is to help practitioners provide care that is concordant with the patient's values, needs and preferences, and that allows patients to provide input and participate actively in decisions regarding their health and health care. PCC is widely endorsed as a central component of high-quality health care, but it is unclear what it is and how to measure it. PCC includes four communication domains: the patient's perspective, the psychosocial context, shared understanding, and sharing power and responsibility. Problems in measuring PCC include lack of theoretical and conceptual clarity, unexamined assumptions, lack of adequate control for patient characteristics and social contexts, modest correlations between survey and observational measures, and overlap of PCC with other constructs. We outline problems in operationalizing PCC, choosing tools for assessing PCC, choosing data sources, identifying mediators of PCC, and clarifying outcomes of PCC. We propose nine areas for improvement: (1) developing theory-based operational definitions of PCC; (2) clarifying what is being measured; (3) accounting for the communication behaviors of each individual in the encounter as well as interactions among them; (4) accounting for context; (5) validating of instruments; (6) interpreting patient ratings of their physicians; (7) doing longitudinal studies; (8) examining pathways and mediators of links between PCC and outcomes; and (9) dealing with the complexity of the construct of PCC. We discuss the use of observational and survey measures, multi-method and mixed-method research, and standardized patients. The increasing influence of the PCC literature to guide medical education, licensure of clinicians, and assessments of quality provides a strong rationale for further clarification of these measurement issues.

Counselling quality in community pharmacies: implementation of the pseudo customer methodology in Germany

OBJECTIVES

To investigate a new method for evaluating counselling performance of staff in community pharmacies and to assess the quality of patient counselling.

METHOD

Trained pseudo customers, instructed to play their role according to two different self-medication scenarios, visited voluntarily participating community pharmacies in Berlin. After documenting the counselling process, immediately after each visit, outside the pharmacy on an assessment form, the pseudo customer re-entered the pharmacy and gave detailed performance feedback to the counsellor and the pharmacist in charge in order to provide support for improving counselling skills and practice behaviour, when appropriate. This was followed with a written summary of the general performance of all participating pharmacies and additional individual feedback and suggestions for improvement. Educational needs were identified for subsequent performance-based educational strategies such as group-workshops, team-training and on-site team-coaching.

RESULTS

Forty-nine community pharmacies in Berlin volunteered to participate in this pilot study. Ninety-eight per cent of the participating pharmacies offered advice. However, in 36% of the cases, advice was only given on request. The different types of scenarios--presentation of a symptom or request for a specific product--made a great difference to the spontaneity of questions and advice. At least one question to check on accuracy of self-diagnosis was asked in 95% of the cases of symptom presentation but in only 47% of the cases of specific product request. Information on appropriate self-medication was provided on at least one item in 74% of pseudo customer visits, but most of the time the information was not sufficient. Communication skills (nonverbal elements, comprehensibility etc.) were very good or good in 54% of the visits. Potential for improvement was mainly in relation to the use of open-ended questions to gain more information and on counselling about appropriate self-medication. Direct feedback was given in 96% of the pharmacies (one person refused to accept feedback and one feedback had to be postponed because of time shortage). All of the participants regarded counselling as an important subject in pharmacy practice.

CONCLUSION

The pseudo customer method was successfully used in this study of German community pharmacies. It was shown that pseudo customer visits and performance feedback following the counselling process, were feasible in daily practice and well accepted by the participants. A training program, focussing on areas in most need of improvement, has been developed. The promising results have led to the Federal Chamber of Pharmacists in Germany adopting this method as part of a continuous quality improvement program in community pharmacies.

Influence of patients' requests for direct-to-consumer advertised antidepressants: a randomized controlled trial

CONTEXT

Direct-to-consumer (DTC) advertising of prescription drugs in the United States is both ubiquitous and controversial. Critics charge that it leads to overprescribing, while proponents counter that it helps avert underuse of effective treatments, especially for conditions that are poorly recognized or stigmatized.

OBJECTIVE

To ascertain the effects of patients' DTC-related requests on physicians' initial treatment decisions in patients with depressive symptoms.

DESIGN

Randomized trial using standardized patients (SPs). Six SP roles were created by crossing 2 conditions (major depression or adjustment disorder with depressed mood) with 3 request types (brand-specific, general, or none).

SETTING

Offices of primary care physicians in Sacramento, Calif; San Francisco, Calif; and Rochester, NY, between May 2003 and May 2004.

PARTICIPANTS

One hundred fifty-two family physicians and general internists recruited from solo and group practices and health maintenance organizations; cooperation rates ranged from 53% to 61%.

INTERVENTIONS

The SPs were randomly assigned to make 298 unannounced visits, with assignments constrained so physicians saw 1 SP with major depression and 1 with adjustment disorder. The SPs made a brand-specific drug request, a general drug request, or no request (control condition) in approximately one third of visits.

MAIN OUTCOME MEASURES

Data on prescribing, mental health referral, and primary care follow-up obtained from SP written reports, visit audiorecordings, chart review, and analysis of written prescriptions and drug samples. The effects of request type on prescribing were evaluated using contingency tables and confirmed in generalized linear mixed models that accounted for clustering and adjusted for site, physician, and visit characteristics.

RESULTS

Standardized patient role fidelity was excellent, and the suspicion rate that physicians had seen an SP was 13%. In major depression, rates of antidepressant prescribing were 53%, 76%, and 31% for SPs making brand-specific, general, and no requests, respectively (P<.001). In adjustment disorder, antidepressant prescribing rates were 55%, 39%, and 10%, respectively (P<.001). The results were confirmed in multivariate models. Minimally acceptable initial care (any combination of an antidepressant, mental health referral, or follow-up within 2 weeks) was offered to 98% of SPs in the major depression role making a general request, 90% of those making a brand-specific request, and 56% of those making no request (P<.001).

CONCLUSIONS

Patients' requests have a profound effect on physician prescribing in major depression and adjustment disorder. Direct-to-consumer advertising may have competing effects on quality, potentially both averting underuse and promoting overuse.

Assessing the accuracy of administrative data in health information systems

BACKGROUND

Administrative data play a central role in health care. Inaccuracies in such data are costly to health systems, they obscure health research, and they affect the quality of patient care.

OBJECTIVES

We sought to prospectively determine the accuracy of the primary and secondary diagnoses recorded in administrative data sets.

RESEARCH DESIGN

Between March and July 2002, standardized patients (SPs) completed unannounced visits at 3 sites. We abstracted the 348 medical records from these visits to obtain the written diagnoses made by physicians. We also examined the patient files to identify the diagnoses recorded on the administrative encounter forms and extracted data from the computerized administrative databases. Because the correct diagnosis was defined by the SP visit, we could determine whether the final diagnosis in the administrative data set was correct and, if not, whether it was caused by physician diagnostic error, missing encounter forms, or incorrectly filled out forms.

SUBJECTS

General internal medicine outpatient clinics at 2 Veterans Administration facilities and a large, private medical center participated in this study.

MEASURES

A total of 45 trained SPs presented to physicians with 4 common outpatient conditions.

RESULTS

The correct primary diagnosis was recorded for 57% of visits. Thirteen percent of errors were caused by physician diagnostic error, 8% to missing encounter forms, and 22% to incorrectly entered data. Findings varied by condition and site but not by level of training. Accuracy of secondary diagnosis data (27%) was even poorer.

CONCLUSIONS

Although more research is needed to evaluate the cause of inaccuracies and the relative contributions of patient, provider, and system level effects, it appears that significant inaccuracies in administrative data are common. Interventions aimed at correcting these errors appear feasible.

Using vignettes to compare the quality of clinical care variation in economically divergent countries

OBJECTIVE

To determine whether clinical vignettes can measure variations in the quality of clinical care in two economically divergent countries.

DATA SOURCE/STUDY SETTING

Primary data collected between February 1997 and February 1998 at two Veterans Affairs facilities in the United States and four government-run outpatient facilities in Macedonia.

STUDY DESIGN

Randomly selected, eligible Macedonian and U.S. physicians (>97 percent participation rate) completed vignettes for four common outpatient conditions. Responses were judged against a master list of explicit quality criteria and scored as percent correct. DATA COLLECTION/ EXTRACTION: An ANOVA model and two-tailed t-tests were used to compare overall scores by case, study site, and country. Principal Findings. The mean score for U.S. physicians was 67 percent (+/-11 percent) compared to 48 percent (+/-11 percent) for Macedonian physicians. The quality of clinical practice, which emphasizes basic skills, varied greatly in both sites, but more so in Macedonia. However, the top Macedonian physicians in all sites approached or-in one case-exceeded the median score in the U.S. sites.

CONCLUSIONS

Vignettes are a useful method for making cross-national comparisons of the quality of care provided in very different settings. The vignette measurements revealed that some physicians in Macedonia performed at a standard comparable to that of their counterparts in the United States, despite the disparity of the two health systems. We infer that in poorer countries, policy that promotes improvements in the quality of clinical practice-not just structural inputs-could lead to rapid improvements in health.

Family Medicine Residents' Performance with Detected Versus Undetected Simulated Patients Posing as Problem Drinkers

BACKGROUND

Simulated patients are commonly used to evaluate medical trainees. Unannounced simulated patients provide an accurate measure of physician performance.

PURPOSE

To determine the effects of detection of SPs on physician performance, and identify factors leading to detection.

METHODS

Fixty-six family medicine residents were each visited by two unannounced simulated patients presenting with alcohol-induced hypertension or insomnia. Residents were then surveyed on their detection of SPs.

RESULTS

SPs were detected on 45 out of 104 visits. Inner city clinics had higher detection rates than middle class clinics. Residents' checklist and global rating scores were substantially higher on detected than undetected visits, for both between-subject and within-subject comparisons. The most common reasons for detection concerned SP demographics and behaviour; the SP "did not act like a drinker" and was of a different social class than the typical clinic patient.

CONCLUSIONS

Multi-clinic studies involving residents experienced with SPs should ensure that the SP role and behavior conform to physician expectations and the demographics of the clinic. SP station testing does not accurately reflect physicians' actual clinical behavior and should not be relied on as the primary method of evaluation. The study also suggests that physicians' poor performance in identifying and managing alcohol problems is not entirely due to lack of skill, as they demonstrated greater clinical skills when they became aware that they were being evaluated. Physicians' clinical priorities, sense of responsibility and other attitudinal determinants of their behavior should be addressed when training physicians on the management of alcohol problems.

Surfing, self-medicating and safety: buying non-prescription and complementary medicines via the internet

OBJECTIVE

To examine whether the sale of medicines via the internet supports their safe and appropriate use.

DESIGN

e-Pharmacy websites were identified using key words and a metasearch engine and the quality of information published on these websites was surveyed using the DISCERN tool. A case scenario and internet pharmacy practice standards were also used to evaluate the quality of care delivered.

SETTING AND PARTICIPANTS

Between July and September 2001 104 websites were surveyed and 27 sent either Sudafed (pseudoephedrine HCl), St John's wort products, or both to a residential address in Melbourne, Australia.

MAIN OUTCOME MEASURES

Quality of health information (DISCERN ratings), information exchanged between e-pharmacy staff and consumers, and product and delivery costs.

RESULTS

Of 104 e-pharmacies from at least 13 different countries, 63 websites provided some health information but overall the quality of the information was poor. Only three website operators provided adequate advice to consumers to avoid a potential drug interaction. The costs for a daily dose of pseudoephedrine HCl (240 mg) ranged from 0.81 Australian dollars to 3.04 Australian dollars, and delivery costs from 3.28 Australian dollars to 62.70 Australian dollars.

CONCLUSION

Consumers who self-select medicines from websites have insufficient access to information and advice at the point of ordering and on delivery to make informed decisions about their safe and appropriate use.

Evaluating medical students' skills in obtaining informed consent for HIV testing

OBJECTIVE

To evaluate fourth-year medical students' abilities to obtain informed consent or refusal for HIV testing through a performance-based evaluation method.

DESIGN

Student competence was assessed in a standardized patient interaction in which the student obtained informed consent or refusal for HIV testing. A previously validated 16-item checklist was completed by the standardized patient. A subset was independently reviewed and scored by a faculty member to calculate interrater reliability for this report. Student feedback on the assessment was elicited.

SETTING

School of Medicine at the University of New Mexico.

PATIENTS/PARTICIPANTS

All senior medical students in the class of 2000 were included.

INTERVENTIONS

A 10-minute standardized patient interaction was administered within the context of a formal comprehensive performance assessment.

MEASUREMENTS AND MAIN RESULTS

Seventy-nine students participated, and most (96%) demonstrated competence on the station. For the 15 specific items, the mean score was 25.5 out of 30 possible points (range, 13 to 30; SD, 3.5) on the checklist. A strong positive correlation (rs =.79) was found between the total score on the 15 Likert-scaled items and the score in response to the global item, "I would return to this clinician" (mean, 3.5; SD, 1.0). Scores given by the standardized patients and the faculty rater were well correlated. The station was generally well received by students, many of whom were stimulated to pursue further learning.

CONCLUSIONS

This method of assessing medical students' abilities to obtain informed consent or refusal for HIV testing can be translated to a variety of clinical settings. Such efforts may help in demonstrating competence in performing key ethics skills and may help ensure ethically sound clinical care for people at risk for HIV infection.

Assessment of clinical skills in medical practice

The introduction of a clinical skills examination (CSE) to Step 2 of the U.S. Medical Licensing Examination (USMLE) has focused attention on the design and delivery of large-scale standardized tests of clinical skills and raised the question of the appropriateness of evaluation of these competencies across the span of a physician's career. This initiative coincides with growing pressure to periodically assess the continued competence of physicians in practice. The USMLE CSE is designed to certify that candidates have the basic clinical skills required for the safe and effective practice of medicine in the supervised environment of postgraduate training. These include history taking, physical examination, effective communication with patients and other members of the health care team, and clear and accurate documentation of diagnostic impressions and plans for further assessment. The USMLE CSE does not assess procedural skills. As physicians progress through training and enter practice, both knowledge base and requisite technical skills become more diverse. A variety of indirect and direct measures are available for evaluating physicians, but, at present, no single method permits high-stake inferences about clinical skills. Systematic and standardized assessments make a contribution to comprehensive evaluations, but they retain an element of assessing capacity rather than authentic performance in practice. Much work is needed to identify the optimal combination of methods to be employed in support of programs to ensure maintenance of competence of practicing physicians.

Using standardised patients to measure physicians' practice: validation study using audio recordings

OBJECTIVE

To assess the validity of standardised patients to measure the quality of physicians' practice.

DESIGN

Validation study of standardised patients' assessments. Physicians saw unannounced standardised patients presenting with common outpatient conditions. The standardised patients covertly tape recorded their visit and completed a checklist of quality criteria immediately afterwards. Their assessments were compared against independent assessments of the recordings by a trained medical records abstractor.

SETTING

Four general internal medicine primary care clinics in California.

PARTICIPANTS

144 randomly selected consenting physicians.

MAIN OUTCOME MEASURES

Rates of agreement between the patients' assessments and independent assessment.

RESULTS

40 visits, one per standardised patient, were recorded. The overall rate of agreement between the standardised patients' checklists and the independent assessment of the audio transcripts was 91% (kappa=0.81). Disaggregating the data by medical condition, site, level of physicians' training, and domain (stage of the consultation) gave similar rates of agreement. Sensitivity of the standardised patients' assessments was 95%, and specificity was 85%. The area under the receiver operator characteristic curve was 90%.

CONCLUSIONS

Standardised patients' assessments seem to be a valid measure of the quality of physicians' care for a variety of common medical conditions in actual outpatient settings. Properly trained standardised patients compare well with independent assessment of recordings of the consultations and may justify their use as a "gold standard" in comparing the quality of care across sites or evaluating data obtained from other sources, such as medical records and clinical vignettes.

Compliance among pharmacies in California with a prescription-drug discount program for Medicare beneficiaries

BACKGROUND

Several states have developed prescription-drug discount programs for Medicare beneficiaries. In California, Senate Bill 393, enacted in 1999, requires pharmacies participating in the state Medicaid program (Medi-Cal) to charge customers who present a Medicare card amounts based on Medi-Cal rates. Because Medicare beneficiaries may not be accustomed to presenting their Medicare cards at pharmacies, we assessed the compliance of pharmacies with Senate Bill 393.

METHODS

Fifteen Medicare beneficiaries who received special training and acted as "standardized patients" visited a random sample of pharmacies in the San Francisco Bay area and Los Angeles County in April and May 2001. According to a script, they asked for the prices of three commonly prescribed drugs: rofecoxib, sertraline, and atorvastatin. The script enabled us to determine whether and when, during their interactions with pharmacists or salespeople, the discounts specified in Senate Bill 393 were offered. Pharmacies at which the appropriate discounts were offered were considered compliant.

RESULTS

The patients completed visits to 494 pharmacies. Seventy-five percent of the pharmacies complied with the prescription-drug discount program; at only 45 percent, however, was the discount offered before it was specifically requested. The discount was offered at 91 percent of pharmacies that were part of a chain, as compared with 58 percent of independent pharmacies (P<0.001). Compliance was higher in the San Francisco Bay area than in Los Angeles County (84 percent vs. 72 percent, P=0.004) and was higher in high-income than low-income neighborhoods (81 percent vs. 69 percent, P=0.002). A Medicare beneficiary taking all three drugs would have saved an average of $55.70 per month as compared with retail prices (a savings of 20 percent).

CONCLUSIONS

Discounts required under California's prescription-drug discount program for Medicare beneficiaries offer substantial savings. Many patients, however, especially those who use independent pharmacies or who live in low-income neighborhoods, may not receive the discounts.

Using standardized patients in advanced practice nursing education

Standardized patient encounters, a pedagogic approach to helping students develop appropriate clinical skills, are widely used in medical education and are being adopted by advanced practice nursing programs. Two types, simulated clinical encounters, in which students complete an episodic or comprehensive visit, and objective structured clinical experiences, multiple stations each presenting a different clinical problem, are used for formative and summative evaluation. Graduating adult and pediatric nurse practitioner students (N = 26) completed a simulated clinical encounter for an episodic visit. Students' performance on the simulated clinical encounter did not reflect their performance on other clinical evaluation measures or their performance on national certifying examinations. This one-time simulated clinical encounter was shown to lack the necessary reliability and validity to appropriately evaluate the clinical skills of nurse practitioner students. Videotaped, simulated patient encounters are useful, however as formative learning experiences. This reinforced our position that multiple site visits by faculty to students and their preceptors at all their clinical sites are necessary to accurately evaluate students' clinical performance.