Does Mental Health Care Integration Affect Primary Care Clinician Burnout? Results from a Longitudinal Veterans Affairs Survey

BACKGROUND

Burnout among primary care clinicians (PCPs) is associated with negative health and productivity consequences. The Veterans Health Administration (VA) embedded mental health specialists and care managers in primary care to manage common psychiatric diseases. While challenging to implement, mental health integration is a team-based care model thought to improve clinician well-being.

OBJECTIVE

To examine the relationships between PCP-reported burnout (and secondarily, job satisfaction) and mental health integration at provider and clinic levels DESIGN: Analysis of 286 cross-sectional surveys in 2012 (n = 171) and 2013 (n = 115) PARTICIPANTS: 210 PCPs in one VA region MAIN MEASURES: Outcomes were PCP-reported burnout (Maslach Burnout Inventory emotional exhaustion subscale), and secondarily, job satisfaction. Two independent variables represented mental health integration: (1) PCP-specialty communication rating and (2) proportion of clinic patients who saw integrated specialists. Using multilevel regression models, we examined PCP-reported burnout (and job satisfaction) and mental health integration, adjusting for PCP characteristics (e.g., gender), PCP ratings of team functioning (communication, knowledge/skills, satisfaction), and organizational factors.

KEY RESULTS

On average, PCPs reported high burnout (29, range = 9-54) across all VA healthcare systems. In total, 46% of PCPs reported "very easy" communication with mental health; 9% of primary clinic patients had seen integrated specialists. Burnout was not significantly associated with mental health communication ratings (β coefficient = - 0.96, standard error [SE] = 1.29, p = 0.46), nor with proportion of clinic patients who saw integrated specialists (β = 0.02, SE = 0.11, p = 0.88). No associations were observed with job satisfaction either. Among study participants, PCPs with poor team functioning, as exhibited by low team communication ratings, reported high burnout (β = - 1.28, SE = 0.22, p < 0.001) and low job satisfaction (β = 0.12, SE = 0.02, p < 0.001).

CONCLUSIONS

As currently implemented, primary care and mental health integration did not appear to impact PCP-reported burnout, nor job satisfaction. More research is needed to explore care model variation among clinics in order to optimize implementation to enhance PCP well-being.

Do Collaborative Care Managers and Technology Enhance Primary Care Satisfaction with Care from Embedded Mental Health Providers?

BACKGROUND

To improve mental health care access, the Veterans Health Administration (VA) implemented Primary Care-Mental Health Integration (PC-MHI) in clinics nationally. Primary care clinical leader satisfaction can inform model implementation and may be facilitated by collaborative care managers and technology supporting cross-specialty collaboration.

OBJECTIVE

(1) To determine primary care clinical leaders' overall satisfaction with care from embedded mental health providers for a range of conditions and (2) to examine the association between overall satisfaction and two program features (care managers, technology).

DESIGN

Cross-sectional organizational survey in one VA region (Southern California, Arizona, and New Mexico), 2018.

PARTICIPANTS

Sixty-nine physicians or other designated clinical leaders in each VA primary care clinic (94% response rate).

MAIN MEASURES

We assessed primary care clinical leader satisfaction with embedded mental health care on four groups of conditions: target, non-target mental health, behavioral health, suicide risk management. They additionally responded about the availability of mental health care managers and the sufficiency of information technology (telemental health, e-consult, instant messaging). We examined relationships between satisfaction and the two program features using χ² tests and multivariable regressions.

KEY RESULTS

Most primary care clinical leaders were "very satisfied" with care for targeted anxiety (71%) and depression (69%), but not for other common conditions (37% alcohol misuse, 19% pain). Care manager availability was significantly associated with "very satisfied" responses for depression (p = .02) and anxiety care by embedded mental health providers (p = .02). Highly rated sufficiency of communication technology (only 19%) was associated with "very satisfied" responses to suicide risk management (p = .002).

CONCLUSIONS

Care from embedded mental health providers for depression and anxiety was highly satisfactory, which may guide improvement among less satisfactory conditions (alcohol misuse, pain). Observed associations between overall satisfaction and collaborative care features may inform clinics on how to optimize staffing and technology based on priority conditions.

Assessing fidelity to evidence-based quality improvement as an implementation strategy for patient-centered medical home transformation in the Veterans Health Administration

Background

Effective implementation strategies might facilitate patient-centered medical home (PCMH) uptake and spread by targeting barriers to change. Evidence-based quality improvement (EBQI) is a multi-faceted implementation strategy that is based on a clinical-researcher partnership. It promotes organizational change by fostering innovation and the spread of those innovations that are successful. Previous studies demonstrated that EBQI accelerated PCMH adoption within Veterans Health Administration primary care practices, compared with standard PCMH implementation. Research to date has not documented fidelity to the EBQI implementation strategy, limiting usefulness of prior research findings. This paper develops and assesses clinical participants’ fidelity to three core EBQI elements for PCMH (EBQI-PCMH), explores the relationship between fidelity and successful QI project completion and spread (the outcome of EBQI-PCMH), and assesses the role of the clinical-researcher partnership in achieving EBQI-PCMH fidelity.

Methods

Nine primary care practice sites and seven across-sites, topic-focused workgroups participated (2010–2014). Core EBQI elements included leadership-frontlines priority-setting for QI, ongoing access to technical expertise, coaching, and mentoring in QI methods (through a QI collaborative), and data/evidence use to inform QI. We used explicit criteria to measure and assess EBQI-PCMH fidelity across clinical participants. We mapped fidelity to evaluation data on implementation and spread of successful QI projects/products. To assess the clinical-researcher partnership role in EBQI-PCMH, we analyzed 73 key stakeholder interviews using thematic analysis.

Results

Seven of 9 sites and 3 of 7 workgroups achieved high or medium fidelity to leadership-frontlines priority-setting. Fidelity was mixed for ongoing technical expertise and data/evidence use. Longer duration in EBQI-PCMH and higher fidelity to priority-setting and ongoing technical expertise appear correlated with successful QI project completion and spread. According to key stakeholders, partnership with researchers, as well as bi-directional communication between leaders and QI teams and project management/data support were critical to achieving EBQI-PCMH fidelity.

Conclusions

This study advances implementation theory and research by developing measures for and assessing fidelity to core EBQI elements in relationship to completion and spread of QI innovation projects or tools for addressing PCMH challenges. These results help close the gap between EBQI elements, their intended outcome, and the finding that EBQI-PCMH resulted in accelerated adoption of PCMH.

A real-time assessment of factors influencing medication events

Reducing medical error is critical to improving the safety and quality of healthcare. Physician stress, fatigue, and excessive workload are performance-shaping factors (PSFs) that may influence medical events (actual administration errors and near misses), but direct relationships between these factors and patient safety have not been clearly defined. This study assessed the real-time influence of emotional stress, workload, and sleep deprivation on self-reported medication events by physicians in academic hospitals. During an 18-month study period, 185 physician participants working at four university-affiliated teaching hospitals reported medication events using a confidential reporting application on handheld computers. Emotional stress scores, perceived workload, patient case volume, clinical experience, total sleep, and demographic variables were also captured via the handheld computers. Medication event reports (n = 11) were then correlated with these demographic and PSFs. Medication events were associated with 36.1% higher perceived workload (p < .05), 38.6% higher inpatient caseloads (p < .01), and 55.9% higher emotional stress scores (p < .01). There was a trend for reported events to also be associated with less sleep (p = .10). These results confirm the effect of factors influencing medication events, and support attention to both provider and hospital environmental characteristics for improving patient safety.

Intern to attending: assessing stress among physicians

PURPOSE

Organizations have raised concerns regarding stress in the medical work environment and effects on health care worker performance. This study's objective was to assess workplace stress among interns, residents, and attending physicians using Ecological Momentary Assessment technology, the gold-standard method for real-time measurement of psychological characteristics.

METHOD

The authors deployed handheld computers with customized software to 185 physicians on the medicine and pediatric wards of four major teaching hospitals. The physicians contemporaneously recorded multiple dimensions of physician work (e.g., type of call day), emotional stress (e.g., worry, stress, fatigue), and perceived workload (e.g., patient volume). The authors performed descriptive statistics and t test and linear regression analyses.

RESULTS

Participants completed 5,673 prompts during an 18-month period from 2004 to 2005. Parameters associated with higher emotional stress in linear regression models included male gender (t = -2.5, P = .01), total patient load (t = 4.2, P < .001), and sleep quality (t = -2.8, P = .006). Stress levels reported by attendings (t = -3.3, P = .001) were lower than levels reported by residents (t = -2.6, P = .009), and emotional stress levels of attendings and residents were both lower compared with interns.

CONCLUSIONS

On inpatient wards, after recent resident duty hours changes, physician trainees continue to show wide-ranging evidence of workplace stress and poor sleep quality. This is among the first studies of medical workplace stress in real time. These results can help residency programs target education in stress and sleep and readdress workload distribution by training level. Further research is needed to clarify behavioral factors underlying variability in housestaff stress responses.

Use of a handheld computer application for voluntary medication event reporting by inpatient nurses and physicians

OBJECTIVE

To determine the feasibility of capturing self-reported medication events using a handheld computer-based Medication Event Reporting Tool (MERT).

DESIGN AND PARTICIPANTS

Handheld computers operating the MERT software application were deployed among volunteer physician (n = 185) and nurse (n = 119) participants on the medical wards of four university-affiliated teaching hospitals. Participants were encouraged to complete confidential reports on the handheld computers for medication events observed during the study period.

MEASUREMENTS AND MAIN RESULTS

Demographic variables including age, gender, education level, and clinical experience were recorded for all participants. Each MERT report included details on the provider, location, timing and type of medication event recorded. Over the course of 2,311 days of clinician participation, 76 events were reported; the median time for report completion was 231 seconds. The average event reporting rate for all participants was 0.033 reports per clinician shift. Nurses had a significantly higher reporting rate compared to physicians (0.045 vs 0.026 reports/shift, p = .02). Subgroup analysis revealed that attending physicians reported events more frequently than resident physicians (0.042 vs 0.021 reports/shift, p = .03), and at a rate similar to that of nurses (p = .80). Only 5% of MERT medication events were reported to require increased monitoring or treatment.

CONCLUSIONS

A handheld-based event reporting tool is a feasible method to record medication events in inpatient hospital care units. Handheld reporting tools may hold promise to augment existing hospital reporting systems.

Direct and indirect links between childhood maltreatment, posttraumatic stress disorder, and women's health

The authors evaluated the relationships among childhood maltreatment, sexual trauma in adulthood, posttraumatic stress disorder (PTSD), and health functioning in women. Female Veterans' Affairs (VA) primary care patients (N = 200) completed self-report measures of childhood maltreatment, adult sexual trauma, PTSD symptoms, and current health functioning. The authors used structural equation modeling to test models of the relationship among these variables. Childhood nonsexual maltreatment and adult sexual assault were positively associated with PTSD. Childhood nonsexual maltreatment (beta = -.20) and PTSD (beta = -.75) were significantly associated with poorer physical and mental health functioning. Adult sexual assault negatively affected health functioning through its association with PTSD. Thus, poor health outcomes associated with childhood maltreatment in women may be conveyed through PTSD. These findings should strengthen efforts directed at identifying and treating PTSD in female victims of childhood maltreatment with the aim of preventing or attenuating poor health outcomes.

Effect of age at onset on the course of major depressive disorder

OBJECTIVE

This report assesses whether age at onset defines a specific subgroup of major depressive disorder in 4,041 participants who entered the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) study.

METHOD

The study enrolled outpatients 18-75 years of age with nonpsychotic major depressive disorder from both primary care and psychiatric care practices. At study entry, participants estimated the age at which they experienced the onset of their first major depressive episode. This report divides the population into five age-at-onset groups: childhood onset (ages <12), adolescent onset (ages 12-17), early adult onset (ages 18-44), middle adult onset (ages 45-59), and late adult onset (ages > or =60).

RESULTS

No group clearly stood out as distinct from the others. Rather, the authors observed an apparent gradient, with earlier ages at onset associated with never being married, more impaired social and occupational function, poorer quality of life, greater medical and psychiatric comorbidity, a more negative view of life and the self, more lifetime depressive episodes and suicide attempts, and greater symptom severity and suicidal ideation in the index episode compared to those with later ages at onset of major depressive disorder.

CONCLUSIONS

Although age at onset does not define distinct depressive subgroups, earlier onset is associated with multiple indicators of greater illness burden across a wide range of indicators. Age of onset was not associated with a difference in treatment response to the initial trial of citalopram.

Relationships among childhood maltreatment, PTSD, and health in female veterans in primary care

OBJECTIVE

Women with histories of childhood maltreatment (CM) have higher rates of physical health problems and greater medical utilization compared to women without abuse histories. This study examined whether current post-traumatic stress disorder (PTSD) symptoms mediate the relationship between CM and indicators of physical health and medical utilization in female veterans.

METHOD

Respondents were 221 female veterans (56% of the potential sample), who received medical care from the San Diego VA Healthcare System during a 12-month period. Respondents provided self-report information about CM, PTSD symptoms, use of pain medication, and physical symptoms and functioning. Additional information about medical utilization was extracted from respondents' medical charts. Regression-based models were conducted to test whether PTSD symptoms mediate the relationships between CM and physical symptoms and between CM and medical utilization.

RESULTS

Emotional abuse was associated with poorer role-physical functioning, increased bodily pain and greater odds of using pain medication in the past 6 months. Physical abuse was associated with poorer general health. Contrary to prediction, emotional neglect was associated with better role-physical functioning, and CM was not associated with increased healthcare utilization. PTSD was shown to mediate the relationship between emotional and physical abuse and health outcomes.

CONCLUSIONS

PTSD, or psychopathology more generally, appears to be an important factor in the negative health impact of CM. Given that several empirically supported interventions are available for PTSD, there may be physical health benefits in early identification and treatment of psychopathology related to CM.

Mental health treatment preferences of older and younger primary care patients

OBJECTIVE

To compare mental health treatment history and preferences in older and younger primary care patients.

METHOD

We surveyed 77 older (60+) and 312 younger adult primary care patients from four outpatient medical clinics about their mental health treatment history and preferences.

RESULTS

Older adults were less likely than younger adults to report a history of mental health treatment (29% vs. 51%) or to be currently receiving treatment (11% vs. 23%). They were also less likely to indicate that they currently desire help with emotional problems (25% vs. 50%). Older adults were more likely to hold a belief in self-reliance that could limit their willingness to accept treatment for mental health problems, although they were less likely than younger adults to identify other barriers to treatment. Older adults reported that they were less likely to attend programs in primary care targeting mental health issues (counseling, stress management) than younger adults, although they were as willing as younger adults to attend programs targeting physical health issues (healthy living class, fitness program). Age remained a significant predictor of mental health treatment history and preferences even after controlling for other demographic variables.

CONCLUSION

These results suggest that older adults in the primary care setting may be less willing to accept mental health services than younger adults. Results further suggest that perceived barriers may differ for older and younger patients, which may indicate the need for age-specific educational messages and services targeted to older adults in primary care.

Mental health and satisfaction with primary health care in female patients

PURPOSE AND OBJECTIVES

Patient satisfaction is an important outcome in patient care and is increasingly being used as an indicator of quality of care within large health systems. This study examined whether consideration of specific mental health factors, including posttraumatic stress disorder (PTSD), can improve our understanding of patient satisfaction in primary care settings.

METHODS

Questionnaires were mailed to all women who used the VA San Diego Healthcare System primary care clinic in 1998. Two hundred twenty-one (56%) women who were invited to participate in this study completed questionnaires. Participants provided information about physical and mental health and satisfaction with their primary medical care.

RESULTS

Age and general mental health were negatively associated and PTSD was positively associated with overall satisfaction with care and satisfaction with the provider. General mental health was significantly related to satisfaction with the clinic.

CONCLUSIONS

These findings support the importance of specific mental health symptoms, and trauma-related symptoms in specific, in determining satisfaction.

An evaluation of vignettes for predicting variation in the quality of preventive care

OBJECTIVE

Clinical vignettes offer an inexpensive and convenient alternative to the benchmark method of chart audits for assessing quality of care. We examined whether vignettes accurately measure and predict variation in the quality of preventive care.

DESIGN

We developed scoring criteria based on national guidelines for 11 prevention items, categorized as vaccine, vascular-related, cancer screening, and personal behaviors. Three measurement methods were used to ascertain the quality of care provided by clinicians seeing trained actors (standardized patients; SPs) presenting with common outpatient conditions: 1) the abstracted medical record from an SP visit; 2) SP reports of physician practice during those visits; and 3) physician responses to matching computerized case scenarios (clinical vignettes).

SETTING

Three university-affiliated (including 2 VA) and one community general internal medicine clinics.

PATIENTS/PARTICIPANTS

Seventy-one randomly selected physicians from among eligible general internal medicine residents and attending physicians.

MEASUREMENTS AND MAIN RESULTS

Physicians saw 480 SPs (120 at each site) and completed 480 vignettes. We calculated the proportion of prevention items for each visit reported or recorded by the 3 measurement methods. We developed a multiple regression model to determine whether site, training level, or clinical condition predicted prevention performance for each measurement method. We found that overall prevention scores ranged from 57% (SP) to 54% (vignettes) to 46% (chart abstraction). Vignettes matched or exceeded SP scores for 3 prevention categories (vaccine, vascular-related, and personal behavior). Prevention quality varied by site (from 40% to 67%) and was predicted similarly by vignettes and SPs.

CONCLUSIONS

Vignettes can measure and predict prevention performance. Vignettes may be a less costly way to assess prevention performance that also controls for patient case-mix.

An evaluation of vignettes for predicting variation in the quality of preventive care

OBJECTIVE

Clinical vignettes offer an inexpensive and convenient alternative to the benchmark method of chart audits for assessing quality of care. We examined whether vignettes accurately measure and predict variation in the quality of preventive care.

DESIGN

We developed scoring criteria based on national guidelines for 11 prevention items, categorized as vaccine, vascular-related, cancer screening, and personal behaviors. Three measurement methods were used to ascertain the quality of care provided by clinicians seeing trained actors (standardized patients; SPs) presenting with common outpatient conditions: 1) the abstracted medical record from an SP visit; 2) SP reports of physician practice during those visits; and 3) physician responses to matching computerized case scenarios (clinical vignettes).

SETTING

Three university-affiliated (including 2 VA) and one community general internal medicine clinics.

PATIENTS/PARTICIPANTS

Seventy-one randomly selected physicians from among eligible general internal medicine residents and attending physicians.

MEASUREMENTS AND MAIN RESULTS

Physicians saw 480 SPs (120 at each site) and completed 480 vignettes. We calculated the proportion of prevention items for each visit reported or recorded by the 3 measurement methods. We developed a multiple regression model to determine whether site, training level, or clinical condition predicted prevention performance for each measurement method. We found that overall prevention scores ranged from 57% (SP) to 54% (vignettes) to 46% (chart abstraction). Vignettes matched or exceeded SP scores for 3 prevention categories (vaccine, vascular-related, and personal behavior). Prevention quality varied by site (from 40% to 67%) and was predicted similarly by vignettes and SPs.

CONCLUSIONS

Vignettes can measure and predict prevention performance. Vignettes may be a less costly way to assess prevention performance that also controls for patient case-mix.

The impact of individual forms of childhood maltreatment on health behavior

OBJECTIVE

This study examines the unique contribution of five types of maltreatment (sexual abuse, physical abuse, emotional abuse, physical neglect, emotional neglect) to adult health behaviors as well as the additive impact of exposure to different types of childhood maltreatment.

METHOD

Two hundred and twenty-one women recruited from a VA primary care clinic completed questionnaires assessing exposure to childhood trauma and adult health behaviors. Regression models were used to test the relationship between childhood maltreatment and adult health behaviors.

RESULTS

Sexual and physical abuse appear to predict a number of adverse outcomes; when other types of maltreatment are controlled, however, sexual abuse and physical abuse do not predict as many poor outcomes. In addition, sexual, physical, and emotional abuse and emotional neglect in childhood were all related to different adult health behaviors. The more types of childhood maltreatment participants were exposed to the more likely they were to have problems with substance use and risky sexual behaviors in adulthood.

IMPLICATIONS

The results indicate that it is important to assess a broad maltreatment history rather than trying to relate specific types of abuse to particular adverse health behaviors or health outcomes.

Relationship of sexual assault history to somatic symptoms and health anxiety in women

Prior reports have pointed to a link between traumatic experiences and health consequences in women. The objective of this study was to determine whether there is an association between sexual assault history and measures of somatic symptoms and illness attitudes in a sample of female Veterans Affairs primary care patients, a group in whom high rates of sexual trauma have been reported. We conducted a cross-sectional study of a representative sample of 219 women in a Veteran's Affairs primary care outpatient clinic. Sexual assault history, somatic symptoms and health anxiety were assessed by self-report questionnaire. Multivariate analyses were used to examine relationships between sexual assault exposure and these outcomes. Ninety-seven women (43.9%) reported experience(s) of sexual assault (i.e., rape, attempted rape or being made to perform any type of sexual act through force or threat of harm). Sexual assault was associated with a significant increase in somatization scores, physical complaints across multiple symptom domains and health anxiety. Sexual assault was also a significant statistical predictor of having multiple sick days in the prior 6 months and of being a high utilizer of primary care visits in the prior 6 months. These data confirm a strong association between sexual trauma exposure and somatic symptoms, illness attitudes and healthcare utilization in women. Causal mechanisms cannot be inferred from these data. Studies in other cohorts are warranted.

Sexual trauma, posttraumatic stress disorder, and health behavior

The authors tested whether sexual traumatization is associated with poorer health behavior and also evaluated the role of posttraumatic stress disorder (PTSD) in this relationship. They mailed questionnaires to 419 women who had visited a San Diego Veterans Administration primary care clinic in 1998 and received 221 responses, a 56% return rate. They found that a history of sexual assault was associated with increased substance use, risky sexual behaviors, less vigorous exercise, and increased preventive healthcare. They then used regression-based techniques to test whether PTSD mediates the relationship between a history of sexual assault and health behaviors and discovered support for this hypothesis in relation to substance use. PTSD symptoms were also associated with less likelihood of conducting regular breast self-examinations. Findings from the study highlight the value of programs designed to (1) identify trauma victims, (2) screen for problematic behaviors, and (3) intervene to improve long-term health outcomes.

Sensitivity and specificity of the PTSD checklist in detecting PTSD in female veterans in primary care

PTSD affects a substantial number of women in medical settings and is associated with significant distress and impairment. There are effective methods of treating trauma-related distress, but a minority seek such care. Thus, primary care is an important setting in which to identify individuals with PTSD. We sent questionnaires, including the PTSD Checklist--Civilian Version (PCL-C), to 419 female veterans who were seen in our primary care clinic in 1998; 56% (N = 221) returned the measures. A random subset (n = 49) was interviewed to establish psychiatric diagnoses. The results provide qualified support for the use of the PCL-C total score with a lowered cutoff score as a screening measure for PTSD in female veterans in primary care.

The ethical problem of false positives: a prospective evaluation of physician reporting in the medical record

OBJECTIVE

To determine if the medical record might overestimate the quality of care through false, and potentially unethical, documentation by physicians.

DESIGN

Prospective trial comparing two methods for measuring the quality of care for four common outpatient conditions: (1) structured reports by standardised patients (SPs) who presented unannounced to the physicians' clinics, and (2) abstraction of the medical records generated during these visits.

SETTING

The general medicine clinics of two veterans affairs medical centres.

PARTICIPANTS

Twenty randomly selected physicians (10 at each site) from among eligible second and third year internal medicine residents and attending physicians.

MAIN MEASUREMENTS

Explicit criteria were used to score the medical records of physicians and the reports of SPs generated during 160 visits (8 cases x 20 physicians). Individual scoring items were categorised into four domains of clinical performance: history, physical examination, treatment, and diagnosis. To determine the false positive rate, physician entries were classified as false positive (documented in the record but not reported by the SP), false negative, true positive, and true negative.

RESULTS

False positives were identified in the medical record for 6.4% of measured items. The false positive rate was higher for physical examination (0.330) and diagnosis (0.304) than for history (0.166) and treatment (0.082). For individual physician subjects, the false positive rate ranged from 0.098 to 0.397.

CONCLUSIONS

These data indicate that the medical record falsely overestimates the quality of important dimensions of care such as the physical examination. Though it is doubtful that most subjects in our study participated in regular, intentional falsification, we cannot exclude the possibility that false positives were in some instances intentional, and therefore fraudulent, misrepresentations. Further research is needed to explore the questions raised but incompletely answered by this research.

Effect of pravastatin-to-simvastatin conversion on low-density-lipoprotein cholesterol

The effects of a pravastatin-to-simvastatin conversion program on low-density-lipoprotein (LDL) cholesterol levels were studied. Patients receiving pravastatin at a Veterans Affairs medical center were switched to simvastatin beginning in 1997. The dosage of simvastatin was based on the additional percent reduction in LDL cholesterol needed to achieve the goal specified by the National Cholesterol Education Program. The primary endpoint was the change in the percentage of patients meeting their LDL cholesterol goal at baseline and follow-up. Changes in lipid indices, the relative risk (RR) of coronary heart disease (CHD), and program costs were also evaluated. A total of 1032 patients completed the program. The mean +/- S.D. daily doses of pravastatin and simvastatin were 25.2 +/- 11.3 and 22.7 +/- 13.3 mg, respectively. Median baseline and follow-up LDL cholesterol concentrations were 116 and 99 mg/dL, respectively (p < 0.001). Overall, 44% of the patients met their LDL cholesterol goal while taking pravastatin, compared with 69% after conversion to simvastatin (p < 0.001). The predicted mean RR of a future CHD event (based on changes in serum lipids) was 0.87 (95% confidence interval, 0.83-0.91) four years after conversion. The total cost of the program was $40,644 in the first year, and there was a net saving thereafter. Therapeutic interchange between pravastatin and simvastatin increased the number of patients meeting their LDL cholesterol goal.

Measuring compliance with preventive care guidelines: standardized patients, clinical vignettes, and the medical record

OBJECTIVE

To determine how accurately preventive care reported in the medical record reflects actual physician practice or competence.

DESIGN

Scoring criteria based on national guidelines were developed for 7 separate items of preventive care. The preventive care provided by randomly selected physicians was measured prospectively for each of the 7 items. Three measurement methods were used for comparison: (1) the abstracted medical record from a standardized patient (SP) visit; (2) explicit reports of physician practice during those visits from the SPs, who were actors trained to present undetected as patients; and (3) physician responses to written case scenarios (vignettes) identical to the SP presentations.

SETTING

The general medicine primary care clinics of two university-afflliated VA medical centers.

PARTICIPANTS

Twenty randomly selected physicians (10 at each site) from among eligible second- and third-year general internal medicine residents and attending physicians.

MEASUREMENTS AND MAIN RESULTS

Physicians saw 160 SPs (8 cases x 20 physicians). We calculated the percentage of visits in which each prevention item was recorded in the chart, determined the marginal percentage improvement of SP checklists and vignettes over chart abstraction alone, and compared the three methods using an analysis-of-variance model. We found that chart abstraction underestimated overall prevention compliance by 16% (P < .01) compared with SP checklists. Chart abstraction scores were lower than SP checklists for all seven items and lower than vignettes for four items. The marginal percentage improvement of SP checklists and vignettes to performance as measured by chart abstraction was significant for all seven prevention items and raised the overall prevention scores from 46% to 72% (P < .0001).

CONCLUSIONS

These data indicate that physicians perform more preventive care than they report in the medical record. Thus, benchmarks of preventive care by individual physicians and institutions that rely solely on the medical record may be misleading, at best.

How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record

PURPOSE

Despite widespread reliance on chart abstraction for quality measurement, concerns persist about its reliability and validity. We prospectively evaluated the validity of chart abstraction by directly comparing it with the gold standard of reports by standardized patients.

SUBJECTS AND METHODS

Twenty randomly selected general internal medicine residents and attending faculty physicians at the primary care clinics of two Veterans Affairs Medical Centers blindly evaluated and treated actor-patients (standardized patients) who had one of four common diseases: diabetes, chronic obstructive pulmonary disease, coronary artery disease, or low back pain. Charts from the visits were abstracted using explicit quality criteria; standardized patients completed a checklist containing the same criteria. For each physician, quality was measured for two different cases of the four conditions (a total of 160 physician-patient encounters). We compared chart abstraction with standardized-patient reports for four aspects of the encounter: taking the history, examining the patient, making the diagnosis, and prescribing appropriate treatment. The sensitivity and specificity of chart abstraction were calculated.

RESULTS

The mean (+/- SD) chart abstraction score was 54% +/- 9%, substantially less than the mean score on the standardized-patient checklist of 68% +/- 9% (P <0.001). This finding was similar for all four conditions and at both sites. "False positives"-chart-recorded necessary care actions not reported by the standardized patients-resulted in a specificity of only 81%. The overall sensitivity of chart abstraction for necessary care was only 70%.

CONCLUSIONS

Chart abstraction underestimates the quality of care for common outpatient general medical conditions when compared with standardized-patient reports. The medical record is neither sensitive nor specific. Quality measurements derived from chart abstraction may have important shortcomings, particularly as the basis for drawing policy conclusions or making management decisions.

Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality

CONTEXT

Better health care quality is a universal goal, yet measuring quality has proven to be difficult and problematic. A central problem has been isolating physician practices from other effects of the health care system.

OBJECTIVE

To validate clinical vignettes as a method for measuring the competence of physicians and the quality of their actual practice.

DESIGN

Prospective trial conducted in 1997 comparing 3 methods for measuring the quality of care for 4 common outpatient conditions: (1) structured reports by standardized patients (SPs), trained actors who presented unannounced to physicians' clinics (the gold standard); (2) abstraction of medical records for those same visits; and (3) physicians' responses to clinical vignettes that exactly corresponded to the SPs' presentations. Setting Outpatient primary care clinics at 2 Veterans Affairs medical centers.

PARTICIPANTS

Ninety-eight (97%) of 101 general internal medicine staff physicians, faculty, and second- and third-year residents consented to be randomized for the study. From this group, 10 physicians at each site were randomly selected for inclusion.

MAIN OUTCOME MEASURES

A total of 160 quality scores (8 cases x 20 physicians) were generated for each method using identical explicit criteria based on national guidelines and local expert panels. Scores were defined as the percentage of process criteria correctly met and were compared among the 3 methods.

RESULTS

The quality of care, as measured by all 3 methods, ranged from 76.2% (SPs) to 71.0% (vignettes) to 65.6% (chart abstraction). Measuring quality using vignettes consistently produced scores closer to the gold standard of SP scores than using chart abstraction. This pattern was robust when the scores were disaggregated by the 4 conditions (P<.001 to <.05), by case complexity (P<.001), by site (P<.001), and by level of physician training (P values from <.001 to <.05). The pattern persisted, although less dominantly, when we assessed the component domains of the clinical encounter--history, physical examination, diagnosis, and treatment. Vignettes were responsive to expected directions of variation in quality between sites and levels of training. The vignette responses did not appear to be sensitive to physicians' having seen an SP presenting with the same case.

CONCLUSIONS

Our data indicate that quality of health care can be measured in an outpatient setting by using clinical vignettes. Vignettes appear to be a valid and comprehensive method that directly focuses on the process of care provided in actual clinical practice. Vignettes show promise as an inexpensive case-mix adjusted method for measuring the quality of care provided by a group of physicians.

Analyzing the time and value of housestaff inpatient work

OBJECTIVE

To determine time allocation and the perceived value to education and patient care of the weekday activities of internal medicine housestaff on inpatient rotations and to compare the work activities of interns and residents.

DESIGN

An observational study. We classified activities along five dimensions (association, location, activity, time, and value), developed a computer-assisted self-interview survey, and demonstrated its face and content validity, internal consistency, and interrater reliability. Subjects were assigned survey computers for 5 consecutive weekdays over a 24-week period, into which they entered data when prompted several times a day.

SETTING

The medical service of a university-affiliated Veterans Administration Medical Center.

PARTICIPANTS

Sixty housestaff (36 interns, 24 residents) rotating on the inpatient wards.

MEASUREMENTS AND MAIN RESULTS

We analyzed activities according to content (direct patient care, indirect patient care, education), association, and location. Likert-scale ratings of perceived value to education and patient care were also obtained. Housestaff provided complete responses to 3,812 (95%) of 3,992 prompts by a median of 11 seconds; 93% of responses were logically consistent across the measured dimensions. Housestaff spent more time in indirect patient care (56%) than in direct patient care (14%) or educational activities (45%). Formal educational activities had the highest educational value (66 on 0-100 scale), and direct care had the highest value to patient care (81). Over 30% of time was spent in administrative activities, which had low educational value(40). Compared with residents, interns allocated significantly less time to educational activities (38% vs 57%) and more time to lower-value activities such as documentation (19% vs 12%).

CONCLUSIONS

Improved data collection methods demonstrate that housestaff in our program, particularly interns, spend much of their workday in activities that are low in educational and patient care value. Selective elimination or delegation of such activities would preserve higher-value experiences during reductions in overall inpatient training time. Planners can use automated random sampling to guide the rational redesign of housestaff work.