Special issues addressed in the CAHPS survey of Medicare managed care beneficiaries. Consumer Assessment of Health Plans Study

Associations of Mail Survey Length and Layout With Response Rates

We assess the association between survey layout and response rates (RRs) in the 2017 Medicare Advantage Consumer Assessment of Healthcare Providers and Systems mail survey. Among 438 Medicare Advantage plans surveyed by six vendors, there was latitude in survey layout, and plans could add up to 12 supplemental items. Regression models predicted survey response from survey characteristics (page count, number of supplemental items, and survey attractiveness), and beneficiary sociodemographics. Beneficiary-age-by-survey-characteristic interactions assessed whether survey characteristics were more strongly related to RRs among older beneficiaries. We found that surveys with more supplemental items and less attractive layouts had lower adjusted odds of response. RRs were more sensitive to format among older beneficiaries. The difference in adjusted RRs for the most favorable versus the least favorable survey design was 14.5%. For a 65-year-old, this difference was 13.6%; for an 80-year-old, it was 21.0%. These findings suggest that even within a relatively standardized survey, formatting can substantially influence RRs.

The effects of survey version on patient experience scores and plan rankings

OBJECTIVE

To assess the effect of changing survey questions on plan-level patient experience measures and ratings.

DATA SOURCE

2015 Medicare Advantage CAHPS Survey respondents.

STUDY DESIGN

Ninety three randomly selected beneficiaries in each of 40 MA plans received a revised (5.0) CAHPS survey; 38 832 beneficiaries received version 4.0. Linear mixed-effect regression predicted CAHPS measures from fixed effects for survey version and beneficiary characteristics and random effects for plan and plan-by-version random slope.

PRINCIPAL FINDINGS

Response rates were 42 percent for both versions. Removal of "try to" from screeners increased the percentage of respondents eligible for follow-up questions. Version 5.0 caused a small increase (1-3 points on a 0-100 scale, P < 0.05) in the mean of three altered measures and a moderate increase (>3 points) in one. There was a small statistically significant increase in two unaltered measures. These changes were uniform across plans, so there would be no expected change compared to results using the legacy survey in the score distributions other than uniform mean shifts, and no expected effect on summary measures.

CONCLUSIONS

These analyses illustrate how to assess the impact of seemingly minor survey modifications for other national surveys considering changes and highlight the importance of screeners in instrument design.

Using predicted Spanish preference to target bilingual mailings in a mail survey with telephone follow-up

OBJECTIVE

Spanish-preferring Medicare beneficiaries are underrepresented in national patient experience surveys. We test a method for improving their representation via higher response rates.

DATA SOURCES/STUDY SETTING

2009-2010 Medicare CAHPS surveys; Medicare population.

STUDY DESIGN

We used surname and address to predict Spanish-language preference for a national sample of 177 139 beneficiaries. We randomized half of the 10 000 non-Puerto Rico beneficiaries with the highest predicted probabilities of Spanish preference (>10 percent) to bilingual mailings (intervention) and half to standard English-only mailings (control).

DATA COLLECTION

Medicare CAHPS Survey data were collected through mail surveys with telephone follow-up of nonrespondents.

PRINCIPAL FINDINGS

Mail response rate was higher for intervention (28.7 percent) than control (23.9 percent) (P < 0.0001); phone response rates among mail nonrespondents were similar in intervention and control arms (15.8 percent vs 15.7 percent, P = 0.90). Targeted bilingual mailings induced 6.5 percent of those who would not have responded to respond by mail and 54.0 percent of those who would have responded in English to respond in Spanish. Beneficiaries with greater Spanish probabilities showed greater increases in response rates, a higher proportion of responses in Spanish, and lower control response rates among.

CONCLUSIONS

Targeted bilingual mailing of mixed-mode surveys using commonly available surname and address information can efficiently increase representation of this underrepresented group.

Medicare Prescription Drug Plan Enrollees Report Less Positive Experiences Than Their Medicare Advantage Counterparts

Since 2006, Medicare beneficiaries have been able to obtain prescription drug coverage through standalone prescription drug plans or their Medicare Advantage (MA) health plan, options exercised in 2015 by 72 percent of beneficiaries. Using data from community-dwelling Medicare beneficiaries older than age sixty-four in 700 plans surveyed from 2007 to 2014, we compared beneficiaries' assessments of Medicare prescription drug coverage when provided by standalone plans or integrated into an MA plan. Beneficiaries in standalone plans consistently reported less positive experiences with prescription drug plans (ease of getting medications, getting coverage information, and getting cost information) than their MA counterparts. Because MA plans are responsible for overall health care costs, they might have more integrated systems and greater incentives than standalone prescription drug plans to provide enrollees medications and information effectively, including, since 2010, quality bonus payments to these MA plans under provisions of the Affordable Care Act.

Unveiling SEER-CAHPS®: a new data resource for quality of care research

BACKGROUND

Since 1990, the National Cancer Institute (NCI) and Centers for Medicare and Medicaid Services (CMS) have collaborated to create linked data resources to improve our understanding of patterns of care, health care costs, and trends in utilization. However, existing data linkages have not included measures of patient experiences with care.

OBJECTIVE

To describe a new resource for quality of care research based on a linkage between the Medicare Consumer Assessment of Healthcare Providers and Systems (CAHPS®) patient surveys and the NCI's Surveillance, Epidemiology and End Results (SEER) data.

DESIGN

This is an observational study of CAHPS respondents and includes both fee-for-service and Medicare Advantage beneficiaries with and without cancer. The data linkage includes: CAHPS survey data collected between 1998 and 2010 to assess patient reports on multiple aspects of their care, such as access to needed and timely care, doctor communication, as well as patients' global ratings of their personal doctor, specialists, overall health care, and their health plan; SEER registry data (1973-2007) on cancer site, stage, treatment, death information, and patient demographics; and longitudinal Medicare claims data (2002-2011) for fee-for-service beneficiaries on utilization and costs of care.

PARTICIPANTS

In total, 150,750 respondents were in the cancer cohort and 571,318 were in the non-cancer cohort.

MAIN MEASURES

The data linkage includes SEER data on cancer site, stage, treatment, death information, and patient demographics, in addition to longitudinal data from Medicare claims and information on patient experiences from CAHPS surveys.

KEY RESULTS

Sizable proportions of cases from common cancers (e.g., breast, colorectal, prostate) and short-term survival cancers (e.g., pancreas) by time since diagnosis enable comparisons across the cancer care trajectory by MA vs. FFS coverage.

CONCLUSIONS

SEER-CAHPS is a valuable resource for information about Medicare beneficiaries' experiences of care across different diagnoses and treatment modalities, and enables comparisons by type of insurance.

Obese older adults report high satisfaction and positive experiences with care

Background

Obese, older adults often have multiple chronic conditions resulting in multiple health care encounters. However, their satisfaction and experiences with care are not well understood. The objective of this study was to examine the independent impact of obesity on patient satisfaction and experiences with care in adults 65 years of age and older with Medigap insurance.

Methods

Surveys were mailed to 53,286 randomly chosen adults with an AARP® Medicare Supplement Insurance Plan insured by UnitedHealthcare Insurance Company (for New York residents, UnitedHealthcare Insurance Company of New York) in 10 states. Following adjustment for non-response bias, multivariate regression modeling was used to adjust for demographic, socioeconomic and health status differences to estimate the independent impact of weight on satisfaction and experiences with care. Outcome variables included four global and four composite measures of satisfaction and experiences with care.

Results

21.4% of the respondents were obese. Relative to normal weight, obesity was significantly associated with higher patient satisfaction and better experiences with care in seven of the eight ratings measured.

Conclusions

Obese individuals were more satisfied and had better experiences with care. Obese individuals had more office visits and discussions about nutrition, exercise and medical checks. This may have led to increased attentiveness to care, explaining the increase in satisfaction and better experiences with care. Given the high level of satisfaction and experiences with care in older, obese adults, opportunities exist for clinicians to address weight concerns in this population.

The effect of item screeners on the quality of patient survey data: a randomized experiment of ambulatory care experience measures

BACKGROUND

The use of item screeners is viewed as an essential feature of quality survey design because only respondents who are 'qualified' to answer questions that apply to a subset of the sample are directed to answer. However, empirical evidence supporting this view is scant.

OBJECTIVE

This study compares data quality resulting from the administration of ambulatory care experience measures that use item screeners versus tailored 'not applicable' options in response scales.

METHODS

Patients from the practices of 367 primary care physicians in 65 medical groups were randomly assigned to receive one of two versions of a well validated ambulatory care experience survey. Respondents (n = 2240) represent random samples of active established patients from participating physicians' panels.The 'screener' survey version included item screeners for five test items and the 'no screener' version included tailored 'not applicable' options in response scales instead of using screeners.The main outcomes measures were data quality resulting from the two item versions, including the mean item scores, the level of missing values, outgoing patient sample sizes needed to achieve adequate medical group-level reliability, and the relative ranking of medical groups.

RESULTS

Mean survey item scores generally did not differ by version. There were consistently fewer respondents to the 'screener' versions than 'no screener' versions. However, because the 'screener' versions improved measurement precision, smaller outgoing patient samples were needed to achieve adequate medical group-level reliability for four of the five items than for the 'no screener' version. The relative ranking of medical groups did not differ by item version.

CONCLUSION

Screeners appear to reduce noise by ensuring that respondents who are not 'qualified' to answer a question are screened out instead of providing unreliable responses. The increased precision resulting from 'screener' versions appears to more than offset the higher item non-response rates compared with 'no screener' versions.

Examining multiple sources of differential item functioning on the Clinician & Group CAHPS® survey

OBJECTIVE

To evaluate psychometric properties of a widely used patient experience survey.

DATA SOURCES

English-language responses to the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (CG-CAHPS®) survey (n = 12,244) from a 2008 quality improvement initiative involving eight southern California medical groups.

METHODS

We used an iterative hybrid ordinal logistic regression/item response theory differential item functioning (DIF) algorithm to identify items with DIF related to patient sociodemographic characteristics, duration of the physician-patient relationship, number of physician visits, and self-rated physical and mental health. We accounted for all sources of DIF and determined its cumulative impact.

PRINCIPAL FINDINGS

The upper end of the CG-CAHPS® performance range is measured with low precision. With sensitive settings, some items were found to have DIF. However, overall DIF impact was negligible, as 0.14 percent of participants had salient DIF impact. Latinos who spoke predominantly English at home had the highest prevalence of salient DIF impact at 0.26 percent.

CONCLUSIONS

The CG-CAHPS® functions similarly across commercially insured respondents from diverse backgrounds. Consequently, previously documented racial and ethnic group differences likely reflect true differences rather than measurement bias. The impact of low precision at the upper end of the scale should be clarified.

Market variations in intensity of Medicare service use and beneficiary experiences with care

OBJECTIVE

Examine associations between patient experiences with care and service use across markets.

DATA SOURCES/STUDY SETTING

Medicare fee-for-service (FFS) and managed care (Medicare Advantage [MA]) beneficiaries in 306 markets from the 2003 Consumer Assessments of Healthcare Providers and Systems (CAHPS) surveys. Resource use intensity is measured by the 2003 end-of-life expenditure index.

STUDY DESIGN

We estimated correlations and linear regressions of eight measures of case-mix-adjusted beneficiary experiences with intensity of service use across markets.

DATA COLLECTION/EXTRACTION

We merged CAHPS data with service use data, excluding beneficiaries under 65 years of age or receiving Medicaid.

PRINCIPAL FINDINGS

Overall, higher intensity use was associated (p<.05) with worse (seven measures) or no better care experiences (two measures). In higher-intensity markets, Medicare FFS and MA beneficiaries reported more problems getting care quickly and less helpful office staff. However, Medicare FFS beneficiaries in higher-intensity markets reported higher overall ratings of their personal physician and main specialist. Medicare MA beneficiaries in higher-intensity markets also reported worse quality of communication with physicians, ability to get needed care, and overall ratings of care.

CONCLUSIONS

Medicare beneficiaries in markets characterized by high service use did not report better experiences with care. This trend was strongest for those in managed care.

Factors associated with health-related quality of life among an older population in a largely rural western region

CONTEXT

As elderly people become a larger proportion of the rural population, it is important to identify those at risk for poor health. Predictors of health-related quality of life can be useful in designing interventions.

PURPOSE

One objective of the present study was to profile the health-related quality of life of community-dwelling, elderly people in a southwestern region of the United States. A related objective was to identify the principal factors associated with health-related quality of life, thereby identifying population subgroups in greatest need of health or social services.

METHODS

A telephone survey of approximately 5,000 individuals 65 years and older collected data on need for assistance with activities of daily living, physical and mental health-related quality of life, and worry about health status measures. A modified version of the Behavioral Model was used to more clearly distinguish the different groups at risk for poor health.

FINDINGS

Those groups of community-dwelling, elderly people in the poorest health were older than 75 years, had less than a high school education, were retired or unemployed, and had low household income. No differences were found by urban, rural, and frontier residence.

CONCLUSIONS

To maintain the physical, social, and psychological health of older people residing in rural and urban areas, social services, medical care, and supportive services are needed, particularly among the most socially and economically disadvantaged.

Plan, geographical, and temporal variation of consumer assessments of ambulatory health care

OBJECTIVE

To quantify contributions of health plans and geography to variation in consumer assessments of health plan quality.

DATA SOURCES

Responses of beneficiaries of Medicare managed care plans to the Consumer Assessment of Health Plans Study (CAHPS(R)) survey. Our data included more than 700,000 survey responses assessing 381 Medicare managed care (MMC) contracts over a period of five years.

STUDY DESIGN

The survey was administered to a nationally representative sample of beneficiaries of Medicare managed care plans.

PRINCIPAL FINDINGS

Member assessments of their health plans, customer service functions, and prescription drug benefits varied most across health plans; these also varied the most over time. Assessments of direct interactions with doctors and their practices were more affected by geographical location, and these assessments were quite stable over time. A health plan's global rating often changed significantly between consecutive years, but only rarely were there such changes in ratings of care or doctor. Nationally, mean assessments tended to decrease over the study period.

CONCLUSIONS

Our findings suggest that ratings of plans and reports about customer service and prescription access are affected by plan policies, benefits design, and administrative structures that can be changed relatively quickly. Conversely, assessments of other aspects of care are largely determined by characteristics of provider networks that are relatively stable. A consumer survey is unlikely to detect meaningful changes in quality of care from year to year unless quality improvement measures are developed that have substantially larger effects, possibly through area-wide initiatives, than historical temporal variations in quality.

Teamwork Culture and Patient Satisfaction in Hospitals

BACKGROUND

A growing line of research indicates a positive relationship between a healthcare organization's culture and various performance measures. In these studies, a key cultural characteristic is the emphasis placed on teamwork. None of the studies, however, have examined teamwork culture relative to patient satisfaction, which is now 1 of the most widely used performance measures for healthcare organizations.

OBJECTIVES

This study investigated the relationship between teamwork culture of hospitals and patient reports of their satisfaction with the care they received.

METHODS

The study setting was the Veterans Health Administration (VHA), Department of Veterans Affairs. The study sample consisted of 125 VHA hospitals for which independent and valid sources of data for culture and patient satisfaction were obtained. Each hospital's culture was assessed relative to 4 dimensions: teamwork, entrepreneurial, bureaucratic, and rational. Patient satisfaction data were available for both inpatient and outpatient settings.

RESULTS

Results from multivariate regression analyses indicated a significant and positive relation between teamwork culture and patient satisfaction for inpatient care, and a significant and negative relation between bureaucratic culture and patient satisfaction for inpatient care. Additional analyses revealed an almost 1 standard deviation difference in patient satisfaction scores between hospitals in the top third and bottom third of the distribution for the teamwork culture measure.

CONCLUSION

Study results suggest that hospitals and possibly other healthcare organizations should strive to develop a culture emphasizing teamwork and deemphasizing those aspects of bureaucracy that are not essential to assuring efficiency and quality care.

Patient reports and ratings of individual physicians: an evaluation of the DoctorGuide and Consumer Assessment of Health Plans Study provider-level surveys

The objective of this study was to compare physician-level survey instruments and estimate the number of patients needed per physician to provide reliable estimates of health care. The setting consisted of 3 health plans and 1 large physician group in the greater Cincinnati metro area. Surveys were mailed to patients of 100 primary care physicians. Patients were mailed either the Consumer Assessment of Health Plans Study (CAHPS) or DoctorGuide survey instrument. A total of 4245 CAHPS surveys and 5519 DoctorGuide surveys were returned. Internal consistency reliability estimates for the multi-item scales (access to care, communication, and preventive care) for both surveys were adequate. The number of patient responses needed to obtain a reliability of 0.70 at the physician level for the access to care, communication, and preventive care scales were 32, 43, and 38, respectively, for the CAHPS survey and 26, 25, and 47, respectively, for the DoctorGuide survey. These results indicate similar and parallel psychometric performance for the DoctorGuide and CAHPS survey instruments.

Dimensions of plan performance for sick and healthy members on the Consumer Assessments of Health Plans Study 2.0 survey

BACKGROUND

The quality of health plan care may differ for members in good and poor health.

OBJECTIVE

To determine whether reports from sick and healthy members reflect distinct aspects of plan performance.

RESEARCH DESIGN

Mean health plan scores were analyzed on the 1998 and 1999 Medicare Managed Care (MMC) Consumer Assessments of Health Plans (CAHPS) surveys, treating responses from sick and healthy members as separate plan measures. Alternative definitions of health were compared and the one that defined groups with the most distinct experiences was selected. Using factor analysis, composites of report items defined for these groups were identified. Mean ratings were regressed on these composites.

SUBJECTS

Two hundred ninety thousand seven hundred thirty-nine Medicare managed care beneficiaries from 381 health plan-reporting units.

MEASURES

MMC-CAHPS survey responses, including four overall ratings and 30 specific report items.

RESULTS

A question about general health status best defined subgroups with distinct experiences. Report items grouped into eight factors: care for healthy members, care for sick members, finding and communicating with a doctor for sick members, plan customer service, plan-provided medical services and equipment, vaccinations, prescriptions, and smoking cessation advice. Ratings by each subgroup were generally most strongly predicted by reports on care for the same subgroup and by customer service and plan-provided services (for ratings of plan) and access to doctors.

CONCLUSIONS

Reports from sick and healthy members measure distinct dimensions of health plan quality, especially in the domain of patient care. Distinguishing these dimensions might help in informing consumers and targeting quality improvement efforts.

Factors affecting response rates to the Consumer Assessment of Health Plans Study survey

OBJECTIVES

Assess the determinants of nonresponse to a consumer health care survey.

METHODS

The first (1997; CAHPS 1.0) and third (1999; CAHPS 2.0) Medicare managed care (MMC) CAHPS surveys collected data on 215 and 365 health plan reporting units, respectively. Data indicated which beneficiaries responded by mail, responded by phone, could not be located, and did not respond. InterStudy data described plan characteristics. chi2 tests and logistic regression models, adjusted for clustering by plan, were used to test associations of individual and plan characteristics with availability of good contact information and response given good contact information.

RESULTS

Response rates in the 1997 and 1999 surveys were 75% and 80%, respectively. Older and disabled beneficiaries, women, nonwhite beneficiaries, and persons living in areas with more residents who were nonwhite, on public assistance, and less educated had lower response rates. These associations were partly explained by the distribution of bad contact information, but even among beneficiaries who could be located plan response rates varied greatly. For-profit plans are significantly more likely to have high rates of bad contact information and lower response rates. Telephone follow-up improved the sociodemographic representativeness of the sample, for both high and low response rate plans.

CONCLUSION

CAHPS-MMC survey procedures, in particular telephone follow-up, have resulted in high response rates, and current case-mix strategies compensate for some of the remaining effects of differing response rates on comparisons among plans. Further efforts to explore the determinants of response rates are warranted.

Confirmatory factor analysis of the Consumer Assessment of Health Plans Study (CAHPS) 1.0 Core Survey

The National Consumer Assessment of Health Plans Study (CAHPS) Benchmarking Database was used to assess the factor structure and invariance of the CAHPS 1.0 Core Survey. Separate analyses were conducted with Latino and non-Latino Caucasian consumers drawn from commercial and Medicaid sectors (N = 15,092). Results demonstrated that the 23 CAHPS 1.0 report items measure consumer reports of experiences with 5 aspects of health plan performance: Access to Care, Timeliness of Care, Provider Communication, Health Plan Consumer Service, and Office Staff Helpfulness. Four items assessed global ratings of care. Analyses revealed an identical pattern of fixed and free factor loadings across all samples. Magnitude of factor loadings and correlations among factors was essentially equivalent within a common health service sector. A higher order factor analysis revealed that rating and reports of care showed marked convergence.

Confirmatory factor analysis of the Consumer Assessment of Health Plans Study (CAHPS) 1.0 Core Survey

The National Consumer Assessment of Health Plans Study (CAHPS) Benchmarking Database was used to assess the factor structure and invariance of the CAHPS 1.0 Core Survey. Separate analyses were conducted with Latino and non-Latino Caucasian consumers drawn from commercial and Medicaid sectors (N = 15,092). Results demonstrated that the 23 CAHPS 1.0 report items measure consumer reports of experiences with 5 aspects of health plan performance: Access to Care, Timeliness of Care, Provider Communication, Health Plan Consumer Service, and Office Staff Helpfulness. Four items assessed global ratings of care. Analyses revealed an identical pattern of fixed and free factor loadings across all samples. Magnitude of factor loadings and correlations among factors was essentially equivalent within a common health service sector. A higher order factor analysis revealed that rating and reports of care showed marked convergence.

Patient satisfaction with hospital care: effects of demographic and institutional characteristics

BACKGROUND

There are a growing number of efforts to compare the service quality of health care organizations on the basis of patient satisfaction data. Such efforts inevitably raise questions about the fairness of the comparisons. Fair comparisons presumably should not penalize (or reward) health care organizations for factors that influence satisfaction scores but are not within the control of managers or clinicians. On the basis of previous research, these factors might include the demographic characteristics of patients (eg, age) and the institutional characteristics (eg, size) of the health care organizations where care was received.

OBJECTIVES

The goal of this study was to examine the extent to which a patient's satisfaction scores are related to both his/her demographic characteristics and the institutional characteristics of the health care organization where care was received.

METHODS

We conducted an analysis of secondary data from the Veterans Health Administration (VHA), US Department of Veterans Affairs. The database contained patient responses to self-administered satisfaction questionnaires and information about demographic characteristics. Additional data from VHA were obtained regarding the institutional characteristics of the hospitals where patients received their care.

RESULTS

Among demographic characteristics, age, health status, and race consistently had a statistically significant effect on satisfaction scores. Among the institutional characteristics, hospital size consistently had a significant effect on patient satisfaction scores.

CONCLUSIONS

Study results can be interpreted as justifying the need to adjust patient satisfaction scores for differences in patient population among health care organizations. However, from a policy perspective, such adjustments may ultimately create a disincentive for health care organizations to customize their care.

Sex differences in assessments of the quality of Medicare managed care

Women rate their care slightly more positively than men on the Medicare managed care Consumer Assessments of Health Plans Study (CAHPS) survey. On four of five composites, women have comparable or slightly more positive composite scores than men. Responses to individual questions in 1997 indicate that women may have slightly more problems getting referrals, equipment, and assistance. In 1998, there were no differences in difficulty getting a referral or assistance, but women were less likely to say their plan provided help, services, and equipment. Women were less likely to get a flu shot in both years. Further monitoring of CAHPS data on sex differences in difficulty getting needed medical equipment are warranted.

Dimensions of consumer-assessed quality of Medicare managed-care health plans

OBJECTIVES

We investigated relationships at the health-plan level among member ratings of and reports on plans in the Consumer Assessment of Health Plans Survey (CAHPS). We sought a more parsimonious description of the reports that can be used in analyses of the distribution and correlates of consumer-assessed quality.

SUBJECTS

There were 89,419 Medicare beneficiaries enrolled in 212 Medicare managed-care health plans who responded to CAHPS in 1998.

MEASURES

There were 39 survey items measuring consumer ratings of and reports on care.

METHODS

We adjusted correlations for sampling variability in the plan means and performed a principal factor analysis of the report items with oblique rotation. We grouped items that loaded heavily on the different factors, formed composites, and regressed rating items on the report composites.

RESULTS

Four factors explained 75% of the variance in the reports. The corresponding groups of items were concerned with the following subjects: (1) interactions around delivery of care in the doctor's office; (2) customer service from the plan; (3) access to medical services provided by the plan, such as specialist care, equipment, therapy, or drugs; and (4) advice on health-promoting activities. Corrected Cronbach alpha for composites were 0.97, 0.93, 0.86, and 0.60. The "delivery" composite was strongly predictive of overall ratings of care, doctor, and specialist; the "customer" composite was strongly predictive of overall ratings of the plan.

CONCLUSIONS

CAHPS distinguishes among dimensions of between-plan variability of consumer-assessed quality. Different global ratings are related to distinct groups of consumer reports on their experiences.

Psychometric properties of the CAHPS 1.0 survey measures. Consumer Assessment of Health Plans Study

OBJECTIVES

Consumer surveys are being used increasingly to assess the quality of care provided by health plans, physician groups, and clinicians. The purpose of the Consumer Assessment of Health Plans Study (CAHPS) is to develop an integrated and standardized set of surveys designed to collect reliable and valid information about health plan performance from consumers. This article reports psychometric results for the CAHPS 1.0 survey items in samples of individuals with Medicaid or private health insurance coverage.

METHODS

Reliability estimates for CAHPS 1.0 measures were estimated in a sample of 5,878 persons on Medicaid and 11,393 persons with private health insurance. Correlations of the CAHPS global rating of the health plan with willingness to recommend the plan and intention to re-enroll were estimated in a sample of 313 persons on Medicaid. The association of the rating of the health plan with ratings using a 5-point Excellent-to-Poor response scale also was investigated in the latter sample and in a sample of 539 persons with private health insurance.

RESULTS

The CAHPS measures appeared to have good reliability, particularly at the health-plan level. Responses from 300 consumers per health plan tend to yield estimates that are reliable enough for health plan comparisons, especially among the privately insured. The global health plan rating was significantly correlated with consumers' willingness to recommend the plan to family and friends and to their intention to re-enroll in the plan if given a choice.

CONCLUSIONS

The CAHPS 1.0 survey instrument appears to have excellent psychometric properties.

The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Consumer Assessment of Health Plans Study

OBJECTIVES

The main goal of the Consumer Assessments of Health Plans Study (CAHPS) is to develop an integrated set of tested, standardized surveys to obtain meaningful information from health plan enrollees about their experiences. The CAHPS project benefits from the complementary strengths of psychometric and cognitive testing.

METHODS

The CAHPS team conducted 150 cognitive interviews across three organizations in different geographic locations using multiple interview methods with different consumer populations. This article explains how cognitive testing was used in the CAHPS survey development process and shares the main findings from the cognitive interviews.

RESULTS

A modified report format is more appropriate when asking about specific aspects of plan enrollees' experiences, whereas a rating format is useful for asking about overall assessments. Specifying a longer reference period is preferable to asking about the most recent visit when capturing experiences with care, because some respondents get frustrated when they cannot include experiences other than the most recent visit. Explicit screeners and tailored inapplicable response categories are beneficial in mail questionnaires, so people know that they should not answer questions about which they have no relevant experience.

CONCLUSION

Cognitive testing was integral in the development and refinement of the CAHPS instrument. The cognitive testing findings contributed to an improved instrument that should capture consumers' health care and plan experiences with less response error than one not subjected to such testing. The cognitive testing process and findings can be useful to other researchers with similar survey development goals.