Consumer Assessment of Health Plans Study (CAHPS). Foreword

Differences in health care quality for children and adults under managed care: justification for separate quality assessments?

OBJECTIVE

To assess reported results of health care quality for children and adults in managed systems of care and to determine if variations exist between reported quality results for adults and children within the same plan.

METHODS

We utilized Consumer Assessment of Health Plan Survey results reported from 424 managed care plans to the National Committee for Quality Assurance in 1999. Responses from 218 530 adults (515 per plan, 424 plans) and 55 081 parents of children 0-12 years of age (304 per plan, 181 plans) were available. Restricting analyses to the 178 plans reporting both adult and child results, we performed matched-pairs analyses to test the hypothesis that child results would be the same as their adult counterparts within the same plan. Regression methods were employed to test for potential demographic differences explaining observed differences.

RESULTS

Within the same plan, reported results for care provided by specialists and primary care physicians to adults and children in the same plan revealed marked variation, including rating of doctor (Spearman correlation coefficient, r(S) =.504) and rating of specialist (r(S) =.326). Conversely, assessments of activities related directly to health plan activities showed little variation, including rating of health plan (r(S) =.850) and claims processing (r(S) =.857). Differences in demographic characteristics between adults and child survey respondents do not appear to explain observed variations.

CONCLUSIONS

Separate quality of care assessments for adults and children within the same managed care system identify significant differences in reported quality. Having health plan quality information about adult care does not serve as a proxy for needed information on children, particularly the care related to primary care and specialist providers. Areas of health plan assessment common to both adults and children (eg, claims processing) could be replaced with more targeted assessments of importance to parents and purchasers (eg, children with chronic conditions).

Similarities and differences in choosing health plans

BACKGROUND

Increasingly, consumers have multiple health insurance options. New information is being developed to help consumers with these choices.

OBJECTIVES

To study similarities and differences in how the publicly and privately insured choose health plans. To explore the effect of traditional enrollment materials and reports developed by the Consumer Assessment of Health Plans Study (CAHPS) on consumers' perceptions and decision-making.

RESEARCH DESIGN

Using data from eight CAHPS demonstrations, we tested for significant differences across consumers with employer-sponsored insurance, Medicaid, and Medicare.

SUBJECTS

Approximately 10,000 consumers with employer-sponsored, Medicaid, and Medicare health plans.

MEASURES

Perceptions of the health plan selection process, use of information sources, and reactions to and use of traditional enrollment materials and CAHPS reports.

RESULTS

Most consumers with all types of insurance thought that choosing a health plan was important and obtained information from multiple sources. Choosing a plan was more difficult for Medicare and Medicaid recipients than for the privately insured. When choosing a plan, Medicaid recipients cared most about convenience and access, whereas the privately insured emphasized providers and costs. The percentage of consumers who looked at and remembered the CAHPS report varied widely from 24% to 77%. In all but one of the demonstration sites, most consumers spent less than 30 minutes looking at the CAHPS report.

CONCLUSIONS

Group sponsors and the developers of information interventions such as CAHPS may need to invest in developing and testing different reporting approaches for Medicare, Medicaid, and privately insured consumers.

Consumer assessments of care for children and adults in health plans: how do they compare?

OBJECTIVES

The Consumer Assessment of Health Plans Survey (CAHPS) includes an adult version and also a child version for parents or caretakers to rate children's care in health plans. This study examined how adult and child assessments differed in ranking health plans and explored whether the differences justified the additional cost and respondent burden in administering both surveys.

METHODS

Data were from 136 commercial health plans participating in the National CAHPS Benchmarking Database, with 80,539 adults and 40,003 children. We compared mean assessments for adults and children on four global ratings and five composites, and determined respondent characteristics predictive of these assessments using regression analysis. We calculated correlations of plan mean scores for adults and children and kappa statistics for agreement when health plans are ranked as above average, average, or below average performers based on adult and child scores.

RESULTS

CAHPS scores for children were significantly (P <0.001) higher than those for adults, except for customer service (lower for children) and specialist ratings. Similar respondent characteristics predicted adult and child scores. Plan-level correlations between corresponding adult and child mean scores were moderate to high (r = 0.60-0.85), which translate into fair to moderate agreement (kappa = 0.27-0.61) in ranking health plans.

CONCLUSIONS

Adult and child CAHPS provide similar scores and plan rankings on many aspects of care. Child reports include information that may be useful for consumer choice and to health plans for targeting quality improvement. Methods should be developed for assessing health care for children that minimize cost and respondent burden.

The characteristics of patient satisfaction measures

Information on patient satisfaction is considered a way of including patients' perspectives in the planning and assessment of services. The study of patient satisfaction is a relatively new field, and despite the surge in popularity and use of satisfaction measures during the past three decades, different issues remain to be explored. This is not meant to dissuade clinicians from using satisfaction measures, but rather to allow them to proceed in a thoughtful way, recognizing what these measures can reasonably show us about patients' perceptions of the care and treatment interventions they receive. The proposed approach to classifying the characteristics of patient satisfaction measures should help to highlight potential reasons for variation in results when satisfaction measures perform differently and will be of value if it increases the specificity with which clinicians select measures to achieve their purposes.

Assessing Medicaid recipient access and satisfaction. Fee-for-service, case management, and capitation

Medicaid increasingly requires enrollment in managed care programs. This study assessed access to care, satisfaction with care, and appointment wait times during the transition from fee for service to managed care using three annual Medicaid recipient surveys. There was little evidence of dissatisfaction or poorer access among managed care recipients. Fee-for-service recipients, compared to primary care case management, reported greater general (91 vs. 78%, p < .01) and specialty care access (92 vs. 80%, p < .01). When appointments were required, adult HMO enrollees, compared to case management, had longer waits for routine care in the second (5.8 +/- 8.2 days vs. 4.0 +/- 6.6) and third surveys (5.5 +/- 6.9 days vs. 3.8 +/- 7.3); waits for other appointments did not consistently differ by program. There were no significant program differences in overall satisfaction. Findings are tempered by the potential for response bias and geographic confounding. Continued monitoring is crucial to assure that access and satisfaction remain high in Medicaid managed care.

Quality and outcomes of care for older women with chronic disease

This is the fourth in a series of six papers that will be published from the 1999 lecture series on "Quality Assessment in Women's Health Care" held at the University of Michigan School of Public Health. The lectures are presented by leaders in women's health research, and they explore key issues in the definition, measurement, and improvement of quality in women's health services. The series is supported by an unrestricted educational grant from Pfizer Inc. and is presented by the Interdepartmental Concentration in Reproductive and Women's Health at the University of Michigan School of Public Health; the University of Michigan National Center of Excellence in Women's Health; and the Michigan Initiative for Women's Health. The series coordinator is Carol S. Weisman, PhD, and Catherine L. Maroney prepared the summary of the discussants' comments.

Evaluating the equivalence of health care ratings by whites and Hispanics

PURPOSE

This study was designed to assess the equivalence of a health care ratings scale administered to non-Hispanic white and Hispanic survey respondents.

METHODS

We sent 18,840 questionnaires to a random sample of patients receiving medical care from a physician group association concentrated in the western United States; 7,093 were returned (59% adjusted response rate). Approximately 90% of survey respondents self-identified as white/Caucasian (n = 5,508) or Hispanic/Latino (n = 713). Interpersonal and technical aspects of medical care were assessed with 9 items, all administered with a 7-point response format: the best, excellent, very good, good, fair, poor, and very poor, with a "not applicable" option. Item response theory procedures were used to test for differential item functioning between white and Hispanic respondents.

RESULTS

Hispanics were found to be significantly more dissatisfied with care than whites (effect size=0.27; P <0.05). Of the 9 test items, 2 had statistically significant differential item functioning (P <0.05): reassurance and support offered by your doctors and staff and quality of examinations received. However, summative scale scores and test characteristic curves for whites and Hispanics were similar whether or not these items were included in the scale.

CONCLUSIONS

Despite some differences in item functioning, valid satisfaction-with-care comparisons between whites and Hispanics are possible. Thus, disparities in satisfaction ratings between whites and Hispanics should not be ascribed to measurement bias but should be viewed as arising from actual differences in experiences with care.

Socioeconomic status and dissatisfaction among HMO enrollees

OBJECTIVES

Member satisfaction is commonly used as an indicator of the quality of care delivered by health plans. Yet few contemporary studies have explored the extent to which individual patient characteristics influence dissatisfaction in HMOs. We sought to determine whether socioeconomic status is associated with enrollee dissatisfaction.

METHODS

Data are from a cross-sectional, telephone survey of a probability sample of adults enrolled in New Jersey HMOs in 1998 (n = 7,983). Health plan ratings were elicited as part of the Consumer Assessment of Health Plans Study (CAHPS) survey, along with income, education, and race/ethnicity. Other factors known to influence satisfaction (age, gender, health status, extent of plan choice, and payment for plan) were also ascertained.

RESULTS

Socioeconomically advantaged enrollees were more likely to give low ratings to their health plans. In a multivariate logistic regression model, those with incomes exceeding $100,000 had 1.65 times the odds of being dissatisfied compared with those with family incomes less than $25,000 (P <0.001); those with a college education had 2.53 times the odds of being dissatisfied than those who had not completed high school (P <0.001). However, among enrollees in their plans for > or =5 years, those in the lowest income group were significantly more dissatisfied than higher-income enrollees.

CONCLUSIONS

Among New Jersey HMO enrollees, higher socioeconomic status (SES) is associated with greater dissatisfaction. Although based on cross-sectional data and thus preliminary, the evidence presented here also suggests that the SES-dissatisfaction relationship varies as a function of duration of enrollment. Further research using longitudinal data could shed additional light on the SES-dissatisfaction link.

Consumer responses to health plan report cards in two markets

BACKGROUND

Health plans can compete on quality when consumers have helpful information. Report cards strive to meet this need, but consumer responses have not been measured.

OBJECTIVES

The objectives of this study were (1) to compare consumer responses to report cards in 2 markets, (2) to determine how personal characteristics relate to exposure, and (3) to assess the perceived helpfulness of the report cards.

RESEARCH DESIGN

A postenrollment survey was used.

SUBJECTS

The study included 784 employees of Monsanto (St Louis, 1996) and 670 employees of a health care purchasing cooperative (Denver, 1997).

DEPENDENT MEASURES

The dependent measures were (1) exposure, specifically remembering the report card, and intensity of reading it and (2) perceived helpfulness in learning about plan quality and in deciding to stay or switch.

RESULTS

Except for remembering seeing the report card (Denver, 47%; St Louis, 55%), the 2 groups did not differ. Forty percent read most or all of the report card; 82% found the report helpful in learning about quality; and 66% found it helpful in deciding to stay or switch. Employees who used patient survey information in their plan decision were more likely to remember seeing the report card (odds ratio [OR], 4.85), to read it intensely (OR, 2.84), and to find it helpful in learning about plan quality (OR, 3.04) and deciding whether to stay or switch plans (OR, 2.64).

CONCLUSIONS

Although the 2 samples differed markedly, their responses to report cards were similar. Exposure and helpfulness were related more to employee preferences for the type of information than to their health care decision needs.

Dimensions of consumer-assessed quality of Medicare managed-care health plans

OBJECTIVES

We investigated relationships at the health-plan level among member ratings of and reports on plans in the Consumer Assessment of Health Plans Survey (CAHPS). We sought a more parsimonious description of the reports that can be used in analyses of the distribution and correlates of consumer-assessed quality.

SUBJECTS

There were 89,419 Medicare beneficiaries enrolled in 212 Medicare managed-care health plans who responded to CAHPS in 1998.

MEASURES

There were 39 survey items measuring consumer ratings of and reports on care.

METHODS

We adjusted correlations for sampling variability in the plan means and performed a principal factor analysis of the report items with oblique rotation. We grouped items that loaded heavily on the different factors, formed composites, and regressed rating items on the report composites.

RESULTS

Four factors explained 75% of the variance in the reports. The corresponding groups of items were concerned with the following subjects: (1) interactions around delivery of care in the doctor's office; (2) customer service from the plan; (3) access to medical services provided by the plan, such as specialist care, equipment, therapy, or drugs; and (4) advice on health-promoting activities. Corrected Cronbach alpha for composites were 0.97, 0.93, 0.86, and 0.60. The "delivery" composite was strongly predictive of overall ratings of care, doctor, and specialist; the "customer" composite was strongly predictive of overall ratings of the plan.

CONCLUSIONS

CAHPS distinguishes among dimensions of between-plan variability of consumer-assessed quality. Different global ratings are related to distinct groups of consumer reports on their experiences.

Special issues in assessing care of Medicaid recipients

OBJECTIVES

The authors describe the process used to develop and test survey items targeted to Medicaid consumers for the Consumer Assessment of Health Plans Study (CAHPS). In addition, the authors highlight the special challenges in locating and surveying Medicaid recipients and provide recommendations for increasing response rates.

METHODS

The RAND CAHPS team reviewed the literature and existing questionnaires to identify health care issues and concepts important to Medicaid consumers. Three focus groups and 66 one-on-one cognitive interviews were conducted to test the relevance of our concepts and items and to identify additional concepts important to Medicaid consumers. After the cognitive interviews, the CAHPS Medicaid consumer survey was field tested using a sample of 930 adults and children receiving both Medicaid and Aid to Families with Dependent Children in Los Angeles County and Oklahoma. To determine if one particular mode were preferable for surveying a Medicaid population, our field test sample was divided randomly into a telephone-mode sample, a mixed-mode sample, and a second telephone-mode sample with enhanced locating procedures. Before finalizing the CAHPS 1.0 surveys, the full CAHPS item set was subjected to a formal literacy review.

RESULTS

The results of the focus groups and cognitive testing informed iterative versions of the list of concepts addressed by the Medicaid-targeted items. Concepts that were not relevant to Medicaid consumers or that consumers were unable to accurately attribute to a health plan were discarded. New concepts addressing important aspects of health care and the health care experience of Medicaid consumers were identified and added. Item wording and format were revised and refined based on the findings from focus groups, cognitive testing, the field test, and the formal literacy review. In the field test, the mixed-mode method achieved the best results with a 56% completion rate.

CONCLUSIONS

The testing and formatting efforts described in this article, in combination with a formal literacy review, led to the development of a Medicaid questionnaire that measures the important health care experiences of Medicaid consumers in a format that is "respondent-friendly." Our recommendations for surveying Medicaid recipients can benefit any survey of a Medicaid population.

Psychometric properties of the CAHPS 1.0 survey measures. Consumer Assessment of Health Plans Study

OBJECTIVES

Consumer surveys are being used increasingly to assess the quality of care provided by health plans, physician groups, and clinicians. The purpose of the Consumer Assessment of Health Plans Study (CAHPS) is to develop an integrated and standardized set of surveys designed to collect reliable and valid information about health plan performance from consumers. This article reports psychometric results for the CAHPS 1.0 survey items in samples of individuals with Medicaid or private health insurance coverage.

METHODS

Reliability estimates for CAHPS 1.0 measures were estimated in a sample of 5,878 persons on Medicaid and 11,393 persons with private health insurance. Correlations of the CAHPS global rating of the health plan with willingness to recommend the plan and intention to re-enroll were estimated in a sample of 313 persons on Medicaid. The association of the rating of the health plan with ratings using a 5-point Excellent-to-Poor response scale also was investigated in the latter sample and in a sample of 539 persons with private health insurance.

RESULTS

The CAHPS measures appeared to have good reliability, particularly at the health-plan level. Responses from 300 consumers per health plan tend to yield estimates that are reliable enough for health plan comparisons, especially among the privately insured. The global health plan rating was significantly correlated with consumers' willingness to recommend the plan to family and friends and to their intention to re-enroll in the plan if given a choice.

CONCLUSIONS

The CAHPS 1.0 survey instrument appears to have excellent psychometric properties.

The use of cognitive testing to develop and evaluate CAHPS 1.0 core survey items. Consumer Assessment of Health Plans Study

OBJECTIVES

The main goal of the Consumer Assessments of Health Plans Study (CAHPS) is to develop an integrated set of tested, standardized surveys to obtain meaningful information from health plan enrollees about their experiences. The CAHPS project benefits from the complementary strengths of psychometric and cognitive testing.

METHODS

The CAHPS team conducted 150 cognitive interviews across three organizations in different geographic locations using multiple interview methods with different consumer populations. This article explains how cognitive testing was used in the CAHPS survey development process and shares the main findings from the cognitive interviews.

RESULTS

A modified report format is more appropriate when asking about specific aspects of plan enrollees' experiences, whereas a rating format is useful for asking about overall assessments. Specifying a longer reference period is preferable to asking about the most recent visit when capturing experiences with care, because some respondents get frustrated when they cannot include experiences other than the most recent visit. Explicit screeners and tailored inapplicable response categories are beneficial in mail questionnaires, so people know that they should not answer questions about which they have no relevant experience.

CONCLUSION

Cognitive testing was integral in the development and refinement of the CAHPS instrument. The cognitive testing findings contributed to an improved instrument that should capture consumers' health care and plan experiences with less response error than one not subjected to such testing. The cognitive testing process and findings can be useful to other researchers with similar survey development goals.

The impact of having parents report about both their own and their children's experiences with health insurance plans

OBJECTIVES

The aim of this study was to determine whether parents rate their children's care differently when they also rate their own care than when they do not.

METHODS

Subjects were employees of Washington State who had been enrolled in a health plan for at least 6 months and who had at least one covered child. Subjects were randomly assigned to four study groups that were surveyed using different protocols. To assess the stability of responses over time, a follow-up telephone interview was conducted with individuals in two of the groups.

RESULTS

Parents or guardians who received both the Adult and Child Surveys were less likely to complete a survey than those who received only one survey. Responses to selected survey questions were quite stable between survey administrations. Parents who rated only their child's health care experiences generally gave more positive responses than those who also rated their own care, although few of these differences were statistically significant. This may have been due, in part, to the lower response rates in the latter group. The pairs of survey questions that ask about the adult's and child's experiences with the same aspects of care had moderate to high levels of association. The pair with the weakest association asked how clearly the doctor or nurse explained things to the adult or the child.

CONCLUSIONS

Sending both an adult and child survey to an adult could have an effect on the pattern of responses and result in lower response rates, but this might be a cost-effective way to collect reports about both adult and child health care.

Special issues addressed in the CAHPS survey of Medicare managed care beneficiaries. Consumer Assessment of Health Plans Study

OBJECTIVES

This article describes the process through which the MMC survey was developed and examines issues in using this survey with Medicare beneficiaries that have implications for all CAHPS surveys. These include the ability of Medicare beneficiaries to use MMC navigational features, whether access measures are meaningful for this population, and whether beneficiaries' familiarity with managed care influences their health plan assessments.

BACKGROUND

The Health Care Financing Administration (HCFA) is mandated to provide comparative plan information, based partly on consumer surveys, to Medicare beneficiaries. The Consumer Assessments of Health Plans Study (CAHPS) is an integrated set of tested, standardized surveys of health plan enrollees. To meet its goal, HCFA has invested in the development of a CAHPS survey of beneficiaries enrolled in Medicare Managed Care plans (MMC).

METHODS

Cognitive interviews were completed with 31 Medicare beneficiaries. A field test also was conducted with beneficiaries to examine patterns of survey response. A sample of 956 eligible individuals was selected from six health plans. Using a combination of mail and telephone data collection, 663 (69%) questionnaires were completed. This article provides selective results from these tests.

RESULTS

The use of screening questions, skip instructions, and tailored "not applicable" response options appeared to facilitate the response task. Some CAHPS access questions were not meaningful to Medicare beneficiaries. The data do not support the need to adjust for length of plan enrollment.

CONCLUSION

Analyses suggested changes to improve the MMC survey and to make other CAHPS surveys consistent with these changes.

Making survey results easy to report to consumers: how reporting needs guided survey design in CAHPS. Consumer Assessment of Health Plans Study

OBJECTIVES

CAHPS is designed to report information about health care quality from the consumer perspective. Enrollees are surveyed about their experiences with their health plan and medical care, and results are reported to other consumers choosing among health plans. Based on survey instruments designed to elicit reliable and valid information about health plan experiences from plan enrollees, the aim of the CAHPS team was to design a series of reporting products that present survey results so that consumers find the information understandable, meaningful, and useful in choosing among health plans.

METHODS

Design of the survey instruments and reporting products were closely linked. The approach to reporting was based on previous research on consumers' information interests and needs in evaluating and choosing among plans. Cognitive tests were conducted with consumers to get their reactions to mock-ups of various approaches to reporting CAHPS survey results.

RESULTS

Findings from previous research and cognitive testing, together with feedback from various experts and the public, were used to modify the survey questions, response options, and reporting formats to make it easier for consumers to understand and use reports. Changes included dropping topics of less interest to consumers, changing question wordings that were hard to understand, minimizing the number of different response categories, and revising questions to make them easier to group together for purposes of reporting.

CONCLUSIONS

The CAHPS focus on reporting results to consumers presented an unusual challenge for survey design, requiring close coordination between instrument design and report development to produce a survey and reporting kit that serves consumers' information needs.