Making comparative performance information more comprehensible: an experimental evaluation of the impact of formats on consumer understanding

Reliability and Validity of the Inpatient Rehabilitation Facility Discharge Mobility and Self-Care Quality Measures

OBJECTIVE

To describe the reliability and validity of the publicly reported facility-level quality measures Inpatient Rehabilitation Facility (IRF) Discharge Mobility Score for Medical Rehabilitation Patients ("Discharge mobility score") and IRF Discharge Self-Care Score for Medical Rehabilitation Patients ("Discharge self-care score").

DESIGN

Observational study using standardized patient assessment data to examine facility-level split-half reliability and construct validity of quality measure scores.

SETTING AND PARTICIPANTS

All IRFs (n = 1117) in the United States with at least 20 Medicare stays. Facility-level quality measure scores were calculated from 2017 data on 428,192 Medicare (fee-for-service and Medicare Advantage) IRF patient stays.

METHODS

Using clinician-reported assessment data, we calculated facility-level mobility and self-care quality measure scores and examined reliability of these scores using split-half analysis and Pearson product-moment correlations, Spearman rank correlations, and intraclass correlation coefficients (ICC_2,1). We examined construct validity of these scores by comparing facility-level quality measure scores by facility stroke disease-specific certification status.

RESULTS

Reported as percentages meeting or exceeding expectations, IRF quality measure scores ranged from 8.3% to 90.1% for mobility and 9.0% to 90.3% for self-care. IRF scores, when split in half to examine reliability, showed strong, positive correlations for the mobility (Pearson = 0.898, Spearman = 0.898, ICC = 0.898) and self-care (Pearson = 0.886, Spearman = 0.874, ICC = 0.886) scores. When stratified by provider volume, ICCs remained strong. Construct validity analyses showed IRFs with stroke disease-specific certification had higher mean and median scores than IRFs without certification, and a greater proportion of IRFs that were certified had higher scores.

CONCLUSION AND IMPLICATIONS

Our results support the reliability and construct validity of the IRF quality measures Discharge mobility and Discharge self-care scores. Reported as percentages meeting or exceeding expectations, these quality measures are designed to be more consumer-friendly compared to change scores.

Understanding quality data correctly: a randomized comparison of presentation formats among cancer patients and relatives

Background: Patient representatives are increasingly engaged in quality in health care, and even though quality data are publicly available, correct interpretation may be challenging. We designed a randomized study with the primary aim to examine the association between preferred data presentation format and the interpretation of quality data among cancer patients and relatives.Material and methods: Surveys were distributed to the Danish Cancer Society Citizens' Panel between 31 March and 14 April 2019 and 55% completed the survey (N = 464) including six storyboards that presented authentic quality data in table format, league table and point estimates. The storyboards were randomized to expose participants to the data in the three different formats and in varying presentation order. Logistic regression models were used to calculate Odds Ratios (ORs) and 95% confidence intervals (CIs) for the association between preferred presentation format, health literacy, education and cohabitation status as exposures and interpretation of quality data as outcome.Results: The majority of participants (97%) had high literacy and 57% had a medium or long higher education. A total of 60% found the questions difficult or very difficult and 33% were not able to correctly interpret at least one format. Correct interpretation was associated with preferred league table (OR = 1.62; 95% CI = 1.04-5.52) and if the data was presented in the preferred format. Medium and long education were associated with correct interpretation of at least one format (OR = 1.93; 95% CI = 1.16-3.21 and OR = 3.89; 95% CI = 1.90-7.95, respectively) while health literacy and cohabitation status were not.Conclusions: More than one third of the participants were not able to correctly interpret the data and the understanding of quality data improved with longer education and if the data was presented in the preferred format. Decision-makers should carefully consider displaying quality data according to preferred presentation format and to guide interpretation for individuals with short education.

Testing the effects on information use by older versus younger women of modality and narration style in a hospital report card

Background

Hospital report cards (HRCs) are usually presented in a textual and factual format, likely hampering information processing.

Objective

This study aimed to investigate the effects of audiovisual and narrative information in HRCs on user responses, and to test differences between older and younger women.

Design

A 2 (modality [textual vs. audiovisual]) × 3 (narration style [factual vs. process narrative vs. experience narrative]) online experiment was conducted. Information about breast cancer care was used as a case example. Age (younger [<65] vs. older [≥65]) was included as a potential effect modifier.

Setting and Participants

A total of 631 disease‐naïve women (M_age = 56.06) completed an online survey. The outcomes were perceived cognitive load, satisfaction, comprehension, information recall and decisional conflict. Data were analysed using AN(C)OVAs.

Results

Audiovisual (vs. textual) information resulted in higher information satisfaction across age groups, but was associated with lower comprehension in older women. An experience narrative (vs. factual information) increased satisfaction with attractiveness and emotional support of the information only in older women. A three‐way interaction effect was found, suggesting that older women were most satisfied with the comprehensibility of audiovisual factual or textual process narrative information. Younger women were most satisfied with the comprehensibility of audiovisual process narrative or textual factual information.

Discussion and Conclusion

Audiovisual and narrative information in an HRC showed beneficial effects on satisfaction measures. In particular, audiovisual information could be incorporated into HRCs to increase satisfaction with information.

Public Contribution

Lay persons helped in optimizing the visuals used in the stimulus materials by checking for clarity.

Consumer Engagement With Information on Performance: A Narrative Review

BACKGROUND AND OBJECTIVES

Efforts to provide information on quality of providers assume that consumers and patients are able to use information effectively to guide their decisions. We conducted a narrative literature review to better understand how consumers use information on quality to ultimately select a provider.

METHODS

We used Berwick's Pathways to Quality Improvement Framework to guide a narrative literature review to synthesize past research on consumer choice and to identify and evaluate factors affecting provider selection.

RESULTS

Reviewed articles reflected factors affecting provider selection, including: information needs, content and delivery of information, use and engagement with information, and social and behavioral mechanisms, such as trust. We also identified gaps in the literature for further study (eg, loyalty to physicians) that have been identified as important for optimal decision-making but are not well explored in the literature.

DISCUSSION

Health care providers and managers can learn from a more complete model of consumers' selection process to systematically evaluate and improve service provision and information for consumers. Administrators and providers may consider using patient feedback to identify ways to improve their quality and should streamline information for consumers to facilitate thorough, informed decision-making.

Best practices for data visualization: creating and evaluating a report for an evidence-based fall prevention program

This case report applied principles from the data visualization (DV) literature and feedback from nurses to develop an effective report to display adherence with an evidence-based fall prevention program. We tested the usability of the original and revised reports using a Health Information Technology Usability Evaluation Scale (Health-ITUES) customized for this project. Items were rated on a 5-point Likert scale, strongly disagree (1) to strongly agree (5). The literature emphasized that the ideal display maximizes the information communicated, minimizes the cognitive efforts involved with interpretation, and selects the correct type of display (eg, bar versus line graph). Semi-structured nurse interviews emphasized the value of simplified reports and meaningful data. The mean (standard deviation [SD]) Health-ITUES score for the original report was 3.86 (0.19) and increased to 4.29 (0.11) in the revised report (Mann Whitney U Test, z = -12.25, P < 0.001). Lessons learned from this study can inform report development for clinicians in implementation science.

Current Best Practice for Presenting Probabilities in Patient Decision Aids: Fundamental Principles

BACKGROUND

Shared decision making requires evidence to be conveyed to the patient in a way they can easily understand and compare. Patient decision aids facilitate this process. This article reviews the current evidence for how to present numerical probabilities within patient decision aids.

METHODS

Following the 2013 review method, we assembled a group of 9 international experts on risk communication across Australia, Germany, the Netherlands, the United Kingdom, and the United States. We expanded the topics covered in the first review to reflect emerging areas of research. Groups of 2 to 3 authors reviewed the relevant literature based on their expertise and wrote each section before review by the full authorship team.

RESULTS

Of 10 topics identified, we present 5 fundamental issues in this article. Although some topics resulted in clear guidance (presenting the chance an event will occur, addressing numerical skills), other topics (context/evaluative labels, conveying uncertainty, risk over time) continue to have evolving knowledge bases. We recommend presenting numbers over a set time period with a clear denominator, using consistent formats between outcomes and interventions to enable unbiased comparisons, and interpreting the numbers for the reader to meet the needs of varying numeracy.

DISCUSSION

Understanding how different numerical formats can bias risk perception will help decision aid developers communicate risks in a balanced, comprehensible manner and avoid accidental "nudging" toward a particular option. Decisions between probability formats need to consider the available evidence and user skills. The review may be useful for other areas of science communication in which unbiased presentation of probabilities is important.

Use of a quantitative data report in a hypothetical decision scenario for health policymaking: a computer-assisted laboratory study

BACKGROUND

Quantitative data reports are widely produced to inform health policy decisions. Policymakers are expected to critically assess provided information in order to incorporate the best available evidence into the decision-making process. Many other factors are known to influence this process, but little is known about how quantitative data reports are actually read. We explored the reading behavior of (future) health policy decision-makers, using innovative methods.

METHODS

We conducted a computer-assisted laboratory study, involving starting and advanced students in medicine and health sciences, and professionals as participants. They read a quantitative data report to inform a decision on the use of resources for long-term care in dementia in a hypothetical decision scenario. Data were collected through eye-tracking, questionnaires, and a brief interview. Eye-tracking data were used to generate 'heatmaps' and five measures of reading behavior. The questionnaires provided participants' perceptions of understandability and helpfulness as well as individual characteristics. Interviews documented reasons for attention to specific report sections. The quantitative analysis was largely descriptive, complemented by Pearson correlations. Interviews were analyzed by qualitative content analysis.

RESULTS

In total, 46 individuals participated [students (85%), professionals (15%)]. Eye-tracking observations showed that the participants spent equal time and attention for most parts of the presented report, but were less focused when reading the methods section. The qualitative content analysis identified 29 reasons for attention to a report section related to four topics. Eye-tracking measures were largely unrelated to participants' perceptions of understandability and helpfulness of the report.

CONCLUSIONS

Eye-tracking data added information on reading behaviors that were not captured by questionnaires or interviews with health decision-makers.

A randomized comparison between league tables and funnel plots to inform health care decision-making

Objective

Comparison of provider performance is commonly used to inform health care decision-making. Little attention has been paid to how data presentations influence decisions. This study analyzes differences in suggested actions by decision-makers informed by league tables or funnel plots.

Design

Decision-makers were invited to a survey and randomized to compare hospital performance using either league tables or funnel plots for four different measures within the area of cancer care. For each measure, decision-makers were asked to suggest actions towards 12-16 hospitals (no action, ask for more information, intervene) and provide feedback related to whether the information provided had been useful.

Setting

Swedish health care.

Participants

Two hundred and twenty-one decision-makers at administrative and clinical levels.

Intervention

Data presentations in the form of league tables or funnel plots.

Main outcome measures

Number of actions suggested by participants. Proportion of appropriate actions.

Results

For all four measures, decision-makers tended to suggest more actions based on the information provided in league tables compared to funnel plots (44% vs. 21%, P < 0.001). Actions were on average more appropriate for funnel plots. However, when using funnel plots, decision-makers more often missed to react even when appropriate.

Conclusions

The form of data presentation had an influence on decision-making. With league tables, decision-makers tended to suggest more actions compared to funnel plots. A difference in sensitivity and specificity conditioned by the form of presentation could also be identified, with different implications depending on the purpose of comparisons. Explanations and visualization aids are needed to support appropriate actions.

Feedback preferences of patients, professionals and health insurers in integrated head and neck cancer care

Background

Audit and feedback on professional practice and health care outcomes are the most often used interventions to change behaviour of professionals and improve quality of health care. However, limited information is available regarding preferred feedback for patients, professionals and health insurers.

Objective

Investigate the (differences in) preferences of receiving feedback between stakeholders, using the Dutch Head and Neck Audit as an example.

Methods

A total of 37 patients, medical specialists, allied health professionals and health insurers were interviewed using semi‐structured interviews. Questions focussed on: “Why,” “On what aspects” and “How” do you prefer to receive feedback on professional practice and health care outcomes?

Results

All stakeholders mentioned that feedback can improve health care by creating awareness, enabling self‐reflection and reflection on peers or colleagues, and by benchmarking to others. Patients prefer feedback on the actual professional practice that matches the health care received, whereas medical specialists and health insurers are interested mainly in health care outcomes. All stakeholders largely prefer a bar graph. Patients prefer a pie chart for patient‐reported outcomes and experiences, while Kaplan‐Meier survival curves are preferred by medical specialists. Feedback should be simple with firstly an overview, and 1‐4 times a year sent by e‐mail. Finally, patients and health professionals are cautious with regard to transparency of audit data.

Conclusions

This exploratory study shows how feedback preferences differ between stakeholders. Therefore, tailored reports are recommended. Using this information, effects of audit and feedback can be improved by adapting the feedback format and contents to the preferences of stakeholders.