Patient assessments and online ratings of quality care: a "wake-up call" for providers

Data Quality Issues With Physician-Rating Websites: Systematic Review

BACKGROUND

In recent years, online physician-rating websites have become prominent and exert considerable influence on patients' decisions. However, the quality of these decisions depends on the quality of data that these systems collect. Thus, there is a need to examine the various data quality issues with physician-rating websites.

OBJECTIVE

This study's objective was to identify and categorize the data quality issues afflicting physician-rating websites by reviewing the literature on online patient-reported physician ratings and reviews.

METHODS

We performed a systematic literature search in ACM Digital Library, EBSCO, Springer, PubMed, and Google Scholar. The search was limited to quantitative, qualitative, and mixed-method papers published in the English language from 2001 to 2020.

RESULTS

A total of 423 articles were screened. From these, 49 papers describing 18 unique data quality issues afflicting physician-rating websites were included. Using a data quality framework, we classified these issues into the following four categories: intrinsic, contextual, representational, and accessible. Among the papers, 53% (26/49) reported intrinsic data quality errors, 61% (30/49) highlighted contextual data quality issues, 8% (4/49) discussed representational data quality issues, and 27% (13/49) emphasized accessibility data quality. More than half the papers discussed multiple categories of data quality issues.

CONCLUSIONS

The results from this review demonstrate the presence of a range of data quality issues. While intrinsic and contextual factors have been well-researched, accessibility and representational issues warrant more attention from researchers, as well as practitioners. In particular, representational factors, such as the impact of inline advertisements and the positioning of positive reviews on the first few pages, are usually deliberate and result from the business model of physician-rating websites. The impact of these factors on data quality has not been addressed adequately and requires further investigation.

Association Between Web-Based Physician Ratings and Physician Disciplinary Convictions: Retrospective Observational Study

BACKGROUND

Physician rating websites are commonly used by the public, yet the relationship between web-based physician ratings and health care quality is not well understood.

OBJECTIVE

The objective of our study was to use physician disciplinary convictions as an extreme marker for poor physician quality and to investigate whether disciplined physicians have lower ratings than nondisciplined matched controls.

METHODS

This was a retrospective national observational study of all disciplined physicians in Canada (751 physicians, 2000 to 2013). We searched ratings (2005-2015) from the country's leading online physician rating website for this group, and for 751 matched controls according to gender, specialty, practice years, and location. We compared overall ratings (out of a score of 5) as well as mean ratings by the type of misconduct. We also compared ratings for each type of misconduct and punishment.

RESULTS

There were 62.7% (471/751) of convicted and disciplined physicians (cases) with web-based ratings and 64.6% (485/751) of nondisciplined physicians (controls) with ratings. Of 312 matched case-control pairs, disciplined physicians were rated lower than controls overall (3.62 vs 4.00; P<.001). Disciplined physicians had lower ratings for all types of misconduct and punishment-except for physicians disciplined for sexual offenses (n=90 pairs; 3.83 vs 3.86; P=.81). Sexual misconduct was the only category in which mean ratings for physicians were higher than those for other disciplined physicians (3.63 vs 3.35; P=.003).

CONCLUSIONS

Physicians convicted for disciplinary misconduct generally had lower web-based ratings. Physicians convicted of sexual misconduct did not have lower ratings and were rated higher than other disciplined physicians. These findings may have future implications for the identification of physicians providing poor-quality care.

Heartburn-Related Internet Searches and Trends of Interest across Six Western Countries: A Four-Year Retrospective Analysis Using Google Ads Keyword Planner

The internet is becoming the main source of health-related information. We aimed to investigate data regarding heartburn-related searches made by Google users from Australia, Canada, Germany, Poland, the United Kingdom, and the United States. We retrospectively analyzed data from Google Ads Keywords Planner. We extracted search volumes of keywords associated with “heartburn” for June 2015 to May 2019. The data were generated in the respective primary language. The number of searches per 1000 Google-user years was as follows: 177.4 (Australia), 178.1 (Canada), 123.8 (Germany), 199.7 (Poland), 152.5 (United Kingdom), and 194.5 (United States). The users were particularly interested in treatment (19.0 to 41.3%), diet (4.8 to 10.7%), symptoms (2.6 to 13.1%), and causes (3.7 to 10.0%). In all countries except Germany, the number of heartburn-related queries significantly increased over the analyzed period. For Canada, Germany, Poland, and the United Kingdom, query numbers were significantly lowest in summer; there was no significant seasonal trend for Australia and the United States. The number of heartburn-related queries has increased over the past four years, and a seasonal pattern may exist in certain regions. The trends in heartburn-related searches may reflect the scale of the complaint, and should be verified through future epidemiological studies.

A Comparison of Online Physician Ratings and Internal Patient-Submitted Ratings from a Large Healthcare System

BACKGROUND

Physician online ratings are ubiquitous and influential, but they also have their detractors. Given the lack of scientific survey methodology used in online ratings, some health systems have begun to publish their own internal patient-submitted ratings of physicians.

OBJECTIVE

The purpose of this study was to compare online physician ratings with internal ratings from a large healthcare system.

DESIGN

Retrospective cohort study comparing online ratings with internal ratings from a large healthcare system.

SETTING

Kaiser Permanente, a large integrated healthcare delivery system.

PARTICIPANTS

Physicians in the Southern California region of Kaiser Permanente, including all specialties with ambulatory clinic visits.

MAIN MEASURES

The primary outcome measure was correlation between online physician ratings and internal ratings from the integrated healthcare delivery system.

RESULTS

Of 5438 physicians who met inclusion and exclusion criteria, 4191 (77.1%) were rated both online and internally. The online ratings were based on a mean of 3.5 patient reviews, while the internal ratings were based on a mean of 119 survey returns. The overall correlation between the online and internal ratings was weak (Spearman's rho .23), but increased with the number of reviews used to formulate each online rating.

CONCLUSIONS

Physician online ratings did not correlate well with internal ratings from a large integrated healthcare delivery system, although the correlation increased with the number of reviews used to formulate each online rating. Given that many consumers are not aware of the statistical issues associated with small sample sizes, we would recommend that online rating websites refrain from displaying a physician's rating until the sample size is sufficiently large (for example, at least 15 patient reviews). However, hospitals and health systems may be able to provide better information for patients by publishing the internal ratings of their physicians.

Using online patient feedback to improve NHS services: the INQUIRE multimethod study

Background

Online customer feedback has become routine in many industries, but it has yet to be harnessed for service improvement in health care.

Objectives

To identify the current evidence on online patient feedback; to identify public and health professional attitudes and behaviour in relation to online patient feedback; to explore the experiences of patients in providing online feedback to the NHS; and to examine the practices and processes of online patient feedback within NHS trusts.

Design

A multimethod programme of five studies: (1) evidence synthesis and stakeholder consultation; (2) questionnaire survey of the public; (3) qualitative study of patients’ and carers’ experiences of creating and using online comment; (4) questionnaire surveys and a focus group of health-care professionals; and (5) ethnographic organisational case studies with four NHS secondary care provider organisations.

Setting

The UK.

Methods

We searched bibliographic databases and conducted hand-searches to January 2018. Synthesis was guided by themes arising from consultation with 15 stakeholders. We conducted a face-to-face survey of a representative sample of the UK population (n = 2036) and 37 purposively sampled qualitative semistructured interviews with people with experience of online feedback. We conducted online surveys of 1001 quota-sampled doctors and 749 nurses or midwives, and a focus group with five allied health professionals. We conducted ethnographic case studies at four NHS trusts, with a researcher spending 6–10 weeks at each site.

Results

Many people (42% of internet users in the general population) read online feedback from other patients. Fewer people (8%) write online feedback, but when they do one of their main reasons is to give praise. Most online feedback is positive in its tone and people describe caring about the NHS and wanting to help it (‘caring for care’). They also want their feedback to elicit a response as part of a conversation. Many professionals, especially doctors, are cautious about online feedback, believing it to be mainly critical and unrepresentative, and rarely encourage it. From a NHS trust perspective, online patient feedback is creating new forms of response-ability (organisations needing the infrastructure to address multiple channels and increasing amounts of online feedback) and responsivity (ensuring responses are swift and publicly visible).

Limitations

This work provides only a cross-sectional snapshot of a fast-emerging phenomenon. Questionnaire surveys can be limited by response bias. The quota sample of doctors and volunteer sample of nurses may not be representative. The ethnographic work was limited in its interrogation of differences between sites.

Conclusions

Providing and using online feedback are becoming more common for patients who are often motivated to give praise and to help the NHS improve, but health organisations and professionals are cautious and not fully prepared to use online feedback for service improvement. We identified several disconnections between patient motivations and staff and organisational perspectives, which will need to be resolved if NHS services are to engage with this source of constructive criticism and commentary from patients.

Future work

Intervention studies could measure online feedback as an intervention for service improvement and longitudinal studies could examine use over time, including unanticipated consequences. Content analyses could look for new knowledge on specific tests or treatments. Methodological work is needed to identify the best approaches to analysing feedback.

Study registration

The ethnographic case study work was registered as Current Controlled Trials ISRCTN33095169.

Funding

This project was funded by the National institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 7, No. 38. See the NIHR Journals Library website for further project information.

Differences in Online Consumer Ratings of Health Care Providers Across Medical, Surgical, and Allied Health Specialties: Observational Study of 212,933 Providers

Background

Health care consumers are increasingly using online ratings to select providers, but differences in the distribution of scores across specialties and skew of the data have the potential to mislead consumers about the interpretation of ratings.

Objective

The objective of our study was to determine whether distributions of consumer ratings differ across specialties and to provide specialty-specific data to assist consumers and clinicians in interpreting ratings.

Methods

We sampled 212,933 health care providers rated on the Healthgrades consumer ratings website, representing 29 medical specialties (n=128,678), 15 surgical specialties (n=72,531), and 6 allied health (nonmedical, nonnursing) professions (n=11,724) in the United States. We created boxplots depicting distributions and tested the normality of overall patient satisfaction scores. We then determined the specialty-specific percentile rank for scores across groupings of specialties and individual specialties.

Results

Allied health providers had higher median overall satisfaction scores (4.5, interquartile range [IQR] 4.0-5.0) than physicians in medical specialties (4.0, IQR 3.3-4.5) and surgical specialties (4.2, IQR 3.6-4.6, P<.001). Overall satisfaction scores were highly left skewed (normal between –0.5 and 0.5) for all specialties, but skewness was greatest among allied health providers (–1.23, 95% CI –1.280 to –1.181), followed by surgical (–0.77, 95% CI –0.787 to –0.755) and medical specialties (–0.64, 95% CI –0.648 to –0.628). As a result of the skewness, the percentages of overall satisfaction scores less than 4 were only 23% for allied health, 37% for surgical specialties, and 50% for medical specialties. Percentile ranks for overall satisfaction scores varied across specialties; percentile ranks for scores of 2 (0.7%, 2.9%, 0.8%), 3 (5.8%, 16.6%, 8.1%), 4 (23.0%, 50.3%, 37.3%), and 5 (63.9%, 89.5%, 86.8%) differed for allied health, medical specialties, and surgical specialties, respectively.

Conclusions

Online consumer ratings of health care providers are highly left skewed, fall within narrow ranges, and differ by specialty, which precludes meaningful interpretation by health care consumers. Specialty-specific percentile ranks may help consumers to more meaningfully assess online physician ratings.

Scope, Breadth, and Differences in Online Physician Ratings Related to Geography, Specialty, and Year: Observational Retrospective Study

Background

Physician ratings websites have emerged as a novel forum for consumers to comment on their health care experiences. Little is known about such ratings in Canada.

Objective

We investigated the scope and trends for specialty, geographic region, and time for online physician ratings in Canada using a national data source from the country’s leading physician-rating website.

Methods

This observational retrospective study used online ratings data from Canadian physicians (January 2005-September 2013; N=640,603). For specialty, province, and year of rating, we assessed whether physicians were likely to be rated favorably by using the proportion of ratings greater than the overall median rating.

Results

In total, 57,412 unique physicians had 640,603 individual ratings. Overall, ratings were positive (mean 3.9, SD 1.3). On average, each physician had 11.2 (SD 10.1) ratings. By comparing specialties with Canadian Institute of Health Information physician population numbers over our study period, we inferred that certain specialties (obstetrics and gynecology, family practice, surgery, and dermatology) were more commonly rated, whereas others (pathology, radiology, genetics, and anesthesia) were less represented. Ratings varied by specialty; cardiac surgery, nephrology, genetics, and radiology were more likely to be rated in the top 50th percentile, whereas addiction medicine, dermatology, neurology, and psychiatry were more often rated in the lower 50th percentile of ratings. Regarding geographic practice location, ratings were more likely to be favorable for physicians practicing in eastern provinces compared with western and central Canada. Regarding year, the absolute number of ratings peaked in 2007 before stabilizing and decreasing by 2013. Moreover, ratings were most likely to be positive in 2007 and again in 2013.

Conclusions

Physician-rating websites are a relatively novel source of provider-level patient satisfaction and are a valuable source of the patient experience. It is important to understand the breadth and scope of such ratings, particularly regarding specialty, geographic practice location, and changes over time.

Association Between Physician Online Rating and Quality of Care

Background

Patients are increasingly using physician review websites to find “a good doctor.” However, to our knowledge, no prior study has examined the relationship between online rating and an accepted measure of quality.

Objective

The purpose of this study was to assess the association between online physician rating and an accepted measure of quality: 30-day risk-adjusted mortality rate following coronary artery bypass graft (CABG) surgery.

Methods

In the US states of California, Massachusetts, New Jersey, New York, and Pennsylvania—which together account for over one-quarter of the US population—risk-adjusted mortality rates are publicly reported for all cardiac surgeons. From these reports, we recorded the 30-day mortality rate following isolated CABG surgery for each surgeon practicing in these 5 states. For each surgeon listed in the state reports, we then conducted Internet-based searches to determine his or her online rating(s). We then assessed the relationship between physician online rating and risk-adjusted mortality rate.

Results

Of the 614 surgeons listed in the state reports, we found 96.1% (590/614) to be rated online. The average online rating was 4.4 out of 5, and 78.7% (483/614) of the online ratings were 4 or higher. The median number of reviews used to formulate each rating was 4 (range 1-89), and 32.70% (503/1538) of the ratings were based on 2 or fewer reviews. Overall, there was no correlation between surgeon online rating and risk-adjusted mortality rate (P=.13). Risk-adjusted mortality rates were similar for surgeons across categories of average online rating (P>.05), and surgeon average online rating was similar across quartiles of surgeon risk-adjusted mortality rate (P>.05).

Conclusions

In this study of cardiac surgeons practicing in the 5 US states that publicly report outcomes, we found no correlation between online rating and risk-adjusted mortality rates. Patients using online rating websites to guide their choice of physician should recognize that these ratings may not reflect actual quality of care as defined by accepted metrics.

General Practitioners' Concerns About Online Patient Feedback: Findings From a Descriptive Exploratory Qualitative Study in England

Background

The growth in the volume of online patient feedback, including online patient ratings and comments, suggests that patients are embracing the opportunity to review online their experience of receiving health care. Very little is known about health care professionals’ attitudes toward online patient feedback and whether health care professionals are comfortable with the public nature of the feedback.

Objective

The aim of the overall study was to explore and describe general practitioners’ attitudes toward online patient feedback. This paper reports on the findings of one of the aims of the study, which was to explore and understand the concerns that general practitioners (GPs) in England have about online patient feedback. This could then be used to improve online patient feedback platforms and help to increase usage of online patient feedback by GPs and, by extension, their patients.

Methods

A descriptive qualitative approach using face-to-face semistructured interviews was used in this study. A topic guide was developed following a literature review and discussions with key stakeholders. GPs (N=20) were recruited from Cambridgeshire, London, and Northwest England through probability and snowball sampling. Interviews were transcribed verbatim and analyzed in NVivo using the framework method, a form of thematic analysis.

Results

Most participants in this study had concerns about online patient feedback. They questioned the validity of online patient feedback because of data and user biases and lack of representativeness, the usability of online patient feedback due to the feedback being anonymous, the transparency of online patient feedback because of the risk of false allegations and breaching confidentiality, and the resulting impact of all those factors on them, their professional practice, and their relationship with their patients.

Conclusions

The majority of GPs interviewed had reservations and concerns about online patient feedback and questioned its validity and usefulness among other things. Based on the findings from the study, recommendations for online patient feedback website providers in England are given. These include suggestions to make some specific changes to the platform and the need to promote online patient feedback more among both GPs and health care users, which may help to reduce some of the concerns raised by GPs about online patient feedback in this study.

Online reviews of orthopedic surgeons: an emerging trend

Various websites are dedicated to rating physicians. The goals of this study were to: (1) evaluate the prevalence of orthopedic surgeon ratings on physician rating websites in the United States and (2) evaluate factors that may affect ratings, such as sex, practice sector (academic or private), years of practice, and geographic location. A total of 557 orthopedic surgeons selected from the 30 most populated US cities were enrolled. The study period was June 1 to July 31, 2013. Practice type (academic vs private), sex, geographic location, and years since completion of training were evaluated. For each orthopedic surgeon, numeric ratings from 7 physician rating websites were collected. The ratings were standardized on a scale of 0 to 100. Written reviews were also collected and categorized as positive or negative. Of the 557 orthopedic surgeons, 525 (94.3%) were rated at least once on 1 of the physician rating websites. The average rating was 71.4. The study included 39 female physicians (7.4%) and 486 male physicians (92.6%). There were 204 (38.9%) physicians in academic practice and 321 (61.1%) in private practice. The greatest number of physicians, 281 (50.4%), practiced in the South and Southeast, whereas 276 (49.6%) practiced in the West, Midwest, and Northeast. Those in academic practice had significantly higher ratings (74.4 vs 71.1; P<.007). No significant difference based on sex (72.5 male physicians vs 70.2 female physicians; P=.17) or geographic location (P=.11) were noted. Most comments (64.6%) were positive or extremely positive. Physicians who were in practice for 6 to 10 years had significantly higher ratings (76.9, P<.01) than those in practice for 0 to 5 years (70.5) or for 21 or more years (70.7).