Learning by Listening-Improving Health Care in the Era of Yelp

Assessing an innovative method to promote learning from patient narratives: Findings from a field experiment in ambulatory care

OBJECTIVE

To assess whether an online interactive report designed to facilitate interpretation of patients' narrative feedback produces change in ambulatory staff learning, behavior at the individual staff and practice level, and patient experience survey scores.

DATA SOURCES AND SETTING

We studied 22 ambulatory practice sites within an academic medical center using three primary data sources: 333 staff surveys; 20 in-depth interviews with practice leaders and staff; and 9551 modified CG-CAHPS patient experience surveys augmented by open-ended narrative elicitation questions.

STUDY DESIGN

We conducted a cluster quasi-experimental study, comparing 12 intervention and 10 control sites. At control sites, narratives were delivered free-form to site administrators via email; at intervention sites, narratives were delivered online with interactive tools for interpretation, accompanied by user training. We assessed control-versus-intervention site differences in learning, behavior, and patient experience scores.

DATA COLLECTION

Staff surveys and interviews were completed at intervention and control sites, 9 months after intervention launch. Patient surveys were collected beginning 4 months pre-launch through 9 months post-launch. We used control-versus-intervention and difference-in-difference analyses for survey data and thematic analysis for interview data.

PRINCIPAL FINDINGS

Interviews suggested that the interface facilitated narrative interpretation and use for improvement. Staff survey analyses indicated enhanced learning from narratives at intervention sites (29% over control sites' mean of 3.19 out of 5 across eight domains, p < 0.001) and greater behavior change at staff and practice levels (31% and 21% over control sites' means of 3.35 and 3.39, p < 0.001, respectively). Patient experience scores for interactions with office staff and wait time information increased significantly at intervention sites, compared to control sites (3.7% and 8.2%, respectively); however, provider listening scores declined 3.3%.

CONCLUSIONS

Patient narratives presented through structured feedback reporting methods can catalyze positive changes in staff learning, promote behavior change, and increase patient experience scores in domains of non-clinical interaction.

Association Between Online Reviews of Substance Use Disorder Treatment Facilities and Drug-Induced Mortality Rates: Cross-Sectional Analysis

BACKGROUND

Drug-induced mortality across the United States has continued to rise. To date, there are limited measures to evaluate patient preferences and priorities regarding substance use disorder (SUD) treatment, and many patients do not have access to evidence-based treatment options. Patients and their families seeking SUD treatment may begin their search for an SUD treatment facility online, where they can find information about individual facilities, as well as a summary of patient-generated web-based reviews via popular platforms such as Google or Yelp. Web-based reviews of health care facilities may reflect information about factors associated with positive or negative patient satisfaction. The association between patient satisfaction with SUD treatment and drug-induced mortality is not well understood.

OBJECTIVE

The objective of this study was to examine the association between online review content of SUD treatment facilities and drug-induced state mortality.

METHODS

A cross-sectional analysis of online reviews and ratings of Substance Abuse and Mental Health Services Administration (SAMHSA)-designated SUD treatment facilities listed between September 2005 and October 2021 was conducted. The primary outcomes were (1) mean online rating of SUD treatment facilities from 1 star (worst) to 5 stars (best) and (2) average drug-induced mortality rates from the Centers for Disease Control and Prevention (CDC) WONDER Database (2006-2019). Clusters of words with differential frequencies within reviews were identified. A 3-level linear model was used to estimate the association between online review ratings and drug-induced mortality.

RESULTS

A total of 589 SAMHSA-designated facilities (n=9597 reviews) were included in this study. Drug-induced mortality was compared with the average. Approximately half (24/47, 51%) of states had below average ("low") mortality rates (mean 13.40, SD 2.45 deaths per 100,000 people), and half (23/47, 49%) had above average ("high") drug-induced mortality rates (mean 21.92, SD 3.69 deaths per 100,000 people). The top 5 themes associated with low drug-induced mortality included detoxification and addiction rehabilitation services (r=0.26), gratitude for recovery (r=-0.25), thankful for treatment (r=-0.32), caring staff and amazing experience (r=-0.23), and individualized recovery programs (r=-0.20). The top 5 themes associated with high mortality were care from doctors or providers (r=0.24), rude and insensitive care (r=0.23), medication and prescriptions (r=0.22), front desk and reception experience (r=0.22), and dissatisfaction with communication (r=0.21). In the multilevel linear model, a state with a 10 deaths per 100,000 people increase in mortality was associated with a 0.30 lower average Yelp rating (P=.005).

CONCLUSIONS

Lower online ratings of SUD treatment facilities were associated with higher drug-induced mortality at the state level. Elements of patient experience may be associated with state-level mortality. Identified themes from online, organically derived patient content can inform efforts to improve high-quality and patient-centered SUD care.

Patient-Centered Insights and Biases Regarding Cardiologists Via Online Review Platform Analysis

Background Online cardiologist reviews, such as those on the Yelp website, are a frequently used method for patients to find a cardiologist. It remains unknown how bias may influence such reviews. Our objectives for this study were to (1) determine which cardiologist- or practice-related factors influence the overall rating of cardiologists and patient satisfaction and (2) discover any associations between sex and race with the overall rating of cardiologists or with cardiologist- or practice-related factors. Methods and Results Cardiologist Yelp reviews from practices in the United States from 2007 to 2020 were analyzed. A total of 563 reviews were coded for positive and negative themes. Binary logistic regression was used to determine whether certain factors increased the likelihood of high ratings. Chi-squared tests were used to determine associations between sex and race with certain factors and overall cardiologist ratings. Cardiologists were more likely to receive higher ratings when reviewers noted the characteristics of competency/knowledge base and thoroughness, positive interactions with staff, and when the cardiologist's name was mentioned in the review. Negative interactions with staff were associated with lower ratings. Female cardiologists received lower ratings and more negative mentions of cardiologist-patient communication than expected. White and Black cardiologists received lower ratings than expected compared with other racial groups. Conclusions Patient-perceived cardiologist competency, thoroughness, and positive staff interactions were associated with positive reviews in online assessments. Sex and racial differences were also found. Further research must be done to confirm these findings and to understand the association of online reviews with clinical care and patient outcomes.

Recency of Online Physician Ratings

This cross-sectional study examines the association between older and more recent online physician ratings and the implications for optimizing the trade-off between reliability and incentives.

Artificial Intelligence and Medical Humanities

The use of artificial intelligence in healthcare has led to debates about the role of human clinicians in the increasingly technological contexts of medicine. Some researchers have argued that AI will augment the capacities of physicians and increase their availability to provide empathy and other uniquely human forms of care to their patients. The human vulnerabilities experienced in the healthcare context raise the stakes of new technologies such as AI, and the human dimensions of AI in healthcare have particular significance for research in the humanities. This article explains four key areas of concern relating to AI and the role that medical/health humanities research can play in addressing them: definition and regulation of "medical" versus "health" data and apps; social determinants of health; narrative medicine; and technological mediation of care. Issues include data privacy and trust, flawed datasets and algorithmic bias, racial discrimination, and the rhetoric of humanism and disability. Through a discussion of potential humanities contributions to these emerging intersections with AI, this article will suggest future scholarly directions for the field.

Patient Experience and Satisfaction in Online Reviews of Obstetric Care: Observational Study

Background

The quality of care in labor and delivery is traditionally measured through the Hospital Consumer Assessment of Healthcare Providers and Systems but less is known about the experiences of care reported by patients and caregivers on online sites that are more easily accessed by the public.

Objective

The aim of this study was to generate insight into the labor and delivery experience using hospital reviews on Yelp.

Methods

We identified all Yelp reviews of US hospitals posted online from May 2005 to March 2017. We used a machine learning tool, latent Dirichlet allocation, to identify 100 topics or themes within these reviews and used Pearson r to identify statistically significant correlations between topics and high (5-star) and low (1-star) ratings.

Results

A total of 1569 hospitals listed in the American Hospital Association directory had at least one Yelp posting, contributing a total of 41,095 Yelp reviews. Among those hospitals, 919 (59%) had at least one Yelp rating for labor and delivery services (median of 9 reviews), contributing a total of 6523 labor and delivery reviews. Reviews concentrated among 5-star (n=2643, 41%) and 1-star reviews (n=1934, 30%). Themes strongly associated with favorable ratings included the following: top-notch care (r=0.45, P<.001), describing staff as comforting (r=0.52, P<.001), the delivery experience (r=0.46, P<.001), modern and clean facilities (r=0.44, P<.001), and hospital food (r=0.38, P<.001). Themes strongly correlated with 1-star labor and delivery reviews included complaints to management (r=0.30, P<.001), a lack of agency among patients (r=0.47, P<.001), and issues with discharging from the hospital (r=0.32, P<.001).

Conclusions

Online review content about labor and delivery can provide meaningful information about patient satisfaction and experiences. Narratives from these reviews that are not otherwise captured in traditional surveys can direct efforts to improve the experience of obstetrical care.

Association Between Crowdsourced Health Care Facility Ratings and Mortality in US Counties

IMPORTANCE

Mortality across US counties varies considerably, from 252 to 1847 deaths per 100 000 people in 2018. Although patient satisfaction with health care is associated with patient- and facility-level health outcomes, the association between health care satisfaction and community-level health outcomes is not known.

OBJECTIVE

To examine the association between online ratings of health care facilities and mortality across US counties and to identify language specific to 1-star (lowest rating) and 5-star (highest rating) reviews in counties with high vs low mortality.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective population-based cross-sectional study examined reviews and ratings of 95 120 essential health care facilities across 1301 US counties. Counties that had at least 1 essential health care facility with reviews available on Yelp, an online review platform, were included. Essential health care was defined according to the 10 essential health benefits covered by Affordable Care Act insurance plans.

MAIN OUTCOMES AND MEASURES

The mean rating of essential health care facilities was calculated by county from January 1, 2015, to December 31, 2019. Ratings were on a scale of 1 to 5 stars, with 1 being the worst rating and 5 the best. County-level composite measures of health behaviors, clinical care, social and economic factors, and physical environment were obtained from the University of Wisconsin School of Medicine and Public Health County Health Rankings database. The 2018 age-adjusted mortality by county was obtained from the Centers for Disease Control and Prevention Wide-ranging Online Data for Epidemiological Research database. Multiple linear regression analysis was used to estimate the association between mean facility rating and mortality, adjusting for county health ranking variables. Words with frequencies of use that were significantly different across 1-star and 5-star reviews in counties with high vs low mortality were identified.

RESULTS

The 95 120 facilities meeting inclusion criteria were distributed across 1301 of 3142 US counties (41.4%). At the county level, a 1-point increase in mean rating was associated with a mean (SE) age-adjusted decrease of 18.05 (3.68) deaths per 100 000 people (P < .001). Words specific to 1-star reviews in high-mortality counties included told, rude, and wait, and words specific to 5-star reviews in low-mortality counties included Dr, pain, and professional.

CONCLUSIONS AND RELEVANCE

This study found that, at the county level, higher online ratings of essential health care facilities were associated with lower mortality. Equivalent online ratings did not necessarily reflect equivalent experiences of care across counties with different mortality levels, as evidenced by variations in the frequency of use of key words in reviews. These findings suggest that online ratings and reviews may provide insight into unequal experiences of essential health care.

Online Reviews of Mental Health Treatment Facilities: Narrative Themes Associated With Positive and Negative Ratings

OBJECTIVE

Previous studies indicate that patients' satisfaction with mental health care is correlated with both treatment outcomes and quality of life. The aims of this study were to describe online reviews of mental health treatment facilities, including key themes in review content, and to evaluate the correlation between narrative review themes, facility characteristics, and review ratings.

METHODS

United States National Mental Health Services Survey (N-MHSS) facilities were linked to corresponding Yelp pages, created between March 2007 and September 2019. Correlations between review ratings and both machine learning-generated latent Dirichlet allocation topics and N-MHSS-reported facility characteristics were measured by using Spearman's rank-order correlation coefficient. Significance was defined by a Bonferroni-adjusted p<0.001.

RESULTS

Of 10,191 unique mental health treatment facilities, 1,383 (13.6%) had relevant Yelp pages with 8,133 corresponding reviews. The number of newly reviewed facilities and the number of new reviews increased throughout the study period. Narrative topics positively correlated with review ratings included caring staff (Spearman's ρ=0.39) and nonpharmacologic treatment (ρ=0.16). Topics negatively correlated with review ratings included rude staff (ρ=-0.14) and safety and abuse (ρ=-0.14). Of 126 N-MHSS survey items, 11 were positively correlated with review rating, including "outpatient mental health facility" (ρ=0.13), and 33 were negatively correlated with review rating, including accepting Medicare (ρ=-0.21).

CONCLUSIONS

Narrative topics provide information beyond what is currently collected through the N-MHSS. Topics associated with positive and negative reviews, such as staff attitude toward patients, can guide improvement in patients' satisfaction and engagement with mental health care.

Online Reviews of Specialized Drug Treatment Facilities-Identifying Potential Drivers of High and Low Patient Satisfaction

BACKGROUND

Despite the importance of high-quality and patient-centered substance use disorder treatment, there are no standardized ratings of specialized drug treatment facilities and their services. Online platforms offer insights into potential drivers of high and low patient experience.

OBJECTIVE

We sought to analyze publicly available online review content of specialized drug treatment facilities and identify themes within high and low ratings.

DESIGN

This was a retrospective analysis of online ratings and reviews of specialized drug treatment facilities in Pennsylvania listed within the 2016 National Directory of Drug and Alcohol Abuse Treatment Facilities. Latent Dirichlet Allocation, a machine learning approach to narrative text, was used to identify themes within reviews. Differential Language Analysis was then used to measure correlations between themes and star ratings.

SETTING

Online reviews of Pennsylvania's specialized drug treatment facilities posted to Google and Yelp (July 2010-August 2018).

RESULTS

A total of 7823 online ratings were posted over 8 years. The distribution was bimodal (43% 5-star and 34% 1-star). The average weighted rating of a facility was 3.3 stars. Online themes correlated with 5-star ratings were the following: focus on recovery (r = 0.53), helpfulness of staff (r = 0.43), compassionate care (r = 0.37), experienced a life-changing moment (r = 0.32), and staff professionalism (r = 0.29). Themes correlated with a 1-star rating were waiting time (r = 0.41), poor accommodations (0.26), poor phone communication (r = 0.24), medications given (0.24), and appointment availability (r = 0.23). Themes derived from review content were similar to 9 of the 14 facility-level services highlighted by the Substance Abuse and Mental Health Services Administration's National Survey of Substance Abuse Treatment Services.

CONCLUSIONS

Individuals are sharing their ratings and reviews of specialized drug treatment facilities on online platforms. Organically derived reviews of the patient experience, captured by online platforms, reveal potential drivers of high and low ratings. These represent additional areas of focus which can inform patient-centered quality metrics for specialized drug treatment facilities.

Association of Online Consumer Reviews of Skilled Nursing Facilities With Patient Rehospitalization Rates

Importance

There are areas of skilled nursing facility (SNF) experience of importance to the public that are not currently included in public reporting initiatives on SNF quality. Whether patients, hospitals, and payers can leverage the information available from unsolicited online reviews to reduce avoidable rehospitalizations from SNFs is unknown.

Objectives

To assess the association between rehospitalization rates and online ratings of SNFs; to compare the association of rehospitalization with ratings from a review website vs Medicare Nursing Home Compare (NHC) ratings; and to identify specific topics consistently reported in reviews of SNFs with the highest vs lowest rehospitalization rates using natural language processing.

Design, Setting, and Participants

A retrospective cross-sectional study of 1536 SNFs with online reviews on Yelp (a website that allows consumers to rate and review businesses and services, scored on a 1- to 5-star rating scale, with 1 star indicating the lowest rating and 5 stars indicating the highest rating) posted between January 1, 2014, and December 31, 2018. The combined data set included 1536 SNFs with 8548 online reviews, NHC ratings, and readmission rates.

Main Outcomes and Measures

A mean rating from the review website was calculated through the end of each year. Risk-standardized rehospitalization rates were obtained from NHC. Linear regression was used to measure the association between the rehospitalization rate of a SNF and the online ratings. Natural language processing was used to identify topics associated with reviews of SNFs in the top and bottom quintiles of rehospitalization rates.

Results

The 1536 SNFs in the sample had a median of 6 reviews (interquartile range, 3-13 reviews), with a mean (SD) review website rating of 2.7 (1.1). The SNFs with the highest rating on both the review website and NHC had 2.0% lower rehospitalization rates compared with the SNFs with the lowest rating on both websites (21.3%; 95% CI, 20.7%-21.8%; vs 23.3%; 95% CI, 22.7%-24.0%; P = .04). Compared with the NHC ratings alone, review website ratings were associated with an additional 0.4% of the variation in rehospitalization rates across SNFs (adjusted R2 = 0.009 vs adjusted R2 = 0.013; P = .003). Thematic analysis of qualitative comments on the review website for SNFs with high vs low rehospitalization rates identified several areas of importance to the reviewers, such as the quality of physical infrastructure and equipment, staff attitudes and communication with caregivers.

Conclusions and Relevance

Skilled nursing facilities with the best rating on both a review website and NHC had slightly lower rehospitalization rates than SNFs with the best rating on NHC alone. However, there was marked variation in the volume of reviews, and many SNF characteristics were underrepresented. Further refinement of the review process is warranted.

Content Analysis of Negative Online Reviews of Hospice Agencies in the United States

IMPORTANCE

As online reviews of health care become increasingly integral to patient decision-making, understanding their content can help health care practices identify and address patient concerns.

OBJECTIVE

To identify the most frequently cited complaints in negative (ie, 1-star) online reviews of hospice agencies across the United States.

DESIGN, SETTING, AND PARTICIPANTS

This qualitative study conducted a thematic analysis of online reviews of US hospice agencies posted between August 2011 and July 2019. The sample was selected from a Hospice Analytics database. For each state, 1 for-profit (n = 50) and 1 nonprofit (n = 50) hospice agency were randomly selected from the category of extra-large hospice agencies (ie, serving >200 patients/d) in the database. Data analysis was conducted from January 2019 to April 2019.

MAIN OUTCOMES AND MEASURES

Reviews were analyzed to identify the most prevalent concerns expressed by reviewers.

RESULTS

Of 100 hospice agencies in the study sample, 67 (67.0%) had 1-star reviews; 33 (49.3%) were for-profit facilities and 34 (50.7%) were nonprofit facilities. Of 137 unique reviews, 68 (49.6%) were for for-profit facilities and 69 (50.4%) were for nonprofit facilities. A total of 5 themes emerged during the coding and analytic process, as follows: discordant expectations, suboptimal communication, quality of care, misperceptions about the role of hospice, and the meaning of a good death. The first 3 themes were categorized as actionable criticisms, which are variables hospice organizations could change. The remaining 2 themes were categorized as unactionable criticisms, which are factors that would require larger systematic changes to address. For both for-profit and nonprofit hospice agencies, quality of care was the most frequently commented-on theme (117 of 212 comments [55.2%]). For-profit hospice agencies received more communication-related comments overall (34 of 130 [26.2%] vs 9 of 82 [11.0%]), while nonprofit hospice agencies received more comments about the role of hospice (23 of 33 [69.7%] vs 19 of 31 [61.3%]) and the quality of death (16 [48.5%] vs 12 [38.7%]).

CONCLUSIONS AND RELEVANCE

Regarding actionable criticisms, hospice agencies could examine their current practices, given that reviewers described these issues as negatively affecting the already difficult experience of losing a loved one. The findings indicated that patients and their families, friends, and caregivers require in-depth instruction and guidance on what they can expect from hospice staff, hospice services, and the dying process. Several criticisms identified in this study may be mitigated through operationalized, explicit conversations about these topics during hospice enrollment.

What Words Convey: The Potential for Patient Narratives to Inform Quality Improvement

Policy Points Narratives about patients' experiences with outpatient care are essential for quality improvement because they convey ample actionable information that both elaborates on existing domains within patient experience surveys and describes multiple additional domains that are important to patients. The content of narrative feedback from patients can potentially be translated to improved quality in multiple ways: clinicians can learn from their own patients, groups of clinicians can learn from the experience of their peers' patients, and health system administrators can identify and respond to patterns in patients' accounts that reflect systemic challenges to quality. Consistent investment by payers and providers is required to ensure that patient narratives are rigorously collected, analyzed fully, and effectively used for quality improvement.

CONTEXT

For the past 25 years, health care providers and health system administrators have sought to improve care by surveying patients about their experiences. More recently, policymakers have acted to promote this learning by deploying financial incentives tied to survey scores. This article explores the potential of systematically elicited narratives about experiences with outpatient care to enrich quality improvement.

METHODS

Narratives were collected from 348 patients recruited from a nationally representative Internet panel. Drawing from the literature on health services innovation, we developed a two-part coding schema that categorized narrative content in terms of (a) the aspects of care being described, and (b) the actionability of this information for clinicians, quality improvement staff, and health system administrators. Narratives were coded using this schema, with high levels of reliability among the coders.

FINDINGS

The scope of outpatient narratives divides evenly among aspects of care currently measured by patient experience surveys (35% of content), aspects related to measured domains but not captured by existing survey questions (31%), and aspects of care that are omitted from surveys entirely (34%). Overall, the narrative data focused heavily on relational aspects of care (43%), elaborating on this aspect of experience well beyond what is captured with communication-related questions on existing surveys. Three-quarters of elicited narratives had some actionable content, and almost a third contained three or more separate actionable elements.

CONCLUSIONS

In a health policy environment that incentivizes attention to patient experience, rigorously elicited narratives hold substantial promise for improving quality in general and patients' experiences with care in particular. They do so in two ways: by making concrete what went wrong or right in domains covered by existing surveys, and by expanding our view of what aspects of care matter to patients as articulated in their own words and thus how care can be made more patient-centered. Most narratives convey experiences that are potentially actionable by those committed to improving health care quality in outpatient settings.

Harvesting the wisdom of the crowd: using online ratings to explore care experiences in regions

Background

Regional population health management (PHM) initiatives need an understanding of regional patient experiences to improve their services. Websites that gather patient ratings have become common and could be a helpful tool in this effort. Therefore, this study explores whether unsolicited online ratings can provide insight into (differences in) patient’s experiences at a (regional) population level.

Methods

Unsolicited online ratings from the Dutch website Zorgkaart Nederland (year = 2008–2017) were used. Patients rated their care providers on six dimensions from 1 to 10 and these ratings were geographically aggregated based on nine PHM regions. Distributions were explored between regions. Multilevel analyses per provider category, which produced Intraclass Correlation Coefficients (ICC), were performed to determine clustering of ratings of providers located within regions. If ratings were clustered, then this would indicate that differences found between regions could be attributed to regional characteristics (e.g. demographics or regional policy).

Results

In the nine regions, 70,889 ratings covering 4100 care providers were available. Overall, average regional scores (range = 8.3–8.6) showed significant albeit small differences. Multilevel analyses indicated little clustering between unsolicited provider ratings within regions, as the regional level ICCs were low (ICC pioneer site < 0.01). At the provider level, all ICCs were above 0.11, which showed that ratings were clustered.

Conclusions

Unsolicited online provider-based ratings are able to discern (small) differences between regions, similar to solicited data. However, these differences could not be attributed to the regional level, making unsolicited ratings not useful for overall regional policy evaluations. At the provider level, ratings can be used by regions to identify under-performing providers within their regions.

A Qualitative, Cross-Sectional Study of Positive and Negative Comments of Residency Programs Across 9 Medical and Surgical Specialties

IMPORTANCE

Residency applicants often use social media to discuss the positive and negative features of prospective training programs. An examination of the content discussed by applicants could provide guidance for how a medical education faculty can better engage with prospective trainees and adapt to meet the educational expectations of a new generation of digital-native physicians.

OBJECTIVE

The objective was to identify unstructured social media data submitted by residency applicants and categorize positive and negative statements to determine key themes.

DESIGN

The study design was qualitative analysis of a retrospective cohort.

SETTING

Publicly available datasets were used.

PARTICIPANTS

The participants were anonymized medical trainees applying to residency training positions in 9 specialties-dermatology, general surgery, internal medicine, obstetrics/gynecology, plastic surgery, otolaryngology, physical medicine and rehabilitation, pediatrics, and radiology-from 2007 to 2017.

MAIN OUTCOMES AND MEASURES

After we developed a standardized coding scheme that broke comments down into major features, themes, and subthemes, all unstructured comments were coded by two independent researchers. Positive and negative comments were coded separately. Frequency counts and percentages were recorded for each identified feature, theme, and subtheme. The percent positive and negative comments by specialty were also calculated.

RESULTS

Of the 6314 comments identified, 4541 were positive and 1773 were negative. Institution was the most commonly cited major feature in both the positive (n = 767 [17%]) and negative (n = 827 [47%]) comments. Geography was the most cited theme, and City, Cost of Living, and Commute were commonly cited subthemes. Training was the next most cited major feature in both positive (n = 1005 [22%]) and negative (n = 291 [16%]) comments, with Clinical Training being more commonly cited compared to Research Opportunities. Overall, 72% of comments from all were positive; however, the percent of comments that were positive comments varied significantly across the 9 specialties. Pediatrics (65%), dermatology (66%), and internal medicine (68%) applicants were more likely to express negative comments compared with the global average, but physical medicine and rehabilitation (85%), radiology (82%), otolaryngology (81%), and plastic surgery (80%) applicants were more likely to express positive comments.

CONCLUSIONS AND RELEVANCE

This qualitative analysis of positive and negative themes as posted by applicants in recent matching years is the first and provides new detailed insights into the motivations and desires of trainees.

Content, Quality, and Assessment Tools of Physician-Rating Websites in 12 Countries: Quantitative Analysis

BACKGROUND

Websites on which users can rate their physician are becoming increasingly popular, but little is known about the website quality, the information content, and the tools they offer users to assess physicians. This study assesses these aspects on physician-rating websites in German- and English-speaking countries.

OBJECTIVE

The objective of this study was to collect information on websites with a physician rating or review tool in 12 countries in terms of metadata, website quality (transparency, privacy and freedom of speech of physicians and patients, check mechanisms for appropriateness and accuracy of reviews, and ease of page navigation), professional information about the physician, rating scales and tools, as well as traffic rank.

METHODS

A systematic Web search based on a set of predefined keywords was conducted on Google, Bing, and Yahoo in August 2016. A final sample of 143 physician-rating websites was analyzed and coded for metadata, quality, information content, and the physician-rating tools.

RESULTS

The majority of websites were registered in the United States (40/143) or Germany (25/143). The vast majority were commercially owned (120/143, 83.9%), and 69.9% (100/143) displayed some form of physician advertisement. Overall, information content (mean 9.95/25) as well as quality were low (mean 18.67/47). Websites registered in the United Kingdom obtained the highest quality scores (mean 26.50/47), followed by Australian websites (mean 21.50/47). In terms of rating tools, physician-rating websites were most frequently asking users to score overall performance, punctuality, or wait time in practice.

CONCLUSIONS

This study evidences that websites that provide physician rating should improve and communicate their quality standards, especially in terms of physician and user protection, as well as transparency. In addition, given that quality standards on physician-rating websites are low overall, the development of transparent guidelines is required. Furthermore, attention should be paid to the financial goals that the majority of physician-rating websites, especially the ones that are commercially owned, pursue.

Patient narratives in Yelp reviews offer insight into opioid experiences and the challenges of pain management

Aim: To characterize Yelp reviews about pain management and opioids. Methods: We manually coded and applied natural language processing to 836 Yelp reviews of US hospitals mentioning an opioid medication. Results: Yelp reviews by patients and caregivers describing experiences with pain management and opioids had lower ratings compared with other reviews. Negative descriptions of pain management and opioid-related experiences were more commonly described than positive experiences, and the number of themes they reflected was more diverse. Conclusion: Yelp reviews offer insights into pain management and opioid use that are not assessed by traditional surveys. As a free, highly utilized source of unstructured narratives, Yelp may allow ongoing assessment of policies related to pain management and opioid use.