Analysis of multicenter clinical trials with very low event rates

INTRODUCTION

In a five-arm randomized clinical trial (RCT) with stratified randomization across 54 sites, we encountered low primary outcome event proportions, resulting in multiple sites with zero events either overall or in one or more study arms. In this paper, we systematically evaluated different statistical methods of accounting for center in settings with low outcome event proportions.

METHODS

We conducted a simulation study and a reanalysis of a completed RCT to compare five popular methods of estimating an odds ratio for multicenter trials with stratified randomization by center: (i) no center adjustment, (ii) random intercept model, (iii) Mantel-Haenszel model, (iv) generalized estimating equation (GEE) with an exchangeable correlation structure, and (v) GEE with small sample correction (GEE-small sample correction). We varied the number of total participants (200, 500, 1000, 5000), number of centers (5, 50, 100), control group outcome percentage (2%, 5%, 10%), true odds ratio (1, > 1), intra-class correlation coefficient (ICC) (0.025, 0.075), and distribution of participants across the centers (balanced, skewed).

RESULTS

Mantel-Haenszel methods generally performed poorly in terms of power and bias and led to the exclusion of participants from the analysis because some centers had no events. Failure to account for center in the analysis generally led to lower power and type I error rates than other methods, particularly with ICC = 0.075. GEE had an inflated type I error rate except in some settings with a large number of centers. GEE-small sample correction maintained the type I error rate at the nominal level but suffered from reduced power and convergence issues in some settings when the number of centers was small. Random intercept models generally performed well in most scenarios, except with a low event rate (i.e., 2% scenario) and small total sample size (n ≤ 500), when all methods had issues.

DISCUSSION

Random intercept models generally performed best across most scenarios. GEE-small sample correction performed well when the number of centers was large. We do not recommend the use of Mantel-Haenszel, GEE, or models that do not account for center. When the expected event rate is low, we suggest that the statistical analysis plan specify an alternative method in the case of non-convergence of the primary method.

The design of a randomized controlled trial to evaluate multi-dimensional effects of a section 1115 Medicaid demonstration waiver with community engagement requirements

Section 1115 demonstration waivers provide a mechanism for states to implement changes to their Medicaid programs. While such waivers are mandated to include evaluations of their impact, randomization - the gold standard for assessing causality - has not typically been a consideration. In a critical departure, the Commonwealth of Kentucky opted to pursue a two-arm randomized controlled trial (RCT) for their controversial 2018 Medicaid Demonstration waiver, which included work requirements as a condition for the subset of beneficiaries deemed able-bodied to maintain eligibility for benefits. Beneficiaries were randomized 9:1 to the new waiver program or a control group who would retain their current benefits as part of the existing Medicaid expansion program. To address potential bias from differential attrition from the Medicaid program that would accrue from solely analyzing administrative data, our team designed a rich, prospective, longitudinal survey to collect primary and secondary outcomes from six categories of interest to policymakers: insurance coverage, health care utilization and quality, health behaviors, socioeconomic measures, personal finances, and health outcomes. At baseline, a subset of survey participants was invited to participate in the collection of biometric samples via in-person follow-up visits, and a cross-section were also invited to participate in qualitative interviews. While the demonstration waiver was terminated before the program began, our study design illustrates that it is possible for other researchers and state agencies seeking to evaluate Medicaid demonstration waivers and other demonstration policies to work together to implement high quality randomized trials - even for controversial policies.

Effect of Passive Choice and Active Choice Interventions in the Electronic Health Record to Cardiologists on Statin Prescribing: A Cluster Randomized Clinical Trial

Importance

Statin therapy is underused for many patients who could benefit.

Objective

To evaluate the effect of passive choice and active choice interventions in the electronic health record (EHR) to promote guideline-directed statin therapy.

Design, Setting, and Participants

Three-arm randomized clinical trial with a 6-month preintervention period and 6-month intervention. Randomization conducted at the cardiologist level at 16 cardiology practices in Pennsylvania and New Jersey. The study included 82 cardiologists and 11 693 patients. Data were analyzed between May 8, 2019, and January 9, 2020.

Interventions

In passive choice, cardiologists had to manually access an alert embedded in the EHR to select options to initiate or increase statin therapy. In active choice, an interruptive EHR alert prompted the cardiologist to accept or decline guideline-directed statin therapy. Cardiologists in the control group were informed of the trial but received no other interventions.

Main Outcomes and Measures

Primary outcome was statin therapy at optimal dose based on clinical guidelines. Secondary outcome was statin therapy at any dose.

Results

The sample comprised 11 693 patients with a mean (SD) age of 63.8 (9.1) years; 58% were male (n = 6749 of 11 693), 66% were White (n = 7683 of 11 693), and 24% were Black (n = 2824 of 11 693). The mean (SD) 10-year atherosclerotic cardiovascular disease (ASCVD) risk score was 15.4 (10.0); 68% had an ASVCD clinical diagnosis. Baseline statin prescribing rates at the optimal dose were 40.3% in the control arm, 39.1% in the passive choice arm, and 41.2% in the active choice arm. In adjusted analyses, the change in statin prescribing rates at optimal dose over time was not significantly different from control for passive choice (adjusted difference in percentage points, 0.2; 95% CI, -2.9 to 2.8; P = .86) or active choice (adjusted difference in percentage points, 2.4; 95% CI, -0.6 to 5.0; P = .08). In adjusted analyses of the subset of patients with clinical ASCVD, the active choice intervention resulted in a significant increase in statin prescribing at optimal dose relative to control (adjusted difference in percentage points, 3.8; 95% CI, 1.0-6.4; P = .008). No other subset analyses were significant. There were no significant changes in statin prescribing at any dose for either intervention.

Conclusions and Relevance

The passive choice and active choice interventions did not change statin prescribing. In the subgroup of patients with clinical ASCVD, the active choice intervention led to a small increase in statin prescribing at the optimal dose, which could inform the design or targeting of future interventions.

Trial Registration

ClinicalTrials.gov Identifier: NCT03271931.

Effect of Patient Financial Incentives on Statin Adherence and Lipid Control: A Randomized Clinical Trial

IMPORTANCE

Financial incentives can improve medication adherence and cardiovascular disease risk, but the optimal design to promote sustained adherence after incentives are discontinued is unknown.

OBJECTIVE

To determine whether 6-month interventions involving different financial incentives to encourage statin adherence reduce low-density lipoprotein cholesterol (LDL-C) levels from baseline to 12 months.

DESIGN, SETTING, AND PARTICIPANTS

This 4-group, randomized clinical trial was conducted from August 2013 to July 2018 among several large US insurer or employer populations and the University of Pennsylvania Health System. The study population included adults with elevated risk of cardiovascular disease, suboptimal LDL-C control, and evidence of imperfect adherence to statin medication. Data analysis was performed from July 2017 to June 2019.

INTERVENTIONS

The interventions lasted 6 months during which all participants received daily medication reminders and an electronic pill bottle. Statin adherence was measured by opening the bottle. For participants randomized to the 3 intervention groups, adherence was rewarded with financial incentives. The sweepstakes group involved incentives for daily adherence. In the deadline sweepstakes group, incentives were reduced if participants were adherent only after a reminder. The sweepstakes plus deposit contract group split incentives between daily adherence and a monthly deposit reduced for each day of nonadherence.

MAIN OUTCOMES AND MEASURES

The primary outcome was change in LDL-C level from baseline to 12 months.

RESULTS

Among 805 participants randomized (199 in the simple daily sweepstakes group, 204 in the deadline sweepstakes group, 201 in the sweepstakes plus deposit contract group, and 201 in the control group), the mean (SD) age was 58.5 (10.3) years; 519 participants (64.5%) were women, 514 (63.9%) had diabetes, and 273 (33.9%) had cardiovascular disease. The mean (SD) baseline LDL-C level was 143.2 (42.5) mg/dL. Measured adherence at 6 months (defined as the proportion of 180 days with electronic pill bottle opening) in the control group (0.69; 95% CI, 0.66-0.72) was lower than that in the simple sweepstakes group (0.84; 95% CI, 0.81-0.87), the deadline sweepstakes group (0.86; 95% CI, 0.83-0.89), and the sweepstakes plus deposit contract group (0.87; 95% CI, 0.84-0.90) (P < .001 for each incentive group vs control). LDL-C levels were measured for 636 participants at 12 months. Mean LDL-C level reductions from baseline to 12 months were 33.6 mg/dL (95% CI, 28.4-38.8 mg/dL) in the control group, 32.4 mg/dL (95% CI, 27.3-37.6 mg/dL) in the sweepstakes group, 33.2 mg/dL (95% CI, 28.1-38.3 mg/dL) in the deadline sweepstakes group, and 36.5 mg/dL (95% CI, 31.3-41.7 mg/dL) in the sweepstakes plus deposit contract group (adjusted P > .99 for each incentive group vs control).

CONCLUSIONS AND RELEVANCE

Compared with the control group, different financial incentives improved measured statin adherence but not LDL-C levels. This result points to the importance of directly measuring health outcomes, rather than simply adherence, in trials aimed at improving health behaviors.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT01798784.

The effect of increased cost-sharing on low-value service use

We examine the effect of a value-based insurance design (VBID) program implemented at a large public employer in the state of Oregon. The program substantially increased cost-sharing for several healthcare services likely to be of low value for most patients: diagnostic services (e.g., imaging services) and surgeries (e.g., spinal surgeries for pain). Using a difference-in-differences design coupled with granular, administrative health insurance claims data over the period 2008-2012, we estimate the change in low-value service use among beneficiaries before and after program implementation relative to a comparison group not exposed to the VBID. Our findings suggest that the VBID significantly reduced the use of targeted services, with an implied elasticity of demand of -0.22. We find no evidence that the VBID led to substitution to non-targeted services or increased overall healthcare costs. However, we also observe no evidence that the program led to cost-savings.

Improving Identification of Patients at Low Risk for Major Cardiac Events After Noncardiac Surgery Using Intraoperative Data

BACKGROUND/OBJECTIVE

Risk-stratification tools for cardiac complications after noncardiac surgery based on preoperative risk factors are used to inform postoperative management. However, there is limited evidence on whether risk stratification can be improved by incorporating data collected intraoperatively, particularly for low-risk patients.

METHODS

We conducted a retrospective cohort study of adults who underwent noncardiac surgery between 2014 and 2018 at four hospitals in the United States. Logistic regression with elastic net selection was used to classify in-hospital major adverse cardiovascular events (MACE) using preoperative and intraoperative data ("perioperative model"). We compared model performance to standard risk stratification tools and professional society guidelines that do not use intraoperative data.

RESULTS

Of 72,909 patients, 558 (0.77%) experienced MACE. Those with MACE were older and less likely to be female. The perioperative model demonstrated an area under the receiver operating characteristic curve (AUC) of 0.88 (95% CI, 0.85-0.92). This was higher than the Lee Revised Cardiac Risk Index (RCRI) AUC of 0.79 (95% CI, 0.74-0.84; P < .001 for AUC comparison). There were more MACE complications in the top decile (n = 1,465) of the perioperative model's predicted risk compared with that of the RCRI model (n = 58 vs 43). Additionally, the perioperative model identified 2,341 of 7,597 (31%) patients as low risk who did not experience MACE but were recommended to receive postoperative biomarker testing by a risk factor-based guideline algorithm.

CONCLUSIONS

Addition of intraoperative data to preoperative data improved prediction of cardiovascular complication outcomes after noncardiac surgery and could potentially help reduce unnecessary postoperative testing.

Implementing nudges for suicide prevention in real-world environments: project INSPIRE study protocol

Background

Suicide is a global health issue. There are a number of evidence-based practices for suicide screening, assessment, and intervention that are not routinely deployed in usual care settings. The goal of this study is to develop and test implementation strategies to facilitate evidence-based suicide screening, assessment, and intervention in two settings where individuals at risk for suicide are especially likely to present: primary care and specialty mental health care. We will leverage methods from behavioral economics, which involves understanding the many factors that influence human decision making, to inform strategy development.

Methods

We will identify key mechanisms that limit implementation of evidence-based suicide screening, assessment, and intervention practices in primary care and specialty mental health through contextual inquiry involving behavioral health and primary care clinicians. Second, we will use contextual inquiry results to systematically design a menu of behavioral economics-informed implementation strategies that cut across settings, in collaboration with an advisory board composed of key stakeholders (i.e., behavioral economists, clinicians, implementation scientists, and suicide prevention experts). Finally, we will conduct rapid-cycle trials to test and refine the menu of implementation strategies. Primary outcomes include clinician-reported feasibility and acceptability of the implementation strategies.

Discussion

Findings will elucidate ways to address common and unique barriers to evidence-based suicide screening, assessment, and intervention practices in primary care and specialty mental health care. Results will yield refined, pragmatically tested strategies that can inform larger confirmatory trials to combat the growing public health crisis of suicide.

0196 Differences in Sleep Duration and Alertness Among Internal Medicine Interns Comparing Intensive Care Unit to General Medicine Rotations: A Secondary Analysis of the ICOMPARE Trial

Introduction

Little is known about the impact of specific rotations on medical residents’ sleep. The purpose of this analysis was to examine the difference in sleep duration and alertness among internal-medicine resident interns during intensive care unit (ICU) compared to general medicine (GM) rotations.

Methods

This is a secondary report of a randomized non-inferiority trial of 63 United States internal-medicine residency programs. Programs were assigned to either standard duty-hour (80h workweek/16h shifts) or flexible (80h workweek/no shift-length limit) policies. Interns were followed for 2 weeks during either a GM or ICU rotation. The primary outcome was sleep duration/24h (actigraphy). Secondary outcomes were sleepiness (Karolinska Sleepiness Scale [KSS]) and alertness (number of Brief Psychomotor Vigilance Test [PVT-B] lapses). Data were averaged across days (thirteen 24-hour periods). Linear mixed-effect models with random program intercept were used to determine the association between each outcome by rotation, controlling for age, sex, and policy followed.

Results

N=386 interns were included (mean age 27.9±2.1y, 194 (50.3%) males), with n=261 (67.6%) in GM, and n=125 (32.4%) in ICU. Average sleep duration was 7.00±0.08h and 6.84±0.10h for GM and ICU respectively (p=.09; 95%CI -0.02;0.33h). Percent of days with self-reports of excessive sleepiness were significantly more likely for ICU vs GM from 12am-6am (ICU: 20.2%; GM: 12.5%) and 6am-12pm (ICU: 20.5%; GM: 14.3%). GM had significantly more days with no excessive sleepiness (GM: 40.5%; ICU: 28.1%). Average KSS was 4.8±0.1 for both GM and ICU (p=.60; 95%CI -0.18;0.32). Average number of PVT-B lapses were 5.5±0.5 and 5.7±0.7 for GM and ICU respectively (p=.83; 95%CI -1.48;1.18 lapses). There were no significant differences in PVT-B response speed or false starts between rotations.

Conclusion

Interns in ICU may experience more excessive sleepiness compared to GM interns, especially in early morning hours. However, sleep duration and alertness were not significantly different between rotations.

Support

Funded by the National Heart, Lung, and Blood Institute and American Council for Graduate Medical Education

0261 A Randomized Trial on The Effects of Standard and Flexible Duty-Hour Rules on Intern Sleep and Alertness

Introduction

Duty hour regulations affect resident sleep, education, and patient care in complex ways. We performed a national cluster-randomized trial (iCOMPARE) in 63 internal medicine residency programs comparing the effects of the 2011 duty-hour standards to a more flexible set of duty hour rules characterized by maintaining an 80-hour workweek but without limits on shift length or mandatory time off between shifts, relative to patient mortality, intern educational outcomes, and intern sleep and alertness.

Methods

In the sleep and alertness sub-study, sleep duration and morning sleepiness and alertness were assessed with actigraphy, the Karolinska Sleepiness Scale, and a 3-minute Psychomotor Vigilance Test (PVT-B) for 14 days in 193 interns from 6 standard programs and 205 interns from 6 flexible programs.

Results

During the 14-day study periods, interns in standard and flexible programs averaged 7.03h sleep/24h (95% confidence interval [CI] 6.78h, 7.27h) and 6.85h sleep/24h (95% CI 6.61h, 7.10h), respectively. Sleep duration (difference between arms of -0.17h/24h; 1-sided lower 95% confidence limit -0.45h; NIM -0.5h; P=0.02 for noninferiority) and KSS sleepiness (difference 0.12 points; 1-sided upper 95% confidence limit 0.31 points; NIM 1 point; P&lt;0.001) were noninferior in flexible versus standard programs. We could not establish noninferiority for PVT-B alertness (difference -0.3 lapses; 1-sided upper 95% confidence limit 1.6 lapses; NIM 1 lapse; P=0.10). Based on analyses by shift type, sleep duration was 1.77h shorter on days when interns in flexible programs finished an overnight shift relative to a regular day shift (p&lt;.001), with significant decreases in subjective and objective alertness, and frequent reports of excessive sleepiness, especially between 12am and 6am.

Conclusion

There were no signs of relevant chronic sleep loss across shifts in interns in flexible programs relative to their standard program counterparts. Interns were able to compensate for the sleep lost during extended overnight shifts by increasing sleep duration on nights prior to day shifts, night shifts, and days off. Increased sleepiness and reduced alertness of interns following extended overnight shifts need to be mitigated and suggest a role for fatigue-risk management programs.

Support

Supported by NHLBI grants U01HL125388 and U01HL126088 and grants from the ACGME.

Rationale and Design of EMPOWER, a Pragmatic Randomized Trial of Automated Hovering in Patients With Congestive Heart Failure

BACKGROUND

Congestive heart failure is a major cause of morbidity, mortality, and cost. Disease management programs have shown promise but lack firm evidence of effectiveness and scalability. We describe the motivation, design, and planned analyses of EMPOWER (Electronic Monitoring of Patients Offers Ways to Enhance Recovery), a randomized clinical trial of an innovative intervention incorporating behavioral economic principles with remote monitoring technology embedded within a healthcare system.

METHODS AND RESULTS

EMPOWER is an ongoing, pragmatic, randomized clinical trial comparing usual care to an automated hovering intervention that includes patient-level incentives for daily weight monitoring and diuretic adherence combined with automated feedback into the clinical care pathway, enabling real-time response to concerning clinical symptoms. Identification of eligible patients began in May 2016, and implementation of the intervention is feasible. Trial processes are embedded into existing clinical pathways. The primary outcome is time to readmission for any cause. Cost-effectiveness analyses are planned to evaluate the healthcare costs and health outcomes of the approach.

CONCLUSIONS

The EMPOWER trial incorporates leading-edge approaches in human motivation, derived from behavioral economics, with contemporary technology to provide scale and exception handling at low cost. The trial is also implemented within the naturalized environment of a health system, as much as possible taking advantage of the existing journeys of patients and workflows of clinicians. A goal of this pragmatic design is to limit resource utilization and also to test an intervention that would need minimal modification to be translated from research into a new way of practice.

CLINICAL TRIAL REGISTRATION

URL: https://www.clinicaltrials.gov . Unique identifier: NCT02708654.

Association Between State Policies Using Medicaid Exclusions to Sanction Noncompliance With Welfare Work Requirements and Medicaid Participation Among Low-Income Adults

This cohort study examines the association of implementation of Medicaid sanctions in the Temporary Assistance for Needy Families program with Medicaid coverage rates among low-income adults.

Effect of Default Options in Advance Directives on Hospital-Free Days and Care Choices Among Seriously Ill Patients: A Randomized Clinical Trial

IMPORTANCE

There is limited evidence regarding how patients make choices in advance directives (ADs) or whether these choices influence subsequent care.

OBJECTIVE

To examine whether default options in ADs influence care choices and clinical outcomes.

DESIGN, SETTING, AND PARTICIPANTS

This randomized clinical trial included 515 patients who met criteria for having serious illness and agreed to participate. Patients were enrolled at 20 outpatient clinics affiliated with the University of Pennsylvania Health System and the University of Pittsburgh Medical Center from February 2014 to April 2016 and had a median follow-up of 18 months. Data analysis was conducted from November 2018 to April 2019.

INTERVENTIONS

Patients were randomly assigned to complete 1 of the 3 following ADs: (1) a comfort-promoting plan of care and nonreceipt of potentially life-sustaining therapies were selected by default (comfort AD), (2) a life-extending plan of care and receipt of potentially life-sustaining therapies were selected by default (life-extending AD), or (3) no choices were preselected (standard AD).

MAIN OUTCOMES AND MEASURES

This trial was powered to rule out a reduction in hospital-free days in the intervention groups. Secondary outcomes included choices in ADs for an overall comfort-oriented approach to care, choices to forgo 4 forms of life support, patients' quality of life, decision conflict, place of death, admissions to hospitals and intensive care units, and costs of inpatient care.

RESULTS

Among 515 patients randomized, 10 withdrew consent and 13 were later found to be ineligible, leaving 492 (95.5%) in the modified intention-to-treat (mITT) sample (median [interquartile range] age, 63 [56-70] years; 279 [56.7%] men; 122 [24.8%] black; 363 [73.8%] with cancer). Of these, 264 (53.7%) returned legally valid ADs and were debriefed about their assigned intervention. Among these, patients completing comfort ADs were more likely to choose comfort care (54 of 85 [63.5%]) than those returning standard ADs (45 of 91 [49.5%]) or life-extending ADs (33 of 88 [37.5%]) (P = .001). Among 492 patients in the mITT sample, 57 of 168 patients [33.9%] who completed the comfort AD, 47 of 165 patients [28.5%] who completed the standard AD, and 35 of 159 patients [22.0%] who completed the life-extending AD chose comfort care (P = .02), with patients not returning ADs coded as not selecting comfort care. In mITT analyses, median (interquartile range) hospital-free days among 168 patients assigned to comfort ADs and 159 patients assigned to life-extending default ADs were each noninferior to those among 165 patients assigned to standard ADs (standard AD: 486 [306-717] days; comfort AD: 554 [296-833] days; rate ratio, 1.05; 95% CI, 0.90-1.23; P < .001; life-extending AD: 550 [325-783] days; rate ratio, 1.03; 95% CI, 0.88-1.20; P < .001). There were no differences among groups in other secondary outcomes.

CONCLUSIONS AND RELEVANCE

In this randomized clinical trial, default options in ADs altered the choices seriously ill patients made regarding their future care without changing clinical outcomes.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02017548.

Risk Stratification for Postoperative Acute Kidney Injury in Major Noncardiac Surgery Using Preoperative and Intraoperative Data

IMPORTANCE

Acute kidney injury (AKI) is one of the most common complications after noncardiac surgery. Yet current postoperative AKI risk stratification models have substantial limitations, such as limited use of perioperative data.

OBJECTIVE

To examine whether adding preoperative and intraoperative data is associated with improved prediction of noncardiac postoperative AKI.

DESIGN, SETTING, AND PARTICIPANTS

A prognostic study using logistic regression with elastic net selection, gradient boosting machine (GBM), and random forest approaches was conducted at 4 tertiary academic hospitals in the United States. A total of 42 615 hospitalized adults with serum creatinine measurements who underwent major noncardiac surgery between January 1, 2014, and April 30, 2018, were included in the study. Serum creatinine measurements from 365 days before and 7 days after surgery were used in this study.

MAIN OUTCOMES AND MEASURES

Postoperative AKI (defined by the Kidney Disease Improving Global Outcomes within 7 days after surgery) was the primary outcome. The area under the receiver operating characteristic curve (AUC) was used to assess discrimination.

RESULTS

Among 42 615 patients who underwent noncardiac surgery, the mean (SD) age was 57.9 (15.7) years, 23 943 (56.2%) were women, 27 857 (65.4%) were white, and the most frequent surgery types were orthopedic (15 718 [36.9%]), general (8808 [20.7%]), and neurologic (6564 [15.4%]). The rate of postoperative AKI was 10.1% (n = 4318). The progressive addition of clinical data improved model performance across all modeling approaches, with GBM providing the highest discrimination by AUC. In GBM models, the AUC increased from 0.712 (95% CI, 0.694-0.731) using prehospitalization variables to 0.804 (95% CI, 0.788-0.819) using preoperative variables (inclusive of prehospitalization variables) (P < .001 for AUC comparison). The AUC further increased to 0.817 (95% CI, 0.802-0.832) when adding intraoperative variables (P < .001 for comparison vs model using preoperative variables). However, the statistically significant improvements in discrimination did not appear to be clinically significant. In particular, the AKI rate among patients classified as high risk improved from 29.1% to 30.0%, a net of 15 patients were appropriately reclassified as high risk, and an additional 15 patients were appropriately reclassified as low risk.

CONCLUSIONS AND RELEVANCE

The findings of the study suggest that electronic health record data may be used to accurately stratify patients at risk of perioperative AKI, but the modest improvements from adding intraoperative data should be weighed against challenges in using intraoperative data.

Studying expressions of loneliness in individuals using twitter: an observational study

OBJECTIVES

Loneliness is a major public health problem and an estimated 17% of adults aged 18-70 in the USA reported being lonely. We sought to characterise the (online) lives of people who mention the words 'lonely' or 'alone' in their Twitter timeline and correlate their posts with predictors of mental health.

SETTING AND DESIGN

From approximately 400 million tweets collected from Twitter in Pennsylvania, USA, between 2012 and 2016, we identified users whose Twitter posts contained the words 'lonely' or 'alone' and compared them to a control group matched by age, gender and period of posting. Using natural-language processing, we characterised the topics and diurnal patterns of users' posts, their association with linguistic markers of mental health and if language can predict manifestations of loneliness. The statistical analysis, data synthesis and model creation were conducted in 2018-2019.

PRIMARY OUTCOME MEASURES

We evaluated counts of language features in the users with posts including the words lonely or alone compared with the control group. These language features were measured by (a) open-vocabulary topics, (b) Linguistic Inquiry Word Count (LIWC) lexicon, (c) linguistic markers of anger, depression and anxiety, and (d) temporal patterns and number of drug words. Using machine learning, we also evaluated if expressions of loneliness can be predicted in users' timelines, measured by area under curve (AUC).

RESULTS

Twitter timelines of users (n=6202) with posts including the words lonely or alone were found to include themes about difficult interpersonal relationships, psychosomatic symptoms, substance use, wanting change, unhealthy eating and having troubles with sleep. Their posts were also associated with linguistic markers of anger, depression and anxiety. A random forest model predicted expressions of loneliness online with an AUC of 0.86.

CONCLUSIONS

Users' Twitter timelines with the words lonely or alone often include psychosocial features and can potentially have associations with how individuals express and experience loneliness. This can inform low-resource online assessment for high-risk individuals experiencing loneliness and interventions focused on addressing morbidities in this condition.

Effect of Escalating Financial Incentive Rewards on Maintenance of Weight Loss: A Randomized Clinical Trial

IMPORTANCE

Identifying effective strategies for treating obesity is a public health priority.

OBJECTIVE

To test an escalating lottery-based incentive tied to daily self-weighing for weight loss maintenance.

DESIGN, SETTING, AND PARTICIPANTS

This 2-phase, 2-arm randomized clinical trial enrolled participants aged 30 to 80 years with an initial body mass index (calculated as weight in kilograms divided by height in meters squared) of 30.0 to 45.0 who lost at least 5 kg during 4 to 6 months in a national online weight management program. Study investigators and outcomes assessors were blinded to participant groups. Data were collected and analyzed from May 23, 2016, through November 13, 2017, based on intention to treat.

INTERVENTIONS

All participants were advised to weigh themselves daily, with a goal of 6 or more days per week, and received text messaging feedback on their performance. Incentive group participants were eligible for a lottery-based incentive worth an expected value of $3.98 in week 1 that escalated by $0.43 each week they achieved their self-weighing goal during months 1 to 6 (phase 1), followed by no incentives during months 7 to 12 (phase 2).

MAIN OUTCOMES AND MEASURES

The primary outcome was weight change at the end of phase 1. Secondary outcomes included weight change at the end of phase 2 and changes in self-reported physical activity and eating behaviors.

RESULTS

A total of 258 participants (128 in the incentive group and 130 in the control group) had a mean (SD) age of 48.0 (10.5) years and mean (SD) body mass index of 32.1 (3.9); 225 (87.2%) were women; 235 (91.1%) were white; and 102 (39.5%) had an annual income of at least $100 000. Weight measurement was transmitted by 250 participants (96.9%) at 6 months and 227 (88.0%) at 12 months. The percentage of weeks that incentive and control participants achieved a mean self-weighing of at least 6 times was 85.3% vs 75.8%, respectively (P = .002) in phase 1 and 37.7% vs 50.2%, respectively (P = .009) in phase 2. Mean weight changes at the end of phase 1 were -1.1 (95% CI, -2.1 to -0.1) kg in the incentive group and -1.9 (95% CI, -2.9 to -0.8) kg in the control group, with a mean difference of 0.7 (95% CI, -0.7 to 2.2) kg (P = .30 for comparison). At the end of phase 2, mean weight changes were 0.2 (95% CI, -1.2 to 1.7) kg in the incentive group and -0.6 (95% CI, -2.0 to 0.8) kg in the control group, with a mean difference of 0.8 (95% CI, -1.2 to 2.8) kg (P = .41 for comparison).

CONCLUSIONS AND RELEVANCE

Compared with an active control condition of weekly messages, escalating lottery-based incentives transiently increased rates of self-weighing but did not significantly enhance weight loss maintenance.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT01900392.

Real-time Feedback in Pay-for-Performance: Does More Information Lead to Improvement?

BACKGROUND

Pay-for-performance (P4P) has been used expansively to improve quality of care delivered by physicians. However, to what extent P4P works through the provision of information versus financial incentives is poorly understood.

OBJECTIVE

To determine whether an increase in information feedback without changes to financial incentives resulted in improved physician performance within an existing P4P program.

INTERVENTION/EXPOSURE

Implementation of a new registry enabling real-time feedback to physicians on quality measure performance.

DESIGN

Observational, predictive piecewise model at the physician-measure level to examine whether registry introduction associated with performance changes. We used detailed physician quality measure data 3 years prior to registry implementation (2010-2012) and 2 years after implementation (2014-2015). We also linked physician-level data including age, gender, and board certification; group-level data including registry click rates; and patient panel data including chronic conditions.

PARTICIPANTS

Four hundred thirty-four physicians continuously affiliated with Advocate from 2010 to 2015.

MAIN MEASURES

Physician performance on ten quality metrics.

KEY RESULTS

We found no consistent pattern of improvement associated with the availability of real-time information across ten measures. Relative to predicted performance without the registry, average performance increased for two measures (childhood immunization status-rotavirus (p < 0.001) and diabetes care-medical attention for nephropathy (p = 0.024)) and decreased for three measures (childhood immunization status-influenza (p < 0.001) and diabetes care-HbA1c testing (p < 0.001) and poor HbA1c control (p < 0.001)). Results were consistent for subgroup analysis on those most able to improve, i.e., physicians in the bottom tertile of performance prior to registry introduction. Physicians who improved most were in groups that accessed the registry more than those who improved least (8.0 vs 10.0 times per week, p = 0.010).

CONCLUSIONS

More frequent provision of information, provided in real-time, was insufficient to improve physician performance in an existing P4P program with high baseline performance. Results suggest that electronic registries may not themselves drive performance improvement. Future work should consider testing information feedback enhancements with financial incentives.

Model Performance Metrics in Assessing the Value of Adding Intraoperative Data for Death Prediction: Applications to Noncardiac Surgery

We tested the value of adding data from the operating room to models predicting in-hospital death. We assessed model performance using two metrics, the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC), to illustrate the differences in information they convey in the setting of class imbalance. Data was collected on 74,147 patients who underwent major noncardiac surgery and 112 unique features were extracted from electronic health records. Sets of features were incrementally added to models using logistic regression, naïve Bayes, random forest, and gradient boosted machine methods. AUROC increased as more features were added, but changes were small for some modeling approaches. In contrast, AUPRC, which reflects positive predicted value, exhibited improvements across all models. Using AUPRC highlighted the added value of intraoperative data, not seen consistently with AUROC, and that with class imbalance AUPRC may serve as the more clinically relevant criterion.

Effect of Sequential or Active Choice for Colorectal Cancer Screening Outreach: A Randomized Clinical Trial

Importance

Colonoscopy and fecal immunochemical testing (FIT) are considered top-tier tests for colorectal cancer (CRC) screening. Behavioral economic insights about "choice architecture" suggest that participation could be influenced by how people are presented test options.

Objective

To investigate response rates for offering colonoscopy only compared with sequential choice (colonoscopy and then FIT) or active choice (colonoscopy or FIT) through mailed outreach.

Design, Setting, and Participants

Three-arm pragmatic randomized clinical trial conducted between November 14, 2017, and May 14, 2018. The setting was primary care practices at an academic health system. Patients aged 50 to 74 years with at least 2 primary care visits in the 2-year preenrollment period were included if they were eligible but not up to date on CRC screening.

Interventions

Eligible patients received mailed outreach about CRC screening. Equal numbers of eligible patients were randomly assigned to 3 outreach groups to receive mailings about CRC screening with the following options: (1) direct phone number to call for scheduling colonoscopy (colonoscopy only), (2) direct phone number to call for colonoscopy and a mailed FIT kit if no response within 4 weeks (sequential choice), or (3) direct phone number to call for colonoscopy and a mailed FIT kit offered at the same time (active choice).

Main Outcomes and Measures

The primary outcome was CRC screening completion (FIT or colonoscopy) within 4 months of initial outreach. The secondary outcomes were CRC screening completion within 6 months of outreach and the choice of colonoscopy as a screening test.

Results

In total, 438 patients were included in the intent-to-treat analysis, with a median age of 56 years (interquartile range, 52-63 years); 55.0% were women. At 4 months, the CRC screening completion rates were 14.4% (95% CI, 8.7%-20.1%) in the colonoscopy-only arm, 17.1% (95% CI, 11.0%-23.2%) in the sequential choice arm, and 19.9% (95% CI, 13.4%-26.4%) in the active choice arm. Neither choice arm achieved a screening rate statistically greater than that in the colonoscopy-alone arm. Among those who completed CRC screening at 4 months, 90.5% (95% CI, 78.0%-103.0%) chose colonoscopy in the colonoscopy-only arm, which was significantly higher than the 52.0% (95% CI, 32.4%-71.6%; P = .005) and 37.9% (95% CI, 20.2%-55.6%; P < .001) in the sequential choice and active choice arms, respectively.

Conclusions and Relevance

There was no significant increase in CRC screening when offering sequential or active choice, but there was a lower rate of colonoscopy in the choice arms than in the colonoscopy-only arm. Subtle changes in sequencing or defaults can alter patient decision making related to preventive health.

Trial Registration

ClinicalTrials.gov identifier: NCT03246438.