1
|
Kohavi R, Tang D, Xu Y, Hemkens LG, Ioannidis JPA. Online randomized controlled experiments at scale: lessons and extensions to medicine. Trials 2020; 21:150. [PMID: 32033614 PMCID: PMC7007661 DOI: 10.1186/s13063-020-4084-y] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2019] [Accepted: 01/18/2020] [Indexed: 12/03/2022] Open
Abstract
Background Many technology companies, including Airbnb, Amazon, Booking.com, eBay, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, and Yahoo!/Oath, run online randomized controlled experiments at scale, namely hundreds of concurrent controlled experiments on millions of users each, commonly referred to as A/B tests. Originally derived from the same statistical roots, randomized controlled trials (RCTs) in medicine are now criticized for being expensive and difficult, while in technology, the marginal cost of such experiments is approaching zero and the value for data-driven decision-making is broadly recognized. Methods and results This is an overview of key scaling lessons learned in the technology field. They include (1) a focus on metrics, an overall evaluation criterion and thousands of metrics for insights and debugging, automatically computed for every experiment; (2) quick release cycles with automated ramp-up and shut-down that afford agile and safe experimentation, leading to consistent incremental progress over time; and (3) a culture of ‘test everything’ because most ideas fail and tiny changes sometimes show surprising outcomes worth millions of dollars annually. Technological advances, online interactions, and the availability of large-scale data allowed technology companies to take the science of RCTs and use them as online randomized controlled experiments at large scale with hundreds of such concurrent experiments running on any given day on a wide range of software products, be they web sites, mobile applications, or desktop applications. Rather than hindering innovation, these experiments enabled accelerated innovation with clear improvements to key metrics, including user experience and revenue. As healthcare increases interactions with patients utilizing these modern channels of web sites and digital health applications, many of the lessons apply. The most innovative technological field has recognized that systematic series of randomized trials with numerous failures of the most promising ideas leads to sustainable improvement. Conclusion While there are many differences between technology and medicine, it is worth considering whether and how similar designs can be applied via simple RCTs that focus on healthcare decision-making or service delivery. Changes – small and large – should undergo continuous and repeated evaluations in randomized trials and learning from their results will enable accelerated healthcare improvements.
Collapse
Affiliation(s)
- Ron Kohavi
- Analysis & Experimentation, Microsoft, One Microsoft way, Redmond, WA, 98052, USA.,Airbnb, 888 Brannan St, San Francisco, CA, 94103, USA
| | - Diane Tang
- Google, 1600 Amphitheatre Parkway, Mountain View, CA, 94043, USA
| | - Ya Xu
- LinkedIn, 950 W Maude Ave, Sunnyvale, CA, 94085, USA
| | - Lars G Hemkens
- Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, 4031, Basel, Switzerland
| | - John P A Ioannidis
- Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Medical School Office Building, Room X306, 1265 Welch Rd, Stanford, CA, 94305, USA. .,Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Palo Alto, CA, 94305, USA. .,Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, 94305, USA. .,Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, 94305, USA. .,Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, 94305, USA.
| |
Collapse
|
2
|
Mc Cord KA, Al-Shahi Salman R, Treweek S, Gardner H, Strech D, Whiteley W, Ioannidis JPA, Hemkens LG. Routinely collected data for randomized trials: promises, barriers, and implications. Trials 2018; 19:29. [PMID: 29325575 PMCID: PMC5765645 DOI: 10.1186/s13063-017-2394-5] [Citation(s) in RCA: 82] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2017] [Accepted: 11/29/2017] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Routinely collected health data (RCD) are increasingly used for randomized controlled trials (RCTs). This can provide three major benefits: increasing value through better feasibility (reducing costs, time, and resources), expanding the research agenda (performing trials for research questions otherwise not amenable to trials), and offering novel design and data collection options (e.g., point-of-care trials and other designs directly embedded in routine care). However, numerous hurdles and barriers must be considered pertaining to regulatory, ethical, and data aspects, as well as the costs of setting up the RCD infrastructure. Methodological considerations may be different from those in traditional RCTs: RCD are often collected by individuals not involved in the study and who are therefore blinded to the allocation of trial participants. Another consideration is that RCD trials may lead to greater misclassification biases or dilution effects, although these may be offset by randomization and larger sample sizes. Finally, valuable insights into external validity may be provided when using RCD because it allows pragmatic trials to be performed. METHODS We provide an overview of the promises, challenges, and potential barriers, methodological implications, and research needs regarding RCD for RCTs. RESULTS RCD have substantial potential for improving the conduct and reducing the costs of RCTs, but a multidisciplinary approach is essential to address emerging practical barriers and methodological implications. CONCLUSIONS Future research should be directed toward such issues and specifically focus on data quality validation, alternative research designs and how they affect outcome assessment, and aspects of reporting and transparency.
Collapse
Affiliation(s)
- Kimberly A. Mc Cord
- Basel Institute for Clinical Epidemiology and Biostatistics (CEB), Department of Clinical Research, University Hospital Basel, University of Basel, Spitalstrasse 12, 4031 Basel, Switzerland
| | | | - Shaun Treweek
- Health Services Research Unit, University of Aberdeen, Aberdeen, AB25 2ZD UK
| | - Heidi Gardner
- Health Services Research Unit, University of Aberdeen, Aberdeen, AB25 2ZD UK
| | - Daniel Strech
- Institute for History, Ethics and Philosophy of Medicine, Hannover Medical School, 30625 Hannover, Germany
| | - William Whiteley
- Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, EH16 4SB UK
| | - John P. A. Ioannidis
- Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305 USA
- Meta-Research Innovation Center at Stanford (METRICS), Stanford School of Medicine, Palo Alto, CA 94304 USA
- Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305 USA
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA 94305 USA
- Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305 USA
| | - Lars G. Hemkens
- Basel Institute for Clinical Epidemiology and Biostatistics (CEB), Department of Clinical Research, University Hospital Basel, University of Basel, Spitalstrasse 12, 4031 Basel, Switzerland
| |
Collapse
|
4
|
Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ 2016; 352:i493. [PMID: 26858277 PMCID: PMC4772787 DOI: 10.1136/bmj.i493] [Citation(s) in RCA: 122] [Impact Index Per Article: 15.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 01/08/2016] [Indexed: 11/04/2022]
Abstract
OBJECTIVE To assess differences in estimated treatment effects for mortality between observational studies with routinely collected health data (RCD; that are published before trials are available) and subsequent evidence from randomized controlled trials on the same clinical question. DESIGN Meta-epidemiological survey. DATA SOURCES PubMed searched up to November 2014. METHODS Eligible RCD studies were published up to 2010 that used propensity scores to address confounding bias and reported comparative effects of interventions for mortality. The analysis included only RCD studies conducted before any trial was published on the same topic. The direction of treatment effects, confidence intervals, and effect sizes (odds ratios) were compared between RCD studies and randomized controlled trials. The relative odds ratio (that is, the summary odds ratio of trial(s) divided by the RCD study estimate) and the summary relative odds ratio were calculated across all pairs of RCD studies and trials. A summary relative odds ratio greater than one indicates that RCD studies gave more favorable mortality results. RESULTS The evaluation included 16 eligible RCD studies, and 36 subsequent published randomized controlled trials investigating the same clinical questions (with 17,275 patients and 835 deaths). Trials were published a median of three years after the corresponding RCD study. For five (31%) of the 16 clinical questions, the direction of treatment effects differed between RCD studies and trials. Confidence intervals in nine (56%) RCD studies did not include the RCT effect estimate. Overall, RCD studies showed significantly more favorable mortality estimates by 31% than subsequent trials (summary relative odds ratio 1.31 (95% confidence interval 1.03 to 1.65; I(2)=0%)). CONCLUSIONS Studies of routinely collected health data could give different answers from subsequent randomized controlled trials on the same clinical questions, and may substantially overestimate treatment effects. Caution is needed to prevent misguided clinical decision making.
Collapse
Affiliation(s)
- Lars G Hemkens
- Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland
| | - Despina G Contopoulos-Ioannidis
- Department of Pediatrics, Division of Infectious Diseases, Stanford University School of Medicine, Stanford, California, USA Meta-Research Innovation Center at Stanford (METRICS)
| | - John P A Ioannidis
- Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Meta-Research Innovation Center at Stanford (METRICS) Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, USA Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA
| |
Collapse
|
5
|
Hospital Readmissions in Patients With Carbapenem-Resistant Klebsiella pneumoniae. Infect Control Hosp Epidemiol 2015; 37:281-8. [PMID: 26686227 DOI: 10.1017/ice.2015.298] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
Abstract
BACKGROUND Various transmission routes contribute to spread of carbapenem-resistant Klebsiella pneumoniae (CRKP) in hospitalized patients. Patients with readmissions during which CRKP is again isolated ("CRKP readmission") potentially contribute to transmission of CRKP. OBJECTIVE To evaluate CRKP readmissions in the Consortium on Resistance against Carbapenems in K. pneumoniae (CRaCKLe). DESIGN Cohort study from December 24, 2011, through July 1, 2013. SETTING Multicenter consortium of acute care hospitals in the Great Lakes region. PATIENTS All patients who were discharged alive during the study period were included. Each patient was included only once at the time of the first CRKP-positive culture. METHODS All readmissions within 90 days of discharge from the index hospitalization during which CRKP was again found were analyzed. Risk factors for CRKP readmission were evaluated in multivariable models. RESULTS Fifty-six (20%) of 287 patients who were discharged alive had a CRKP readmission. History of malignancy was associated with CRKP readmission (adjusted odds ratio [adjusted OR], 3.00 [95% CI, 1.32-6.65], P<.01). During the index hospitalization, 160 patients (56%) received antibiotic treatment against CRKP; the choice of regimen was associated with CRKP readmission (P=.02). Receipt of tigecycline-based therapy (adjusted OR, 5.13 [95% CI, 1.72-17.44], using aminoglycoside-based therapy as a reference in those treated with anti-CRKP antibiotics) was associated with CRKP readmission. CONCLUSION Hospitalized patients with CRKP-specifically those with a history of malignancy-are at high risk of readmission with recurrent CRKP infection or colonization. Treatment during the index hospitalization with a tigecycline-based regimen increases this risk.
Collapse
|