The Expanding Role of Real-World Evidence Trials in Health Care Decision Making

Key results from observational studies and real-world evidence of sodium-glucose cotransporter-2 inhibitor effectiveness and safety in reducing cardio-renal risk

Sodium-glucose cotransporter 2 (SGLT2) inhibitors, originally designed to manage blood sugar levels in individuals with type 2 diabetes (T2D), have emerged as a crucial class of drugs for managing cardio-renal diseases. These drugs work by targeting the SGLT2 protein in the kidneys, promoting the excretion of glucose and influencing metabolic pathways beyond glucose control. The relationship between cardio-renal diseases and SGLT2 inhibitors has been explored through landmark trials and real-world evidence (RWE) studies, demonstrating significant reductions in cardio-renal complications. This review discusses the importance of RWE studies alongside randomized controlled trials in understanding the real-world effectiveness and safety of SGLT2 inhibitors. It outlines the advantages and disadvantages of RWE compared to RCTs, highlighting their complementary roles in providing comprehensive insights into treatment outcomes. By examining a range of RWE studies, the review underscores the cardio-renal benefits of SGLT2 inhibitors across various patient populations. Safety assessments indicate that SGLT2 inhibitors are generally well tolerated, with severe adverse events being rare. Common issues, such as genital mycotic infections and urinary tract infections, are acknowledged, alongside less frequent but significant adverse events including diabetic ketoacidosis, lower-limb amputations, and bone fractures. In summary, SGLT2 inhibitors show promising cardio-renal protective effects in real-world scenarios across diverse populations in T2D, indicating their potential as early intervention measures. Continued research is essential for gaining a thorough understanding of their long-term effects and safety profiles.

Dark Data in Real-World Evidence: Challenges, Implications, and the Imperative of Data Literacy in Medical Research

Randomized controlled trials (RCTs) and real-world evidence (RWE) studies are crucial and complementary in generating clinical evidence. RCTs provide controlled settings to validate the clinical effect of specific drugs or medical devices, while RWE integrates extrinsic factors, encompassing external influences affecting real-world scenarios, thus challenging RCT results in practical applications. In this study, we explore the impact of extrinsic factors on RWE outcomes, focusing on "dark data," which refers to data collected but not used or excluded from the analyses. Dark data can arise in many ways during research process, from selecting study samples to data collection and analysis. However, even unused or unanalyzed dark data hold potential insights, providing a comprehensive view of clinical contexts. Extrinsic factors lead to divergent RWE outcomes that could differ from RCTs beyond statistical correction's scope. Two main types of dark data exist: "known-unknown" and "unknown-unknown." The distinction between these dark data types highlights RWE's complexity. The transformation of unknown into known depends on data literacy-powerful utilization capabilities that can be interpreted based on medical expertise. Shifting the focus to excluded subjects or unused data in real-world contexts reveals unexplored potential. Understanding the significance of dark data is vital in reflecting the complexity of clinical settings. Connecting RCTs and RWEs requires medical data literacy, enabling clinicians to decipher meaningful insights. In the big data and artificial intelligence era, medical staff must navigate data complexities while promoting the core role of medicine. Prepared clinicians will lead this transformative journey, ensuring data value shapes the medical landscape.

Clinical trials are becoming more complex: a machine learning analysis of data from over 16,000 trials

The past decade has seen substantial innovation in clinical trials, including new trial formats, endpoints, and others. Also there have been regulatory changes, increasing competitive pressures and other external factors which impact clinical trials. In parallel, trial timelines have increased and success rates remain stubbornly low. This has led many observers to question whether clinical trials have become overly complex and if this complexity is always needed. Here we present a large-scale analysis of protocols and other data from over 16,000 trials. Using a machine learning algorithm, we automatically assessed key features of these trials, such as number of endpoints, number of inclusion-exclusion criteria and others. Using a regression analysis we combined these features into a new metric, the Trial Complexity Score, which correlates with overall clinical trial duration. Looking at this score across different clinical phases and therapeutic areas we see substantial increases over time, suggesting that clinical trials are indeed becoming more complex. We discuss drivers of increasing trial complexity, necessary or helpful ('good') complexity versus unnecessary ('bad') complexity, and we explore mechanisms of how sponsors of clinical trials can reduce trial complexity where appropriate.

Statistical and Scientific Considerations Concerning the Interpretation, Replicability, and Transportability of Research Findings

To advance scientific understanding of disease processes and related intervention effects, study results should be free from bias and replicable. More broadly, investigators seek results that are transportable, that is, applicable to a perceived study population as well as in other environments and populations. We review fundamental statistical issues that arise in the analysis of observational data from disease cohorts and other sources and discuss how these issues affect the transportability and replicability of research results. Much of the literature focuses on estimating average exposure or intervention effects at the population level, but we argue for more nuanced analyses of conditional effects that reflect the complexity of disease processes.

Effectiveness of pharmacogenomics on the response and remission of treatment-resistant depression: a meta-analysis of randomised controlled trials

Background

Pharmacogenomics (PGx) is a promising tool to realise tailored drug therapy for depression.

Aims

To investigate the treatment efficacy of PGx for treatment-resistant depression (TRD) compared with treatment as usual.

Methods

A systematic search was conducted in PubMed, Embase, the Cochrane Library, Web of Science and PsycINFO to identify relevant studies published from inception to 15 April 2023. Two-arm randomised controlled trials (RCTs) exploring the efficacy of PGx-guided versus unguided treatment for TRD were included. The risk of bias in the included studies was evaluated using the Cochrane risk of bias assessment tool. The overall quality of evidence was assessed using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach.

Results

Seven RCTs (n=3003) comparing PGx-guided (n=1492) and unguided (n=1511) groups were identified and analysed. PGx-guided treatment was superior to treatment as usual in response (relative risk (RR)=1.31; 95% confidence interval (95% CI): 1.15 to 1.49; p<0.001) and remission (RR=1.40; 95% CI: 1.09 to 1.80; p=0.009) improvements. Effect sizes for acceptability (RR=0.90; 95% CI: 0.80 to 1.02; p=0.100) and side effect burden (RR=0.58; 95% CI: 0.29 to 1.15; p=0.120) between the two groups were not statistically different. The overall quality of evidence was rated from 'very low' (25%) to 'low' (75%) based on the GRADE criteria.

Conclusions

PGx-guided treatment has shown a small overall effect in improving the response and remission rates for patients with TRD. However, these results should be interpreted cautiously because of the few included studies and the low quality of evidence. Further high-quality clinical trials are warranted to confirm the findings.

PROSPERO registration number

CRD42022340182.

Using public clinical trial reports to probe non-experimental causal inference methods

BACKGROUND

Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study.

METHODS

We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo "ground truths" about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset.

RESULTS

From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline.

CONCLUSIONS

We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data.

Tretinoin 0.1% and Benzoyl Peroxide 3% Cream for the Treatment of Facial Acne Vulgaris

OBJECTIVE

To assess the efficacy, safety, and clinical application of tretinoin 0.1%-benzoyl peroxide 3% cream for the topical treatment of acne vulgaris.

DATA SOURCES

A systematic review of the literature was performed using the terms Twyneo OR tretinoin and benzoyl peroxide OR S6G5T-3 in MEDLINE (PubMed) and EMBASE. ClinicalTrials.gov was searched to obtain completed clinical trial results not published elsewhere.

STUDY SELECTION AND DATA EXTRACTION

All human studies published in English prior to November 2022 related to pharmacology, clinical trials, safety, and efficacy were evaluated for inclusion.

DATA SYNTHESIS

In two 12-week, phase 3, randomized, vehicle-controlled clinical trials, tretinoin 0.1%-benzoyl peroxide 3% cream significantly reduced inflammatory and noninflammatory facial acne lesions and significantly improved Investigator Global Assessment (IGA) rating to clear or almost clear. The cream has a suitable safety profile, with application site pain and dryness as the most common adverse events.

RELEVANCE TO PATIENT CARE AND CLINICAL PRACTICE IN COMPARISON TO EXISTING AGENTS

Tretinoin-BPO had similar IGA success compared to other topical retinoid and retinoid-BPO treatments for acne vulgaris. Compared to individual tretinoin and benzoyl peroxide therapy, the combination product streamlines application, which will improve medication adherence; however, the cost of tretinoin-BPO cream may be prohibitive.

CONCLUSIONS

Tretinoin 0.1%-benzoyl peroxide 3% cream is safe and effective for the treatment of moderate-to-severe acne. Long-term trial data on efficacy and tolerability are not yet available.

Evaluating and improving real-world evidence with Targeted Learning

BACKGROUND

The Targeted Learning roadmap provides a systematic guide for generating and evaluating real-world evidence (RWE). From a regulatory perspective, RWE arises from diverse sources such as randomized controlled trials that make use of real-world data, observational studies, and other study designs. This paper illustrates a principled approach to assessing the validity and interpretability of RWE.

METHODS

We applied the roadmap to a published observational study of the dose-response association between ritodrine hydrochloride and pulmonary edema among women pregnant with twins in Japan. The goal was to identify barriers to causal effect estimation beyond unmeasured confounding reported by the study's authors, and to explore potential options for overcoming the barriers that robustify results.

RESULTS

Following the roadmap raised issues that led us to formulate alternative causal questions that produced more reliable, interpretable RWE. The process revealed a lack of information in the available data to identify a causal dose-response curve. However, under explicit assumptions the effect of treatment with any amount of ritodrine versus none, albeit a less ambitious parameter, can be estimated from data.

CONCLUSIONS

Before RWE can be used in support of clinical and regulatory decision-making, its quality and reliability must be systematically evaluated. The TL roadmap prescribes how to carry out a thorough, transparent, and realistic assessment of RWE. We recommend this approach be a routine part of any decision-making process.

Retrospective Analysis With Propensity Score Matching of Peripheral T-Cell Lymphoma Treated Frontline With Brentuximab Vedotin and Chemotherapy

BACKGROUND

Since Food and Drug Administration approval of brentuximab vedotin in combination with cyclophosphamide, doxorubicin, and prednisone (A + CHP) as initial therapy for previously untreated CD30-expressing peripheral T-cell lymphoma (PTCL), there has been limited research on real-world patient characteristics, treatment patterns, and clinical outcomes.

METHODS

We retrospectively analyzed claims of patients with PTCL treated with frontline A + CHP or CHOP (cyclophosphamide, doxorubicin, vincristine, prednisone) using the Symphony Health Solutions database. Adults with International Classification of Diseases-9/10 PTCL diagnosis codes who initiated A + CHP or CHOP between November 2018 and July 2021 were included. A 1:1 propensity score matching analysis was performed that adjusted for potential confounders between groups.

RESULTS

A total of 1344 patients were included (A + CHP, n = 749; CHOP, n = 595). Before matching, 61% were men; median age at index was 62 (A + CHP) and 69 (CHOP) years. The most common A + CHP-treated PTCL subtypes were systemic anaplastic large cell lymphoma (sALCL; 51%), PTCL-not otherwise specified (NOS; 30%), and angioimmunoblastic T-cell lymphoma (AITL; 12%); the most common CHOP-treated subtypes were PTCL-NOS (51%) and AITL (19%). After matching, similar proportions of patients treated with A + CHP and CHOP received granulocyte colony-stimulating factor (89% vs. 86%, P = .3). Fewer patients treated with A + CHP received subsequent therapy than CHOP overall (20% vs. 30%, P < .001) and specifically with the sALCL subtype (15% vs. 28%, P = .025).

CONCLUSIONS

Characteristics and management of this real-world PTCL population who were older and had a higher comorbidity burden than that in the ECHELON-2 trial demonstrate the importance of retrospective studies when assessing the impact of new regimens on clinical practice.

The Effect of Daily Meditative Practices Based on Mindfulness and Self-Compassion on Emotional Distress under Stressful Conditions: A Randomized Controlled Trial

Intervention programs based on self-compassion have demonstrated their efficacy both in reducing psychological distress and increasing well-being. The goal of this study was to test the efficacy of an online intervention to increase mindfulness and self-compassion levels in a non-clinical sample in a highly stressful context: the ten weeks of lockdown imposed in the early stages of the COVID-19 pandemic. The intervention sessions consisted of thirty-minute guided meditations followed by thirty minutes of inquiry. Sixty-one participants completed two thirds of the sessions or more, and 65 individuals participated in a waiting-list (WL) control group. Self-compassion, anxiety, depression and stress levels were assessed. The analysis of pre-post results suggests that the interventions increased self-compassion levels and decreased anxiety, depression and stress levels, whereas the WL group did not show any significant changes. The emotional changes in the intervention group were associated with the increase in self-compassion. However, at follow-up, the scores of emotional distress variables returned to the initial pre-intervention scores. These data can be interpreted in line with previous results that have shown the efficacy of self-compassion-based intervention programs. Given that this efficacy was not maintained at follow-up, data are discussed according to the pervasive role of a highly stressful context and-as described in other studies-the need for regular practice to maintain the benefits obtained.

Guidance on using real-world evidence from Western Europe in Central and Eastern European health policy decision making

Aim: Real-world data and real-world evidence (RWE) are becoming more important for healthcare decision making and health technology assessment. We aimed to propose solutions to overcome barriers preventing Central and Eastern European (CEE) countries from using RWE generated in Western Europe. Materials & methods: To achieve this, following a scoping review and a webinar, the most important barriers were selected through a survey. A workshop was held with CEE experts to discuss proposed solutions. Results: Based on survey results, we selected the nine most important barriers. Multiple solutions were proposed, for example, the need for a European consensus, and building trust in using RWE. Conclusion: Through collaboration with regional stakeholders, we proposed a list of solutions to overcome barriers on transferring RWE from Western Europe to CEE countries.

Learning end-to-end patient representations through self-supervised covariate balancing for causal treatment effect estimation

A causal effect can be defined as a comparison of outcomes that result from two or more alternative actions, with only one of the action-outcome pairs actually being observed. In healthcare, the gold standard for causal effect measurements is randomized controlled trials (RCTs), in which a target population is explicitly defined and each study sample is randomly assigned to either the treatment or control cohorts. The great potential to derive actionable insights from causal relationships has led to a growing body of machine-learning research applying causal effect estimators to observational data in the fields of healthcare, education, and economics. The primary difference between causal effect studies utilizing observational data and RCTs is that for observational data, the study occurs after the treatment, and therefore we do not have control over the treatment assignment mechanism. This can lead to massive differences in covariate distributions between control and treatment samples, making a comparison of causal effects confounded and unreliable. Classical approaches have sought to solve this problem piecemeal, first by predicting treatment assignment and then treatment effect separately. Recent work extended part of these approaches to a new family of representation-learning algorithms, showing that the upper bound of the expected treatment effect estimation error is determined by two factors: the outcome generalization-error of the representation and the distance between the treated and control distributions induced by the representation. To achieve minimal dissimilarity in learning such distributions, in this work we propose a specific auto-balancing, self-supervised objective. Experiments on real and benchmark datasets revealed that our approach consistently produced less biased estimates than previously published state-of-the-art methods. We demonstrate that the reduction in error can be directly attributed to the ability to learn representations that explicitly reduce such dissimilarity; further, in case of violations of the positivity assumption (frequent in observational data), we show our approach performs significantly better than the previous state of the art. Thus, by learning representations that induce similar distributions of the treated and control cohorts, we present evidence to support the error bound dissimilarity hypothesis as well as providing a new state-of-the-art model for causal effect estimation.

Real-World Patient Characteristics, Treatment Patterns, and Outcomes for Patients With Stage III or IV Classic Hodgkin Lymphoma Treated With Frontline ABVD: A Retrospective Database Review in the United States

In newly diagnosed stage III/IV classic Hodgkin lymphoma (cHL), A+AVD (brentuximab vedotin, doxorubicin, vinblastine, dacarbazine) improved overall survival (OS) versus ABVD (doxorubicin, bleomycin, vinblastine, dacarbazine). As clinical trial and real-world populations may differ, real-world treatment characteristics and OS (rwOS) were assessed for patients with stage III/IV cHL treated with frontline ABVD. This retrospective, observational analysis of deidentified electronic health record data (1/1/2011-8/31/2020) evaluated baseline disease and clinical characteristics, treatment patterns, and rwOS in patients with stage III/IV cHL treated with frontline ABVD. Data for 167 patients were analyzed. A median of 6 ABVD cycles were received. Baseline/interim positron emission tomography (PET) scans were obtained for 60.5%/89.8% of patients. Of patients diagnosed in 2016 or later (n = 73), 89% received an interim PET scan; 15/46 patients with no documented Deauville score, 6/15 with a score of 1 to 3, and 3/4 with a score of 4 to 5 de-escalated to AVD. Following frontline ABVD, 55.1% of patients received subsequent systemic therapy and 31.7% stem cell transplantation (SCT). At a median follow-up of 31.8 months, 82.0% of patients were alive (median rwOS, 101.2 months). Patients with stage III/IV cHL treated with frontline ABVD in the real world versus in clinical trials receive more subsequent therapy, including SCTs. Interim PET scans and Deauville scores were not universally obtained after treatment cycle 2, yet treatment de-escalation was observed. Patients with stage III/IV cHL may benefit from frontline A+AVD versus ABVD, as it improves OS and reduces the burden of subsequent therapy, including SCTs.

Clustering of prostate cancer healthcare pathways in the French National Healthcare database

BACKGROUND

Healthcare pathways of patients with prostate cancer are heterogeneous and complex to apprehend using traditional descriptive statistics. Clustering and visualization methods can enhance their characterization.

METHODS

Patients with prostate cancer in 2014 were identified in the French National Healthcare database (Système National des Données de Santé-SNDS) and their data were extracted with up to 5 years of history and 4 years of follow-up. Fifty-one-specific encounters constitutive of prostate cancer management were synthesized into four macro-variables using a clustering approach. Their values over patient follow-ups constituted healthcare pathways. Optimal matching was applied to calculate distances between pathways. Partitioning around medoids was then used to define consistent groups across four exclusive cohorts of incident prostate cancer patients: Hormone-sensitive (HSPC), metastatic hormone-sensitive (mHSPC), castration-resistant (CRPC), and metastatic castration-resistant (mCRPC). Index plots were used to represent pathways clusters.

RESULTS

The repartition of macro-variables values-surveillance, local treatment, androgenic deprivation, and advanced treatment-appeared to be consistent with prostate cancer status. Two to five clusters of healthcare pathways were observed in each of the different cohorts, corresponding for most of them to relevant clinical patterns, although some heterogeneity remained. For instance, clustering allowed to distinguish patients undergoing active surveillance, or treated according to cancer progression risk in HSPC, and patients receiving treatment for potentially curative or palliative purposes in mHSPC and mCRPC.

CONCLUSION

Visualization methods combined with a clustering approach enabled the identification of clinically relevant patterns of prostate cancer management. Characterization of these care pathways is an essential element for the comprehension and the robust assessment of healthcare technology effectiveness.

Examining Endpoint Concordance in Clinical Trials and Real-World Clinical Practice to Advance Real-World Evidence Utilization

Real-world evidence (RWE) is increasingly contributing to more informed decisions regarding the optimal access to and use of therapeutics to improve patient outcomes. However, in many cases, a disconnect between evidence derived from clinical trials and the RWE that follows market approval impedes the potential value and widespread adoption of RWE to optimize patient care. Collaborators with the Learning Ecosystems Accelerator for Patient-centered, Sustainable innovation (LEAPS), a major project of the Tufts Medical Center [formally Massachusetts Institute of Technology (MIT)] NEW Drug Development ParadIGmS (NEWDIGS) initiative, propose assessing the relationship between efficacy endpoints used in randomized controlled trials (RCTs) and effectiveness measures that inform treatment decisions within real-world clinical settings as one way to bridge this divide and further leverage RWE to improve care and patient outcomes. This commentary outlines elements of an endpoint concordance study using Rheumatoid Arthritis as a case study. The authors describe the ways in which better understanding of the relationship between effectiveness and RCT endpoints could improve the confidence in and adoption of RWE by both contextualizing existing RWE as well as identifying ways in which to improve the value of RWE in improving care and outcomes.

Effect of prior treatments on post-CDK 4/6 inhibitor survival in hormone receptor-positive breast cancer

PURPOSE

Multiple treatment options exist for patients with metastatic breast cancer (MBC). However, limited information is available on the impact of prior treatment duration and class on survival outcome for novel therapies, such as cyclin-dependent kinase 4/6 inhibitors (CDK4/6i) for patients with hormone receptor-positive, human epidermal growth factor receptor 2-negative (HR+ HER2-) MBC.

METHODS

This study used a nationwide, de-identified electronic health record-derived database to identify women with HR+ HER2- MBC who received at least one CDK 4/6i between 2011 and 2020. Hazard ratios (HR) and 95% confidence intervals (CI) were estimated for the association between prior duration and class of cancer treatment (both early-stage and metastatic) and prior CDK 4/6i survival as well as for those with multiple CDK 4/6i.

RESULTS

Of 5363 patients, the median survival from first CDK 4/6 inhibitor administration was 3.3 years. When compared to patients with no prior treatments, patients with < 1 year of prior treatment duration had a 30% increased hazard of death (HR, 1.30; 95% CI 1.15-1.46), those with 1 to < 3 years a 68% increased hazard of death (HR 1.68; 95% CI 1.49-1.88), and those with 3 or more years a 55% increased hazard of death (HR 1.55; 95% CI 1.36, 1.76). Patients who received prior therapy (endocrine or chemotherapy) before their CDK 4/6i had worse outcomes than those who received no prior therapy. Similar results were seen when comparing patients in the metastatic setting alone. Finally, patients who received a different CDK 4/6i after their first saw a lower hazard of death compared to patients who received subsequent endocrine or chemotherapy after their first CDK 4/6i.

CONCLUSION

Prior treatment duration and class are associated with a decreased overall survival after CDK 4/6 inhibitor administration. This highlights the importance for clinicians to consider prior treatment and duration in treatment decision-making and for trialists to stratify by these factors when randomizing patients or reporting results of future studies.

Post-Market Surveillance Assessment of the Clinical Accuracy of a Blood Glucose Monitoring System with an Improved Algorithm for Enhanced Product Performance

BACKGROUND

On-going manufacturer-led post-market surveillance (PMS), assessing the clinical accuracy of blood glucose monitoring (BGM) systems, is critical to substantiate the performance of such products for people with diabetes.

MATERIALS AND METHODS

Batches of Verio test-strip product were randomly and routinely selected over the period from launch of an improved-algorithm product to reporting date and sent to 3 clinic sites for clinician-led accuracy assessment. Accuracy is reported as per recently adopted FDA guidance for BGM systems, EN ISO 15197:2015 and MARD/MAD (Mean absolute relative difference/Mean absolute difference).

RESULTS

Thirty-three individual test-strip batches were evaluated corresponding to 506 unique donors. Accuracy performance - FDA: 98.9% of values within ±15% of comparator; ISO: 99.0% within ±15 mg/dL or ±15% at <100 mg/dL (<5.55 mmol/L) or ≥100 mg/dL (≥5.55 mmol/L) glucose, respectively. Overall MARD was 4.19% with a MARD range of 3.54%-5.73% across all test strip batches.

CONCLUSIONS

This post-market surveillance program demonstrates the new BGM system consistently meets measures of clinical accuracy specified by regulators. This program supports a growing demand by regulators for real-world evidence demonstrating consistent in-market product efficacy as opposed to the current largely passive approach that relies on assessment of reports filed by device users.

Post-Market Surveillance of a Blood Glucose Test Strip Demonstrates No Evidence of Interference on Clinical Accuracy in a Large Cohort of People with Type 1 or Type 2 Diabetes

BACKGROUND

Regulations and industry guidance relating to testing for interference in blood glucose monitoring (BGM) systems continue to focus on in vitro laboratory bench tests. Post-market surveillance (PMS) in a clinical setting allows for BGM accuracy assessments to evaluate the impact of real-world exposure to polypharmacy in people with diabetes. This study evaluated the OneTouch Select Plus® BGM test-strip accuracy with respect to polypharmacy using a clinical registry dataset.

METHODS

Medication profiles were analysed for 1023 subjects (425 with type 1 (T1D) and 598 with type 2 diabetes (T2D)) attending 3 UK hospitals. Blood samples were analysed to determine clinical accuracy of the BGM test-strip against a laboratory comparator.

RESULTS

538 different medications (48 diabetes and 490 non-diabetes) were recorded across the 1023 subjects. Patients took on average 6.9 (n = 1-36) individual medications and 4.1 (n = 1-13) unique medication classes. Clinical accuracy to EN ISO 15197:2015 criteria were met irrespective of increasing average number of individual medications, categorized from 1-3, 4-6, 7-9, 10-12 and >12 taken per subject (97.7%, 97.7%, 97.8%, 97.8%, and 98.4%, respectively). Clinical accuracy criteria were met across 15 classes of medication using the combined dataset (97.9%; 29784/30433). Surveillance Error Grid (SEG) analysis showed 98.7% (29959/30368) of readings presented no clinical risk. No individual class or combination of medication classes impacted clinical accuracy of the BGM test-strip.

CONCLUSIONS

Clinical performance for the test strip under assessment demonstrated no evidence of interference from over 500 prescription medications, with clinical accuracy maintained across a range of polypharmacy conditions in people with diabetes.

Development of an Electronic Health Record Registry to Facilitate Collection of Commission on Cancer Metrics for Patients Undergoing Surgery for Breast Cancer

PURPOSE

Accurate and efficient data collection is a challenge for quality improvement initiatives and clinical research. We describe the development of a custom electronic health record (EHR)–based registry to automatically extract structured Commission on Cancer axillary surgery-specific metrics from a custom synoptic note template included in the operative reports for patients with breast cancer undergoing surgery.

METHODS

The smart functionality of our enterprise-based EHR system was leveraged to create a custom smart phrase to capture axillary surgery-specific variables. A multidisciplinary team developed structured data elements correlating to each axillary surgery-specific variable. These data elements were then included in a note template for the operative report. Each variable could be aggregated and converted into a single flat database through the EHR's reporting workbench and serve as a live, prospective registry for all users within the EHR.

RESULTS

The final axillary surgery-specific note template in a synoptic format allowed for efficient and easy entry and automatic collection of breast cancer–specific metrics. From initial adoption in February 2021-December 2021, there were 1,254 patients who underwent breast surgery with axillary surgery. The operative notes allowed for automatic capture of metrics from 60.5% (n = 759) of patients. Data capture improved from 37.6% in the initial adoption period of 6 months to 86.2% in the last 5 months.

CONCLUSION

We were able to demonstrate successful implementation of provider-driven structured data entry into EHR systems that permits automatic data capture. The end result is a custom synoptic note template and a real-time, prospective registry of breast cancer–specific Commission on Cancer metrics that are robust enough to use for quality improvement initiatives and clinical research.

Development and validation of a novel model for characterizing migraine outcomes within real-world data

BACKGROUND

In disease areas with 'soft' outcomes (i.e., the subjective aspects of a medical condition or its management) such as migraine or depression, extraction and validation of real-world evidence (RWE) from electronic health records (EHRs) and other routinely collected data can be challenging due to how the data are collected and recorded. In this study, we aimed to define and validate a scalable framework model to measure outcomes of migraine treatment and prevention by use of artificial intelligence (AI) algorithms within EHR data.

METHODS

Headache specialists defined descriptive features based on routinely collected clinical data. Data elements were weighted to define a 10-point scale encompassing headache severity (1-7 points) and associated features (0-3 points). A test data set was identified, and a reference standard was manually produced by trained annotators. Automation (i.e., AI) was used to extract features from the unstructured data of patient encounters and compared to the reference standard. A threshold of 70% close agreement (within 1 point) between the automated score and the human annotator was considered to be a sufficient extraction accuracy. The accuracy of AI in identifying features used to construct the outcome model was also evaluated and success was defined as achieving an F1 score (i.e., the weighted harmonic mean of the precision and recall) of 80% in identifying encounters.

RESULTS

Using data from 2,006 encounters, 11 features were identified and included in the model; the average F1 scores for automated extraction were 92.0% for AI applied to unstructured data. The outcome model had excellent accuracy in characterizing migraine status with an exact match for 77.2% of encounters and a close match (within 1 point) for 82.2%, compared with manual extraction scores-well above the 70% match threshold set prior to the study.

CONCLUSION

Our findings indicate the feasibility of technology-enabled models for validated determination of soft outcomes such as migraine progression using the data elements typically captured in the real-world clinical setting, providing a scalable approach to credible EHR-based clinical studies.

Using real-world evidence in healthcare from Western to Central and Eastern Europe: a review of existing barriers

As part of the HTx (Next Generation Health Technology Assessment) project, this study was aimed at identifying the main barriers for application of real-world evidence (RWE) for the purposes of health technology assessment in the Central and Eastern European countries. A mixed methods approach was employed to identify the main barriers: a scoping review of the literature and a series of discussions with stakeholders. Based on the applied approaches, we attempted to summarize the main barriers and challenges related to transferability of RWE in five main groups: technical, regulatory, clinical, scientific and perceptional barriers. Further research should pursue the development of detailed, consensus-based guidelines to improve the harmonization and standardization of RWE.

Analysis of a Real-World Progression Variable and Related Endpoints for Patients with Five Different Cancer Types

Introduction

We previously demonstrated that real-world progression (rwP) can be ascertained from unstructured electronic health record (EHR)-derived documents using a novel abstraction approach for patients with advanced non-small cell lung cancer (base case). The objective of this methodological study was to assess the reliability, clinical relevance, and the need for disease-specific adjustments of this abstraction approach in five additional solid tumor types.

Methods

Patients with metastatic breast cancer (mBC), advanced melanoma (aMel), small cell lung cancer (SCLC), metastatic renal cell carcinoma (mRCC), and advanced gastric/esophageal cancer (aGEC) were selected from a real-world database. Disease-specific additions to the base case were implemented as needed. The resulting abstraction approach was applied to each disease cohort to capture rwP events and dates. To provide comprehensive clinical context, real-world progression-free survival (rwPFS) and time to progression (rwTTP) were compared to real-world overall survival (rwOS), time to next treatment (rwTTNT), and time to treatment discontinuation (rwTTD). Endpoint estimates were assessed using the Kaplan–Meier method. Correlations between real-world endpoints and rwOS were calculated using Spearman’s ρ.

Results

Additions to the base-case rwP abstraction approach were required for mBC, aMel, and SCLC. Inter-abstractor agreement for rwP occurrence, irrespective of date, ranged from 88% to 97%. Occurrence of clinically relevant downstream events (new antineoplastic systemic therapy start, antineoplastic systemic therapy end, or death relative to the rwP event) ranged from 59% (aMel) to 72% (mBC). Median rwPFS ranged from 3.7 (aMel) to 7.7 (mBC) months, and median rwTTP ranged from 4.6 (aMel) to 8.3 (mRCC) months. Correlations between rwOS and rwPFS ranged from 0.52 (aMel) to 0.82 (SCLC). The correlation between rwOS and rwTTD was often lower relative to other comparisons (range 0.40–0.62).

Conclusion

Derivation of a rwP variable from EHR documentation is feasible and reliable across the five solid tumors. Endpoint analyses show that rwP produces clinically meaningful information.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12325-022-02091-8.

Dimethyl Fumarate in Multiple Sclerosis Patients: A Real World Study

Amaç: Çalışmamızda dimetil fumarat (DMF) kullanan ya da daha önce kullanmış olan ataklı multipl skleroz (MS) hastalarında etkinlik ve güvenlik verilerinin elde edilmesi amaçlandı. Yöntem: Çalışma gözlemsel retrospektif kohort çalışması olarak tasarlandı. İstanbul Tıp Fakültesi Nöroloji Ana Bilim Dalında takip edilen ve 2014–2021 yılları arasında DMF kullanmış ataklı yineleyici (RRMS) ve ataklı progresif (RPMS) MS hastalarının verileri incelendi. Hastalarda DMF başlandıktan sonraki ilk genişletilmiş özürlülük durum ölçeği (EDSS) kötüleşmesi, ilk atak ve ilk yeni MRG lezyonu bilgileri elde edilerek hastalık aktivitesi olmaması durumu (NEDA-3) belirlendi. Bulgular: Toplam 126 ataklı MS hastası (85 kadın, 41 erkek; 116 RRMS, 10 RPMS) çalışmaya dahil edildi. DMF öncesi başka bir tedavi kullanmış olan, ataklı progresif olan ve DMF öncesi hastalık süresi daha uzun olan hastalarda DMF daha yüksek oranda kesilmişti (p=0,002, p=0,03 ve p=0,002). Beyin MRG bilgisi olan 65 RRMS ve RPMS tanılı hastanın %25’inde DMF başlandıktan sonraki takip döneminde NEDA-3 durumu sağlandığı görüldü. DMF öncesinde EDSS skorunun 3’ün altında olması NEDA-3 durumunun yitimine, ilk atağa ve ilk özürlülük birikiminin olmasına kadar geçen süreleri anlamlı ölçüde uzattı (NEDA-3 için p=0,008, HR=1,95). Sonuç: Çalışmamızda elde edilen gerçek yaşam verileri güncel bilgiler ile büyük oranda benzerlik göstermektedir. Elde ettiğimiz bulgular EDSS skoru 3’ün altında olan hastalarda başlanan DMF’nin atak, yeni beyin MRG lezyonu ve engellilik birikimi gelişme riskinin azalttığına işaret etmektedir.

Continuous Ketone Monitoring Consensus Report 2021

This article is the work product of the Continuous Ketone Monitoring Consensus Panel, which was organized by Diabetes Technology Society and met virtually on April 20, 2021. The panel consisted of 20 US-based experts in the use of diabetes technology, representing adult endocrinology, pediatric endocrinology, advanced practice nursing, diabetes care and education, clinical chemistry, and bioengineering. The panelists were from universities, hospitals, freestanding research institutes, government, and private practice. Panelists reviewed the medical literature pertaining to ten topics: (1) physiology of ketone production, (2) measurement of ketones, (3) performance of the first continuous ketone monitor (CKM) reported to be used in human trials, (4) demographics and epidemiology of diabetic ketoacidosis (DKA), (5) atypical hyperketonemia, (6) prevention of DKA, (7) non-DKA states of fasting ketonemia and ketonuria, (8) potential integration of CKMs with pumps and automated insulin delivery systems to prevent DKA, (9) clinical trials of CKMs, and (10) the future of CKMs. The panelists summarized the medical literature for each of the ten topics in this report. They also developed 30 conclusions (amounting to three conclusions for each topic) about CKMs and voted unanimously to adopt the 30 conclusions. This report is intended to support the development of safe and effective continuous ketone monitoring and to apply this technology in ways that will benefit people with diabetes.

Precision rehabilitation: optimizing function, adding value to health care

Precision medicine efforts are underway in many medical disciplines; however, the power of precision rehabilitation has not yet been explored. Precision medicine aims to deliver the right intervention, at the right time, in the right setting, for the right person, ultimately, bolstering the value of the care that we provide. To date precision medicine efforts have rarely focused on function at the level of a person, but precision rehabilitation is poised to change this and bring the focus on function to the broader precision medicine enterprise. To do this, subgroups of individuals must be identified based on their level of function via precise measurement of their abilities in the physical, cognitive, and psychosocial domains. Adoption of electronic health records, advances in data storage and analytics, and improved measurement technology make this shift possible. Here we detail critical components of the precision rehabilitation framework, including 1) the synergistic use of various study designs, 2) the need for standardized functional measurements, 3) the importance of precise and longitudinal measures of function, 4) the utility of comprehensive databases, 5) the importance of predictive analyses, and 6) the need for system and team science. Precision rehabilitation has the potential to revolutionize clinical care, optimize function for all individuals, and magnify the value of rehabilitation in healthcare; however, to reap the benefits of precision rehabilitation, the rehabilitation community must actively pursue this shift.

Minimizing bias in massive multi-arm observational studies with BCAUS: balancing covariates automatically using supervision

BACKGROUND

Observational studies are increasingly being used to provide supplementary evidence in addition to Randomized Control Trials (RCTs) because they provide a scale and diversity of participants and outcomes that would be infeasible in an RCT. Additionally, they more closely reflect the settings in which the studied interventions will be applied in the future. Well-established propensity-score-based methods exist to overcome the challenges of working with observational data to estimate causal effects. These methods also provide quality assurance diagnostics to evaluate the degree to which bias has been removed and the estimates can be trusted. In large medical datasets it is common to find the same underlying health condition being treated with a variety of distinct drugs or drug combinations. Conventional methods require a manual iterative workflow, making them scale poorly to studies with many intervention arms. In such situations, automated causal inference methods that are compatible with traditional propensity-score-based workflows are highly desirable.

METHODS

We introduce an automated causal inference method BCAUS, that features a deep-neural-network-based propensity model that is trained with a loss which penalizes both the incorrect prediction of the assigned treatment as well as the degree of imbalance between the inverse probability weighted covariates. The network is trained end-to-end by dynamically adjusting the loss term for each training batch such that the relative contributions from the two loss components are held fixed. Trained BCAUS models can be used in conjunction with traditional propensity-score-based methods to estimate causal treatment effects.

RESULTS

We tested BCAUS on the semi-synthetic Infant Health & Development Program dataset with a single intervention arm, and a real-world observational study of diabetes interventions with over 100,000 individuals spread across more than a hundred intervention arms. When compared against other recently proposed automated causal inference methods, BCAUS had competitive accuracy for estimating synthetic treatment effects and provided highly concordant estimates on the real-world dataset but was an order-of-magnitude faster.

CONCLUSIONS

BCAUS is directly compatible with trusted protocols to estimate treatment effects and diagnose the quality of those estimates, while making the established approaches automatically scalable to an arbitrary number of simultaneous intervention arms without any need for manual iteration.

Philadelphia-positive B-lymphoblastic leukemia in a middle-income country - A real-world multicenter cohort

Outside of clinical trials, few studies have addressed the outcomes of Ph+ acute lymphoblastic leukemia (ALL) in adults, especially from developing world. In this study, we conducted a multicenter analysis on the outcomes of patients aged > 15 years with Ph+ ALL, aiming to get to know an overview of the Brazilian experience as well as to explore baseline factors associated with relapse and mortality in our setting. Over these 10 years, patients were treated with diverse protocols, all of them always combined with a frontline tyrosine-kinase inhibitor. A total of 123 Ph+ ALL patients was included. Imatinib was the first line TKI in 97 %. The complete response rate was 79 %. The early death rate was 15 %, being associated with increasing age at diagnosis (p = 0.06). The use of intensive versus attenuated induction regimen was not associated with higher induction mortality (p = 0.99). Overall, 29 % of patients aged ≤ 60 years underwent allogeneic transplantation, 87 % in first CR. 4-year overall survival (OS) and relapse-free survival were 25 % and 24 %, respectively. The incidence of relapse (death as a competitor) was 29 %, while the non-relapse mortality was 42 %. Only age was independently associated with OS, and lactate dehydrogenase level and central nervous disease at diagnosis were related to relapse in our cohort. This is the first historical cohort multicenter study on Ph+ ALL from Brazil. Reporting these outcomes is essential to encourage public policies to expand access to new drugs and transplantation in middle-income countries.

Demographic Characteristics and Treatment Patterns Among Patients Receiving Palbociclib for HR+/HER2- Advanced Breast Cancer: A Nationwide Real-World Experience

Background

This nationwide retrospective study reports data on the real-world use of the selective cyclin-dependent kinase 4 and 6 (CDK4/6) inhibitor palbociclib in a large population of advanced breast cancer (ABC) patients during a 2-year period in Hungary.

Methods

All patients with ABC who received palbociclib between May 1, 2017 and June 30, 2019 were included in the analysis. Patient demographic and clinical characteristics, disease-related factors and treatment patterns were examined during the early access program and in the regular reimbursement period.

Results

Altogether, 962 patients were included (mean age: 60.6 years). A total of 399 patients (41%) were treated with palbociclib plus aromatase inhibitors (P+AI), and 563 patients (59%) received palbociclib and fulvestrant (P+F). The most commonly prescribed AI was letrozole (n=359; 90%). Of those with metastatic disease (n=733; 76%), 241 patients (33%) had visceral metastases and 449 (61%) had bone-only disease. The majority of patients (79%) received palbociclib as first- or second-line therapy for ABC. The starting dose of palbociclib was 125 mg in 98% of patients; dose reductions were required in 32% of patients receiving P+AI and 31% of those treated with P+F. At the time of data collection, palbociclib therapy was ongoing in 270 patients (68%) in the P+AI group and 245 patients (44%) in the P+F group.

Conclusions

This nationwide analysis is the first to provide insights into the real-world use of palbociclib in a large patient population from a Central-Eastern European country. The findings confirm the good tolerability of palbociclib with similar dose reduction rates to those reported from registration trials.

The Benefit of Insulin Degludec/Liraglutide (IDegLira) Compared With Basal-Bolus Insulin Therapy is Consistent Across Participant Subgroups With Type 2 Diabetes in the DUAL VII Randomized Trial

BACKGROUND

Insulin degludec/liraglutide (IDegLira) results in glycated hemoglobin (HbA1c) levels comparable with basal-bolus (BB) therapy. Here, we assessed the effect of once-daily IDegLira compared with BB (once-daily insulin glargine 100 U/mL and insulin aspart ≤4 times/day) across subgroups with varying characteristics.

MATERIALS AND METHODS

DUAL VII trial participants (type 2 diabetes [T2D], HbA1c 53-86 mmol/mol [7.0%-10.0%]) were subgrouped post hoc based on the following baseline characteristics: HbA1c (≤58.5, >58.5 to ≤69.4, and >69.4 mmol/mol; ≤7.5%, >7.5 to ≤8.5%, and >8.5%), body mass index (<30, ≥30 to <35, and ≥35 kg/m²), age (18 to <65 and ≥65 years), duration of diabetes (≥0 to 10 and ≥10 years), total pretrial daily basal insulin dose (20 to <30, ≥30 to <40, and ≥40 to ≤50 U), and fasting plasma glucose (<7.2 mmol/L/<130 mg/dL and ≥7.2 mmol/L/≥130 mg/dL).

RESULTS

Compared with BB, and in all subgroups, IDegLira treatment consistently gave similar HbA1c reductions, less severe or blood glucose-confirmed hypoglycemia, lower end-of-trial (EOT) total daily insulin dose, and weight loss. In all subgroups, mean EOT HbA1c was ≤53 mmol/mol (≤7.0%). The greatest HbA1c reduction occurred in the highest baseline HbA1c subgroup. Overall, mean EOT daily insulin dose was 0.43 to 0.52 U/kg with IDegLira and 0.74 to 1.07 U/kg with BB. More participants achieved the triple composite endpoint (HbA1c <53 mmol/mol [<7.0%] without weight gain or hypoglycemia) with IDegLira vs BB across the baseline HbA1c subgroups (≤58.5 mmol/mol [44.6% vs 7.0%], >58.5 to ≤69.4 mmol/mol [41.1% vs 8.3%], and >69.4 mmol/mol [23.8% vs 3.4%]).

CONCLUSION

These results support initiating IDegLira in patients with varying baseline characteristics and uncontrolled T2D on basal insulin.

CLINICALTRIALS.GOV REGISTRATION

NCT02420262.

Generating real-world evidence from unstructured clinical notes to examine clinical utility of genetic tests: use case in BRCAness

Background

Next-generation sequencing provides comprehensive information about individuals’ genetic makeup and is commonplace in oncology clinical practice. However, the utility of genetic information in the clinical decision-making process has not been examined extensively from a real-world, data-driven perspective. Through mining real-world data (RWD) from clinical notes, we could extract patients’ genetic information and further associate treatment decisions with genetic information.

Methods

We proposed a real-world evidence (RWE) study framework that incorporates context-based natural language processing (NLP) methods and data quality examination before final association analysis. The framework was demonstrated in a Foundation-tested women cancer cohort (N = 196). Upon retrieval of patients’ genetic information using NLP system, we assessed the completeness of genetic data captured in unstructured clinical notes according to a genetic data-model. We examined the distribution of different topics regarding BRCA1/2 throughout patients’ treatment process, and then analyzed the association between BRCA1/2 mutation status and the discussion/prescription of targeted therapy.

Results

We identified seven topics in the clinical context of genetic mentions including: Information, Evaluation, Insurance, Order, Negative, Positive, and Variants of unknown significance. Our rule-based system achieved a precision of 0.87, recall of 0.93 and F-measure of 0.91. Our machine learning system achieved a precision of 0.901, recall of 0.899 and F-measure of 0.9 for four-topic classification and a precision of 0.833, recall of 0.823 and F-measure of 0.82 for seven-topic classification. We found in result-containing sentences, the capture of BRCA1/2 mutation information was 75%, but detailed variant information (e.g. variant types) is largely missing. Using cleaned RWD, significant associations were found between BRCA1/2 positive mutation and targeted therapies.

Conclusions

In conclusion, we demonstrated a framework to generate RWE using RWD from different clinical sources. Rule-based NLP system achieved the best performance for resolving contextual variability when extracting RWD from unstructured clinical notes. Data quality issues such as incompleteness and discrepancies exist thus manual data cleaning is needed before further analysis can be performed. Finally, we were able to use cleaned RWD to evaluate the real-world utility of genetic information to initiate a prescription of targeted therapy.

The Ethics of Emergent Health Technologies: Implications of the 21st Century Cures Act for Nursing

The 21st Century Cures Act, passed in December 2016 by the United States Congress, is a public law aimed at accelerating the time it takes to get pharmaceutical drugs and medical devices into the market, in addition to shifting connected review processes from randomized controlled trials to real-world efficacy tests. As of December 2019, efforts are underway to introduce a "Cures Act 2.0" bill, with particular attention to the implementation of digital health within health systems. Research on the development of emergent health technologies is nascent; research examining health technology implications of 21st Century Cures Act for the health care workforce is nonexistent. This article fills a crucial gap in public awareness, discussing ethical implications of the 21st Century Cures Act and centering nursing. Nursing is a profession frequently acknowledged as practicing on "the front lines of care" and frequently responsible for the trialing of products in clinical settings. The article summarizes and evaluates key components of the 21st Century Cures Act related to health technology development. Discrete health technologies addressed are (a) breakthrough devices, (b) digital health software, and (c) combination products. It then connects these provisions to ethical considerations for nursing practice, research, and policy. The article concludes by discussing the relevance of emerging digital health technologies to the crafting of a "Cures 2.0" bill, with particular attention to this moment in light of digital care precedents set during the COVID-19 pandemic.

Economic Evaluations Informed Exclusively by Real World Data: A Systematic Review

Economic evaluations using Real World Data (RWD) has been increasing in the very recent years, however, this source of information has several advantages and limitations. The aim of this review was to assess the quality of full economic evaluations (EE) developed using RWD. A systematic review was carried out through articles from the following databases: PubMed, Embase, Web of Science and Centre for Reviews and Dissemination. Included were studies that employed RWD for both costs and effectiveness. Methodological quality of the studies was assessed using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist. Of the 14,011 studies identified, 93 were included. Roughly half of the studies were carried out in a hospital setting. The most frequently assessed illnesses were neoplasms while the most evaluated interventions were pharmacological. The main source of costs and effects of RWD were information systems. The most frequent clinical outcome was survival. Some 47% of studies met at least 80% of CHEERS criteria. Studies were conducted with samples of 100-1000 patients or more, were randomized, and those that reported bias controls were those that fulfilled most CHEERS criteria. In conclusion, fewer than half the studies met 80% of the CHEERS checklist criteria.

Generalizing findings from a randomized controlled trial to a real-world study of the iLookOut, an online education program to improve early childhood care and education providers' knowledge and attitudes about reporting child maltreatment

In recent years, real-world studies (RWS) are gaining increasing interests, because they can generate more realistic and generalizable results than randomized controlled clinical trials (RCT). In 2017, we published a RCT in 741 early childhood care and education providers (CCPs). It is the Phase I of our iLookOut for Child Abuse project (iLookOut), an online, interactive learning module about reporting suspected child maltreatment. That study demonstrated that in a RCT setting, the iLookOut is efficient at improving CCPs’ knowledge of and attitudes towards child maltreatment reporting. However, the generalizability of that RCT’s results in a RWS setting remains unknown. To address this question, we design and conduct this large RWS in 11,065 CCPs, which is the Phase II of the iLookOut. We hypothesize replication of the earlier RCT findings, i.e., the iLookOut can improve CCPs’ knowledge of and attitudes toward child maltreatment reporting in a real world setting. In addition, this RWS also explores whether demographic factors affect CCPs’ performance. Results of this RWS confirmed the generalizability of the previous RCT’s results in a real world setting. It yielded similar effect sizes for knowledge and attitudes as were found in the earlier RCT. Cohen’s d for knowledge improvement was 0.95 in that RCT, 0.96 in this RWS; Cohen’s d for attitude improvement was 0.98 in that RCT, 0.80 in this RWS. Also, we found several significant differences in knowledge and attitude improvement with regard to age, race, education, and employment status. In conclusion, iLookOut improves knowledge and attitudes of CCPs about child maltreatment prevention and reporting in a real-world setting. The generalizability of the initial RCT findings to this RWS provides strong evidence that the iLookout will be effective in other real world settings. It can be a useful model for other interventions aimed at preventing child maltreatment.

Clinical trial registration for the original RCT: NCT02225301 (ClinicalTrials.gov Identifier)