Regulatory Considerations on the use of Machine Learning based tools in Clinical Trials

Background

The widespread increasing use of machine learning (ML) based tools in clinical trials (CTs) impacts the activities of Regulatory Agencies (RAs) that evaluate the development of investigational medicinal products (IMPs) in clinical studies to be carried out through the use of data-driven technologies. The fast progress in this field poses the need to define new approaches and methods to support an agile and structured assessment process.

Method

The assessment of key information, characteristics and challenges deriving from the application of ML tools in CTs and their link with the principles for a trustworthy artificial intelligence (AI) that directly affect the decision-making process is investigated.

Results

Potential issues are identified during the assessment and areas of greater interaction combining key regulatory points and principles for a trustworthy AI are highlighted. The most impacted areas are those related to technical robustness and safety of the ML tool, in relation to data used and the level of evidence generated. Additional areas of attention emerged, like the ones related to data and algorithm transparency.

Conclusion

We evaluate the applicability of a new method to further support the assessment of medicinal products developed using data-driven tools in a CT setting. This is a first step and new paradigms should be adopted to support policy makers and regulatory decisions, capitalizing on technology advancements, considering stakeholders’ feedback and still ensuring a regulatory framework on safety and efficacy.

Graphical Abstract

Clinical Trials and Machine Learning: Regulatory Approach Review

Machine Learning, a fast-growing technology, is an application of Artificial Intelligence that has significantly contributed to drug discovery and clinical development. In the last few years, the number of clinical applications based on Machine Learning has constantly been growing. Moreover, it is now also impacting National Competent Authorities during the assessment of most recently submitted Clinical Trials that are designed, managed, or generating data deriving from the use of Machine Learning or Artificial Intelligence technologies. We review current information available on the regulatory approach to Clinical Trials and Machine Learning. We also provide inputs for further reasoning and potential indications, including six actionable proposals for regulators to proactively drive the upcoming evolution of Clinical Trials within a strong regulatory framework, focusing on patient safety, health protection, and fostering immediate access to effective treatments.

T20QoL: an observational multicenter cohort study to evaluate the quality of life in HIV-patients treated with enfuvirtide (ENF, T-20) in combination with an optimized background therapy

AIM

To evaluate the impact of health-related quality of life (HRQoL) enfuvirtide-based (ENF-based) salvage regimens of treatment-experienced HIV patients, in an observational multicenter cohort study.

METHODS

HRQoL was measured in a cohort of 16 patients over a 6-month follow-up using 2 instruments: the ISSQoL (Istituto Superiore di Sanità Quality of Life), a recently validated HIV-specific questionnaire; the EQ-5D (EuroQol), a generic widely used instrument. ENF was given at standard dosage along with an optimized background regimen.

RESULTS

Most of HRQoL dimensions showed improvement in ENF-treated patients at the post-baseline time points. Social functioning was the only dimension showing a negative effect. Monthly care costs of antiretroviral drugs for HIV patients taking ENF plus an optimized background regimen were approximately euro2,348 per patient-month (range euro382-euro2,940).

CONCLUSION

Our results show that the addition of ENF to an optimized background salvage-HAART may positively affect HRQoL not only in clinical trials but also in a sample population of patients used in a routine clinical practice.

Cost of surgical intervention for reconstructive therapy of HIV-associated facial lipoatrophy

This study aims to assess direct cost of reconstructive interventions with facial fillers for treatment of HIV (human immunodeficiency virus)-associated facial lipoatrophy (FLA). Evaluation was performed on data from patients enrolled in one arm of a comparative study of immediate versus delayed reconstructive treatment of facial lipoatrophy. Median costs were standardized for efficacy, estimated using data reported by physicians and patient reported outcomes. The variations of the results were evaluated with a sensitivity analysis. Evaluation was performed on 66 patients characterized by significant differences in terms of severity of FLA. Total cost resulted of €140,416.15, with a median cost per patient of €2126.04 (interquartile range [IQR]: 1599-2822). Taking into consideration severity of disease, median costs were €1641.67 (IQR: 1326.67-2126.04) and 2557.12 (IQR: 1939.34-2872.04) (P = 0.0) respectively for patients with low and high severity scores at baseline. Significant differences in term of cost-effectiveness ratios were also found between patients with different severity of FLA, and sensitivity analysis showed that these ratios increase with higher severity scores at baseline and vary widely depending on the costs of filler. Although these results cannot be considered representative because of important limitations, the present study suggests the severity of disease as an important determinant of costs.

A factor analytic study of the Italian National Institute of Health Quality of Life - Core Evaluation Form (ISSQoL-CEF)

OBJECTIVES

The Italian National Institute of Health Quality of Life - Core Evaluation Form (ISSQoL-CEF) is a specific questionnaire measuring health-related quality of life for human immunodeficiency virus-infected people in the era of highly active antiretroviral therapy. The main goal of this study was to examine the construct validity of this questionnaire by confirmation of its hypothesized dimensional structure.

METHODS

Baseline quality of life data from four clinical studies were collected and a confirmatory factor analysis of the ISSQoL-CEF items was carried out. Both first-order and second-order factor models were tested: Model 1 with nine correlated first-order factors; Model 2 with three correlated second-order factors (Physical, Mental, and Social Health); Model 3 with two correlated second-order factors (Physical and Mental/Social Health); Model 4 with only one second-order factor (General Health).

RESULTS

A total of 261 patients were surveyed. Model 1 had a good fit to the data. Model 2 had an acceptable fit to the data and it was the best of all hierarchical models. However, Model 2 fitted the data worse than Model 1.

CONCLUSIONS

The findings of in this study, consistent with the results of previous study, pointed out the construct validity of the ISSQoL-CEF.

Immediate versus delayed surgical intervention for reconstructive therapy of HIV-associated facial lipoatrophy: a randomized open-label study

We assessed the safety and efficacy of reconstructive therapy with facial fillers for the treatment of HIV-associated facial lipoatrophy (FLA) through a randomized, controlled, open-label single-center study. A total of 134 HIV-infected patients with severe FLA were randomly assigned to receive immediate (67 patients) or delayed (67 patients) facial injections of poly-l-lactic acid (PLA) or polyacrylamide gel (PAIG). Outcome measures included changes in physician and patient FLA severity scale, adverse events, and changes in health-related quality of life (HRQoL) and anxiety using validated measures. The mean average study follow-up was 27 weeks for the immediate and 25 weeks for the delayed subjects. Adverse events were mild and resolved after a mean of 4 days. Compared to patients randomized to the delayed treatment group, patients assigned to the immediate treatment group had significantly lower physician-rated (0.0 versus -3.0; p < 0.0001) and patient-rated (0.1 versus -1.8; p < 0.0001) FLA severity scores. By contrast, measures exploring HRQoL and anxiety did not show any significant difference between patients randomized to the immediate and deferred groups. Reconstructive therapy with facial fillers was effective and safe and led to significant improvements in FLA severity. However, no significant gains in HRQoL, relational and psychological consequences of body changes, and anxiety-related concerns were observed. Studies should be performed to identify patients who could maximally benefit from filling interventions for FLA.

Health-related quality of life outcomes in HIV-infected patients starting different combination regimens in a randomized multinational trial: the INITIO-QoL substudy

The health-related quality of life (HRQoL) outcomes in HIV-infected, treatment-naive patients starting different HAART regimens in a 3-year, randomized, multinational trial were compared. HRQoL was measured in a subgroup of patients enrolled in the INITIO study (153/911), using a modified version of the MOS-HIV questionnaire. The regimens compared in the INITIO trial were composed by two NRTIs (didanosine + stavudine) plus either an NNRTI (efavirenz) or a PI (nelfinavir), or both (efavirenz + nelfinavir). Primary HRQoL outcomes were Physical and Mental Health Summary scores (PHS and MHS, respectively). During follow-up, an increase of PHS score was observed in all treatment arms. The MHS score remained substantially unchanged with the four-drug combination and showed with both NNRTI- and PI-based three-drug regimens a marked trend toward improvement, which became statistically significant when a multiple imputation method was used to adjust for missing data. Overall, starting all the combination regimens compared in the INITIO study was associated with a maintained or slightly improved HRQOL status, consistently with the positive immunological and virological changes observed in the main study. The observed differences in the MHS indicate a possible HRQoL benefit associated to the use of three-drug, two-class regimens and no additional benefit for the use of four-drug, three-class regimens, confirming that three-drug, two-class regimens that include two NRTIs plus either an NNRTI or a PI should be preferred as initial treatment of HIV infection.

ISSQoL: A New Questionnaire for Evaluating the Quality of Life of People Living with HIV in the HAART Era*

OBJECTIVE

To design a Health-related Quality of Life (HRQoL) instrument for HIV-infected people in the era of highly active antiretroviral therapy (HAART).

METHODS

The self-administered questionnaire was developed by an Italian network including researchers, physicians, people living with HIV, national institutions and community-based organizations (CBO) through several steps: (1) review of existing HRQoL literature and questionnaires for HIV-infected people; (2) selection of relevant domains measuring HRQoL in HIV-infected people, and identification of new domains related to new aspects of HRQoL concerning HAART-treated individuals; (3) conduction of two pre-test analyses in independent groups of Italian HIV-positive people (n approximately =100) distributed throughout the country. The objectives of the first pre-test were to verify the usefulness of the questionnaire, to construct a form easily understandable by everyone, to define the domains and their significance; the second pre-test aimed at evaluating and reshaping the questionnaire based on a statistical analysis of the outcomes of first pre-test; (4) validation analysis. A large cohort of people with HIV infection was recruited for the last step.

RESULTS

The internal consistence reliability (Cronbach's alpha) was >or=0.70 for all domains. Most domains had Cronbach's coefficient >0.80. All domains demonstrated convergent and discriminant validity. The final version of ISSQoL includes two sections: HRQoL Core Evaluation Form (9 domains) and Additional Important Areas for HRQoL (6 domains). The ISSQoL was administered together with two additional forms: a Daily Impact of Symptoms Form and a Demographic Information Form. The Additional Important Areas for HRQoL include social support, interaction with medical staff, treatment impact, body changes, life planning, and motherhood/fatherhood.

CONCLUSION

The data reported in the present paper provide preliminary evidence of the reliability and validity of the ISSQoL questionnaire for the measurement of HRQoL in HIV-infected people. The direct involvement of HIV-positive people in all the phases of the project was a key aspect of our work.

[Prevalence of the use of old and new drugs among new entrants in Italian prisons]

An epidemiological survey was conducted to evaluate the prevalence of drug use among new entrants in Italian prisons. Overall, 1267 inmates were recruited by 9 prisons. Drug use before imprisonment was reported by 55.8% of the participants. Cocaine was the most commonly used drug (42%), followed by heroin (34%), marijuana/cannabis (33%), ecstasy (7%), hallucinogens (6%), amphetamines (5%); more than one drug was reported by 68% of abusers. Recent use (1 month before imprisonment) was admitted by up to 27% of inmates. Alcohol or tobacco use was reported by 38 and 77% of the inmates, respectively. Our findings indicate that a high proportion of inmates has ever used drugs; adequate intervention is needed to reduce the risk of addictive behaviour in this population group.

Quality of life outcomes of combination zidovudine- didanosine-nevirapine and zidovudine-didanosine for antiretroviral-naive advanced HIV-infected patients

OBJECTIVES

To evaluate the quality of life outcomes in antiretroviral-naive patients randomized to zidovudine plus didanosine versus zidovudine plus didanosine plus nevirapine for treatment of advanced HIV disease (the Istituto Superiore di Sanità 047 trial).

DESIGN

A 48-week randomized, double-blind trial.

METHODS

Sixty patients were enrolled and evaluated over 24 weeks. Quality of life was assessed using a modified version of the Medical Outcomes Study-HIV Health Survey. For analysis, we calculated two summary scores reflecting the physical (PHS) and the mental (MHS) components of health.

RESULTS

Although the three-drug combination was superior at inducing immunologic and virologic responses, the two-drug regimen was superior for both PHS and MHS, especially at week 8 where differences were both statistically and clinically significant (5.8 and 9.2 points, respectively, P< 0.02 for both). Quality of life changes paralleled trends in body weight and Karnofsky performance status score.

CONCLUSION

Although a three-drug antiretroviral therapy regimen was superior in terms of short term virologic/immunologic response, the two-drug regimen was better in terms of quality of life. In general, triple therapy remains the most effective treatment option. However, quality of life assessments can yield results that may be discordant with and complementary to those obtained using conventional endpoints. Comparative trials should collect a comprehensive range of outcome measures, including patient-reported quality of life, in order to provide clinicians and patients with additional information that may influence treatment decisions.

Hospitalizations and costs of treatment for protease inhibitor-based regimens in patients with very advanced HIV-infection (CD4 < 50/mm(3))

PURPOSE

To describe the cost of hospitalization and treatment in patients with very advanced disease who tart different regimens based on a protease inhibitor (PI).

METHOD

An observational retrospective analysis was performed on data from a 48-week randomized, multicenter study. Analysis was based on a subgroup of centers that were geographically defined. Costs of ordinary hospital admissions and of antiretroviral treatment were considered. Incidence of hospitalization and number of days free from hospitalization during the period of observation were calculated. Cost and hospitalization measures were compared among patients receiving three different therapeutic regimens: only PI, PI plus one nucleoside, or PI plus two nucleosides. A multivariate analysis was used to assess cost differences, controlling for variables potentially able to influence outcome.

RESULTS

Overall, among 166 patients starting PI (PI plus two nucleosides, 71;PI plus one nucleoside, 65; only PI, 30), 162 ordinary hospital admissions were observed during about 1 year of follow-up. Monthly rates of admission per person and incidence of first hospitalization on 100 person-months showed a clear inverse relationship with the number of drugs comprising the baseline treatment regimen, with the lower rates for the triple therapy group (0.06 and 3.9, respectively), intermediate values for the dual therapy group (0.10 and 8.1, respectively), and higher rates for the PI monotherapy group (0.15 and 13.7, respectively). The average number of days free from hospitalization per month was 29.5 in the triple therapy group, 28.6 in the dual therapy group, and 27.9 in the monotherapy group. The results of cost analysis showed, despite higher cost of antiretroviral treatment, that global costs were progressively lower using regimens of increasing potency: Compared to PI monotherapy, global cost (costs of antiretroviral treatment and of hospitalizations combined) per month per patient was 31.9% lower for the triple therapy group and 19.3% lower for the dual therapy. Global cost for the triple therapy was 15.7% lower compared to global cost for dual therapy. After adjustment for CD4 count, AIDS status, and Karnofsky score, both hospitalization costs and global costs were significantly lower for triple therapy compared to monotherapy (p =.002 and.039, respectively).

CONCLUSION

In advanced and nucleoside-experienced patients, PI-containing regimens have a differential impact according to the overall strength of the regimen, with the best effects on both hospitalizations and treatment costs obtained using PI within potent combination regimens.