1
|
Dennstädt F, Zink J, Putora PM, Hastings J, Cihoric N. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Syst Rev 2024; 13:158. [PMID: 38879534 PMCID: PMC11180407 DOI: 10.1186/s13643-024-02575-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/17/2023] [Accepted: 05/30/2024] [Indexed: 06/19/2024] Open
Abstract
BACKGROUND Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.
Collapse
Affiliation(s)
- Fabio Dennstädt
- Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland.
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland.
| | - Johannes Zink
- Institute for Computer Science, University of Würzburg, Würzburg, Germany
| | - Paul Martin Putora
- Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| | - Janna Hastings
- Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland
- School of Medicine, University of St. Gallen, St. Gallen, Switzerland
- Swiss Institute of Bioinformatics, Lausanne, Switzerland
| | - Nikola Cihoric
- Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland
| |
Collapse
|
2
|
Meijboom RW, Gardarsdottir H, Becker ML, Movig KLL, Kuijvenhoven J, Egberts TCG, Giezen TJ. Discontinuation of infliximab treatment in patients with inflammatory bowel disease who retransitioned to originator and those who remained on biosimilar. Therap Adv Gastroenterol 2023; 16:17562848231197923. [PMID: 37706094 PMCID: PMC10496466 DOI: 10.1177/17562848231197923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Accepted: 08/12/2023] [Indexed: 09/15/2023] Open
Abstract
Background Many patients with inflammatory bowel disease (IBD) have transitioned from an infliximab originator to a biosimilar. However, some patients retransition to the originator (i.e. stop biosimilar and reinitiate the originator). Whether this sign of potential unsatisfactory treatment response is specifically related to the infliximab biosimilar or the patient and/or the disease including patients' beliefs on the biosimilar is unclear. Objectives We aimed to compare the risk of and reasons for infliximab discontinuation between retransitioned patients and those remaining on biosimilar. Design Non-interventional, multicentre cohort study. Methods IBD patients who transitioned from infliximab originator to biosimilar between January 2015 and September 2019 in two Dutch hospitals were eligible for this study. Retransitioned patients (retransitioning cohort) were matched with patients remaining on biosimilar (biosimilar remainder cohort). Reasons for discontinuation were categorised as the unwanted response (i.e. loss of effect or adverse events) or remission. Risk of unwanted discontinuation was compared using Cox proportional hazards models. Results Patients in the retransitioning cohort (n = 44) were younger (median age 39.9 versus 44.0 years), more often female (65.9% versus 48.9%) and had shorter dosing intervals (median 48.5 versus 56.0 days) than in the biosimilar remainder cohort (n = 127). Infliximab discontinuation due to unwanted response was 22.7% in the retransitioning and 13.4% in the biosimilar remainder cohort, and due to remission was 2.3% and 9.4%, respectively. Retransitioned patients are at increased risk of discontinuing due to unwanted response compared with biosimilar remainder patients (adjusted HR 3.7, 95% CI: 1.0-13.9). Patients who retransitioned due to an increase in objective disease markers had higher discontinuation rates than patients who retransitioned due to symptoms only (66.7% versus 23.7%). Conclusion Retransitioned patients are at increased risk of infliximab discontinuation due to unwanted response. Retransitioning appeared related to the patient and/or disease and not the product. Clinicians might switch patients opting for retransitioning to other treatment regimens.
Collapse
Affiliation(s)
- Rosanne W. Meijboom
- Pharmacy Foundation of Haarlem Hospitals, Haarlem, the Netherlands
- Division of Pharmacoepidemiology & Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands
| | - Helga Gardarsdottir
- Division of Pharmacoepidemiology & Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands
- Department of Clinical Pharmacy, University Medical Center Utrecht, Utrecht, the Netherlands
- Department of Pharmaceutical Sciences, University of Iceland, Reykjavik, Iceland
| | - Matthijs L. Becker
- Pharmacy Foundation of Haarlem Hospitals, Haarlem, the Netherlands
- Department of Clinical Pharmacy, Spaarne Gasthuis, Haarlem, the Netherlands
| | - Kris L. L. Movig
- Department of Clinical Pharmacy, Medisch Spectrum Twente, Enschede, the Netherlands
| | - Johan Kuijvenhoven
- Department of Gastroenterology and Hepatology, Spaarne Gasthuis, Haarlem, the Netherlands
| | - Toine C. G. Egberts
- Division of Pharmacoepidemiology & Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands
- Department of Clinical Pharmacy, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Thijs J. Giezen
- Pharmacy Foundation of Haarlem Hospitals, Boerhaavelaan 24, Haarlem 2035 RC, the Netherlands
- Division of Pharmacoepidemiology & Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht University, Utrecht, the Netherlands
- Department of Clinical Pharmacy, Spaarne Gasthuis, Haarlem, the Netherlands
| |
Collapse
|
3
|
Meijboom RW, Barbier L, Druedahl LC, Sarnola K, Tolonen HM, Gardarsdottir H, Egberts TCG, Giezen TJ. Recommendations on TNFα inhibitor biosimilar use in clinical practice: a comparison of European gastroenterology IBD guidance. Expert Opin Biol Ther 2023; 23:801-817. [PMID: 36947408 DOI: 10.1080/14712598.2023.2189008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2022] [Accepted: 03/06/2023] [Indexed: 03/23/2023]
Abstract
BACKGROUND Professional associations publish guidance advising gastroenterologists on prescribing biosimilars; however, guidelines differ between countries and change over time. This study aimed to map the presence and content of guidance from European gastroenterology associations on TNFα inhibitor biosimilar use and its development over time. RESEARCH DESIGN AND METHODS Guidelines on biosimilar prescribing from national gastroenterology associations in the European Economic Area (EEA) partnered with the European Crohn's and Colitis Organization (ECCO) were collected. Treatment guidelines and biosimilar position papers from 2010 to 2022 were included. Data were extracted using a template. RESULTS 26 of 30 EEA countries have an ECCO-partnered gastroenterology association, of which 14 (53.8%) had national guidelines addressing biosimilars, four (15.4%) followed ECCO's position, and three (11.6%) had treatment guidelines without mentioning biosimilars. From five countries (19.2%) no guidelines were retrieved. Among 18 countries with guidance, 14 (77.8%) associations endorsed initiating biological treatment with biosimilars, and 13 (72.2%) endorsed transitioning from originator to biosimilar. Nine associations published multiple guidelines over time addressing biosimilars; overall, their positions became more encouraging. CONCLUSIONS The majority of gastroenterology associations endorsed biosimilar use. The lack of (up-to-date) guidelines for some associations indicates an area of improvement to support biosimilar use in clinical practice.
Collapse
Affiliation(s)
- Rosanne W Meijboom
- Pharmacy Foundation of Haarlem Hospitals, Haarlem, The Netherlands
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, The Netherlands
| | - Liese Barbier
- Department of Pharmaceutical and Pharmacological Sciences, KU Leuven, Leuven, Belgium
| | - Louise C Druedahl
- Centre for Advanced Studies in Biomedical Innovation Law (CeBIL), Faculty of Law, University of Copenhagen, Copenhagen, Denmark
| | - Kati Sarnola
- Research at Kela, Social Insurance Institution of Finland (Kela), Helsinki, Finland
| | - Hanna M Tolonen
- HUS Pharmacy, HUS Helsinki University Hospital, and University of Helsinki, Helsinki, Finland
- Clinical Pharmacy Group, Division of Pharmacology and Pharmacotherapy, Faculty of Pharmacy, University of Helsinki, Helsinki, Finland
| | - Helga Gardarsdottir
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, The Netherlands
- Department of Clinical Pharmacy, University Medical Centre Utrecht, Utrecht, The Netherlands
- Department of Pharmaceutical Sciences, University of Iceland, Reykjavík, Iceland
| | - Toine C G Egberts
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, The Netherlands
- Department of Clinical Pharmacy, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Thijs J Giezen
- Pharmacy Foundation of Haarlem Hospitals, Haarlem, The Netherlands
- Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical Sciences, Utrecht, The Netherlands
- Department of Clinical Pharmacy, Spaarne Gasthuis, Haarlem and Hoofddorp, The Netherlands
| |
Collapse
|
4
|
Jourdain H, Hoisnard L, Sbidian E, Zureik M. TNF-alpha inhibitors biosimilar use in France: a nationwide population-based study using the French National Health Data System. Sci Rep 2022; 12:19569. [PMID: 36380105 PMCID: PMC9666557 DOI: 10.1038/s41598-022-24050-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Accepted: 11/09/2022] [Indexed: 11/16/2022] Open
Abstract
TNF-alpha inhibitors have revolutionized the therapeutic care in chronic inflammatory diseases. Several biosimilar products were commercialized at their patent expiry, substantially decreasing the cost of treatment. This longitudinal descriptive study aimed at assessing infliximab, etanercept and adalimumab biosimilar penetration rates using data of the French National Health Data System. A total of 207,118 new or prevalent users from the date of first biosimilar commercialization in France (respectively January 2015, May 2016 and October 2018) were included in the study and followed until September 30, 2021. Biosimilars represented respectively 78%, 46% and 53% of the overall initiations, and 94%, 66% and 60% last year's initiations. A total of 46%, 19% and 17% of originator product prevalent users switched for a biosimilar during the follow-up. Biosimilar penetration rate was much higher for infliximab than for its counterparts, due to its hospital delivery modality. Biosimilar initiation and originator-to-biosimilar switch tended to be observed more in rheumatology than in the other specialties. Biosimilar use was mostly consistent across patient socio-demographic characteristics. Biosimilar initiation rate increased rapidly from their market arrival and originator-to-biosimilar switch rate remained moderate, highlighting the need and usefulness of political action and biosimilar use tracking.
Collapse
Affiliation(s)
- Hugo Jourdain
- grid.512012.5EPI-PHARE, French National Agency for Medicines and Health Products Safety (ANSM) and French National Health Insurance (CNAM), 143-147 Boulevard Anatole France, 93285 Saint-Denis, France
| | - Léa Hoisnard
- grid.412116.10000 0001 2292 1474Fédération Hospitalo-Universitaire TRUE InnovaTive theRapy for immUne disordErs, Assistance Publique-Hôpitaux de Paris (AP-HP), Henri Mondor Hospital, 94010 Créteil, France ,grid.7429.80000000121866389INSERM, Centre d’Investigation Clinique 1430, 94010 Créteil, France ,grid.410511.00000 0001 2149 7878EpiDermE Epidemiology in Dermatology and Evaluation of Therapeutics, EA7379, Paris Est Créteil University UPEC, F-94010 Créteil, France
| | - Emilie Sbidian
- grid.512012.5EPI-PHARE, French National Agency for Medicines and Health Products Safety (ANSM) and French National Health Insurance (CNAM), 143-147 Boulevard Anatole France, 93285 Saint-Denis, France ,grid.412116.10000 0001 2292 1474Fédération Hospitalo-Universitaire TRUE InnovaTive theRapy for immUne disordErs, Assistance Publique-Hôpitaux de Paris (AP-HP), Henri Mondor Hospital, 94010 Créteil, France ,grid.7429.80000000121866389INSERM, Centre d’Investigation Clinique 1430, 94010 Créteil, France ,grid.410511.00000 0001 2149 7878EpiDermE Epidemiology in Dermatology and Evaluation of Therapeutics, EA7379, Paris Est Créteil University UPEC, F-94010 Créteil, France ,grid.412116.10000 0001 2292 1474Department of Dermatology, Hôpital Henri Mondor, 51 Avenue du Maréchal de Lattre de Tassigny, 94000 Créteil, France
| | - Mahmoud Zureik
- grid.512012.5EPI-PHARE, French National Agency for Medicines and Health Products Safety (ANSM) and French National Health Insurance (CNAM), 143-147 Boulevard Anatole France, 93285 Saint-Denis, France ,grid.463845.80000 0004 0638 6872University Paris-Saclay, UVSQ, University Paris-Sud, Inserm, Anti-Infective Evasion and Pharmacoepidemiology, CESP, Montigny le Bretonneux, France
| |
Collapse
|