1
|
Hughes SE, Aiyegbusi OL, McMullan C, Turner GM, Anderson N, Cruz Rivera S, Collis P, Glasby J, Lasserson D, Calvert M. Patient-reported outcomes in integrated health and social care: A scoping review. JRSM Open 2024; 15:20542704241232866. [PMID: 38529208 PMCID: PMC10962043 DOI: 10.1177/20542704241232866] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/27/2024] Open
Abstract
Background Patient-reported outcomes (PROs) have potential to support integrated health and social care research and practice; however, evidence of their utilisation has not been synthesised. Objective To identify PRO measures utilised in integrated care and adult social care research and practice and to chart the evidence of implementation factors influencing their uptake. Design Scoping review of peer-reviewed literature. Data sources Six databases (01 January 2010 to 19 May 2023). Study selection Articles reporting PRO use with adults (18+ years) in integrated care or social care settings. Review methods We screened articles against pre-specified eligibility criteria; 36 studies (23%) were extracted in duplicate for verification. We summarised the data using thematic analysis and descriptive statistics. Results We identified 159 articles reporting on 216 PRO measures deployed in a social care or integrated care setting. Most articles used PRO measures as research tools. Eight (5.0%) articles used PRO measures as an intervention. Articles focused on community-dwelling participants (35.8%) or long-term care home residents (23.9%), with three articles (1.9%) focussing on integrated care settings. Stakeholders viewed PROs as feasible and acceptable, with benefits for care planning, health and wellbeing monitoring as well as quality assurance. Patient-reported outcome measure selection, administration and PRO data management were perceived implementation barriers. Conclusion This scoping review showed increasing utilisation of PROs in adult social care and integrated care. Further research is needed to optimise PROs for care planning, design effective training resources and develop policies and service delivery models that prioritise secure, ethical management of PRO data.
Collapse
Affiliation(s)
- Sarah E Hughes
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Blood and Transplant Research Unit in Precision Cellular Therapeutics, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Olalekan L Aiyegbusi
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Blood and Transplant Research Unit in Precision Cellular Therapeutics, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Christel McMullan
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Blood and Transplant Research Unit in Precision Cellular Therapeutics, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Surgical Reconstruction and Microbiology Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Grace M Turner
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
| | - Nicola Anderson
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
| | - Samantha Cruz Rivera
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- DEMAND Hub, University of Birmingham, Birmingham, UK
| | | | - Jon Glasby
- School of Social Policy, University of Birmingham, Birmingham, UK
- IMPACT (Improving Adult Social Care Together), University of Birmingham, Birmingham, UK
| | - Daniel Lasserson
- National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, Birmingham, UK
- Warwick Medical School, University of Warwick, Coventry, UK
- Department of Geriatric Medicine, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Melanie Calvert
- Centre for Patient Reported Outcomes Research, Institute of Applied Health Research, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, Birmingham, UK
- Birmingham Health Partners Centre for Regulatory Science and Innovation, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Blood and Transplant Research Unit in Precision Cellular Therapeutics, University of Birmingham, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- National Institute for Health and Care Research (NIHR) Surgical Reconstruction and Microbiology Centre, University of Birmingham and University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
- DEMAND Hub, University of Birmingham, Birmingham, UK
| |
Collapse
|
2
|
van Hoorn ES, Ye L, van Leeuwen N, Raat H, Lingsma HF. Value-Based Integrated Care: A Systematic Literature Review. Int J Health Policy Manag 2024; 13:8038. [PMID: 38618830 PMCID: PMC11016279 DOI: 10.34172/ijhpm.2024.8038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Accepted: 01/30/2024] [Indexed: 04/16/2024] Open
Abstract
BACKGROUND Healthcare services worldwide are transforming themselves into value-based organizations. Integrated care is an important aspect of value-based healthcare (VBHC), but practical evidence-based recommendations for the successful implementation of integrated care within a VBHC context are lacking. This systematic review aims to identify how value-based integrated care (VBIC) is defined in literature, and to summarize the literature regarding the effects of VBIC, and the facilitators and barriers for its implementation. METHODS Embase, Medline ALL, Web of Science Core Collection, and Cochrane Central Register of Controlled Trails databases were searched from inception until January 2022. Empirical studies that implemented and evaluated an integrated care intervention within a VBHC context were included. Non-empirical studies were included if they described either a definition of VBIC or facilitators and barriers for its implementation. Theoretical articles and articles without an available full text were excluded. All included articles were analysed qualitatively. The Rainbow Model of Integrated Care (RMIC) was used to analyse the VBIC interventions. The quality of the articles was assessed using the Mixed Methods Appraisal Tool (MMAT). RESULTS After screening 1328 titles/abstract and 485 full-text articles, 24 articles were included. No articles were excluded based on quality. One article provided a definition of VBIC. Eleven studies reported-mostly positive- effects of VBIC, on clinical outcomes, patient-reported outcomes, and healthcare utilization. Nineteen studies reported facilitators and barriers for the implementation of VBIC; factors related to reimbursement and information technology (IT) infrastructure were reported most frequently. CONCLUSION The concept of VBIC is not well defined. The effect of VBIC seems promising, but the exact interpretation of effect evaluations is challenged by the precedence of multicomponent interventions, multiple testing and generalizability issues. For successful implementation of VBIC, it is imperative that healthcare organizations consider investing in adequate IT infrastructure and new reimbursement models. Systematic Review Registration: PROSPERO (CRD42021259025).
Collapse
Affiliation(s)
- Evelien S. van Hoorn
- Department of Public Health, Erasmus MC, Erasmus University Medical Centre, Rotterdam, The Netherlands
| | | | | | | | | |
Collapse
|
3
|
Alamrani S, Gardner A, Falla D, Russell E, Rushton AB, Heneghan NR. Outcome measures for young people with adolescent idiopathic scoliosis: A qualitative exploration of healthcare professionals' perceptions and practices. PLoS One 2024; 19:e0297339. [PMID: 38277344 PMCID: PMC10817127 DOI: 10.1371/journal.pone.0297339] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Accepted: 01/03/2024] [Indexed: 01/28/2024] Open
Abstract
BACKGROUND Limited knowledge exists on current use of patient reported outcome measures (PROMs) and performance measures for adolescents with idiopathic scoliosis (AIS), as well as health care professionals' (HCPs) perceived barriers and facilitators towards their use. This study's objectives were: 1) to explore current practice of HCPs when assessing outcomes for AIS 2) to understand perceived barriers and facilitators of HCPs to use PROMs 3) to understand perceived barriers and facilitators of HCPs to use performance measures. METHODS A qualitative study recruited a purposive sample of HCPs from a tertiary hospital in the United Kingdom. Mean years of experience managing individuals with AIS was 11.8 years; and included surgeons, physiotherapists and nurses, educated at Bachelor, Masters and Doctoral level. Consent to participate and demographic information were collected in advance of the interviews. In-depth, virtual semi-structured interviews were informed by a topic guide based on current evidence. Interviews of approximately 45 minutes were audio and video recorded and transcribed verbatim alongside written field notes. Data were coded and analysed using inductive thematic analysis, involving researchers with topic and methodological expertise and input from a patient representative. RESULTS Two themes emerged regarding current practice of using PROMs routine practice and personal evaluations. Four themes emerged as barriers to using PROMs for individuals with AIS: priority and support (e.g., HCPs focus on providing care), practical challenges (e.g., inadequate PROMs), patient-related challenges (e.g., patient preferences) and knowledge, education, and perceived value. Two themes emerged as facilitators: quality existing measure (e.g., sufficient psychometric properties), and priority and support (e.g., research department/culture). Themes for barriers to use performance measures were practicality (e.g., need physical space) and perceived value and knowledge (e.g., PROMs are more important), while the one theme for facilitators was practical consideration (e.g., acceptability). CONCLUSIONS Although HCPs perceived the value of using outcome measures, current practice indicates limited use for individuals with AIS. The findings revealed different barriers and facilitators to implement PROMs in practice. Adopting performance measure are limited due to lack of knowledge and perceived value alongside the practicality, while considering practical factors can improve the use of these measures in practice.
Collapse
Affiliation(s)
- Samia Alamrani
- Centre of Precision Rehabilitation for Spinal Pain (CPR Spine), School of Sport, Exercise and Rehabilitation Sciences, College of Life and Environmental Sciences, University of Birmingham, Birmingham, United Kingdom
- Physical Therapy Department, College of Applied Medical Science, University of Tabuk, Tabuk, Saudi Arabia
| | - Adrian Gardner
- Spine Unit, The Royal Orthopaedic Hospital NHS Foundation Trust, Northfield, Birmingham, United Kingdom
| | - Deborah Falla
- Centre of Precision Rehabilitation for Spinal Pain (CPR Spine), School of Sport, Exercise and Rehabilitation Sciences, College of Life and Environmental Sciences, University of Birmingham, Birmingham, United Kingdom
| | - Emily Russell
- Oxford University Hospitals NHS Foundation Trust, Oxford, United Kingdom
| | - Alison B. Rushton
- School of Physical Therapy, Western University, London, Ontario, Canada
| | - Nicola R. Heneghan
- Centre of Precision Rehabilitation for Spinal Pain (CPR Spine), School of Sport, Exercise and Rehabilitation Sciences, College of Life and Environmental Sciences, University of Birmingham, Birmingham, United Kingdom
| |
Collapse
|
4
|
Maharaj AD, Roberts N, Jefford M, Ng J, Rutherford C, Koczwara B. The use of patient reported outcome measures in oncology clinical practice across Australia and New Zealand. J Patient Rep Outcomes 2024; 8:1. [PMID: 38165502 PMCID: PMC10761654 DOI: 10.1186/s41687-023-00664-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Accepted: 11/17/2023] [Indexed: 01/03/2024] Open
Abstract
BACKGROUND While there is increasing evidence on the benefits of PROMs in cancer care, the extent of routine collection and use of PROMs in clinical cancer practice across Australia and New Zealand (ANZ) is unknown. This study examined the prevalence and characteristics of PROMs use in routine clinical cancer care in ANZ. METHODS An online survey was designed and disseminated via professional societies and organisations using a snowball sampling approach to clinical and health administration professionals managing cancer care in ANZ. A poster advertising the study was also circulated on professional social media networks via LinkedIn and Twitter inviting health professionals from ANZ to participate if they were using or intending to use PROMs in clinical cancer practice. Responders opted into the survey via the survey link. RESULTS From 132 survey views, 91(response rate, 69%) respondents from 56 clinical practices across ANZ agreed to participate in the survey, and of these 55 (n = 55/91, 60%) respondents reported collecting PROMs within their clinical practice. The majority of the respondents were from the State of New South Wales in Australia (n = 21/55, 38%), hospital (n = 35/55, 64%), and a public setting (n = 46/55, 83%). PROMs were collected in all cancer types (n = 21/36, 58%), in all stages of the disease (n = 31/36, 86%), in an adult population (n = 33/36, 92%), applied in English (n = 33/36, 92%), and used to facilitate communication with other reasons (27/36, 75%). A geospatial map analysis provided insights into the variation in PROMs uptake between the two countries and in certain jurisdictions within Australia. This study also highlights the limited resources for PROMs implementation, and a lack of systematic priority driven approach. CONCLUSION PROM use across Australia and New Zealand seems variable and occurring predominantly in larger metropolitan centres with limited standardisation of approach and implementation. A greater focus on equitable adoption of PROMs in diverse cancer care settings is urgently needed.
Collapse
Affiliation(s)
- Ashika D Maharaj
- Public Health and Preventative Medicine, Monash University, Clayton, VIC, Australia.
- Technology, Policy and Management, Delft University of Technology, Delft, The Netherlands.
| | - Natasha Roberts
- The University of Queensland Centre for Clinical Research, Brisbane, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Herston, QLD, Australia
| | - Michael Jefford
- Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia
- Australian Cancer Survivorship Centre, Peter MacCallum Cancer Centre, Melbourne, VIC, Australia
- Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, VIC, Australia
| | - Jerome Ng
- Te Whatu Ora Counties Manukau, Auckland, New Zealand
- School of Pharmacy, University of Auckland, Auckland, New Zealand
| | - Claudia Rutherford
- Cancer Care Research Unit, Sydney Nursing School, University of Sydney, Sydney, NSW, Australia
- The Daffodil Centre, The University of Sydney, a Joint Venture with Cancer Council NSW, Sydney, Australia
| | - Bogda Koczwara
- Flinders Medical Centre and Flinders Health and Medical Research Institute, Flinders University, Adelaide, SA, Australia
| |
Collapse
|
5
|
Consolo L, Basile I, Colombo S, Rusconi D, Pasquot L, Campa T, Caraceni A, Lusignani M. Exploring patient perspectives on electronic patient-reported outcome measures in home-based cancer palliative care: A qualitative study. Digit Health 2024; 10:20552076241249962. [PMID: 38665884 PMCID: PMC11044777 DOI: 10.1177/20552076241249962] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/10/2024] [Indexed: 04/28/2024] Open
Abstract
Background Electronic patient-reported outcomes (ePROMs) enhance symptom management and patients' engagement in palliative cancer care. However, integrating them into this setting brings challenges, including patients' familiarity with technological devices and declining health status. Prioritizing the patient's acceptability and feasibility is crucial for their adoption. However, more knowledge is needed about patients' perspectives on the adoption of ePROMs in the community, especially for home-based palliative care. Aim Explore patient viewpoints on utilizing ePROMs for symptom reporting in home-based oncology palliative care. Design A qualitative interpretative approach was used to evaluate patients' points of view on using ePROMs in this specific care setting. Semistructured interviews were carried out. Data were analyzed using a reflexive thematic analysis. Setting/participants A total of 25 patients receiving oncological home palliative care from the advanced palliative care unit of the Fondazione IRCCS Istituto Nazionale dei Tumori in Milan, Italy, were invited to participate. Twenty interviews were conducted, as five patients declined due to deteriorating health. Results Four themes were identified: (1) strategic value of ePROMs and subjective appreciation; (2) enhancing patient centeredness through ePROMs; (3) exploring and addressing concerns about the use of ePROMs and (4) intersecting factors influencing the efficacy of ePROMs. Conclusion Despite initial reticence, home palliative care patients consider ePROMs as potentially valuable allies monitoring symptoms, enhancing their quality of life, and amplifying their voices on less explored aspects of care. Continuous dialog between healthcare professionals and patients is crucial for addressing patient skepticism about ePROMs and their impact on the human aspect of care.
Collapse
Affiliation(s)
- Letteria Consolo
- Department of Biomedicine and Prevention, University of Rome “Tor Vergata”, Rome, Italy
- Bachelor School of Nursing, Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
| | - Ilaria Basile
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
| | - Stella Colombo
- Intensive Care Unit, Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
| | - Daniele Rusconi
- Urologic Oncology Unit, Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
| | - Loredana Pasquot
- Department of Clinical Sciences and Community Health, University of Milan, Milan, Italy
| | - Tiziana Campa
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
| | - Augusto Caraceni
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, Milan, Italy
- Department of Clinical Sciences and Community Health, University of Milan, Milan, Italy
| | - Maura Lusignani
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
| |
Collapse
|
6
|
Silveira Bianchim M, Crane E, Jones A, Neukirchinger B, Roberts G, Mclaughlin L, Noyes J. The implementation, use and impact of patient reported outcome measures in value-based healthcare programmes: A scoping review. PLoS One 2023; 18:e0290976. [PMID: 38055759 DOI: 10.1371/journal.pone.0290976] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Accepted: 08/08/2023] [Indexed: 12/08/2023] Open
Abstract
BACKGROUND Value-Based Healthcare (VBHC) focuses on the value of patient outcomes and is achieved by ensuring resources already available are managed to realise the best possible individual and population health outcomes. Patient reported outcome measures (PROMs) measure the impact of illnesses from the patient perspective. We conducted a scoping review to understand how PROMs were implemented and used, and their impact in the context of VBHC. METHODS Arksey and O'Malley's overarching framework supplemented by principles from mixed-methods Framework Synthesis were used. CINAHL, Cochrane Library, EMBASE, MEDLINE, PsycINFO, Web of Science, Google Scholar and reference lists were searched. An a priori data extraction framework was created using the review question and objectives as key domains against which to extract data. Mixed-methods data were organised, integrated and preserved in original format and reported for each domain. RESULTS Forty-three studies were included with 60,200 participants. Few studies reported a well-developed programme theory and we found little robust evidence of effect. PROMs were universally considered to have the potential to increase patient satisfaction with treatment and services, enhance patient awareness of symptoms and self-management, and improve health outcomes such as quality of life and global health status. Evidence is currently limited on how PROMs work and how best to optimally implement PROMs to achieve the target outcome. Implementation challenges commonly prevented the realisation of optimal outcomes and patients generally needed better and clearer communication about why PROMs were being given and how they could optimally be used to support their own self-management. CONCLUSION PROMSs have yet to demonstrate their full potential in a VBHC context. Optimal PROMs implementation is poorly understood by clinicians and patients. Future studies should explore different models of PROM implementation and use within VBHC programmes to understand what works best and why for each specific context, condition, and population.
Collapse
Affiliation(s)
| | - Ellie Crane
- School of Medical and Health Sciences, Bangor University, Bangor, United Kingdom
| | - Anwen Jones
- School of Medical and Health Sciences, Bangor University, Bangor, United Kingdom
| | | | - Gareth Roberts
- Aneurin Bevan University Health Board, Newport, United Kingdom
| | - Leah Mclaughlin
- School of Medical and Health Sciences, Bangor University, Bangor, United Kingdom
| | - Jane Noyes
- School of Medical and Health Sciences, Bangor University, Bangor, United Kingdom
| |
Collapse
|
7
|
Shunmugasundaram C, Sundaresan P, White K, King MT, Milross C, Rodd L, Fuller M, Rutherford C. Development and implementation barriers of a new patient-reported measure: The Radiation therapy-related Inconvenience Questionnaire (RIQ). J Med Imaging Radiat Oncol 2023; 67:777-788. [PMID: 37742323 DOI: 10.1111/1754-9485.13586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2023] [Accepted: 09/04/2023] [Indexed: 09/26/2023]
Abstract
INTRODUCTION Radiation therapy (RT) can benefit approximately 50% of cancer patients and contribute to 40% of all cancer cures, yet its utilisation in cancer is low globally. Several factors contribute to this including perceived inconvenience related to accessing and utilising RT. To quantitatively assess the latter, a new tool - the Radiation therapy-related Inconvenience Questionnaire (RIQ) - was developed. This study aimed to pre-test the RIQ and explore barriers and facilitators to implementing it in routine clinical practice and clinical trials. METHODS Semi-structured cognitive interviews were conducted with patients undertaking RT, recruited via three hospitals to examine content validity, acceptability, and comprehensibility of the RIQ. Interviews identified inconsistencies, relevance, and clarity of items. Semi-structured interviews with healthcare professionals involved in the delivery of care to individuals undertaking RT explored barriers and facilitators to routine usage. Thematic analysis was used to identify themes. RESULTS Patients (N = 15) identified problems in content, instructions, layout, length, and response options; consequently, 25 items were revised and eight removed, resulting in a final 29-item questionnaire. Clinicians identified staff- and patient-specific barriers to implementing RIQ in clinical practice. Perceived facilitators included the following: (a) workplace factors, (b) mode of administration, and (c) imparting knowledge. CONCLUSION This study demonstrated acceptability and comprehensibility of the 29-item RIQ amongst cancer patients undertaking RT and treating clinicians. The next phase will evaluate the RIQ's measurement properties in a larger clinical study. The barriers and facilitators identified can guide future implementation of RIQ in clinical practice and clinical trials.
Collapse
Affiliation(s)
- Chindhu Shunmugasundaram
- Faculty of Science, School of Psychology, Centre for Medical Psychology and Evidence-based Decision-making, The University of Sydney, New South Wales, Sydney, Australia
- Faculty of Medicine and Health, Susan Wakil School of Nursing and Midwifery, Sydney Quality of Life Office, The University of Sydney, New South Wales, Sydney, Australia
| | - Puma Sundaresan
- Radiation Oncology Network, Western Sydney Local Health District, New South Wales, Sydney, Australia
- Sydney Medical School, The University of Sydney, New South Wales, Sydney, Australia
| | - Kate White
- Cancer Nursing Research Unit, The University of Sydney, Susan Wakil School of Nursing and Midwifery, New South Wales, Sydney, Australia
| | - Madeleine T King
- Faculty of Medicine and Health, Susan Wakil School of Nursing and Midwifery, Sydney Quality of Life Office, The University of Sydney, New South Wales, Sydney, Australia
| | - Chris Milross
- Sydney Medical School, The University of Sydney, New South Wales, Sydney, Australia
- Radiation Oncology and Medical Services, Chris O'Brien Lifehouse, New South Wales, Sydney, Australia
| | - Lucinda Rodd
- Radiation Oncology, Central West Cancer Centre, New South Wales, Orange, Australia
| | - Matthew Fuller
- Radiation Oncology, Central West Cancer Centre, New South Wales, Orange, Australia
| | - Claudia Rutherford
- Faculty of Medicine and Health, Susan Wakil School of Nursing and Midwifery, Sydney Quality of Life Office, The University of Sydney, New South Wales, Sydney, Australia
- Cancer Nursing Research Unit, The University of Sydney, Susan Wakil School of Nursing and Midwifery, New South Wales, Sydney, Australia
| |
Collapse
|
8
|
Burner-Fritsch I, Kolmhuber S, Hodiamont F, Bausewein C, Hriskova K. Implementing ePROM in specialist palliative home care: the professionals' perspective - a mixed-methods study. Palliat Care Soc Pract 2023; 17:26323524231186827. [PMID: 37560175 PMCID: PMC10408331 DOI: 10.1177/26323524231186827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2022] [Accepted: 06/21/2023] [Indexed: 08/11/2023] Open
Abstract
Background Over the last decades, patient-reported outcome measures (PROM) have been developed for a better understanding of patient needs. The Integrated Palliative Care Outcome Scale (IPOS) is an internationally recommended PROM in palliative care. The validated electronic version of IPOS (eIPOS) was implemented in four German specialist palliative home care (SPHC) teams for use in everyday clinical practice. Patients reported symptoms and concerns via eIPOS, which were transmitted directly to the electronic patient record of the respective SPHC team. Objectives The aim of the study was to describe and explore the health care professionals' (HCPs') experiences regarding acceptance and use of eIPOS in clinical practice in SPHC. Design The mixed-methods sequential explanatory design comprised an anonymized quantitative online survey followed by qualitative focus groups. Methods The online survey asked in both closed and open questions for HCP's experience with eIPOS. Ambiguous results from the survey were discussed in two focus groups. Survey data were analysed with descriptive and univariable statistics, and the framework approach was used for qualitative data. In a further step, we conducted integrated analysis of quantitative and qualitative results using joint displays. Results All HCPs of the four SPHC teams (n = 52) were invited to participate. HCPs participating in the survey (n = 32) and the focus groups (n = 7) saw potentials for implementing ePROM in palliative home care - as far as it is technically easy to handle and can be easily integrated into clinical practice. Conclusion Successful use of ePROMs is affected by the possibility of easy integration into the teams' different structures and processes and the HCPs' perceptions of potentials regarding ePROM use in SPHC. Registration The study is registered on clinicaltrials.org (NCT03879668).
Collapse
Affiliation(s)
- Isabel Burner-Fritsch
- Department of Palliative Medicine, University Hospital, LMU Munich, Bahnhofstraße 20, 83673 Bichl, Germany
| | - Stefanie Kolmhuber
- Department of Palliative Medicine, University Hospital, LMU Munich, Munich, Germany
| | - Farina Hodiamont
- Department of Palliative Medicine, University Hospital, LMU Munich, Munich, Germany
| | - Claudia Bausewein
- Department of Palliative Medicine, University Hospital, LMU Munich, Munich, Germany
| | - Katerina Hriskova
- Department of Palliative Medicine, University Hospital, LMU Munich, Munich, Germany
| |
Collapse
|
9
|
Consolo L, Colombo S, Basile I, Rusconi D, Campa T, Caraceni A, Lusignani M. Barriers and facilitators of electronic patient-reported outcome measures (e-PROMs) for patients in home palliative cancer care: a qualitative study of healthcare professionals' perceptions. BMC Palliat Care 2023; 22:111. [PMID: 37542264 PMCID: PMC10401773 DOI: 10.1186/s12904-023-01234-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Accepted: 07/26/2023] [Indexed: 08/06/2023] Open
Abstract
BACKGROUND Patient-reported outcomes in palliative care enable early monitoring and management of symptoms that most impact patients' daily lives; however, there are several barriers to adopting electronic Patient-reported Outcome Measures (e-PROMs) in daily practice. This study explored the experiences of health care professionals (HCPs) regarding potential barriers and facilitators in implementing e-PROMs in palliative cancer care at home. METHODS This was a qualitative descriptive study. The data were collected from two focus groups structured according to the conceptual framework of Grol. HCPs involved in home palliative cancer care of Fondazione IRCCS Istituto Nazionale dei Tumori of Milan were enrolled. Data were analyzed using a reflexive thematic analysis. RESULTS A total of 245 codes were generated, 171 for the first focus group and 74 for the second focus group. The results were subdivided into subthemes according to Grol's themes: Innovation, Individual professional, Patient, Social context, Organizational context, except Economic Political context. Nine HCPs attended the first focus group, and ten attended the second. According to these participants, e-PROMs could be integrated into clinical practice after adequate training and support of HCPs at all stages of implementation. They identified barriers, especially in the social and organizational contexts, due to the uniqueness of the oncological end-of-life setting and the intangible care interventions, as well as many facilitators for the innovation that these tools bring and for improved communication with the patient and the healthcare team. CONCLUSIONS e-PROMs are perceived by HCPs as adding value to patient care and their work; however, barriers remain especially related to the fragility of these patients, the adequacy of technological systems, lack of education, and the risk of low humanization of care.
Collapse
Affiliation(s)
- Letteria Consolo
- Department of Biomedicine and Prevention, University of Rome "Tor Vergata", Rome, Italy.
- Bachelor School of Nursing, IRCCS, National Cancer Institute, Milan, Italy.
| | - Stella Colombo
- Intensive Care Unit, IRCCS, National Cancer Institute, Milan, Italy
| | - Ilaria Basile
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation, IRCCS, National Cancer Institute, Milan, Italy
| | - Daniele Rusconi
- Urology Unit, IRCCS, National Cancer Institute, Milan, Italy
| | - Tiziana Campa
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation, IRCCS, National Cancer Institute, Milan, Italy
| | - Augusto Caraceni
- High-Complexity Unit of Palliative Care, Pain Therapy and Rehabilitation, IRCCS, National Cancer Institute, Milan, Italy
- University of Milan, Milan, Italy
| | - Maura Lusignani
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
| |
Collapse
|
10
|
Warnecke E, Salvador Comino MR, Kocol D, Hosters B, Wiesweg M, Bauer S, Welt A, Heinzelmann A, Müller S, Schuler M, Teufel M, Tewes M. Electronic Patient-Reported Outcome Measures (ePROMs) Improve the Assessment of Underrated Physical and Psychological Symptom Burden among Oncological Inpatients. Cancers (Basel) 2023; 15:cancers15113029. [PMID: 37296991 DOI: 10.3390/cancers15113029] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2023] [Revised: 05/19/2023] [Accepted: 05/29/2023] [Indexed: 06/12/2023] Open
Abstract
For advanced cancer inpatients, the established standard for gathering information about symptom burden involves a daily assessment by nursing staff using validated assessments. In contrast, a systematic assessment of patient-reported outcome measures (PROMs) is required, but it is not yet systematically implemented. We hypothesized that current practice results in underrating the severity of patients' symptom burden. To explore this hypothesis, we have established systematic electronic PROMs (ePROMs) using validated instruments at a major German Comprehensive Cancer Center. In this retrospective, non-interventional study, lasting from September 2021 to February 2022, we analyzed collected data from 230 inpatients. Symptom burden obtained by nursing staff was compared to the data acquired by ePROMs. Differences were detected by performing descriptive analyses, Chi-Square tests, Fisher's exact, Phi-correlation, Wilcoxon tests, and Cohen's r. Our analyses pointed out that pain and anxiety especially were significantly underrated by nursing staff. Nursing staff ranked these symptoms as non-existent, whereas patients stated at least mild symptom burden (pain: meanNRS/epaAC = 0 (no); meanePROM = 1 (mild); p < 0.05; r = 0.46; anxiety: meanepaAC = 0 (no); meanePROM = 1 (mild); p < 0.05; r = 0.48). In conclusion, supplementing routine symptom assessment used daily by nursing staff with the systematic, e-health-enabled acquisition of PROMs may improve the quality of supportive and palliative care.
Collapse
Affiliation(s)
- Eva Warnecke
- Department of Palliative Medicine, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Maria Rosa Salvador Comino
- Department of Palliative Medicine, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Dilara Kocol
- Department of Psychosomatic Medicine and Psychotherapy, West German Cancer Center, LVR-Klinikum Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | | | - Marcel Wiesweg
- Department of Medical Oncology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Sebastian Bauer
- Department of Medical Oncology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Anja Welt
- Department of Medical Oncology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Anna Heinzelmann
- Department of Palliative Medicine, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Sandy Müller
- Department of Palliative Medicine, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Martin Schuler
- Department of Medical Oncology, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Martin Teufel
- Department of Psychosomatic Medicine and Psychotherapy, West German Cancer Center, LVR-Klinikum Essen, University of Duisburg-Essen, 45122 Essen, Germany
| | - Mitra Tewes
- Department of Palliative Medicine, West German Cancer Center, University Hospital Essen, University of Duisburg-Essen, 45122 Essen, Germany
| |
Collapse
|
11
|
Depla AL, Pluut B, Lamain-de Ruiter M, Kersten AW, Evers IM, Franx A, Bekker MN. PROMs and PREMs in routine perinatal care: mixed methods evaluation of their implementation into integrated obstetric care networks. J Patient Rep Outcomes 2023; 7:26. [PMID: 36894797 PMCID: PMC9998006 DOI: 10.1186/s41687-023-00568-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Accepted: 02/23/2023] [Indexed: 03/11/2023] Open
Abstract
BACKGROUND In the transition towards value-based healthcare, patient-reported outcome and experience measures (PROM and PREM) are recommended by international collaborations and government programs to guide clinical practice and quality improvement. For many conditions, using PROM/PREM over the complete continuum of care requires implementation across care organizations and disciplines. Along PROM/PREM implementation in obstetric care networks (OCN), we aimed to evaluate implementation outcomes and the processes influencing these outcomes in the complex context of care networks across the continuum of perinatal care. METHODS Three OCN in the Netherlands implemented PROM/PREM in routine practice, using an internationally developed outcomes set with care professionals and patient advocates. Their aim was to use PROM/PREM results individually to guide patient-specific care decisions and at group-level to improve quality of care. The implementation process was designed following the principles of action research: iteratively planning implementation, action, data generation and reflection to refine subsequent actions, involving both researchers and care professionals. During the one-year implementation period in each OCN, implementation outcomes and processes were evaluated in this mixed-methods study. Data generation (including observation, surveys and focus groups) and analysis were guided by two theoretical implementation frameworks: the Normalization Process Theory and Proctor's taxonomy for implementation outcomes. Qualitative findings were supplemented with survey data to solidify findings in a broader group of care professionals. RESULTS Care professionals in OCN found the use of PROM/PREM acceptable and appropriate, recognized their benefits and felt facilitated in their patient-centered goals and vision. However, feasibility for daily practice was low, mainly due to IT issues and time constraints. Hence PROM/PREM implementation did not sustain, but strategies for future PROM/PREM implementation were formulated in all OCN. Processes contributing positively to implementation outcomes were internalization (understand the value) and initiation (driven by key-participants), whereas challenges in relational integration (maintain confidence) and reconfiguration (refine activities) affected implementation negatively. CONCLUSION Although implementation did not sustain, network-broad PROM/PREM use in clinic and quality improvement matched professionals' motivation. This study provides recommendations to implement PROM/PREM meaningfully in practice in ways that support professionals in their drive towards patient-centered care. In order for PROM/PREM to fulfill their potential for value-based healthcare, our work highlights the need for sustainable IT infrastructures, as well as an iterative approach to refine their complex implementation into local contexts.
Collapse
Affiliation(s)
- Anne L Depla
- Department of Obstetrics and Gynecology, Wilhelmina Children's Hospital, University Medical Centre Utrecht, KE.04.123.1, Lundlaan 6, 3584 EA, Utrecht, The Netherlands
| | - Bettine Pluut
- Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Marije Lamain-de Ruiter
- Department of Obstetrics and Gynecology, Wilhelmina Children's Hospital, University Medical Centre Utrecht, KE.04.123.1, Lundlaan 6, 3584 EA, Utrecht, The Netherlands
- Department of Obstetrics and Gynecology, Erasmus MC Sophia, Rotterdam, The Netherlands
| | - Anna W Kersten
- Department of Public Health, Julius Centre, University Medical Centre Utrecht, Utrecht, The Netherlands
| | - Inge M Evers
- Department of Obstetrics and Gynecology, Meander Medical Centre, Amersfoort, The Netherlands
| | - Arie Franx
- Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Mireille N Bekker
- Department of Obstetrics and Gynecology, Wilhelmina Children's Hospital, University Medical Centre Utrecht, KE.04.123.1, Lundlaan 6, 3584 EA, Utrecht, The Netherlands.
| |
Collapse
|
12
|
Klootwijk A, Bakx P, Franx A, Bijma H, Ernst-Smelt H, Lamain-de Ruiter M, Posthumus A, van Rijn B. Patient-reported outcome and experience measures for quality improvement in pregnancy and childbirth care: a retrospective cohort study. BMJ Open Qual 2023; 12:bmjoq-2022-001922. [PMID: 36889814 PMCID: PMC10008327 DOI: 10.1136/bmjoq-2022-001922] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2022] [Accepted: 01/11/2023] [Indexed: 03/10/2023] Open
Abstract
BACKGROUND Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) can highlight issues that remain unnoticed when using standard clinical quality indicators. However, estimations of the potential power of measuring PROMs and PREMs to identify unrecognised areas suitable for quality improvement are often limited by a lack of reliable real-world data. Here, we report on how the indicator set for PROMs and PREMs that was recently developed by the International Consortium for Health Outcome Measures can change perspectives on quality assessment in women receiving care for pregnancy and childbirth. METHODS PROMs and PREMs were captured 6 months after childbirth via an online survey in a single academic maternity unit in the Netherlands between 2018 and 2019. Indicators of abnormality were scored using predefined cut-off values established by a national consensus group. We used regression analysis to identify associations between PROMs, PREMs and healthcare use, and further stratified data to explore the distribution of indicators among relevant patient subgroups. RESULTS Of 2775 questionnaires, 645 were completed and linked to medical health records. Despite only 5% of women reporting overall dissatisfaction with care, suboptimal scores were often found; in birth experience for 32% of the population, and 42% who experienced painful sexual intercourse. Subgroup analysis further revealed associations with relevant indicators of quality of care; inadequate pain relief among women with preterm birth (OR 8.8), pain with sexual intercourse among women undergoing vaginal assisted delivery (OR 2.2) and women living in a deprived area had problematic birth experiences (coefficient -3.2). CONCLUSION Use of PROMs and PREMs in pregnancy and childbirth care provides new insights on quality of care, resulting in potentially actionable targets for improvement not normally identified with standard clinical quality indicators. Implementation strategies and follow-up are needed to act on these findings.
Collapse
Affiliation(s)
- Anouk Klootwijk
- Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Pieter Bakx
- Erasmus School of Health Policy and Management, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Arie Franx
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Hilmar Bijma
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Hiske Ernst-Smelt
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Marije Lamain-de Ruiter
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Anke Posthumus
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| | - Bas van Rijn
- Department of Obstetrics and Gynecology, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
| |
Collapse
|
13
|
Ng VL, Dunphy C, Shemesh E, Lobritto S, Eisenberg E, Pomponi C, Szolna J, Wilkerson D, Gupta N, Romero R, Perito ER, DiPaola F, Gonzalez-Peralta RP, Hsu E, Saarela K, Mohammad S, Superina R, Logan S, Miller DW, Krise-Confair C, Swami N, Mazariegos G. Feasibility of using a patient-reported outcome measure into clinical practice following pediatric liver transplantation: The Starzl Network experience. Pediatr Transplant 2023; 27:e14409. [PMID: 36272132 DOI: 10.1111/petr.14409] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Revised: 09/19/2022] [Accepted: 09/23/2022] [Indexed: 01/24/2023]
Abstract
BACKGROUND Patient-reported outcome measures (PROMs) are not routinely used in clinical care by pediatric liver transplant (LT) teams. The Starzl Network for Excellence in Pediatric Transplantation (SNEPT) assessed feasibility of using a disease-specific Quality of Life (QoL) questionnaire in the ambulatory setting at 10 SNEPT sites. METHODS A mixed methods feasibility project assessing administration processes, barriers, and user experiences with the Pediatric Liver Transplant Quality of Life (PeLTQL) tool. Iterative processes sought stakeholder feedback across four phases (Pilot, Extended Pilot, Development of a Mobile App PeLTQL version, and Pilot App use). RESULTS A total of 149 patient-parent dyads completed the PeLTQL during LT clinic follow-up. Clinicians, parents, and patients evaluated and reported on feasibility of operationalization. Only two of 10 SNEPT sites continued PeLTQL administration after the initial two pilot phases. Reasons include limited clinical time and available personnel aggravated by the COVID-19 pandemic. In response, a mobile application version of the PeLTQL was initiated. Providing PeLTQL responses electronically was "very easy" or "easy" as reported by 96% (22/23) parents. CONCLUSIONS Administration of a PROM into post-pediatric LT clinical care was feasible, but ongoing utilization stalled. Use of a mobile app towards facilitating completion of the PeLTQL outside of clinic hours may address the time and work-flow barriers identified.
Collapse
Affiliation(s)
- Vicky Lee Ng
- The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada
| | - Claire Dunphy
- Icahn School of Medicine at Mount Sinai, Mount Sinai Kravis Children's Hospital, New York, New York, USA
| | - Eyal Shemesh
- Icahn School of Medicine at Mount Sinai, Mount Sinai Kravis Children's Hospital, New York, New York, USA
| | - Steven Lobritto
- Columbia University Medical Center, Morgan Stanley Children's Hospital of New York, New York, New York, USA
| | - Elizabeth Eisenberg
- Patient and Family Voice, Starzl Network for Excellence in Pediatric Transplantation, Pittsburgh, Pennsylvania, USA
| | - Claudia Pomponi
- Icahn School of Medicine at Mount Sinai, Mount Sinai Kravis Children's Hospital, New York, New York, USA
| | - Jonathan Szolna
- Children's Hospital of Pittsburgh, UPMC, Pittsburgh, Pennsylvania, USA
| | - Dawn Wilkerson
- Children's Hospital of Pittsburgh, UPMC, Pittsburgh, Pennsylvania, USA
| | - Nitika Gupta
- Children's Healthcare of Atlanta and Emory University School of Medicine, Atlanta, Georgia, USA
| | - Rene Romero
- Children's Healthcare of Atlanta and Emory University School of Medicine, Atlanta, Georgia, USA
| | - Emily R Perito
- Benioff Children's Hospital, University of California San Francisco, San Francisco, California, USA
| | - Frank DiPaola
- Division of Pediatric Gastroenterology, Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA
| | | | - Evelyn Hsu
- Division of Gastroenterology and Hepatology, Department of Pediatrics, Seattle Children's Hospital, University of Washington School of Medicine, Seattle, Washington, USA
| | - Katelyn Saarela
- Division of Gastroenterology and Hepatology, Department of Pediatrics, Seattle Children's Hospital, University of Washington School of Medicine, Seattle, Washington, USA
| | - Saeed Mohammad
- Feinberg School of Medicine, The Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University, Chicago, Illinois, USA.,Vanderbilt Unversity Medical Center, Nashville, Tennessee, USA
| | - Riccardo Superina
- Feinberg School of Medicine, The Ann & Robert H. Lurie Children's Hospital of Chicago, Northwestern University, Chicago, Illinois, USA
| | - Sherrie Logan
- Patient and Family Voice, Starzl Network for Excellence in Pediatric Transplantation, Pittsburgh, Pennsylvania, USA
| | - Daniel W Miller
- The Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada
| | | | | | - George Mazariegos
- Children's Hospital of Pittsburgh, UPMC, Pittsburgh, Pennsylvania, USA
| | | |
Collapse
|
14
|
Ong ALK, Knight K, Panettieri V, Dimmock M, Tuan JKL, Tan HQ, Wright C. Predictive modelling for late rectal and urinary toxicities after prostate radiotherapy using planned and delivered dose. Front Oncol 2022; 12:1084311. [PMID: 36591496 PMCID: PMC9800591 DOI: 10.3389/fonc.2022.1084311] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2022] [Accepted: 11/30/2022] [Indexed: 12/23/2022] Open
Abstract
Background and purpose Normal tissue complication probability (NTCP) parameters derived from traditional 3D plans may not be ideal in defining toxicity outcomes for modern radiotherapy techniques. This study aimed to derive parameters of the Lyman-Kutcher-Burman (LKB) NTCP model using prospectively scored clinical data for late gastrointestinal (GI) and genitourinary (GU) toxicities for high-risk prostate cancer patients treated using volumetric-modulated-arc-therapy (VMAT). Dose-volume-histograms (DVH) extracted from planned (DP) and accumulated dose (DA) were used. Material and methods DP and DA obtained from the DVH of 150 prostate cancer patients with pelvic-lymph-nodes irradiation treated using VMAT were used to generate LKB-NTCP parameters using maximum likelihood estimations. Defined GI and GU toxicities were recorded up to 3-years post RT follow-up. Model performance was measured using Hosmer-Lemeshow goodness of fit test and the mean area under the receiver operating characteristics curve (AUC). Bootstrapping method was used for internal validation. Results For mild-severe (Grade ≥1) GI toxicity, the model generated similar parameters based on DA and DP DVH data (DA-D50:71.6 Gy vs DP-D50:73.4; DA-m:0.17 vs DP-m:0.19 and DA/P-n 0.04). The 95% CI for DA-D50 was narrower and achieved an AUC of >0.6. For moderate-severe (Grade ≥2) GI toxicity, DA-D50 parameter was higher and had a narrower 95% CI (DA-D50:77.9 Gy, 95% CI:76.4-79.6 Gy vs DP-D50:74.6, 95% CI:69.1-85.4 Gy) with good model performance (AUC>0.7). For Grade ≥1 late GU toxicity, D50 and n parameters for DA and DP were similar (DA-D50: 58.8 Gy vs DP-D50: 59.5 Gy; DA-n: 0.21 vs DP-n: 0.19) with a low AUC of<0.6. For Grade ≥2 late GU toxicity, similar NTCP parameters were attained from DA and DP DVH data (DA-D50:81.7 Gy vs DP-D50:81.9 Gy; DA-n:0.12 vs DP-n:0.14) with an acceptable AUCs of >0.6. Conclusions The achieved NTCP parameters using modern RT techniques and accounting for organ motion differs from QUANTEC reported parameters. DA-D50 of 77.9 Gy for GI and DA/DP-D50 of 81.7-81.9 Gy for GU demonstrated good predictability in determining the risk of Grade ≥2 toxicities especially for GI derived D50 and are recommended to incorporate as part of the DV planning constraints to guide dose escalation strategies while minimising the risk of toxicity.
Collapse
Affiliation(s)
- Ashley Li Kuan Ong
- Division of Radiation Oncology, National Cancer Centre, Singapore, Singapore,Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC, Australia,*Correspondence: Ashley Li Kuan Ong,
| | - Kellie Knight
- Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC, Australia
| | - Vanessa Panettieri
- Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC, Australia,Alfred Health Radiation Oncology, Alfred Hospital, Melbourne, VIC, Australia
| | - Mathew Dimmock
- Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC, Australia,School of Allied Health Professions, Keele University, Staffordshire, United Kingdom
| | | | - Hong Qi Tan
- Division of Radiation Oncology, National Cancer Centre, Singapore, Singapore
| | - Caroline Wright
- Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC, Australia
| |
Collapse
|
15
|
Girgis A, Bamgboje-Ayodele A, Rincones O, Vinod SK, Avery S, Descallar J, Smith A‘B, Arnold B, Arnold A, Bray V, Durcinoska I, Rankin NM, Chang CF, Eifler B, Elliott S, Hardy C, Ivimey B, Jansens W, Kaadan N, Koh ES, Livio N, Lozenkovski S, McErlean G, Nasser E, Ryan N, Smeal T, Thomas T, Tran T, Wiltshire J, Delaney GP. Stepping into the real world: a mixed-methods evaluation of the implementation of electronic patient reported outcomes in routine lung cancer care. J Patient Rep Outcomes 2022; 6:70. [PMID: 35723827 PMCID: PMC9207870 DOI: 10.1186/s41687-022-00475-6] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2022] [Accepted: 06/03/2022] [Indexed: 11/10/2022] Open
Abstract
Abstract
Background
To realize the broader benefits of electronic patient-reported outcome measures (ePROMs) in routine care, we used the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to inform the translation of a clinically effective ePROM system (hereafter referred to as the PRM system) into practice. The study aimed to evaluate the processes and success of implementing the PRM system in the routine care of patients diagnosed with lung cancer.
Method
A controlled before-and-after mixed-methods study was undertaken. Data sources included a self-report questionnaire and interviews with healthcare providers, electronic health record data for PRMs patients and historical controls, and field notes. Descriptive statistics, logistic regression modelling, negative binomial models, generalized estimating equations and repeated measures ANOVA were used to analyze quantitative data. Qualitative data was thematically analyzed.
Results
A total of 48/79 eligible people diagnosed with lung cancer completed 90 assessments during the 5-month implementation period (RE-AIM reach). Every assessment breached the pre-defined threshold and care coordinators reviewed and actioned 95.6% of breaches, resulting in 146 referrals to allied health services, most frequently for social work (25.3%), dietetics (18.5%), physiotherapy (18.5%) and occupational therapy (17.1%). PRMs patients had significantly fewer visits to the cancer assessment unit for problematic symptoms (M = 0.23 vs. M = 0.43; p = 0.035), and were significantly more likely to be offered referrals (71% vs. 29%, p < 0.0001) than historical controls (RE-AIM effect). The levels of ‘organizational readiness for implementing change’ (ORIC) did not show much differences between baseline and follow-up, though this was already high at baseline; but significantly more staff reported improved confidence when asking patients to complete assessments (64.7% at baseline vs. 88.2% at follow-up, p = 0.0046), and when describing the assessment tool to patients (64.7% at baseline vs. 76.47% at follow-up, p = 0.0018) (RE-AIM adoption). A total of 78 staff received PRM system training, and 95.6% of the PRM system alerts were actioned (RE-AIM implementation); and all lung cancer care coordinators were engaged with the PRM system beyond the end of the study period (RE-AIM maintenance).
Conclusion
This study demonstrates the potential of the PRM system in enhancing the routine care of lung cancer patients, through leveraging the capabilities of automated web-based care options.
Plain English summary
Research has shown the clear benefits of using electronically collected patient-reported outcome measures (ePROMs) for cancer patients and health services. However, we need to better understand how to implement ePROMs as part of routine care. This study evaluated the processes and outcomes of implementing an ePROMs system in the routine care of patients diagnosed with lung cancer. Key findings included: (a) a majority of eligible patients completed the scheduled assessments; (b) patient concerns were identified in every assessment, and care coordinators reviewed and actioned almost all of these, including making significantly more referrals to allied health services; (c) patients completing assessments regularly were less likely to present to the cancer assessment unit with problematic symptoms, suggesting that ePROMs identified patient concerns early and this led to a timely response to concerns; (d) staff training and engagement was high, and staff reporting increased confidence when asking patients to complete assessments and when describing the assessment tool to patients at the end of the implementation period. This study shows that implementing ePROMs in routine care is feasible and can lead to improvements in patient care.
Collapse
|
16
|
Gentizon J, Fleury M, Pilet E, Büla C, Mabire C. Conceptualization and content validation of the MEDication literacy assessment of geriatric patients and informal caregivers (MED-fLAG). J Patient Rep Outcomes 2022; 6:87. [PMID: 35984575 PMCID: PMC9389474 DOI: 10.1186/s41687-022-00495-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Accepted: 08/10/2022] [Indexed: 11/25/2022] Open
Abstract
Background The assessment of patients’ medication literacy skills (i.e., abilities to access, comprehend and interact with medication-related information) is an important step in assisting clinicians to plan for appropriate care. Despite several attempts by researchers to develop measures of medication literacy, an instrument tailored to the specific needs of older adults remains a significant shortfall. Therefore, an interprofessional team that included a citizen co-researcher conceptualized a new standardised measure of medication literacy—the MEDedication Literacy Assessment of Geriatric patients and informal caregivers (MED-fLAG). MED-fLAG was designed as a three-dimensional self-reported measure of functional, interactive and critical skills. This study describes the conceptualization process and provides the results of an evaluation of MED-fLAG’s content validity, acceptability, and feasibility during a hospital stay. Methods MED-fLAG was developed in accordance with the guidance on scale development and standards for good content validity, by using the following steps: (I) conceptualization of a provisional version of MED-fLAG; (II) iterative qualitative evaluation of its content validity by older adults, informal caregivers and healthcare professionals. Results The qualitative assessment of the initial 54-item MED-fLAG was conducted in 36 participants, namely 13 home-dwelling older adults and/or informal caregivers and 23 healthcare professionals. Six rounds of revisions were performed to achieve content validity and to propose a 56-item revised MED-fLAG. Participants reported benefits of using a standardized assessment of medication literacy during a hospital stay but warned about certain limitations and prerequisites. The extent to which MED-fLAG could be integrated into discharge planning needs to be further investigated. Conclusions MED-fLAG is the first medication literacy measure tailored to the specific needs of older patients and informal caregivers. A unique feature of this measure is that it includes prescribed and non-prescribed medications, irrespective of the galenic form. Additional studies are required to evaluate the other measurement properties of MED-fLAG, and to reduce the number of items before considering its clinical application. Supplementary Information The online version contains supplementary material available at 10.1186/s41687-022-00495-2. On the basis of what has been written about medication literacy and the experiences of experts, we developed a new questionnaire to measure medication literacy (MED-fLAG) in older adults and/or informal caregivers. MED-fLAG was then submitted to older adults, informal caregivers and healthcare professionals to retrieve their feedback concerning the relevance, comprehensibility and exhaustiveness of the proposed items. In future, MED-fLAG will allow health professionals to evaluate medication literacy skills in older patients during hospitalization and/or in their informal caregivers when they are responsible for preparing or administering the medications, and then propose individualised support.
Collapse
|
17
|
Surgeon Engagement with Patient-Reported Measures in Australian and Aotearoa New Zealand Bariatric Practices. Obes Surg 2022; 32:3410-3418. [PMID: 35974291 PMCID: PMC9532331 DOI: 10.1007/s11695-022-06237-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Revised: 07/27/2022] [Accepted: 08/04/2022] [Indexed: 10/30/2022]
Abstract
PURPOSE Patient-reported measures are an important emerging metric in outcome monitoring; however, they remain ill-defined and underutilized in bariatric clinical practice. This study aimed to determine the characteristics of patient-reported measures employed in bariatric practices across Australia and Aotearoa New Zealand, including barriers to their implementation and to what extent clinicians are receptive to their use. METHODS An online survey was distributed to all bariatric surgeons actively contributing to the Australian and Aotearoa New Zealand Bariatric Surgery Registry (n = 176). Participants reported their use of patient-reported measures and identified the most important and useful outcomes of patient-reported data for clinical practice. RESULTS Responses from 64 participants reported on 120 public and private bariatric practices across Australia and Aotearoa New Zealand. Most participants reported no collection of any patient-reported measure (39 of 64; 60.9%), citing insufficient staff time or resources as the primary barrier to the collection of both patient-reported experience measures (34 of 102 practices; 33.3%) and patient-reported outcome measures (30 of 84 practices; 35.7%). Participants indicated data collection by the Registry would be useful (47 of 57; 82.5%), highlighting the most valuable application to be a monitoring tool, facilitating increased understanding of patient health needs, increased reporting of symptoms, and enhanced patient-physician communication. CONCLUSION Despite the current lack of patient-reported measures, there is consensus that such data would be valuable in bariatric practices. Widespread collection of patient-reported measures by registries could improve the collective quality of the data, while avoiding implementation barriers faced by individual surgeons and hospitals.
Collapse
|
18
|
Cossio-Gil Y, Omara M, Watson C, Casey J, Chakhunashvili A, Gutiérrez-San Miguel M, Kahlem P, Keuchkerian S, Kirchberger V, Luce-Garnier V, Michiels D, Moro M, Philipp-Jaschek B, Sancini S, Hazelzet J, Stamm T. The Roadmap for Implementing Value-Based Healthcare in European University Hospitals-Consensus Report and Recommendations. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2022; 25:1148-1156. [PMID: 35779941 DOI: 10.1016/j.jval.2021.11.1355] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2021] [Revised: 10/26/2021] [Accepted: 11/03/2021] [Indexed: 05/11/2023]
Abstract
OBJECTIVES Value-based healthcare (VBHC) aims at improving patient outcomes while optimizing the use of hospitals' resources among medical personnel, administrations, and support services through an evidence-based, collaborative approach. In this article, we present a blueprint for the implementation of VBHC in hospitals, based on our experience as members of the European University Hospital Alliance. METHODS The European University Hospital Alliance is a consortium of 9 large hospitals in Europe and aims at increasing the quality and efficiency of care to ultimately drive better outcomes for patients. RESULTS The blueprint describes how to prepare hospitals for VBHC implementation; analyzes gaps, barriers, and facilitators; and explores the most effective ways to turn patient pathways into a process that results in high-value care. Using a patient-centric approach, we identified 4 core minimum components that must be established as cornerstones and 7 organizational enablers to waive the barriers to implementation and ensure sustainability. CONCLUSION The blueprint guides through pathway implementation and establishment of key performance indicators in 6 phases, which hospitals can tailor to their current status on their way to implement VBHC.
Collapse
Affiliation(s)
- Yolima Cossio-Gil
- Department of Information Systems, Vall d'Hebron, Barcelona Hospital Campus, Barcelona, Spain
| | - Maisa Omara
- Section for Outcomes Research, Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Austria and Ludwig Boltzmann Institute for Arthritis and Rehabilitation, Vienna, Austria
| | - Carolina Watson
- Department of Information Systems, Vall d'Hebron, Barcelona Hospital Campus, Barcelona, Spain
| | | | | | | | - Pascal Kahlem
- Scientific Network Management S.L., Barcelona, Spain
| | | | | | | | | | - Matteo Moro
- Chief Medical Office, Ospedale San Raffaele, Milan, Italy
| | | | - Simona Sancini
- Chief Medical Office, Ospedale San Raffaele, Milan, Italy
| | - Jan Hazelzet
- Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Tanja Stamm
- Section for Outcomes Research, Center for Medical Statistics, Informatics and Intelligent Systems, Medical University of Vienna, Austria and Ludwig Boltzmann Institute for Arthritis and Rehabilitation, Vienna, Austria.
| |
Collapse
|
19
|
Robstad N, Westergren T, Mølland E, Abildsnes E, Haraldstad K, Stamnes Köpp UM, Håland ÅT, Fegran L. Experiences of Norwegian child and school health nurses with the "Starting Right™" child health assessment innovation: a qualitative interview study. BMC Health Serv Res 2022; 22:728. [PMID: 35650581 PMCID: PMC9157476 DOI: 10.1186/s12913-022-08088-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2022] [Accepted: 05/13/2022] [Indexed: 11/10/2022] Open
Abstract
Background Although child health services are well established in Norway, the use of information technology for the systematic collection of evidence-based child- and proxy-reported health measures may be beneficial in the early identification of child development problems. The Norwegian “Starting Right™” health service innovation consists of parent- and child-reported online structured health assessments tools, including practical routines for child and school health assessments. The aim of this study was to explore the experiences of child and school health nurses with the Starting Right innovation. Methods We used a qualitative design and conducted three focus group interviews with 18 child and school health nurses from three child health centres one year after the implementation of the innovation. Results The experiences of professionals with the Starting Right innovation were captured by three themes: (1) the digital innovation could be used to obtain a good overview of a child’s health and development; (2) interpreting the questionnaires was a challenge; and (3) implementing the new digital innovation was time-consuming. Conclusions Overall, the child and school health nurses experienced that the Starting Right innovation was useful for providing a comprehensive overview of child development and health. The challenges related to interpreting the parents’ scores and follow-up of children, as well as providing the questionnaires in relevant foreign languages, should be addressed to allow all children and families to be reached.
Collapse
Affiliation(s)
- Nastasja Robstad
- Department of Health and Nursing Science, Faculty of Health and Sports Sciences, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway.
| | - Thomas Westergren
- Department of Health and Nursing Science, Faculty of Health and Sports Sciences, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway.,NORCE, Universitetsveien 19, 4630, Kristiansand, Norway
| | - Eirin Mølland
- NORCE, Universitetsveien 19, 4630, Kristiansand, Norway.,Department of Economics and Finance, School of Business and Law, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway
| | - Eirik Abildsnes
- Kristiansand municipality, P.O. Box 4, 4685, Nodeland, Norway.,Department of Psychosocial Health, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway
| | - Kristin Haraldstad
- Department of Health and Nursing Science, Faculty of Health and Sports Sciences, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway
| | - Unni Mette Stamnes Köpp
- Department of Pediatrics, Sørlandet Hospital, P.O. Box 416, Lundsiden, 4604, Kristiansand, Norway
| | - Åshild Tellefsen Håland
- Department of Health and Nursing Science, Faculty of Health and Sports Sciences, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway.,Department of Pediatrics, Sørlandet Hospital, P.O. Box 416, Lundsiden, 4604, Kristiansand, Norway
| | - Liv Fegran
- Department of Health and Nursing Science, Faculty of Health and Sports Sciences, University of Agder, P.O. Box 422, 4604, Kristiansand, Norway
| |
Collapse
|
20
|
Melly C, McGeehan G, O’Connor N, Johnston A, Bass G, Mohseni S, Donohoe C, Bucholc M, Sugrue M. OUP accepted manuscript. BJS Open 2022; 6:6603491. [PMID: 35668711 PMCID: PMC9171002 DOI: 10.1093/bjsopen/zrac062] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2021] [Revised: 02/24/2022] [Accepted: 04/06/2022] [Indexed: 11/24/2022] Open
Abstract
Background Healthcare requires patient feedback to improve outcomes and experience. This study undertook a systematic review of the depth, variability, and digital suitability of current patient-reported outcome measures (PROMs) in patients undergoing laparoscopic cholecystectomy. Methods A PROSPERO-registered (registration number CRD42021261707) systematic review was undertaken for all relevant English language articles using PubMed version of MEDLINE, Scopus, and Web of Science electronic databases in June 2021. The search used Boolean operators and wildcards and included the keywords: laparoscopic cholecystectomy AND patient outcome OR patient-reported outcome OR patient-reported outcome measure OR PRO OR PROM. Medical Subjects Heading terms were used to search PubMed and Scopus. Articles published from 1 January 2011 to 2 June 2021 were included. Results A total of 4960 individual articles were reviewed in this study, of which 44 were found to evaluate PROMs in patients undergoing laparoscopic cholecystectomy and underwent methodological index for non-randomized studies (MINORS) grading. Twenty-one articles spanning 19 countries and four continents met all inclusion criteria and were included in the qualitative data synthesis. There was significant heterogeneity in PROMs identified with eight different comprehensive PROM tools used in the 21 studies. There was wide variation in the time points at which PROMs were recorded. Fourteen of 21 studies recorded PROMs before and after surgery, and 7 of 21 recorded PROMs only after surgery. Follow-up intervals ranged from 3 days to 2 years after surgery. Conclusions This study identified that while post-laparoscopic cholecystectomy PROMs are infrequently measured currently, tools are widely available to achieve this in clinical practice. PROMs may not capture all the outcomes but should be incorporated into future cholecystectomy outcome research. The EQ-5D™ (EuroQoL Group, Rotterdam, the Netherlands) provides a simple platform for the modern digital era.
Collapse
Affiliation(s)
| | - Gearoid McGeehan
- Department of Surgery, Letterkenny University Hospital and Donegal Clinical Research Academy, Donegal, Ireland
- University of Limerick School of Medicine, University of Limerick, Limerick, Ireland
| | - Niall O’Connor
- Department of Surgery, Letterkenny University Hospital and Donegal Clinical Research Academy, Donegal, Ireland
| | - Alison Johnston
- Department of Surgery, Letterkenny University Hospital and Donegal Clinical Research Academy, Donegal, Ireland
| | - Gary Bass
- Division of Traumatology, Emergency Surgery and Surgical Critical Care, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- School of Medical Sciences, Orebro University, Orebro, Sweden
| | - Shahin Mohseni
- Division of Trauma and Emergency Surgery, Department of Surgery Orebro University Hospital, & School of Medical Sciences, Orebro University, Orebro, Sweden
| | - Claire Donohoe
- Department of Surgery, Trinity College Dublin, St James’ Hospital, Dublin, Ireland
| | - Magda Bucholc
- Intelligent Systems Research Centre, School of Computing, Engineering and Intelligent Systems, Ulster University, Magee Campus, Derry-Londonderry, UK
| | - Michael Sugrue
- Department of Surgery, Letterkenny University Hospital and Donegal Clinical Research Academy, Donegal, Ireland
- EU INTERREG Centre for Personalized Medicine, Intelligent Systems Research Centre, School of Computing, Engineering and Intelligent Systems, Ulster University, Magee Campus, Derry-Londonderry, UK
| |
Collapse
|