1
|
Mirpuri E, García-Álvarez L, Acín-Gericó MT, Bartolomé B, Delgado Bolton RC, San-Martín M, Vivanco L. Characterization of Factors Predicting a Favorable Opinion of Research Applications Submitted for an Ethical Review Process. Front Med (Lausanne) 2022; 9:878786. [PMID: 35783641 PMCID: PMC9243650 DOI: 10.3389/fmed.2022.878786] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Accepted: 05/30/2022] [Indexed: 11/13/2022] Open
Abstract
IntroductionIn Spain, biomedical research applications must receive a positive ethical opinion from Research Ethics Committees (RECs) before being executed. There is limited information on how to optimize the ethical review process to reduce delays. This study was performed to characterize variables predicting favorable opinions at the first ethical review performed by a REC.Material and MethodsThe study assessed all research applications revised by a REC in 2019–2020. Data was extracted from REC's database of La Rioja, Spain. Variables collected covered three areas: (i) principal investigator's profile; (ii) study design; and (iii) ethical review process. A model based on multiple logistic regression analysis was created to identify variables explaining favorable opinions in first rounds of ethical review processes.ResultsThe sample included 125 applications (41 submitted in 2019, and 84 in 2020). At the first review, nine (7%) applications were rejected, 56 (45%) were approved, and the remaining 60 (48%) required at least two reviews prior to approval. When comparing both years, a 2-fold increase in the number of applications submitted, and a difference in the ratio of applications with a favorable vs. non-favorable opinion were observed. Furthermore, a model predicted 71% of probability of obtaining a favorable opinion in the first ethical review. Three variables appeared as being explanatory: if the principal investigator is either the group leader or the department's head (OR = 17.39; p < 0.001), and if the informed consent (OR = 11.79; p = 0.01), and methods and procedures (OR = 34.15; p < 0.001) are well done.ConclusionsThese findings confirm an increase in the number of submissions and a difference in the ratio of applications approved by year. Findings observed also confirm deficiencies in “informed consent” and in “methods and procedures” are the two main causes of delay for favorable ethical opinions. Additionally, findings highlight the need that group leaders and heads of departments should be more involved in guiding and supervising their research teams, especially when research applications are led by less experienced researchers. Based on these findings, it is suggested that an adequate mentoring and targeted training in research could derive in more robust research applications and in smoother ethical review processes.
Collapse
Affiliation(s)
- Eduardo Mirpuri
- Research Ethics Committee of La Rioja (CEImLAR), Rioja Health Foundation, Logroño, Spain
| | - Lara García-Álvarez
- Research Ethics Committee of La Rioja (CEImLAR), Rioja Health Foundation, Logroño, Spain
- Unit for Clinical Research Support, Center for Biomedical Research of La Rioja (CIBIR), Logroño, Spain
| | - María Teresa Acín-Gericó
- Research Ethics Committee of La Rioja (CEImLAR), Rioja Health Foundation, Logroño, Spain
- Subdirectorate of Pharmacy and Provisions, Navarre Health Service, Pamplona, Spain
| | - Blanca Bartolomé
- Platform of Bioethics and Medical Education, Center for Biomedical Research of La Rioja (CIBIR), Logroño, Spain
- National Centre of Documentation on Bioethics, Rioja Health Foundation, Logroño, Spain
| | - Roberto C. Delgado Bolton
- Platform of Bioethics and Medical Education, Center for Biomedical Research of La Rioja (CIBIR), Logroño, Spain
- National Centre of Documentation on Bioethics, Rioja Health Foundation, Logroño, Spain
- Department of Diagnostic Imaging (Radiology) and Nuclear Medicine, University Hospital San Pedro, Logroño, Spain
| | - Montserrat San-Martín
- Platform of Bioethics and Medical Education, Center for Biomedical Research of La Rioja (CIBIR), Logroño, Spain
- Department of Statistics and Operational Research, University of Granada, Melilla, Spain
- Scientific Computing & Technological Innovation Center (SCoTIC), University of La Rioja, Logroño, Spain
| | - Luis Vivanco
- Research Ethics Committee of La Rioja (CEImLAR), Rioja Health Foundation, Logroño, Spain
- Platform of Bioethics and Medical Education, Center for Biomedical Research of La Rioja (CIBIR), Logroño, Spain
- National Centre of Documentation on Bioethics, Rioja Health Foundation, Logroño, Spain
- *Correspondence: Luis Vivanco
| |
Collapse
|
2
|
Seykora A, Coleman C, Rosenfeld SJ, Bierer BE, Lynch HF. Steps toward a System of IRB Precedent: Piloting Approaches to Summarizing IRB Decisions for Future Use. Ethics Hum Res 2021; 43:2-18. [PMID: 34676693 DOI: 10.1002/eahr.500106] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Institutional review boards (IRBs) have been criticized for inconsistency and lack of transparency in decision-making, problems that undermine both trust in their ability to protect human research participants and respect for their decisions among researchers. The absence of robust documentation of their decisions and the inability or unwillingness to share those decisions together represent a missed opportunity for IRBs to learn from one another and advance debates about challenging ethical issues. The concept of IRB precedent, modeled upon the system of legal precedent, has been proposed as a potential solution to these problems. In theory, an IRB faced with a review decision could look back at previous IRB decisions, either its own or those of other boards, made in similar studies or circumstances to guide the present decision. Some IRBs attempt this informally within their institution, but few examples of a structured system of IRB precedent have been described in the literature, and none has been widely adopted. This article describes a pilot project to summarize IRB decisions in a way that could facilitate their use as precedent by creating a documentation tool that meets four criteria-comprehensiveness, validity, searchability, and efficiency. Though this process turned out to be more challenging than expected, we identified key features of such a tool that holds promise for future development and could promote more consistent, robust IRB decision-making and advance discourse in human research ethics.
Collapse
Affiliation(s)
- Andrea Seykora
- Research compliance manager at Kaiser Permanente Northwest at the time of submission
| | | | | | - Barbara E Bierer
- Faculty director at the Multi-Regional Clinical Trials Center of Brigham and Women's Hospital and Harvard and a professor of medicine at Harvard Medical School and Brigham and Women's Hospital
| | - Holly Fernandez Lynch
- John Russell Dickson, MD, Presidential Assistant Professor of Medical Ethics at Perelman School of Medicine at the University of Pennsylvania
| |
Collapse
|
3
|
Abstract
A standard of evidence is a rule or norm pertaining to the type or amount of evidence that is required to prove or support a conclusion. Standards of evidence play an important role in institutional review board (IRB) decision-making, but they are not mentioned in the federal research regulations. In this article, I examine IRB standards of evidence from a normative, epistemological perspective and argue that IRBs should rely on empirical evidence for making decisions, but that other sources of evidence, such as intuition, emotion, and rational reflection, can also play an important role in decision-making, because IRB decisions involve an ethical component which is not reducible to science. I also argue that an IRB should approve a study only if it has clear and convincing evidence that the study meets all the approval criteria and other relevant, ethical considerations; and that for studies which expose healthy volunteers to significant risks, an IRB should require that evidence be more than clear and convincing as a condition for approval. Additional empirical research is needed on how IRBs use evidence to make decisions and how standards of evidence influence IRB decision-making at the individual and group level.
Collapse
Affiliation(s)
- David B Resnik
- National Institute of Environmental Health Sciences, National Institutes of Health, Bethesda, MD, USA
| |
Collapse
|
4
|
Hodge J, Foley S, Brankaert R, Kenning G, Lazar A, Boger J, Morrissey K. Relational, Flexible, Everyday: Learning from Ethics in Dementia Research. PROCEEDINGS OF THE SIGCHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS. CHI CONFERENCE 2020; 2020. [PMID: 32705092 DOI: 10.1145/3313831.3376627] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Engaging in participatory research in HCI raises numerous ethical complexities such as consent, researcher relationships, and participant compensation. Doing HCI work in the area of dementia amplifies these issues, and researchers in this area are modelling ethical stances to ensure researcher-participant relationships focus on meaningful engagement and care. This paper presents an insight into the kinds of ethical foci required when doing design research with people living with dementia and their carers. We interviewed 22 HCI researchers with experience working in dementia care contexts. Our qualitative analysis outlines subsequent lessons-learned, such as recognition of the participants, self-care, research impact, and subjectivity in ethical review boards. Furthermore, we found the complexity of navigating both "everyday" and more formal, institutional ethics in dementia research has implications beyond the context of working with people with dementia and outline key considerations for ethical practices in socially orientated HCI research.
Collapse
Affiliation(s)
- James Hodge
- Open Lab, Newcastle University, Newcastle upon Tyne, UK
| | - Sarah Foley
- School of Applied Psychology, University College Cork, Ireland
| | - Rens Brankaert
- Industrial Design, University of Technology & Institute of Allied Health Professions, Fontys University of Applied Sciences, Eindhoven, Netherlands
| | | | - Amanda Lazar
- College of Information Studies, University of Maryland, College Park, Maryland, United States
| | | | | |
Collapse
|
5
|
Friesen P, Yusof ANM, Sheehan M. Should the Decisions of Institutional Review Boards Be Consistent? Ethics Hum Res 2020; 41:2-14. [PMID: 31336039 DOI: 10.1002/eahr.500022] [Citation(s) in RCA: 19] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
In response to increasing concerns regarding inconsistency in the decision-making of institutional review boards (IRBs), we introduce the decision-maker's dilemma, which arises when complex, normative decisions must be made regularly. Those faced with such decisions can either develop a process of algorithmic decision-making, in which consistency is ensured but many morally relevant factors are excluded from the process, or embrace discretionary decision-making, which makes space for morally relevant factors to shape decisions but leads to decisions that are inconsistent. Based on an exploration of similarities between systems of criminal sentencing and of research ethics review, we argue for a discretionary system of decision-making, even though it leads to more inconsistency than does an algorithmic system. We conclude with a discussion of some safeguards that could improve consistency while still making space for discretion to enter IRBs' decision-making processes.
Collapse
Affiliation(s)
- Phoebe Friesen
- Postdoctoral fellow at the Ethox Centre at the University of Oxford
| | | | - Mark Sheehan
- Oxford Biomedical Research Centre ethics fellow at the Ethox Centre at the University of Oxford
| |
Collapse
|
6
|
Vadeboncoeur C, Foster C, Townsend N. Challenges of research recruitment in a university setting in England. Health Promot Int 2019; 33:878-886. [PMID: 28531308 DOI: 10.1093/heapro/dax025] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
The recruitment is an integral part of most research projects in medical sciences involving human participants. In health promotion research, there is increasing work on the impact of environments. Settings represent environments such as schools where social, physical and psychological development unfolds. In this study, we investigated weight gain in students within a university setting. Barriers to access and recruitment of university students within a specific setting, in the context of health research are discussed. An online survey on health behaviours of first year students across 101 universities in England was developed. Ethics committees of each institutions were contacted to obtain permission to recruit and access their students. Recruitment adverts were standardized and distributed within restrictions imposed by universities. Three time points and incentives were used. Several challenges in recruiting from a university setting were found. These included (i) ethics approval, (ii) recruitment approval, (iii) navigating restrictions on advertisement and (iv) logistics of varying university academic calendars. We also faced challenges of online surveys including low recruitment, retention and low eligibility of respondents. From the 101 universities, 28 allowed dissemination of adverts. We obtained 1026 responses at T1, 599 at T2 and 497 at T3. The complete-case sample represented 13% of those originally recruited at T1. Conducting research on students within the university setting is a time consuming and challenging task. To improve research-based health promotion, universities could work together to increase consistency as to their policies on student recruitment.
Collapse
Affiliation(s)
- Claudia Vadeboncoeur
- British Heart Foundation Centre on Population Approaches for Non-Communicable Disease Prevention, Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Charlie Foster
- British Heart Foundation Centre on Population Approaches for Non-Communicable Disease Prevention, Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Nick Townsend
- British Heart Foundation Centre on Population Approaches for Non-Communicable Disease Prevention, Nuffield Department of Population Health, University of Oxford, Oxford, UK
| |
Collapse
|
7
|
Sheehan M, Dunn M, Sahan K. In defence of governance: ethics review and social research. JOURNAL OF MEDICAL ETHICS 2018; 44:710-716. [PMID: 29018176 DOI: 10.1136/medethics-2017-104443] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2017] [Revised: 07/31/2017] [Accepted: 09/24/2017] [Indexed: 06/07/2023]
Abstract
There is a growing body of literature that has sought to undermine systems of ethical regulation, and governance more generally, within the social sciences. In this paper, we argue that any general claim for a system of research ethics governance in social research depends on clarifying the nature of the stake that society has in research. We show that certain accounts of this stake-protecting researchers' freedoms; ensuring accountability for resources; safeguarding welfare; and supporting democracy-raise relevant ethical considerations that are reasonably contested. However, these accounts cannot underpin a general claim in favour of, or against, a system of research ethics governance. Instead, we defend governance in social research on the grounds that research, as an institutionalised form of enquiry, is a constitutive element of human flourishing, and that society ought to be concerned with the flourishing of its members. We conclude by considering the governance arrangements that follow from, and are justified by, our arguments.
Collapse
Affiliation(s)
- Mark Sheehan
- Nuffield Department of Population Health, The Ethox Centre, University of Oxford, Oxford, UK
| | - Michael Dunn
- Nuffield Department of Population Health, The Ethox Centre, University of Oxford, Oxford, UK
| | - Kate Sahan
- Nuffield Department of Population Health, The Ethox Centre, University of Oxford, Oxford, UK
| |
Collapse
|
8
|
Abstract
When considering submissions ethics committees should be consistent in all aspects of their review. A wide variation in performance is likely to result in the unfair dismissal of good research on the one hand with inadequate ethical review on the other, neither of which is acceptable. The recent annual reports for UK MRECs suggest that the level of unfavourable opinion ranges from 6.9% to 24.2% Although a certain level of inconsistency is inherent in the system of ethical review there is little research to determine how much of that inconsistency in inevitable and how much of it is the result of poor process. The time has come for high quality research to be done to assess the consistency and hence the validity of ethical review.
Collapse
|
9
|
Moore A, Donnelly A. The job of 'ethics committees'. JOURNAL OF MEDICAL ETHICS 2018; 44:481-487. [PMID: 26566944 DOI: 10.1136/medethics-2015-102688] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/05/2015] [Revised: 09/22/2015] [Accepted: 10/21/2015] [Indexed: 06/05/2023]
Abstract
What should authorities establish as the job of ethics committees and review boards? Two answers are: (1) review of proposals for consistency with the duly established and applicable code and (2) review of proposals for ethical acceptability. The present paper argues that these two jobs come apart in principle and in practice. On grounds of practicality, publicity and separation of powers, it argues that the relevant authorities do better to establish code-consistency review and not ethics-consistency review. It also rebuts bad code and independence arguments for the opposite view. It then argues that authorities at present variously specify both code-consistency and ethics-consistency jobs, but most are also unclear on this issue. The paper then argues that they should reform the job of review boards and ethics committees, by clearly establishing code-consistency review and disestablishing ethics-consistency review, and through related reform of the basic orientation, focus, name, and expertise profile of these bodies and their actions.
Collapse
|
10
|
Ning N, Yan J, Dietrich MF, Xie XJ, Gerber DE. Institutional Scientific Review of Cancer Clinical Research Protocols: A Unique Requirement That Affects Activation Timelines. J Oncol Pract 2017; 13:e982-e991. [PMID: 29019706 PMCID: PMC5728362 DOI: 10.1200/jop.2017.024299] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/30/2022] Open
Abstract
PURPOSE The National Cancer Institute (NCI) requirement that clinical trials at NCI-designated cancer centers undergo institutional scientific review in addition to institutional review board evaluation is unique among medical specialties. We sought to evaluate the effect of this process on protocol activation timelines. METHODS We analyzed oncology clinical trials that underwent full board review by the Harold C. Simmons Comprehensive Cancer Center Protocol Review and Monitoring Committee (PRMC) from January 1, 2009, through June 30, 2013. We analyzed associations between trial characteristics, PRMC decisions, protocol modifications, and process timelines using the χ2 test, Fisher's exact test, Wilcoxon rank sum test, Kruskal-Wallis test, and logistic regression. RESULTS A total of 226 trials were analyzed. Of these, 77% were industry sponsored and 23% were investigator initiated. The median time from submission to PRMC approval was 55 days. The length of review was associated with trial phase, timing of approval, and number of committee changes/clarifications requested. The median process time was 35 days for those approved at first decision, 68 days for second decision, and 116 days for third decision ( P < .001). The median process time was 39 days if no changes/clarifications were requested, 64 days for one to three changes/clarifications, and 73 days for four or more changes/clarifications ( P < .001). Requested changes/clarifications had a greater effect on industry-sponsored trials than on investigator-initiated trials. CONCLUSION NCI-mandated institutional scientific review of oncology clinical trials contributes substantially to protocol activation timelines. Further evaluation of this process and the value added to research quality is warranted.
Collapse
Affiliation(s)
- Ning Ning
- University of Texas Southwestern Medical Center, Dallas, TX
| | - Jingsheng Yan
- University of Texas Southwestern Medical Center, Dallas, TX
| | | | - Xian-Jin Xie
- University of Texas Southwestern Medical Center, Dallas, TX
| | - David E. Gerber
- University of Texas Southwestern Medical Center, Dallas, TX,Corresponding author: David E. Gerber, MD, Division of Hematology-Oncology, Harold C. Simmons Comprehensive Cancer Center, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Mail Code 8852, Dallas, TX 75390-8852; e-mail:
| |
Collapse
|
11
|
Grinnell F, Sadler JZ, McNamara V, Senetar K, Reisch J. Confidence of IRB/REC Members in Their Assessments of Human Research Risk: A Study of IRB/REC Decision Making in Action. J Empir Res Hum Res Ethics 2017; 12:140-149. [DOI: 10.1177/1556264617710386] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Understanding how institutional review boards/research ethics committees (IRBs/RECs) perform risk/benefit assessment is important to help improve their function. In environmental ethics, uncertainty about potential outcomes and the precautionary principle play important roles in regulatory oversight but have received little attention in the context of human research ethics. We carried out an empirical study to gain insight into uncertainty by asking IRB/REC members about confidence in their risk assessments immediately after discussion of new protocols under review. Based on 12 meetings carried out by four IRBs/RECs over a 6-month period, we found a robust, inverse relationship between risk and confidence. As risk increased, confidence decreased. We detected different patterns of consensus between different IRBs/RECs and their members. Our study introduces a novel and relatively easy to implement approach to begin to understand IRB/REC decision making in real time that can be used within or across institutions.
Collapse
Affiliation(s)
| | | | | | | | - Joan Reisch
- UT Southwestern Medical Center, Dallas, TX, USA
| |
Collapse
|
12
|
Williamson L. Empirical assessments of clinical ethics services: implications for clinical ethics committees. ACTA ACUST UNITED AC 2016; 2:187-192. [PMID: 18677421 DOI: 10.1258/147775007783560184] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The need to evaluate the performance of clinical ethics services is widely acknowledged although work in this area is more developed in the United States. In the USA many studies that assess clinical ethics services have utilized empirical methods and assessment criteria. The value of these approaches is thought to rest on their ability to measure the value of services in a demonstrable fashion. However, empirical measures tend to lack ethical content, making their contribution to developments in ethical governance unclear. The steady increase of clinical ethics committees in the UK must be accompanied by efforts to evaluate their performance. As part of this evaluative work it is important to examine how the practice of clinical ethics committees can be informed by empirical measures.
Collapse
|
13
|
Vadeboncoeur C, Townsend N, Foster C, Sheehan M. Variation in university research ethics review: Reflections following an inter-university study in England. RESEARCH ETHICS REVIEW 2016. [DOI: 10.1177/1747016116652650] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Conducting large multi-site research within universities highlights inconsistencies between universities in approaches, requirements and responses of research ethics committees. Within the context of a social science research study, we attempted to obtain ethical approval from 101 universities across England to recruit students for a short online survey. We received varied responses from research ethics committees of different universities with the steps to obtaining ethics approval ranging from those that only required proof of approval from our home institution, to universities that facilitated fast-track applications to those that required a full ethics review. Some universities also completely refused. After contacting all 101 universities in England, 60 universities gave clearance to our study. In this article, we present the different approaches universities adopted in response to our application to sample from students in their institution. We consider a number of conceptual and ethical issues pertinent to considering ethics approval for researchers from other universities in England and critically discuss three possible models of ethics governance that would cover all universities in England.
Collapse
Affiliation(s)
| | - Nick Townsend
- Nuffield Department of Population Health, University of Oxford, UK
| | - Charlie Foster
- Nuffield Department of Population Health, University of Oxford, UK
| | - Mark Sheehan
- Nuffield Department of Population Health, University of Oxford, UK
| |
Collapse
|
14
|
Ning N, Yan J, Xie XJ, Gerber DE. Impact of NCI-mandated scientific review on protocol development and content. J Natl Compr Canc Netw 2016; 13:409-16. [PMID: 25870377 DOI: 10.6004/jnccn.2015.0056] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
PURPOSE The NCI requirement that clinical trials at NCI-designated cancer centers undergo scientific review in addition to Institutional Review Board review is unique among medical specialties. We evaluated the impact of this process on protocol development and content. METHODS We analyzed cancer clinical trials that underwent full board review by the Harold C. Simmons Cancer Center Protocol Review and Monitoring Committee (PRMC) from January 1, 2009, through June 30, 2013. We analyzed associations between trial characteristics, PRMC decisions, and protocol modifications using Chi-square testing, Fishers exact testing, and logistic regression. RESULTS A total of 226 trials were analyzed. Of these, 77% were industry-sponsored and 23% were investigator-initiated. Initial PRMC decisions were: approval (40%), provisional approval (52%), deferral (7%), and disapproval (1%). These decisions were associated with study sponsor (P<.001) and phase (P<.001). Ultimately, 97% of industry-sponsored and 90% of investigator-initiated trials were approved (P=.05). Changes were requested for 27% of industry-sponsored trials compared with 54% of investigator-initiated trials (P<.001). Total changes requested (mean, 5.6 vs 2.4; P<.001) and implemented (mean, 4.6 vs 2.1; P=.008) per protocol were significantly greater for investigator-initiated trials. Changes related to study design were more commonly requested (35% vs 13% of trials) and implemented (40% vs 5% of trials) for investigator-initiated trials compared with industry-sponsored trials (P=.03). CONCLUSIONS NCI-mandated scientific protocol review seems to have a substantial impact on investigator-initiated trials but less effect on industry-sponsored trials. These findings may provide guidance on development and prioritization of institutional protocol review policies.
Collapse
Affiliation(s)
- Ning Ning
- From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas
| | - Jingsheng Yan
- From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas
| | - Xian-Jin Xie
- From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas. From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas
| | - David E Gerber
- From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas. From School of Medicine, Harold C. Simmons Cancer Center, Department of Clinical Sciences, and Department of Internal Medicine, Division of Hematology-Oncology, University of Texas Southwestern Medical Center, Dallas, Texas
| |
Collapse
|
15
|
Gallagher B, Berman AH, Bieganski J, Jones AD, Foca L, Raikes B, Schiratzki J, Urban M, Ullman S. National Human Research Ethics: A Preliminary Comparative Case Study of Germany, Great Britain, Romania, and Sweden. ETHICS & BEHAVIOR 2015; 26:586-606. [PMID: 27746664 PMCID: PMC5044765 DOI: 10.1080/10508422.2015.1096207] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
Although international research is increasing in volume and importance, there remains a dearth of knowledge on similarities and differences in “national human research ethics” (NHREs), that is, national ethical guidelines (NEGs), Institutional Review Boards (IRBs), and research stakeholder’ ethical attitudes and behaviors (EABs). We begin to address this situation by reporting upon our experiences in conducting a multinational study into the mental health of children who had a parent/carer in prison. The study was conducted in 4 countries: Germany, Great Britain, Romania, and Sweden. Data on NHREs were gathered via a questionnaire survey, two ethics-related seminars, and ongoing contact between members of the research consortium. There was correspondence but even more so divergence between countries in the availability of NEGs and IRBs and in researcher’ EABs. Differences in NHREs have implications particularly in terms of harmonization but also for ethical philosophy and practice and for research integrity.
Collapse
Affiliation(s)
| | - Anne H Berman
- Department of Clinical Neurosciences, Karolinska Institutet
| | | | - Adele D Jones
- School of Human and Health Sciences, University of Huddersfield
| | - Liliana Foca
- Department of Psychology, Asociația Alternative Sociale
| | - Ben Raikes
- Division of Social Work, University of Huddersfield
| | | | - Mirjam Urban
- Department of Medicine, Technische Universitaet Dresden
| | - Sara Ullman
- Department of Investigation, The Swedish Police
| |
Collapse
|
16
|
Halila R. Evaluation of the work of hospital districts' research ethics committees in Finland. JOURNAL OF MEDICAL ETHICS 2014; 40:866-868. [PMID: 25246637 DOI: 10.1136/medethics-2012-101313] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
The main task of research ethics committees (RECs) is to assess research studies before their start. In this study, 24 RECs that evaluate medical research were sent questionnaires about their structure and functions. The RECs were divided into two separate groups: those working in university hospital districts (uRECs) and those in central hospital districts (non-uRECs). The two groups were different in many respects: the uRECs were bigger in size, covered a wider range of disciplines (both medical and non-medical), had better resources and more frequent and regular meetings. After the survey was performed and analysed, the Medical Research Act was amended so that only hospital districts with a medical faculty in their region had a duty to establish ethics committees. After the amendment, the number of RECs evaluating medical research in Finland decreased from 25 to 9. The ethics committees that remained had wider expertise and were better equipped already by the time of this survey. Only one non-uREC was continuing its work, and this was being done under the governance of a university hospital district. Simple measures were used for qualitative analysis of the work of RECs that evaluate medical research. These showed differences between RECs. This may be helpful in establishing an ethics committee network in a research field or administrational area.
Collapse
|
17
|
Taljaard M, Brehaut JC, Weijer C, Boruch R, Donner A, Eccles MP, McRae AD, Saginur R, Zwarenstein M, Grimshaw JM. Variability in research ethics review of cluster randomized trials: a scenario-based survey in three countries. Trials 2014; 15:48. [PMID: 24495542 PMCID: PMC3925119 DOI: 10.1186/1745-6215-15-48] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2013] [Accepted: 01/21/2014] [Indexed: 11/25/2022] Open
Abstract
Background Cluster randomized trials (CRTs) present unique ethical challenges. In the absence of a uniform standard for their ethical design and conduct, problems such as variability in procedures and requirements by different research ethics committees will persist. We aimed to assess the need for ethics guidelines for CRTs among research ethics chairs internationally, investigate variability in procedures for research ethics review of CRTs within and among countries, and elicit research ethics chairs’ perspectives on specific ethical issues in CRTs, including the identification of research subjects. The proper identification of research subjects is a necessary requirement in the research ethics review process, to help ensure, on the one hand, that subjects are protected from harm and exploitation, and on the other, that reviews of CRTs are completed efficiently. Methods A web-based survey with closed- and open-ended questions was administered to research ethics chairs in Canada, the United States, and the United Kingdom. The survey presented three scenarios of CRTs involving cluster-level, professional-level, and individual-level interventions. For each scenario, a series of questions was posed with respect to the type of review required (full, expedited, or no review) and the identification of research subjects at cluster and individual levels. Results A total of 189 (35%) of 542 chairs responded. Overall, 144 (84%, 95% CI 79 to 90%) agreed or strongly agreed that there is a need for ethics guidelines for CRTs and 158 (92%, 95% CI 88 to 96%) agreed or strongly agreed that research ethics committees could be better informed about distinct ethical issues surrounding CRTs. There was considerable variability among research ethics chairs with respect to the type of review required, as well as the identification of research subjects. The cluster-cluster and professional-cluster scenarios produced the most disagreement. Conclusions Research ethics committees identified a clear need for ethics guidelines for CRTs and education about distinct ethical issues in CRTs. There is disagreement among committees, even within the same countries, with respect to key questions in the ethics review of CRTs. This disagreement reflects variability of opinion and practices pointing toward possible gaps in knowledge, and supports the need for explicit guidelines for the ethical conduct and review of CRTs.
Collapse
Affiliation(s)
- Monica Taljaard
- Ottawa Hospital Research Institute, Clinical Epidemiology Program, 1053 Carling Avenue, Civic Campus, C409, Ottawa, ON K1Y 4E9, Canada.
| | | | | | | | | | | | | | | | | | | |
Collapse
|
18
|
Jaspers P, Houtepen R, Horstman K. Ethical review: Standardizing procedures and local shaping of ethical review practices. Soc Sci Med 2013; 98:311-8. [DOI: 10.1016/j.socscimed.2013.03.043] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2012] [Revised: 12/18/2012] [Accepted: 03/20/2013] [Indexed: 10/26/2022]
|
19
|
Varga O. Critical Analysis of Assessment Studies of the Animal Ethics Review Process. Animals (Basel) 2013; 3:907-22. [PMID: 26479540 PMCID: PMC4494455 DOI: 10.3390/ani3030907] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2013] [Revised: 08/27/2013] [Accepted: 08/29/2013] [Indexed: 11/27/2022] Open
Abstract
In many countries the approval of animal research projects depends on the decisions of Animal Ethics Committees (AEC's), which review the projects. An animal ethics review is performed as part of the authorization process and therefore performed routinely, but comprehensive information about how well the review system works is not available. This paper reviews studies that assess the performance of animal ethics committees by using Donabedian's structure-process-outcome model. The paper points out that it is well recognised that AECs differ in structure, in their decision-making methods, in the time they take to review proposals and that they also make inconsistent decisions. On the other hand, we know little about the quality of outcomes, and to what extent decisions have been incorporated into daily scientific activity, and we know almost nothing about how well AECs work from the animal protection point of view. In order to emphasise this viewpoint in the assessment of AECs, the paper provides an example of measures for outcome assessment. The animal suffering is considered as a potential measure for outcome assessment of the ethics review. Although this approach has limitations, outcome assessment would significantly increase our understanding of the performance of AECs.
Collapse
Affiliation(s)
- Orsolya Varga
- Laboratory Animal Science Group, IBMC (Instituto de Biologia Molecular e Celular), Universidade do Porto, Rua do Campo Alegre 823, 4150-180 Porto, Portugal.
| |
Collapse
|
20
|
|
21
|
Anderson EE, DuBois JM. IRB decision-making with imperfect knowledge: a framework for evidence-based research ethics review. THE JOURNAL OF LAW, MEDICINE & ETHICS : A JOURNAL OF THE AMERICAN SOCIETY OF LAW, MEDICINE & ETHICS 2012; 40:951-969. [PMID: 23289698 DOI: 10.1111/j.1748-720x.2012.00724.x] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
Here we describe the five steps of evidence-based practice as applied to research ethics review and apply these steps to three exemplar dilemmas: incentive payments in substance abuse research; informed consent for biobanking; and placebo-controlled trials involving pregnant women in order to demonstrate the potential of empirical data to inform and improve IRB decision-making.
Collapse
Affiliation(s)
- Emily E Anderson
- Neiswanger Institute for Bioethics, Health Sciences Division, Loyola University Chicago, USA
| | | |
Collapse
|
22
|
Abstract
For decades, scholars in the social sciences and humanities have questioned the appropriateness and utility of prior review of their research by human subjects' ethics committees. This essay seeks to organize thematically some of their published complaints and to serve as a brief restatement of the major critiques of ethics review. In particular, it argues that 1) ethics committees impose silly restrictions, 2) ethics review is a solution in search of a problem, 3) ethics committees lack expertise, 4) ethics committees apply inappropriate principles, 5) ethics review harms the innocent, and 6) better options exist.
Collapse
Affiliation(s)
- Zachary M. Schrag
- Department of History and Art History. George Mason University, Fairfax, VA, USA
| |
Collapse
|
23
|
Waiver of informed consent in prehospital emergency health research in Australia. Monash Bioeth Rev 2011; 29:07.1-16. [PMID: 22031986 DOI: 10.1007/bf03351323] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
Informed consent is a vital part of ethical research. In emergency health care research environments such as ambulance services and emergency departments, it is sometimes necessary to conduct trial interventions or observations without patient consent. At times where treatment is time critical, it may be impossible or inappropriate to seek consent from next of kin. Emergency medicine is one of the few areas where the process of informed consent can be waived to allow research to proceed without patient consent. This article will explore the ethics of informed consent in the prehospital emergency research context. This will include an overview of current Australian guidelines for ethical research, and recent changes in law internationally which have affected the conduct of international emergency health research. An overview of the ethical reasoning behind the waiver of informed consent in emergency research is presented, also addressing issues relating to emergency health research such as proxy consent, unconscious patients, and patient decision making capacity. The unusual circumstances encountered in the prehospital ambulance environment will also be discussed, including the dependent and coercive relationship between patients and ambulance professionals, and a lack of alternatives for care and transport for patients who refuse consent. The conflict arising from differences in medical culture and values between patients and health care professionals will also briefly be discussed. It will be argued that, while emergency care research should not require informed consent due to the restrictions of time and dependent nature of the relationship between patient and health professional, emergency health researchers still have a responsibility to consider the patients' perspective when considering the ethical issues of an emergency research project, particularly in the prehospital environment.
Collapse
|
24
|
Abbott L, Grady C. A systematic review of the empirical literature evaluating IRBs: what we know and what we still need to learn. J Empir Res Hum Res Ethics 2011; 6:3-19. [PMID: 21460582 DOI: 10.1525/jer.2011.6.1.3] [Citation(s) in RCA: 159] [Impact Index Per Article: 12.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Institutional review boards (IRBs) are integral to the U.S. system of protection of human research participants. Evaluation of IRBs, although difficult, is essential. To date, no systematic review of IRB studies has been published. We conducted a systematic review of empirical studies of U.S. IRBs to determine what is known about the function of IRBs and to identify gaps in knowledge. A structured search in PubMed identified forty-three empirical studies evaluating U.S. IRBs. Studies were included if they reported an empirical investigation of the structure, process, outcomes, effectiveness, or variation of U.S. IRBs. The authors reviewed each study to extract information about study objectives, sample and methods, study results, and conclusions. Empirical evidence collected in forty-three published studies shows that for review of a wide range of types of research, U.S. IRBs differ in their application of the federal regulations, in the time they take to review studies, and in the decisions made. Existing studies show evidence of variation in multicenter review, inconsistent or ambiguous interpretation of the federal regulations, and inefficiencies in review. Despite recognition of a need to evaluate effectiveness of IRB review, no identified published study included an evaluation of IRB effectiveness. Multiple studies evaluating the structure, process, and outcome of IRB review in the United States have documented inconsistencies and inefficiencies. Efforts should be made to address these concerns. Additional research is needed to understand how IRBs accomplish their objectives, what issues they find important, what quality IRB review is, and how effective IRBs are at protecting human research participants.
Collapse
Affiliation(s)
- Lura Abbott
- Department of Bioethics, National Institutes of Health, Clinical Center, Building 10/1C118, Bethesda, MD 20892, USA
| | | |
Collapse
|
25
|
Pritchard IA. How Do IRB Members Make Decisions? A Review and Research Agenda. J Empir Res Hum Res Ethics 2011; 6:31-46. [DOI: 10.1525/jer.2011.6.2.31] [Citation(s) in RCA: 27] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Many factors have been found to influence the nature and quality of the human research ethics review process. These are reviewed along with discussion of ways in which normal psychological characteristics and group decision-making processes may affect the decisions of institutional review board (IRB) members when reviewing proposed research activities, and may contribute to the acknowledged variability of IRB responses to identical research proposals. Three salient features of human judgment and decision-making illuminated by the existing psychological research literature are used to illustrate this idea: Research findings related to (a) risk perception and acceptance, (b) the standards people use to make decisions, and (c) some nonrational influences on group decision-making suggest how psychological characteristics may affect some outcomes of convened IRB meetings. Recognizing such influences may enable the improvement of IRB decision-making.
Collapse
|
26
|
Abstract
Regulation of biomedical research is the subject of considerable debate in the bioethics and health policy worlds. The ethics and governance of medical student projects is becoming an increasingly important topic in its own right, especially in the U.K., where there are periodic calls to change it. My main claim is that there seems to be no good reason for treating student projects differently from projects led by qualified and more experienced scientists and hence no good grounds for changing the current system of ethics review. I first suggest that the educational objectives cannot be met without laying down standards of good science, whatever they may be. Weak science is unnecessary for educational purposes, and it is, in any case, unlikely to produce good researchers in the future. Furthermore, it is curious to want to change the system of ethics review specifically for students when it is the science that is at stake, and when the science now falls largely outside the ethics remit. I further show that ethics review is nevertheless important since students carry a new potential conflict of interests that warrants independent oversight which supervisory support does not offer. This potential conflict may become more morally troublesome the greater the risks to the subjects of the research, and students may impose greater risks on their subjects (relative to professional researchers) by virtue of being inexperienced, whatever the nature of the project. Pragmatic concerns may finally be allayed by organizing the current system more efficiently at critical times of the university calendar.
Collapse
|
27
|
Edwards SJL. The Role, Remit and Function of the Research Ethics Committee — 1. The Rationale for Ethics Review of Research by Committee. RESEARCH ETHICS 2009. [DOI: 10.1177/174701610900500405] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
This is the first in a series of five papers on the role, remit and function of research ethics committees which are intended to provide for REC members a broad understanding of the most important issues in research ethics and governance. The first considers the rationale for having ethics review by committee at all; seeking to explain why ethics committees, as we currently have them, are so important to the wider system of governing research.
Collapse
|
28
|
Varelius J. Is ethical expertise possible? MEDICINE, HEALTH CARE, AND PHILOSOPHY 2008; 11:127-32. [PMID: 17712608 DOI: 10.1007/s11019-007-9089-8] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/15/2007] [Accepted: 07/06/2007] [Indexed: 05/16/2023]
Abstract
Services of ethics committees are nowadays commonly used in such various spheres of life as health care, public administration, business, law, engineering, and scientific research. It is taken that as their members have expertise in ethics, these committees can have valuable contributions to make in solving practical moral problems. It has, however, also been maintained that it is simply absurd to claim that one has some special knowledge and skills in moral matters; in connection with moral questions there is no expertise to be had. In this paper, I assess this criticism of the use of ethics committees and ethics consultants. I argue that there is no sufficient reason to reject the possibility of ethical expertise.
Collapse
Affiliation(s)
- Jukka Varelius
- Department of Philosophy, University of Turku, Turku, Finland.
| |
Collapse
|
29
|
Boyd EA, Bero LA. Defining financial conflicts and managing research relationships: an analysis of university conflict of interest committee decisions. SCIENCE AND ENGINEERING ETHICS 2007; 13:415-435. [PMID: 18008185 DOI: 10.1007/s11948-007-9041-6] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/03/2006] [Revised: 10/03/2006] [Accepted: 10/22/2007] [Indexed: 05/25/2023]
Abstract
Despite a decade of federal regulation and debate over the appropriateness of financial ties in research and their management, little is known about the actual decision-making processes of university conflict of interest (COI) committees. This paper analyzes in detail the discussions and decisions of three COI committees at three public universities in California. University committee members struggle to understand complex financial relationships and reconcile institutional, state, and federal policies and at the same time work to protect the integrity of the scientific process, the autonomy and intellectual freedom of their faculty colleagues and students, and the financial interests of the university.
Collapse
Affiliation(s)
- Elizabeth A Boyd
- Department of Clinical Pharmacy, University of California San Francisco, San Francisco, CA 94143-0613, USA.
| | | |
Collapse
|
30
|
Dixon-Woods M, Angell E, Ashcroft RE, Bryman A. Written work: The social functions of Research Ethics Committee letters. Soc Sci Med 2007; 65:792-802. [PMID: 17490795 DOI: 10.1016/j.socscimed.2007.03.046] [Citation(s) in RCA: 32] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2006] [Indexed: 10/23/2022]
Abstract
Research Ethics Committees (RECs) are increasingly institutionalised as a feature of research practice, but have remained strangely neglected by social scientists. In this paper, we argue that analysis of letters from RECs to researchers offers important insights into how RECs operate. We report a traditional content analysis and an ethnographic content analysis of 141 letters to researchers, together with an analysis of the organisational and institutional arrangements for RECs in the UK. We show that REC letters perform three important social functions. First, they define what is deemed by a REC to be ethical practice for any particular application, and confer authority on that definition. They do this actively, through comments on particular aspects of proposals, and passively, through silences about other aspects. Second, they provide an account of the work of the REC, and function as a form of institutional display. Third, they specify the nature of the relationship between the REC and the applicant, casting the applicant in a supplicant role and requiring forms of docility. Writing and reading REC letters require highly specific competences, and engage both parties in a Bourdieusian "game" that discourages challenges from researchers. The authority of RECs' decisions derives not from their appeal to the moral superiority of any ethical position, but through their place in the organisational structure and the social positioning of the parties to the process thus implied. Letters are the critical point at which RECs act on researchers and their projects.
Collapse
|
31
|
Abstract
Objectives:To examine differences in the ethical judgments made by Research Ethics Committees (RECs) or Institutional Review Boards (IRBs).Methods:We did a review of the literature and included any study that attempted to compare the ethical judgments made by different RECs or IRBs when reviewing one or more protocol.Results:There were twenty-six articles reporting such discrepancies across Europe, within the United Kingdom, Spain, and United States. Of these studies, there were only five reports of some RECs approving while others rejecting the same protocol. All studies, however, reported differences in the clarifications and revisions asked of researchers regarding consent, recruitment, risks and benefits, compensation arrangements, and scientific issues.Conclusions:The studies were generally anecdotal reports of researchers trying to do research. New rules requiring a single ethical opinion for multi-site research at least in European Member States may simply conceal problematic issues in REC decision making. In the last analysis, we should expect a certain degree of variation and differences if we are to keep a committee system of review, although there is a pressing need to investigate the way in which RECs make these judgments. In particular, we need to identify the source of any aberrations, distortions, or confusions that could arbitrarily affect these judgments. Furthermore, local conditions remain important ethical considerations and should not be sidelined in pursuit of greater “consistency.”
Collapse
Affiliation(s)
- Sarah J L Edwards
- University College London/University of College London Hospitals, NHS Trust, Tottenham Court Road, London, UK.
| | | | | |
Collapse
|
32
|
|
33
|
Abstract
Decisions about the need for ethical review should be based on the morality of all actions rather than arbitrary distinctions between audit and research
Collapse
|
34
|
Lawson L. Furthering the search for truth and justice. JOURNAL OF FORENSIC NURSING 2005; 1:149-50. [PMID: 17073115 DOI: 10.1111/j.1939-3938.2005.tb00036.x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
|