Evaluating reliability and risk of bias of in vivo animal data for risk assessment of chemicals - Exploring the use of the SciRAP tool in a systematic review context

Within the field of health risk assessment, it is essential that evaluations of reliability or validity of toxicity data are conducted with structure and transparency. To this end, different tools for evaluating toxicity studies have been developed by different groups and organizations, for different specific purposes. The Science in Risk Assessment and Policy (SciRAP) tool was developed for use in the regulatory health risk assessment of chemicals and to promote structured and transparent evaluation of study reliability within European regulatory frameworks. As such, the SciRAP tool is not specifically tailored for use in a systematic review context. However, in light of the current movement towards applying systematic review in the field of environmental health and chemical assessments and European chemicals regulation, we were interested in exploring how SciRAP could be applied in such a context. To achieve this, the scope of the SciRAP tool was first compared to two tools developed based on systematic review principles at the US Environmental Protection Agency's IRIS program and the National Toxicology Program's Office of Health Assessment and Translation (OHAT). Next, the SciRAP and IRIS tools were both applied in a case study to evaluate the same nine in vivo animal studies and the resulting evaluations were compared. The SciRAP tool was found to address the majority of the elements included for study evaluation in the OHAT and IRIS tools. In the case study, no major differences were found in the conclusions drawn when using SciRAP or IRIS tools. However, future developments to bring the SciRAP tool more in line with systematic review principles were identified and are discussed. Overall, this work illustrates the advantages of applying structured and pre-defined methods for study evaluation and provides a unique case study comparing the impact of using different tools for evaluating animal toxicity studies.

Evaluation of the ToxRTool's ability to rate the reliability of toxicological data for human health hazard assessments

Regulatory agencies often utilize results from peer reviewed publications for hazard assessments. A problem in doing so is the lack of well-accepted tools to objectively, efficiently and systematically assess the quality of published toxicological studies. Herein, we evaluated the publicly available software-based ToxRTool (Toxicological data Reliability assessment Tool) for use in human health hazard assessments. The ToxRTool was developed by the European Commission's Joint Research Center in 2009. It builds on Klimisch categories, a rating system established in 1997, by providing additional criteria and guidance for assessing the reliability of toxicological studies. It also transparently documents the study-selection process. Eight scientists used the ToxRTool to rate the same 20 journal articles on thyroid toxicants. Results were then compared using the Finn coefficient and "AC1" to determine inter-rater consistency. Ratings were most consistent for high-quality journal articles, but less consistent as study quality decreased. Primary reasons for inconsistencies were that some criteria were subjective and some were not clearly described. It was concluded, however, that the ToxRTool has potential and, with refinement, could provide a more objective approach for screening published toxicology studies for use in health risk evaluations, although the ToxRTool ratings are primarily based on study reporting quality.