Testing COVID-19 tests faces methodological challenges

Meta-DiSc 2.0: a web application for meta-analysis of diagnostic test accuracy data

BACKGROUND

Diagnostic evidence of the accuracy of a test for identifying a target condition of interest can be estimated using systematic approaches following standardized methodologies. Statistical methods for the meta-analysis of diagnostic test accuracy (DTA) studies are relatively complex, presenting a challenge for reviewers without extensive statistical expertise. In 2006, we developed Meta-DiSc, a free user-friendly software to perform test accuracy meta-analysis. This statistical program is now widely used for performing DTA meta-analyses. We aimed to build a new version of the Meta-DiSc software to include statistical methods based on hierarchical models and an enhanced web-based interface to improve user experience.

RESULTS

In this article, we present the updated version, Meta-DiSc 2.0, a web-based application developed using the R Shiny package. This new version implements recommended state-of-the-art statistical models to overcome the limitations of the statistical approaches included in the previous version. Meta-DiSc 2.0 performs statistical analyses of DTA reviews using a bivariate random effects model. The application offers a thorough analysis of heterogeneity, calculating logit variance estimates of sensitivity and specificity, the bivariate I-squared, the area of the 95% prediction ellipse, and the median odds ratios for sensitivity and specificity, and facilitating subgroup and meta-regression analyses. Furthermore, univariate random effects models can be applied to meta-analyses with few studies or with non-convergent bivariate models. The application interface has an intuitive design set out in four main menus: file upload; graphical description (forest and ROC plane plots); meta-analysis (pooling of sensitivity and specificity, estimation of likelihood ratios and diagnostic odds ratio, sROC curve); and summary of findings (impact of test through downstream consequences in a hypothetical population with a given prevalence). All computational algorithms have been validated in several real datasets by comparing results obtained with STATA/SAS and MetaDTA packages.

CONCLUSION

We have developed and validated an updated version of the Meta-DiSc software that is more accessible and statistically sound. The web application is freely available at www.metadisc.es .

Evaluation of Diagnostic Strategies for Identifying SARS-CoV-2 Infection in Clinical Practice: a Systematic Review and Compliance with the Standards for Reporting Diagnostic Accuracy Studies Guideline (STARD)

We aimed to review strategies for identifying SARS-CoV-2 infection before the availability of molecular test results, and to assess the reporting quality of the studies identified through the application of the STARD guideline. We screened 3,821 articles published until 30 April 2021, of which 23 met the inclusion criteria: including at least two diagnostic variables, being designed for use in clinical practice or in a public health context and providing diagnostic accuracy rates. Data extraction and application of STARD criteria were performed independently by two researchers and discrepancies were discussed with a third author. Most of the studies (16, 69.6%) included symptomatic patients with suspected infection, six studies (26.1%) included patients already diagnosed and one study (4.3%) included individuals with close contact to a COVID-positive patient. The main variables considered in the studies, which included symptomatic patients, were imaging and demographic characteristics, symptoms, and lymphocyte count. The values for area under the receiver operating characteristic curve (AUC)ranged from 53-97.4. Seven studies (30.4%) validated the diagnostic model in an independent sample. The average number of STARD criteria fulfilled was 17.6 (maximum, 27 and minimum, 5). High diagnostic accuracy values are shown when more than one diagnostic variable is considered, mainly imaging and demographic characteristics, symptoms, and lymphocyte count. This could offer the potential to identify individuals with SARS-CoV-2 infection with high accuracy when molecular testing is not available. However, external validation for developed models and evaluations in populations as similar as possible to those in which they will be applied is urgently needed. IMPORTANCE According to this review, the inclusion of more than one diagnostic test in the diagnostic process for COVID-19 infection shows high diagnostic accuracy values. Imaging characteristics, patients' symptoms, demographic characteristics, and lymphocyte count were the variables most frequently included in the diagnostic models. However, developed models should be externally validated before reaching conclusions on their utility in practice. In addition, it is important to bear in mind that the test should be evaluated in populations as similar as possible to those in which it will be applied in practice.

Bayesian latent class analysis produced diagnostic accuracy estimates that were more interpretable than composite reference standards for extrapulmonary tuberculosis tests

BACKGROUND

Evaluating the accuracy of extrapulmonary tuberculosis (TB) tests is challenging due to lack of a gold standard. Latent class analysis (LCA), a statistical modeling approach, can adjust for reference tests' imperfect accuracies to produce less biased test accuracy estimates than those produced by commonly used methods like composite reference standards (CRSs). Our objective is to illustrate how Bayesian LCA can address the problem of an unavailable gold standard and demonstrate how it compares to using CRSs for extrapulmonary TB tests.

METHODS

We re-analyzed a dataset of presumptive extrapulmonary TB cases in New Delhi, India, for three forms of extrapulmonary TB. Results were available for culture, smear microscopy, Xpert MTB/RIF, and a non-microbiological test, cytopathology/histopathology, or adenosine deaminase (ADA). A diagram was used to define assumed relationships between observed tests and underlying latent variables in the Bayesian LCA with input from an inter-disciplinary team. We compared the results to estimates obtained from a sequence of CRSs defined by increasing numbers of positive reference tests necessary for positive disease status.

RESULTS

Data were available from 298, 388, and 230 individuals with presumptive TB lymphadenitis, meningitis, and pleuritis, respectively. Using Bayesian LCA, estimates were obtained for accuracy of all tests and for extrapulmonary TB prevalence. Xpert sensitivity neared that of culture for TB lymphadenitis and meningitis but was lower for TB pleuritis, and specificities of all microbiological tests approached 100%. Non-microbiological tests' sensitivities were high, but specificities were only moderate, preventing disease rule-in. CRSs' only provided estimates of Xpert and these varied widely per CRS definition. Accuracy of the CRSs also varied by definition, and no CRS was 100% accurate.

CONCLUSION

Unlike CRSs, Bayesian LCA takes into account known information about test performance resulting in accuracy estimates that are easier to interpret. LCA should receive greater consideration for evaluating extrapulmonary TB diagnostic tests.

COVID-19 Impact on Diagnostic Innovations: Emerging Trends and Implications

Diagnostic testing remains the backbone of the coronavirus disease 2019 (COVID-19) response, supporting containment efforts to mitigate the outbreak. The severity of this crisis and increasing capacity issues associated with polymerase chain reaction (PCR)-based testing, accelerated the development of diagnostic solutions to meet demands for mass testing. The National Institute for Health Research (NIHR) Innovation Observatory is the national horizon scanning organization in England. Since March, the Innovation Observatory has applied advanced horizon scanning methodologies and tools to compile a diagnostic landscape, based upon data captured for molecular (MDx) and immunological (IDx) based diagnostics (commercialized/in development), for the diagnosis of SARS-CoV-2. In total we identified and tracked 1608 diagnostics, produced by 1045 developers across 54 countries. Our dataset shows the speed and scale in which diagnostics were produced and provides insights into key periods of development and shifts in trends between MDx and IDx solutions as the pandemic progressed. Stakeholders worldwide required timely and detailed intelligence to respond to major challenges, including testing capacity and regulatory issues. Our intelligence assisted UK stakeholders with assessing priorities and mitigation options throughout the pandemic. Here we present the global evolution of diagnostic innovations devised to meet changing needs, their regulation and trends across geographical regions, providing invaluable insights into the complexity of the COVID-19 phenomena.

False-negative results of initial RT-PCR assays for COVID-19: A systematic review

BACKGROUND

A false-negative case of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is defined as a person with suspected infection and an initial negative result by reverse transcription-polymerase chain reaction (RT-PCR) test, with a positive result on a subsequent test. False-negative cases have important implications for isolation and risk of transmission of infected people and for the management of coronavirus disease 2019 (COVID-19). We aimed to review and critically appraise evidence about the rate of RT-PCR false-negatives at initial testing for COVID-19.

METHODS

We searched MEDLINE, EMBASE, LILACS, as well as COVID-19 repositories, including the EPPI-Centre living systematic map of evidence about COVID-19 and the Coronavirus Open Access Project living evidence database. Two authors independently screened and selected studies according to the eligibility criteria and collected data from the included studies. The risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. We calculated the proportion of false-negative test results using a multilevel mixed-effect logistic regression model. The certainty of the evidence about false-negative cases was rated using the GRADE approach for tests and strategies. All information in this article is current up to July 17, 2020.

RESULTS

We included 34 studies enrolling 12,057 COVID-19 confirmed cases. All studies were affected by several risks of bias and applicability concerns. The pooled estimate of false-negative proportion was highly affected by unexplained heterogeneity (tau-squared = 1.39; 90% prediction interval from 0.02 to 0.54). The certainty of the evidence was judged as very low due to the risk of bias, indirectness, and inconsistency issues.

CONCLUSIONS

There is substantial and largely unexplained heterogeneity in the proportion of false-negative RT-PCR results. The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability. Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection given that up to 54% of COVID-19 patients may have an initial false-negative RT-PCR (very low certainty of evidence).

SYSTEMATIC REVIEW REGISTRATION

Protocol available on the OSF website: https://tinyurl.com/vvbgqya.

False-negative results of initial RT-PCR assays for COVID-19: A systematic review

BACKGROUND

A false-negative case of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is defined as a person with suspected infection and an initial negative result by reverse transcription-polymerase chain reaction (RT-PCR) test, with a positive result on a subsequent test. False-negative cases have important implications for isolation and risk of transmission of infected people and for the management of coronavirus disease 2019 (COVID-19). We aimed to review and critically appraise evidence about the rate of RT-PCR false-negatives at initial testing for COVID-19.

METHODS

We searched MEDLINE, EMBASE, LILACS, as well as COVID-19 repositories, including the EPPI-Centre living systematic map of evidence about COVID-19 and the Coronavirus Open Access Project living evidence database. Two authors independently screened and selected studies according to the eligibility criteria and collected data from the included studies. The risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. We calculated the proportion of false-negative test results using a multilevel mixed-effect logistic regression model. The certainty of the evidence about false-negative cases was rated using the GRADE approach for tests and strategies. All information in this article is current up to July 17, 2020.

RESULTS

We included 34 studies enrolling 12,057 COVID-19 confirmed cases. All studies were affected by several risks of bias and applicability concerns. The pooled estimate of false-negative proportion was highly affected by unexplained heterogeneity (tau-squared = 1.39; 90% prediction interval from 0.02 to 0.54). The certainty of the evidence was judged as very low due to the risk of bias, indirectness, and inconsistency issues.

CONCLUSIONS

There is substantial and largely unexplained heterogeneity in the proportion of false-negative RT-PCR results. The collected evidence has several limitations, including risk of bias issues, high heterogeneity, and concerns about its applicability. Nonetheless, our findings reinforce the need for repeated testing in patients with suspicion of SARS-Cov-2 infection given that up to 54% of COVID-19 patients may have an initial false-negative RT-PCR (very low certainty of evidence).

SYSTEMATIC REVIEW REGISTRATION

Protocol available on the OSF website: https://tinyurl.com/vvbgqya.

IFCC Interim Guidelines on Molecular Testing of SARS-CoV-2 Infection

The diagnosis of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection globally has relied extensively on molecular testing, contributing vitally to case identification, isolation, contact tracing, and rationalization of infection control measures during the coronavirus disease 2019 (COVID-19) pandemic. Clinical laboratories have thus needed to verify newly developed molecular tests and increase testing capacity at an unprecedented rate. As the COVID-19 pandemic continues to pose a global health threat, laboratories continue to encounter challenges in the selection, verification, and interpretation of these tests. This document by the International Federation for Clinical Chemistry and Laboratory Medicine (IFCC) Task Force on COVID-19 provides interim guidance on: (A) clinical indications and target populations, (B) assay selection, (C) assay verification, and (D) test interpretation and limitations for molecular testing of SARS-CoV-2 infection. These evidence-based recommendations will provide practical guidance to clinical laboratories worldwide and highlight the continued importance of laboratory medicine in our collective pandemic response.