Estimation of sensitivity and specificity of three conditionally dependent diagnostic tests in the absence of a gold standard

Evaluation using latent class models of the diagnostic performances of three ELISA tests commercialized for the serological diagnosis of Coxiella burnetii infection in domestic ruminants

ELISA methods are the diagnostic tools recommended for the serological diagnosis of Coxiella burnetii infection in ruminants but their respective diagnostic performances are difficult to assess because of the absence of a gold standard. This study focused on three commercial ELISA tests with the following objectives (1) assess their sensitivity and specificity in sheep, goats and cattle, (2) assess the between- and within-herd seroprevalence distribution in these species, accounting for diagnostic errors, and (3) estimate optimal sample sizes considering sensitivity and specificity at herd level. We comparatively tested 1413 cattle, 1474 goat and 1432 sheep serum samples collected in France. We analyzed the cross-classified test results with a hierarchical zero-inflated beta-binomial latent class model considering each herd as a population and conditional dependence as a fixed effect. Potential biases and coverage probabilities of the model were assessed by simulation. Conditional dependence for truly seropositive animals was high in all species for two of the three ELISA methods. Specificity estimates were high, ranging from 94.8% [92.1; 97.8] to 99.2% [98.5; 99.7], whereas sensitivity estimates were generally low, ranging from 39.3 [30.7; 47.0] to 90.5% [83.3; 93.8]. Between- and within-herd seroprevalence estimates varied greatly among geographic areas and herds. Overall, goats showed higher within-herd seroprevalence levels than sheep and cattle. The optimal sample size maximizing both herd sensitivity and herd specificity varied from 3 to at least 20 animals depending on the test and ruminant species. This study provides better interpretation of three widely used commercial ELISA tests and will make it possible to optimize their implementation in future studies. The methodology developed may likewise be applied to other human or animal diseases.

True malaria prevalence in children under five: Bayesian estimation using data of malaria household surveys from three sub-Saharan countries

BACKGROUND

Malaria is one of the major causes of childhood death in sub-Saharan countries. A reliable estimation of malaria prevalence is important to guide and monitor progress toward control and elimination. The aim of the study was to estimate the true prevalence of malaria in children under five in the Democratic Republic of the Congo, Uganda and Kenya, using a Bayesian modelling framework that combined in a novel way malaria data from national household surveys with external information about the sensitivity and specificity of the malaria diagnostic methods used in those surveys-i.e., rapid diagnostic tests and light microscopy.

METHODS

Data were used from the Demographic and Health Surveys (DHS) and Malaria Indicator Surveys (MIS) conducted in the Democratic Republic of the Congo (DHS 2013-2014), Uganda (MIS 2014-2015) and Kenya (MIS 2015), where information on infection status using rapid diagnostic tests and/or light microscopy was available for 13,573 children. True prevalence was estimated using a Bayesian model that accounted for the conditional dependence between the two diagnostic methods, and the uncertainty of their sensitivities and specificities obtained from expert opinion.

RESULTS

The estimated true malaria prevalence was 20% (95% uncertainty interval [UI] 17%-23%) in the Democratic Republic of the Congo, 22% (95% UI 9-32%) in Uganda and 1% (95% UI 0-3%) in Kenya. According to the model estimations, rapid diagnostic tests had a satisfactory sensitivity and specificity, and light microscopy had a variable sensitivity, but a satisfactory specificity. Adding reported history of fever in the previous 14 days as a third diagnostic method to the model did not affect model estimates, highlighting the poor performance of this indicator as a malaria diagnostic.

CONCLUSIONS

In the absence of a gold standard test, Bayesian models can assist in the optimal estimation of the malaria burden, using individual results from several tests and expert opinion about the performance of those tests.

Evaluation by latent class analysis of a magnetic capture based DNA extraction followed by real-time qPCR as a new diagnostic method for detection of Echinococcus multilocularis in definitive hosts

A new method, based on a magnetic capture based DNA extraction followed by qPCR, was developed for the detection of the zoonotic parasite Echinococcus multilocularis in definitive hosts. Latent class analysis was used to compare this new method with the currently used phenol-chloroform DNA extraction followed by single tube nested PCR. In total, 60 red foxes and coyotes from three different locations were tested with both molecular methods and the sedimentation and counting technique (SCT) or intestinal scraping technique (IST). Though based on a limited number of samples, it could be established that the magnetic capture based DNA extraction followed by qPCR showed similar sensitivity and specificity as the currently used phenol-chloroform DNA extraction followed by single tube nested PCR. All methods have a high specificity as shown by Bayesian latent class analysis. Both molecular assays have higher sensitivities than the combined SCT and IST, though the uncertainties in sensitivity estimates were wide for all assays tested. The magnetic capture based DNA extraction followed by qPCR has the advantage of not requiring hazardous chemicals like the phenol-chloroform DNA extraction followed by single tube nested PCR. This supports the replacement of the phenol-chloroform DNA extraction followed by single tube nested PCR by the magnetic capture based DNA extraction followed by qPCR for molecular detection of E. multilocularis in definitive hosts.

Bovine and Caprine Brucellosis in Bangladesh: Bayesian evaluation of four serological tests, true prevalence, and associated risk factors in household animals

A cross-sectional study was carried out to estimate the true prevalence of Brucella spp. and identify allied risk factors/indicators associated with brucellosis in the Dinajpur and Mymensingh districts of Bangladesh. A total 320 stratified random blood samples were collected and tested in parallel for Brucella antibodies using Rose Bengal (RBT), slow agglutination (SAT), and indirect and competitive ELISA. In addition, a structured questionnaire was administered to each household herd owner to gather information regarding potential risk factors. Both univariate and multivariate logistic regression analyses were used to identify potential risk factors or indicators at animal level. A Bayesian approach was used to estimate the true prevalence of brucellosis along with the test performances (Se and Sp). The estimated animal level true prevalence in cattle was 9.70 % (95 % CPI 5.0-16 %) and in goat 6.3 % (95 % CPI 2.8-11.0 %). The highest sensitivity was achieved by SAT ranges from 69.6 to 78.9 %, and iELISA was found to be more specific (97.4 to 98.8 %) in comparison with other tests. On the other hand, a significant level of (P < 0.05) Brucella seropositivity was found in cattle that breed naturally compared with those that undergo artificial insemination. In goats, exotic breeds were significantly associated (P < 0.05) with Brucella seroprevalence compared with indigenous breeds. Goats with a previous records of abortion and/or retained placenta were also found to have significant levels (P < 0.05). Cows with previous abortion records showed higher odds (18 times) of being seropositive. None of the evaluated tests can be recommended to apply alone for the diagnosis of bovine and caprine brucellosis.

Within herd transmission and evaluation of the performance of clinical and serological diagnosis of foot-and-mouth disease in partially immune cattle herds

The control of foot-and-mouth disease (FMD) in vaccinated populations relies upon surveillance activities such as clinical inspections (CI) and serological monitoring. New evidence to refine current surveillance guidelines has been provided by evaluating (1) the diagnostic performance of CI and serological tests for detection of FMD virus (FMDV) non-structural proteins (NSP), and (2) the within-herd transmission of the virus in partially immune cattle. Data came from 23 affected herds during an epidemic of FMDV type O in Bolivia, in 2007. All cattle (n=957) in these herds were clinically inspected and serum samples were collected one month after the last animal with clinical signs was detected. Samples were tested for the presence of antibodies against NSP using the PANAFTOSA 3ABC-ELISA test and a subset of samples were tested using the enzyme-linked immunoelectrotransfer blot assay (EITB). Data from clinical and serological diagnoses were analysed using a Bayesian model. The sensitivity Se and specificity Sp of the tests, as well as the prevalence and the within-herd reproduction ratio R of FMDV were estimated. In addition, risk factors for infection were identified. The Se of CI, the 3ABC-ELISA and the EITB tests were estimated to be 0.30, 0.88 and 0.96 respectively. The estimated Sp, in the same order, were 0.88, 0.93 and 0.97. The within-herd prevalence of infected animals ranged from 0.04 to 0.91 and R ranged from 1.02 to 2.68. It was observed that cattle coming from areas with high vaccination coverage had a lower risk of becoming infected than home-bred cattle from the affected herds, where vaccination coverage was thought to be low. Although these estimates come from herds kept under specific conditions, they provide a reference for future surveillance design and can inform simulation models for surveillance and control of FMD in similar cattle populations.

Estimation of diagnostic test accuracy without full verification: a review of latent class methods

The performance of a diagnostic test is best evaluated against a reference test that is without error. For many diseases, this is not possible, and an imperfect reference test must be used. However, diagnostic accuracy estimates may be biased if inaccurately verified status is used as the truth. Statistical models have been developed to handle this situation by treating disease as a latent variable. In this paper, we conduct a systematized review of statistical methods using latent class models for estimating test accuracy and disease prevalence in the absence of complete verification.

Development and validation of a genotype 3 recombinant protein-based immunoassay for hepatitis E virus serology in swine

Hepatitis E virus (HEV) is classified within the family Hepeviridae, genus Hepevirus. HEV genotype 3 (Gt3) infections are endemic in pigs in Western Europe and in North and South America and cause zoonotic infections in humans. Several serological assays to detect HEV antibodies in pigs have been developed, at first mainly based on HEV genotype 1 (Gt1) antigens. To develop a sensitive HEV Gt3 ELISA, a recombinant baculovirus expression product of HEV Gt3 open reading frame-2 was produced and coated onto polystyrene ELISA plates. After incubation of porcine sera, bound HEV antibodies were detected with anti-porcine anti-IgG and anti-IgM conjugates. For primary estimation of sensitivity and specificity of the assay, sets of sera were used from pigs experimentally infected with HEV Gt3. For further validation of the assay and to set the cutoff value, a batch of 1100 pig sera was used. All pig sera were tested using the developed HEV Gt3 assay and two other serologic assays based on HEV Gt1 antigens. Since there is no gold standard available for HEV antibody testing, further validation and a definite setting of the cutoff of the developed HEV Gt3 assay were performed using a statistical approach based on Bayes' theorem. The developed and validated HEV antibody assay showed effective detection of HEV-specific antibodies. This assay can contribute to an improved detection of HEV antibodies and enable more reliable estimates of the prevalence of HEV Gt3 in swine in different regions.

A general latent class model for performance evaluation of diagnostic tests in the absence of a gold standard: an application to Chagas disease

We propose a new general Bayesian latent class model for evaluation of the performance of multiple diagnostic tests in situations in which no gold standard test exists based on a computationally intensive approach. The modeling represents an interesting and suitable alternative to models with complex structures that involve the general case of several conditionally independent diagnostic tests, covariates, and strata with different disease prevalences. The technique of stratifying the population according to different disease prevalence rates does not add further marked complexity to the modeling, but it makes the model more flexible and interpretable. To illustrate the general model proposed, we evaluate the performance of six diagnostic screening tests for Chagas disease considering some epidemiological variables. Serology at the time of donation (negative, positive, inconclusive) was considered as a factor of stratification in the model. The general model with stratification of the population performed better in comparison with its concurrents without stratification. The group formed by the testing laboratory Biomanguinhos FIOCRUZ-kit (c-ELISA and rec-ELISA) is the best option in the confirmation process by presenting false-negative rate of 0.0002% from the serial scheme. We are 100% sure that the donor is healthy when these two tests have negative results and he is chagasic when they have positive results.

Reconciling multiple data sources to improve accuracy of large-scale prediction of forest disease incidence

Ecological spatial data often come from multiple sources, varying in extent and accuracy. We describe a general approach to reconciling such data sets through the use of the Bayesian hierarchical framework. This approach provides a way for the data sets to borrow strength from one another while allowing for inference on the underlying ecological process. We apply this approach to study the incidence of eastern spruce dwarf mistletoe (Arceuthobium pusillum) in Minnesota black spruce (Picea mariana). A Minnesota Department of Natural Resources operational inventory of black spruce stands in northern Minnesota found mistletoe in 11% of surveyed stands, while a small, specific-pest survey found mistletoe in 56% of the surveyed stands. We reconcile these two surveys within a Bayesian hierarchical framework and predict that 35-59% of black spruce stands in northern Minnesota are infested with dwarf mistletoe.

Evaluation of three commercial enzyme-linked immunosorbent assays for the detection of antibodies against Salmonella spp. in meat juice from finishing pigs in Spain

The control of animal salmonellosis is considered as a major objective in Europe and indirect ELISAs will be important tools for the implementation of control programs for this infection in pigs. We analyse the results yielded by three commercial ELISAs (Herdcheck Swine Salmonella, SALMOTYPE Pig Screen, and PrioCHECK Salmonella) on meat juice samples from a population of slaughter pigs of Aragon, NW Spain, to assess their efficacy using traditional and latent-class approaches. Overall, the Herdcheck Swine Salmonella detected more Salmonella-infected pigs than the other two tests, but its relative sensitivity was low (65.9%). A similar result was observed when only serotypes detectable by this test were considered (69.1%). When a Bayesian approach was used the Herdcheck Swine Salmonella showed also the highest overall accuracy (sensitivity = 88% and specificity = 74%). Our results suggest that a relatively small proportion of the observed prevalence in herds would be explained by using these ELISAs. Also, this study points out that when different ELISA tests are used within the same herd, results may differ substantially. Thus, caution is advised if it is decided to use these assays for herd health classification in Spanish Salmonella control programs.

Evaluation of ELISA test characteristics and estimation of Toxoplasma gondii seroprevalence in Dutch sheep using mixture models

Lamb and mutton are considered important sources of human Toxoplasma gondii infections, but actual data on the prevalence of T. gondii in sheep in The Netherlands is lacking. The aim of this study was to investigate the prevalence of T. gondii in slaughtered sheep to get more insight in the importance of sheep as a source of human infection. In addition, regional variation in prevalence was studied, as this may indicate differences in environmental contamination. An in-house ELISA that detects antibodies against T. gondii was developed and used to test 1179 sera collected from sheep presented at 11 Dutch slaughterhouses between October and December 2007. Since validation of the serological assay was hampered by a lack of appropriate reference sera, the diagnostic performance and seroprevalence were estimated by fitting a binormal mixture model. ROC-curve analysis on the fitted distributions showed high discriminatory power (AUC=0.995), and high sensitivity and specificity of the ELISA. The overall prevalence was estimated at 27.8% (25.6-29.9%), but was significantly higher in sheep over 1 year old, and in sheep from the central provinces. The high sensitivity and specificity of the in-house ELISA were confirmed by Bayesian analysis together with three commercially available assays: Toxo-Screen DA (bioMérieux), Chekit Toxotest Antibody ELISA (IDEXX), and Toxoplasmosis serum screening ELISA (Institut Pourquier). In conclusion, the binormal mixture model proved a useful method to obtain estimates of diagnostic performance and seroprevalence without use of reference sera. The seroprevalence in sheep was high, and as sheep with antibodies usually carry tissue cysts, this indicates that undercooked lamb and mutton may indeed be important sources of human toxoplasmosis in The Netherlands.

Validation of diagnostic tests for detection of avian influenza in vaccinated chickens using Bayesian analysis

Vaccination is an attractive tool for the prevention of outbreaks of highly pathogenic avian influenza in domestic birds. It is known, however, that under certain circumstances vaccination may fail to prevent infection, and that the detection of infection in vaccinated birds can be problematic. Here, we investigate the characteristics of three serological tests (immunofluorescent antibody test (iIFAT), neuraminidase inhibition (NI) assay, and NS1 ELISA) that are able to differentiate infected from vaccinated animals. To this end, data of H7N7 infection experiments are analyzed using Bayesian methods of inference. These Bayesian methods enable validation of the tests in the absence of a gold standard, and allow one to take into account that infected birds do not always develop antibodies after infection. The results show that the N7 iIFAT and the NI assay have sensitivities for detecting antibodies of 0.95 (95% CI: 0.89-0.98) and 0.93 (95% CI: 0.78-0.99), but substantially lower sensitivities for detecting infection: 0.64 (95% CI: 0.52-0.75) and 0.63 (95% CI: 0.49-0.75). The NS1 ELISA has a low sensitivity for both detecting antibodies 0.55 (95% CI: 0.34-0.74) and infection 0.42 (95% CI: 0.28-0.56). The estimated specificities of the N7 iIFAT and the NI assay are 0.92 (95% CI: 0.87-0.95) and 0.91 (95% CI: 0.85-0.95), and 0.82 (95% CI: 0.74-0.87) for the NS1 ELISA. Additionally, our analyses suggest a strong association between the duration of virus excretion of infected birds and the probability to develop antibodies.

Bayesian approach to average power calculations for binary regression models with misclassified outcomes

We develop a simulation-based procedure for determining the required sample size in binomial regression risk assessment studies when response data are subject to misclassification. A Bayesian average power criterion is used to determine a sample size that provides high probability, averaged over the distribution of potential future data sets, of correctly establishing the direction of association between predictor variables and the probability of event occurrence. The method is broadly applicable to any parametric binomial regression model including, but not limited to, the popular logistic, probit, and complementary log-log models. We detail a common medical scenario wherein ascertainment of true disease status is impractical or otherwise impeded, and in its place the outcome of a single binary diagnostic test is used as a surrogate. These methods are then extended to the two diagnostic test setting. We illustrate the method with categorical covariates using one example that involves screening for human papillomavirus. This example coupled with results from simulated data highlights the utility of our Bayesian sample size procedure with error prone measurements.

Bayesian estimation of hepatitis E virus seroprevalence for populations with different exposure levels to swine in The Netherlands

Hepatitis E virus (HEV) is ubiquitous in pigs worldwide and may be zoonotic. Previous HEV seroprevalence estimates for groups of people working with swine were higher than for control groups. However, discordance among results of anti-HEV assays means that true seroprevalence estimates, i.e. seroprevalence due to previous exposure to HEV, depends on choice of seroassay. We tested blood samples from three subpopulations (49 swine veterinarians, 153 non-swine veterinarians and 644 randomly selected individuals from the general population) with one IgM and two IgG ELISAs, and subsets with IgG and/or IgM Western blots. A Bayesian stochastical model was used to combine results of all assays. The model accounted for imperfection of each assay by estimating sensitivity and specificity, and accounted for dependence between serological assays. As expected, discordance among assay results occurred. Applying the model yielded seroprevalence estimates of approximately 11% for swine veterinarians,approximately 6% for non-swine veterinarians and approximately 2% for the general population. By combining the results of five serological assays in a Bayesian stochastical model we confirmed that exposure to swine or their environment was associated with elevated HEV seroprevalence.