Evaluation of the Accuracy of Diagnostic Tests From Repeated Measurements Without a Gold Standard

Estimation of diagnostic test accuracy: A "Rule of Three" for data with repeated observations but without a gold standard

This article considers how to estimate the accuracy of a diagnostic test when there are repeated observations, but without the availability of a gold standard or reference test. We identify conditions under which the structure of the observed data is rich enough to provide sufficient degrees of freedom, such that a suitable latent class model can be fitted with identifiable accuracy parameters. We show that a Rule of Three applies, specifying that accuracy can be evaluated as long as there are at least three observations per individual with the given test. This rule also applies if the three observations arise from combinations of different test methods, or from a sequential design in which individuals are tested for a maximum number of times with the same test but stopping if a positive (or negative) result occurs. The rule pertains to tests having an arbitrary number of response categories. Accuracy is evaluated by parameters reflecting rates of misclassification among the response categories, and the model also provides estimates of the underlying distribution of the true disease state. These ideas are illustrated by data from two medical studies. Issues discussed include the advantages and disadvantages of analyzing the response variable as binary or multinomial, as well as the feasibility of testing goodness of fit when the model incorporates a large number of parameters. Comparisons are possible between models that do or do not assume equal accuracy rates for the observations, and between models where certain misclassification parameters are or are not assumed to be zero.

Evaluation of Mycobacterium tuberculosis lipoarabinomannan antigen assay and rapid serology blood test for the diagnosis of bovine tuberculosis in Ethiopia

Background

Bovine tuberculosis (bTB) is prevalent in dairy cattle in Ethiopia. Currently used diagnostic tools such as the single intradermal comparative tuberculin test (SICTT) are time consuming and labor intensive. A rapid, easy-to-use and cost-effective diagnostic test would greatly contribute to the control of bTB in developing countries like Ethiopia. In the present study, two point-of-care diagnostic tests were evaluated for the detection of bTB: LIONEX® Animal TB Rapid test, a membrane-based test for the detection of antibodies to Mycobacterium bovis in blood and ALERE® Determine TB Lipoarabinomannan (LAM) Ag, an immunoassay for the detection of lipoarabinomannan (LAM) antigen (Ag) of mycobacteria in urine. A combination of the SICTT and gamma interferon (IFN-γ) test was used as the gold standard for the validation of these point-of-care tests, as it was not feasible to slaughter the study animals to carry out the historical gold standard of mycobacterial culture. A total of 175 heads of cattle having three different bTB infection categories (positive SICTT, negative SICTT, and unknown SICTT status) were used for this study.

Result

The sensitivity and specificity of TB LAM Ag were 72.2% (95% CI = 62.2, 80.4) and 98.8% (95% CI = 93.6, 99.7), respectively, while the sensitivity and specificity of the LIONEX Animal TB rapid test assay were 54% (95% CI = 44.1 64.3) and 98.8% (95% CI = 93.6, 99.7) respectively. The agreement between TB LAM Ag and SICTT was higher (κ = 0.85; 95% CI = 0.65–0.94) than between TB LAM Ag and IFN-γ (κ = 0.67; 95% CI = 0.52–0.81). The agreement between LIONEX Animals TB Rapid blood test and SICTT was substantial, (κ = 0.63; 95% CI = 0.49–0.77) while the agreement between LIONEX Animal TB rapid blood test and IFN-γ test was moderate (κ = 0.53; 95% CI = 0.40–0.67). Analysis of receiver operating curve (ROC) indicated that the area under the ROC curve (AUC) for TB LAM Ag was 0.85 (95% CI = 0.79–0.91) while it was 0.76 (95% CI; =0.69–0.83) for LIONEX Animal TB rapid test assay.

Conclusion

This study showed that TB LAM Ag had a better diagnostic performance and could potentially be used as ancillary either to SICTT or IFN-γ test for diagnosis of bTB.

Evaluation of Two PCR Tests for Coxiella burnetii Detection in Dairy Cattle Farms Using Latent Class Analysis

Different tests performed on bulk tank milk samples (BTM) are available to determine the C. burnetii status of herds. However, these tests, which are based on the detection of either antibodies directed against C. burnetii (ELISA) or bacterial DNA (PCR), have limitations. A currently disease-free herd infected in the past may continue to test positive with ELISA due to the persistence of antibodies in animals that were infected and that subsequently cleared the infection. Infectious herds can also be misclassified using PCR because of the absence of bacteria in the BTM when the test is performed. Recently, PCR has been used for bacterial DNA detection in the farm environment, which constitutes the main reservoir of C. burnetii. The objectives of this study were to assess and compare the sensitivities and specificities of one commonly used PCR test in BTM (PCR BTM) and of a PCR applied to environmental samples (PCR DUST) in dairy cattle farms. BTM and dust samples were collected (using environmental swabs) in 95 herds. The evaluation of the performance of the 2 tests was conducted using latent class models accounting for within herd disease dynamics. Parameter estimation was carried out using MCMC, within a Bayesian framework. Two types of priors were used for the specificity of PCR DUST. A model with a uniform prior on 0-1 fitted the data better than a model with a uniform prior on 0.95-1. With the best model PCR DUST had a lower sensitivity than PCR BTM (0.75 versus 0.83) and a specificity of 0.72. The moderately low value for the specificity of PCR DUST suggests that the presence of bacteria on farm is not always associated with persistent infections and shedding of bacteria in milk.

Evaluation of the sensitivity and specificity of an enzyme-linked immunosorbent assay for diagnosing brucellosis in African buffalo (Syncerus caffer)

Brucellosis is a disease of veterinary and public health importance worldwide. In sub-Saharan Africa, where the bacterium Brucella abortus has been identified in several free-ranging wildlife species, successful disease control may be dependent on accurate detection in wildlife reservoirs, including African buffalo (Syncerus caffer). We estimated the sensitivity and specificity of a commercial enzyme-linked immunosorbent assay (ELISA) (IDEXX Brucellosis Serum Ab test, IDEXX Laboratories, Westbrook, Maine, USA) for B. abortus based on a data set of 571 serum samples from 258 buffalo in the Kruger National Park, South Africa. We defined a pseudogold standard test result as those buffalo that were consistently positive or negative on two additional serologic tests, namely, the rose bengal test (RBT) and the complement fixation test (CFT). The ELISA's cutoff value was selected using receiver operating characteristics analysis, the pseudogold standard, and a threshold criterion that maximizes the total sensitivity and specificity. Then, we estimated the sensitivity and specificity of all three tests using Bayesian inference and latent class analysis. The ELISA had an estimated sensitivity of 0.928 (95% Bayesian posterior credibility interval [95% BCI] = 0.869-0.974) and specificity of 0.870 (95% BCI = 0.836-0.900). Compared with the ELISA, the RBT had a higher estimated sensitivity of 0.986 (95% BCI = 0.928-0.999), and both the RBT and CFT had higher specificities, estimated to be 0.992 (95% BCI = 0.971-0.996) and 0.998 (95% BCI = 0.992-0.999), respectively. Therefore, no single serologic test perfectly detected the antibody. However, after adjustment of cutoff values for South African conditions, the IDEXX Brucellosis Serum Ab Test may be a valuable additional screening test for brucellosis in Kruger National Park's African buffalo.

Estimation of diagnostic test accuracy without full verification: a review of latent class methods

The performance of a diagnostic test is best evaluated against a reference test that is without error. For many diseases, this is not possible, and an imperfect reference test must be used. However, diagnostic accuracy estimates may be biased if inaccurately verified status is used as the truth. Statistical models have been developed to handle this situation by treating disease as a latent variable. In this paper, we conduct a systematized review of statistical methods using latent class models for estimating test accuracy and disease prevalence in the absence of complete verification.

An application of Bayesian growth mixture modelling to estimate infection incidences from repeated serological tests

Diagnoses of infectious diseases are often performed using antibody detection through enzyme-linked immunosorbent assay techniques. These data are usually dichotomized into positive and negative samples using a fixed cut-off and prevalences of infection are subsequently estimated assuming perfect correspondence between the dichotomized test results and infection status. In contrast to this approach, in this case study, we estimate the effect of distributing insecticide impregnated bednets to prevent Leishmania infection through mixture modelling of the original continuous data. We analyze the data from a cluster randomized intervention trial using a generalized latent variable model consisting of a longitudinal mixture model for the observed outcome and a Hidden Markov model for the underlying unobserved disease status to estimate the effect of an intervention. The response and structural models are jointly estimated in a Bayesian framework. This model has the advantage that it avoids the need to choose an arbitrary cut-off and allows for uncertainty in the infection status. In this paper, we describe the development of the model and selection of priors, the application to the motivating data, model checking and simulation results.

A framework for the joint modeling of longitudinal diagnostic outcome data and latent infection status: application to investigating the temporal relationship between infection and disease

For many diseases the infection status of individuals cannot be observed directly, but can only be inferred from biomarkers that are subject to measurement error. Diagnosis of infection based on observed symptoms can itself be regarded as an imperfect test of infection status. The temporal relationship between infection and marker outcomes may be complex, especially for recurrent diseases where individuals can experience multiple bouts of infection. We propose an approach that first models the unobserved longitudinal infection status of individuals conditional on relevant covariates, and then jointly models the longitudinal sequence of biomarker outcomes conditional on infection status and covariate information through time, thus resulting in a joint model for longitudinal infection and biomarker sequences. This model can be used to investigate the temporal dynamics of infection, and to evaluate the usefulness of biomarkers for monitoring purposes. Our work is motivated and illustrated by a longitudinal study of bovine digital dermatitis (BDD) on commercial dairy farms in North West England and North Wales, in which the infection of interest is Treponeme spp., and the biomarkers of interest are a continuous enzyme-linked immunosorbent assay test outcome and a dichotomous outcome, foot lesion status. BDD is known to be one of the possible causes of foot lesions in cows.