Evaluation of diagnostic tests using information theory for multi-class diagnostic problems and its application for the detection of occlusal caries lesions

Fifty years of Shannon information theory in assessing the accuracy and agreement of diagnostic tests

Since 1948, Shannon theoretic methods for modeling information have found a wide range of applications in several areas where information plays a key role, which goes well beyond the original scopes for which they have been conceived, namely data compression and error correction over a noisy channel. Among other uses, these methods have been applied in the broad field of medical diagnostics since the 1970s, to quantify diagnostic information, to evaluate diagnostic test performance, but also to be used as technical tools in image processing and registration. This review illustrates the main contributions in assessing the accuracy of diagnostic tests and the agreement between raters, focusing on diagnostic test performance measurements and paired agreement evaluation. This work also presents a recent unified, coherent, and hopefully, final information-theoretical approach to deal with the flows of information involved among the patient, the diagnostic test performed to appraise the state of disease, and the raters who are checking the test results. The approach is assessed by considering two case studies: the first one is related to evaluating extra-prostatic cancers; the second concerns the quality of rapid tests for COVID-19 detection.

Imaging modalities to inform the detection and diagnosis of early caries

BACKGROUND

The detection and diagnosis of caries at the earliest opportunity is fundamental to the preservation of tooth tissue and maintenance of oral health. Radiographs have traditionally been used to supplement the conventional visual-tactile clinical examination. Accurate, timely detection and diagnosis of early signs of disease could afford patients the opportunity of less invasive treatment with less destruction of tooth tissue, reduce the need for treatment with aerosol-generating procedures, and potentially result in a reduced cost of care to the patient and to healthcare services.

OBJECTIVES

To determine the diagnostic accuracy of different dental imaging methods to inform the detection and diagnosis of non-cavitated enamel only coronal dental caries.

SEARCH METHODS

Cochrane Oral Health's Information Specialist undertook a search of the following databases: MEDLINE Ovid (1946 to 31 December 2018); Embase Ovid (1980 to 31 December 2018); US National Institutes of Health Ongoing Trials Register (ClinicalTrials.gov, to 31 December 2018); and the World Health Organization International Clinical Trials Registry Platform (to 31 December 2018). We studied reference lists as well as published systematic review articles.

SELECTION CRITERIA

We included diagnostic accuracy study designs that compared a dental imaging method with a reference standard (histology, excavation, enhanced visual examination), studies that evaluated the diagnostic accuracy of single index tests, and studies that directly compared two or more index tests. Studies reporting at both the patient or tooth surface level were included. In vitro and in vivo studies were eligible for inclusion. Studies that explicitly recruited participants with more advanced lesions that were obviously into dentine or frankly cavitated were excluded. We also excluded studies that artificially created carious lesions and those that used an index test during the excavation of dental caries to ascertain the optimum depth of excavation.

DATA COLLECTION AND ANALYSIS

Two review authors extracted data independently and in duplicate using a standardised data extraction form and quality assessment based on QUADAS-2 specific to the clinical context. Estimates of diagnostic accuracy were determined using the bivariate hierarchical method to produce summary points of sensitivity and specificity with 95% confidence regions. Comparative accuracy of different radiograph methods was conducted based on indirect and direct comparisons between methods. Potential sources of heterogeneity were pre-specified and explored visually and more formally through meta-regression.

MAIN RESULTS

We included 104 datasets from 77 studies reporting a total of 15,518 tooth sites or surfaces. The most frequently reported imaging methods were analogue radiographs (55 datasets from 51 studies) and digital radiographs (42 datasets from 40 studies) followed by cone beam computed tomography (CBCT) (7 datasets from 7 studies). Only 17 studies were of an in vivo study design, carried out in a clinical setting. No studies were considered to be at low risk of bias across all four domains but 16 studies were judged to have low concern for applicability across all domains. The patient selection domain had the largest number of studies judged to be at high risk of bias (43 studies); the index test, reference standard, and flow and timing domains were judged to be at high risk of bias in 30, 12, and 7 studies respectively. Studies were synthesised using a hierarchical bivariate method for meta-analysis. There was substantial variability in the results of the individual studies, with sensitivities that ranged from 0 to 0.96 and specificities from 0 to 1.00. For all imaging methods the estimated summary sensitivity and specificity point was 0.47 (95% confidence interval (CI) 0.40 to 0.53) and 0.88 (95% CI 0.84 to 0.92), respectively. In a cohort of 1000 tooth surfaces with a prevalence of enamel caries of 63%, this would result in 337 tooth surfaces being classified as disease free when enamel caries was truly present (false negatives), and 43 tooth surfaces being classified as diseased in the absence of enamel caries (false positives). Meta-regression indicated that measures of accuracy differed according to the imaging method (Chi2(4) = 32.44, P < 0.001), with the highest sensitivity observed for CBCT, and the highest specificity observed for analogue radiographs. None of the specified potential sources of heterogeneity were able to explain the variability in results. No studies included restored teeth in their sample or reported the inclusion of sealants. We rated the certainty of the evidence as low for sensitivity and specificity and downgraded two levels in total for risk of bias due to limitations in the design and conduct of the included studies, indirectness arising from the in vitro studies, and the observed inconsistency of the results.

AUTHORS' CONCLUSIONS

The design and conduct of studies to determine the diagnostic accuracy of methods to detect and diagnose caries in situ are particularly challenging. Low-certainty evidence suggests that imaging for the detection or diagnosis of early caries may have poor sensitivity but acceptable specificity, resulting in a relatively high number of false-negative results with the potential for early disease to progress. If left untreated, the opportunity to provide professional or self-care practices to arrest or reverse early caries lesions will be missed. The specificity of lesion detection is however relatively high, and one could argue that initiation of non-invasive management (such as the use of topical fluoride), is probably of low risk. CBCT showed superior sensitivity to analogue or digital radiographs but has very limited applicability to the general dental practitioner. However, given the high-radiation dose, and potential for caries-like artefacts from existing restorations, its use cannot be justified in routine caries detection. Nonetheless, if early incidental carious lesions are detected in CBCT scans taken for other purposes, these should be reported. CBCT has the potential to be used as a reference standard in diagnostic studies of this type. Despite the robust methodology applied in this comprehensive review, the results should be interpreted with some caution due to shortcomings in the design and execution of many of the included studies. Future research should evaluate the comparative accuracy of different methods, be undertaken in a clinical setting, and focus on minimising bias arising from the use of imperfect reference standards in clinical studies.

Fluorescence devices for the detection of dental caries

BACKGROUND

Caries is one of the most prevalent and preventable conditions worldwide. If identified early enough then non-invasive techniques can be applied, and therefore this review focusses on early caries involving the enamel surface of the tooth. The cornerstone of caries detection is a visual and tactile dental examination, however alternative methods of detection are available, and these include fluorescence-based devices. There are three categories of fluorescence-based device each primarily defined by the different wavelengths they exploit; we have labelled these groups as red, blue, and green fluorescence. These devices could support the visual examination for the detection and diagnosis of caries at an early stage of decay.

OBJECTIVES

Our primary objectives were to estimate the diagnostic test accuracy of fluorescence-based devices for the detection and diagnosis of enamel caries in children or adults. We planned to investigate the following potential sources of heterogeneity: tooth surface (occlusal, proximal, smooth surface or adjacent to a restoration); single point measurement devices versus imaging or surface assessment devices; and the prevalence of more severe disease in each study sample, at the level of caries into dentine.

SEARCH METHODS

Cochrane Oral Health's Information Specialist undertook a search of the following databases: MEDLINE Ovid (1946 to 30 May 2019); Embase Ovid (1980 to 30 May 2019); US National Institutes of Health Ongoing Trials Register (ClinicalTrials.gov, to 30 May 2019); and the World Health Organization International Clinical Trials Registry Platform (to 30 May 2019). We studied reference lists as well as published systematic review articles.

SELECTION CRITERIA

We included diagnostic accuracy study designs that compared a fluorescence-based device with a reference standard. This included prospective studies that evaluated the diagnostic accuracy of single index tests and studies that directly compared two or more index tests. Studies that explicitly recruited participants with caries into dentine or frank cavitation were excluded.

DATA COLLECTION AND ANALYSIS

Two review authors extracted data independently using a piloted study data extraction form based on the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Sensitivity and specificity with 95% confidence intervals (CIs) were reported for each study. This information has been displayed as coupled forest plots and summary receiver operating characteristic (SROC) plots, displaying the sensitivity-specificity points for each study. We estimated diagnostic accuracy using hierarchical summary receiver operating characteristic (HSROC) methods. We reported sensitivities at fixed values of specificity (median 0.78, upper quartile 0.90).

MAIN RESULTS

We included a total of 133 studies, 55 did not report data in the 2 x 2 format and could not be included in the meta-analysis. 79 studies which provided 114 datasets and evaluated 21,283 tooth surfaces were included in the meta-analysis. There was a high risk of bias for the participant selection domain. The index test, reference standard, and flow and timing domains all showed a high proportion of studies to be at low risk of bias. Concerns regarding the applicability of the evidence were high or unclear for all domains, the highest proportion being seen in participant selection. Selective participant recruitment, poorly defined diagnostic thresholds, and in vitro studies being non-generalisable to the clinical scenario of a routine dental examination were the main reasons for these findings. The dominance of in vitro studies also means that the information on how the results of these devices are used to support diagnosis, as opposed to pure detection, was extremely limited. There was substantial variability in the results which could not be explained by the different devices or dentition or other sources of heterogeneity that we investigated. The diagnostic odds ratio (DOR) was 14.12 (95% CI 11.17 to 17.84). The estimated sensitivity, at a fixed median specificity of 0.78, was 0.70 (95% CI 0.64 to 0.75). In a hypothetical cohort of 1000 tooth sites or surfaces, with a prevalence of enamel caries of 57%, obtained from the included studies, the estimated sensitivity of 0.70 and specificity of 0.78 would result in 171 missed tooth sites or surfaces with enamel caries (false negatives) and 95 incorrectly classed as having early caries (false positives). We used meta-regression to compare the accuracy of the different devices for red fluorescence (84 datasets, 14,514 tooth sites), blue fluorescence (21 datasets, 3429 tooth sites), and green fluorescence (9 datasets, 3340 tooth sites) devices. Initially, we allowed threshold, shape, and accuracy to vary according to device type by including covariates in the model. Allowing consistency of shape, removal of the covariates for accuracy had only a negligible effect (Chi2 = 3.91, degrees of freedom (df) = 2, P = 0.14). Despite the relatively large volume of evidence we rated the certainty of the evidence as low, downgraded two levels in total, for risk of bias due to limitations in the design and conduct of the included studies, indirectness arising from the high number of in vitro studies, and inconsistency due to the substantial variability of results.

AUTHORS' CONCLUSIONS

There is considerable variation in the performance of these fluorescence-based devices that could not be explained by the different wavelengths of the devices assessed, participant, or study characteristics. Blue and green fluorescence-based devices appeared to outperform red fluorescence-based devices but this difference was not supported by the results of a formal statistical comparison. The evidence base was considerable, but we were only able to include 79 studies out of 133 in the meta-analysis as estimates of sensitivity or specificity values or both could not be extracted or derived. In terms of applicability, any future studies should be carried out in a clinical setting, where difficulties of caries assessment within the oral cavity include plaque, staining, and restorations. Other considerations include the potential of fluorescence devices to be used in combination with other technologies and comparative diagnostic accuracy studies.

Beyond kappa: an informational index for diagnostic agreement in dichotomous and multivalue ordered-categorical ratings

Agreement measures are useful tools to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Cohen's kappa (κ) certainly is the most popular agreement method between two raters, and proved its effectiveness in the last sixty years. In spite of that, this method suffers from some alleged issues, which have been highlighted since the 1970s; moreover, its value is strongly dependent on the prevalence of the disease in the considered sample. This work introduces a new agreement index, the informational agreement (IA), which seems to avoid some of Cohen's kappa's flaws, and separates the contribution of the prevalence from the nucleus of agreement. These goals are achieved by modelling the agreement-in both dichotomous and multivalue ordered-categorical cases-as the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. In order to test its fair behaviour and the effectiveness of the method, IA has been tested on some cases known to be problematic for κ, in the machine learning context and in a clinical scenario to compare ultrasound (US) and automated breast volume scanner (ABVS) in the setting of breast cancer imaging. Graphical Abstract To evaluate the agreement between the two raters [Formula: see text] and [Formula: see text] we create an agreement channel, based on Shannon Information Theory, that directly connects the random variables X and Y, that express the raters outcomes. They are the terminals of the chain X⇔ diagnostic test performed by [Formula: see text] ⇔ patient condition[Formula: see text] ⇔ diagnostic test performed by [Formula: see text] ⇔ Y.

Validity of micro-CT for in vitro caries detection: a systematic review and meta-analysis

OBJECTIVE

To investigate the validity of micro-CT for in vitro caries detection in comparison with histology as the reference standard.

METHODS

A systematic search was conducted in the databases Latin American and Caribbean Health Sciences (LILACS), LIVIVO, PubMed, Scopus and Web of Science from their inception to 16 January 2019. Grey literature was searched on Open Grey, ProQuest Dissertations and Theses Database and Google Scholar. In vitro studies assessing the validity of micro-CT for caries detection were included when compared with histology as the reference standard were included. Two authors independently collected the information and sensitivity, specificity, positive and negative likelihood ratios, as well as diagnostic odds ratios were calculated. The risk of bias of the included studies was assessed using the QUADAS-2 tool. Certainty of evidence was assessed with GRADE.

RESULTS

A total of 270 papers were identified, and after a 2-phase selection, 12 studies were included in qualitative and three in quantitative synthesis. For enamel caries diagnostic, sensitivity values ranged from 29.0 to 84.0% indicating high variability while specificity varied from 88.0 to 95.0% indicating good to excellent micro-CT capability do identify the true negative. For dentine caries diagnostic, sensitivity values ranged from 61.0 to 77.0% indicating fair-to-good probability of micro-CT to identify the true positives, while specificity varied from 88.0 to 94.0%. The majority of the included studies presented low risk of bias and moderate certainty of evidence.

CONCLUSIONS

This study demonstrated the validity of micro-CT for in vitro caries detection in comparison with histology.