A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard

Estimation of Diagnostic Test Accuracy Without Gold Standards

The ideal evaluation of diagnostic test performance requires a reference test that is free of errors. However, for many diseases, obtaining such a "gold standard" reference is either impossible or prohibitively expensive. Estimating test accuracy in the absence of a gold standard is therefore a significant challenge. In this article, we introduce and categorize existing methods for evaluating diagnostic tests without a gold standard, considering factors such as the type and number of tests, as well as the structure of the observed data. For each method, we provide a comprehensive introduction and analysis of its underlying assumptions, model architecture, identifiability, estimation techniques, and inference procedures. We use R to conduct simulations for widely applicable models, validating assumptions, comparing models, and assessing their reliability. Additionally, we present real-world examples along with the corresponding R code for these models, enabling readers to better understand how to apply them effectively. Beyond diagnostic medicine, we underscore that the issue of imperfect gold standards affects other fields, drawing parallels to the noisy label problem in machine learning. By highlighting similarities and differences across these domains, we open pathways for further research. The primary aim of this article is to consolidate existing methods for assessing test accuracy in the absence of a gold standard and to provide practical guidance for researchers seeking to apply these methods effectively.

Evaluation of the recombinant protein Sh-TSP-2 for the serological diagnosis of imported urogenital schistosomiasis and comparison with commercially available tests

Different agencies have emphasized the need to evaluate current serological methods for screening patients with suspected urogenital schistosomiasis. However, there is still a lack of evidence regarding the most appropriate methods for this purpose. Here we assessed the diagnostic efficacy of a newly developed serological technique that utilizes the recombinant protein Sh-TSP-2, applied to the urine and serum of migrants suspected of having urogenital schistosomiasis. The sensitivity, specificity, positive and negative predictive values of an in-house enzyme-linked immunosorbent assay (ELISA) using the recombinant protein Sh-TSP-2 were analysed and compared with other commercial serological methods. Due to the limitations of microscopy as a perfect reference method, a latent class analysis (LCA) and composite reference standard (CRS) approach was used to determine the sensitivity and specificity of each test. According to the LCA model, the commercial tests NovaLisa® and immunochromatography test (ICT) immunoglobulin G–immunoglobulin M (IgG–IgM) presented the highest sensitivity (100%), whereas the Sh-TSP-2 serum ELISA test had 79.2%. The Sh-TSP-2 urine and serum ELISA tests had the highest specificities among the serological methods (87.5 and 75%, respectively). CRS modelling showed that the ICT IgG–IgM, NovaLisa® and Sh-TSP-2 serum tests led in sensitivity at 97.1, 88.6 and 71.4%, respectively, with all tests except that the ICT IgG–IgM test having a specificity >90%. Sh-TSP-2 has been validated as a screening tool for patients suspected of having urogenital schistosomiasis. Although commercial serological tests have shown higher sensitivities, Sh-TSP-2 could be valuable for confirming results from tests with lower specificity. Nevertheless, further studies with larger patient cohorts are necessary to fully verify its potential.

Optic Nerve Sheath Measurement on Ultrasound: A Novel Diagnostic Test for Giant Cell Arteritis

OBJECTIVE

Optic nerve sheath enhancement on magnetic resonance imaging has been reported in patients with giant cell arteritis (GCA), with or without visual manifestations. Whether similar findings can be documented on ultrasound is unknown. Optic nerve ultrasound is a point-of-care, easy to learn, rapid, and noninvasive technique. This study aims to investigate whether optic nerve sheath diameter (ONSD) measured on ultrasound is useful in the diagnosis of active, new-onset GCA.

METHODS

A single-center, diagnostic accuracy study was performed from June to November 2022 on consecutive eligible patients referred for suspected GCA. Optic nerve ultrasound was performed on both eyes. The ONSD (includes the optic nerve and its sheath) and optic nerve diameter (OND) were measured 3 mm behind the ocular globe. The presence or absence of GCA was confirmed clinically 6 months later. Multivariable linear regression, adjusting for age and sex, was used to determine the association between optic nerve ultrasound measures and final GCA diagnosis.

RESULTS

Thirty participants were enrolled, including nine participants with a final diagnosis of GCA. Mean ± SD ONSD was 5.98 ± 1.17 mm in patients with GCA and 4.02 ± 0.99 mm in patients without GCA. Mean ONSD was greater by 1.26 mm in patients with GCA (95% confidence interval 0.30-2.21 mm, P = 0.01) compared with those without GCA, adjusting for age and sex. Mean ± SD OND was 2.97 ± 0.46 mm in patients with GCA and 2.47 ± 0.58 mm in patients without GCA. There was no evidence of an association between GCA diagnosis and OND.

CONCLUSION

Patients with GCA had a significantly greater ONSD on ultrasound than patients without GCA. Optic nerve ultrasound may represent a novel, rapid, bedside diagnostic test for GCA. A large prospective study is required to confirm these findings and evaluate whether ONSD can be used as a disease activity biomarker in GCA.

The LEADING Guideline: Reporting Standards for Expert Panel, Best-Estimate Diagnosis, and Longitudinal Expert All Data (LEAD) Studies

Background

Accurate assessments of symptoms and illnesses are essential for health research and clinical practice but face many challenges. The absence of a single error-free measure is currently addressed by assessment methods involving experts reviewing several sources of information to achieve a more accurate or best-estimate assessment. Three bodies of work spanning medicine, psychiatry, and psychology propose similar assessment methods: The Expert Panel, the Best-Estimate Diagnosis, and the Longitudinal Expert All Data (LEAD) method. However, the quality of such best-estimate assessments is typically very difficult to evaluate due to poor reporting of the assessment methods and when they are reported, the reporting quality varies substantially. Here, we tackle this gap by developing reporting guidelines for such best-estimate assessment studies.

Methods

The development of the reporting guidelines followed a four-stage approach: 1) drafting reporting standards accompanied by rationales and empirical evidence, which were further developed with a patient organization for depression, 2) incorporating expert feedback through a two-round Delphi procedure, 3) refining the guideline based on an expert consensus meeting, and 4) testing the guideline by i) having two researchers test it and ii) using it to examine the extent previously published studies report the standards. The last step also provides evidence for the need for the guideline: 10 to 63% (Mean = 33%) of the standards were not reported across thirty randomly selected studies.

Results

The LEADING guideline comprises 20 reporting standards related to four groups: The Longitudinal design (four standards); the Appropriate data (four standards); the Evaluation - experts, materials, and procedures (ten standards); and the Validity group (two standards).

Conclusions

We hope that the LEADING guideline will be useful in assisting researchers in planning, conducting, reporting, and evaluating research aiming to achieve best-estimate assessments.

A review of methods for the analysis of diagnostic tests performed in sequence

BACKGROUND

Many clinical pathways for the diagnosis of disease are based on diagnostic tests that are performed in sequence. The performance of the full diagnostic sequence is dictated by the diagnostic performance of each test in the sequence as well as the conditional dependence between them, given true disease status. Resulting estimates of performance, such as the sensitivity and specificity of the test sequence, are key parameters in health-economic evaluations. We conducted a methodological review of statistical methods for assessing the performance of diagnostic tests performed in sequence, with the aim of guiding data analysts towards classes of methods that may be suitable given the design and objectives of the testing sequence.

METHODS

We searched PubMed, Scopus and Web of Science for relevant papers describing methodology for analysing sequences of diagnostic tests. Papers were classified by the characteristics of the method used, and these were used to group methods into themes. We illustrate some of the methods using data from a cohort study of repeat faecal immunochemical testing for colorectal cancer in symptomatic patients, to highlight the importance of allowing for conditional dependence in test sequences and adjustment for an imperfect reference standard.

RESULTS

Five overall themes were identified, detailing methods for combining multiple tests in sequence, estimating conditional dependence, analysing sequences of diagnostic tests used for risk assessment, analysing test sequences in conjunction with an imperfect or incomplete reference standard, and meta-analysis of test sequences.

CONCLUSIONS

This methodological review can be used to help researchers identify suitable analytic methods for studies that use diagnostic tests performed in sequence.

A standardized protocol using clinical adjudication to define true infection status in patients presenting to the emergency department with suspected infections and/or sepsis

In absence of a "gold standard", a standardized clinical adjudication process was developed for a registrational trial of a transcriptomic host response (HR) test. Two physicians independently reviewed clinical data to adjudicate presence and source of bacterial and viral infections in emergency department patients. Discordant cases were resolved by a third physician. Agreement among 955 cases was 74.1% (708/955) for bacterial, 75.6% (722/955) for viral infections, and 71.2% (680/955) overall. Most discordances were minor (85.2%; 409/480) versus moderate (11.7%; 56/480) or complete (3.3%; 16/480). Concordance levels were lowest for bacterial skin and soft tissue infections (8.2%) and for viral respiratory tract infections (4.5%). This robust adjudication process can be used to evaluate HR tests and other diagnostics by regulatory agencies and for educating clinicians, laboratorians, and clinical researchers. Clinicaltrials.gov NCT04094818. SUMMARY: Without a gold standard for evaluating host response tests, clinical adjudication is a robust reference standard that is essential to determine the true infection status in diagnostic registrational clinical studies.

Pilot deployment of a cloud-based universal medical image repository in a large public health system: A protocol study

This paper outlines the protocol for the deployment of a cloud-based universal medical image repository system. The proposal aims not only at the deployment but also at the automatic expansion of the platform, incorporating Artificial Intelligence (AI) for the analysis of medical image examinations. The methodology encompasses efficient data management through a universal database, along with the deployment of various AI models designed to assist in diagnostic decision-making. By presenting this protocol, the goal is to overcome technical challenges and issues that impact all phases of the workflow, from data management to the deployment of AI models in the healthcare sector. These challenges include ethical considerations, compliance with legal regulations, establishing user trust, and ensuring data security. The system has been deployed, with a tested and validated proof of concept, possessing the capability to receive thousands of images daily and to sustain the ongoing deployment of new AI models to expedite the analysis process in medical image exams.

Practical and analytical considerations when performing interim analyses in diagnostic test accuracy studies

Interim analysis is a common methodology in randomised clinical trials but has received less attention in studies of diagnostic test accuracy. In such studies, early termination for futility may be beneficial if early evidence indicates that a diagnostic test is unlikely to achieve a clinically useful level of diagnostic performance, as measured by the sensitivity and specificity. In this paper, we describe relevant practical and analytical considerations when planning and performing interim analysis in diagnostic accuracy studies, focusing on stopping rules for futility. We present an adaptation of the exact group sequential method for diagnostic testing, with R code provided for implementing this method in practice. The method is illustrated using two simulated data sets and data from a published diagnostic accuracy study for point-of-care testing for SARS-CoV-2. The considerations described in this paper can be used to guide decisions as to when an interim analysis in a diagnostic accuracy study is suitable and highlight areas for further methodological development.

Bibliometric Analysis on Cholecystectomy Surgery in the Nursing Field

PURPOSE

This study aimed to bibliometrically examine nursing publications regarding cholecystectomy surgery.

DESIGN

The study is a bibliometric analysis.

METHODS

The study was conducted in the Web of Science database. The keywords "cholecystectomy AND nursing" were used during the scanning. The data were evaluated in quantitative aspects.

FINDINGS

In the study, 481 publications were identified, publications were made between 1987 and 2023, the average number of years since the release of the publication was 10.2, and the annual increase rate was 8.19%. The most active country was the United States, and the journal most often published in was the "Journal of Perianesthesia Nursing." In the last 3 years, the themes of "meta-analysis, complication, pain, anxiety, quality of life" have been trending, the most working and continuing to develop our "laparoscopic cholecystectomy" and "pain" themes, and the leading theme in the field was "anxiety".

CONCLUSIONS

This study is the first bibliometric study examining nursing studies on cholecystectomy and provides a comprehensive overview of cholecystectomy and nursing issues over 36 years. In quantitative data, it was concluded that although studies in this field have a long history, the number of publications is low, and the annual growth rate is low. According to the quantitative evaluation results, it was found that the most studied topic in this field was laparoscopic cholecystectomy and pain, minimally invasive intervention and perioperative period topics remained isolated, and computerized tomography and gallbladder themes were among the newly emerging themes.

Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses

BACKGROUND

Systematic reviews are performed manually despite the exponential growth of scientific literature.

OBJECTIVE

To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews.

DESIGN

Diagnostic test accuracy study.

SETTING

Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations.

PARTICIPANTS

None.

MEASUREMENTS

A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened.

RESULTS

Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level.

LIMITATIONS

Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time.

CONCLUSION

The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level.

PRIMARY FUNDING SOURCE

None.

Is the Comparator in Your Diagnostic Cost-Effectiveness Model "Standard of Care"? Recommendations from Literature Reviews and Expert Interviews on How to Identify and Operationalize It

OBJECTIVES

This research aimed to develop best-practice recommendations for identifying the "standard of care" (SoC) and integrate it when it is the comparator in diagnostic economic models (SoC comparator).

METHODS

A multi-methods approach comprising 2 pragmatic literature reviews and 9 expert interviews was used. Experts rated their agreement with draft recommendations based on the authors' analysis of the reviews. These were refined iteratively to produce final recommendations.

RESULTS

Fourteen best-practice recommendations are provided. Care pathway mapping (using quantitative, qualitative, or mixed-methods approaches) should be used for identifying the SoC comparator. Guidelines analysis can be integrated with expert opinion to identify pathway variability and discrepancies from clinical practice. For integrating the SoC comparator into the model, recommendations around structure, input sourcing, data aggregation and reporting, input uncertainty, and model variability are presented. For example, modelers should consider that the reference standard is not synonymous with the SoC, and the SoC may not be the only comparator. The comparator limitations should be discussed with clinical experts, but elicitation of its diagnostic accuracy is not recommended. Probabilistic sensitivity analysis is recommended when evaluating the overall input uncertainty, and deterministic sensitivity analysis is useful when there is high model uncertainty or SoC variability. Consensus could not be reached for some topics (eg, the role of real-world data, model averaging, and alternative model structures), but the reported discussions provide points for consideration.

CONCLUSIONS

To our knowledge, this is the first guidance to support modelers when identifying and operationalizing the SoC comparator in diagnostic cost-effectiveness models.

DiTA: helping you search for evidence of diagnostic test accuracy in physical therapy

BACKGROUND

Physical therapists use diagnostic tests in a variety of settings. Choosing the best diagnostic test to apply in a particular situation can be difficult. The choice of diagnostic test should be informed, at least in part, by evidence of test accuracy. Finding evidence of diagnostic test accuracy has, until recently, been challenging. Ideally, there would exist a database that comprehensively indexes evidence on diagnostic tests relevant to physical therapy practice, is free to access, and is easy to use.

OBJECTIVE

This Masterclass will describe the DiTA (Diagnostic Test Accuracy) database (dita.org.au) including its development and search interface, and provide advice on how to search and retrieve records.

DISCUSSION

DiTA indexes more than 2400 primary studies and systematic reviews of diagnostic test accuracy relevant to physical therapy practice. Users can search DiTA using text fields and dropdown lists to find evidence of diagnostic test accuracy. The database is freely accessible on the internet. Since its launch, DiTA has been accessed from almost every country in the world, the largest number of searches having been conducted from Brazil.

Integrity of randomized clinical trials: Performance of integrity tests and checklists requires assessment

The integrity of randomized clinical trials (RCT) has become a concern owing to a recent rise in the number of retractions and the repercussions this has for evidence-based patient care. However, there is little research on the subject of RCT integrity assessment. Recent literature reviews have revealed that journals' authors' instructions concerning integrity and their investigation policies concerning allegations of misconduct are heterogeneous. The judicious use of integrity tests applied to RCT manuscripts is hampered by an absence of data concerning misconduct prevalence (pre-test probability), a failure to evaluate test performance (validity) and a lack of consensus over a gold standard (against which test accuracy can be evaluated). These deficiencies hinder the post-publication correction of RCT records, the integrity evaluations in systematic reviews of RCTs and the prospective application of preventive solutions in RCT peer-review and preprint assessment. Dealing with the current controversy about trustworthiness of RCT evidence requires a strong investment in research, reform and education concerning research integrity. The purpose of this review article is to highlight the current limitations in dealing with trial integrity-related concerns and to propose solutions to some of these issues.

The Accuracy of Computed Tomography Angiography Compared With Technetium-99m Labelled Red Blood Cell Scintigraphy for the Diagnosis and Localization of Acute Gastrointestinal Bleeding: A Systematic Review and Meta-Analysis

Imaging tests are commonly used as an initial or early investigation for patients presenting with suspected acute gastrointestinal bleeding (AGIB). However, controversy remains regarding which of two frequently used modalities, computed tomography angiography (CTA) or technetium-99m labelled red blood cell scintigraphy (RBCS), is most accurate. This systematic review and meta-analysis was performed to compare the accuracy of CTA and RBCS for the detection and localization of AGIB. Five electronic databases were searched with additional manual searching of reference lists of relevant publications identified during the search. Two reviewers independently performed screening, data extraction and methodological assessments. Where appropriate, the bivariate model was used for meta-analysis of sensitivities and specificities for the detection of bleeding and Freeman-Tukey double-arcsine transformation used for meta-analysis of proportions of correctly localized bleeding sites. Forty-four unique primary studies were included: twenty-two investigating CTA, seventeen investigating RBCS and five investigating both modalities. Meta-analysis produced similar pooled sensitivities; 0.83 (95% CI 0.74-0.90) and 0.84 (0.68-0.92) for CTA and RBCS respectively. Pooled specificity for CTA was higher than RBCS; 0.90 (0.72-0.97) and 0.84 (0.71-0.91) respectively. However, differences were not statistically significant. CTA was superior to RBCS in correctly localizing bleeding; pooled proportions of 1.00 (0.98-1.00) and 0.90 (0.83-0.96) respectively (statistically significant difference, P < 0.001). There is no evidence that CTA and RBCS have different diagnostic performance with respect to the detection of AGIB. However, CTA is superior to RBCS in terms of correctly localising the bleeding site, supporting usage of CTA over RBCS as the first line imaging investigation.

Screening for Neglected Tropical Diseases and other infections in African refugees and asylum seekers in Rome and Lazio region, Italy

BACKGROUND

Few reliable data are available on Neglected Tropical Diseases (NTDs) and other infections among African refugees and asylum seekers in Italy. We aimed to estimate the prevalence of NTDs and other infections in a large cohort of African refugees and asylum seekers living in reception centers in Lazio, Italy.

MATERIAL AND METHODS

This is an observational, prospective prevalence study on infectious diseases in a large population of African refugees and asylum seekers (936 overall) consecutively enrolled for screening purpose at the Infectious and Tropical diseases outpatient clinic of the National Institute of Migrant and Poverty (INMP), Rome from August 2019 to December 2020.

RESULTS

We found a prevalence of 8.8 % and 31 % for Strongyloides and schistosoma infection, respectively, while the prevalence of human immunodeficiency virus (HIV) infection was 0.7 %, HCV antibodies 2.5%, hepatitis B virus surface antigen 10.8 % and syphilis serological tests 2.9 %.

CONCLUSION

Strongyloidiasis and schistosomiasis are highly prevalent among African refugees and asylum seekers in Italy, in contrast to communicable diseases (with the exception of hepatitis B). Raising awareness of NTDs among health professionals and implementing guidelines seems to be of paramount importance to prevent these diseases and their sufferers from becoming even more "neglected".

Challenges in the real world use of classification accuracy metrics: From recall and precision to the Matthews correlation coefficient

The accuracy of a classification is fundamental to its interpretation, use and ultimately decision making. Unfortunately, the apparent accuracy assessed can differ greatly from the true accuracy. Mis-estimation of classification accuracy metrics and associated mis-interpretations are often due to variations in prevalence and the use of an imperfect reference standard. The fundamental issues underlying the problems associated with variations in prevalence and reference standard quality are revisited here for binary classifications with particular attention focused on the use of the Matthews correlation coefficient (MCC). A key attribute claimed of the MCC is that a high value can only be attained when the classification performed well on both classes in a binary classification. However, it is shown here that the apparent magnitude of a set of popular accuracy metrics used in fields such as computer science medicine and environmental science (Recall, Precision, Specificity, Negative Predictive Value, J, F1, likelihood ratios and MCC) and one key attribute (prevalence) were all influenced greatly by variations in prevalence and use of an imperfect reference standard. Simulations using realistic values for data quality in applications such as remote sensing showed each metric varied over the range of possible prevalence and at differing levels of reference standard quality. The direction and magnitude of accuracy metric mis-estimation were a function of prevalence and the size and nature of the imperfections in the reference standard. It was evident that the apparent MCC could be substantially under- or over-estimated. Additionally, a high apparent MCC arose from an unquestionably poor classification. As with some other metrics of accuracy, the utility of the MCC may be overstated and apparent values need to be interpreted with caution. Apparent accuracy and prevalence values can be mis-leading and calls for the issues to be recognised and addressed should be heeded.

Missing values and inconclusive results in diagnostic studies - A scoping review of methods

Most diagnostic studies exclude missing values and inconclusive results from the analysis or apply simple methods resulting in biased accuracy estimates. This may be due to the lack of availability or awareness of appropriate methods. This scoping review aimed to provide an overview of strategies to handle missing values and inconclusive results in the reference standard or index test in diagnostic accuracy studies. Conducting a systematic literature search in MEDLINE, Cochrane Library, and Web of Science, we could identify many articles proposing methods for addressing missing values in the reference standard. There are also several articles describing methods regarding missing values or inconclusive results in the index test. The latter encompass imputation, frequentist and Bayesian likelihood, model-based, and latent class methods. While methods for missing values in the reference standard are regularly applied in practice, this is not true for methods addressing missing values and inconclusive results in the index test. Our comprehensive overview and description of available methods may raise further awareness of these methods and will enhance their application. Future research is needed to compare the performance of these methods under different conditions to give valid and robust recommendations for their usage in various diagnostic accuracy research scenarios.

Agreement between Five Experts and the Laguna ONhE Automatic Colourimetric Application Interpreting the Glaucomatous Aspect of the Optic Nerve

BACKGROUND

Optic nerve head (ONH) interpretation is a glaucoma screening method which may be influenced by criteria variability. Laguna ONhE software is a low-cost and non-invasive method of ONH analysis.

METHODS

We analysed the results of the Laguna ONhE application, interpreting 552 ONH images from the ACRIMA database, publicly available on the Internet, and compared them with the opinion of five experts. Diagnostic agreement was investigated using Cohen's kappa (κ) with 95% confidence.

RESULTS

The kappa concordance index obtained with Laguna ONhE and the majority of the experts' criterion (0.77) was significantly higher compared to that obtained with ACRIMA and the majority of the experts' criterion (0.61). In 44.7% of the cases there was absolute agreement among the 5 experts and the Laguna ONhE program. Removing borderline cases from the analysis yielded increased diagnostic agreement (0.81). The area under the receiver operating characteristic (AUROC) of the Laguna ONhE program (0.953, p < 0.001) was not significantly different than AUROC of the majority of the experts' criterion (0.925, p < 0.001), p = 0.052. Individually obtained expert's AUROCs were significantly lower (0.636 to 0.913; p < 0.01).

CONCLUSIONS

Laguna ONhE's agreement with the experts is high, particularly where the diagnosis may be more obvious by the appearance of the ONH.

Development and internal validation of a Neonatal Healthcare-associated infectiOn Prediction score (NeoHoP score) for very low birthweight infants in low-resource settings: a retrospective case-control study

BACKGROUND AND OBJECTIVES

Early diagnosis of neonatal infection is essential to prevent serious complications and to avoid unnecessary use of antibiotics. The prevalence of healthcare-associated infections (HAIs) among very low birthweight (VLBW; <1500 g) infants is 20%; and the mortality in low-resource settings can be as high as 70%. This study aimed to develop an Infection Prediction Score to diagnose bacterial HAIs.

METHODS

A retrospective cohort of VLBW infants investigated for HAI was randomised into two unmatched cohorts. The first cohort was used for development of the score, and the second cohort was used for the internal validation thereof. Potential predictors included risk factors, clinical features, interventions, and laboratory data. The model was developed based on logistic regression analysis.

RESULTS

The study population of 655 VLBW infants with 1116 episodes of clinically suspected HAIs was used to develop the model. The model had five significant variables: capillary refill time >3 s, lethargy, abdominal distention, presence of a central venous catheter in the previous 48 hours and a C reactive protein ≥10 mg/L. The area below the receiver operating characteristic curve was 0.868. A score of ≥2 had a sensitivity of 54.2% and a specificity of 96.4%.

CONCLUSION

A novel Infection Prediction Score for HAIs among VLBW infants may be an important tool for healthcare providers working in low-resource settings but external validation needs to be performed before widespread use can be recommended.

Stool DNA testing for early detection of colorectal cancer: systematic review using the HTA Core Model® for Rapid Relative Effectiveness Assessment

Background

Stool DNA testing for early detection of colorectal cancer (CRC) is a non-invasive technology with the potential to supplement established CRC screening tests. The aim of this health technology assessment was to evaluate effectiveness and safety of currently CE-marked stool DNA tests, compared to other CRC tests in CRC screening strategies in an asymptomatic screening population.

Methods

The assessment was carried out following the guidelines of the European Network for Health Technology Assessment (EUnetHTA). This included a systematic literature search in MED-LINE, Cochrane and EMBASE in 2018. Manufacturers were asked to provide additional data. Five patient interviews helped assessing potential ethical or social aspects and patients' experiences and preferences. We assessed the risk of bias using QUADAS-2, and the quality of the body of evidence using GRADE.

Results

We identified three test accuracy studies, two of which investigated a multitarget stool DNA test (Cologuard®, compared fecal immunochemical test (FIT)) and one a combined DNA stool assay (ColoAlert®, compared to guaiac-based fecal occult blood test (gFOBT), Pyruvate Kinase Isoenzyme Type M2 (M2-PK) and combined gFOBT/M2-PK). We found five published surveys on patient satisfaction. No primary study investigating screening effects on CRC incidence or on overall mortality was found. Both stool DNA tests showed in direct comparison higher sensitivity for the detection of CRC and (advanced) adenoma compared to FIT, or gFOBT, respectively, but had lower specificity. However, these comparative results may depend on the exact type of FIT used. The reported test failure rates were higher for stool DNA testing than for FIT. The certainty of evidence was moderate to high for Cologuard® studies, and low to very low for the ColoAlert® study which refers to a former version of the product and yielded no direct evidence on the test accuracy for ad-vanced versus non-advanced adenoma.

Conclusions

ColoAlert® is the only stool DNA test currently sold in Europe and is available at a lower price than Cologuard®, but reliable evidence is lacking. A screening study including the current product version of ColoAlert® and suitable comparators would, therefore, help evaluate the effectiveness of this screening option in a European context.

Lung Ultrasound Prediction Model for Acute Respiratory Distress Syndrome: A Multicenter Prospective Observational Study

Rationale: Lung ultrasound (LUS) is a promising tool for diagnosis of acute respiratory distress syndrome (ARDS), but adequately sized studies with external validation are lacking. Objectives: To develop and validate a data-driven LUS score for diagnosis of ARDS and compare its performance with that of chest radiography (CXR). Methods: This multicenter prospective observational study included invasively ventilated ICU patients who were divided into a derivation cohort and a validation cohort. Three raters scored ARDS according to the Berlin criteria, resulting in a classification of "certain no ARDS," or "certain ARDS" when experts agreed or "uncertain ARDS" when evaluations conflicted. Uncertain cases were classified in a consensus meeting. Results of a 12-region LUS exam were used in a logistic regression model to develop the LUS-ARDS score. Measurements and Main Results: Three hundred twenty-four (16% certain ARDS) and 129 (34% certain ARDS) patients were included in the derivation cohort and the validation cohort, respectively. With an ARDS diagnosis by the expert panel as the reference test, the LUS-ARDS score, including the left and right LUS aeration scores and anterolateral pleural line abnormalities, had an area under the receiver operating characteristic (ROC) curve of 0.90 (95% confidence interval [CI], 0.85-0.95) in certain patients of the derivation cohort and 0.80 (95% CI, 0.72-0.87) in all patients of the validation cohort. Within patients who had imaging-gold standard chest computed tomography available, diagnostic accuracy of eight independent CXR readers followed the ROC curve of the LUS-ARDS score. Conclusions: The LUS-ARDS score can be used to accurately diagnose ARDS also after external validation. The LUS-ARDS score may be a useful adjunct to a diagnosis of ARDS after further validation, as it showed performance comparable with that of the current practice with experienced CXR readers but more objectifiable diagnostic accuracy at each cutoff.

Feasibility of an online consensus approach for the diagnosis of cognitive impairment and dementia in rural South Africa

INTRODUCTION

We describe the development and feasibility of using an online consensus approach for diagnosing cognitive impairment and dementia in rural South Africa.

METHODS

Cognitive assessments, clinical evaluations, and informant interviews from Cognition and Dementia in the Health and Aging in Africa Longitudinal Study (HAALSI Dementia) were reviewed by an expert panel using a web-based platform to assign a diagnosis of cognitively normal, mild cognitive impairment (MCI), or dementia.

RESULTS

Six hundred thirty-five participants were assigned a final diagnostic category, with 298 requiring adjudication conference calls. Overall agreement between each rater's independent diagnosis and final diagnosis (via the portal or consensus conference) was 78.3%. A moderate level of agreement between raters' individual ratings and the final diagnostic outcomes was observed (average κ coefficient = 0.50).

DISCUSSION

Findings show initial feasibility in using an online consensus approach for the diagnosis of cognitive impairment and dementia in remote, rural, and low-resource settings.

Clinically Adjudicated Reference Standards for Evaluation of Infectious Diseases Diagnostics

Lack of a gold standard can present a challenge for evaluation of diagnostic test accuracy of some infectious diseases tests, particularly when the test's accuracy potentially exceeds that of its predecessors. This approach may measure agreement with an imperfect reference, rather than correctness, because the right answer is unknown. Solutions consist of multitest comparators, including those that involve a test under evaluation if multiple new tests are being evaluated together, using latent class modeling, and clinically adjudicated reference standards. Clinically adjudicated reference standards may be considered as comparator methods when no predefined test or composite of tests is sufficiently accurate; they emulate clinical practice in that multiple data pieces are clinically assessed together.

Heart rate and respiratory rate in predicting risk of serious bacterial infection in febrile children given antipyretics: prospective observational study

Clinical algorithms used in the assessment of febrile children in the Paediatric Emergency Departments are commonly based on threshold values for vital signs, which in children with fever are often outside the normal range. Our aim was to assess the diagnostic value of heart and respiratory rate for serious bacterial infection (SBI) in children after temperature lowering following administration of antipyretics. A prospective cohort of children presenting with fever between June 2014 and March 2015 at the Paediatric Emergency Department of a large teaching hospital in London, UK, was performed. Seven hundred forty children aged 1 month-16 years presenting with a fever and ≥ 1 warning signs of SBI given antipyretics were included. Tachycardia or tachypnoea were defined by different threshold values: (a) APLS threshold values, (b) age-specific and temperature-adjusted centiles charts and (c) relative difference in z-score. SBI was defined by a composite reference standard (cultures from a sterile site, microbiology and virology results, radiological abnormalities, expert panel). Persistent tachypnoea after body temperature lowering was an important predictor of SBI (OR 1.92, 95% CI 1.15, 3.30). This effect was only observed for pneumonia but not other SBIs. Threshold values for tachypnoea > 97th centile at repeat measurement achieved high specificity (0.95 (0.93, 0.96)) and positive likelihood ratios (LR + 3.25 (1.73, 6.11)) and may be useful for ruling in SBI, specifically pneumonia. Persistent tachycardia was not an independent predictor of SBI and had limited value as a diagnostic test.  Conclusion: Among children given antipyretics, tachypnoea at repeat measurement had some value in predicting SBI and was useful to rule in pneumonia. The diagnostic value of tachycardia was poor. Overreliance on heart rate as a diagnostic feature following body temperature lowering may not be justified to facilitate safe discharge. What is Known: • Abnormal vital signs at triage have limited value as a diagnostic test to identify children with SBI, and fever alters the specificity of commonly used threshold values for vital signs. • The observed temperature response after antipyretics is not a clinically useful indicator to differentiate the cause of febrile illness. What is New: • Persistent tachycardia following reduction in body temperature was not associated with an increased risk of SBI and of poor value as a diagnostic test, whilst persistent tachypnoea may indicate the presence of pneumonia.

Utility of investigation for suspected microbial keratitis: a diagnostic accuracy study

PURPOSE

The true disease status of a population with suspected microbial keratitis (MK) cannot be verified. There is not an accurate (gold) reference standard to confirm infection and inter-test comparisons of sensitivity and specificity therefore lead to bias with questionable estimates of test utility. We present an alternative method to report results.

METHODS

We used a decision to treat as the definition for MK. We retrospectively compared the results of corneal culture and polymerase chain reaction (PCR) as these are objective tests available for the three principal groups of pathogens. We then estimated the potential contribution of positive results, either alone or in combination, to support the working diagnosis.

RESULTS

We included 2021 (77.4%) eyes with suspected bacterial keratitis, 365 (14.0%) with suspected acanthamoeba keratitis, and 226 (8.6%) with suspected fungal keratitis, all treated between July 2013 and December 2019. In these groups, there were 51.6% positive culture and 6.5% positive PCR results for bacteria, 19.0% and 40.5% for acanthamoeba, and 28.3% and 15.0% for fungi. Between groups the differences in the proportions of positive results from culture and PCR was statistically significant (P < 0.001). The added benefit of PCR to the result of culture in identifying a potential pathogen was 1.4% for bacteria (P = 0.6292), 24.4% for acanthamoeba (P = 0.0001), and 5.8% for fungi (P = 0.3853).

CONCLUSIONS

For suspected MK a comparison of the test positivity rate is an easily comprehensible outcome measure of test utility.

Reliability of visual review of intracranial electroencephalogram in identifying the seizure onset zone: A systematic review and implications for the accuracy of automated methods

Visual review of intracranial electroencephalography (iEEG) is often an essential component for defining the zone of resection for epilepsy surgery. Unsupervised approaches using machine and deep learning are being employed to identify seizure onset zones (SOZs). This prompts a more comprehensive understanding of the reliability of visual review as a reference standard. We sought to summarize existing evidence on the reliability of visual review of iEEG in defining the SOZ for patients undergoing surgical workup and understand its implications for algorithm accuracy for SOZ prediction. We performed a systematic literature review on the reliability of determining the SOZ by visual inspection of iEEG in accordance with best practices. Searches included MEDLINE, Embase, Cochrane Library, and Web of Science on May 8, 2022. We included studies with a quantitative reliability assessment within or between observers. Risk of bias assessment was performed with QUADAS-2. A model was developed to estimate the effect of Cohen kappa on the maximum possible accuracy for any algorithm detecting the SOZ. Two thousand three hundred thirty-eight articles were identified and evaluated, of which one met inclusion criteria. This study assessed reliability between two reviewers for 10 patients with temporal lobe epilepsy and found a kappa of .80. These limited data were used to model the maximum accuracy of automated methods. For a hypothetical algorithm that is 100% accurate to the ground truth, the maximum accuracy modeled with a Cohen kappa of .8 ranged from .60 to .85 (F-2). The reliability of reviewing iEEG to localize the SOZ has been evaluated only in a small sample of patients with methodologic limitations. The ability of any algorithm to estimate the SOZ is notably limited by the reliability of iEEG interpretation. We acknowledge practical limitations of rigorous reliability analysis, and we propose design characteristics and study questions to further investigate reliability.

Developing a reference standard for pertussis by applying a stratified sampling strategy to electronic medical record data

PURPOSE

Pertussis surveillance remains essential in Canada, but ascertainment bias limits the accuracy of surveillance data. Introducing other sources to improve detection has highlighted the importance of validation. However, challenges arise due to low prevalence, and oversampling suspected cases can introduce partial verification bias. The aim of this study was to build a reference standard for pertussis validation studies that provides adequate analytic precision and minimizes bias.

METHODS

We used a stratified strategy to sample the reference standard from a primary care electronic medical record cohort. We incorporated abstractor notes into definite, possible, ruled-out, and no mention of pertussis classifications which were based on surveillance case definitions.

RESULTS

We abstracted eight hundred records from the cohort of 404,922. There were 208 (26%) definite and 261 (32.6%) possible prevalent pertussis cases. Classifications demonstrated a wide variety of case severities. Abstraction reliability was moderate to substantial based on Cohen's kappa and raw percent agreement.

CONCLUSIONS

When conducting validation studies for pertussis and other low prevalence diseases, this stratified sampling strategy can be used to develop a reference standard using limited resources. This approach mitigates verification and spectrum bias while providing sufficient precision and incorporating a range of case severities.

Core concepts in pharmacoepidemiology: Validation of health outcomes of interest within real-world healthcare databases

Real-world healthcare data, including administrative and electronic medical record databases, provide a rich source of data for the conduct of pharmacoepidemiologic studies but carry the potential for misclassification of health outcomes of interest (HOIs). Validation studies are important ways to quantify the degree of error associated with case-identifying algorithms for HOIs and are crucial for interpreting study findings within real-world data. This review provides a rationale, framework, and step-by-step approach to validating case-identifying algorithms for HOIs within healthcare databases. Key steps in validating a case-identifying algorithm within a healthcare database include: (1) selecting the appropriate health outcome; (2) determining the reference standard against which to validate the algorithm; (3) developing the algorithm using diagnosis codes, diagnostic tests or their results, procedures, drug therapies, patient-reported symptoms or diagnoses, or some combinations of these parameters; (4) selection of patients and sample sizes for validation; (5) collecting data to confirm the HOI; (6) confirming the HOI; and (7) assessing the algorithm's performance. Additional strategies for algorithm refinement and methods to correct for bias due to misclassification of outcomes are discussed. The review concludes by discussing factors affecting the transportability of case-identifying algorithms and the need for ongoing validation as data elements within healthcare databases, such as diagnosis codes, change over time or new variables, such as patient-generated health data, are included in these data sources.

Brain natriuretic peptide and N-terminal brain natriuretic peptide for the diagnosis of haemodynamically significant patent ductus arteriosus in preterm neonates

BACKGROUND

Echocardiogram is the reference standard for the diagnosis of haemodynamically significant patent ductus arteriosus (hsPDA) in preterm infants. A simple blood assay for brain natriuretic peptide (BNP) or amino-terminal pro-B-type natriuretic peptide (NT-proBNP) may be useful in the diagnosis and management of hsPDA, but a summary of the diagnostic accuracy has not been reviewed recently.

OBJECTIVES

Primary objective: To determine the diagnostic accuracy of the cardiac biomarkers BNP and NT-proBNP for diagnosis of haemodynamically significant patent ductus arteriosus (hsPDA) in preterm neonates. Our secondary objectives were: to compare the accuracy of BNP and NT-proBNP; and to explore possible sources of heterogeneity among studies evaluating BNP and NT-proBNP, including type of commercial assay, chronological age of the infant at testing, gestational age at birth, whether used to initiate medical or surgical treatment, test threshold, and criteria of the reference standard (type of echocardiographic parameter used for diagnosis, clinical symptoms or physical signs if data were available).

SEARCH METHODS

We searched the following databases in September 2021: MEDLINE, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Science. We also searched clinical trial registries and conference abstracts. We checked references of included studies and conducted cited reference searches of included studies. We did not apply any language or date restrictions to the electronic searches or use methodological filters, so as to maximise sensitivity.

SELECTION CRITERIA

We included prospective or retrospective, cohort or cross-sectional studies, which evaluated BNP or NT-proBNP (index tests) in preterm infants (participants) with suspected hsPDA (target condition) in comparison with echocardiogram (reference standard).

DATA COLLECTION AND ANALYSIS

Two authors independently screened title/abstracts and full-texts, resolving any inclusion disagreements through discussion or with a third reviewer. We extracted data from included studies to create 2 × 2 tables. Two independent assessors performed quality assessment using the Quality Assessment of Diagnostic-Accuracy Studies-2 (QUADAS 2) tool. We excluded studies that did not report data in sufficient detail to construct 2 × 2 tables, and where this information was not available from the primary investigators. We used bivariate and hierarchical summary receiver operating characteristic (HSROC) random-effects models for meta-analysis and generated summary receiver operating characteristic space (ROC) curves. Since both BNP and NTproBNP are continuous variables, sensitivity and specificity were reported at multiple thresholds. We dealt with the threshold effect by reporting summary ROC curves without summary points.

MAIN RESULTS

We included 34 studies: 13 evaluated BNP and 21 evaluated NT-proBNP in the diagnosis of hsPDA. Studies varied by methodological quality, type of commercial assay, thresholds, age at testing, gestational age and whether the assay was used to initiate medical or surgical therapy. We noted some variability in the definition of hsPDA among the included studies. For BNP, the summary curve is reported in the ROC space (13 studies, 768 infants, low-certainty evidence). The estimated specificities from the ROC curve at fixed values of sensitivities at median (83%), lower and upper quartiles (79% and 92%) were 93.6% (95% confidence interval (CI) 77.8 to 98.4), 95.5% (95% CI 83.6 to 98.9) and 81.1% (95% CI 50.6 to 94.7), respectively. Subgroup comparisons revealed differences by type of assay and better diagnostic accuracy at lower threshold cut-offs (< 250 pg/ml compared to ≥ 250 pg/ml), testing at gestational age < 30 weeks and chronological age at testing at one to three days. Data were insufficient for subgroup analysis of whether the BNP testing was indicated for medical or surgical management of PDA. For NT-proBNP, the summary ROC curve is reported in the ROC space (21 studies, 1459 infants, low-certainty evidence). The estimated specificities from the ROC curve at fixed values of sensitivities at median (92%), lower and upper quartiles (85% and 94%) were 83.6% (95% CI 73.3 to 90.5), 90.6% (95% CI 83.8 to 94.7) and 79.4% (95% CI 67.5 to 87.8), respectively. Subgroup analyses by threshold (< 6000 pg/ml and ≥ 6000 pg/ml) did not reveal any differences. Subgroup analysis by mean gestational age (< 30 weeks vs 30 weeks and above) showed better accuracy with < 30 weeks, and chronological age at testing (days one to three vs over three) showed testing at days one to three had better diagnostic accuracy. Data were insufficient for subgroup analysis of whether the NTproBNP testing was indicated for medical or surgical management of PDA. We performed meta-regression for BNP and NT-proBNP using the covariates: assay type, threshold, mean gestational age and chronological age; none of the covariates significantly affected summary sensitivity and specificity.

AUTHORS' CONCLUSIONS

Low-certainty evidence suggests that BNP and NT-proBNP have moderate accuracy in diagnosing hsPDA and may work best as a triage test to select infants for echocardiography. The studies evaluating the diagnostic accuracy of BNP and NT-proBNP for hsPDA varied considerably by assay characteristics (assay kit and threshold) and infant characteristics (gestational and chronological age); hence, generalisability between centres is not possible. We recommend that BNP or NT-proBNP assays be locally validated for specific populations and outcomes, to initiate therapy or follow response to therapy.

Assessing Bias and Limitations of Clinical Validation Studies of Molecular Diagnostic Tests for Indeterminate Thyroid Nodules: Systematic Review and Meta-Analysis

Background: Molecular tests for thyroid nodules with indeterminate fine needle aspiration results are increasingly used in clinical practice; however, true diagnostic summaries of these tests are unknown. A systematic review and meta-analysis were completed to (1) evaluate the accuracy of commercially available molecular tests for malignancy in indeterminate thyroid nodules and (2) quantify biases and limitations in studies that validate those tests. Summary: PubMed, EMBASE, and Web of Science were systematically searched through July 2021. English language articles that reported original clinical validation attempts of molecular tests for indeterminate thyroid nodules were included if they reported counts of true-negative, true-positive, false-negative, and false-positive results. We performed screening and full-text review, followed by assessment of eight common biases and limitations, extraction of diagnostic and histopathological information, and meta-analysis of clinical validity using a bivariate linear mixed-effects model. Forty-nine studies were included. Meta-analysis of Afirma Gene expression classifiers (GEC; n = 38 studies) revealed a sensitivity of 0.92 (confidence interval: 0.90-0.94), specificity of 0.26 (0.20-0.32), negative likelihood ratio (LR-) of 0.32 (0.23-0.44), positive LR+ of 1.24 (1.15-1.35), and area under the curve (AUC) of 0.83 (0.74-0.89). Afirma Genomic Sequencing Classifier (GSC; n = 10) had a sensitivity of 0.94 (0.89-0.96), specificity of 0.38 (0.27-0.50), LR- of 0.18 (0.10-0.30), LR+ of 1.52 (1.28-1.87), and AUC of 0.91 (0.62-0.92). ThyroSeq v1 and v2 (n = 10) had a sensitivity of 0.86 (0.82-0.90), specificity of 0.74 (0.59-0.85), LR- of 0.19 (0.13-0.26), LR+ of 3.52 (2.08-5.92), and AUC of 0.86 (0.81-0.90). ThyroSeq v3 (n = 6) had a sensitivity of 0.92 (0.86-0.95), specificity of 0.41 (0.18-0.69), LR- of 0.24 (0.09-0.62), LR+ of 1.67 (1.09-2.98), and AUC of 0.90 (0.63-0.92). Fourteen percent of studies conducted a blinded histopathologic review of excised thyroid nodules, and 8% made the decision to go to surgery blind to molecular test results. Conclusions: Meta-analyses reveal a high diagnostic accuracy of molecular tests for thyroid nodule assessment of malignancy risk; however, these studies are subject to several limitations. Limitations and their potential clinical impacts must be addressed and, when feasible, adjusted for using valid statistical methodologies.

Diagnostic accuracy of rapid point-of-care tests for diagnosis of current SARS-CoV-2 infections in children: a systematic review and meta-analysis

OBJECTIVE

To systematically assess the diagnostic accuracy of rapid point-of-care tests for diagnosis of current SARS-CoV-2 infections in children under real-life conditions.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

MEDLINE, Embase, Cochrane Database for Systematic Reviews, INAHTA HTA database, preprint servers (via Europe PMC), ClinicalTrials.gov, WHO ICTRP from 1 January 2020 to 7 May 2021; NICE Evidence Search, NICE Guidance, FIND Website from 1 January 2020 to 24 May 2021.

REVIEW METHODS

Diagnostic cross-sectional or cohort studies were eligible for inclusion if they had paediatric study participants and compared rapid point-of care tests for diagnosing current SARS-CoV-2 infections with reverse transcription polymerase chain reaction (RT-PCR) as the reference standard. The Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool was used to assess the risk of bias and the applicability of the included studies. Bivariate meta-analyses with random effects were performed. Variability was assessed by subgroup analyses.

RESULTS

17 studies with a total of 6355 paediatric study participants were included. All studies compared antigen tests against RT-PCR. Overall, studies evaluated eight antigen tests from six different brands. Only one study was at low risk of bias. The pooled overall diagnostic sensitivity and specificity in paediatric populations was 64.2% (95% CI 57.4% to 70.5%) and 99.1% (95% CI 98.2% to 99.5%), respectively. In symptomatic children, the pooled diagnostic sensitivity was 71.8% (95% CI 63.6% to 78.8%) and the pooled diagnostic specificity was 98.7% (95% CI 96.6% to 99.5%). The pooled diagnostic sensitivity in asymptomatic children was 56.2% (95% CI 47.6% to 64.4%) and the pooled diagnostic specificity was 98.6% (95% CI 97.3% to 99.3%).

CONCLUSIONS

The performance of current antigen tests in paediatric populations under real-life conditions varies broadly. Relevant data were only identified for very few antigen tests on the market, and the risk of bias was mostly unclear due to poor reporting. Additionally, the most common uses of these tests in children (eg, self-testing in schools or parents testing their toddlers before kindergarten) have not been addressed in clinical performance studies yet. The observed low diagnostic sensitivity may impact the planned purpose of the broad implementation of testing programmes.

PROSPERO REGISTRATION NUMBER

CRD42021236313.

Commentary: Statistical analysis of 2 x 2 tables in Biomarker studies 1) The Four "indices of test validity"

Abstract/SummaryDiagnostic statistics such as sensitivity and specificity are widely used in the assessment of biomarkers. Interpretation of these and other statistics derived from a 2 x 2 table can be complex. The properties of the commonly used statistics are discussed. The object is to provide help in their interpretation for authors designing studies and the subsequent reporting of results and to referees and others who assess such papers.

Bayesian latent class analysis produced diagnostic accuracy estimates that were more interpretable than composite reference standards for extrapulmonary tuberculosis tests

BACKGROUND

Evaluating the accuracy of extrapulmonary tuberculosis (TB) tests is challenging due to lack of a gold standard. Latent class analysis (LCA), a statistical modeling approach, can adjust for reference tests' imperfect accuracies to produce less biased test accuracy estimates than those produced by commonly used methods like composite reference standards (CRSs). Our objective is to illustrate how Bayesian LCA can address the problem of an unavailable gold standard and demonstrate how it compares to using CRSs for extrapulmonary TB tests.

METHODS

We re-analyzed a dataset of presumptive extrapulmonary TB cases in New Delhi, India, for three forms of extrapulmonary TB. Results were available for culture, smear microscopy, Xpert MTB/RIF, and a non-microbiological test, cytopathology/histopathology, or adenosine deaminase (ADA). A diagram was used to define assumed relationships between observed tests and underlying latent variables in the Bayesian LCA with input from an inter-disciplinary team. We compared the results to estimates obtained from a sequence of CRSs defined by increasing numbers of positive reference tests necessary for positive disease status.

RESULTS

Data were available from 298, 388, and 230 individuals with presumptive TB lymphadenitis, meningitis, and pleuritis, respectively. Using Bayesian LCA, estimates were obtained for accuracy of all tests and for extrapulmonary TB prevalence. Xpert sensitivity neared that of culture for TB lymphadenitis and meningitis but was lower for TB pleuritis, and specificities of all microbiological tests approached 100%. Non-microbiological tests' sensitivities were high, but specificities were only moderate, preventing disease rule-in. CRSs' only provided estimates of Xpert and these varied widely per CRS definition. Accuracy of the CRSs also varied by definition, and no CRS was 100% accurate.

CONCLUSION

Unlike CRSs, Bayesian LCA takes into account known information about test performance resulting in accuracy estimates that are easier to interpret. LCA should receive greater consideration for evaluating extrapulmonary TB diagnostic tests.

LOw-dose CT Or Lung UltraSonography versus standard of care based-strategies for the diagnosis of pneumonia in the elderly: protocol for a multicentre randomised controlled trial (OCTOPLUS)

INTRODUCTION

Pneumonia is a leading cause of mortality and a common indication for antibiotic in elderly patients. However, its diagnosis is often inaccurate. We aim to compare the diagnostic accuracy, the clinical and cost outcomes and the use of antibiotics associated with three imaging strategies in patients >65 years old with suspected pneumonia in the emergency room (ER): chest X-ray (CXR, standard of care), low-dose CT scan (LDCT) or lung ultrasonography (LUS).

METHODS AND ANALYSIS

This is a multicentre randomised superiority clinical trial with three parallel arms. Patients will be allocated in the ER to a diagnostic strategy based on either CXR, LDCT or LUS. All three imaging modalities will be performed but the results of two of them will be masked during 5 days to the patients, the physicians in charge of the patients and the investigators according to random allocation. The primary objective is to compare the accuracy of LDCT versus CXR-based strategies. As secondary objectives, antibiotics prescription, clinical and cost outcomes will be compared, and the same analyses repeated to compare the LUS and CXR strategies. The reference diagnosis will be established a posteriori by a panel of experts. Based on a previous study, we expect an improvement of 16% of the accuracy of pneumonia diagnosis using LDCT instead of CXR. Under this assumption, and accounting for 10% of drop-out, the enrolment of 495 patients is needed to prove the superiority of LDCT over CRX (alpha error=0.05, beta error=0.10).

ETHICS AND DISSEMINATION

Ethical approval: CER Geneva 2019-01288.

TRIAL REGISTRATION NUMBER

NCT04978116.

Accuracy of a CBCT-based virtual injection software for vessel detection during hepatic arterial embolization

PURPOSE

To assess the accuracy, sensitivity, positive predictive value (PPV) and interobserver agreement of a virtual injection (VI) software that simulates selective arterial injection from nonselective cone-beam CT (CBCT) arteriography.

METHODS

From March 2019 to May 2020, 20 consecutive patients in whom a nonselective injected CBCT and a selective CT angiography (CTA) were completed in the same procedure, were retrospectively included. The position of the microcatheter tip used for selective CTA injection was identified. The VI was simulated from the exact same point on the nonselective CBCT and the two volumes were merged. VI was compared to the real injection on the selective CTA. Three interventional radiologists evaluated the accuracy using a 6-point scale (Perfect; Good; Fair; Incorrect Origin; False Negative; Non existing). Sensitivity, PPV, and Fleiss' kappa were calculated. Numerical variables were presented as means ± standard deviations.

RESULTS

Twenty procedures and 195 vessel segments were analyzed. Most vessels were 4th order (57/195; 29%) and 5th order (96/195; 49%). VI was classified as perfect to good in 96.8% ± 1.4 of 1st-3rd order arteries and in 83.4% ± 0.4 of 4th-5th order arteries. Interobserver agreement was substantial (Fleiss' kappa = 0.79; 95% confidence interval = 0.73-0.84, P < 0.01). False negatives were reported with a mean of 9.4% ± 0.3. Average sensitivity was 90.6% ± 0.3 and average PPV was 92.7% ± 0.02. Fourteen false positives were noted.

CONCLUSION

CBCT-based VI software accurately simulated distal injections in the liver with high sensitivity and a substantial interobserver agreement.

Clinical validity of the smallest oblique sagittal area of the neural foramen in patients with suspected cervical spondylotic radiculopathy

OBJECTIVE

To evaluate the smallest oblique sagittal area of the neural foramen in detecting cervical spondylotic radiculopathy (CSR) and to determine its potential significance for treatment decisions.

METHODS

The subjects of the study were patients with CSR who visited the spine surgery from 2016 to 2019. All patients were compared according to the minimum oblique sagittal area and the cut-off point value, and they were divided into positive and negative parameters. The changes in neck disability index (NDI), Japanese Orthopaedic Association score (JOA), and visual analog scale (VAS) during the two treatment groups from baseline to at least 24 months of follow-up were compared.

RESULTS

In the surgery group, there was no significant difference in symptom improvement between patients with positive and negative parameters. In the non-surgical group, for patients with positive parameters, NDI decreased by 2.35, JOA increased by 0.88, and neck VAS score improved by 0.42. For patients with negative parameters, NDI decreased by 10.32, JOA increased by 2.86 on average, and neck VAS score improved by 2.46 points on average (both p<0.01 on t test).

CONCLUSIONS

Patients with both positive and negative parameters showed significant improvement in their symptoms after surgery, and the smallest oblique sagittal area of the neural foramen seems to be unable to predict the outcome of the surgery. However, in non-surgical patients, symptomatic improvement was more limited in patients with positive parameters than in those with negative parameters. This suggests that patients with positive parameters may be more suitable for surgery and those with negative parameters are more suitable for conservative treatment.

A Dynamic Nomogram to Predict the Risk of Stroke in Emergency Department Patients With Acute Dizziness

Objective

To develop a risk prediction tool for acute ischemic stroke (AIS) for patients presenting to the emergency department (ED) with acute dizziness/vertigo or imbalance.

Method

A prospective, multicenter cohort study was designed, and adult patients presenting with dizziness/vertigo or imbalance within 14 days were consecutively enrolled from the EDs of 4 tertiary hospitals between August 10, 2020, and June 10, 2021. Stroke was diagnosed by CT or MRI performed within 14 days of symptom onset. Participants were followed-up for 30 days. The least absolute shrinkage and selection operator (LASSO) logistic regression analysis was conducted to extract predictive factors that best identified patients at high risk of stroke to establish a prediction model. Model discrimination and calibration were assessed and its prediction performance was compared with the age, blood pressure, clinical features, duration, and diabetes (ABCD2) score, nystagmus scheme, and finger to nose test.

Results

In this study, 790 out of 2,360 patients were enrolled {median age, 60.0 years [interquartile range (IQR), 51–68 years]; 354 (44.8%) men}, with complete follow-up data available. AIS was identified in 80 patients. An online web service tool (https://neuroby.shinyapps.io/dynnomapp/) was developed for stroke risk prediction, including the variables of sex, trigger, isolated symptom, nausea, history of brief dizziness, high blood pressure, finger to nose test, and tandem gait test. The model exhibited excellent discrimination with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.889 (95% CI: 0.855–0.923), compared with the ABCD2 score, nystagmus scheme, and finger to nose test [0.712 (95% CI, 0.652–0.771), 0.602 (95% CI, 0.556–0.648), and 61.7 (95% CI, 0.568–0.666) respectively].

Conclusion

Our new prediction model exhibited good performance and could be useful for stroke identification in patients presenting with dizziness, vertigo, or imbalance. Further externally validation study is needed to increase the strength of our findings.

Development and evaluation of safety and effectiveness of novel cancer screening tests for routine clinical use with applications to multicancer detection technologies

Multicancer screening is a promising approach to improving the detection of preclinical disease, but current technologies have limited ability to identify precursor or early stage lesions, and approaches for developing the evidentiary chain are unclear. Frameworks to enable development and evaluation from discovery through evidence of clinical effectiveness are discussed.

Comparing ascertainment of chronic condition status with problem lists versus encounter diagnoses from electronic health records

OBJECTIVE

To assess and compare electronic health record (EHR) documentation of chronic disease in problem lists and encounter diagnosis records among Community Health Center (CHC) patients.

MATERIALS AND METHODS

We assessed patient EHR data in a large clinical research network during 2012-2019. We included CHCs who provided outpatient, older adult primary care to patients age ≥45 years, with ≥2 office visits during the study. Our study sample included 1 180 290 patients from 545 CHCs across 22 states. We used diagnosis codes from 39 Chronic Condition Warehouse algorithms to identify chronic conditions from encounter diagnoses only and compared against problem list records. We measured correspondence including agreement, kappa, prevalence index, bias index, and prevalence-adjusted bias-adjusted kappa.

RESULTS

Overlap of encounter diagnosis and problem list ascertainment was 59.4% among chronic conditions identified, with 12.2% of conditions identified only in encounters and 28.4% identified only in problem lists. Rates of coidentification varied by condition from 7.1% to 84.4%. Greatest agreement was found in diabetes (84.4%), HIV (78.1%), and hypertension (74.7%). Sixteen conditions had <50% agreement, including cancers and substance use disorders. Overlap for mental health conditions ranged from 47.4% for anxiety to 59.8% for depression.

DISCUSSION

Agreement between the 2 sources varied substantially. Conditions requiring regular management in primary care settings may have a higher agreement than those diagnosed and treated in specialty care.

CONCLUSION

Relying on EHR encounter data to identify chronic conditions without reference to patient problem lists may under-capture conditions among CHC patients in the United States.

Statistical methods for evaluating the fine needle aspiration cytology procedure in breast cancer diagnosis

Background

Statistical issues present while evaluating a diagnostic procedure for breast cancer are non rare but often ignored, leading to biased results. We aimed to evaluate the diagnostic accuracy of the fine needle aspiration cytology(FNAC), a minimally invasive and rapid technique potentially used as a rule-in or rule-out test, handling its statistical issues: suspect test results and verification bias.

Methods

We applied different statistical methods to handle suspect results by defining conditional estimates. When considering a partial verification bias, Begg and Greenes method and multivariate imputation by chained equations were applied, however, and a Bayesian approach with respect to each gold standard was used when considering a differential verification bias. At last, we extended the Begg and Greenes method to be applied conditionally on the suspect results.

Results

The specificity of the FNAC test above 94%, was always higher than its sensitivity regardless of the proposed method. All positive likelihood ratios were higher than 10, with variations among methods. The positive and negative yields were high, defining precise discriminating properties of the test.

Conclusion

The FNAC test is more likely to be used as a rule-in test for diagnosing breast cancer. Our results contributed in advancing our knowledge regarding the performance of FNAC test and the methods to be applied for its evaluation.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-022-01506-y).

Usefulness of computed tomography in the diagnosis of acute pyelonephritis in older patients suspected of infection with unknown focus

BACKGROUND

In older adults, the diagnosis of acute pyelonephritis is challenging because of non-specific symptoms and false-positive urine test results. Few studies have investigated the diagnostic performance of computed tomography (CT) signs.

PURPOSE

To evaluate the diagnostic performance of CT signs for acute pyelonephritis in older patients suspected of infection with unknown focus.

MATERIAL AND METHODS

This cross-sectional study was conducted between 2015 and 2018. Patients aged ≥65 years who underwent blood cultures, urine culture, and non-contrast or contrast-enhanced CT on admission were included. Cases with clinically presumable infection focus before CT were excluded. Two radiologists blinded to clinical information independently reviewed five CT signs: perirenal fat stranding; pelvicalyceal wall thickening and enhancement; renal enlargement; thickening of Gerota's fascia; and area(s) of decreased attenuation. The final diagnoses were made by a clinical expert panel.

RESULTS

Among 473 eligible patients, 61 were diagnosed with acute pyelonephritis. When the laterality of findings between the left and right kidneys were considered, the positive and negative likelihood ratios of perirenal fat stranding were 4.0 (95% confidence interval [CI] = 2.3-7.0) and 0.8 (95% CI = 0.7-0.9) in non-contrast CT, respectively. The other signs in non-contrast CT showed similar diagnostic performance with positive and negative likelihood ratios of 3.5-11.3 and 0.8-0.9, respectively.

CONCLUSION

CT signs can help physicians diagnose acute pyelonephritis in older patients suspected of infection with unknown focus.

Quality assessment standards in artificial intelligence diagnostic accuracy systematic reviews: a meta-research study

Artificial intelligence (AI) centred diagnostic systems are increasingly recognised as robust solutions in healthcare delivery pathways. In turn, there has been a concurrent rise in secondary research studies regarding these technologies in order to influence key clinical and policymaking decisions. It is therefore essential that these studies accurately appraise methodological quality and risk of bias within shortlisted trials and reports. In order to assess whether this critical step is performed, we undertook a meta-research study evaluating adherence to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool within AI diagnostic accuracy systematic reviews. A literature search was conducted on all studies published from 2000 to December 2020. Of 50 included reviews, 36 performed the quality assessment, of which 27 utilised the QUADAS-2 tool. Bias was reported across all four domains of QUADAS-2. Two hundred forty-three of 423 studies (57.5%) across all systematic reviews utilising QUADAS-2 reported a high or unclear risk of bias in the patient selection domain, 110 (26%) reported a high or unclear risk of bias in the index test domain, 121 (28.6%) in the reference standard domain and 157 (37.1%) in the flow and timing domain. This study demonstrates the incomplete uptake of quality assessment tools in reviews of AI-based diagnostic accuracy studies and highlights inconsistent reporting across all domains of quality assessment. Poor standards of reporting act as barriers to clinical implementation. The creation of an AI-specific extension for quality assessment tools of diagnostic accuracy AI studies may facilitate the safe translation of AI tools into clinical practice.