Meta-analysis of test accuracy studies using imputation for partial reporting of multiple thresholds

Clinical judgement by primary care physicians for the diagnosis of all-cause dementia or cognitive impairment in symptomatic people

BACKGROUND

In primary care, general practitioners (GPs) unavoidably reach a clinical judgement about a patient as part of their encounter with patients, and so clinical judgement can be an important part of the diagnostic evaluation. Typically clinical decision making about what to do next for a patient incorporates clinical judgement about the diagnosis with severity of symptoms and patient factors, such as their ideas and expectations for treatment. When evaluating patients for dementia, many GPs report using their own judgement to evaluate cognition, using information that is immediately available at the point of care, to decide whether someone has or does not have dementia, rather than more formal tests.

OBJECTIVES

To determine the diagnostic accuracy of GPs' clinical judgement for diagnosing cognitive impairment and dementia in symptomatic people presenting to primary care. To investigate the heterogeneity of test accuracy in the included studies.

SEARCH METHODS

We searched MEDLINE (Ovid SP), Embase (Ovid SP), PsycINFO (Ovid SP), Web of Science Core Collection (ISI Web of Science), and LILACs (BIREME) on 16 September 2021.

SELECTION CRITERIA

We selected cross-sectional and cohort studies from primary care where clinical judgement was determined by a GP either prospectively (after consulting with a patient who has presented to a specific encounter with the doctor) or retrospectively (based on knowledge of the patient and review of the medical notes, but not relating to a specific encounter with the patient). The target conditions were dementia and cognitive impairment (mild cognitive impairment and dementia) and we included studies with any appropriate reference standard such as the Diagnostic and Statistical Manual of Mental Disorders (DSM), International Classification of Diseases (ICD), aetiological definitions, or expert clinical diagnosis.

DATA COLLECTION AND ANALYSIS

Two review authors screened titles and abstracts for relevant articles and extracted data separately with differences resolved by consensus discussion. We used QUADAS-2 to evaluate the risk of bias and concerns about applicability in each study using anchoring statements. We performed meta-analysis using the bivariate method.

MAIN RESULTS

We identified 18,202 potentially relevant articles, of which 12,427 remained after de-duplication. We assessed 57 full-text articles and extracted data on 11 studies (17 papers), of which 10 studies had quantitative data. We included eight studies in the meta-analysis for the target condition dementia and four studies for the target condition cognitive impairment. Most studies were at low risk of bias as assessed with the QUADAS-2 tool, except for the flow and timing domain where four studies were at high risk of bias, and the reference standard domain where two studies were at high risk of bias. Most studies had low concern about applicability to the review question in all QUADAS-2 domains. Average age ranged from 73 years to 83 years (weighted average 77 years). The percentage of female participants in studies ranged from 47% to 100%. The percentage of people with a final diagnosis of dementia was between 2% and 56% across studies (a weighted average of 21%). For the target condition dementia, in individual studies sensitivity ranged from 34% to 91% and specificity ranged from 58% to 99%. In the meta-analysis for dementia as the target condition, in eight studies in which a total of 826 of 2790 participants had dementia, the summary diagnostic accuracy of clinical judgement of general practitioners was sensitivity 58% (95% confidence interval (CI) 43% to 72%), specificity 89% (95% CI 79% to 95%), positive likelihood ratio 5.3 (95% CI 2.4 to 8.2), and negative likelihood ratio 0.47 (95% CI 0.33 to 0.61). For the target condition cognitive impairment, in individual studies sensitivity ranged from 58% to 97% and specificity ranged from 40% to 88%. The summary diagnostic accuracy of clinical judgement of general practitioners in four studies in which a total of 594 of 1497 participants had cognitive impairment was sensitivity 84% (95% CI 60% to 95%), specificity 73% (95% CI 50% to 88%), positive likelihood ratio 3.1 (95% CI 1.4 to 4.7), and negative likelihood ratio 0.23 (95% CI 0.06 to 0.40). It was impossible to draw firm conclusions in the analysis of heterogeneity because there were small numbers of studies. For specificity we found the data were compatible with studies that used ICD-10, or applied retrospective judgement, had higher reported specificity compared to studies with DSM definitions or using prospective judgement. In contrast for sensitivity, we found studies that used a prospective index test may have had higher sensitivity than studies that used a retrospective index test.

AUTHORS' CONCLUSIONS

Clinical judgement of GPs is more specific than sensitive for the diagnosis of dementia. It would be necessary to use additional tests to confirm the diagnosis for either target condition, or to confirm the absence of the target conditions, but clinical judgement may inform the choice of further testing. Many people who a GP judges as having dementia will have the condition. People with false negative diagnoses are likely to have less severe disease and some could be identified by using more formal testing in people who GPs judge as not having dementia. Some false positives may require similar practical support to those with dementia, but some - such as some people with depression - may suffer delayed intervention for an alternative treatable pathology.

A semiparametric approach for meta-analysis of diagnostic accuracy studies with multiple cut-offs

The accuracy of a diagnostic test is often expressed using a pair of measures: sensitivity (proportion of test positives among all individuals with target condition) and specificity (proportion of test negatives among all individuals without target condition). If the outcome of a diagnostic test is binary, results from different studies can easily be summarized in a meta-analysis. However, if the diagnostic test is based on a discrete or continuous measure (e.g., a biomarker), several cut-offs within one study as well as among different studies are published. Instead of taking all information of the cut-offs into account in the meta-analysis, a single cut-off per study is often selected arbitrarily for the analysis, even though there are statistical methods for the incorporation of several cut-offs. For these methods, distributional assumptions have to be met and/or the models may not converge when specific data structures occur. We propose a semiparametric approach to overcome both problems. Our simulation study shows that the diagnostic accuracy is under-estimated, although this underestimation in sensitivity and specificity is relatively small. The comparative approach of Steinhauser et al. is better in terms of coverage probability, but may lead to convergence problems. In addition to the simulation results, we illustrate the application of the semiparametric approach using a published meta-analysis for a diagnostic test differentiating between bacterial and viral meningitis in children.

Effectiveness of walking training on balance, motor functions, activity, participation and quality of life in people with chronic stroke: a systematic review with meta-analysis and meta-regression of recent randomized controlled trials

PURPOSE

To review and quantify the effects of walking training for the improvement of various aspects of physical function of people with chronic stroke.

METHODS

We conducted a systematic search and meta-analysis of randomized controlled trials (RCTs) of chronic stroke rehabilitation interventions published from 2008 to 2020 in English or French. Of the 6476-screened articles collated from four databases, 15 RCTs were included and analyzed. We performed a meta-regression with the total training time as dependent variable in order to have a better understanding of how did the training dosage affect the effect sizes.

RESULTS

Treadmill walking training was more effective on balance and motor functions (standardized mean difference (SMD)=0.70[0.02, 1.37], p = 0.04) and 0.56[0.15, 0.96], p = 0.007 respectively). Overground walking training improved significantly walking endurance (SMD = 0.38[0.16, 0.59], p < 0.001), walking speed (MD = 0.12[0.05, 0.18], p < 0.001), participation (SMD = 0.35[0.02, 0.68], p = 0.04) and quality of life (SMD = 0.46[0.12, 0.80], p = 0.008). Aquatic training improved balance (SMD = 2.41[1.20, 3.62], p < 0.001). The Meta-regression analysis did not show significant effect of total training time on the effect sizes.

CONCLUSION

Treadmill and overground walking protocols consisting of ≥30 min sessions conducted at least 3 days per week for about 8 weeks are beneficial for improving motor impairments, activity limitations, participation, and quality of life in people with chronic stroke.Implications for rehabilitationTreadmill walking training is effective for improving balance and motor functions.Overground walking training improved significantly walking endurance, walking speed, participation and quality of life.Treadmill and overground walking protocols consisting of ≥30 min sessions conducted at least 3 days per week for about 8 weeks are beneficial for improving motor impairments, activity limitations, participation, and quality of life in patient with chronic stroke.

Incomplete high dimensional data streams clustering

Many recent applications such as sensor networks generate continuous and time varying data streams that are often gathered from multiple data sources with some incompleteness and high dimensionality. Clustering such incomplete high dimensional streaming data faces four constraints which are 1) data incompleteness, 2) high dimensionality of data, 3) data distribution, 4) data streams’ continuous nature. Thus, in this paper, we propose the Subspace clustering for Incomplete High dimensional Data streams (SIHD) framework that overcomes the above clustering issues. The proposed SIHD provides continuous missing values imputation for incomplete streams based on the corresponding nearest-neighbors’ intervals. An adaptive subspace clustering mechanism is proposed to deal with such incomplete high dimensional data streams. Our experimental results using two different data sets prove the efficiency of the proposed SIHD framework in clustering such incomplete high dimensional data streams in terms of accuracy, precision, sensitivity, specificity, and F-score compared to five algorithms GFCM, GBDC-P2P, DS, Ensemble, and DMSC. The proposed SIHD improved: 1) the accuracy on average over the five algorithms in the same mentioned order by 11.3%, 10.8%, 6.5%, 4.1%, and 3.6%, 2) the precision by 15%, 10.6%, 6.4%, 4%, and 3.5%, 3) the sensitivity by 16.6%, 10.6%, 5.8%, 4.2%, and 3.6%, 4) the specificity by 16.8%, 10.9%, 6.5%, 4%, and 3.5%, 5) the F-score by 16.6%, 10.7%, 6.6%, 4.1%, and 3.6%.

An empirical comparison of three methods for multiple cutoff diagnostic test meta-analysis of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool using published data vs individual level data

Selective cutoff reporting in primary diagnostic accuracy studies with continuous or ordinal data may result in biased estimates when meta-analyzing studies. Collecting individual participant data (IPD) and estimating accuracy across all relevant cutoffs for all studies can overcome such bias but is labour intensive. We meta-analyzed the diagnostic accuracy of the Patient Health Questionnaire-9 (PHQ-9) depression screening tool. We compared results for two statistical methods proposed by Steinhauser and by Jones to account for missing cutoffs, with results from a series of bivariate random effects models (BRM) estimated separately at each cutoff. We applied the methods to a dataset that contained information only on cutoffs that were reported in the primary publications and to the full IPD dataset that contained information for all cutoffs for every study. For each method, we estimated pooled sensitivity and specificity and associated 95% confidence intervals for each cutoff and area under the curve (AUC). The full IPD dataset comprised data from 45 studies, 15 020 subjects, and 1972 cases of major depression and included information on every possible cutoff. When using data available in publications, using statistical approaches outperformed the BRM applied to the same data. AUC was similar for all approaches when using the full IPD dataset, though pooled estimates were slightly different. Overall, using statistical methods to fill in missing cutoff data recovered the receiver operating characteristic (ROC) curve from the full IPD dataset well when using only the published subset. All methods performed similarly when applied to the full IPD dataset.

A pseudo-likelihood approach for multivariate meta-analysis of test accuracy studies with multiple thresholds

Multivariate meta-analysis of test accuracy studies when tests are evaluated in terms of sensitivity and specificity at more than one threshold represents an effective way to synthesize results by fully exploiting the data, if compared to univariate meta-analyses performed at each threshold independently. The approximation of logit transformations of sensitivities and specificities at different thresholds through a normal multivariate random-effects model is a recent proposal that straightforwardly extends the bivariate models well recommended for the one threshold case. However, drawbacks of the approach, such as poor estimation of the within-study correlations between sensitivities and between specificities, and severe computational issues can make it unappealing. We propose an alternative method for inference on common diagnostic measures using a pseudo-likelihood constructed under a working independence assumption between sensitivities and between specificities at different thresholds in the same study. The method does not require within-study correlations, overcomes the convergence issues and can be effortlessly implemented. Simulation studies highlight a satisfactory performance of the method, remarkably improving the results from the multivariate normal counterpart under different scenarios. The pseudo-likelihood approach is illustrated in the evaluation of a test used for diagnosis of preeclampsia as a cause of maternal and perinatal morbidity and mortality.

Quantifying how diagnostic test accuracy depends on threshold in a meta-analysis

Tests for disease often produce a continuous measure, such as the concentration of some biomarker in a blood sample. In clinical practice, a threshold C is selected such that results, say, greater than C are declared positive and those less than C negative. Measures of test accuracy such as sensitivity and specificity depend crucially on C, and the optimal value of this threshold is usually a key question for clinical practice. Standard methods for meta‐analysis of test accuracy (i) do not provide summary estimates of accuracy at each threshold, precluding selection of the optimal threshold, and furthermore, (ii) do not make use of all available data. We describe a multinomial meta‐analysis model that can take any number of pairs of sensitivity and specificity from each study and explicitly quantifies how accuracy depends on C. Our model assumes that some prespecified or Box‐Cox transformation of test results in the diseased and disease‐free populations has a logistic distribution. The Box‐Cox transformation parameter can be estimated from the data, allowing for a flexible range of underlying distributions. We parameterise in terms of the means and scale parameters of the two logistic distributions. In addition to credible intervals for the pooled sensitivity and specificity across all thresholds, we produce prediction intervals, allowing for between‐study heterogeneity in all parameters. We demonstrate the model using two case study meta‐analyses, examining the accuracy of tests for acute heart failure and preeclampsia. We show how the model can be extended to explore reasons for heterogeneity using study‐level covariates.

Red cell distribution width to platelet ratio for liver fibrosis: a systematic review and meta-analysis of diagnostic accuracy

Introduction: Red cell distribution width to platelet ratio (RPR) may be a useful marker for the evaluation of liver fibrosis in chronic liver disease (CLD). We sought to investigate its value in fibrosis-related outcomes in a meta-analysis of diagnostic accuracy. Areas covered: We searched MEDLINE (1966-2019), Clinicaltrials.gov (2008-2019), Cochrane Central Register of Controlled Trials (CENTRAL) (1999-2019), Google Scholar (2004-2019) and WHO (International Clinical Trials Register Platform) databases using a structured algorithm. The articles were assessed by Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2). In over 1,800 patients for each outcome, pooled sensitivity and specificity for a) significant fibrosis, b) advanced fibrosis and c) cirrhosis were: a) 0.635 and 0.769 with an AUC of 0.747, b) 0.607 and 0.783 with an AUC of 0.773, c) 0.739 and 0.768 with an AUC of 0.818 respectively. Similar results were found for chronic hepatitis B in all outcomes. Subgroup analysis indicated a high specificity for advanced fibrosis detection in primary biliary cirrhosis. Sensitivity analysis did not alter the results. Expert opinion: RPR is a good predictor of fibrosis, especially as severity of chronic liver disease progresses. Future research should elucidate its value in specific etiologies of chronic liver disease.

Biochemical tests of placental function versus ultrasound assessment of fetal size for stillbirth and small-for-gestational-age infants

BACKGROUND

Stillbirth affects 2.6 million pregnancies worldwide each year. Whilst the majority of cases occur in low- and middle-income countries, stillbirth remains an important clinical issue for high-income countries (HICs) - with both the UK and the USA reporting rates above the mean for HICs. In HICs, the most frequently reported association with stillbirth is placental dysfunction. Placental dysfunction may be evident clinically as fetal growth restriction (FGR) and small-for-dates infants. It can be caused by placental abruption or hypertensive disorders of pregnancy and many other disorders and factorsPlacental abnormalities are noted in 11% to 65% of stillbirths. Identification of FGA is difficult in utero. Small-for-gestational age (SGA), as assessed after birth, is the most commonly used surrogate measure for this outcome. The degree of SGA is associated with the likelihood of FGR; 30% of infants with a birthweight < 10th centile are thought to be FGR, while 70% of infants with a birthweight < 3rd centile are thought to be FGR. Critically, SGA is the most significant antenatal risk factor for a stillborn infant. Correct identification of SGA infants is associated with a reduction in the perinatal mortality rate. However, currently used tests, such as measurement of symphysis-fundal height, have a low reported sensitivity and specificity for the identification of SGA infants.

OBJECTIVES

The primary objective was to assess and compare the diagnostic accuracy of ultrasound assessment of fetal growth by estimated fetal weight (EFW) and placental biomarkers alone and in any combination used after 24 weeks of pregnancy in the identification of placental dysfunction as evidenced by either stillbirth, or birth of a SGA infant. Secondary objectives were to investigate the effect of clinical and methodological factors on test performance.

SEARCH METHODS

We developed full search strategies with no language or date restrictions. The following sources were searched: MEDLINE, MEDLINE In Process and Embase via Ovid, Cochrane (Wiley) CENTRAL, Science Citation Index (Web of Science), CINAHL (EBSCO) with search strategies adapted for each database as required; ISRCTN Registry, UK Clinical Trials Gateway, WHO International Clinical Trials Portal and ClinicalTrials.gov for ongoing studies; specialist abstract and conference proceeding resources (British Library's ZETOC and Web of Science Conference Proceedings Citation Index). Search last conducted in Ocober 2016.

SELECTION CRITERIA

We included studies of pregnant women of any age with a gestation of at least 24 weeks if relevant outcomes of pregnancy (live birth/stillbirth; SGA infant) were assessed. Studies were included irrespective of whether pregnant women were deemed to be low or high risk for complications or were of mixed populations (low and high risk). Pregnancies complicated by fetal abnormalities and multi-fetal pregnancies were excluded as they have a higher risk of stillbirth from non-placental causes. With regard to biochemical tests, we included assays performed using any technique and at any threshold used to determine test positivity.

DATA COLLECTION AND ANALYSIS

We extracted the numbers of true positive, false positive, false negative, and true negative test results from each study. We assessed risk of bias and applicability using the QUADAS-2 tool. Meta-analyses were performed using the hierarchical summary ROC model to estimate and compare test accuracy.

MAIN RESULTS

We included 91 studies that evaluated seven tests - blood tests for human placental lactogen (hPL), oestriol, placental growth factor (PlGF) and uric acid, ultrasound EFW and placental grading and urinary oestriol - in a total of 175,426 pregnant women, in which 15,471 pregnancies ended in the birth of a small baby and 740 pregnancies which ended in stillbirth. The quality of included studies was variable with most domains at low risk of bias although 59% of studies were deemed to be of unclear risk of bias for the reference standard domain. Fifty-three per cent of studies were of high concern for applicability due to inclusion of only high- or low-risk women.Using all available data for SGA (86 studies; 159,490 pregnancies involving 15,471 SGA infants), there was evidence of a difference in accuracy (P < 0.0001) between the seven tests for detecting pregnancies that are SGA at birth. Ultrasound EFW was the most accurate test for detecting SGA at birth with a diagnostic odds ratio (DOR) of 21.3 (95% CI 13.1 to 34.6); hPL was the most accurate biochemical test with a DOR of 4.78 (95% CI 3.21 to 7.13). In a hypothetical cohort of 1000 pregnant women, at the median specificity of 0.88 and median prevalence of 19%, EFW, hPL, oestriol, urinary oestriol, uric acid, PlGF and placental grading will miss 50 (95% CI 32 to 68), 116 (97 to 133), 124 (108 to 137), 127 (95 to 152), 139 (118 to 154), 144 (118 to 161), and 144 (122 to 161) SGA infants, respectively. For the detection of pregnancies ending in stillbirth (21 studies; 100,687 pregnancies involving 740 stillbirths), in an indirect comparison of the four biochemical tests, PlGF was the most accurate test with a DOR of 49.2 (95% CI 12.7 to 191). In a hypothetical cohort of 1000 pregnant women, at the median specificity of 0.78 and median prevalence of 1.7%, PlGF, hPL, urinary oestriol and uric acid will miss 2 (95% CI 0 to 4), 4 (2 to 8), 6 (6 to 7) and 8 (3 to 13) stillbirths, respectively. No studies assessed the accuracy of ultrasound EFW for detection of pregnancy ending in stillbirth.

AUTHORS' CONCLUSIONS

Biochemical markers of placental dysfunction used alone have insufficient accuracy to identify pregnancies ending in SGA or stillbirth. Studies combining U and placental biomarkers are needed to determine whether this approach improves diagnostic accuracy over the use of ultrasound estimation of fetal size or biochemical markers of placental dysfunction used alone. Many of the studies included in this review were carried out between 1974 and 2016. Studies of placental substances were mostly carried out before 1991 and after 2013; earlier studies may not reflect developments in test technology.

Meta-analysis of test accuracy studies using imputation for partial reporting of multiple thresholds

Introduction

For tests reporting continuous results, primary studies usually provide test performance at multiple but often different thresholds. This creates missing data when performing a meta‐analysis at each threshold. A standard meta‐analysis (no imputation [NI]) ignores such missing data. A single imputation (SI) approach was recently proposed to recover missing threshold results. Here, we propose a new method that performs multiple imputation of the missing threshold results using discrete combinations (MIDC).

Methods

The new MIDC method imputes missing threshold results by randomly selecting from the set of all possible discrete combinations which lie between the results for 2 known bounding thresholds. Imputed and observed results are then synthesised at each threshold. This is repeated multiple times, and the multiple pooled results at each threshold are combined using Rubin's rules to give final estimates. We compared the NI, SI, and MIDC approaches via simulation.

Results

Both imputation methods outperform the NI method in simulations. There was generally little difference in the SI and MIDC methods, but the latter was noticeably better in terms of estimating the between‐study variances and generally gave better coverage, due to slightly larger standard errors of pooled estimates. Given selective reporting of thresholds, the imputation methods also reduced bias in the summary receiver operating characteristic curve. Simulations demonstrate the imputation methods rely on an equal threshold spacing assumption. A real example is presented.

Conclusions

The SI and, in particular, MIDC methods can be used to examine the impact of missing threshold results in meta‐analysis of test accuracy studies.