Correcting the 3MS for bias does not improve accuracy when screening for cognitive impairment or dementia

Should Cognitive Screening Tests Be Corrected for Age and Education? Insights From a Causal Perspective

Cognitive screening tests such as the Mini-Mental State Examination are widely used in clinical routine to predict cognitive impairment. The raw test scores are often corrected for age and education, although documented poorer discrimination performance of corrected scores has challenged this practice. Nonetheless, test correction persists, perhaps due to the seemingly counterintuitive nature of the underlying problem. We used a causal framework to inform the long-standing debate from a more intuitive angle. We illustrate and quantify the consequences of applying the age-education correction of cognitive tests on discrimination performance. In an effort to bridge theory and practical implementation, we computed differences in discrimination performance under plausible causal scenarios using Open Access Series of Imaging Studies (OASIS)-1 data. We show that when age and education are causal risk factors for cognitive impairment and independently also affect the test score, correcting test scores for age and education removes meaningful information, thereby diminishing discrimination performance.

Comparing Detection of Alzheimer's and Vascular Disease-Related Cognitive Impairment With Brief Cognitive Screens

Objectives: The study compared the accuracy of the Mini-Mental State Examination (MMSE) with its modified version (3MS) in distinguishing healthy older adults from adults with cognitive impairment due to suspected Alzheimer’s disease (AD) or vascular disease (VaD). Method: Participants were 98 veterans who underwent comprehensive neuropsychological evaluation due to concern for cognitive decline. Participants were selected via retrospective chart review on the basis of diagnosis. They had diagnoses of mild or major neurocognitive disorder due to suspected AD (N=20), mild or major neurocognitive disorder due to suspected VaD (N=44), or no neurocognitive diagnosis (i.e., healthy adult comparisons; HC, N=34). Results: The 3MS demonstrated superior detection of cognitive impairment. The extent of this enhanced detection was influenced by the suspected etiology of cognitive impairment. The 3MS and MMSE had comparable discrimination of AD and HC. With respect to VaD, the 3MS showed superior discriminability compared to the MMSE. Conclusions: Overall, results support the adoption of the 3MS over that of the MMSE. The 3MS is a superior (and free) tool for detecting cognitive impairment in geriatric populations. Its use is recommended for first-line screening of cognitive symptoms in older adult populations, especially those with concern for VaD.

The Italian Version of the Test Your Memory (TYM-I): A Tool to Detect Mild Cognitive Impairment in the Clinical Setting

The Test Your Memory (TYM) is a brief self-administered, cognitive screening test, currently used in several settings. It requires minimal administrator supervision and the computation of the final test score takes approximately 2 min. We assessed the discrimination ability of the Italian version of the TYM (TYM-I) in detecting Mild Cognitive Impairment (MCI) in clinical setting. TYM-I was administered to 94 MCI patients and 134 healthy controls. The clinical diagnosis of MCI was considered as the gold standard. An extended formal neuropsychological test battery was used to define MCI subtypes. Receiver Operating Characteristic (ROC) analyses were conducted to find the optimal cut-off and measure discrimination ability of TYM-I in detecting MCI. TYM-I had a similar area under the curve (AUC = 0.85) point estimate as Mini Mental State Examination (MMSE) (AUC = 0.83). A TYM-I score lower or equal to 36 was found to be optimal cut off to detect MCI. The TYM-I showed the highest discrimination ability among individuals aged more than 70 and high educational level (AUC = 0.89). The amnestic MCI subtype patients, compared to non-amnestic MCI patients, had worse performance in recall, orientation and visuospatial abilities TYM-I subscores. The TYM-I is a valid screening test in detecting cognitive dysfunction, easily carried out in clinical practice. The TYM-I subscores may allow to identify amnestic and non-amnestic MCI subtypes.

The utility of the Modified Mini-Mental State Examination in inpatient rehabilitation for traumatic brain injury: preliminary findings

OBJECTIVE

The Modified Mini-Mental State Examination (3MS) could provide useful information about cognitive status in traumatic brain injury (TBI), yet has not been validated in this population. We studied the reliability, construct validity, clinical responsiveness, and cognitive impairment classification-ability of the 3MS compared to the Mini-Mental State Examination (MMSE).

METHODS

Adult participants receiving inpatient rehabilitation services for TBI were administered the 3MS and MMSE at admission and discharge (n = 72). Construct validity and classification agreement were assessed through relationships of each measure with cognitive items of the Functional Independence Measure^TM (FIM) and rehabilitation length of stay (LOS).

RESULTS

3MS reliability at admission and discharge (Cronbach's alphas = .871 and .839, respectively) exceeded that for the MMSE (Cronbach's alpha = .748 and .653, respectively). 3MS construct validity was marginally better than for the MMSE as assessed through correlations with FIM cognitive scores and LOS. Standard error of measurement as a percentage of the total scale was lower for the 3MS; responsiveness of the 3MS was superior as assessed by the 95% confidence interval for minimal detectable change. Cognitive impairment classification-ability was superior for the 3MS.

CONCLUSIONS

While both instruments had reasonable psychometric properties, the 3MS had a superior psychometric profile in the acute phase of TBI.

Mini-Mental State Examination (MMSE) for the detection of dementia in clinically unevaluated people aged 65 and over in community and primary care populations

BACKGROUND

The Mini Mental State Examination (MMSE) is a cognitive test that is commonly used as part of the evaluation for possible dementia.

OBJECTIVES

To determine the diagnostic accuracy of the Mini-Mental State Examination (MMSE) at various cut points for dementia in people aged 65 years and over in community and primary care settings who had not undergone prior testing for dementia.

SEARCH METHODS

We searched the specialised register of the Cochrane Dementia and Cognitive Improvement Group, MEDLINE (OvidSP), EMBASE (OvidSP), PsycINFO (OvidSP), LILACS (BIREME), ALOIS, BIOSIS previews (Thomson Reuters Web of Science), and Web of Science Core Collection, including the Science Citation Index and the Conference Proceedings Citation Index (Thomson Reuters Web of Science). We also searched specialised sources of diagnostic test accuracy studies and reviews: MEDION (Universities of Maastricht and Leuven, www.mediondatabase.nl), DARE (Database of Abstracts of Reviews of Effects, via the Cochrane Library), HTA Database (Health Technology Assessment Database, via the Cochrane Library), and ARIF (University of Birmingham, UK, www.arif.bham.ac.uk). We attempted to locate possibly relevant but unpublished data by contacting researchers in this field. We first performed the searches in November 2012 and then fully updated them in May 2014. We did not apply any language or date restrictions to the electronic searches, and we did not use any methodological filters as a method to restrict the search overall.

SELECTION CRITERIA

We included studies that compared the 11-item (maximum score 30) MMSE test (at any cut point) in people who had not undergone prior testing versus a commonly accepted clinical reference standard for all-cause dementia and subtypes (Alzheimer disease dementia, Lewy body dementia, vascular dementia, frontotemporal dementia). Clinical diagnosis included all-cause (unspecified) dementia, as defined by any version of the Diagnostic and Statistical Manual of Mental Disorders (DSM); International Classification of Diseases (ICD) and the Clinical Dementia Rating.

DATA COLLECTION AND ANALYSIS

At least three authors screened all citations.Two authors handled data extraction and quality assessment. We performed meta-analysis using the hierarchical summary receiver-operator curves (HSROC) method and the bivariate method.

MAIN RESULTS

We retrieved 24,310 citations after removal of duplicates. We reviewed the full text of 317 full-text articles and finally included 70 records, referring to 48 studies, in our synthesis. We were able to perform meta-analysis on 28 studies in the community setting (44 articles) and on 6 studies in primary care (8 articles), but we could not extract usable 2 x 2 data for the remaining 14 community studies, which we did not include in the meta-analysis. All of the studies in the community were in asymptomatic people, whereas two of the six studies in primary care were conducted in people who had symptoms of possible dementia. We judged two studies to be at high risk of bias in the patient selection domain, three studies to be at high risk of bias in the index test domain and nine studies to be at high risk of bias regarding flow and timing. We assessed most studies as being applicable to the review question though we had concerns about selection of participants in six studies and target condition in one study.The accuracy of the MMSE for diagnosing dementia was reported at 18 cut points in the community (MMSE score 10, 14-30 inclusive) and 10 cut points in primary care (MMSE score 17-26 inclusive). The total number of participants in studies included in the meta-analyses ranged from 37 to 2727, median 314 (interquartile range (IQR) 160 to 647). In the community, the pooled accuracy at a cut point of 24 (15 studies) was sensitivity 0.85 (95% confidence interval (CI) 0.74 to 0.92), specificity 0.90 (95% CI 0.82 to 0.95); at a cut point of 25 (10 studies), sensitivity 0.87 (95% CI 0.78 to 0.93), specificity 0.82 (95% CI 0.65 to 0.92); and in seven studies that adjusted accuracy estimates for level of education, sensitivity 0.97 (95% CI 0.83 to 1.00), specificity 0.70 (95% CI 0.50 to 0.85). There was insufficient data to evaluate the accuracy of the MMSE for diagnosing dementia subtypes.We could not estimate summary diagnostic accuracy in primary care due to insufficient data.

AUTHORS' CONCLUSIONS

The MMSE contributes to a diagnosis of dementia in low prevalence settings, but should not be used in isolation to confirm or exclude disease. We recommend that future work evaluates the diagnostic accuracy of tests in the context of the diagnostic pathway experienced by the patient and that investigators report how undergoing the MMSE changes patient-relevant outcomes.

Age-correction of test scores reduces the validity of mild cognitive impairment in predicting progression to dementia

Objectives

A phase of mild cognitive impairment (MCI) precedes most forms of neurodegenerative dementia. Many definitions of MCI recommend the use of test norms to diagnose cognitive impairment. It is, however, unclear whether the use of norms actually improves the detection of individuals at risk of dementia. Therefore, the effects of age- and education-norms on the validity of test scores in predicting progression to dementia were investigated.

Methods

Baseline cognitive test scores (Syndrome Short Test) of dementia-free participants aged ≥65 were used to predict progression to dementia within three years. Participants were comprehensively examined one, two, and three years after baseline. Test scores were calculated with correction for (1) age and education, (2) education only, (3) age only and (4) without correction. Predictive validity was estimated with Cox proportional hazard regressions. Areas under the curve (AUCs) were calculated for the one-, two-, and three-year intervals.

Results

82 (15.3%) of initially 537 participants, developed dementia. Model coefficients, hazard ratios, and AUCs of all scores were significant (p<0.001). Predictive validity was the lowest with age-corrected scores (−2 log likelihood  = 840.90, model fit χ² (1)  = 144.27, HR  = 1.33, AUCs between 0.73 and 0.87) and the highest with education-corrected scores (−2 log likelihood  = 815.80, model fit χ² (1)  = 171.16, HR  = 1.34, AUCs between 0.85 and 0.88).

Conclusion

The predictive validity of test scores is markedly reduced by age-correction. Therefore, definitions of MCI should not recommend the use of age-norms in order to improve the detection of individuals at risk of dementia.

Operationalizing diagnostic criteria for Alzheimer's disease and other age-related cognitive impairment-Part 1

In this article, the challenges faced by several noted population studies for Alzheimer dementia in operationalizing current clinical diagnostic criteria for Alzheimer's disease (AD) have been reviewed. Differences in case ascertainment, methodological biases, cultural and educational influences on test performance, inclusion of special populations such as underrepresented minorities and the oldest old, and detection of the earliest symptomatic stages of underlying AD have been considered. Classification of Alzheimer dementia may be improved by the incorporation of biomarkers for AD if the sensitivity, specificity, and predictive value of the biomarkers are established and if they are appropriate for epidemiological studies, as may occur should a plasma biomarker be developed. Biomarkers for AD could also facilitate studies of the interactions of various forms of neurodegenerative disorders with cerebrovascular disease, resulting in "mixed dementia".

Demographic corrections appear to compromise classification accuracy for severely skewed cognitive tests

Demographic corrections for cognitive tests should improve classification accuracy by reducing age or education biases, but empirical support has been equivocal. Using a simulation procedure, we show that creating moderate or extreme skewness in cognitive tests compromises the classification accuracy of demographic corrections, findings that appear replicated within clinical data for the few neuropsychological test scores with an extreme degree of skew. For most neuropsychological tests, the dementia classification accuracy of raw and demographically corrected scores was equivalent. These findings suggest that the dementia classification accuracy of demographic corrections is robust to slight degrees of skew (i.e., skewness <1.5).

Age corrections and dementia classification accuracy

In contrast to expectations, demographic corrections to reduce biases against those of advanced age or few years of education does not universally improve diagnostic classification accuracy. Age corrections may be particularly problematic because age is also a risk factor for a dementia diagnosis. We found that simulating increased risk for dementia based on demographic variables, such as age, reduced the overall classification accuracy for demographically corrected simulated scores relative to the raw, uncorrected test scores. In clinical data with a small magnitude of association between age and dementia diagnosis, we found equivalent overall classification accuracy for demographically corrected and raw test scores. Regardless of the overall classification accuracy results, cutoff comparisons (16th and 9th percentiles) in clinical and simulated data demonstrated that for the most part, the sensitivity of raw scores was higher than the sensitivity of demographically corrected scores, but the specificity of scores corrected with normative data was superior.

Review of normative data for common screening measures used to evaluate cognitive functioning in elderly individuals

When conducting neuropsychological evaluations of the elderly, it is important to compare patients' test scores to appropriate normative data to maximize diagnostic and descriptive accuracy. Many sets of normative data are now available for screening measures that assess cognitive functioning in the elderly. This article systematically reviewed available norms for 6 widely used screening measures of cognitive functioning in elderly patients. Details regarding the sample characteristics and data collection methods are provided for each set of norms, thereby providing a useful reference for clinicians.

A review of screening tests for cognitive impairment

The merit of screening for dementia and cognitive impairment has been the subject of recent debate. One of the main limitations in this regard is the lack of robust evidence to support the many screening tests available. Although plentiful in number, few such instruments have been well validated in the populations for which they are intended to be used. In addition, it is likely that "one size does not fit all" in cognitive screening, leading to the development of many specialised tests for particular types of impairment. In this review, we sought to ascertain the number of screening tools currently available, and to examine the evidence for their validity in detecting different diagnoses in a variety of populations. A further consideration was whether each screen elicited indices of a range of cognitive, affective and functional domains or abilities, as such information is a valuable adjunct to simple cut-off scores. Thirty-nine screens were identified and discussed with reference to three purposes: brief assessment in the doctor's office; large scale community screening programmes; and identifying profiles of impairment across different cognitive, psychiatric and functional domains/abilities, to guide differential diagnosis and further assessment. A small number of screens rated highly for both validity and content. This review is intended to serve as an evaluative resource, to guide clinicians and researchers in choosing among the wide range of screens which are currently available.

Brief cognitive assessment in a UK population sample -- distributional properties and the relationship between the MMSE and an extended mental state examination

Background

Despite the MMSE's known flaws, it is still used extensively as both a screening instrument for dementia and a population measure of cognitive ability. The aim of this paper is to provide data on the distribution of MMSE scores in a representative sample from the UK population and to compare it with an extended cognitive assessment (EMSE) which covers a wider range of cognitive domains and provides a wider range of difficulty levels.

Methods

The MMSE and the EMSE were administered to over 12,000 participants at the screening stage of the MRC Cognitive Function and Ageing Study (MRC CFAS). MRC CFAS is a multi-centre population-based study in England and Wales with respondents aged 65 years and older.

Results

Normative values on the MMSE and EMSE are presented by age group, sex and level of education. There are very large differences between age groups, with smaller differences seen between the sexes and by level of education. The EMSE extends the scores at the high end of the ability range, but is no better than the MMSE at differentiating between dementia and non-dementia.

Conclusion

Population-derived norms are valuable for comparing an individual's score to the score that would be expected among the general population, given the individual's specific demographic characteristics.