Protocol for a construct and clinical validation study of MyCog Mobile: a remote smartphone-based cognitive screener for older adults

INTRODUCTION

Annual cognitive screening in older adults is essential for early detection of cognitive impairment, yet primary care settings face time constraints that present barriers to routine screening. A remote cognitive screener completed on a patient's personal smartphone before a visit has the potential to save primary care clinics time, encourage broader screening practices and increase early detection of cognitive decline. MyCog Mobile is a promising new remote smartphone-based cognitive screening app for primary care settings. We propose a combined construct and clinical validation study of MyCog Mobile.

METHODS AND ANALYSIS

We will recruit a total sample of 300 adult participants aged 65 years and older. A subsample of 200 healthy adult participants and a subsample of 100 adults with a cognitive impairment diagnosis (ie, dementia, mild cognitive impairment, cognitive deficits or other memory loss) will be recruited from the general population and specialty memory care centres, respectively. To evaluate the construct validity of MyCog Mobile, the healthy control sample will self-administer MyCog Mobile on study-provided smartphones and be administered a battery of gold-standard neuropsychological assessments. We will compare correlations between performance on MyCog Mobile and measures of similar and dissimilar constructs to evaluate convergent and discriminant validity. To assess clinical validity, participants in the clinical sample will self-administer MyCog Mobile on a smartphone and be administered a Mini-Cog screener and these data will be combined with the healthy control sample. We will then apply several supervised model types to determine the best predictors of cognitive impairment within the sample. Area under the receiver operating characteristic curve, accuracy, sensitivity and specificity will be the primary performance metrics for clinical validity.

ETHICS AND DISSEMINATION

The Institutional Review Board at Northwestern University (STU00214921) approved this study protocol. Results will be published in peer-reviewed journals and summaries provided to the study's funders.

Development and Validation of a Vocabulary Measure in the Mobile Toolbox

OBJECTIVE

We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies.

METHOD

Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones.

RESULTS

The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001).

CONCLUSION

The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.

Remote Self-Administration of Cognitive Screeners for Older Adults Prior to a Primary Care Visit: Pilot Cross-Sectional Study of the Reliability and Usability of the MyCog Mobile Screening App

BACKGROUND

Routine cognitive screening is essential in the early detection of dementia, but time constraints in primary care settings often limit clinicians' ability to conduct screenings. MyCog Mobile is a newly developed cognitive screening system that patients can self-administer on their smartphones before a primary care visit, which can help save clinics' time, encourage broader screening practices, and increase early detection of cognitive decline.

OBJECTIVE

The goal of this pilot study was to examine the feasibility, acceptability, and initial psychometric properties of MyCog Mobile. Research questions included (1) Can older adults complete MyCog Mobile remotely without staff support? (2) Are the internal consistency and test-retest reliability of the measures acceptable? and (3) How do participants rate the user experience of MyCog Mobile?

METHODS

A sample of adults aged 65 years and older (N=51) self-administered the MyCog Mobile measures remotely on their smartphones twice within a 2- to 3-week interval. The pilot version of MyCog Mobile includes 4 activities: MyFaces measures facial memory, MySorting measures executive functioning, MySequences measures working memory, and MyPictures measures episodic memory. After their first administration, participants also completed a modified version of the Simplified System Usability Scale (S-SUS) and 2 custom survey items.

RESULTS

All participants in the sample passed the practice items and completed each measure. Findings indicate that the Mobile Toolbox assessments measure the constructs well (internal consistency 0.73 to 0.91) and are stable over an approximately 2-week delay (test-retest reliability 0.61 to 0.71). Participants' rating of the user experience (mean S-SUS score 73.17, SD 19.27) indicated that older adults found the usability of MyCog Mobile to be above average. On free-response feedback items, most participants provided positive feedback or no feedback at all, but some indicated a need for clarity in certain task instructions, concerns about participants' abilities, desire to be able to contact a support person or use in-app technical support, and desire for additional practice items.

CONCLUSIONS

Pilot evidence suggests that the MyCog Mobile cognitive screener can be reliably self-administered by older adults on their smartphones. Participants in our study generally provided positive feedback about the MyCog Mobile experience and rated the usability of the app highly. Based on participant feedback, we will conduct further usability research to improve support functionality, optimize task instructions and practice opportunities, and ensure that patients feel comfortable using MyCog Mobile. The next steps include a clinical validation study that compares MyCog Mobile to gold-standard assessments and tests the sensitivity and specificity of the measures for identifying dementia.

Primary care detection of cognitive impairment leveraging health and consumer technologies in underserved US communities: protocol for a pragmatic randomised controlled trial of the MyCog paradigm

INTRODUCTION

Early identification of cognitive impairment (CI), including Alzheimer's disease and related dementias (ADRD), is a top public health priority. Yet, CI/ADRD is often undetected and underdiagnosed within primary care settings, and in health disparate populations. The MyCog paradigm is an iPad-based, self-administered, validated cognitive assessment based on the National Institutes of Health (NIH) Toolbox Cognition Battery and coupled with clinician decision-support tools that is specifically tailored for CI/ADRD detection within diverse, primary care settings.

METHODS AND ANALYSIS

We will conduct a two-arm, primary care practice-randomised (N=24 practices; 45 257 active patients at the proposed practices), pragmatic trial among geographically diverse Oak Street Health sites to test the effectiveness of the MyCog paradigm to improve early detection CI/ADRD among low socioeconomic, black and Hispanic older adults compared with usual care. Participating practices randomised to the intervention arm will impart the MyCog paradigm as a new standard of care over a 3-year implementation period; as the cognitive component for Annual Wellness Visits and for any patient/informant-reported or healthcare provider-suspected cognitive concern. Rates of detected (cognitive test suggesting impairment) and/or diagnosed (relevant International Classification of Diseases-9/10 [ICD-9/10] code) cognitive deficits, impairments or dementias including ADRD will be our primary outcome of study compared between arms. Secondary outcomes will include ADRD severity (ie, mild or later stage), rates of cognitive-related referrals and rates of family member or caregiver involvement in ADRD care planning. We will use generalised linear mixed models to account for clustered study design. Secondary models will adjust for subject, clinic or visit-specific characteristics. We will use mixed-methods approaches to examine fidelity and cost-effectiveness of the MyCog paradigm.

ETHICS AND DISSEMINATION

The Institutional Review Board at Advarra has approved the study protocol (Pro00064339). Results will be published in peer-reviewed journals and summaries will be provided to the funders of the study.

TRIAL REGISTRATION NUMBER

NCT05607732.

AO Patient Outcomes Center: Design, Implementation, and Evaluation of a Software Application for the Collection of Patient-Reported Outcome Measures in Orthopedic Outpatient Clinics

BACKGROUND

Patient-reported outcomes are increasingly utilized in routine orthopedic clinical care. Computer adaptive tests (CATs) from the Patient-Reported Outcomes Measurement Information System (PROMIS) offer a brief and precise assessment that is well suited for collection within busy clinical environments. However, software apps that support the administration and scoring of CATs, provide immediate access to patient-reported outcome (PRO) scores, and minimize clinician burden are not widely available.

OBJECTIVE

Our objective was to design, implement, and test the feasibility and usability of a Web-based system for collecting CATs in orthopedic clinics.

METHODS

AO Patient Outcomes Center (AOPOC) was subjected to 2 rounds of testing. Alpha testing was conducted in 3 orthopedic clinics to evaluate ease of use and feasibility of integration in clinics. Patients completed an assessment of PROMIS CATs and a usability survey. Clinicians participated in a brief semistructured interview. Beta-phase testing evaluated system performance through load testing and usability of the updated version of AOPOC. In both rounds of testing, user satisfaction, bugs, change requests, and performance of PROMIS CATs were captured.

RESULTS

Patient feedback supported the ease of use in completing an assessment in AOPOC. Across both phases of testing, clinicians rated AOPOC as easy to use but noted difficulties in integrating a Web-based software application within their clinics. PROMIS CATs performed well; the default assessment of 2 CATs was completed quickly (mean 9.5 items) with a satisfactory range of measurement.

CONCLUSION

AOPOC was demonstrated to be an easy-to-learn and easy-to-use software application for patients and clinicians that can be integrated into orthopedic clinical care. The workflow disruption in integrating any type of PRO collection must be addressed if patients' voices are to be better integrated in clinical care.

Bringing PROMIS to practice: brief and precise symptom screening in ambulatory cancer care

BACKGROUND

Supportive oncology practice can be enhanced by the integration of a brief and validated electronic patient-reported outcome assessment into the electronic health record (EHR) and clinical workflow.

METHODS

Six hundred thirty-six women receiving gynecologic oncology outpatient care received instructions to complete clinical assessments through Epic MyChart, an EHR patient communication portal. Patient Reported Outcomes Measurement Information System (PROMIS) computer adaptive tests (CATs) were administered to assess fatigue, pain interference, physical function, depression, and anxiety. Checklists identified psychosocial concerns, informational and nutritional needs, and risk factors for inadequate nutrition. Assessment results, including PROMIS T scores with documented severity thresholds, were immediately populated in the EHR. Clinicians were notified of clinically elevated symptoms through EHR messages. EHR integration was designed to provide automated triage to social work providers for psychosocial concerns, to health educators for information, and to dietitians for nutrition-related concerns.

RESULTS

Four thousand forty-two MyChart messages sent, and 3203 (79%) were reviewed by patients. The assessment was started by 1493 patients (37%), and once they started, 93% (1386 patients) completed the assessment. According to first assessments only, 49.8% of the patients who reviewed the MyChart message completed the assessment. Mean PROMIS CAT T scores indicated a lower level of physical function and elevated anxiety in comparison with the general population. Fatigue, pain, and depression scores were comparable to those of the general population. Impaired physical functioning was the most common basis for clinical alerts and occurred in 4% of the patients.

CONCLUSIONS

PROMIS CATs were used to measure common cancer symptoms in routine oncology outpatient care. Immediate EHR integration facilitated the use of symptom reporting as the basis for referral to psychosocial and supportive care.