Maximizing the Potential of Patient-Reported Assessments by Using the Open-Source Concerto Platform With Computerized Adaptive Testing and Machine Learning

Establishing language and ethnic equivalence for health-related quality of life item banks and testing their efficiency via computerised adaptive testing simulations

PURPOSE

We aimed to (1) establish linguistic and ethnic equivalence (i.e. lack of bias) for the items in the English and Chinese versions of the Singapore Health and Well Being (SHAWS) Physical Functioning (PF), Positive Mindset (PM) and Social Relationship (SR) item banks (IBs); and (2) evaluate the preliminary efficiency of these IBs using Computer Adaptive Testing (CAT) simulations.

METHODS

In this cross-sectional study, 671, 670, and 672 subjects answered 55, 48 and 30 items of the PF, PM, and SR IBs, respectively. Rasch analysis was conducted to assess each IB's psychometric properties, particularly the presence of differential item functioning (DIF) for language and ethnicity. A set of performance criteria related to removing items that displayed notable DIF were employed. CAT simulations determined the mean number of items for high, moderate, and moderate-low measurement precisions (stopping rule: SEM 0.300, 0.387. 0.521, respectively).

RESULTS

Half of subjects were >50 years old (40.9% PF, 42.1% PM, 41.4% SR), Chinese (50.7% PF, 51.0% PM, 50.6% SR) and female (50.0% PF. 49.4% PM, 52.8% SR) respectively. Rasch analysis revealed 4 items with DIF for the PF IB, 9 items with DIF for the PM IB and 2 items with DIF for the SR IB. In CAT simulations, the mean number of items administered was 8.5, 21.6 and 14.5 for the PF, PM and SR IBs, respectively (SEM 0.300), 5.1, 13.0, 8.0 for PF, PM and SR IBs, respectively (SEM 0.387) and 3.1, 5.3 and 4.1 for PF, PM and SR IBs, respectively (SEM 0.521).

CONCLUSION

The PF, PM and SR IBs to measure health-related quality of life revealed minimal DIF for language and ethnicity after remedial efforts. CAT simulations demonstrated that these IBs were efficient, especially when the stopping rule was set at moderate precision, and support the implementation of the SHAWS IBs into routine clinical care.

Efficiency, Precision, Validity, and Reliability of GlauCAT-Asian Computerized Adaptive Tests in Measuring Glaucoma-Related Quality of Life

Purpose

To determine the efficiency, precision, and agreement of GlauCAT-Asian and its corresponding validity and reliability.

Methods

In this cross-sectional study, 219 participants (mean ± standard deviation age, 66.59 ± 8.61 years; 34% female) across the spectrum of glaucoma severity and 50 glaucoma suspects were recruited from glaucoma clinics in Singapore. Participants answered seven computerized adaptive testing (CAT) evaluations (Ocular Comfort, Activity Limitation, Lighting, Mobility, Concerns, Psychosocial, Glaucoma Management) and underwent eye examinations. Efficiency (mean number of items required for each CAT and time taken for CAT versus full item banks [IBs]), agreement (concordance between CATs and full IB person measures, henceforth referred to as scores), and precision (standard error of measurement [SE]) were evaluated. Other validity and reliability metrics were also assessed.

Results

The mean number of items administered ranged from 9 (Mobility/Glaucoma Management) to 12 (Ocular Comfort). Compared to answering the full IBs, CATs provided an average time saving of 38.3% (range, 10% to 70.6% for Lighting and Activity Limitation, respectively). Agreement between scores obtained by CAT versus full IB was high (intracorrelation coefficient ≥0.75), as was precision of score estimates (mean SE range: 0.35 for Psychosocial to 0.29 for Mobility). Scores from Activity Limitation, Mobility, Lighting, and Concerns decreased significantly as glaucoma severity increased (criterion validity; P-trend <0.05). All tests displayed good convergent/divergent validity and test-retest reliability.

Conclusions

GlauCAT-Asian provides efficient, precise, accurate, valid, and reliable measurement of the patient-centered impact of glaucoma.

Translational Relevance

GlauCAT-Asian may provide a valuable clinical tool for ophthalmologists to monitor impact of disease progression and the effectiveness of therapies.

Vanderbilt Mini-PROM-Breast for Breast Reconstruction: A Short-Form, Patient-Reported Outcomes Measure

BACKGROUND

Breast reconstruction procedures are among the most commonly performed plastic surgery operations. Although there are well-validated patient-reported outcomes measures (PROMs) for breast reconstruction, there are several patient-, provider-, and service-level barriers to their implementation in routine clinical settings. Therefore, we developed a short-form PROM to evaluate breast reconstruction outcomes.

METHODS

Using a mixed-methods approach, the Vanderbilt Mini-PROM-Breast (VMP-B) was constructed and validated to assess patient-reported outcomes from breast reconstruction. Classic test theory methods were used to evaluate acceptability, reliability, and validity. External validation was subsequently performed using the BREAST-Q as a reference standard.

RESULTS

The VMP-B is a 16-item instrument composed of three domains: quality of life, body image, and breast satisfaction. Psychometric properties including acceptability, reliability, and validity exceeded reference criteria. When tested with 104 patients, the authors found significant benefits of breast reconstruction on quality of life, body image, and breast satisfaction. These results were associated with sizeable effect sizes (g) (g = 0.421, g = 0.520, and g = 1.25) demonstrative of clinically meaningful results. When tested concurrently in 70 patients, the VMP-B and the BREAST-Q showed similar results, exhibiting excellent convergent validity.

CONCLUSIONS

The VMP-B is a validated short-form PROM that reliably assesses breast reconstruction outcomes. As a short form, the VMP-B decreases both patient and provider burden, which allows for routine, point-of-care collection of breast reconstruction outcomes.

Computerized adaptive testing for the patient evaluation measure (PEM) in patients undergoing cubital tunnel syndrome surgery

In outcome measures, item response theory (IRT) validation can deliver interval-scaled high-quality measurement that can be harnessed using computerized adaptive tests (CATs) to pose fewer questions to patients. We aimed to develop a CAT by developing an IRT model for the Patient Evaluation Measure (PEM) for patients undergoing cubital tunnel syndrome (CuTS) surgery. Nine hundred and seventy-nine completed PEM responses of patients with CuTS in the United Kingdom Hand Registry were used to develop and calibrate the CAT. Its performance was then evaluated in a simulated cohort of 1000 patients. The CAT reduced the original PEM length from ten to a median of two questions (range two to four), while preserving a high level of precision (median standard error of measurement of 0.27). The mean error between the CAT score and full-length score was 0.08%. A Bland-Altman analysis showed good agreement with no signs of bias. The CAT version of the PEM can substantially reduce patient burden while enhancing construct validity by harnessing IRT for patients undergoing CuTS surgery.

Remote Symptom Monitoring With Ecological Momentary Computerized Adaptive Testing: Pilot Cohort Study of a Platform for Frequent, Low-Burden, and Personalized Patient-Reported Outcome Measures

BACKGROUND

Remote patient-reported outcome measure (PROM) data capture can provide useful insights into research and clinical practice and deeper insights can be gained by administering assessments more frequently, for example, in ecological momentary assessment. However, frequent data collection can be limited by the burden of multiple, lengthy questionnaires. This burden can be reduced with computerized adaptive testing (CAT) algorithms that select only the most relevant items from a PROM for an individual respondent. In this paper, we propose "ecological momentary computerized adaptive testing" (EMCAT): the use of CAT algorithms to reduce PROM response burden and facilitate high-frequency data capture via a smartphone app. We develop and pilot a smartphone app for performing EMCAT using a popular hand surgery PROM.

OBJECTIVE

The aim of this study is to determine the feasibility of EMCAT as a system for remote PROM administration.

METHODS

We built the EMCAT web app using Concerto, an open-source CAT platform maintained by the Psychometrics Centre, University of Cambridge, and hosted it on an Amazon Web Service cloud server. The platform is compatible with any questionnaire that has been parameterized with item response theory or Rasch measurement theory. For this study, the PROM we chose was the patient evaluation measure, which is commonly used in hand surgery. CAT algorithms were built using item response theory models derived from UK Hand Registry data. In the pilot study, we enrolled 40 patients with hand trauma or thumb-base arthritis, across 2 sites, between July 13, 2022, and September 14, 2022. We monitored their symptoms with the patient evaluation measure, via EMCAT, over a 12-week period. Patients were assessed thrice weekly, once daily, or thrice daily. We additionally administered full-length PROM assessments at 0, 6, and 12 weeks, and the User Engagement Scale at 12 weeks.

RESULTS

The use of EMCAT significantly reduced the length of the PROM (median 2 vs 11 items) and the time taken to complete it (median 8.8 seconds vs 1 minute 14 seconds). Very similar scores were obtained when EMCAT was administered concurrently with the full-length PROM, with a mean error of <0.01 on a logit (z score) scale. The median response rate in the daily assessment group was 93%. The median perceived usability score of the User Engagement Scale was 4.0 (maximum possible score 5.0).

CONCLUSIONS

EMCAT reduces the burden of PROM assessments, enabling acceptable high-frequency, remote PROM data capture. This has potential applications in both research and clinical practice. In research, EMCAT could be used to study temporal variations in symptom severity, for example, recovery trajectories after surgery. In clinical practice, EMCAT could be used to monitor patients remotely, prompting early intervention if a patient's symptom trajectory causes clinical concern.

TRIAL REGISTRATION

ISRCTN 19841416; https://www.isrctn.com/ISRCTN19841416.

Measuring glaucoma quality of life in an Asian population using item banking: psychometric evaluation and computerized adaptive testing simulations

PURPOSE

To assess the psychometric properties of glaucoma-specific health-related quality of life (HRQoL) item banks (IBs), and explore their efficiency using computerized adaptive testing (CAT) simulations.

METHODS

In this cross-sectional, clinical study, 300 Asian glaucoma patients answered 221 items within seven IBs: Ocular Comfort Symptoms (OS); Activity Limitation (AL); Lighting (LT); Mobility (MB); Glaucoma Management (GM); Psychosocial (PSY); and Work (WK). Rasch analysis was conducted to assess each IB's psychometric properties (e.g., item "fit" to the construct; unidimensionality) and a set of analytic performance criteria guiding decision making relating to retaining or dropping domains and items was employed. CAT simulations determined the mean number of items for 'high' and 'moderate' measurement precision (stopping rule: SEM 0.3 and 0.387, respectively).

RESULTS

Participants' mean age was 67.2 ± 9.2 years (62% male; 87% Chinese). LT, MB, and GM displayed good psychometric properties overall. To optimize AL's psychometric properties, 16 items were deleted due to poor "fit", high missing data, item bias, low discrimination and/or a low clinical/patient importance rating. To resolve multidimensionality in PSY, we rehomed 16 items into a "Concern (CN)" domain. PSY and CN required further amendment, including collapsing of response categories, and removal of poorly functioning items (N = 7). Due to poor measurement precision, low applicability and high ceiling effect, low test information indices, and low item separation index the WK IB was not considered further. In CAT simulations on the final seven IBs (n = 182 items total), an average of 12.1 and 15.7 items per IB were required for moderate and high precision measurement, respectively.

CONCLUSIONS

After reengineering our seven IBs, they displayed robust psychometric properties and good efficiency in CAT simulations. Once finalized, GlauCAT™-Asian may enable comprehensive assessment of the HRQoL impact of glaucoma and associated treatments.

Item response theory assumptions were adequately met by the Oxford hip and knee scores

OBJECTIVES

To develop item response theory (IRT) models for the Oxford hip and knee scores which convert patient responses into continuous scores with quantifiable precision and provide these as web applications for efficient score conversion.

STUDY DESIGN AND SETTING

Data from the National Health Service patient-reported outcome measures program were used to test the assumptions of IRT (unidimensionality, monotonicity, local independence, and measurement invariance) before fitting models to preoperative response patterns obtained from patients undergoing primary elective hip or knee arthroplasty. The hip and knee datasets contained 321,147 and 355,249 patients, respectively.

RESULTS

Scree plots, Kaiser criterion analyses, and confirmatory factor analyses confirmed unidimensionality and Mokken analysis confirmed monotonicity of both scales. In each scale, all item pairs shared a residual correlation of ≤ 0.20. At the test level, both scales showed measurement invariance by age and gender. Both scales provide precise measurement in preoperative settings but demonstrate poorer precision and ceiling effects in postoperative settings.

CONCLUSION

We provide IRT parameters and web applications that can convert Oxford Hip Score or Oxford Knee Score response sets into continuous measurements and quantify individual measurement error. These can be used in sensitivity analyses or to administer truncated and individualized computerized adaptive tests.

Italian cross-cultural adaptation of the patient-reported outcome measure of ataxia

BACKGROUND

The patient-reported outcome measure of ataxia (PROM-Ataxia) is the first patient-reported questionnaire specifically developed for use in patients with cerebellar ataxia. The scale was recently designed and validated in English language, and it consists of 70 items encompassing all aspects associated with the patient experience, including physical and mental health and their consequences on activities of daily living. The aim of the study was to translate and culturally adapt into Italian the PROM-Ataxia questionnaire, before assessing its psychometric properties.

METHODS

We translated and culturally adapted into Italian the PROM-Ataxia following the ISPOR TCA Task Force guidelines. The questionnaire was field tested via cognitive interviews with users.

RESULTS

The Italian patients found that the questionnaire was complete, and no significant contents related to the physical, mental, and functional dimensions were missing. Some items were found redundant or ambiguous. Most of the identified issues pertained to semantic equivalence, and a few to conceptual and normative equivalence, while the questionnaire did not contain any idiomatic expression.

CONCLUSIONS

The translation and cultural adaptation of the PROM-Ataxia questionnaire in the Italian patient population represent the pre-requisite for the subsequent psychometric validation of the scale. This instrument may be valuable for cross-country comparability that would allow the merging of the data in collaborative multinational research studies.

Intelligent multi-modal shear wave elastography to reduce unnecessary biopsies in breast cancer diagnosis (INSPiRED 002): a retrospective, international, multicentre analysis

BACKGROUND

Breast ultrasound identifies additional carcinomas not detected in mammography but has a higher rate of false-positive findings. We evaluated whether use of intelligent multi-modal shear wave elastography (SWE) can reduce the number of unnecessary biopsies without impairing the breast cancer detection rate.

METHODS

We trained, tested, and validated machine learning algorithms using SWE, clinical, and patient information to classify breast masses. We used data from 857 women who underwent B-mode breast ultrasound, SWE, and subsequent histopathologic evaluation at 12 study sites in seven countries from 2016 to 2019. Algorithms were trained and tested on data from 11 of the 12 sites and externally validated using the additional site's data. We compared findings to the histopathologic evaluation and compared the diagnostic performance between B-mode breast ultrasound, traditional SWE, and intelligent multi-modal SWE.

RESULTS

In the external validation set (n = 285), intelligent multi-modal SWE showed a sensitivity of 100% (95% CI, 97.1-100%, 126 of 126), a specificity of 50.3% (95% CI, 42.3-58.3%, 80 of 159), and an area under the curve of 0.93 (95% CI, 0.90-0.96). Diagnostic performance was significantly higher compared to traditional SWE and B-mode breast ultrasound (P < 0.001). Unlike traditional SWE, positive-predictive values of intelligent multi-modal SWE were significantly higher compared to B-mode breast ultrasound. Unnecessary biopsies were reduced by 50.3% (79 versus 159, P < 0.001) without missing cancer compared to B-mode ultrasound.

CONCLUSION

The majority of unnecessary breast biopsies might be safely avoided by using intelligent multi-modal SWE. These results may be helpful to reduce diagnostic burden for patients, providers, and healthcare systems.

Short-form adaptive measure of financial toxicity from the Economic Strain and Resilience in Cancer (ENRICh) study: Derivation using modern psychometric techniques

OBJECTIVES

This study sought to evaluate advanced psychometric properties of the 15-item Economic Strain and Resilience in Cancer (ENRICh) measure of financial toxicity for cancer patients.

METHODS

We surveyed 515 cancer patients in the greater Houston metropolitan area using ENRICh from March 2019 to March 2020. We conducted a series of factor analyses alongside parametric and non-parametric item response theory (IRT) assessments using Mokken analysis and the graded response model (GRM). We utilized parameters derived from the GRM to run a simulated computerized adaptive test (CAT) assessment.

RESULTS

Among participants, mean age was 58.49 years and 278 (54%) were female. The initial round factor analysis results suggested a one-factor scale structure. Negligible levels of differential item functioning (DIF) were evident between eight items. Three items were removed due to local interdependence (Q3>+0.4). The original 11-point numerical rating scale did not function well, and a new 3-point scoring system was implemented. The final 12-item ENRICh had acceptable fit to the GRM (p<0.001; TLI = 0.94; CFI = 0.95; RMSEA = 0.09; RMSR = 0.06) as well as good scalability and dimensionality. We observed high correlation between CAT version scores and the 12-item measure (r = 0.98). During CAT, items 2 (money you owe) and 4 (stress level about finances) were most frequently administered, followed by items 1 (money in savings) and 5 (ability to pay bills). Scores from these four items alone were strongly correlated with that of the 12-item ENRICh (r = 0.96).

CONCLUSION

These CAT and 4-item versions provide options for quick screening in clinical practice and low-burden assessment in research.

Psychometric Evaluation of Glaucoma Quality of Life Item Banks (GlauCAT) and Initial Assessment Using Computerized Adaptive Testing

Purpose

To evaluate the psychometric properties of glaucoma-specific quality of life (QoL) item banks (GlauCAT) and assess their performance using computerized adaptive testing (CAT) simulations.

Methods

In this cross-sectional study, 293 participants with glaucoma (mean age ± SD, 70.7 ± 13.2 years; 45% female) answered 342 items in 12 QoL item banks (IBs): Activity Limitation (AL); Driving (DV); Convenience (CV); Economic (EC); Emotional (EM); General Symptoms (GS); Health Concerns (HC); Lighting (LT); Mobility (MB); Ocular Surface Symptoms (OS); Social (SC); and Visual Symptoms (VS). These IBs were assessed using Rasch analysis, and CAT simulations with 1000 simulated respondents were utilized to determine the average number of items to be administered to achieve moderate and high precision levels.

Results

The AL, DV, EM, HC, LT, MB, EC, OS, SC, and VS IBs required relatively minor amendments to achieve satisfactory psychometric fit. To resolve multidimensionality, we split CV into Treatment Convenience (TCV) and General Convenience (GCV). Due to poor measurement precision, the GS IB was not pursued further. This resulted in 12 total IBs. In CAT simulations, an average of 3.7 and 7.3 items per IB were required to attain measurement at moderate and high precision, respectively.

Conclusions

Following rigorous psychometric assessment, we developed 12 valid glaucoma-specific QoL domains that can obtain highly precise person measure estimates using a small number of items.

Translational Relevance

GlauCAT will enable researchers and clinicians to quickly and comprehensively assess the impact of glaucoma and its associated interventions across a range of QoL domains.

Implementation of an Online Glaucoma-Specific Quality of Life Computerized Adaptive Test System in a US Glaucoma Hospital

Purpose

The feasibility of implementing a computerized adaptive test (CAT) system in routine clinical care in ophthalmology has not been assessed. We evaluated the implementation of a glaucoma-specific CAT (GlauCAT) in outpatients at Massachusetts Eye and Ear Institute.

Methods

In this implementation study (July 2020–April 2021), 216 adults (mean ± SD age 64.8 ± 15.3 years; 56.0% women) completed six adaptive GlauCAT quality of life (QOL) tests on an internet-enabled tablet at the clinic. A real-time printable report summarizing domain scores was shared with physicians prior to consultation. The implementation was evaluated using Proctor's outcomes: acceptability (patient satisfaction); appropriateness (independent complete rate [%]); feasibility (acceptance rate [%]; completion time); and fidelity (percentage of patients discussing GlauCAT results with their physician). Physician barriers/facilitators were explored using open-ended questions.

Results

Patients’ mean ± SD satisfaction score was 3.5 ± 0.5 of 4, with >95% of patients willing to recommend it to others. Of the 216 (89.2%) patients accepting to participate, 173 (80%) completed GlauCAT independently. Patients took 8 minutes and 5 seconds (median) to complete all 6 GlauCAT tests. Almost two-thirds (n = 136/216) of the patients reported discussing their GlauCAT results with their doctor. Physicians described the GlauCAT summary report as helpful and user-friendly, although lack of time and uncertainty about how to action information were reported.

Conclusions

Pilot implementation of six GlauCAT QOL tests in glaucoma outpatient clinics was feasible and acceptable. Integration of GlauCAT with electronic medical records (EMRs) and evaluation of long-term implementation outcomes are needed.

Translational Relevance

GlauCAT's multiple outcomes and low test-taking burden makes it attractive for measuring glaucoma-specific QOL in routine clinical care.

Feasibility and performance of smartphone-based daily micro-surveys among patients recovering from cancer surgery

AIMS

Daily micro-surveys, or the high-frequency administration of patient-reported outcome measures (PROMs), may provide real-time, unbiased assessments of health-related quality of life (HRQoL). We evaluated the feasibility and accuracy of daily micro-surveys using a smartphone platform among patients recovering from cancer surgery.

METHODS

In a prospective study (2017-2019), patients undergoing cancer surgery downloaded a smartphone application that administered daily micro-surveys comprising five randomly selected items from the Short Form-36 (SF-36). Micro-surveys were administered without replacement until the entire SF-36 was administered weekly. The full-length SF-36 was also administered preoperatively and 4, 12, and 24 weeks postoperatively. We assessed response and completion rates between the micro-surveys and full-length SF-36, as well as agreement of responses using Bland-Altman (B&A) analyses.

RESULTS

Ninety-five patients downloaded the application and were followed for a mean of 131 days [SD ± 85]. Response rates for the full-length SF-36 and micro-surveys was 76% [95%CI 69, 83], and 34% [95%CI 26, 39]. Despite lower response rates, more SF-36 surveys were collected using the daily micro-surveys compared to the intermittent full-length SF-36 (9.9 [95%CI 8.4, 12.6] vs. 3.0 [95%CI 2.8, 3.3], respectively). B&A analyses demonstrated lack of agreement between micro-surveys and SF-36. However, agreement improved with higher micro-survey completion rate. Eighty-five percent of participants reported that daily micro-surveys were not burdensome.

CONCLUSION

This study suggests that collection of daily micro-surveys among patients recovering from cancer surgery is feasible using smartphones in the early postoperative period. Future implementation of daily micro-surveys may more granularly describe momentary HRQoL changes through a greater volume of self-reported survey data.

Maximizing the Potential of Patient-Reported Assessments by Using the Open-Source Concerto Platform With Computerized Adaptive Testing and Machine Learning

Patient-reported assessments are transforming many facets of health care, but there is scope to modernize their delivery. Contemporary assessment techniques like computerized adaptive testing (CAT) and machine learning can be applied to patient-reported assessments to reduce burden on both patients and health care professionals; improve test accuracy; and provide individualized, actionable feedback. The Concerto platform is a highly adaptable, secure, and easy-to-use console that can harness the power of CAT and machine learning for developing and administering advanced patient-reported assessments. This paper introduces readers to contemporary assessment techniques and the Concerto platform. It reviews advances in the field of patient-reported assessment that have been driven by the Concerto platform and explains how to create an advanced, adaptive assessment, for free, with minimal prior experience with CAT or programming.