Item Response Theory Validation of the Forgotten Joint Score for Persons Undergoing Total Knee Replacement

BACKGROUND

The Forgotten Joint Score (FJS), a commonly used patient-reported outcome measure, was developed without fully confirming assumptions such as unidimensionality (all items reflect 1 underlying factor), appropriate weighting of each item in scoring, absence of differential item functioning (in which different groups, e.g., men and women, respond differently), local dependence (pairs of items are measuring only 1 underlying factor), and monotonicity (persons with higher function have a higher score). We applied item response theory (IRT) to perform validation of the FJS according to contemporary standards, and thus support its ongoing use. We aimed to confirm that the FJS reflects a single latent trait. In addition, we aimed to determine whether an IRT model could be fitted to the FJS.

METHODS

Participants undergoing primary total knee replacement provided responses to the FJS items preoperatively and at 6 and 12 months postoperatively. An exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and Mokken analysis were conducted. A graded response model (GRM) was fitted to the data.

RESULTS

A total of 1,774 patient responses were analyzed. EFA indicated a 1-factor model (all 12 items reflecting 1 underlying trait). CFA demonstrated an excellent model fit. Items did not have equal weighting. The FJS demonstrated good monotonicity and no differential item functioning by sex, age, or body mass index. GRM parameters are reported in this paper.

CONCLUSIONS

The FJS meets key validity assumptions, supporting its use in clinical practice and research. The IRT-adapted FJS has potential advantages over the traditional FJS: it provides continuous measurements with finer granularity between health states, includes individual measurement error, and can compute scores despite more missing data (with only 1 response required to estimate a score). It can be applied retrospectively to existing data sets or used to deliver individualized computerized adaptive tests.

LEVEL OF EVIDENCE

Prognostic Level II . See Instructions for Authors for a complete description of levels of evidence.

A Qualitative Study of Patients' Lived Experiences of Free Tissue Transfer for Diabetic Foot Disease

Background

Free tissue transfer (FTT) for reconstruction of diabetic foot disease (DFD) is an emerging field to preserve the lower limb within this patient group. The design of future quantitative research and clinical services in this area must consider the needs, expectations and concerns of patients. This qualitative study explores patient experiences of FTT for reconstruction of DFD.

Methods

Semistructured interviews were conducted to explore patients' lived experiences of FTT for DFD. A purposive sampling strategy identified six patients who underwent FTT for recalcitrant DFD between September 2019 and December 2021 in a single center in the United Kingdom.

Results

Three experiential themes emerged. Theme 1: "negative lived experiences of living with DFD" included frustration with the chronic management of nonhealing ulcers and fear regarding limb amputation. Theme 2: "surgery related concerns" included fears of reconstructive failure and subsequent amputation, as well as foot cosmesis and donor-site morbidity. Theme 3: "positive lived experiences following reconstruction" included the positive impact the reconstruction had on their overall life and diabetic control. All patients would repeat the process to obtain their current results.

Conclusions

This qualitative study provides first-hand insight into the lived experience of FTT for DFD, exploring both the negative and positive experiences and reasons for these. We found that FTT for DFD can be positively life-changing for affected individuals.

Hand and Wrist trauma: Antimicrobials and Infection Audit of Clinical Practice (HAWAII ACP) protocol

Aims

Hand trauma, consisting of injuries to both the hand and the wrist, are a common injury seen worldwide. The global age-standardized incidence of hand trauma exceeds 179 per 100,000. Hand trauma may require surgical management and therefore result in significant costs to both healthcare systems and society. Surgical site infections (SSIs) are common following all surgical interventions, and within hand surgery the risk of SSI is at least 5%. SSI following hand trauma surgery results in significant costs to healthcare systems with estimations of over £450 per patient. The World Health Organization (WHO) have produced international guidelines to help prevent SSIs. However, it is unclear what variability exists in the adherence to these guidelines within hand trauma. The aim is to assess compliance to the WHO global guidelines in prevention of SSI in hand trauma.

Methods

This will be an international, multicentre audit comparing antimicrobial practices in hand trauma to the standards outlined by WHO. Through the Reconstructive Surgery Trials Network (RSTN), hand surgeons across the globe will be invited to participate in the study. Consultant surgeons/associate specialists managing hand trauma and members of the multidisciplinary team will be identified at participating sites. Teams will be asked to collect data prospectively on a minimum of 20 consecutive patients. The audit will run for eight months. Data collected will include injury details, initial management, hand trauma team management, operation details, postoperative care, and antimicrobial techniques used throughout. Adherence to WHO global guidelines for SSI will be summarized using descriptive statistics across each criteria.

Discussion

The Hand and Wrist trauma: Antimicrobials and Infection Audit of Clinical Practice (HAWAII ACP) will provide an understanding of the current antimicrobial practice in hand trauma surgery. This will then provide a basis to guide further research in the field. The findings of this study will be disseminated via conference presentations and a peer-reviewed publication.

Health-related quality of life in patients with conditions affecting the hand: meta-analysis

BACKGROUND

Health state utility values provide the quality component of quality-adjusted life years and are essential for health economic analyses, such as the National Institute for Health and Care Excellence Technology Appraisal. The aims of this systematic review were to: catalogue utility values for health states experienced by patients with hand conditions; provide pooled utility estimates for common hand conditions; and determine how utilities have been estimated.

METHODS

A PRISMA-compliant systematic review and meta-analysis was conducted (registered in PROSPERO, the international prospective register of systematic reviews (CRD42021226098)). Five databases were searched from inception until April 2023 (Embase, MEDLINE, PsycINFO, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane Central Register of Controlled Trials (CENTRAL)). All studies that reported primary utility values for hand health states in adult patients were eligible for inclusion. Pooled utility estimates were determined across conditions and intervention status using random-effects meta-analysis.

RESULTS

A total of 10 254 articles were identified; 57 studies met the full inclusion criteria and reported 363 distinct health state utility values. Health state utility values were estimated using a range of methods; the most common measure was the EQ-5D. Pooled utility estimates for carpal tunnel syndrome and hand osteoarthritis before surgical intervention were 0.69 (95% c.i. 0.66 to 0.73) and 0.63 (95% c.i. 0.60 to 0.67) respectively.

CONCLUSION

Pooled utility estimates for patients with untreated carpal tunnel syndrome and hand osteoarthritis are 11% and 18% lower than age-matched population norms respectively. Hand conditions have a significant detrimental impact on health-related quality of life and this study provides catalogued utility values for use in future economic analyses to support the delivery of value-based hand surgery.

FACE-Q for Measuring Patient-reported Outcomes after Facial Skin Cancer Surgery: Cross-cultural Validation

Background

Facial skin cancer and its surgical treatment can affect health-related quality of life. The FACE-Q Skin Cancer Module is a patient-reported outcome measure that measures different aspects of health-related quality of life and has recently been translated into Dutch. This study aimed to evaluate the performance of the translated version in a Dutch cohort using modern psychometric measurement theory (Rasch).

Methods

Dutch participants with facial skin cancer were prospectively recruited and asked to complete the translated FACE-Q Skin Cancer Module. The following assumptions of the Rasch model were tested: unidimensionality, local independence, and monotonicity. Response thresholds, fit statistics, internal consistency, floor and ceiling effects, and targeting were assessed for all scales and items within the scales. Responsiveness was tested for the "cancer worry" scale.

Results

In total, 259 patients completed the preoperative questionnaire and were included in the analysis. All five scales assessed showed a good or sufficient fit to the Rasch model. Unidimensionality and monotonicity were present for all scales. Some items showed a local dependency. Most of the scales demonstrate ordered item thresholds and appropriate fit statistics.

Conclusions

The FACE-Q Skin Cancer Module is a well-designed patient-reported outcome measure that shows psychometric validity for the translated version in a Dutch cohort, using classical and modern test theory.

Structural validation of the Manchester-Oxford Foot Questionnaire for use in foot and ankle surgery

Aims

The Manchester-Oxford Foot Questionnaire (MOxFQ) is an anatomically specific patient-reported outcome measure (PROM) currently used to assess a wide variety of foot and ankle pathology. It consists of 16 items across three subscales measuring distinct but related traits: walking/standing ability, pain, and social interaction. It is the most used foot and ankle PROM in the UK. Initial MOxFQ validation involved analysis of 100 individuals undergoing hallux valgus surgery. This project aimed to establish whether an individual's response to the MOxFQ varies with anatomical region of disease (measurement invariance), and to explore structural validity of the factor structure (subscale items) of the MOxFQ.

Methods

This was a single-centre, prospective cohort study involving 6,637 patients (mean age 52 years (SD 17.79)) presenting with a wide range of foot and ankle pathologies between January 2013 and December 2021. To assess whether the MOxFQ responses vary by anatomical region of foot and ankle disease, we performed multigroup confirmatory factor analysis. To assess the structural validity of the subscale items, exploratory and confirmatory factor analyses were performed.

Results

Measurement invariance by pathology was confirmed, suggesting the same model can be used across all foot and ankle anatomical regions. Exploratory factor analysis demonstrated a two- to three-factor model, and suggested that item 13 (inability to carry out work/everyday activities) and item 14 (inability to undertake social/recreational activities) loaded more positively onto the "walking/standing" subscale than their original "social interaction" subscale.

Conclusion

This large cohort study supports the current widespread use of the MOxFQ across a broad range of foot and ankle pathologies. Our analyses found indications that could support alterations to the original factor structure (items 13 and 14 might be moved from the "social interaction" to the "walking/standing" subscale). However, this requires further work to confirm.

Modified Scoring of the QuickDASH Can Achieve Previously-unattained Interval-level Measurement in Dupuytren Disease and Carpal Tunnel Syndrome

Background

Rasch measurement theory can be used to identify scales within questionnaires and to map responses to more precise continuous scales. The aim of this article was to use RMT to refine the scoring of the QuickDASH in patients with Dupuytren disease and carpal tunnel syndrome (CTS).

Methods

Data were collected between 2013 and 2019 from a single center in the UK. Preoperative QuickDASH responses from patients diagnosed with Dupuytren disease and CTS were used. RMT was used to reduce the number of items in the QuickDASH and examine the reliability and validity of each subscale.

Results

The preoperative QuickDASH responses of 750 patients with Dupuytren disease and 1916 patients with CTS were used. The median age of participants was 61 years, and 46% were men. Exploratory factor analysis suggested two distinct subscales within the QuickDASH: task items 1-6 and symptom items 9-11. These items were fitted to the Rasch model, and disordered response thresholds were collapsed. In Dupuytren disease, the two worst responses or each item were disordered. After collapsing these options, good Rasch model fit was demonstrated. CTS responses fitted without modification. Item targeting was more appropriate for CTS than Dupuytren disease.

Conclusions

This study proposes a modification to the scoring system for the QuickDASH that provides high-quality, continuous, and condition-specific scales for the QuickDASH. The identification of distinct subscales within the QuickDASH can be used to identify distinct improvements in hand function and/or symptoms in previous, current, and future work.

Psychometric sensitivity analyses can identify bias related to measurement properties in trials that use patient-reported outcome measures: a secondary analysis of a clinical trial using the disabilities of the arm, shoulder, and hand questionnaire

OBJECTIVES

Demonstrate psychometric sensitivity analyses for testing the stability of study findings to assumptions made about patient-reported outcome measures.

STUDY DESIGN AND SETTING

We performed secondary analyses of Disability of Arm, Shoulder, and Hand (DASH) data collected within the Prevention of Shoulder Problems clinical trial, which compared upper limb function scores in women who had undergone breast cancer surgery, randomized to either an exercise program or usual care. We repeated the principal trial analyses after grouping DASH items into subscales suggested by factorial analyses in this dataset and applied item response theory to account for unequal item weighting. We checked for measurement invariance by participant age and response shift bias using established techniques.

RESULTS

Our analyses suggested that the DASH measured two constructs: motor function and sensory symptoms. The majority of the six-month difference in DASH score was driven by motor function. With item response theory scoring, we found differences in both constructs at 12 months (P = 0.019 and P = 0.007), but in neither construct at 6 months, contrary to the original trial results. We found no differential item function by age or between baseline and 12-month measurements.

CONCLUSIONS

Psychometric sensitivity analyses aid in the interpretation of the Prevention of Shoulder Problems trial's results.

Computerized adaptive testing for the patient evaluation measure (PEM) in patients undergoing cubital tunnel syndrome surgery

In outcome measures, item response theory (IRT) validation can deliver interval-scaled high-quality measurement that can be harnessed using computerized adaptive tests (CATs) to pose fewer questions to patients. We aimed to develop a CAT by developing an IRT model for the Patient Evaluation Measure (PEM) for patients undergoing cubital tunnel syndrome (CuTS) surgery. Nine hundred and seventy-nine completed PEM responses of patients with CuTS in the United Kingdom Hand Registry were used to develop and calibrate the CAT. Its performance was then evaluated in a simulated cohort of 1000 patients. The CAT reduced the original PEM length from ten to a median of two questions (range two to four), while preserving a high level of precision (median standard error of measurement of 0.27). The mean error between the CAT score and full-length score was 0.08%. A Bland-Altman analysis showed good agreement with no signs of bias. The CAT version of the PEM can substantially reduce patient burden while enhancing construct validity by harnessing IRT for patients undergoing CuTS surgery.

Overcoming floor and ceiling effects in knee arthroplasty outcome measurement

Aims

To map the Oxford Knee Score (OKS) and High Activity Arthroplasty Score (HAAS) items to a common scale, and to investigate the psychometric properties of this new scale for the measurement of knee health.

Methods

Patient-reported outcome measure (PROM) data measuring knee health were obtained from the NHS PROMs dataset and Total or Partial Knee Arthroplasty Trial (TOPKAT). Assumptions for common scale modelling were tested. A graded response model (fitted to OKS item responses in the NHS PROMs dataset) was used as an anchor to calibrate paired HAAS items from the TOPKAT dataset. Information curves for the combined OKS-HAAS model were plotted. Bland-Altman analysis was used to compare common scale scores derived from OKS and HAAS items. A conversion table was developed to map between HAAS, OKS, and the common scale.

Results

We included 3,329 response sets from 528 patients undergoing knee arthroplasty. These generally met the assumptions of unidimensionality, monotonicity, local independence, and measurement invariance. The HAAS items provided more information than OKS items at high levels of knee health. Combining both instruments resulted in higher test-level information than either instrument alone. The mean error between common scale scores derived from the OKS and HAAS was 0.29 logits.

Conclusion

The common scale allowed more precise measurement of knee health than use of either the OKS or HAAS individually. These techniques for mapping PROM instruments may be useful for the standardization of outcome reporting, and pooling results across studies that use either PROM in individual-patient meta-analysis.

Improving the structural validity of the QuickDASH questionnaire: Exploratory factor analysis and structural equation modelling in 1798 patients with carpal tunnel syndrome

STUDY DESIGN

Retrospective cohort.

BACKGROUND

The QuickDASH is a commonly used questionnaire for the assessment of carpal tunnel patients, although it is unclear whether the questionnaire has suitable structural validity PURPOSE: This study aimed to evaluate the structural validity of the QuickDASH patient-reported outcome measure (PROM), when used in CTS, through exploratory factor analysis (EFA) and structural equation modelling (SEM).

METHODS

Between 2013 and 2019, we recorded preoperative QuickDASH scores of 1916 patients undergoing carpal tunnel decompressions at a single unit. One hundred and eighteen patients with incomplete datasets were excluded leaving a final study group of 1798 patients with complete data. EFA was undertaken using the R statistical computing environment. We then conducted SEM in a random sample of 200 patients. Model fit was assessed using the chi-square (χ2) test, comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA) and standardized root mean square residuals (SRMR). A second "validation" SEM analysis was undertaken by repeating the analysis with a separate sample of 200 randomly-selected patients.

RESULTS

EFA revealed a 2-factor model: items 1-6 represented the first factor ("function") and items 9-11 measured a different factor ("symptoms"). SEM demonstrated excellent fit (χ2 p value 0.167, CFI 0.999, TLI 0.999, RMSEA 0.032, SRMR 0.046) and this was supported in our "validation" sample.

CONCLUSIONS

This study demonstrates that the QuickDASH PROM measures 2 distinct factors in CTS. This is comparable with the findings of a previous EFA that assessed the full-length Disabilities of the Arm, Shoulder and Hand PROM in patients with Dupuytren's disease.

Effectiveness of routine provision of feedback from patient-reported outcome measurements for cancer care improvement: a systematic review and meta-analysis

BACKGROUND

Research shows that feeding back patient-reported outcome information to clinicians and/or patients could be associated with improved care processes and patient outcomes. Quantitative syntheses of intervention effects on oncology patient outcomes are lacking.

OBJECTIVE

To determine the effects of patient-reported outcome measure (PROM) feedback intervention on oncology patient outcomes.

DATA SOURCES

We identified relevant studies from 116 references included in our previous Cochrane review assessing the intervention for the general population. In May 2022, we conducted a systematic search in five bibliography databases using predefined keywords for additional studies published after the Cochrane review.

STUDY SELECTION

We included randomized controlled trials evaluating the effects of PROM feedback intervention on processes and outcomes of care for oncology patients.

DATA EXTRACTION AND SYNTHESIS

We used the meta-analytic approach to synthesize across studies measuring the same outcomes. We estimated pooled effects of the intervention on outcomes using Cohen's d for continuous data and risk ratio (RR) with a 95% confidence interval for dichotomous data. We used a descriptive approach to summarize studies which reported insufficient data for a meta-analysis.

MAIN OUTCOME(S) AND MEASURES(S)

Health-related quality of life (HRQL), symptoms, patient-healthcare provider communication, number of visits and hospitalizations, number of adverse events, and overall survival.

RESULTS

We included 29 studies involving 7071 cancer participants. A small number of studies was available for each metanalysis (median = 3 studies, ranging from 2 to 9 studies) due to heterogeneity in the evaluation of the trials. We found that the intervention improved HRQL (Cohen's d = 0.23, 95% CI 0.11-0.34), mental functioning (Cohen's d = 0.14, 95% CI 0.02-0.26), patient-healthcare provider communication (Cohen's d = 0.41, 95% CI 0.20-0.62), and 1-year overall survival (OR = 0.64, 95% CI 0.48-0.86). The risk of bias across studies was considerable in the domains of allocation concealment, blinding, and intervention contamination.

CONCLUSIONS AND RELEVANCE

Although we found evidence to support the intervention for highly relevant outcomes, our conclusions are tempered by the high risk of bias relating mainly to intervention design. PROM feedback for oncology patients may improve processes and outcomes for cancer patients but more high-quality evidence is required.

Item response theory assumptions were adequately met by the Oxford hip and knee scores

OBJECTIVES

To develop item response theory (IRT) models for the Oxford hip and knee scores which convert patient responses into continuous scores with quantifiable precision and provide these as web applications for efficient score conversion.

STUDY DESIGN AND SETTING

Data from the National Health Service patient-reported outcome measures program were used to test the assumptions of IRT (unidimensionality, monotonicity, local independence, and measurement invariance) before fitting models to preoperative response patterns obtained from patients undergoing primary elective hip or knee arthroplasty. The hip and knee datasets contained 321,147 and 355,249 patients, respectively.

RESULTS

Scree plots, Kaiser criterion analyses, and confirmatory factor analyses confirmed unidimensionality and Mokken analysis confirmed monotonicity of both scales. In each scale, all item pairs shared a residual correlation of ≤ 0.20. At the test level, both scales showed measurement invariance by age and gender. Both scales provide precise measurement in preoperative settings but demonstrate poorer precision and ceiling effects in postoperative settings.

CONCLUSION

We provide IRT parameters and web applications that can convert Oxford Hip Score or Oxford Knee Score response sets into continuous measurements and quantify individual measurement error. These can be used in sensitivity analyses or to administer truncated and individualized computerized adaptive tests.

Item response theory may account for unequal item weighting and individual-level measurement error in trials that use PROMs: a psychometric sensitivity analysis of the TOPKAT trial

OBJECTIVES

To apply item response theory as a framework for studying measurement error in superiority trials which use patient-reported outcome measures (PROMs).

METHODS

We reanalyzed data from the The Total or Partial Knee Arthroplasty Trial, which compared the Oxford Knee Score (OKS) responses of patients undergoing partial or total knee replacement, using traditional sum-scoring, after accounting for OKS item characteristics with expected a posteriori (EAP) scoring, and after accounting for individual-level measurement error with plausible value imputation (PVI). We compared the marginalized mean scores of each group at baseline, 2 months, and yearly for 5 years. We used registry data to estimate the minimal important difference (MID) of OKS scores with sum-scoring and EAP scoring.

RESULTS

With sum-scoring, we found statistically significant differences in mean OKS score at 2 months (P = 0.030) and 1 year (P = 0.030). EAP scores produced slightly different results, with statistically significant differences at 1 year (P = 0.041) and 3 years (P = 0.043). With PVI, there were no statistically significant differences.

CONCLUSION

Psychometric sensitivity analyses can be readily performed for superiority trials using PROMs and may aid the interpretation of results.

The Use of Botulinum Toxin in the Management of Hidradenitis Suppurativa: A Systematic Review

Hidradenitis suppurativa (HS) is a chronic inflammatory skin condition characterized by suppurative infection, sinus tract, and abscess formation. International management guidelines are largely consensus-based. Botulinum toxin (BTX) has been widely used in the treatment of apocrine and eccrine gland disorders, such as hyperhidrosis, although the effectiveness of BTX in the treatment of HS remains unknown. The aim of this systematic review was to understand the published evidence of BTX safety and effectiveness in the management of HS.

METHODS

We conducted a PRISMA-compliant, prospectively registered (PROSPERO, CRD42021228732), systematic review. We devised bespoke search strategy and applied it to the Cochrane Central Register of Controlled Trials, Medline, Embase, and OpenGrey up until March 2022. We included all clinical studies that reported outcomes following BTX treatment in patients diagnosed with HS (both adult and pediatric).

RESULTS

A total of 4658 studies were identified, of which six met full inclusion criteria reporting data on 26 patients. The six identified studies included one randomized control trial, one case series, and four case studies. The one included randomized control trial demonstrated a significant reduction in the Dermatology Life Quality Index score at 3 months following treatment with BTX.

CONCLUSIONS

The effectiveness and safety of BTX in the treatment of HS remain unknown. This systematic review identified a paucity of high-quality clinical data. Evidence of treatment effectiveness is likely to come from registry-based cohort studies using established core outcome sets in the first instance.

Computerized adaptive testing for the Oxford Hip, Knee, Shoulder, and Elbow scores : accurate measurement from fewer, and more patient-focused, questions

AIMS

The aim of this study was to develop and evaluate machine-learning-based computerized adaptive tests (CATs) for the Oxford Hip Score (OHS), Oxford Knee Score (OKS), Oxford Shoulder Score (OSS), and the Oxford Elbow Score (OES) and its subscales.

METHODS

We developed CAT algorithms for the OHS, OKS, OSS, overall OES, and each of the OES subscales, using responses to the full-length questionnaires and a machine-learning technique called regression tree learning. The algorithms were evaluated through a series of simulation studies, in which they aimed to predict respondents' full-length questionnaire scores from only a selection of their item responses. In each case, the total number of items used by the CAT algorithm was recorded and CAT scores were compared to full-length questionnaire scores by mean, SD, score distribution plots, Pearson's correlation coefficient, intraclass correlation (ICC), and the Bland-Altman method. Differences between CAT scores and full-length questionnaire scores were contextualized through comparison to the instruments' minimal clinically important difference (MCID).

RESULTS

The CAT algorithms accurately estimated 12-item questionnaire scores from between four and nine items. Scores followed a very similar distribution between CAT and full-length assessments, with the mean score difference ranging from 0.03 to 0.26 out of 48 points. Pearson's correlation coefficient and ICC were 0.98 for each 12-item scale and 0.95 or higher for the OES subscales. In over 95% of cases, a patient's CAT score was within five points of the full-length questionnaire score for each 12-item questionnaire.

CONCLUSION

Oxford Hip Score, Oxford Knee Score, Oxford Shoulder Score, and Oxford Elbow Score (including separate subscale scores) CATs all markedly reduce the burden of items to be completed without sacrificing score accuracy.Cite this article: Bone Jt Open 2022;3(10):786-794.

Computerized adaptive testing of symptom severity: a registry-based study of 924 patients with trapeziometacarpal arthritis

We aimed to develop a computerized adaptive testing (CAT) version of the 11 item Patient Evaluation Measure (PEM), using an item response theory model. This model transformed the ordinal scores into ratio-interval scores. We obtained PEM responses from 924 patients with trapeziometacarpal osteoarthritis to build a CAT model and tested its performance on a simulated cohort of 1000 PEM response sets. The CAT achieved high precision (median standard error or measurement 0.26) and reduced the number of questions needed for accurate scoring from 11 to median two. The CAT scores and item-response-theory-based 15-item PEM scores were similar, and a Bland-Altman analysis demonstrated a mean score difference of 0.2 between the CAT and the full-length PEM scores on a scale from 0 to 100. We conclude that the CAT substantially reduced the burden of the PEM while also harnessing the validity of item response theory scoring.

Developing Machine Learning Algorithms to Support Patient-centered, Value-based Carpal Tunnel Decompression Surgery

Background

Carpal tunnel syndrome (CTS) is extremely common and typically treated with carpal tunnel decompression (CTD). Although generally an effective treatment, up to 25% of patients do not experience meaningful benefit. Given the prevalence, this amounts to considerable morbidity and cost without return. Being able to reliably predict which patients would benefit from CTD preoperatively would support more patient-centered and value-based care.

Methods

We used registry data from 1916 consecutive patients undergoing CTD for CTS at a regional hand center between 2010 and 2019. Improvement was defined as change exceeding the respective QuickDASH subscale's minimal important change estimate. Predictors included a range of clinical, demographic and patient-reported variables. Data were split into training (75%) and test (25%) sets. A range of machine learning algorithms was developed using the training data and evaluated with the test data. We also used a machine learning technique called chi-squared automatic interaction detection to develop flowcharts that could help clinicians and patients to understand the chances of a patient improving with surgery.

Results

The top performing models predicted functional and symptomatic improvement with accuracies of 0.718 (95% confidence interval 0.660, 0.771) and 0.759 (95% confidence interval 0.708, 0.810), respectively. The chi-squared automatic interaction detection flowcharts could provide valuable clinical insights from as little as two preoperative questions.

Conclusions

Patient-reported outcome measures and machine learning can support patient-centered and value-based healthcare. Our algorithms can be used for expectation management and to rationalize treatment risks and costs associated with CTD.

Sample size calculations in high-profile surgical trials that use patient-reported outcome measures: systematic review

Sample size calculations from high-profile surgical RCTs that used a patient-reported outcome measure as primary outcome were reviewed systematically against Difference ELicitation in TriAls (DELTA2) standards, with a focus on target differences. In this sample of trials, there was frequent use of suboptimal methods to determine the target difference, and sample size calculations were generally not reported to DELTA2 standards. This risks over-recruitment and/or erroneous trial conclusions, which clinicians should be aware of when interpreting published trials.

Construct validity of the anglicised FACE-Q skin cancer module

OBJECTIVES

The FACE-Q Skin Cancer module is a patient-reported outcome measure (PROM) for facial skin cancer. It has been anglicised for the UK population and undergone psychometric testing using classical test theory. In this study, further evaluation of construct validity using Rasch measurement theory and hypothesis testing was performed.

METHODS

Patients were prospectively recruited to the Patient-Reported Outcome Measures In Skin Cancer Reconstruction (PROMISCR) study and asked to complete the anglicised FACE-Q Skin Cancer module. The scalability and unidimensionality of the data were assessed with a Mokken analysis prior to Rasch analysis. Response thresholds, targeting, fit statistics, local dependency, and internal consistency were examined for all items and subscales. Four a priori hypotheses were tested to evaluate the convergent and divergent validity. We additionally hypothesised that the median 'cancer worry' score would be lower in post-operative than pre-operative patients.

RESULTS

239 patients self-completed the questionnaire between August 2017 and May 2019. Of the ten subscales assessed, five showed relative fit to the Rasch model. Unidimensionality was present for all five subscales, with most demonstrating ordered item thresholds and appropriate fit statistics. Two items in the 'cancer worry' subscale had either disordered or very close response thresholds. Subscales of the FACE-Q Skin Cancer module demonstrated convergent and divergent validity with relevant Skin Cancer Index comparators (p < 0.001). Median 'cancer worry' was lower in post-operative patients (44 vs 39, p < 0.001).

CONCLUSION

The anglicised FACE-Q Skin Cancer module shows psychometric validity through hypothesis testing, and both classical and modern test theory.

Perspectives: A surgeon's guide to machine learning

The exponential increase in the volume and complexity of healthcare data presents new challenges to researchers and clinicians in analysis and interpretation. The requirement for new strategies to extract meaningful information from large, noisy datasets has led to the development of the field of big data analytics. Artificial intelligence (AI) is a general-purpose technology in which machines carry out tasks traditionally thought to be only achievable by humans. Machine learning (ML) is an approach to AI in which machines can "learn" to perform tasks in an automated process, rather than being explicitly programmed by a human. Research aiming to apply ML techniques to classification, prediction and decision-making problems in healthcare has increased 61-fold from 2005 to 2019, mirroring this sense of early promise. The field of healthcare ML is relatively young, and many critical steps are needed before adoption into clinical practice, including transparent, unbiased development and reporting of algorithms. Articles claiming that machines can outperform, or replace, doctors in high-level tasks, such as diagnosis or prognostication, must be carefully appraised. It is critical that surgeons have an understanding of the principles and terminology of AI and ML to evaluate these claims and to take an active role in directing research. This article is an up-to-date review and primer for surgeons covering the core tenets of ML applied to surgical problems, including algorithm types and selection, model training and validation, interpretation of common outcome metrics, current and future reporting guidelines and discussion of the challenges and limitations in this field.

Machine learning in medicine: a practical introduction to natural language processing

Background

Unstructured text, including medical records, patient feedback, and social media comments, can be a rich source of data for clinical research. Natural language processing (NLP) describes a set of techniques used to convert passages of written text into interpretable datasets that can be analysed by statistical and machine learning (ML) models. The purpose of this paper is to provide a practical introduction to contemporary techniques for the analysis of text-data, using freely-available software.

Methods

We performed three NLP experiments using publicly-available data obtained from medicine review websites. First, we conducted lexicon-based sentiment analysis on open-text patient reviews of four drugs: Levothyroxine, Viagra, Oseltamivir and Apixaban. Next, we used unsupervised ML (latent Dirichlet allocation, LDA) to identify similar drugs in the dataset, based solely on their reviews. Finally, we developed three supervised ML algorithms to predict whether a drug review was associated with a positive or negative rating. These algorithms were: a regularised logistic regression, a support vector machine (SVM), and an artificial neural network (ANN). We compared the performance of these algorithms in terms of classification accuracy, area under the receiver operating characteristic curve (AUC), sensitivity and specificity.

Results

Levothyroxine and Viagra were reviewed with a higher proportion of positive sentiments than Oseltamivir and Apixaban. One of the three LDA clusters clearly represented drugs used to treat mental health problems. A common theme suggested by this cluster was drugs taking weeks or months to work. Another cluster clearly represented drugs used as contraceptives. Supervised machine learning algorithms predicted positive or negative drug ratings with classification accuracies ranging from 0.664, 95% CI [0.608, 0.716] for the regularised regression to 0.720, 95% CI [0.664,0.776] for the SVM.

Conclusions

In this paper, we present a conceptual overview of common techniques used to analyse large volumes of text, and provide reproducible code that can be readily applied to other research studies using open-source software.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01347-1.

Recursive Partitioning vs Computerized Adaptive Testing to Reduce the Burden of Health Assessments in Cleft Lip and/or Palate: Comparative Simulation Study

BACKGROUND

Computerized adaptive testing (CAT) has been shown to deliver short, accurate, and personalized versions of the CLEFT-Q patient-reported outcome measure for children and young adults born with a cleft lip and/or palate. Decision trees may integrate clinician-reported data (eg, age, gender, cleft type, and planned treatments) to make these assessments even shorter and more accurate.

OBJECTIVE

We aimed to create decision tree models incorporating clinician-reported data into adaptive CLEFT-Q assessments and compare their accuracy to traditional CAT models.

METHODS

We used relevant clinician-reported data and patient-reported item responses from the CLEFT-Q field test to train and test decision tree models using recursive partitioning. We compared the prediction accuracy of decision trees to CAT assessments of similar length. Participant scores from the full-length questionnaire were used as ground truth. Accuracy was assessed through Pearson's correlation coefficient of predicted and ground truth scores, mean absolute error, root mean squared error, and a two-tailed Wilcoxon signed-rank test comparing squared error.

RESULTS

Decision trees demonstrated poorer accuracy than CAT comparators and generally made data splits based on item responses rather than clinician-reported data.

CONCLUSIONS

When predicting CLEFT-Q scores, individual item responses are generally more informative than clinician-reported data. Decision trees that make binary splits are at risk of underfitting polytomous patient-reported outcome measure data and demonstrated poorer performance than CATs in this study.

Automated conversational agents for post-intervention follow-up: a systematic review

BACKGROUND

Advances in natural language processing and other machine learning techniques have led to the development of automated agents (chatbots) that mimic human conversation. These systems have mainly been used in commercial settings, and within medicine, for symptom checking and psychotherapy. The aim of this systematic review was to determine the acceptability and implementation success of chatbots in the follow-up of patients who have undergone a physical healthcare intervention.

METHODS

A systematic review of MEDLINE, MEDLINE In-process, EMBASE, PsychINFO, CINAHL, CENTRAL and the grey literature using a PRISMA-compliant methodology up to September 2020 was conducted. Abstract screening and data extraction were performed in duplicate. Risk of bias and quality assessments were performed for each study.

RESULTS

The search identified 904 studies of which 10 met full inclusion criteria: three randomised control trials, one non-randomised clinical trial and six cohort studies. Chatbots were used for monitoring after the management of cancer, hypertension and asthma, orthopaedic intervention, ureteroscopy and intervention for varicose veins. All chatbots were deployed on mobile devices. A number of metrics were identified and ranged from a 31 per cent chatbot engagement rate to a 97 per cent response rate for system-generated questions. No study examined patient safety.

CONCLUSION

A range of chatbot builds and uses was identified. Further investigation of acceptability, efficacy and mechanistic evaluation in outpatient care pathways may lend support to implementation in routine clinical care.