Measuring agreement between healthcare survey instruments using mutual information

Nonresponse Bias on Inpatient Rehabilitation Hospitals' Experience of Care Quality Measure Scores

Objective

To describe the magnitude of nonresponse bias on inpatient rehabilitation facility (IRF) experience of care survey data in patients with neurologic disorders.

Design

Cohort study of patients at 2 IRFs. Patients reported experience of care via an IRF-administered survey as part of routine operations approximately 2 weeks after discharge. A partially overlapping sample of research participants completed a similar survey approximately 2 weeks and 30 days after discharge.

Setting

Two inpatient rehabilitation facilities.

Participants

Patients aged ≥18 years with neurologic disorders who were discharged from 2 IRFs.

Interventions

None.

Main Outcome Measures

Experience of care data collected via an IRF Survey (self-report or proxy responses) and a Research Survey (self-report only).

Results

Of the 1055 patients admitted during the study period who met the age and diagnosis criteria, 490 (46.4%) completed one or both of the surveys. Of the 325 IRF Survey respondents, 202 were self-report, 99 were proxy respondents, and 24 were unknown respondents. Only patients completed the Research Survey (N=285). One hundred twenty patients completed both surveys, of which 7 were proxy IRF Survey respondents. IRF Survey respondents had higher cognitive function than nonrespondents; patients with spinal cord injuries were more likely to complete the IRF Survey than other patients. There were no differences in the proportions of patients answering favorably on the IRF Survey (all respondents) compared with the Research Survey, except for physician communication and discharge information. Mutual information analysis revealed agreement between the scores produced by the 2 data sources.

Conclusions

There were subtle, potentially important differences in quality measure results across surveys, reflecting the extent to which patients are encouraged to complete experience of care surveys. There was higher agreement on questions about global hospital perceptions than specific aspects of patients' experience.

Evaluating the Psychometric Properties and Clinical Utility of a Digital Psychosocial Self-Screening Tool (HEARTSMAP-U) for Postsecondary Students: Prospective Cohort Study

BACKGROUND

Existing screening tools for mental health issues among postsecondary students have several challenges, including a lack of standardization and codevelopment by students. HEARTSMAP-U was adapted to address these issues.

OBJECTIVE

This study aimed to evaluate the suitability of HEARTSMAP-U as a self-screening tool for psychosocial issues among postsecondary students by evaluating its validity evidence and clinical utility.

METHODS

A prospective cohort study was conducted with University of British Columbia Vancouver students to evaluate HEARTSMAP-U's predictive validity and convergent validity. Participating students completed baseline and 3-month follow-up assessments via HEARTSMAP-U and a clinician-administered interview.

RESULTS

In a diverse student sample (n=100), HEARTSMAP-U demonstrated high sensitivity (95%-100%) in identifying any psychiatric concerns that were flagged by a research clinician, with lower specificity (21%-25%). Strong convergent validity (r=0.54-0.68) was demonstrated when relevant domains and sections of HEARTSMAP-U were compared with those of other conceptually similar instruments.

CONCLUSIONS

This preliminary evaluation suggests that HEARTSMAP-U may be suitable for screening in the postsecondary educational setting. However, a larger-scale evaluation is necessary to confirm and expand on these findings.

Convergent Validity of Myheartsmap: A Pediatric Psychosocial Health Screening Tool

Recognition of pediatric mental health concerns often depends on assessment by parents, educators, and primary care professionals. Therefore, a psychosocial screening instrument suitable for routine use in schools and primary care is needed. The Pediatric Quality of Life (PedsQL) and the Strengths and Difficulties Questionnaire (SDQ) are widely used for screening but lack adolescent-specific mental health measures. MyHEARTSMAP is an instrument assessing aspects of youth psychosocial health via four domains: Psychiatry, Function, Social, and Youth Health. We evaluated MyHEARTSMAP convergent validity with PedsQL and SDQ among 122 child-parent dyads participating in a larger concussion study. Convergent validity was assessed via correlations: MyHEARTSMAP Psychiatry and Function domains correlated strongly (r ≥ 0.44) and Social domain correlated weakly (r ≤ 0.25) to corresponding PedsQL and SDQ subscales, while Youth Health domain correlated moderately (r ≥ 0.31) to the tools' total scales. In conclusion, MyHEARTSMAP converges with PedsQL and SDQ, and benefits from the inclusion of adolescent-specific psychosocial measures.

Fifty years of Shannon information theory in assessing the accuracy and agreement of diagnostic tests

Since 1948, Shannon theoretic methods for modeling information have found a wide range of applications in several areas where information plays a key role, which goes well beyond the original scopes for which they have been conceived, namely data compression and error correction over a noisy channel. Among other uses, these methods have been applied in the broad field of medical diagnostics since the 1970s, to quantify diagnostic information, to evaluate diagnostic test performance, but also to be used as technical tools in image processing and registration. This review illustrates the main contributions in assessing the accuracy of diagnostic tests and the agreement between raters, focusing on diagnostic test performance measurements and paired agreement evaluation. This work also presents a recent unified, coherent, and hopefully, final information-theoretical approach to deal with the flows of information involved among the patient, the diagnostic test performed to appraise the state of disease, and the raters who are checking the test results. The approach is assessed by considering two case studies: the first one is related to evaluating extra-prostatic cancers; the second concerns the quality of rapid tests for COVID-19 detection.

Beyond kappa: an informational index for diagnostic agreement in dichotomous and multivalue ordered-categorical ratings

Agreement measures are useful tools to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. Cohen's kappa (κ) certainly is the most popular agreement method between two raters, and proved its effectiveness in the last sixty years. In spite of that, this method suffers from some alleged issues, which have been highlighted since the 1970s; moreover, its value is strongly dependent on the prevalence of the disease in the considered sample. This work introduces a new agreement index, the informational agreement (IA), which seems to avoid some of Cohen's kappa's flaws, and separates the contribution of the prevalence from the nucleus of agreement. These goals are achieved by modelling the agreement-in both dichotomous and multivalue ordered-categorical cases-as the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. In order to test its fair behaviour and the effectiveness of the method, IA has been tested on some cases known to be problematic for κ, in the machine learning context and in a clinical scenario to compare ultrasound (US) and automated breast volume scanner (ABVS) in the setting of breast cancer imaging. Graphical Abstract To evaluate the agreement between the two raters [Formula: see text] and [Formula: see text] we create an agreement channel, based on Shannon Information Theory, that directly connects the random variables X and Y, that express the raters outcomes. They are the terminals of the chain X⇔ diagnostic test performed by [Formula: see text] ⇔ patient condition[Formula: see text] ⇔ diagnostic test performed by [Formula: see text] ⇔ Y.