A New Interpretation of the Weighted Kappa Coefficients

Large language models facilitate the generation of electronic health record phenotyping algorithms

OBJECTIVES

Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts.

MATERIALS AND METHODS

We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (ie, type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.

RESULTS

GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).

CONCLUSION

GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

SWI versus GRE-T2*: Assessing cortical superficial siderosis in advanced cerebral amyloid angiopathy

BACKGROUND AND PURPOSE

Cortical superficial siderosis (cSS) is a key neuroimaging marker of cerebral amyloid angiopathy (CAA) detected on blood-sensitive magnetic resonance imaging (MRI). We aimed to assess cSS in advanced CAA patients and explore differences in its evaluation between susceptibility weighted imaging (SWI) and gradient recalled echo-T2* (GRE-T2*).

MATERIALS AND METHODS

Neuroimaging data gathered from a prospective cohort of CAA patients with probable or definite CAA were retrospectively analyzed by two independent raters. SWI and GRE-T2* were used to assess presence and severity (absent, focal [≤3 sulci] or disseminated [>3 sulci]) of cSS and number of foci. Ratings were compared between sequences and inter-rater agreement was determined. Post hoc analysis explored differences in cSS multifocality scores.

RESULTS

We detected cSS in 38 patients with SWI and in 36 with GRE-T2* (70.4% versus 66.7%; P=0.5). The two raters agreed in detecting more disseminated cSS when using SWI: 16 focal (29.63%) and 20 disseminated (37.04%) cases of cSS seen on GRE-T2* and 11 (20.37%) focal and 27 (50%) disseminated cSS cases seen using SWI (P=0.008). Inter-rater agreement was equivalent for the two sequences (κpresence 0.7 versus 0.69; κseverity 0.74 versus 0.66) for assessing both presence and severity of cSS. Post hoc analysis showed higher multifocality scores from both raters' SWI evaluations, with agreement equivalent to that for T2* evaluations.

CONCLUSIONS

Our findings suggest that SWI ratings could show more disseminated cSS and higher multifocality scores in advanced CAA patients with inter-rater reliability equivalent to that obtained using GRE-T2*, regardless of level of experience.

The barometer of moral distress in midwifery: A pilot study

BACKGROUND

Moral distress is a phenomena that occurs following a compromise to moral beliefs. Moral distress has been reported across health professions, including midwifery. Although there are validated tools to assess for moral distress, none have been identified that suit the Australian healthcare system or midwifery.

AIM

The aim of this study was to pilot the Barometer of Moral Distress in Midwifery.

METHODS

This study was the fourth stage of a mixed method project. Using a cross-sectional approach, a survey tool including demographic questions, the Barometer of Moral Distress in Midwifery, and the Copenhagen Burnout Inventory assessed tool stability, reliability, and validity.

FINDINGS

A total of 103 surveys were completed. A test-retest demonstrated tool reliability and stability (a =.97). Factor analysis confirmed internal consistency; Factor 1 - Professional Identity (a=.91), Factor 2 - Inadequate Resources (a=.85), and Factor 3 - Unethical Cultures (a=.88). Concurrent validity was demonstrated through positive correlations between self-reported types of moral distress with mean scores for each Factor. Strong correlations were identified between work-related burnout and mean scores, while only weak correlations were noted between client-related burnout and mean scores. Only Factor 1 demonstrated a correlation between leaving the profession and mean scores.

DISCUSSION/CONCLUSION

This was the first moral distress tool that assessed both frequency of exposure and psychological outcomes to score moral distress. Findings indicate that moral distress in midwifery is not associated with caring work but with occupational environments. Further research is required to assess self-sacrifice in moral distress.

Translating and establishing the psychometric properties of the Jenkins Sleep Scale for Arabic-speaking individuals

BACKGROUND

The Jenkins Sleep Scale is a widely used self-report questionnaire that assesses sleep quality and disturbances. This study aimed to translate the scale into Arabic and evaluate its psychometric properties in an Arabic-speaking population.

METHODS

The Jenkins Sleep Scale was translated into Arabic using forward and backward translation procedures. The Arabic version was administered to a convenience sample of 420 adults along with the Pittsburgh Sleep Quality Index (PSQI) and Athens Insomnia Scale (AIS) for validation purposes. Reliability was examined using Cronbach's alpha and McDonald's omega coefficients. Confirmatory factor analysis (CFA) was also conducted to test the unidimensional factor structure. Convergent validity was assessed using correlations with PSQI and AIS scores.

RESULTS

The Cronbach's alpha and McDonald's omega values for the Arabic Jenkins Sleep Scale were 0.74 and 0.75, respectively, indicating good internal consistency. The 2-week and 4-week test-retest intraclass correlation coefficients were both 0.94 (p < 0.001), indicating excellent test-retest reliability. The CFA results confirmed the unidimensional factor structure (CFI = 0.99, TLI = 0.96, RMSEA = 0.08). The measurement model had an equivalent factor structure, loadings, intercepts, and residuals across sex, age, and marital status. Significant positive correlations were found between the Arabic Jenkins scale score and the PSQI (r = 0.80, p < 0.001) and AIS (r = 0.74, p < 0.001), supporting convergent validity.

CONCLUSION

The Arabic version of the Jenkins Sleep Scale demonstrated good psychometric properties. The findings support its use as a valid and reliable measure for evaluating sleep quality and disturbances among Arabic-speaking populations.

Evaluating inter-rater reliability in the context of "Sysmex UN2000 detection of protein/creatinine ratio and of renal tubular epithelial cells can be used for screening lupus nephritis": a statistical examination

BACKGROUND

The evaluation of inter-rater reliability (IRR) is integral to research designs involving the assessment of observational ratings by two raters. However, existing literature is often heterogeneous in reporting statistical procedures and the evaluation of IRR, although such information can impact subsequent hypothesis testing analyses.

METHODS

This paper evaluates a recent publication by Chen et al., featured in BMC Nephrology, aiming to introduce an alternative statistical approach to assessing IRR and discuss its statistical properties. The study underscores the crucial need for selecting appropriate Kappa statistics, emphasizing the accurate computation, interpretation, and reporting of commonly used IRR statistics between two raters.

RESULTS

The Cohen's Kappa statistic is typically used for two raters dealing with two categories or for unordered categorical variables having three or more categories. On the other hand, when assessing the concordance between two raters for ordered categorical variables with three or more categories, the commonly employed measure is the weighted Kappa.

CONCLUSION

Chen and colleagues might have underestimated the agreement between AU5800 and UN2000. Although the statistical approach adopted in Chen et al.'s research did not alter their findings, it is important to underscore the importance of researchers being discerning in their choice of statistical techniques to address their specific research inquiries.

Large Language Models Facilitate the Generation of Electronic Health Record Phenotyping Algorithms

Objectives

Phenotyping is a core task in observational health research utilizing electronic health records (EHRs). Developing an accurate algorithm demands substantial input from domain experts, involving extensive literature review and evidence synthesis. This burdensome process limits scalability and delays knowledge discovery. We investigate the potential for leveraging large language models (LLMs) to enhance the efficiency of EHR phenotyping by generating high-quality algorithm drafts.

Materials and Methods

We prompted four LLMs-GPT-4 and GPT-3.5 of ChatGPT, Claude 2, and Bard-in October 2023, asking them to generate executable phenotyping algorithms in the form of SQL queries adhering to a common data model (CDM) for three phenotypes (i.e., type 2 diabetes mellitus, dementia, and hypothyroidism). Three phenotyping experts evaluated the returned algorithms across several critical metrics. We further implemented the top-rated algorithms and compared them against clinician-validated phenotyping algorithms from the Electronic Medical Records and Genomics (eMERGE) network.

Results

GPT-4 and GPT-3.5 exhibited significantly higher overall expert evaluation scores in instruction following, algorithmic logic, and SQL executability, when compared to Claude 2 and Bard. Although GPT-4 and GPT-3.5 effectively identified relevant clinical concepts, they exhibited immature capability in organizing phenotyping criteria with the proper logic, leading to phenotyping algorithms that were either excessively restrictive (with low recall) or overly broad (with low positive predictive values).

Conclusion

GPT versions 3.5 and 4 are capable of drafting phenotyping algorithms by identifying relevant clinical criteria aligned with a CDM. However, expertise in informatics and clinical experience is still required to assess and further refine generated algorithms.

Agreement Between Single Raters and Team Rating When Applying the International Classification of Functioning, Disability and Health's Rehabilitation Set

OBJECTIVE

To quantify the agreement between functional assessments by a single rater and a team using the Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set in a clinical situation.

DESIGN

Inter-rater, multi-centre agreement study.

SUBJECTS

A total of 193 adult inpatients admitted to 5 rehabilitation centres at 5 hospitals in China Methods: The Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set was used by either a single rater or a team to assess 193 patients at 5 Chinese hospitals. Percentage of agreement and quadratic-weighted kappa coefficients were computed. Evaluation times were compared with paired t-tests.

RESULTS

The mean team and individual evaluation times were not significantly different. The percentage of agreement ranged from 46.1% to 94.2% depending on the item, and the quadratic-weighted kappas ranged from 0.43 to 0.92. Eight categories (26.6%) showed a weighted kappa exceeding 0.4, 11 others (36.7%) exceeded 0.6, and another 11 (36.7%) produced kappas of more than 0.8.

CONCLUSION

Either a single rater or a team of raters can produce valid and consistent ratings when using the Chinese version of the International Classification of Functioning, Disability and Health Rehabilitation Set to assess patients in a rehabilitation department. The team rating approach is suitable for clinical application.

Validity and reliability of the Danish version of the Pittsburgh Sleep Quality Index

AIM

The Pittsburgh Sleep Quality Index is a widely used measure of sleep quality. The validity and reliability of the Danish version of the instrument has not yet been established. The aim of this study is to establish the internal consistency, test-retest reliability and convergent validity of the Danish version of the Pittsburgh Sleep Quality Index in adolescents.

METHODS

Data were collected from 719 students from 17 different upper secondary schools. The sample consisted of 55% women, and the mean age of the sample was 17.87 (2.52) years. Data was collected on two occasions approximately 6 weeks apart using online surveys. Apart from the Pittsburgh Sleep Quality Index, the students also completed the five-item World Health Organization Well-being Index and the 10-item Perceived Stress Scale. For internal consistency the Cronbach's alpha was calculated, for test-retest reliability the interclass correlation coefficient was computed, and for validity the Pearson's correlation was calculated.

RESULTS

The mean global Pittsburgh Sleep Quality Index score among Danish adolescents was 5.86 (3.13). The internal consistency for the Pittsburgh Sleep Quality Index was satisfactory with a Cronbach's alpha of 0.72. Test-retest reliability was adequate with an interclass correlation coefficient of 0.68. Finally, the Pittsburgh Sleep Quality Index showed large positive correlation with the 10-item Perceived Stress Scale (0.55) and a large negative correlation with the five-item World Health Organization Well-being Index (-0.59).

CONCLUSIONS

The Danish version of the Pittsburgh Sleep Quality Index showed adequate reliability and validity among Danish adolescents.

Oxygen comfort evaluation method based on symptom index for short-term internal migrants to Tibet

In Tibet, the hypobaric-hypoxic environment found at high altitudes leads to dysfunction in short-term internal migrants and has noticeable effects on physiology, psychological health, and comfort level. Therefore, it is essential to accurately determine the degree of hypoxia and improve the hypoxic environment of plateaus. Despite advances in the medical diagnosis and treatment of pathological hypoxic injuries, there are some limitations in the oxygenic evaluation of internal migrants with mild hypoxia. An oxygen comfort evaluation method (OCEM) based on typical anoxic symptomatology and physiological indices is proposed in this study. Experiments with different oxygen concentrations were conducted to measure anoxic symptomatology and physiological indices. Using item and exploratory factor analyses, 19 symptom indices were screened to predict oxygen sensation in humans. Finally, the OCEM was established using an artificial neural network and fuzzy mathematics method and its accuracy was verified through a field survey. The results showed that the artificial neural network model using symptomatologic indices could predict human oxygen sensation, with an area under the receiver operating characteristic curve of 0.630-0.913 and prediction accuracy of 93 %. Oxygen comfort can be predicted from the oxygen sensation and typical physiological indices using the fuzzy mathematics method; the weighted kappa coefficient was 0.825, indicating a strong correlation between the predicted and actual values. The proposed OCEM can help determine the oxygen comfort conditions of high-altitude internal migrants and provide a basis for indoor oxygen environment regulation in high-altitude buildings.

Using Augmented Reality Toward Improving Social Skills: Scoping Review

BACKGROUND

Augmented reality (AR) has emerged as a promising technology in educational settings owing to its engaging nature. However, apart from applications aimed at the autism spectrum disorder population, the potential of AR in social-emotional learning has received less attention.

OBJECTIVE

This scoping review aims to map the range of AR applications that improve social skills and map the characteristics of such applications.

METHODS

In total, 2 independent researchers screened 2748 records derived from 3 databases in December 2021-PubMed, IEEE Xplore, and ACM Guide to Computing Literature. In addition, the reference lists of all the included records and existing reviews were screened. Records that had developed a prototype with the main outcome of improving social skills were included in the scoping review. Included records were narratively described for their content regarding AR and social skills, their target populations, and their outcomes. Evaluation studies were assessed for methodological quality.

RESULTS

A total of 17 records met the inclusion criteria for this study. Overall, 10 records describe applications for children with autism, primarily teaching about reading emotions in facial expressions; 7 records describe applications for a general population, targeting both children and adults, with a diverse range of outcome goals. The methodological quality of evaluation studies was found to be weak.

CONCLUSIONS

Most applications are designed to be used alone, although AR is well suited to facilitating real-world interactions during a digital experience, including interactions with other people. Therefore, future AR applications could endorse social skills in a general population in more complex group settings.

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

BACKGROUND

In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing.

METHODS

This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved.

RESULTS

The Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure.

CONCLUSION

Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research.

The Dependence of Chance-Corrected Weighted Agreement Coefficients on the Power Parameter of the Weighting Scheme: Analysis and Measurement

We consider the dependence of a broad class of chance-corrected weighted agreement coefficients on the weighting scheme that penalizes rater disagreements. The considered class encompasses many existing coefficients with any number of raters, and one real-valued power parameter defines the weighting scheme that includes linear, quadratic, identity, and radical weights. We obtain the first-order and second-order derivatives of the coefficients with respect to the power parameter and decompose them into components corresponding to all pairs of different category distances. Each component compares its two distances in terms of the ratio of observed to expected-by-chance frequency. A larger ratio for the smaller distance than the larger distance contributes to a positive relationship between the power parameter and the coefficient value; the opposite contributes to a negative relationship. We provide necessary and sufficient conditions for the coefficient value to increase or decrease and the relationship to intensify or weaken as the power parameter increases. We use the first-order and second-order derivatives for corresponding measurement. Furthermore, we show how these two derivatives allow other researchers to obtain quite accurate estimates of the coefficient value for unreported values of the power parameter, even without access to the original data.

Development and Narrow Validation of Computer Vision Approach to Facilitate Assessment of Change in Pigmented Cutaneous Lesions

The documentation of the change in the number and appearance of pigmented cutaneous lesions over time is critical to the early detection of skin cancers and may provide preliminary signals of efficacy in early-phase therapeutic prevention trials for melanoma. Despite substantial progress in computer-aided diagnosis of melanoma, automated methods to assess the evolution of lesions are relatively undeveloped. This report describes the development and narrow validation of mathematical algorithms to register nevi between sequential digital photographs of large areas of skin and to align images for improved detection and quantification of changes. Serial posterior truncal photographs from a pre-existing database were processed and analyzed by the software, and the results were evaluated by a panel of clinicians using a separate Extensible Markup Language‒based application. The software had a high sensitivity for the detection of cutaneous lesions as small as 2 mm. The software registered lesions accurately, with occasional errors at the edges of the images. In one pilot study with 17 patients, the use of the software enabled clinicians to identify new and/or enlarged lesions in 3‒11 additional patients versus the unregistered images. Automated quantification of size change performed similarly to that of human raters. These results support the further development and broader validation of this technique.

A Prospective, Longitudinal and Exploratory Study of Head and Neck Lymphoedema and Dysphagia Following Chemoradiotherapy for Head and Neck Cancer

The aim of the study was to examine the following: (a) the trajectory of external and internal head and neck lymphoedema (HNL) in patients with head and neck cancer (HNC) up to 12 months post-chemoradiotherapy (CRT) and (b) the relationship between HNL and swallowing function. Using a prospective longitudinal cohort study, external/internal HNL and swallowing were examined in 33 participants at 3, 6 and 12 months post-CRT. External HNL was assessed using the Assessment of Lymphoedema of the Head and Neck and the MD Anderson Cancer Centre Lymphoedema Rating Scale. Internal HNL was rated using Patterson’s Radiotherapy Oedema Rating Scale. Swallowing was assessed via clinical, instrumental and patient-reported measures. Associations between HNL and swallowing were examined using multivariable regression models. External HNL was prevalent at 3 months (71%), improved by 6 months (58%) and largely resolved by 12 months (10%). In contrast, moderate/severe internal HNL was prevalent at 3 months (96%), 6 months (84%) and at 12 months (65%). More severe penetration/aspiration and increased diet modification were associated with higher severities of external HNL (p=0.006 and p=0.031, respectively) and internal HNL (p<0.001 and p=0.007, respectively), and more diffuse internal HNL (p=0.043 and p=0.001, respectively). Worse patient-reported swallowing outcomes were associated with a higher severity of external HNL (p=0.001) and more diffuse internal HNL (p=0.002). External HNL largely resolves by 12 months post-CRT, but internal HNL persists. Patients with a higher severity of external and/or internal HNL and those with more diffuse internal HNL can be expected to have more severe dysphagia.

Validation of the Swedish version of PROMIS-29v2 and FACIT-Dyspnea Index in patients with systemic sclerosis

PURPOSE

To evaluate the reliability, internal consistency, and construct validity of the Swedish versions of PROMIS-29 and Functional Assessment of Chronic Illness Therapy-Dyspnea (FACIT-Dyspnea) instruments in patients with systemic sclerosis (SSc).

METHODS

In a cross-sectional study, consecutive SSc patients completed a paper-based survey. Internal consistency was assessed using Cronbach's alpha. Test-retest reliability was tested employing weighted Kappa (K_w) and intra-class correlation coefficient (ICC). Construct validity was evaluated by hypotheses testing using RAND-36, MRC Dyspnea score, Scleroderma Health Assessment Questionnaire (SHAQ) and clinical measurements.

RESULTS

Forty-nine patients (86% female; 73% limited cutaneous SSc) completed the survey. The mean disease duration was 11 years and mean SHAQ was 0.5. Internal consistency and test-retest reliability were good with the exception of PROMIS-29 anxiety. PROMIS-29, FACIT-Dyspnea, and Functional limitation showed strong correlations to corresponding RAND-36 domains (|r_s|=0.67 to -0.85). Relevant PROMIS-29 domains, FACIT-Dyspnea and Functional limitation correlated strongly to SHAQ and VAS overall disease severity (|r_s|=0.60 to -0.75). Ceiling effects (>15%) were found in six PROMIS-29 domains and in both FACIT-Dyspnea and Functional limitations. Four (4/5) hypotheses were confirmed.

CONCLUSIONS

PROMIS-29 and FACIT-Dyspnea meet the requirements for reliability and have adequate construct validity in Swedish patients with SSc.Implications for rehabilitationPROMIS-29v2 and Functional Assessment of Chronic Illness Therapy-Dyspnea (FACIT-Dyspnea) Index are patient outcome measures that gain increasing interest for the evaluation of patient with rheumatologic diseases.PROMIS-29v2 and FACIT-Dyspnea Index meet the requirements for reliability and have adequate construct validity compared to legacy measures in Swedish patients with systemic sclerosis.Translation and validation of PROMs is important for studies of rare diseases in multi-center collaborations.

Graduate Student Clinicians' Perceptions of Child Speech Sound Errors

Purpose

Speech-language pathologists (SLPs) rely on auditory perception to form judgments on child speech. This can be challenging for graduate student clinicians with limited clinical experience as they often need to judge children's speech errors using their auditory perception. This study examined how consistently graduate student clinicians used a 3-point perceptual rating scale to judge child speech.

Method

Twenty-four graduate student clinicians rated single words produced by children with typically developing speech and language skills and children with speech sound disorders. All participants rated the productions using a 3-point scale, where "2" was an accurate production, "1" was a close approximation, and "0" was an inaccurate production. Ratings were solely based on the auditory signal. These ratings were compared to a consensus rating formed by two experienced SLPs.

Results

Graduate student clinicians reached substantial agreement with the expert SLP rating. They reached the highest percentage agreement when rating accurate productions, and the lowest agreement when rating inaccurate productions.

Conclusions

Graduate student clinicians reached substantial agreement with expert SLP rating in judging child speech using a 3-point scale when provided with detailed descriptions of each rating category. These results are consistent with previous findings on the role that clinical experience plays in speech error perception tasks and highlight the need for additional listening training in speech-language pathology graduate programs.

A Measurement-Based Protocol Improves Inter-Rater Agreement And Accuracy of Right Ventricular Systolic Pressure Assessment by Echocardiography in Children: A Call for Quality Improvement

BACKGROUND

Echocardiographic measurements carry the promise of improving inter-rater agreement (IR) over subjective assessment. In this study we assess the effect of implementing a measurement-based protocol on the IR agreement and accuracy in reporting of right ventricular (RV) systolic pressure in children. We also evaluate the effect of this reporting protocol on the IR agreement in reporting RV dilation, hypertrophy and systolic function.

METHODS

Five echocardiography readers reported their assessment of the RV systolic pressure, dilation, hypertrophy and systolic function on 40 deidentified echocardiograms using their individual accustomed methods and then using an agreed-upon protocol based solely on RV measurements. The IR agreement was assessed using kappa statistics. Accuracy of the RV systolic pressure ratings was assessed using McNemar's test in comparison to hemodynamic data obtained by cardiac catheterization. The reliability of the RV measurements was assessed using intraclass correlation coefficient (ICC) and coefficient of variation (CoV).

RESULTS

The IR agreement and accuracy of RV systolic pressure assessment improved after employing the measurement-based protocol [agreement: 0.39 (95% confidence interval: 0.27,0.5) to 0.62 (0.48,0.76) & accuracy 18/40 to 29/40 p = 0.03]. IR agreement of RV dilation improved [0.36 (0.25,0.48) to 0.63 (0.48,0.79)] while the IR agreement of RV hypertrophy [0.29 (0.17,0.42) to 0.35 (0.15,0.55)] and RV systolic function [0.57 (0.45,0.69) to 0.53 (0.41,0.66)] did not improve. The reliability of the measurements was good (ICC > 0.8) except for RV free wall thickness (ICC: 0.62, CoV: 24%) and RV fractional area change (ICC: 0.47, CoV: 22%) proposing a possible reason for lack of improvement in IR agreement of RV hypertrophy and RV systolic function. Heteroscedasticity was observed in the reliability of RV measurements with the ICC being significantly lower at larger magnitudes for all RV measurements.

CONCLUSIONS

Standardization of reporting protocols using RV measurements in place of subjective assessment improved IR agreement and accuracy of RV systolic pressure assessment. Reliable measurements, RV systolic pressure and dilation, resulted in improvement in IR agreement while unreliable measurements, RV hypertrophy and systolic function, did not. Special attention to measurements' reliability and heteroscedasticity of reliability is required when designing clinical protocols to decrease IR disagreement as a source of error.

Injury prevention for older adults: A dataset of safety concern narratives from online reviews of mobility-related products

Older adults are among the fastest-growing demographic groups in the United States, increasing by over a third this past decade. Consequently, the older adult consumer product market has quickly become a multi-billion-dollar industry in which millions of products are sold every year. However, the rapidly growing market raises the potential for an increasing number of product safety concerns and consumer product-related injuries among older adults. Recent manufacturer and consumer injury prevention efforts have begun to turn towards online reviews, as these provide valuable information from which actionable, timely intelligence can be derived and used to detect safety concerns and prevent injury. The presented dataset contains 1966 curated online product reviews from consumers, equally distributed between safety concerns and non-concerns, pertaining to product categories typically intended for older adults. Identified safety concerns were manually sub-coded across thirteen dimensions designed to capture relevant aspects of the consumer's experience with the purchased product, facilitate the safety concern identification and sub-classification process, and serve as a gold-standard, balanced dataset for text classifier learning.

Cross-cultural adaptation and reliability of the Arabic version of Children's Hand-use Experience Questionnaire (CHEQ)

Background

Validated outcome measures are essential for assessment and treatment of children with disabilities. The Children's Hand-use Experience Questionnaire (CHEQ) was developed and validated for use in Western countries for children with unilateral hand dysfunction. This study aimed to perform a cross-cultural adaptation and investigate reliability for the Arabic CHEQ.

Methods

Translation and cross-cultural adaptation were performed in four phases: (i) forward-translation and reconciliation with feedback from parents and typically developing children from Jordan (n = 14); (ii) backward-translation and review; (iii) cognitive debriefing with parents and/or their children with unilateral hand dysfunction (n = 17); and (iv) review and proofreading. In the psychometric analyses, 161 children from Jordan (mean age [SD] 10y 8 m [5y 8 m]; 88 males) participated. Internal consistency was evaluated with Cronbach's alpha. Test-retest reliability was evaluated in 39 children with intraclass correlation coefficient (ICC) and weighted kappa (κ).

Results

Synonyms of four words were added to accommodate for different Arabic dialects. On average, 93% of children with unilateral hand dysfunction and their parents understood the CHEQ items. One response alternative, 'Get help', to the opening question was unclear for 70% of the respondents and need further explanation. Two items about using a knife and fork were difficult to comprehend and culturally irrelevant. High internal consistency was demonstrated (Cronbach's alphas 0.94- 0.97) and moderate to excellent ICC (0.77-0.93). For 18 individual items, κ indicated poor to good agreement (κ between 0.28 and 0.66).

Conclusions

After the suggested minor adjustments, the Arabic CHEQ will be comprehensible, culturally relevant and reliable for assessing children with unilateral hand dysfunction in Jordan.

Improving the patient-reported outcome sections of clinical trial protocols: a mixed methods evaluation of educational workshops

Introduction

Failure to incorporate key patient-reported outcome (PRO) content in trial protocols affects the quality and interpretability of the collected data, contributing to research waste. Our group developed evidence-based training specifically addressing PRO components of protocols. We aimed to assess whether 2-day educational workshops improved the PRO completeness of protocols against consensus-based minimum standards provided in the SPIRIT-PRO Extension in 2018.

Method

Annual workshops were conducted 2011–2017. Participants were investigators/trialists from cancer clinical trials groups. Although developed before 2018, workshops covered 15/16 SPIRIT-PRO items. Participant feedback immediately post-workshop and, retrospectively, in November 2017 was summarised descriptively. Protocols were evaluated against SPIRIT-PRO by two independent raters for workshop protocols (developed post-workshop by participants) and control protocols (contemporaneous non-workshop protocols). SPIRIT-PRO items were assessed for completeness (0 = not addressed, 10 = fully addressed). Mann–Whitney U tests assessed whether workshop protocols scored higher than controls by item and overall.

Results

Participants (n = 107) evaluated the workshop positively. In 2017, 16/41 survey responders (39%) reported never applying in practice; barriers included role restrictions (14/41, 34%) and lack of time (5/41, 12%). SPIRIT-PRO overall scores did not differ between workshop (n = 13, median = 3.81/10, interquartile range = 3.24) and control protocols (n = 9, 3.51/10 (2.14)), (p = 0.35). Workshop protocols scored higher than controls on two items: ‘specify PRO concepts/domains’ (p = 0.05); ‘methods for handling missing data’ (p = 0.044).

Conclusion

Although participants were highly satisfied with these workshops, the completeness of PRO protocol content generally did not improve. Additional knowledge translation efforts are needed to assist protocol writers address SPIRIT-PRO guidance and avoid research waste that may eventuate from sub-optimal PRO protocol content.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11136-022-03127-w.

The Utility of a Novel Proximal Femur Maturity Index for Staging Skeletal Growth in Patients with Idiopathic Scoliosis

BACKGROUND

For growing patients, it is ideal to have a growth plate visible in routine radiographs for skeletal maturity assessment without additional radiation. The proximal femoral epiphyseal ossification is in proximity to the spine; however, whether it can be used for assessing a patient's growth status remains unknown.

METHODS

Two hundred and twenty sets of radiographs of the spine and the left hand and wrist of patients with idiopathic scoliosis were assessed for skeletal maturity and reliability testing. Risser staging, Sanders staging (SS), distal radius and ulna (DRU) classification, the proximal humeral ossification system (PHOS), and the novel proximal femur maturity index (PFMI) were used. The PFMI was newly developed on the basis of the radiographic appearances of the femoral head, greater trochanter, and triradiate cartilage. It consists of 7 grades (0 to 6) associated with increasing skeletal maturity. The PFMI was evaluated through its relationship with pubertal growth (i.e., the rate of changes of standing and sitting body height [BH] and arm span [AS]) and with established skeletal maturity indices. Longitudinal growth data and 780 corresponding spine radiographs were assessed to detect peak growth using receiver operating characteristic (ROC) curve analysis.

RESULTS

The PFMI was found to be correlated with chronological age (τ b = 0.522), growth rates based on standing BH (τ b = -0.303), and AS (τ b = -0.266) (p < 0.001 for all). The largest growth rate occurred at PFMI grade 3, with mean standing BH growth rates (and standard deviations) of 0.79 ± 0.44 cm/month for girls and 1.06 ± 0.67 cm/mo for boys. Growth rates of 0.12 ± 0.23 cm/mo (girls) and 0 ± 0 cm/mo (boys) occurred at PFMI grade 6, indicating growth cessation. Strong correlations were found between PFMI gradings and Risser staging (τ b = 0.743 and 0.774 for girls and boys), Sanders staging (τ b = 0.722 and 0.736, respectively), and radius (τ b = 0.792 and 0.820) and ulnar gradings (τ b = 0.777 and 0.821), and moderate correlations were found with PHOS stages (τ b = 0.613 and 0.675) (p < 0.001 for all). PFMI gradings corresponded to as young as SS1, R4, U1, and PHOS stage 1. Fair to excellent interrater and intrarater reliabilities were observed. PFMI grade 3 was most prevalent and predictive for peak growth based on ROC results.

CONCLUSIONS

The PFMI demonstrated clear pubertal growth phases with satisfactory reliability. Grade 3 indicates peak growth and grade 6 indicates growth cessation.

CLINICAL RELEVANCE

The use of PFMI can benefit patients by avoiding additional radiation in skeletal maturity assessment and can impact current clinical protocol of patient visits. PFMI gradings had strong correlations with SS, DRU gradings, and Risser staging, and they cross-referenced to their established grades at peak growth and growth cessation. PFMI may aid in clinical decision making.

Comparing Fire Extent and Severity Mapping between Sentinel 2 and Landsat 8 Satellite Sensors

Mapping of fire extent and severity across broad landscapes and timeframes using remote sensing approaches is valuable to inform ecological research, biodiversity conservation and fire management. Compiling imagery from various satellite sensors can assist in long-term fire history mapping; however, inherent sensor differences need to be considered. The New South Wales Fire Extent and Severity Mapping (FESM) program uses imagery from Sentinel and Landsat satellites, along with supervised classification algorithms, to produce state-wide fire maps over recent decades. In this study, we compared FESM outputs from Sentinel 2 and Landsat 8 sensors, which have different spatial and spectral resolutions. We undertook independent accuracy assessments of both Sentinel 2 and Landsat 8 sensor algorithms using high-resolution aerial imagery from eight training fires. We also compared the FESM outputs from both sensors across 27 case study fires. We compared the mapped areas of fire severity classes between outputs and assessed the classification agreement at random sampling points. Our independent accuracy assessment demonstrated very similar levels of accuracy for both sensor algorithms. We also found that there was substantial agreement between the outputs from the two sensors. Agreement on the extent of burnt versus unburnt areas was very high, and the severity classification of burnt areas was typically either in agreement between the sensors or in disagreement by only one severity class (e.g., low and moderate severity or high and extreme severity). Differences between outputs are likely partly due to differences in sensor resolution (10 m and 30 m pixel sizes for Sentinel 2 and Landsat 8, respectively) and may be influenced by landscape complexity, such as terrain roughness and foliage cover. Overall, this study supports the combined use of both sensors in remote sensing applications for fire extent and severity mapping.

Drug-related hospital admissions in older adults: comparison of the Naranjo algorithm and an adjusted version of the Kramer algorithm

PURPOSE

Drug-related admissions (DRAs) are an important cause of preventable harm in older adults. Multiple algorithms exist to assess causality of adverse drug reactions, including the Naranjo algorithm and an adjusted version of the Kramer algorithm. The performance of these tools in assessing DRA causality has not been robustly shown. This study aimed to evaluate the ability of the adjusted Kramer algorithm to adjudicate DRA causality in geriatric inpatients.

METHODS

DRAs were assessed in a convenience sample of patients admitted to the acute geriatric wards of an academic hospital. DRAs were identified by expert consensus and causality was evaluated using the Naranjo and the adjusted Kramer algorithms. Positive agreement with expert consensus was calculated for both algorithms. A multivariable logistic regression analysis was performed to explore determinants for a DRA.

RESULTS

A total of 218 geriatric inpatients was included of whom 65 (29.8%) experienced a DRA. Positive agreement was 72.3% (95% confidence interval (CI), 59.6-82.3%) and 100% (95% CI, 93.0-100%) for the Naranjo and the adjusted Kramer algorithm, respectively. Diuretics were the main culprits and most DRAs were attributed to a fall (n = 18; 27.7%). A fall-related principal diagnosis was independently associated with a DRA (odds ratio 20.11; 95% CI, 5.60-72.24).

CONCLUSION

The adjusted Kramer algorithm demonstrated a higher positive agreement with expert consensus in assessing DRA causality in geriatric inpatients compared to the Naranjo algorithm. Our results further support implementation of the adjusted Kramer algorithm as part of a standardized DRA assessment in older adults.

Automated post scoring: evaluating posts with topics and quoted posts in online forum

Online forumpost evaluationis an effective way for instructors to assess students' knowledge understanding and writing mechanics. Manually evaluating massive posts costs a lot of time. Automatically grading online posts could significantly alleviate instructors' burden. Similar text assessment tasks like Automated Text Scoring evaluate the writing quality of independent texts or relevance between text and prompt. And Automatic Short Answer Grading measures the semantic matching of short answers according to given problems and correct answers. Different from existing tasks, we propose a novel task, Automated Post Scoring (APS), which grades all online discussion posts in each thread of each student with given topics and quoted posts. APS evaluates not only the writing quality of posts automatically but also the relevance to topics. To measure the relevance, we model the semantic consistency between posts and topics. Supporting arguments are also extracted from quoted posts to enhance posts evaluation. Specifically, we propose a mixture model including a hierarchical text model to measure the writing quality, a semantic matching model to model topic relevance, and a semantic representation model to integrate quoted posts. We also construct a new dataset called Online Discussion Dataset containing 2,542 online posts from 694 students of a social science course. The proposed models are evaluated on the dataset with correlation and residual based evaluation metrics. Compared with measuring posts alone, experimental results demonstrate that incorporating topics and quoted posts could improve the performance of APS by a large margin, more than 9 percent on QWK.

Inter-rater reliability of the Australian Spasticity Assessment Scale in poststroke spasticity

To investigate the inter-rater reliability of the Australian Spasticity Assessment Scale (ASAS) in adult stroke patients with spasticity, two experienced clinicians rated the elbow flexor, wrist flexor, and ankle plantar flexor spasticity by using the ASAS in 85 persons with stroke. Unweighted and weighted (linear and quadratic) kappa statistics were used to calculate the inter-rater reliability for each muscle group. Unweighted kappa coefficients for elbow flexors (n = 83), wrist flexors (n = 80), and ankle plantar flexors (n = 77) were 0.67, 0.60, and 0.55, respectively. Linear and quadratic weighted kappa coefficients, respectively, were 0.77 and 0.87 for elbow flexors, 0.72 and 0.82 for wrist flexors, and 0.72 and 0.85 for ankle plantar flexors. The raters never disagreed by more than a single score in the rating of elbow flexors. On the contrary, the raters disagreed by more than a single score in three patients in the rating of ankle plantar flexors and in one patient in the rating of wrist flexors. The results suggested that inter-rater reliability of the ASAS differed according to the spastic muscle group assessed and the statistical method used. The strength of the agreement on the ASAS, an ordinal scale, ranged from good to very good when the weighted kappa values were considered.

Speech-Language Pathologists' Ratings of Speech Accuracy in Children With Speech Sound Disorders

PURPOSE

Forming accurate and consistent speech judgments can be challenging when working with children with speech sound disorders who produce a large number and varied types of error patterns. Rating scales offer a systematic approach to assessing the whole word rather than individual sounds. Thus, these scales can be an efficient way for speech-language pathologists (SLPs) to monitor treatment progress. This study evaluated the interrater reliability of an existing 3-point rating scale using a large group of SLPs as raters.

METHOD

Utilizing an online platform, 30 SLPs completed a brief training and then rated single words produced by children with typical speech patterns and children with speech sound disorders. Words were closely balanced across the three rating categories of the scale. The interrater reliability of the SLPs ratings to a consensus judgment was examined.

RESULTS

The majority of SLPs (87%) reached substantial interrater reliability to a consensus judgment using the 3-point rating scale. Correct productions had the highest interrater reliability. Productions with extensive errors had higher agreement than those with minor errors. Certain error types, such as vowel distortions, were especially challenging for SLPs to judge.

CONCLUSIONS

This study demonstrated substantial interrater reliability to a consensus judgment among a large majority of 30 SLPs using a 3-point rating. The clinical implications of the findings are discussed along with proposed modifications to the training procedure to guide future research.

Quality analysis for reliable complex multiclass neuroscience signal classification via electroencephalography

Purpose

The aim of this paper is to explore the brain–computer interface (BCI) as a methodology for generating awareness and increasing reliable use cases of the same so that an individual's quality of life can be enhanced via neuroscience and neural networks, and risk evaluation of certain experiments of BCI can be conducted in a proactive manner.

Design/methodology/approach

This paper puts forward an efficient approach for an existing BCI device, which can enhance the performance of an electroencephalography (EEG) signal classifier in a composite multiclass problem and investigates the effects of sampling rate on feature extraction and multiple channels on the accuracy of a complex multiclass EEG signal. A one-dimensional convolutional neural network architecture is used to further classify and improve the quality of the EEG signals, and other algorithms are applied to test their variability. The paper further also dwells upon the combination of internet of things multimedia technology to be integrated with a customized design BCI network based on a conventionally used system known as the message query telemetry transport.

Findings

At the end of our implementation stage, 98% accuracy was achieved in a binary classification problem of classifying digit and non-digit stimuli, and 36% accuracy was observed in the classification of signals resulting from stimuli of digits 0 to 9.

Originality/value

BCI, also known as the neural-control interface, is a device that helps a user reliably interact with a computer using only his/her brain activity, which is measured usually via EEG. An EEG machine is a quality device used for observing the neural activity and electric signals generated in certain parts of the human brain, which in turn can help us in studying the different core components of the human brain and how it functions to improve the quality of human life in general.

Construction and validation of an instrument for classifying nipple and areola complex lesions resulting from breastfeeding

OBJECTIVES

to construct and validate a classification of nipple and areola complex lesions resulting from breastfeeding, according to content and appearance.

METHODS

this is a methodological study, developed in four stages: operational definition, instrument construction, content and appearance validation. The Delphi technique and a Likert-type scale were used to validate content and appearance, through the participation of ten and sixteen judges, respectively. For analysis, Content Validity Index and Kappa Coefficient were applied.

RESULTS

Content Validity Index obtained an overall value of 0.93 and, for appearance, 0.94. Kappa values ranged between 0.46 and 1. The high rates of agreement among judges demonstrated the quality of the proposed content validity.

CONCLUSIONS

the Nipple and Areola Complex Lesions Classification Instrument developed obtained acceptable values of its indexes, proving to be valid in terms of content and appearance.

The utility of household Grocery Purchase Quality Index scores as an individual diet quality metric

The Grocery Purchase Quality Index (GPQI) reflects concordance between household grocery purchases and US dietary recommendations. However, it is unclear whether GPQI scores calculated from partial purchasing records reflect individual-level diet quality. This secondary analysis of a 9-month randomised controlled trial examined concordance between the GPQI (range 0-75, scaled to 100) calculated from 3 months of loyalty-card linked partial (≥50 %) household grocery purchasing data and individual-level Healthy Eating Index (HEI) scores at baseline and 3 months calculated from FFQ (n 209). Concordance was assessed with overall and demographic-stratified partially adjusted correlations; covariate-adjusted percentage score differences, cross-classification and weighted κ coefficients assessed concordance across GPQI tertiles (T). Participants were middle aged (55·4 (13·9) years), female (90·3 %), from non-smoking households (96·4 %) and without children (70·7 %). Mean GPQI (54·8 (9·1) %) scores were lower than HEI scores (baseline: 73·2 (9·1) %, 3 months: 72·4 (9·4) %) and moderately correlated (baseline r 0·41 v. 3 months r 0·31, P < 0·001). Correlations were stronger among participants with ≤ bachelor's degree, obesity and children. Scores showed moderate agreement (κ = 0·25); concordance was highest in T3. Participants with high (T3) v. low (T1) GPQI scores had 7·3-10·6 higher odds of having HEI scores >80 % at both time points. Household-level GPQI was moderately correlated with self-reported intake, indicating their promise for evaluating diet quality. Partial purchasing data appear to moderately reflect individual diet quality and may be useful in interventions monitoring changes in diet quality.

Development and initial psychometric validation of a patient-reported outcome measure for Crohn's perianal fistula: the Crohn's Anal Fistula Quality of Life (CAF-QoL) scale

INTRODUCTION

Crohn's perianal fistulas are challenging for patients and clinicians. Many do not respond to available treatments and despite recommendations by a global consensus, there are currently no specific patient-derived quality of life tools to measure response to treatment. We present a new validated patient-reported outcome measure (PROM) for this complicated disease phenotype.

METHODS

A draft questionnaire was generated using unstructured qualitative patient interviews on the experience of living with Crohn's perianal fistula, a nationwide multidisciplinary consensus exercise, a systematic review of outcomes assessing medical/surgical/combined treatment and a patient and public involvement day. Psychometric properties were assessed including construct validity (by comparison with the Hospital Anxiety and Depression Scale (HADS) and the UK Inflammatory Bowel Disease Questionnaire (UK-IBDQ)), and reliability and responsiveness was assessed by test-retest analysis.

RESULTS

Data from 211 patients contributed to development of a final 28-item questionnaire. The Crohn's Anal Fistula Quality of Life (CAF-QoL) demonstrated good internal consistency (Cronbach's alpha 0.88), excellent stability (intraclass correlation 0.98) and good responsiveness and construct validity, with positive correlation with the UK-IBDQ and HADS.

CONCLUSION

The CAF-QoL scale is ready for use as a PROM in research and clinical practice. It complements objective clinical evaluation of fistula by capturing impact on the patient.

Intelligence Beliefs Predict Spatial Performance in Virtual Environments and Graphical Creativity Performance

Although intelligence beliefs have been applied to explain the influence of cognition, behavior, and creativity, the research on creativity is still limited. Therefore, in order to effectively expand the understanding of the influence of intelligence beliefs on the creative performance of learners' graphics, the implicit theories of intelligence were exploited as the basis of this study. Three hypothetical pathways were proposed to be explored, and a research model was validated. First- and second-year students from a technical high school in New Taipei City were invited to participate. There were 273 valid data (88.9% of complete data). Reliability and validity analyses were performed, as well as overall model fit analysis and research model validation, and descriptive statistical analysis of the learners' performance in applying the operational virtual reality (VR). The results of this study showed that: (1) Incremental beliefs of aesthetic intelligence had a positive effect on spatial performance; (2) entity belief of spatial intelligence (EBSI) had a negative effect on spatial performance; and (3) spatial performance had a positive effect on graphical design performance. From the results, it is clear that design teachers can assess students' implicit beliefs in the early stages of teaching to actively promote better spatial performance when students show high levels of entity beliefs.

An enzyme immunoassay to determine human chorionic gonadotropin (HCG) in serum and urine samples using an ultra-microanalytical system

The determination of Human Chorionic Gonadotropin (HCG) in biological fluids is of great interest in the early pregnancy diagnostics, the evaluation of pregnancy disorders, as a tumor marker, as a screening procedure for anti-doping control, and many other purposes. A simple sandwich-type UltraMicro Enzyme-Linked ImmunoSorbent Assay (UMELISA) has been developed for the measurement of HCG in serum and urine samples. Strips coated with a high affinity MAb directed against HCG are used as solid phase, to ensure the specificity of the assay. The HCG assay was completed in 1.5 h, with a measuring range of 0.76-400 mIU/mL. The intra- and inter-assay coefficients of variation were lower than 10 %, depending on the HCG concentrations evaluated. Recovery percentages were 96.43-97.16 % (serum) and 98.10-99.04 % (urine). The assay detected intact HCG, nicked HCG, HCG β, and nicked HCG β, and did not recognize any of the interfering molecules tested. Regression analysis showed a good correlation with Elecsys in serum (n = 1459, r = 0.952, ρc = 0.948) and urine (n = 869, r = 0.988, ρc = 0.978). A good correlation was also found with 84 RIQAS samples analyzed with the kits Elecsys (r = 0.969, ρc = 0.957), Architect (r = 0.982, ρc = 0.970), Dimension (r = 0.989, ρc = 0.977), and Bioscience (r = 0.992, ρc = 0.980), all with a p < 0.01. Comparison with transvaginal ultrasonography in early pregnancy detection showed a specificity and a sensitivity of 100 % (n = 2385, κ = 1). The analytical performance characteristics of UMELISA HCG endorse its use for the quantification of HCG in serum and urine samples. This assay will make a cost-effective diagnostic kit accessible to low-income countries and is now available in the Cuban Public Health System.

The impact of the narrative mindset on effectivity in social problem solving

The narrative mindset is a tendency to interpret social information in the frame of stories. Two experiments were conducted to determine if and why the narrative mindset increases social problem-solving effectivity. The experiments consisted of two parts: the experimental manipulation (inducing the narrative mindset or control condition) and the observation of effects. In the second part, presented as a separate study, a participant was asked to advise other people facing interpersonal problems (experiment 1) or emotional problems (experiment 2). Three pairs of coders judged each piece of advice independently on three scales: Effectivity of the advice, empathy, and personalization (using their own experiences in providing the advice). The results indicate that the narrative mindset increases empathy, supported by the co-occurring increase in the problem's personalization, which leads to higher effectivity. The results reveal the positive real-life implications of structuring social information within a story frame. It may encourage the introduction of the narrative mindset effects into an area of social cognition research. Finally, the experiments show that the narrative mindset may be activated experimentally, providing an effective instrument to test the impact of narrative knowledge on social cognition.

Adaptation and Validation of the Dynamic Imaging Grade of Swallowing Toxicity for Flexible Endoscopic Evaluation of Swallowing: DIGEST-FEES

Purpose While flexible endoscopic evaluation of swallowing (FEES) is a common clinical procedure used in the head and neck cancer (HNC) population, extant outcome measures for FEES such as bolus-level penetration-aspiration and residue scores are not well suited as global patient-level endpoint measures of dysphagia severity in cooperative group trials or clinical outcomes research. The Dynamic Imaging Grade of Swallowing Toxicity (DIGEST) was initially developed and validated for use during videofluoroscopic evaluations as a way to grade safety, efficiency, and overall pharyngeal swallowing impairment. The purpose of this study was to adapt and validate DIGEST for use with FEES. Method A modified Delphi exercise was conducted for content validation, expert consensus, adaptation, and operationalization of DIGEST-FEES. Three blinded, expert raters then evaluated 100 de-identified post-HNC treatment FEES examinations. Intra- and interrater reliability were tested with quadratic weighted kappa. Criterion validity against the MD Anderson Dysphagia Inventory, Functional Oral Intake Scale, Secretion Severity Scale, and Yale Residue Rating Scale was assessed with Spearman correlation coefficients. Results Interrater reliability was almost perfect for overall DIGEST-FEES grade (κ_w = 0.83) and safety grade (κ_w = 0.86) and substantial for efficiency grade (κ_w = 0.74). Intrarater reliability was excellent for all raters (0.9-0.91). Overall DIGEST-FEES grade correlated with MD Anderson Dysphagia Inventory (r = -.43, p < .0001), Functional Oral Intake Scale (r = -.43, p < .0001), Secretion Severity Scale (r = .47, p < .0001), Yale Vallecular Residue (r = .73, p < .0001), and Yale Pyriform Sinus Residue (r = .65, p < .0001). Conclusion DIGEST-FEES is a valid and reliable scale to describe the severity of pharyngeal dysphagia in patients with HNC. Supplemental Material https://doi.org/10.23641/asha.14642787.

National guidelines on nutrient reference values for the healthy adult population and for pregnant or lactating women are based on heterogeneous sources of evidence: review of guidelines

Many countries provide dietary guidelines for health practitioners and/or the general population. However, there is no general, international guideline serving as a template for national dietary guidelines, and there is little to no consensus regarding reference values for different nutrients. The present review compared 27 national dietary guidelines for healthy adults as well as for pregnant and/or breastfeeding women, and analysed their quality and the evidence behind their recommendations. The guidelines were evaluated for their quality using the instrument Agree II, and found to be heterogeneous (overall quality score 14%-100%) and often insufficient (quality score < 50%) due to missing information about their methodology and sources of evidence. We analysed the evidence (number of studies, study types and publication years) of reference values of a number of nutrients using the five guidelines that provided the highest scores in the Agree II assessment. The reference values varied among guidelines, were rarely based on up-to-date meta-analyses, and were often based on insufficiently reported evidence (22/27 guidelines with quality score < 50%). We recommend systematic reviews of high quality studies to formulate future guidelines, and to use guidelines on how to write guidelines.

Comparison of metrics for the evaluation of medical segmentations using prostate MRI dataset

Nine previously proposed segmentation evaluation metrics, targeting medical relevance, accounting for holes, and added regions or differentiating over- and under-segmentation, were compared with 24 traditional metrics to identify those which better capture the requirements for clinical segmentation evaluation. Evaluation was first performed using 2D synthetic shapes to highlight features and pitfalls of the metrics with known ground truths (GTs) and machine segmentations (MSs). Clinical evaluation was then performed using publicly-available prostate images of 20 subjects with MSs generated by 3 different deep learning networks (DenseVNet, HighRes3DNet, and ScaleNet) and GTs drawn by 2 readers. The same readers also performed the 2D visual assessment of the MSs using a dual negative-positive grading of -5 to 5 to reflect over- and under-estimation. Nine metrics that correlated well with visual assessment were selected for further evaluation using 3 different network ranking methods - based on a single metric, normalizing the metric using 2 GTs, and ranking the network based on a metric then averaging, including leave-one-out evaluation. These metrics yielded consistent ranking with HighRes3DNet ranked first then DenseVNet and ScaleNet using all ranking methods. Relative volume difference yielded the best positivity-agreement and correlation with dual visual assessment, and thus is better for providing over- and under-estimation. Interclass Correlation yielded the strongest correlation with the absolute visual assessment (0-5). Symmetric-boundary dice consistently yielded good discrimination of the networks for all three ranking methods with relatively small variations within network. Good rank discrimination may be an additional metric feature required for better network performance evaluation.

Chinesisation, adaptation and validation of the Chelsea Critical Care Physical Assessment Tool in critically ill patients: a cross-sectional observational study

PURPOSE

To translate and adapt the Chelsea Critical Care Physical Assessment Tool (CPAx) into Chinese version ('CPAx-Chi'), test the reliability and validity of CPAx-Chi, and verify the cut-off point for the diagnosis of intensive care unit-acquired weakness (ICU-AW).

STUDY DESIGN

Cross-sectional observational study.

METHODS

Forward and back translation, cross-cultural adaptation and pretesting of CPAx into CPAx-Chi were based on the Brislin model. Participants were recruited from the general ICU of five third-grade class-A hospitals in western China. Two hundred critically ill adult patients (median age: 53 years; 64% men) with duration of ICU stay ≥48 hours and Glasgow Coma Scale ≥11 were included in this study. Two researchers simultaneously and independently assessed eligible patients using the Medical Research Council Muscle Score (MRC-Score) and CPAx-Chi.

RESULTS

The content validity index of items was 0.889. The content validity index of scale was 0.955. Taking the MRC-Score scale as standard, the criterion validity of CPAx-Chi was r=0.758 (p<0.001) for researcher A, and r=0.65 (p<0.001) for researcher B. Cronbach's α was 0.939. The inter-rater reliability was 0.902 (p<0.001). The area under the receiver operating characteristic curves of CPAx-Chi for diagnosing ICU-AW based on MRC-Score ≤48 were 0.899 (95% CI 0.862 to 1.025) and 0.874 (95% CI 0.824 to 0.925) for researcher B. The best cut-off point for CPAx-Chi for the diagnosis of ICU-AW was 31.5. The sensitivity was 87% and specificity was 77% for researcher A, whereas it was 0.621, 31.5, 75% and 87% for researcher B, respectively. The consistency was high when taking CPAx-Chi ≤31 and MRC-Score ≤48 as the cut-off points for the diagnosis of ICU-AW. Cohen's kappa=0.845 (p=0.02) in researcher A and 0.839 (p=0.04) for researcher B.

CONCLUSIONS

CPAx-Chi demonstrated content validity, criterion-related validity and reliability. CPAx-Chi showed the best accuracy in assessment of patients at risk of ICU-AW with good sensitivity and specificity at a recommended cut-off of 31.

Developmental Differences Between the Limbic and Neocortical Telencephalic Wall: An Intrasubject Slice-Matched 3 T MRI-Histological Correlative Study in Humans

The purpose of the study was to investigate the interrelation of the signal intensities and thicknesses of the transient developmental zones in the cingulate and neocortical telencephalic wall, using T2-weighted 3 T-magnetic resonance imaging (MRI) and histological scans from the same brain hemisphere. The study encompassed 24 postmortem fetal brains (15-35 postconceptional weeks, PCW). The measurements were performed using Fiji and NDP.view2. We found that T2w MR signal-intensity curves show a specific regional and developmental stage profile already at 15 PCW. The MRI-histological correlation reveals that the subventricular-intermediate zone (SVZ-IZ) contributes the most to the regional differences in the MRI-profile and zone thicknesses, growing by a factor of 2.01 in the cingulate, and 1.78 in the neocortical wall. The interrelations of zone or wall thicknesses, obtained by both methods, disclose a different rate and extent of shrinkage per region (highest in neocortical subplate and SVZ-IZ) and stage (highest in the early second half of fetal development), distorting the zones' proportion in histological sections. This intrasubject, slice-matched, 3 T correlative MRI-histological study provides important information about regional development of the cortical wall, critical for the design of MRI criteria for prenatal brain monitoring and early detection of cortical or other brain pathologies in human fetuses.

Validation of the telephone-based assessment of locomotion ability after stroke

The aim of the study was to validate the telephone-based application of the ABILOCO questionnaire for the assessment of locomotion ability after stroke. Individuals after stroke answered the ABILOCO-Brazil questionnaire on two randomized occasions, face-to-face and by telephone, 5-7 days apart. The mean difference between the interviews was reported. Intraclass correlation coefficient (ICC) was calculated to investigate the agreement between the total scores, and weighted-Kappa statistics to investigate the agreement between the individual items. A total of 92 individuals were included. There was no significant difference in the mean scores between face-to-face and telephone-based applications of the ABILOCO [mean difference 0.17 logits; 95% confidence interval (CI), -0.88 to 0.54]. A high level of agreement was found between the ABILOCO total scores obtained between both applications (ICC = 0.90; 95% CI, 0.84-0.93), and most of the individual items had, on average, moderate agreement. The ABILOCO showed to be a reliable questionnaire for telephone assessment of locomotion ability after stroke.

Diagnostic Performance of GeneChip for the Rapid Detection of Drug-Resistant Tuberculosis in Different Subgroups of Patients

Objective

Drug-resistant tuberculosis (DR-TB) is a growing problem worldwide. The rapid drug susceptibility test (DST) of DR-TB enables the timely administration of a chemotherapy regimen that effectively treats DR-TB. GeneChip has been reported as a novel molecular diagnostic tool for rapid diagnosis but has limited data on the performance of subgroup patients with DR-TB. This study aims to assess the diagnostic value of GeneChip in patients with different sexes, ages, treatment histories, treatment outcomes, and places of residence.

Methods

We recruited newly registered sputum smear-positive pulmonary TB patients from January 2011 to September 2020 in Lianyungang City, Jiangsu Province, China. We applied both GeneChip and DST to measure drug resistance to rifampin (RIF) and isoniazid (INH). The kappa value, sensitivity, specificity, and agreement rate (AR) were calculated. We also applied a Classification and Regression Tree to explore factors related to the performance of GeneChip.

Results

We observed that sex, age, treatment history, treatment outcomes, and drug resistance type were significantly associated with the performance of GeneChip. For RIF resistance, there was significant accordance in young patients (kappa: 0.79) and cases with the treatment failure outcome (kappa: 0.92). For multidrug resistance (MDR), there was significant accordance in young cases (kappa: 0.77). Compared with previously treated patients, the newly treated patients had a significantly higher AR in detecting RIF resistance (0.97 vs 0.92), INH resistance (0.95 vs 0.89), and MDR (0.98 vs 0.92). The overall sensitivity, specificity, AR and kappa value for the diagnosis of MDR-TB were 0.70 (95% CI: 0.63–0.70), 0.99 (95% CI: 0.98–0.99), 0.98 (95% CI: 0.97–0.98), and 0.72 (95% CI: 0.67–0.78), respectively.

Conclusion

We observed a high concordance between GeneChip and DST among TB patients with different characteristics, indicating that GeneChip can be a potential alternative tool for rapid MDR-TB detection.

Functional Correlations in the Pursuit of Performance Assessment of Classifiers

In statistical classification and machine learning, as well as in social and other sciences, a number of measures of association have been proposed for assessing and comparing individual classifiers, raters, as well as their groups. In this paper, we introduce, justify, and explore several new measures of association, which we call CO-, ANTI-, and COANTI-correlation coefficients, that we demonstrate to be powerful tools for classifying confusion matrices. We illustrate the performance of these new coefficients using a number of examples, from which we also conclude that the coefficients are new objects in the sense that they differ from those already in the literature.

Transforming self-reported outcomes from a stroke register to the modified Rankin Scale: a cross-sectional, explorative study

The aim was to create an algorithm to transform self-reported outcomes from a stroke register to the modified Rankin Scale (mRS). Two stroke registers were used: the Väststroke, a local register in Gothenburg, Sweden, and the Riksstroke, a Swedish national register. The reference variable, mRS (from Väststroke), was mapped with seven self-reported questions from Riksstroke. The transformation algorithm was created as a result of manual mapping performed by healthcare professionals. A supervised machine learning method—decision tree—was used to further evaluate the transformation algorithm. Of 1145 patients, 54% were male, the mean age was 71 y. The mRS grades 0, 1 and 2 could not be distinguished as a result of manual mapping or by using the decision tree analysis. Thus, these grades were merged. With manual mapping, 78% of the patients were correctly classified, and the level of agreement was almost perfect, weighted Kappa (K_w) was 0.81. With the decision tree, 80% of the patients were correctly classified, and substantial agreement was achieved, K_w = 0.67. The self-reported outcomes from a stroke register can be transformed to the mRS. A mRS algorithm based on manual mapping might be useful for researchers using self-reported questionnaire data.

Association between external and internal lymphedema and chronic dysphagia following head and neck cancer treatment

BACKGROUND

To examine the relationship between chronic external and internal head and neck lymphedema (HNL) and swallowing function in patients following head and neck cancer (HNC) treatment.

METHODS

Seventy-nine participants, 1-3 years post treatment were assessed for external HNL using the MD Anderson Cancer Centre Lymphedema Rating Scale, and internal HNL using Patterson's Radiotherapy Edema Rating Scale. Swallowing was assessed via instrumental, clinical and patient-reported outcome measures.

RESULTS

HNL presented as internal only (68%), combined external/internal (29%), and external only (1%). Laryngeal penetration/aspiration was confirmed in 20%. Stepwise multivariable regression models, that accounted for primary site, revealed that a higher severity of external HNL and internal HNL was associated with more severe penetration/aspiration (P < .004 and P = .006, respectively), diet modification (P < .001 both), and poorer patient-reported outcomes (P = .037 and P = .014, respectively).

CONCLUSION

Increased swallowing issues can be expected in patients presenting with more severe external HNL and/or internal HNL following HNC treatment.

Kappa coefficients for dichotomous-nominal classifications

Two types of nominal classifications are distinguished, namely regular nominal classifications and dichotomous-nominal classifications. The first type does not include an ‘absence’ category (for example, no disorder), whereas the second type does include an ‘absence’ category. Cohen’s unweighted kappa can be used to quantify agreement between two regular nominal classifications with the same categories, but there are no coefficients for assessing agreement between two dichotomous-nominal classifications. Kappa coefficients for dichotomous-nominal classifications with identical categories are defined. All coefficients proposed belong to a one-parameter family. It is studied how the coefficients for dichotomous-nominal classifications are related and if the values of the coefficients depend on the number of categories. It turns out that the values of the new kappa coefficients can be strictly ordered in precisely two ways. The orderings suggest that the new coefficients are measuring the same thing, but to a different extent. If one accepts the use of magnitude guidelines, it is recommended to use stricter criteria for the new coefficients that tend to produce higher values.

Anterior Chamber Angle Evaluation Using Gonioscopy: Consistency and Agreement between Optometrists and Ophthalmologists

SIGNIFICANCE

In our intermediate-tier glaucoma care clinic, we demonstrate fair to moderate agreement in gonioscopy examination between optometrists and ophthalmologists, but excellent agreement when considering open versus closed angles. We highlight the need for increased consistency in the evaluation and recording of angle status using gonioscopy.

PURPOSE

The consistency of gonioscopy results obtained by different clinicians is not known but is important in moving toward practice modalities such as telemedicine and collaborative care clinics. The purpose of this study was to evaluate the description and concordance of gonioscopy results among different practitioners.

METHODS

The medical records of 101 patients seen within a collaborative care glaucoma clinic who had undergone gonioscopic assessment by two clinicians (one optometrist and either one general ophthalmologist [n = 50] or one glaucoma specialist [n = 51]) were reviewed. The gonioscopy records were evaluated for their descriptions of deepest structure seen, trabecular pigmentation, iris configuration, and other features. These were compared between clinicians (optometrist vs. ophthalmologist) and against the final diagnosis.

RESULTS

Overall, 51.9 and 59.8% of angles were graded identically in terms of deepest visible structure when comparing between optometrist versus general ophthalmologist and optometrist versus glaucoma specialist, respectively. The concordance increased when considering ±1 of the grade (67.4 and 78.5%, respectively), and agreement with the final diagnosis was high (>90%). Variations in angle grading other than naming structures were observed (2.0, 30, and 3.9% for optometrist, general ophthalmologist, and glaucoma specialist, respectively). Most of the time, trabecular pigmentation or iris configuration was not described.

CONCLUSIONS

Fair to moderate concordance in gonioscopy was achieved between optometrists and ophthalmologists in a collaborative care clinic in which there is consistent feedback and clinical review. To move toward unified medical records and a telemedicine model, improved consistency of record keeping and angle description is required.