Appropriate Statistics for Determining Chance-Removed Interpractitioner Agreement

Interrater Agreement of BT-RADS for Evaluation of Follow-up MRI in Patients with Treated Primary Brain Tumor

BACKGROUND AND PURPOSE

The Brain Tumor Reporting and Data System (BT-RADS) is a structured radiology reporting algorithm that was introduced to provide uniformity in posttreatment primary brain tumor follow-up and reporting, but its interrater reliability (IRR) assessment has not been widely studied. Our goal is to evaluate the IRR among neuroradiologists and radiology residents in the use of BT-RADS.

MATERIALS AND METHODS

This retrospective study reviewed 103 consecutive MR studies in 98 adult patients previously diagnosed with and treated for primary brain tumor (January 2019 to February 2019). Six readers with varied experience (4 neuroradiologists and 2 radiology residents) independently evaluated each case and assigned a BT-RADS score. Readers were blinded to the original score reports and the reports from other readers. Cases in which at least 1 neuroradiologist scored differently were subjected to consensus scoring. After the study, a post hoc reference score was also assigned by 2 readers by using future imaging and clinical information previously unavailable to readers. The interrater reliabilities were assessed by using the Gwet AC2 index with ordinal weights and percent agreement.

RESULTS

Of the 98 patients evaluated (median age, 53 years; interquartile range, 41-66 years), 53% were men. The most common tumor type was astrocytoma (77%) of which 56% were grade 4 glioblastoma. Gwet index for interrater reliability among all 6 readers was 0.83 (95% CI: 0.78-0.87). The Gwet index for the neuroradiologists' group (0.84 [95% CI: 0.79-0.89]) was not statistically different from that for the residents' group (0.79 [95% CI: 0.72-0.86]) (χ2 = 0.85; P = .36). All 4 neuroradiologists agreed on the same BT-RADS score in 57 of the 103 studies, 3 neuroradiologists agreed in 21 of the 103 studies, and 2 neuroradiologists agreed in 21 of the 103 studies. Percent agreement between neuroradiologist blinded scores and post hoc reference scores ranged from 41%-52%.

CONCLUSIONS

A very good interrater agreement was found when tumor reports were interpreted by independent blinded readers by using BT-RADS criteria. Further study is needed to determine if this high overall agreement can translate into greater consistency in clinical care.

Low Inter-Rater Reliability and Reproducibility of Neck Reflex/"Adler-Langer" Points in Neural Therapy Diagnostics but Increased Pressure Pain Threshold after Therapy: Results of a Randomized Controlled Observer-Blind Trial

BACKGROUND

Neck reflex points or Adler-Langer points are commonly used in neural therapy to detect so-called interference fields. Chronic irritations or inflammations in the sinuses, teeth, tonsils, or ears are supposed to induce tension and tenderness of the soft tissues and short muscles in the upper cervical spine. The individual treatment strategy is based on the results of diagnostic Adler-Langer point palpation. This study investigated the inter- and intra-rater reliability and explored treatment effects.

METHODS

We performed a randomized controlled trial with 104 inpatients (80.8% female, 51.8 ± 12.74 years) of a German department for internal and integrative medicine. Patients were randomized to individual neural therapy according to the pathological findings (n = 48) or no treatment (n = 56). In each patient, three experienced raters (20-45 years of experience in neural therapy) and two novice raters (medical students) rated Adler-Langer points rigidity on a standardized rating scale ("strong," "weak," "none"). The patients independently evaluated the tenderness on palpation of the eight points using the same scale. Pressure pain thresholds were assessed at the eight Adler-Langer points. All patients were retested after 30 min. The five raters were blinded to treatment allocation and assessments of the other raters. Video recordings were obtained to assess the consistency of the areas tested by the different raters.

RESULTS

Agreement between patients and raters (Cohen's kappa = 0.161-0.400) and inter-rater reliability were low (Fleiss kappa = 0.132-0.150). Moreover, the individual agreement (pre-post comparisons in untreated patients) was similarly low even in experienced raters (Cohen's kappa = 0.099-0.173). Video documentation suggests that raters do not place their fingers in the correct segments (percentage of correct position: 42.0-60.6%). Pressure pain thresholds at five of the eight Adler-Langer points showed significant changes after treatment compared to none in the control group.

CONCLUSION

Under this artificial experimental setting, this method of Adler-Langer point palpation has not proven to be a reliable diagnostic tool. But it could be shown that, as claimed by the method, the tenderness in five of eight Adler-Langer points decreased after neural therapy.

Exploring the level of agreement among different drug-drug interaction checkers: a comparative study on direct oral anticoagulants

BACKGROUND

Direct oral anticoagulants (DOACs) may be involved in drug-drug interactions (DDIs) potentially increasing the risk of adverse drug reactions. This study aimed to evaluate the level of agreement among interaction checkers (ICs) and DOACs' summary of product characteristics (SPCs), in listing DDIs and in attributing DDIs' severity.

RESEARCH DESIGN AND METHODS

The level of agreement among five ICs (i.e. INTERCheck WEB, Micromedex, Lexicomp, Epocrates, and drugs.com) in identifying potential DDIs and in attributing severity categories was evaluated using Gwet's AC1 on all five ICs and by comparing groups of four- and two-pair sets of ICs.

RESULTS

A total of 486 potentially interacting drugs with dabigatran, 556 for apixaban, 444 for edoxaban, and 561 for rivaroxaban were reported. The level of agreement among the ICs in identifying potential DDIs was poor (range: 0.12-0.16). Similarly, it was low in 4 and 2 sets analyses. The level of agreement among the ICs in classifying the severity of potential DDIs was poor (range: 0.32-0.34), also in 4 and 2 sets analyses.

CONCLUSIONS

The heterogeneity among different ICs and SPCs underscores the need to standardize DDI datasets and to conduct real-world studies to generate evidence regarding the frequency and clinical relevance of potential DOAC-related DDIs.

Assessing food availability and healthier options in an urban Chinese university: a case study using the Chinese Nutrition Environment Measurement Survey for Stores (C-NEMS-S)

Young adults (18-24 years) in universities are frequently exposed to an environment that promotes unhealthy eating behaviors. Using a validated tool, the Chinese Nutrition Environment Measurement Survey for Stores (C-NEMS-S), we assess the food availability and healthier options in a large, urban Chinese university. We employed C-NEMS-S for scoring criteria and weighting. A total of 52 on-campus canteen outlets were audited in an urban university located in Shijiazhuang City, China. General food outlets (n 43) and self-served food outlets (n 7) were further categorized into eight subtypes. Beverage outlets (n 2) were discussed separately from food outlets. C-NEMS-S scores were significantly different across food outlet types (P = 0.0024), especially between noodle and rice outlets (P = 0.0415). Food availability scores for starchy tubers (P < 0.001), dry beans (P < 0.001), vegetables (P = 0.0225), and fruits (P < 0.001) were significantly different across food outlet subtypes. Healthier options were scarce and only appeared in "grains" (n 2) and "meat and poultry" (n 2) categories. Further research on improving the accustomed audit tool and assessing university student diet quality is warranted.

Impact of high-density EEG in presurgical evaluation for refractory epilepsy patients

OBJECTIVE

Electrical source localization (ESI) can help to identify the seizure onset zone or propagation zone, but it is unclear how dipole localization techniques influence surgical planning.

METHODS

Patients who received a high density (HD)-EEG from 7/2014-7/2019 at Stanford were included if they met the following inclusion criteria: (1) adequate epileptiform discharges were recorded for source localization analysis, (2) underwent surgical treatment, which was at least 6 months before the survey. Interictal ESI was performed with the LORETA method on age matched MRIs. Six neurophysiologists from the Stanford Epilepsy Program independently reviewed each case through an HIPPA-protected online survey. The same cases were presented again with additional data from the HD-EEG study. Ratings of how much the HD-EEG findings added value and in what way were recorded.

RESULTS

Fifty out of 202 patients met the inclusion criteria, providing a total of 276 h of HDEEG recordings. All patients had video EEG recordings and at least one brain MRI, 88 % had neuropsychological testing, 78 % had either a PET or SPECT scan. Additional HD-EEG information was rated as helpful in 83.8 %, not useful in 14.4 % and misleading in 1.8 % of cases. In 20.4 % of cases the HD-EEG information altered decision-making in a major way, such as choosing a different surgical procedure, avoidance of invasive recording or suggesting placement of invasive electrodes in a lobe not previously planned. In 21.5 % of cases, HD-EEG changed the plan in a minor way, e.g., extra invasive electrodes near the previously planned sites in the same sub-lobar region. In 42.3 % cases, HD-EEG did not change their plan but provided confirmation. In cases with normal MRI, additional HD-EEG information was more likely to change physicians' decision making during presurgical process when compared to the cases with MRI-visible lesions (53.3 % vs. 34.3 %, p = 0.002). Among patients achieving Engel class I/II outcome, the concordance rate of HD-EEG and resection zone was 64.7 % versus 35.3 % with class III/IV (p = 0.028).

CONCLUSION

HD-EEG assists presurgical planning for refractory epilepsy patients, with a higher yield in patients with non-lesional MRIs. Concordance of HD-EEG dipole analysis localization and resection site is a favorable outcome indicator.

Inter-Rater Reliability of Kampo Diagnosis for Chronic Diseases

This single-center observational study aimed to assess the inter-rater reliability (IRR) of Kampo medicine pattern diagnosis, which is modularized into three modules for chronic diseases, using 64 participants' information documents. The linearly weighted percentage of agreement and Gwet's agreement coefficient (AC) 2 for the deficiency–excess module, among three specialists, were 85.9% and 0.708, respectively. The unweighted percentage of agreement and Gwet's AC1 were 64.6% and 0.542 for the cold–heat, and 35.9% and 0.254 for the qi–blood–fluid modules, respectively. Our findings suggest that our modularization method may improve the IRR of Kampo medicine pattern diagnosis.

Autobiographical internal dialogue on TQM

Purpose

To present the internal dialogue of a TQM practitioner using the conceptual lens of reflective practice.

Design/methodology/approach

This study used a pragmatic philosophical approach to collect qualitative critical reflection data and quantitative career construction data from stratified and purposively sampled respondents, using structured questionnaires. The qualitative data were analyzed through reconstruction, while the quantitative data were analyzed through co-construction using the percentage agreement value and Wilcoxon–Mann Whitney test.

Findings

Five questions were found to be valuable for steering the internal dialogue for critical reflection, thus recommended as a must-have in a TQM practitioner's toolkit. This study found the career adapt–abilities scale to be a valuable tool for assessing the career construction of a TQM practitioner. This was supported by a 64% agreement and non-significant difference between the two groups of raters used, p < 0.05 (U = 3356.5, W = 7451.5, Z = 1.9826), two-tailed.

Research limitations/implications

The pragmatic philosophical stance used in this study lends it to a certain level of subjectivity. However, the inputs from the three other participants neutralize the subjectivity. Most notably, this study is not about consensus-seeking but rather verifiable/testable self-reflection.

Practical implications

The theory-informed results presented in this study are useful for the continuing professional development of TQM practitioners.

Originality/value

This study provides insights for applying an individual-level self-assessment tool for TQM implementation.

A Novel Approach to Describing Traditional Chinese Medical Patterns: The "Traditional Chinese Medical Diagnostic Descriptor"

Objectives: In the first of a series of three articles by the present authors, diagnostic agreement between Traditional Chinese Medicine (TCM) practitioners was found to be low. This was the first time that TCM diagnoses had been evaluated with an open population of patients and this result is a cause of concern. In the second article, incorrect statistics were shown to have often been used to calculate chance-removed inter-rater agreement, and appropriate statistics such as Gwet's Agreement Coefficient 2 (AC2) was recommended for future studies. In this, the third article, a novel approach to recording TCM diagnostic patterns, the Traditional Chinese Medical Diagnostic Descriptor (TCMDD), is presented that allows chance-removed agreement calculation. An example of mapping TCM diagnostic patterns to the TCMDD format is given and diagnostic agreement is evaluated. Design, Settings, Subjects: The same 35 subjects used to report agreement in our first article were also diagnosed by additional practitioners using the TCMDD format during the same experimental sessions at the University of Technology, Sydney Clinic. TCM diagnoses from the first article were also mapped to the TCMDD format. Outcome measures: Linearly weighted simple agreement and the AC2 statistic were utilized and all results compared. Results: Linearly weighted simple agreement using the TCMDD and TCM mapped to TCMDD format averaged 0.80 ± 0.02 compared with 0.19 for TCM. TCMDD and TCM mapped to TCMDD chance-removed agreement, as calculated with AC2, ranged between 0.67 and 0.73 ± 0.03. Conclusions: The TCMDD allows the essence of diagnoses expressed by TCM practitioners to be appropriately compared. This was confirmed by the TCM mapped to TCMDD results. In both cases, simple agreement was significantly greater than that obtained with the TCM format. Chance-removed statistics and error estimates can be reliably calculated with the AC2 and the TCMDD in open populations.