Nonparametric Limits of Agreement in Method Comparison Studies: A Simulation Study on Extreme Quantile Estimation

Measurement properties of Brief-BESTest scores from children, adolescents, and youth with visual impairments

Purpose: This study aimed to vet the measurement properties of Brief-BESTest scores in children, adolescents, and youth with visual impairment and blindness (YVI). Methods: A cross-sectional sample of YVI (N = 101) completed the Brief-BESTest, a modified version of the Y-Balance Test, the 360-degree turn test, bipedal quiet stance, and Activity-Specific Balance Confidence Scales. Thirty-seven YVI completed test-retest of the Brief-BESTest using a one-year interval. Using classical test theory, various forms of total and item-level Brief-BESTest score reliability and validity were investigated in YVI. Results: All inter-rater reliability coefficients were ≥ .80. When considering the eight items of the Brief-BESTest, 27 of the 28 possible correlations were statistically significant (p<.05). Various internal consistency and item difficulty results were strong. When taking total Brief-BESTest scores and their association with the complementary balance tasks/metrics into account, 11 of 13 associations were statistically significant (p<.05) providing strong convergent validity evidence. Being multimorbid and degree of vision significantly predicted total Brief-BESTest scores (p<.001) suggesting construct (i.e. known groups) validity. Numerous test-retest results (e.g. coefficients, limits of agreement) following the one-year interval were indicative of score stability. Conclusion: Practitioners and researchers should have confidence in, and consider adopting, the Brief-BESTest to examine multidimensional balance in YVI.

Comparing Device-Generated and Calculated Bioimpedance Variables in Community-Dwelling Older Adults

Despite BIA emerging as a clinical tool for assessing older adults, it remains unclear how to calculate whole-body impedance (Z), reactance (Xc), resistance (R), and phase angle (PhA) from segmental values using modern BIA devices that place electrodes on both sides of the body. This investigation aimed to compare both the whole-body and segmental device-generated phase angle (PhADG) with the phase angle calculated using summed Z, Xc, and R from the left, right, and combined sides of the body (PhACalc) and to compare bioelectric variables between sides of the body. A sample of 103 community-dwelling older adults was assessed using a 50 kHz direct segmental multifrequency BIA device. Whole-body PhACalc values were assessed for agreement with PhADG using 2.5th and 97.5th quantile nonparametric limits of agreement and Spearman's rho. Bioelectrical values between sides of the body were compared using Wilcoxon rank and Spearman's rho. A smaller mean difference was observed between PhADG and right PhACalc (-0.004°, p = 0.26) than between PhACalc on the left (0.107°, p = 0.01) and on the combined sides (0.107°, p < 0.001). The sum of Z, R, and PhACalc was significantly different (p < 0.01) between the left (559.66 ± 99.55 Ω, 556.80 ± 99.52 Ω, 5.51 ± 1.5°, respectively) and the right sides (554.60 ± 94.52 Ω, 552.02 ± 94.23 Ω, 5.41 ± 0.8°, respectively). Bilateral BIA values do not appear to be interchangeable when determining whole-body measurements. Present data suggest that using right-sided segmental values would be the most appropriate choice for calculating whole-body bioelectrical variables.

Evaluating the Usability and Equivalence of Electronic Patient-Reported Outcome Measures for Individuals with a Lower-Limb Amputation

Introduction

Electronic versions of patient-reported outcome measures (PROMs) seem to have a clear administrative logging advantage to traditional paper versions. However, most of them have not been formally evaluated for their suitability to replace paper outcome measures for assessment of individuals with lower-limb amputations. The aim of this study is to examine the usability and equivalence of electronic to paper versions of PROMs suitable for use in prosthetic clinical care and research for persons with lower-limb loss.

Methods

In this cross-sectional study, 10 participants remotely completed the following PROMs online and then on paper: Orthotic and Prosthetic User Survey (OPUS), Modified Falls Efficacy Scale (MFES), Prosthetic Evaluation Questionnaire (PEQ), Patient Health Questionnaire–9 (PHQ-9), and Community Participation Indicators (CPI). Participants also answered open-ended and standardized questions regarding the usability of the electronic surveys. Wilcoxon signed rank tests, comparisons to minimum detectable change, intraclass correlation coefficients, and Bland-Altman plots were used to evaluate differences between the two survey versions, meaningful changes in scores, reliability, and systematic biases, respectively.

Results

Electronic surveys had fewer missing or ambiguous responses than paper surveys; however, the PEQ Social Burdens subscale could not be evaluated due to error in the creation of the electronic survey. No significant differences were found between scores of the two versions for any of the measures, but multiple participants had meaningful changes in the Appearance and Sounds PEQ subscales. All measures demonstrated acceptable reliability between versions, except the Appearance, Perceived Response, and Sounds subscales of the PEQ. No systematic biases in scores or usability concerns were found for any measures.

Conclusions

This study analysis showed that most of the electronic PROMs studied are easily used and demonstrate equivalence to the paper versions. However, the PEQ Appearance, Perceived Response, Sounds, and Social Burden subscales require further evaluation.

Clinical Relevance

Except for the PEQ, electronic versions of the PROMs in this study can likely be used interchangeably with paper versions among individuals with lower-limb loss.

Accuracy and Consistency of Confidence Limits for Monosyllable Identification Scores Derived Using Simulation, the Harrell-Davis Estimator, and Nonlinear Quantile Regression

BACKGROUND

Audiological diagnosis and rehabilitation often involve the assessment of whether the maximum speech identification score (PBmax) is poorer than expected from the pure-tone average (PTA) threshold. This requires the estimation of the lower boundary of the PBmax values expected for a given PTA (one-tailed 95% confidence limit, CL). This study compares the accuracy and consistency of three methods for estimating the 95% CL.

METHOD

The 95% CL values were estimated using a simulation method, the Harrell-Davis (HD) estimator, and non-linear quantile regression (nQR); the latter two are both distribution-free methods. The first two methods require the formation of sub-groups with different PTAs. Accuracy and consistency in the estimation of the 95% CL were assessed by applying each method to many random samples of 50% of the available data and using the fitted parameters to predict the data for the remaining 50%.

STUDY SAMPLE

A total of 642 participants aged 17 to 84 years with sensorineural hearing loss were recruited from audiology clinics. Pure-tone audiograms were obtained and PBmax scores were measured using monosyllables at 40 dB above the speech recognition threshold or at the most comfortable level.

RESULTS

For the simulation method, 6.7 to 8.2% of the PBmax values fell below the 95% CL for both ears, exceeding the target value of 5%. For the HD and nQR methods, the PBmax values fell below the estimated 95% CL for approximately 5% of the ears, indicating good accuracy. Consistency, estimated from the standard deviation of the deviations from the target value of 5%, was similar for all the methods.

CONCLUSIONS

The nQR method is recommended because it has good accuracy and consistency, and it does not require the formation of arbitrary PTA sub-groups.

Cadiax Compact 2 and MODJAW comparative analysis of condylar inclination: Innovative digital approaches in dentistry

STATEMENT OF PROBLEM

Artificial intelligence has been used to enhance the digitalized workflow, especially when undergoing complex oral rehabilitations. However, the reliability of real-time jaw motion registration devices is unclear, and no standard measurement method of the sagittal condylar inclination (SCI) and Bennett angle (BA) has been established.

PURPOSE

The purpose of this clinical study was to compare and evaluate the reliability of the SCI and BA values recorded by using 2 different digital devices.

MATERIAL AND METHODS

A total of 17 participants, aged between 20 and 30 years (10 women and 7 men) were included in the study. For each participant, the Cadiax Compact 2 and MODJAW tracking devices were used to measure the SCI and BA values at 3 mm and 5 mm of condylar displacement during 3 separate recording sessions. The intraclass correlation coefficient (ICC) was used to assess the reliability of the recordings. Comparisons between the devices were performed with the Wilcoxson rank-sum test (α=.05). The Bland-Altman plot was used to evaluate the quantitative agreement between the 2 devices.

RESULTS

All ICC intrarater reliability values for Cadiax and MODJAW were higher than 0.90. Both at 3 and 5 mm, the SCI measurements were higher for MODJAW registrations than for those recorded by Cadiax. The Bland-Altman plot showed that the SCI values were higher for MODJAW than for Cadiax by 5.9 (95% CI 3.9 to 8.2) and that the BA differences between the MODJAW and the Cadiax were not consistent with the measured value.

CONCLUSIONS

Excellent reliability was found with the MODJAW and Cadiax recordings. The SCI and BA values for MODJAW measurements were higher at 3 mm and 5 mm than those acquired with Cadiax. MODJAW showed higher values than Cadiax, and the discrepancies were more pronounced for 3 mm than for 5 mm of condylar displacement, highlighting the need for precision in measurements at lower ranges of motion.

Assessing total erythrocyte count as a potential attribute for estimating hemoglobin in Cholistani cattle

The main objective of the study was to confirm whether a specific and constant interrelationship exists between total erythrocyte count (TEC) and hemoglobin (Hb) concentration in Cholistani cattle blood and to navigate the potential of TEC for estimating Hb level in Cholistani cattle (n = 264) grouped as per gender (males, n = 122; females, n = 142) and age (young, n = 140; adults, n = 124). The TEC and Hb (HbD) estimation was carried out through veterinary hematology analyzer. The Hb was also calculated as TEC × 3 and was termed as HbC. Linear regression was implied, and accordingly, scatterplots were drawn between TEC, HbD, HbC, and corrected Hb (CHB). The regression prediction equation hence attained was used to deduce corrected hemoglobin (CHb). A significant (P ≤ 0.05) difference was noticed between HbD and HbC. A non-significant (P ≥ 0.05) difference was noticed, however, between HbD and CHb. Tests of level of agreement indicated a higher Cronbach's alpha and intraclass correlation coefficient (0.682 for average measures) for HbD and CHb as compared to that for HbD and HbC (0.559 for average measures). A convention of Hb concentration as three times of TEC (× 3) is not valid for Cholistani cattle. A different pen-side hematological formula, i.e., Hb (g/dL) = 0.66(TEC) + 6.1, however, provides a better estimate of Hb from the TEC in cattle blood. Using hemocytometry for TEC in the field, all the stakeholders associated with veterinary research, academics, and practice may benefit from this formula in resource-poor countries.

Assessing Agreement When Agreement Is Hard to Assess-The Agatston Score for Coronary Calcification

Method comparison studies comprised simple scatterplots of paired measurements, a 45-degree line as benchmark, and correlation coefficients up to the advent of Bland-Altman analysis in the 1980s. The Agatston score for coronary calcification is based on computed tomography of the heart, and it originated in 1990. A peculiarity of the Agatston score is the often-observed skewed distribution in screening populations. As the Agatston score has manifested itself in preventive cardiology, it is of interest to investigate how reproducibility of the Agatston score has been established. This review is based on literature findings indexed in MEDLINE/PubMed before 20 November 2021. Out of 503 identified articles, 49 papers were included in this review. Sample sizes were highly variable (10-9761), the main focus comprised intra- and interrater as well as intra- and interscanner variability assessments. Simple analysis tools such as scatterplots and correlation coefficients were successively supplemented by first difference, later Bland-Altman plots; however, only very few publications were capable of deriving Limits of Agreement that fit the observed data visually in a convincing way. Moreover, several attempts have been made in the recent past to improve the analysis and reporting of method comparison studies. These warrant increased attention in the future.

A Method to Combine Neurofilament Light Measurements From Blood Serum and Plasma in Clinical and Population-Based Studies

Introduction

Neurofilament light (NfL) can be detected in blood of healthy individuals and at elevated levels in those with different neurological diseases. We investigated if the choice of biological matrix can affect results when using NfL as biomarker in epidemiological studies.

Method

We obtained paired serum and EDTA-plasma samples of 299 individuals aged 37–67 years (BiDirect study) and serum samples of 373 individuals aged 65–83 years (MEMO study). In BiDirect, Passing–Bablok analyses were performed to assess proportional and systematic differences between biological matrices. Associations between serum or EDTA-plasma NfL and renal function (serum creatinine, serum cystatin C, glomerular filtration rate, and kidney disease) were investigated using linear or logistic regression, respectively. All regression coefficients were estimated (1) per one ng/L increase and (2) per one standard deviation increase (standardization using z-scores). In MEMO, regression coefficients were estimated (1) per one ng/L increase of serum or calculated EDTA-plasma NfL and (2) per one standard deviation increase providing a comparison to the results from BiDirect.

Results

We found proportional and systematic differences between paired NfL measurements in BiDirect, i.e., serum NfL [ng/L] = −0.33 [ng/L] + 1.11 × EDTA-plasma NfL [ng/L]. Linear regression coefficients for the associations between NfL and renal function did not vary between the different NfL measurements. In MEMO, one standard deviation increase in serum NfL was associated with greater changes in the outcomes than in BiDirect.

Conclusion

Although there are differences between serum and EDTA-plasma NfL, results can be used interchangeably if standardized values are used.

The Effect of High-Altitude Acclimatisation on Ultra-Short Heart Rate Variability

Introduction

High-altitude (HA) exposure affects heart rate variability (HRV) and has been inconsistently linked to acute mountain sickness (AMS). The influence of increasing HA exposure on ultra-short HRV and its relationship to gold standard HRV measures at HA has not been examined.

Methods

This was a prospective observational study of adults aged ≥ 18 years undertaking a HA trek in the Dhaulagiri region of the Himalayas. Cardiac inter-beat-intervals were obtained from a 10-s recording of supra-systolic blood pressure (Uscom BP⁺ device) immediately followed by 300 s single lead ECG recording (CheckMyHeart device). HRV was measured using the RMSSD (root mean square of successive differences of NN intervals) at sea level (SL) in the United Kingdom and at 3,619, 4,600, and 5,140 m at HA. Oxygen saturations (SpO₂) were measured using finger-based pulse oximetry. The level of agreement between the 10 and 300 s RMSSD values were examined using a modified Bland–Altman relative-difference analysis.

Results

Overall, 89 participants aged 32.2 ± 8.8 years (range 18–56) were included of which 70.8% were men. HA exposure (SL vs. 3,619 m) was associated with an initial increase in both 10 s (45.0 [31.0–82.0]) vs. 58.0 [33.0–119.0] ms) and 300 s (45.67 [33.24–70.32] vs. 56.48 [36.98–102.0] ms) in RMSSD. Thereafter at 4,600 and 5,140 m both 10 and 300 s RMSSD values were significantly lower than SL. From a total of 317 paired HRV measures the 10 and 300 s RMSSD measures were moderately correlated (Spearman r = 0.66; 95% CI: 0.59–0.72; p < 0.0001). The median difference (bias) in RMSSD values (300 s − 10 s) was −2.3 ms with a lower and upper limit of agreement of −107.5 and 88.61 ms, respectively with no differences with altitude. Overall, 293/317 (92.4%) of all paired HRV values fell within the 95% CI limits of agreement. Neither HRV method was predictive of AMS.

Conclusion

Increasing HA affects ultra-short HRV in a similar manner to gold-standard 300 s. Ultra-short HRV has a moderate agreement with 300 s measurements. HRV did not predict AMS.

Semi-Automated Quantification of Retinal and Choroidal Biomarkers in Retinal Vascular Diseases: Agreement of Spectral-Domain Optical Coherence Tomography with and without Enhanced Depth Imaging Mode

BACKGROUND

We compared with and without enhanced depth imaging mode (EDI) in semi-automated quantification of retinal and choroidal biomarkers in optical coherence tomography (OCT) in patients with diabetic retinopathy (DR) or retinal vein occlusion (RVO) complicated by macular edema. We chose to study three OCT biomarkers: the numbers of hyperreflective foci (HF), the ellipsoid zone reflectivity ratio (EZR) and the choroidal vascularity index (CVI), all known to be correlated with visual acuity changes or treatment outcomes.

METHODS

In a single examination, one eye of each patient (n = 60; diabetic retinopathy: n = 27, retinal vein occlusion: n = 33) underwent macular 870 nm spectral domain-OCT (SD-OCT) B-scans without and with EDI mode. Semi-automated quantification of HF, EZR and CVI was applied according to preexisting published protocols. Paired Student's t-test or Wilcoxon rank-sum test was used to test for differences in subgroups. Intraclass correlation coefficient (ICC) and Bland-Altman plots were applied to describe the agreement between quantification in EDI and conventional OCT mode. The effect of macular edema on semi-automated quantification was evaluated.

RESULTS

For the entire cohort, quantification of all three biomarkers was not significantly different in SD-OCT scans with and without EDI mode (p > 0.05). ICC was 0.78, 0.90 and 0.80 for HF, EZR and CVI. The presence of macular edema led to significant differences in the quantification of hyperreflective foci (without EDI: 80.00 ± 33.70, with EDI: 92.08 ± 38.11; mean difference: 12.09, p = 0.03), but not in the quantification of EZR and CVI (p > 0.05).

CONCLUSION

Quantification of EZR and CVI was comparable whether or not EDI mode was used. In conclusion, both retinal and choroidal biomarkers can be quantified from one single 870 nm SD-OCT EDI image.

Sex Differences in Factors Associated With Progression of Aortic Valve Calcification in the General Population

BACKGROUND

Guidelines recommend measurement of the aortic valve calcification (AVC) score to help differentiate between severe and nonsevere aortic stenosis, but a paucity exists in data about AVC in the general population. The aim of this study was to describe the natural history of AVC progression in the general population and to identify potential sex differences in factors associated with this progression rate.

METHODS

Noncontrast cardiac computed tomography was performed in 1298 randomly selected women and men aged 65 to 74 years who participated in the DANCAVAS trial (Danish Cardiovascular Screening). Participants were invited to attend a reexamination after 4 years. The AVC score was measured at the computed tomography, and AVC progression (ΔAVC) was defined as the difference between AVC scores at baseline and follow-up. Multivariable regression analyses were performed to identify factors associated with ΔAVC.

RESULTS

Among the 1298 invited citizens, 823 accepted to participate in the follow-up examination. The mean age at follow-up was 73 years. Men had significantly higher AVC scores at baseline (median AVC score 13 Agatston Units [AU; interquartile range, 0-94 AU] versus 1 AU [interquartile range, 0-22 AU], P<0.001) and a higher ΔAVC (median 26 AU [interquartile range, 0-101 AU] versus 4 AU [interquartile range, 0-37 AU], P<0.001) than women. In the fully adjusted model, the most important factor associated with ΔAVC was the baseline AVC score. However, hypertension was associated with ΔAVC in women (incidence rate ratios, 1.58 [95% CI, 1.06-2.34], P=0.024) but not in men, whereas dyslipidemia was associated with ΔAVC in men (incidence rate ratio: 1.66 [95% CI, 1.18-2.34], P=0.004) but not in women.

CONCLUSIONS

The magnitude of the AVC score was the most important marker of AVC progression. However, sex differences were significant; hence, dyslipidemia was associated with AVC progression only among men; hypertension with AVC progression only among women. Registration: URL: https://www.isrctn.com; Unique identifier: ISRCTN12157806.

Bland–Altman Limits of Agreement from a Bayesian and Frequentist Perspective

Bland–Altman agreement analysis has gained widespread application across disciplines, last but not least in health sciences, since its inception in the 1980s. Bayesian analysis has been on the rise due to increased computational power over time, and Alari, Kim, and Wand have put Bland–Altman Limits of Agreement in a Bayesian framework (Meas. Phys. Educ. Exerc. Sci. 2021, 25, 137–148). We contrasted the prediction of a single future observation and the estimation of the Limits of Agreement from the frequentist and a Bayesian perspective by analyzing interrater data of two sequentially conducted, preclinical studies. The estimation of the Limits of Agreement θ1 and θ2 has wider applicability than the prediction of single future differences. While a frequentist confidence interval represents a range of nonrejectable values for null hypothesis significance testing of H0: θ1 ≤ −δ or θ2 ≥ δ against H1: θ1 > −δ and θ2 < δ, with a predefined benchmark value δ, Bayesian analysis allows for direct interpretation of both the posterior probability of the alternative hypothesis and the likelihood of parameter values. We discuss group-sequential testing and nonparametric alternatives briefly. Frequentist simplicity does not beat Bayesian interpretability due to improved computational resources, but the elicitation and implementation of prior information demand caution. Accounting for clustered data (e.g., repeated measurements per subject) is well-established in frequentist, but not yet in Bayesian Bland–Altman analysis.

Magnetic Resonance Assessment of Ejection Fraction Versus Echocardiography for Cardioverter-Defibrillator Implantation Eligibility

Simple Summary

Nonischemic cardiomyopathies with low left ventricular ejection fractions (LVEF) are eligible for an implantable cardioverter defibrillator. However, the guidelines do not specify which method should be used to assess LVEF. In our study we investigated the potential impact of performing two-dimensional echocardiography (2DE) compared to cardiovascular magnetic resonance (CMR) for LVEF regarding ICD eligibility. We found that 2DE both overestimated and especially underestimated the need for implantation, which can have serious implications in the quality of life and the prevention of death events.

Abstract

Background: The aim of this study was to investigate the potential impact of performing two-dimensional echocardiography (2DE) compared to cardiovascular magnetic resonance (CMR) for left ventricular ejection fraction (LVEF) on implantable cardioverter defibrillator (ICD) eligibility. Methods: A prospective cohort of 166 consecutive patients with nonischemic cardiomyopathy (NICM) was designed to compare transthoracic 2DE and CMR imaging. Results: Echocardiography measurements have important differences and large limits of agreement compared to CMR, especially when assessing ventricle volumes, and smaller but relevant differences when assessing LVEF. The agreement between CMR and 2DE regarding the identification of subjects with EF <= 35, respectively <= 30, and thus eligible for an ICD measured by Cohen’s Kappa was 0.78 (95% CI: 0.68–0.88), p < 0.001, respectively 0.65 (95% CI: 0.52–0.78), p < 0.001. The disagreement represented 7.9%/11.3% of the subjects who had EF < 35%/< 30% as observed by CMR, who would have been classified as eligible for an ICD, resulting in an additional need to use an ICD. Moreover, 2.6%/3.3% would have been deemed eligible by echocardiography for an ICD. Conclusions: These measurement problems result in incorrect assignments of eligibility that may have serious implications on the quality of life and the prevention of death events for patients assessed for eligibility of an ICD.

Prevalence and extent of coronary artery calcification in the middle-aged and elderly population

AIMS

Coronary artery calcification (CAC) measured on cardiac computed tomography (CT) is an important risk marker for cardiovascular disease (CVD) and has been included in the prevention guidelines. The aim of this study was to describe CAC score reference values in the middle-aged and elderly population and to develop a freely available CAC calculator.

METHODS AND RESULTS

All participants from two population-based cardiac CT screening cohorts (DanRisk and DANCAVAS) were included. The CAC score was measured as a part of a screening session. Positive CAC scores were log-transformed and non-parametrically regressed on age for each gender, and percentile curves were transposed according to proportions of zero CAC scores. Men had higher CAC scores than women, and the prevalence and extend of CAC increased steadily with age. An online CAC calculator was developed, http://flscripts.dk/cacscore. After entering sex, age, and CAC score, the CAC score percentile and the coronary age are depicted including a figure with the specific CAC score and 25%, 50%, 75%, and 90% percentiles. The specific CAC score can be compared to the entire background population or only those without prior CVD.

CONCLUSION

This study provides modern population-based reference values of CAC scores in men and woman and a freely accessible online CAC calculator. Physicians and patients are very familiar with blood pressure and lipids, but unfamiliar with CAC scores. Using the calculator makes it easy to see if a CAC value is low, moderate, or high, when a physician in the future communicate and discusses a CAC score with a patient.