Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model

Reliability of measurement instruments providing quantitative outcomes is usually assessed by an intraclass correlation coefficient. When participants are repeatedly measured by a single rater or device, or, are each rated by a different group of raters, the intraclass correlation coefficient is based on a one-way analysis of variance model. When planning a reliability study, it is essential to determine the number of participants and measurements per participant (i.e. number of raters or number of repeated measurements). Three different sample size determination approaches under the one-way analysis of variance model were identified in the literature, all based on a confidence interval for the intraclass correlation coefficient. Although eight different confidence interval methods can be identified, Wald confidence interval with Fisher's large sample variance approximation remains most commonly used despite its well-known poor statistical properties. Therefore, a first objective of this work is comparing the statistical properties of all identified confidence interval methods-including those overlooked in previous studies. A second objective is developing a general procedure to determine the sample size using all approaches since a closed-form formula is not always available. This procedure is implemented in an R Shiny app. Finally, we provide advice for choosing an appropriate sample size determination method when planning a reliability study.

Reliability and validity testing of the medicines related - consultation assessment tool for assessing pharmacists' consultations

BACKGROUND

Demonstrating a person-centred approach in a consultation is a key component of delivering high-quality healthcare. To support development of such an approach requires training underpinned by valid assessment tools. Given the lack of a suitable pharmacy-specific tool, a new global consultation skills assessment tool: the medicines related-consultation assessment tool (MR-CAT) was designed and tested.

AIM

This study aimed to test the validity and reliability of the MR-CAT using psychometric methods.

METHOD

Psychometric testing involved analysis of participants' (n = 13) assessment of fifteen pre-recorded simulated consultations using the MR-CAT. Analysis included discriminant validity testing, intrarater and interrater reliability testing for each of the five sections of the MR-CAT and for the overall global assessment of the consultation. Analysis also included internal consistency testing for the whole tool.

RESULTS

Internal consistency for the overall global assessment of the consultation was good (Cronbach's alpha = 0.97). The MR-CAT discriminated well for the overall global assessment of the consultation (p < 0.001). Moderate to high intrarater reliability was observed for the overall global assessment of the consultation and for all five sections of the MR-CAT (rho = 0.64-0.84) in the test-retest analysis. Moderate to good interrater reliability (Kendall's W = 0.68-0.90) was observed for the overall global assessment of the consultation and for all five sections of the MR-CAT.

CONCLUSION

The MR-CAT is a valid and reliable tool for assessing person-centred pharmacist's consultations. Moreover, its unique design means that the MR-CAT can be used in both formative and summative assessment.

Injuries to the anterolateral ligament are observed more frequently compared to lesions to the deep iliotibial tract (Kaplan fibers) in anterior cruciate ligamant deficient knees using magnetic resonance imaging

PURPOSE

To determine the accuracy of detection, injury rate and inter- and intrarater reproducibility in visualizing lesions to the anterolateral ligament (ALL) and the deep portion of the iliotibial tract (dITT) in anterior cruciate ligament (ACL) deficient knees.

METHODS

Ninety-one consecutive patients, out of those 25 children (age 14.3 ± 3.5 years), with diagnosed ACL tears were included. Two musculoskeletal radiologists retrospectively reviewed MRI data focusing on accuracy of detection and potential injuries to the ALL or dITT. Lesion were diagnosed in case of discontinued fibers in combination with intra- or peri-ligamentous edema and graded as intact, partial or complete tears. Cohen's Kappa and 95% confidence intervals (95% CI) were determined for inter- and intrarater reliability measures.

RESULTS

The ALL and dITT were visible in 52 (78.8%) and 56 (84.8%) of adult-and 25 (100%) and 19 (76.0%) of pediatric patients, respectively. The ALL was injured in 45 (58.5%; partial: 36.4%, compleate: 22.1%) patients. Partial and comleate tears, where visualized in 21 (40.4%) and 16 (30.8%) adult- and seven (28.0%) and one (4%) peditric patients. A total of 16 (21.3%; partial: 13.3%, compleate: 8.0%) dITT injuries were identified. Partal and complete lesions were seen in seven (12.5%) and five (8.9%) adult- and three (15.8%) and one (5.3%) pediatric patients. Combined injuries were visualized in nine (12.7%) patients. Inter-observer (0.91-0.95) and intra-observer (0.93-0.95) reproducibility was high.

CONCLUSION

In ACL injured knees, tears of the ALL are observed more frequently compared to lesions to the deep iliotibial tract. Combined injuries of both structures are rare. Clinically, the preoperative visualization of potentially injured structures of the anterolateral knee is crucial and is important for a more personalized preoperative planning and tailored anatomical reconstruction. The clinical implication of injuries to the anterolateral complex of the knee needs further investigation.

LEVEL OF EVIDENCE

II.

Interrater and Intrarater Reliability of the Colloid Cyst Risk Score

BACKGROUND

The Colloid Cyst Risk Score (CCRS) was developed to identify symptomatic patients and stratify risk of hydrocephalus among patients with colloid cysts. Its components consider patient age, cyst diameter, presence/absence of headache, fluid-attenuated inversion recovery (FLAIR) hyperintensity, and location within the third ventricle.

OBJECTIVE

To independently evaluate the inter- and intrarater reliability of the CCRS.

METHODS

Patients with a colloid cyst were identified from billing records and radiology archives. Three independent raters reviewed electronic medical records to determine age, presence/absence of headache, cyst diameter (mm), FLAIR hyperintensity, and risk zone location. Raters made 53 observations, including 5 repeat observations.Fleiss' generalized kappa (κ) was calculated for all of the nominal criteria, whereas Kendall's coefficient of concordance (W) and the intraclass correlation coefficient (ICC) were calculated for the overall score.

RESULTS

Total CCRS score demonstrated extremely strong agreement (W = 0.83) using Kendall's W coefficient and good agreement (ICC = 0.74) using the ICC (P < .001). For interrater reliability of individual criteria, age (κ = 1.00) and FLAIR hyperintensity (κ = 0.89) demonstrated near perfect agreement. Axial diameter (κ = 0.63) demonstrated substantial agreement, whereas agreement was moderate for risk zone (κ = 0.51) and fair for headache (κ = 0.26). Intrarater reliability for total CCRS score was extremely strong using Kendall's W, good to excellent using ICC, and fair to substantial using weighted kappa.

CONCLUSION

The CCRS has good inter- and intrarater reliability when tested in an independent sample of patients, though strength of agreement varies among individual criteria. The validity of the CCRS requires independent evaluation.

Reliability of High-speed Videoendoscopic Ratings of Essential Voice Tremor and Adductor Spasmodic Dysphonia

OBJECTIVES

The main objectives of this study were to (1) examine intrarater and inter-rater reliabilities in perceptual ratings of vocal fold vibratory patterns and supraglottic characteristics for essential vocal tremor and adductor spasmodic dysphonia (AdSD) using high-speed videoendoscopy (HSV), and (2) to investigate the specificity of the parameters in differentiating these two voice disorders.

METHODS

HSV recordings of 34 cases diagnosed with essential vocal tremor, AdSD, or AdSD with vocal tremor were evaluated blindly by two voice speech pathologists. The two raters examined all HSV video segments twice across nine supraglottic and vocal fold vibratory characteristics for inter-rater and intrarater reliabilities. A separate consensus rating was then developed, with the results analyzed to explore differentiation.

RESULTS

Raters demonstrated moderate intrarater reliability with mean Spearman's rho correlation coefficients of 0.68 (rater 1) and 0.73 (rater 2). Moderate inter-rater reliability for the two raters was seen across all parameters with a mean Cohen's kappa coefficient of 0.51. Raters showed higher intrarater and inter-rater reliabilities for supraglottic parameters. Only the presence of tremor differentiated between the two voice disorders in cases with a consistent diagnosis.

CONCLUSIONS

The high level of concomitance between vocal tremor and AdSD may affect subjective perceptual analysis of supraglottic and vocal fold vibratory patterns. Results indicate similar global involvement of supraglottic laryngeal structures for both vocal tremor and AdSD.

Intrarater reliability of hand held dynamometry in measuring lower extremity isometric strength using a portable stabilization device

Hand held dynamometry (HHD) is a more objective way to quantify muscle force production (MP) compared to traditional manual muscle testing. HHD reliability can be negatively impacted by both the strength of the tester and the subject particularly in the lower extremities due to larger muscle groups. The primary aim of this investigation was to assess intrarater reliability of HHD with use of a portable stabilization device for lower extremity MP in an athletic population. Isometric lower extremity strength was measured for bilateral lower extremities including hip abductors, external rotators, adductors, knee extensors, and ankle plantar flexors was measured in a sample of healthy recreational runners (8 male, 7 females, = 30 limbs) training for a marathon. These measurements were assessed using an intrasession intrarater reliability design. Intraclass correlation coefficients (ICC) were calculated using 3,1 model based on the single rater design. The standard error of measurement (SEM) for each muscle group was also calculated. ICC were excellent ranging from ICC (3,1) = 0.93-0.98 with standard error of measurements ranging from 0.58 to 17.2 N. This study establishes the use of a HHD with a portable stabilization device as demonstrating good reliability within testers for measuring lower extremity muscle performance in an active healthy population.

German version, inter- and intrarater reliability and internal consistency of the "Agitated Behavior Scale" (ABS-G) in patients with moderate to severe traumatic brain injury

Background

Agitation is frequently observed during early recovery after traumatic brain injury (TBI). Agitated behaviour often interferes with a goal-orientated rehabilitation and can be a substantial hindrance to therapy. Despite the relatively high occurance of agitation in TBI population there is no objective assessement in German (G) available. An existing scale with excellent psychometric properties is the “Agitated Behavior Scale (ABS)” developed by Corrigan in 1989. The aim of the study was to translate the Agitated Behavior Scale (ABS) into German (ABS-G) and investigate the inter- and intrarater reliability and internal consistency in patients with moderate to severe TBI.

Methods

A formal nine-step translation and cross-cultural adaptation procedure (TCCA) was applied. Subsequently a prospective observational patient study was conducted. To examine the interrater reliability and internal consistency, two therapists rated 20 patients independently after a therapy session. This procedure was repeated twice on a weekly basis. The intrarater reliability was assessed through video recordings from three patients. Nine raters scored the demonstrated behaviour on the videotape with the ABS-G independently twice within one month. The inter- and intrarater reliability were evaluated with the Spearman rank correlation coefficient and the quadratic weighted kappa. The internal consistency was tested with Cronbach’s alpha.

Results

Behaviour of 20 patients (18 males; mean age 41 ± 20.7; mean Functional Independence Measure (FIM) cognitive score on admission 7.1 ± 4.04; mean ABS-G score at first observation 17.3 ± 2.83) was assessed threefold. Interrater reliability yielded a correlation coefficient for ABS-G total score of all 60 paired observations of r_s 0.845 and a weighted Kappa of 0.738. Intrarater reliability for ABS-G total score ranged between r_s 0.719 and 0.953 and showed a weighted Kappa between 0.871 and 0.953. Cronbach’s alpha indicated moderate internal consistency with 0.661.

Conclusion

This study demonstrates that the ABS-G is a reliable instrument for evaluating agitation in patients with moderate to severe TBI. Hereby it would be possible to monitor agitation objectively and optimise the management of agitated patients according to international recommendations.

Intrarater reliabilities of shoulder joint horizontal adductor muscle strength measurements using a handheld dynamometer for geriatric and stroke patients

[Purpose] This study aimed to verify the appropriate number of measurements and the intrarater reliabilities of shoulder joint horizontal adductor muscle strength measurements using a handheld dynamometer (HHD) for geriatric and stroke patients. [Subjects and Methods] The subjects were 40 inpatients, who were divided into two groups: 20 stroke patients in the stroke group (SG), and 20 geriatric patients in the no-stroke group (N-SG). Measurements were performed three times using an HHD with a belt. The reliability was verified using Bland-Altman analysis and the intraclass correlation coefficient (ICC). [Results] ICC (1, 1) was >0.9. A systematic bias was not observed between the first and second measurement values except for the right side in N-SG. A systematic bias between the maximum value obtained during the first and second measurements and third measurement value was observed on the left side in N-SG, and on the non-paralyzed side in SG: the third measurement values were small in both cases. [Conclusion] Intrarater reliabilities were high for shoulder horizontal adductor strength measurements using an HHD with a belt for geriatric and stroke patients. Taking the systematic bias into consideration, these findings suggest that the required number of measurements is two.

Pneumothorax size measurements on digital chest radiographs: Intra- and inter- rater reliability

PURPOSE

Detailed and reliable methods may be important for discussions on the importance of pneumothorax size in clinical decision-making. Rhea's method is widely used to estimate pneumothorax size in percent based on chest X-rays (CXRs) from three measure points. Choi's addendum is used for anterioposterior projections. The aim of this study was to examine the intrarater and interrater reliability of the Rhea and Choi method using digital CXR in the ward based PACS monitors.

MATERIALS AND METHODS

Three physicians examined a retrospective series of 80 digital CXRs showing pneumothorax, using Rhea and Choi's method, then repeated in a random order two weeks later. We used the analysis of variance technique by Eliasziw et al. to assess the intrarater and interrater reliability in altogether 480 estimations of pneumothorax size.

RESULTS

Estimated pneumothorax sizes ranged between 5% and 100%. The intrarater reliability coefficient was 0.98 (95% one-sided lower-limit confidence interval C 0.96), and the interrater reliability coefficient was 0.95 (95% one-sided lower-limit confidence interval 0.93).

CONCLUSION

This study has shown that the Rhea and Choi method for calculating pneumothorax size has high intrarater and interrater reliability. These results are valid across gender, side of pneumothorax and whether the patient is diagnosed with primary or secondary pneumothorax.

Intra- and inter-rater reliability of maximum inspiratory pressure measured using a portable capsule-sensing pressure gauge device in healthy adults

BACKGROUND

Measurement of maximum inspiratory pressure is the most prevalent method used in clinical practice to assess the strength of the inspiratory muscles. Although there are many devices available for the assessment of inspiratory muscle strength, there is a dearth of literature describing the reliability of devices that can be used in clinical patient assessment. The capsule-sensing pressure gauge (CSPG-V) is a new tool that measures the strength of inspiratory muscles; it is easy to use, noninvasive, inexpensive and lightweight.

OBJECTIVE

To test the intra- and inter-rater reliability of a CSPG-V device in healthy adults.

METHODS

A cross-sectional study involving 80 adult subjects with a mean (± SD) age of 22±3 years was performed. Using simple randomization, 40 individuals (20 male, 20 female) were used for intrarater and 40 (20 male, 20 female) were used for inter-rater reliability testing of the CSPG-V device. The subjects performed three inspiratory efforts, which were sustained for at least 3 s; the best of the three readings was used for intra- and inter-rater comparison. The intra- and inter-rater reliability were calculated using intraclass correlation coefficients (ICCs).

RESULTS

The intrarater reliability ICC was 0.962 and the inter-rater reliability ICC was 0.922.

CONCLUSION

Results of the present study suggest that maximum inspiratory pressure measured using a CSPG-V device has excellent intraand inter-rater reliability, and can be used as a diagnostic and prognostic tool in patients with respiratory muscle impairment.

Inter- and intrarater reliability of Minimal Eating Observation and Nutrition Form - version II (MEONF-II) nurse assessments among hospital inpatients

Background

The Minimal Eating Observation and Nutrition form – version II (MEONF – II) is a recently developed nursing nutritional screening tool. However, its inter- and intrarater reliability has not been assessed.

Methods

Inpatients (n = 24; median age, 69 years; 11 women) were assessed by eight nurses (interrater reliability, two nurses scored each patient independently) using the MEONF-II on two consecutive days (intrarater reliability, each patient was scored by the same nurse day 1 and day 2).

Results

Six patients were at moderate/high undernutrition risk. Inter- and intrarater reliabilities (Gwet’s agreement coefficient) for the MEONF-II 2-category classification (no/low risk versus moderate/high risk) were 0.93 and 0.81; for the 3-category classification (no/low – moderate – high risk) reliabilities (Gwet’s weighted agreement coefficient) were 0.98 and 0.88; and total score inter- and intrarater reliabilities (intraclass correlation) were 0.92 and 0.84.

Conclusion

Reliability of MEONF-II nurse assessments among adult hospital inpatients was supported and the tool can be used in research and clinical practice.

EEG interpretation reliability and interpreter confidence: a large single-center study

The intrarater and interrater reliability (I&IR) of EEG interpretation has significant implications for the value of EEG as a diagnostic tool. We measured both the intrarater reliability and the interrater reliability of EEG interpretation based on the interpretation of complete EEGs into standard diagnostic categories and rater confidence in their interpretations and investigated sources of variance in EEG interpretations. During two distinct time intervals, six board-certified clinical neurophysiologists classified 300 EEGs into one or more of seven diagnostic categories and assigned a subjective confidence to their interpretations. Each EEG was read by three readers. Each reader interpreted 150 unique studies, and 50 studies were re-interpreted to generate intrarater data. A generalizability study assessed the contribution of subjects, readers, and the interaction between subjects and readers to interpretation variance. Five of the six readers had a median confidence of ≥99%, and the upper quartile of confidence values was 100% for all six readers. Intrarater Cohen's kappa (κc) ranged from 0.33 to 0.73 with an aggregated value of 0.59. Cohen's kappa ranged from 0.29 to 0.62 for the 15 reader pairs, with an aggregated Fleiss kappa of 0.44 for interrater agreement. Cohen's kappa was not significantly different across rater pairs (chi-square=17.3, df=14, p=0.24). Variance due to subjects (i.e., EEGs) was 65.3%, due to readers was 3.9%, and due to the interaction between readers and subjects was 30.8%. Experienced epileptologists have very high confidence in their EEG interpretations and low to moderate I&IR, a common paradox in clinical medicine. A necessary, but insufficient, condition to improve EEG interpretation accuracy is to increase intrarater and interrater reliability. This goal could be accomplished, for instance, with an automated online application integrated into a continuing medical education module that measures and reports EEG I&IR to individual users.