Extraction of the pull force from inertial sensors during the pull test for Parkinson's disease: A reliability study

OBJECTIVE

To examine the inter- and intra-rater reliability of the pull test in patients with Parkinson's disease (PD) using the extracted pull force.

METHODS

In this inter- and intra-rater reliability study, two raters performed a pull test on 30 patients with PD. The pull force was quantified using inertial sensors attached to the rater's right hand and the patient's lower trunk. In this study, the pull force was calculated as an extracted three-dimensional vector quantity, the resultant acceleration, and was expressed in m/s2. Inter- and intra-rater reliabilities were analyzed using the interclass correlation coefficient (ICC) for the pull force and Cohen's weighted kappa (κw) for the pull test score. Furthermore, Bland-Altman analysis was used to investigate systematic errors.

RESULTS

The inter- and intra-rater reliability of the pull force was very poor (ICC = 0.033-0.214). Bland-Altman analysis revealed no systematic errors in the pull forces between the two test points. Conversely, κw for the pull test scores ranged from 0.763 to 0.920, indicating substantial to almost perfect agreement.

CONCLUSION

The pull test score was reliable despite variations in the quantified pull force for inter- and intra-rater reliability. Our findings suggest that the pull test is a robust tool for evaluating postural instability in patients with PD and that the pull force probably does not affect scoring performance.

Qualitative visual assessment of the J-sign demonstrates high inter-rater reliability

OBJECTIVES

To assess the inter- and intra-rater reliability of the classification of the J-sign as "large" versus "small or none" as compared to another two-level system ("present" versus "absent") and a three-level system ("large," "small," or "none") and to identify anatomical and patient factors associated with the presence of a large J-sign.

METHODS

Forty patients (40 knees) with recurrent patellar instability were prospectively enrolled and recorded on video actively extending their knee while seating. Four raters classified patellar tracking on two separate occasions using three systems: 1) two groups: J-sign versus no J-sign; 2) three groups: large J-sign, small J-sign, or no J-sign; and 3) two groups: large J-sign versus small or no J-sign. The intra- and inter-rater reliability of each system was assessed using kappa statistics. Anatomical (trochlear dysplasia, tibial tubercle-trochlear groove (TT-TG) distance, patellar height) and patient (Beighton score) factors as well as Knee Injury and Osteoarthritis Outcome Score (KOOS) subscales were compared between patients with a large J-sign and patients with a small or no J-sign.

RESULTS

Inter- and intra-rater reliability were found to be highest with the two-level classification system of a large J-sign versus a small or no J-sign (inter-rater kappa ​= ​0.76, intra-rater kappa ​= ​0.75). Patients with a large J-sign had more severe trochlear dysplasia as assessed with the sulcus angle (p ​= ​0.042) and were more likely to have a tight lateral retinaculum (p ​= ​0.032) and an elevated Beighton score (p ​= ​0.009). No significant differences in KOOS subscales were noted based on the presence of a large J-sign versus a small J-sign or no J-sign.

CONCLUSION

Qualitative visual assessment of patellar tracking with the J-sign demonstrates substantial inter- and intra-rater reliability, particularly when utilizing a two-group classification system to identify knees with a large J-sign. Patients with a large J-sign demonstrate an increased incidence of a tight lateral retinaculum, generalized ligamentous laxity, and trochlear dysplasia.

LEVEL OF EVIDENCE

Level III - cross-sectional study.

Assessment of the iridocorneal angle pigmentation and structures after the implantation of EVO+ Visian Implantable Collamer Lens

PURPOSE

To evaluate the effect of EVO+ Visian Implantable Collamer Lens (ICL) implantation on the iridocorneal angle pigmentation and structures.

METHODS

Myopic refractive surgery candidates who underwent posterior chamber EVO+ ICL implantation were evaluated preoperatively and 3 and 6 months postoperatively. High-resolution images of the iridocorneal angle (nasal, superior, temporal and inferior quadrants) were acquired during gonioscopy. A masked observer, blinded to study visits and patients' information, evaluated the angle width, apparent iris root insertion, iris configuration, and trabecular meshwork pigmentation according to the Spaeth scale in two different occasions. The intra-rater reliability was estimated using the weighted Gwet's Agreement Coefficient (AC2). Differences between visits were analyzed using the Cochran Q test or the Friedman test.

RESULTS

Twenty-one patients (13 females and 8 males) aging 31.3 ± 6.3 years old were recruited. The intra-rater reliability of gonioscopy assessment was excellent for every parameter assessed (AC2 ≥ 0.97). No significant differences were found among the study visits for any quadrant in the angle width (p ≥ 0.74), apparent iris root insertion (p ≥ 0.22), iris configuration (p ≥ 0.21) and trabecular meshwork pigmentation (p ≥ 0.24). Mean pigmentation of trabecular meshwork for the four quadrants did not vary either among visits (p = 0.25).

CONCLUSIONS

The EVO+ ICL implantation after uneventful procedures appears not to clinically affect the iridocorneal angle pigmentation and/or structures during a short-medium follow-up using gonioscopy assessment. The intra-rater reliability of glaucoma specialists assessing iridocorneal angle structures using gonioscopy images is very high.

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

BACKGROUND

In research designs that rely on observational ratings provided by two raters, assessing inter-rater reliability (IRR) is a frequently required task. However, some studies fall short in properly utilizing statistical procedures, omitting essential information necessary for interpreting their findings, or inadequately addressing the impact of IRR on subsequent analyses' statistical power for hypothesis testing.

METHODS

This article delves into the recent publication by Liu et al. in BMC Cancer, analyzing the controversy surrounding the Kappa statistic and methodological issues concerning the assessment of IRR. The primary focus is on the appropriate selection of Kappa statistics, as well as the computation, interpretation, and reporting of two frequently used IRR statistics when there are two raters involved.

RESULTS

The Cohen's Kappa statistic is typically utilized to assess the level of agreement between two raters when there are two categories or for unordered categorical variables with three or more categories. On the other hand, when it comes to evaluating the degree of agreement between two raters for ordered categorical variables comprising three or more categories, the weighted Kappa is a widely used measure.

CONCLUSION

Despite not substantially affecting the findings of Liu et al.?s study, the statistical dispute underscores the significance of employing suitable statistical methods. Rigorous and accurate statistical results are crucial for producing trustworthy research.

Inter- and intra-rater reliability of handpump functionality field tests

Sustaining the functionality of drinking water supplies in low- and middle-income countries is a longstanding challenge. Growing awareness of this problem has motivated increased attention to validly and reliably measuring water point functionality, including among handpumps, which serve approximately 9 % of the global population. Yet the most widely used indicator of functionality, whether a water point provides water, has limited validity, reliability, and utility. We tested the inter-rater (agreement among measurements taken by different people) and intra-rater (agreement among repeated measurements taken by the same person) reliability of three handpump functionality field tests in Uganda: pump capacity, 10-minute leakage rate, and flowrate. One person equipped with a stopwatch and a 20-liter container can complete the tests for one handpump in 15 min. The same three to four raters each conducted the tests three times on 28 handpumps. Different sets of four to five raters each conducted the tests once on 32 handpumps. Intraclass correlation coefficients were estimated to indicate inter- and intra-rater reliability. Ten-minute leakage rate had the highest inter-rater reliability, followed by pump capacity. Flowrate, which is commonly measured manually as part of handpump functionality assessments, had poor inter-rater reliability. Indicators derived from all three tests had high intra-rater reliability. Drawing on our inter-rater reliability results, we propose a fully quantitative procedure and validate an ordinal scale of physical handpump functionality based on the 10-minute leakage rate and pump capacity tests. This measurement procedure can be usefully incorporated into service delivery monitoring and research to enhance the objectivity, utility, and comparability of global handpump functionality data. Future studies can test the reliability of these indicators in other contexts and their value for predicting handpump breakdown.

Proposal and multicentric validation of a laparoscopic Roux-en-Y gastric bypass surgery ontology

BACKGROUND

Phase and step annotation in surgical videos is a prerequisite for surgical scene understanding and for downstream tasks like intraoperative feedback or assistance. However, most ontologies are applied on small monocentric datasets and lack external validation. To overcome these limitations an ontology for phases and steps of laparoscopic Roux-en-Y gastric bypass (LRYGB) is proposed and validated on a multicentric dataset in terms of inter- and intra-rater reliability (inter-/intra-RR).

METHODS

The proposed LRYGB ontology consists of 12 phase and 46 step definitions that are hierarchically structured. Two board certified surgeons (raters) with > 10 years of clinical experience applied the proposed ontology on two datasets: (1) StraBypass40 consists of 40 LRYGB videos from Nouvel Hôpital Civil, Strasbourg, France and (2) BernBypass70 consists of 70 LRYGB videos from Inselspital, Bern University Hospital, Bern, Switzerland. To assess inter-RR the two raters' annotations of ten randomly chosen videos from StraBypass40 and BernBypass70 each, were compared. To assess intra-RR ten randomly chosen videos were annotated twice by the same rater and annotations were compared. Inter-RR was calculated using Cohen's kappa. Additionally, for inter- and intra-RR accuracy, precision, recall, F1-score, and application dependent metrics were applied.

RESULTS

The mean ± SD video duration was 108 ± 33 min and 75 ± 21 min in StraBypass40 and BernBypass70, respectively. The proposed ontology shows an inter-RR of 96.8 ± 2.7% for phases and 85.4 ± 6.0% for steps on StraBypass40 and 94.9 ± 5.8% for phases and 76.1 ± 13.9% for steps on BernBypass70. The overall Cohen's kappa of inter-RR was 95.9 ± 4.3% for phases and 80.8 ± 10.0% for steps. Intra-RR showed an accuracy of 98.4 ± 1.1% for phases and 88.1 ± 8.1% for steps.

CONCLUSION

The proposed ontology shows an excellent inter- and intra-RR and should therefore be implemented routinely in phase and step annotation of LRYGB.

Inter-rater and intra-rater reliability of isotonic exercise monitoring device for measuring active knee extension

Background

The goal of this study was to assess the reliability of electromyography and range of motion measurements obtained using a knee exercise monitoring system. This device was developed to collect data on knee exercise activities.

Methods

Twenty healthy individuals performed isotonic quadriceps exercises in this study. The vastus medialis surface electromyography (sEMG) and range of motion (ROM) of the knee were recorded during the exercise using the isotonic knee exercise monitoring device, the Mobi6-6b, and a video camera system. Each subject underwent a second measuring session at least 24 h after the first session. To determine reliability, the intraclass correlation coefficients (ICCs) and standard error of measurement (SEM) at the 95% confidence interval were calculated, and a Bland-Altman analysis was performed.

Results

For inter-rater reliability, the ICCs of the mean absolute value (MAV) and root mean square (RMS) of sEMG were 0.73 (0.49, 0.86) and 0.79 (0.61, 0.89), respectively. ROM had an ICC of 0.93 (0.02, 0.98). The intra-rater reliability of the MAV of the sEMG was 0.89 (0.71, 0.96) and the intra-rater reliability of RMS of the sEMG was 0.88 (0.70, 0.95). The ROM between days had an intra-rater reliability of 0.82 (0.54, 0.93). The Bland-Altman analysis demonstrated no systematic bias in the MAV and RMS of sEMG, but revealed a small, systematic bias in ROM (-0.8311 degrees).

Conclusion

For sEMG and range of motion measures, the isotonic knee exercise monitoring equipment revealed moderate to excellent inter- and intra-rater agreement. However, the confidence interval of ROM inter-rater reliability was quite large, indicating a small agreement bias; hence, the isotonic knee exercise monitor may not be suitable for measuring ROM. This isotonic knee exercise monitor could detect and collect information on a patient's exercise activity for the benefit of healthcare providers.

Intra-rater reliability of goniometry to measure scapular protraction and retraction

STUDY DESIGN

Observational, descriptive design.

BACKGROUND

Despite scapular mobility being essential for the completion of activities of daily living (ADLs), there is currently no established, a reliable goniometric technique to measure scapular protraction and retraction. A proposed method has shown clinically significant inter-rater reliability for a goniometric technique for these measurements.

PURPOSE

This observational descriptive study examined the intra-rater reliability of a goniometric technique to assess scapular protraction and retraction among a sample of healthy adults.

METHODS

An occupational therapist who is a certified hand therapist (CHT) and an occupational therapy student used goniometry to measure the neutral (resting), protracted, and retracted positions of the right and left scapula for a sample of healthy young adults (n = 54; a total of 108 data points for each measurement). These measurements were compared to analyze intra-rater and inter-rater reliability.

RESULTS

For measurements of the scapula in neutral, protraction, and retraction, the standard error of measure (SEM) for repeat measures by the expert and novice raters was < 4.5° and < 3.9° respectively and ICC values ranged from poor to moderate (0.37-0.63). The SEM for measures between evaluators was < 5.0° and the ICC was poor (0.16-0.35). Minimum detectable change (MDC) values as a percentage of the mean (% MDC) ranged between 15.9 and 43.7% for intra-rater reliability and 21.9-52.8% for inter-rater reliability.

DISCUSSION

The results of the study were mixed; variance of less than 5° for repeat measures by the same rater and measures between raters suggest clinically acceptable reliability. However, variance as a proportion of available motion (%MDC) demonstrates a broader range both above and below the threshold of 30% suggested for rehabilitation assessments.

CONCLUSION

There are few efficient, reliable techniques to measure scapular mobility in clinical practice. The absolute reliability of goniometry to measure scapular protraction and retraction is similar to measurements of other joints. However, additional research and possible refinement of the technique is recommended to further address relative reliability and validity.

Intra-rater reliability and feasibility of the HexCom advanced disease complexity assessment model

Aim

To study the intra-rater reliability and feasibility of the HexCom complexity assessment model by analyzing internal consistency, intra-rater reliability and response time.

Design

Test–retest study with a selection of 11 clinical situations that cover the full scope of situations assessed by the HexCom model and which are responded to individually.

Location

Home care, primary care, hospital and sociosanitary care. Two specialized palliative home care teams (PADES).

Participants

A total of 20 professionals comprising 10 experts in palliative home care (PADES) and 10 professionals from general palliative care (primary care, hospital and sociosanitary care). These professionals came from the fields of family medicine (5), internal medicine (2), geriatrics (2), nursing (9), psychology (1) and social work (1).

Main measurements

Cronbach's alpha, weighted kappa, response time.

Results

Cronbach's alpha of 0.91 for HexCom-Red and 0.87 for HexCom-Clin. Intra-rater reliability ranging from good to very good for HexCom-Red (kappa: 0.78–1) and from moderate to very good for HexCom-Clin (kappa: 0.58–0.91). Average response time of 0:57 for HexCom-Red and 3:80 min for HexComClin.

Conclusions

HexCom-Red and HexCom-Clin are reliable tools and feasible for use by all professionals involved in both general and specialized palliative care at different levels.

The feasibility and reliability of measuring forearm muscle thickness by ultrasound in a geriatric inpatient setting: a cross-sectional pilot study

Background

Given the potential benefits of introducing ultrasound in the clinical assessment of muscle disorders, this study aimed to assess the feasibility and reliability of measuring forearm muscle thickness by ultrasound in a geriatric clinical setting.

Methods

Cross-sectional pilot study in 25 participants (12 patients aged ≥ 70 years in an acute geriatric ward and 13 healthy volunteers aged 25–50 years), assessed by three raters. Muscle thickness measurement was estimated as the distance between the subcutaneous adipose tissue-muscle interface and muscle-bone interface of the radius at 30% proximal of the distance between the styloid process and distal insertion of the biceps brachii muscle of the dominant forearm. Examinations were repeated three times by each rater and intra- and inter-rater reliability was calculated. Feasibility analysis included consideration of technological, economic, legal, operational, and scheduling (TELOS) components.

Results

Mean muscle-thickness measurement difference between groups was 4.4 mm (95% confidence interval [CI] 2.4 mm to 6.3 mm], p < 0.001). Intra-rater reliability of muscle-thickness assessment was excellent, with intraclass correlation coefficient (ICC) of 0.947 (95%CI 0.902 to 0.974), 0.969 (95%CI 0.942 to 0.985), and 0.950 (95%CI 0.907 to 0.975) for observer A, B, and C, respectively. Inter-rater comparison showed good agreement (ICC of 0.873 [95%CI 0.73 to 0.94]). Four of the 17 TELOS components considered led to specific recommendations to improve the procedure’s feasibility in clinical practice.

Conclusion

Our findings suggest that US is a feasible tool to assess the thickness of the forearm muscles with good inter-rater and excellent intra-rater reliability in a sample of hospitalized geriatric patients, making it a promising option for use in clinical practice.

Extraction of gait parameters from marker-free video recordings of Timed Up-and-Go tests: Validity, inter- and intra-rater reliability

BACKGROUND

We study dual-task performance with marker-free video recordings of Timed Up-and-Go tests (TUG) and TUG combined with a cognitive/verbal task (TUG dual-task, TUGdt).

RESEARCH QUESTION

Can gait parameters be accurately estimated from video-recorded TUG tests by a new semi-automatic method aided by a technique for human 2D pose estimation based on deep learning?

METHODS

Thirty persons aged 60-85 years participated in the study, conducted in a laboratory environment. Data were collected by two synchronous video-cameras and a marker-based optoelectronic motion capture system as gold standard, to evaluate the gait parameters step length (SL), step width (SW), step duration (SD), single-stance duration (SSD) and double-stance duration (DSD). For reliability evaluations, data processing aided by a deep neural network model, involved three raters who conducted three repetitions of identifying anatomical keypoints in recordings of one randomly selected step from each of the participants. Validity was analysed using 95 % confidence intervals (CI) and p-values for method differences and Bland-Altman plots with limits of agreement. Inter- and intra-rater reliability were calculated as intraclass correlation coefficients (ICC) and standard errors of measurement. Smallest detectable change was calculated for inter-rater reliability.

RESULTS

Mean ddifferences between video and the motion capture system data for SW, DSD, and SSD were significant (p < 0.001). However, mean differences for all parameters were small (-6.4%-13.0% of motion capture system) indicating good validity. Concerning reliability, almost all 95 % CI of the ICC estimates exceeded 0.90, indicating excellent reliability. Only inter-rater reliability for SW (95 % CI = 0.892;0.973) and one rater's intra-rater reliability for SSD (95 % CI = 0.793;0.951) were lower, but still showed good to excellent reliability.

SIGNIFICANCE

The presented method for extraction of gait parameters from video appears suitable for valid and reliable quantification of gait. This opens up for analyses that may contribute to the knowledge of cognitive-motor interference in dual-task testing.

Measurement error of the Mini-Mental State Examination among individuals with dementia that reside in nursing homes

Few studies have investigated the measurement error of the Mini-Mental State Examination (MMSE) in the same unit of measurement, also known as absolute reliability. This measurement can help determine whether an observed score change for an individual is likely to represent true change. The aim of this study was to investigate the absolute reliability of the MMSE among individuals with dementia that reside in nursing homes. Among 88 participants, 19 (21.6%) were men, 35 (39.8%) had Alzheimer’s disease, 35 (39.8%) had vascular dementia, and the mean age was 84.0 years (range 65–98). The participants were tested and retested with the MMSE within 1–6 days. Both tests were administered by the same assessor at the same time of day. The mean MMSE score was 13.7 (range 0–28). The absolute difference between MMSE scores varied from 0 to 6 points, and the differences did not correlate with the corresponding score means (p = 0.874). The smallest detectable change (SDC) between two measurements was 4.00. The SDC was independent of depression, impaired vision and hearing, delirium within the last week, dementia type and age. However, the SDC was 5.56 among men and 3.50 among women (p = 0.003). In conclusion, for individuals with dementia that reside in nursing homes, it seems like their MMSE score needs to change by four or more points between two measurements in order for their score change to be reliably higher than the measurement error.

Reliability in the assessment of paediatric somatosensory evoked potentials post cardiac arrest

OBJECTIVE

To measure inter- and intra-rater agreement in the interpretation of cortical somatosensory evoked potential (SSEP) components following paediatric cardiac arrest (CA) in multi-professional neurophysiology teams.

METHODS

Thirteen professionals blinded to patient outcome interpreted 96 SSEPs in paediatric patients 24-/48-/72-hours following CA. Of these, 34 were duplicates used to assess intra-rater agreement. Consistent interpretations (absent/present/indeterminate) between scientists (who record/identify SSEP components) and neurophysiologists (who provide prognostic SSEP interpretation) were expressed as percentages. Rates of agreement were calculated using Fleiss' kappa coefficient (K).

RESULTS

Unanimous agreement between professionals was present in 40% (95%CI: 28-54%) of the interpreted SSEPs, with a K value of 0.62 (95%CI: 0.55-0.70) based on average agreement. Agreement was similar between neurophysiologists (K = 0.67; 95%CI: 0.57-0.77) and scientists (K = 0.62; 95%CI: 0.54-0.70) but lower in patients < 2 years old (K = 0.23; 95%CI: 0.14-0.33) and in those with poor outcome (K = 0.21; 95%CI: 0.07-0.35). No SSEP was unanimously interpreted as absent and 92% (95%CI: 89-95%) of duplicate SSEPs were interpreted consistently.

CONCLUSION

Despite substantial agreement when interpreting prognostic SSEPs, this was significantly lower in children with poor outcome and of younger age.

SIGNIFICANCE

Clinicians using SSEPs in the intensive care unit should be aware of the inter-rater variability when interpreting SSEPs as absent.

The inter- and intra-rater reliability of Modified Tardieu Scale in assessing spasticity of knee extensors in patients with multiple sclerosis

INTRODUCTION

Spasticity is one of the most common and disabling symptoms in Multiple sclerosis (MS). The is a clinical tool for assessing spasticity. This study aimed to investigate the inter- and intra-rater reliability of the modified Tardieu scale for assessing knee extensors spasticity in MS patients.

METHODS

Twenty-six patients with MS (12 females and 14 males) with a mean age of 40 ± 11.39 years participated in this study. The extensor muscles of both knees were evaluated using the MTS in two sessions. At first session, two examiners randomly assessed the knee extensor spasticity to study the inter-rater reliability and 3-4 days later the first examiner assessed the patients again, to determine intra-rater reliability. Intra-class Correlation Coefficient (ICC) analysis, two-way random effect model was used to determine the reliability of various components of the modified Tardieu scale.

RESULTS

The inter-rater reliability for quality of muscle reaction of knee extensor muscles was very good (ICC = 0.89) and for the difference between the angle of muscle response (R1) and full range(R2) of movement (R2- R1), as spasticity intensity criterion, was good (ICC = 0.73). ICC values for R2-R1 and muscle response quality assessments by one rater were 0.73 and 0.82, respectively.

CONCLUSION

The findings of the current study showed that the MTS has good to very good inter- and intra-rater reliability for assessing knee extensors spasticity in MS patients.

German version of the Chedoke McMaster arm and hand activity inventory (CAHAI-G): intra-rater reliability and responsiveness

BACKGROUND

The English version of the Chedoke Arm and Hand Activity Inventory is a validated, upper-limb measure with the purpose of assessing functional recovery of the arm and hand after a stroke. A German translation and cross-cultural adaptation was recently produced and demonstrated high validity, inter-rater reliability and internal consistency. As a follow-up, the present study evaluated the intra-rater reliability and responsiveness of the CAHAI-G for the long and all shortened versions.

METHODS

The CAHAI-G and the Action Research Arm Test were assessed on three different measurement events: upon entry (ME1), two to 3 days after entry (ME2), and after three to 4 weeks (ME3). For the intra-rater reliability analysis, the ME1 CAHAI assessments were recorded on video and rated by three therapists to obtain the intraclass coefficients (ICC). The data of all three MEs were analysed in a group of stroke inpatients for the evaluation of responsiveness. To test for responsiveness, the CAHAI-G change data were compared to concurrent instruments: The Global Rating of Change-questionnaire and the Global Rating of Concept-questionnaire. Both served as external criteria. For all CAHAI-G versions (7, 8, 9 or 13 items), the same analysis procedures for the evaluation of the responsiveness parameter were performed.

RESULTS

In total, 27 patients (9 females, age 63 ± 13.7) were enrolled in the study. The ICCs for the intra-rater reliability were calculated to be between 0.988 and 0.998 for all CAHAI versions. Responsiveness parameters were as follows from CAHAI-G 7 to 13: Minimal Detectable Change (MDC₉₀₎ 5.3, 6.0, 6.1, 8.2; Pearson's correlation coefficients CAHAI-Gs with ARAT 0.365, 0.409^*, 0.500^**, 0.597^**. The Area und Under the Curve and the Minimal Clinical Important Difference values for all CAHAI-G versions and the three external criteria ranged between 0.483 to 0.603 and 2.5 to 9.0, respectively.

CONCLUSION

In addition to the high validity, inter-rater reliability and internal consistency, the CAHAI-G revealed high intra-rater reliability. The data also suggest an adequate responsiveness of the CAHAI-G versions 9 and 13.

Inter- and intra-rater reliability of the OCRA checklist method in video-recorded manual work tasks

Observation-based risk assessments are often used for evaluation of biomechanical workload and the condition of the workplace ergonomics. However, the inter- and intra-rater reliability of the methods used for the assessments are insufficiently studied. The OCRA checklist method, included as a reference method in the ISO and CEN standards regarding upper limb repetitive risk assessment, was in this study investigated regarding its reliability. Eleven ergonomists were educated within the method. Each ergonomist risk-assessed ten video-recorded manual work tasks, on two occasions. The average inter-rater agreement of the five overall risk levels was 39% and the linearly weighted kappa was 0.43, the intra-rater dittos were 45% and 0.52. Both indicating a moderate reliability, but lower than what could be expected. The levels might be sufficient for coarse surveys. However, for risk assessments before and after an intervention, complementing technical methods should be considered.

Creating a healthy eating and active environment survey (CHEERS) for childcare: an inter-rater, intra-rater reliability and validity study

BACKGROUND

The CHEERS is a self-administered tool to measure gaps, weaknesses, and strengths of an early childhood education and care (ECEC) centre-based nutrition and physical activity environment. ECEC settings have the potential to profoundly influence early dietary and physical activity behaviours. Content validation of the CHEERS tool has been previously reported. The purpose of this study was to develop reliability and validity evidence for the CHEERS audit tool and the proposed subscales of food served, healthy eating environment, program planning, and physical activity environment in ECEC centre-based programs.

METHODS

This cross-sectional study consisted of 2 phases: Phase 1 included inter-, intra-rater and Cronbach's α. A subset of this sample was invited to participate in a second survey (Trial 2) for intra-rater assessment within 3 weeks of completing the first survey (Trial 1). Phase 2 included concurrent validity assessment between a nutrition expert and the ECEC director using within a one-week period.

RESULTS

One hundred two directors and 85 educators (total of 187) returned the survey. Of these, there were 75 matched pairs for inter-rater reliability analysis providing a CHEERS ICC score of 0.59 and ICC scores ranging from 0.40 to 0.58 for the subscales. The ICC for intra-rater reliability of the CHEERS score was 0.81 for 40 participants completing the survey a second time and a range of 0.72 to 0.79 for the subscales. The CHEERS tool demonstrated very good internal consistency (α = 0.91) and α scores ranging from 0.73 to 0.79 for the subscales. In phase 2, concurrent validation was ICC = 0.65 (n = 30) CHEERS scores with a range of 0.42 to 0.69 for the subscales.

CONCLUSIONS

This study provides evidence of inter-, intra-rater reliability, internal consistency, and concurrent validity of an environmental assessment audit tool to assess the nutrition and physical activity environment of ECEC centre-based programs. The results demonstrate that the self-administered CHEERS instrument is stable overtime and between evaluators at the same ECEC centre. The scores obtained with CHEERS self-administered audit tool are reasonably accurate compared to an expert rater (dietitian) assessment. This study adds additional support to establishing the psychometric soundness of the CHEERS tool.

Are Ultrasound Measurements of Achilles Tendon Size Reliable? A Systematic Review of Rater Reliability

Ultrasound measurements of Achilles tendon size are used to assess the tendon's response to exercise, aging, rehabilitation, tendon loading and healing. It is important to understand and minimise the measurement error that occurs with these measurements. This review identified and synthesised studies reporting on intra- and inter-rater reliability of ultrasound measurements of Achilles tendon size. Analysis of 21 studies revealed that good to excellent intra- and inter-rater reliability can be achieved for ultrasound measurements of Achilles tendon size. Reliability can be optimised by using one experienced operator, standardising transducer pressure and orientation and averaging two or three measurements. There was a high risk of methodological bias across the included studies. Reporting of reliability studies needs to be improved by the use of existing reporting guidelines and expansion of these guidelines to include important elements of ultrasound imaging.

Development of the Finnish neurological function testing battery for dogs and its intra- and inter-rater reliability

Background

The Finnish neurological function testing battery for dogs (FINFUN) was developed to meet the increasing demand for objective outcome measures in veterinary physiotherapy. The testing battery should provide consistent, reproducible results and have established face and content validity. Internal consistency and intra- and inter-rater reliability of the FINFUN were also investigated.

Results

The FINFUN comprised 11 tasks: lying, standing up from lying, sitting, standing up from sitting, standing, proprioceptive positioning, starting to walk, walking, trotting, walking turns and walking stairs. A score from 0 to 4, (0: unable to perform task, 4: performing task with normal motor function) was given for each task, the maximum score being 44. Twenty-six dogs were filmed when performing the FINFUN. Seven observers scored the performances from the video recordings. The FINFUN was considered to have appropriate face and content validity based on a pilot study, clinical experience and critical reflection of the development process. Its internal consistency was excellent, with no Cronbach’s alpha values below 0.922. The intra-rater reliability for total score of experienced observers was almost perfect: 0.999 (observer 1) and 0.994 (observer 2). The inter-rater reliability for both experienced and novice observers’ total scores was also almost perfect (0.919–0.993). Analysis of each individual task showed substantial intra-rater and inter-rater agreement for the tasks “lying” and “sitting”.

Conclusions

The FINFUN is an objective, valid and reliable tool with standardized scoring criteria for evaluation of motor function in dogs recovering from spinal cord injury.

Electronic supplementary material

The online version of this article (10.1186/s13028-018-0408-2) contains supplementary material, which is available to authorized users.

Is the reliability of 3D kinematics of young obese participants dependent on the hip joint center localization method used?

The aim of this study was to investigate if the test-retest reliability for three-dimensional (3D) gait kinematics in a young obese population is affected by using either a predictive (Davis) or a functional (SCoRE) hip joint center (HJC) localization approach. A secondary goal was to analyze how consistent both methods perform in estimating the HJC position. A convenience sample of ten participants, two females and eight males with an age-based body mass index (BMI) above the 97th percentile (mean±SD: 34.2±3.9kg/m²) was recruited. Participants underwent two 3D gait analysis sessions separated by a minimum of one day and a maximum of seven days. The standard error of measurement (SEM) and the root mean square error (RMSE) of key kinematic parameters along with the root mean square deviation (RMSD) of the entire waveforms were used to analyze the test-retest reliability. To get an estimate of the consistency of both HJC localization methods, the HJC positions determined by both methods were compared to each other. SEM, RMSE, and RMSD results indicate that the HJC position estimations between both methods are not different and demonstrate moderate to good reliability to estimate joint kinematics. With respect to the localization of the HJC, notable inconsistencies ranging from 0 to 5.4cm were observed. In conclusion, both approaches appear equally reliable. However, the inconsistent HJC estimation points out, that accuracy seems to be a big issue in these methods. Future research should attend to this matter.

The reliability of flexible nasolaryngoscopy in the identification of vocal fold movement impairment in young infants

OBJECTIVE

Flexible nasolaryngoscopy (FNL) is considered the gold standard for evaluation of vocal fold mobility but there has been no data on the reliability of interpretation in the infant population. Visualization may be limited by excessive movement, secretions, or floppy supraglottic structures that prevent accurate diagnosis of vocal fold movement impairment (VFMI). We sought to evaluate the inter- and intra-rater reliability of FNL for the evaluation of VFMI in young infants.

STUDY TYPE

Case-control.

METHODS

Twenty infants were identified: 10 with VFMI and 10 normal as seen on FNL. Three pediatric otolaryngologists reviewed the video without sound and rated the presence and/or degree of vocal fold mobility. Twelve videos were repeated to assess intra-rater reliability.

RESULTS

There was substantial agreement between the reviewers regarding the identification normal vs. any type of VFMI (kappa = 0.67) but only moderate agreement regarding the degree of vocal fold movement (kappa = 0.49). Intra-rater reliability ranges from moderate to perfect agreement (kappa = 0.48-1).

CONCLUSION

FNL in infants is an extremely challenging procedure. Clinically, physicians frequently use the quality of the cry and the past medical and surgical history to help make a judgment of vocal fold movement when the view is suboptimal. These other factors, however, may bias the interpretation of the FNL. Without sound, there is only moderate inter-rater and variable intra-rater reliability for the identification of degree of movement on FNL. Otolaryngologists must be cognizant of the limitations of FNL when using it as a clinical tool or as a "gold standard" against which other modalities are measured.

Within-assessor reliability and minimal detectable change of gait kinematics in a young obese demographic

INTRODUCTION

Three-dimensional gait analysis (3DGA) in obese populations is a difficult task due to a great amount of subcutaneous fat. This makes it more challenging to identify anatomical landmarks, thus leading to inconsistent marker placement. Therefore, the purpose of this study was to investigate the test-retest reliability for kinematic measurements of obese children and adolescents.

METHODS

Nine males and two females with an age-based BMI above the 97th percentile (age: 14.6±2.6years, BMI: 33.4±4.4kg/m²) were administered to two 3DGA sessions. To quantify reliability of discrete parameters the intraclass correlation coefficient (ICC_2,k), standard error of measurement (SEM) and minimal detectable change (MDC) were calculated. To quantify waveform similarity, the coefficient of multiple correlation (CMC) and the linear fit method (LFM) were used.

RESULTS

From 28 kinematic parameters, 23 showed acceptable ICCs (≥0.70) and the remaining parameters demonstrated moderate values. These were peak hip extension during stance (0.58), mean pelvis rotation (0.60), mean anterior pelvic tilt (0.64), peak knee flexion during swing (0.67) and peak hip abduction during swing (0.69). The SEM was below 5° for all parameters. The MDC for the sagittal, frontal, and transversal plane were on average 7.5°±2.2, 4.6°±1.3 and 6.0°±0.9 respectively. Both the LFM and CMC showed, in general, moderate to good reliability except for pelvis tilt and hip rotation.

CONCLUSION

Data demonstrated acceptable error margins especially for the sagittal and frontal plane. Low reliability for the pelvis tilt indicates that great effort is necessary to position the pelvic markers consistently during repeated sessions.

Reproducibility of African giant pouched rats detecting Mycobacterium tuberculosis

Background

African pouched rats sniffing sputum samples provided by local clinics have significantly increased tuberculosis case findings in Tanzania and Mozambique. The objective of this study was to determine the reproducibility of rat results.

Methods

Over an 18-month period 11,869 samples were examined by the rats. Intra-rater reliability was assessed through Yule’s Q. Inter-rater reliability was assessed with Krippendorff’s alpha.

Results

Intra-rater reliability was high, with a mean Yule’s Q of 0.9. Inter-rater agreement was fair, with Krippendorf’s alpha ranging from 0.15 to 0.45. Both Intra- and Inter-rater reliability was independent of the sex of the animals, but they were positively correlated with age. Both intra- and inter-rater agreement was lowest for samples designated as smear-negative by the clinics.

Conclusion

Overall, the reproducibility of tuberculosis detection rat results was fair and diagnostic results were therefore independent of the rats used.

Electronic supplementary material

The online version of this article (doi:10.1186/s12879-017-2347-3) contains supplementary material, which is available to authorized users.

Six of one, half a dozen of the other: A measure of multidisciplinary inter/intra-rater reliability of the society for fetal urology and urinary tract dilation grading systems for hydronephrosis

INTRODUCTION

The urinary tract dilation (UTD) classification system was introduced to standardize terminology in the reporting of hydronephrosis (HN), and bridge a gap between pre- and postnatal classification such as the Society for Fetal Urology (SFU) grading system. Herein we compare the intra/inter-rater reliability of both grading systems.

MATERIALS AND METHODS

SFU (I-IV) and UTD (I-III) grades were independently assigned by 13 raters (9 pediatric urology staff, 2 nephrologists, 2 radiologists), twice, 3 weeks apart, to 50 sagittal postnatal ultrasonographic views of hydronephrotic kidneys. Data regarding ureteral measurements and bladder abnormalities were included to allow proper UTD categorization. Ten images were repeated to assess intra-rater reliability. Krippendorff's alpha coefficient was used to measure overall and by grade intra/inter-rater reliability. Reliability between specialties and training levels were also analyzed.

RESULTS

Overall inter-rater reliability was slightly higher for SFU (α = 0.842, 95% CI 0.812-0.879, in session 1; and α = 0.808, 95% CI 0.775-0.839, in session 2) than for UTD (α = 0.774, 95% CI 0.715-0.827, in session 1; and α = 0.679, 95% CI 0.605-0.750, in session 2). Reliability for intermediate grades (SFU II/III and UTD 2) of HN was poor regardless of the system. Reliabilities for SFU and UTD classifications among Urology, Nephrology, and Radiology, as well as between training levels were not significantly different.

DISCUSSION

Despite the introduction of HN grading systems to standardize the interpretation and reporting of renal ultrasound in infants with HN, none have been proven superior in allowing clinicians to distinguish between "moderate" grades. While this study demonstrated high reliability in distinguishing between "mild" (SFU I/II and UTD 1) and "severe" (SFU IV and UTD 3) grades of HN, the overall reliability between specialties was poor. This is in keeping with a previous report of modest inter-rater reliability of the SFU system. This drawback is likely explained by the subjective interpretation required to assign grades, which can be impacted by experience, image quality, and scanning technique. As shown in the figure, which demonstrates SFU II (a) and SFU III (b), as assigned by a radiologist, it is possible to make an argument that either of these images can be classified into both categories that were observed during the grading sessions of this study.

CONCLUSION

Although both systems have acceptable reliability, the SFU grading system showed higher overall intra/inter-rater reliability regardless of rater specialty than the UTD classification. Inter-rater reliability for SFU grades II/III and UTD 2 was low, highlighting the limitations of both classifications in regards to properly segregating moderate HN grades.

Home-based anthropometric, blood pressure and pulse measurements in young children by trained data collectors in the National Children's Study

OBJECTIVE

The current study assessed whether home-based data collection by trained data collectors can produce high-quality physical measurement data in young children.

DESIGN

The study assessed the quality of intra-examiner measurements of blood pressure, pulse rate and anthropometric dimensions using intra-examiner reliability and intra-examiner technical error of measurement (TEM).

SETTING

Non-clinical, primarily private homes of National Children's Study participants in twenty-two study locations across the USA.

SUBJECTS

Children in four age groups: 5-7 months (n 91), 11-16 months (n 393), 23-28 months (n 1410) and 35-40 months (n 800).

RESULTS

Absolute TEM ranged in value from 0·09 to 16·21, varying widely by age group and measure, as expected. Relative TEM spanned from 0·27 to 13·71 across age groups and physical measures. Reliabilities for anthropometric measurements by age group and measure ranged from 0·46 to >0·99 with most exceeding 0·90, suggesting that the large majority of anthropometric measures can be collected in a home-based setting on young children by trained data collectors. Reliabilities for blood pressure and pulse rate measurements by age group ranged from 0·21 to 0·74, implying these are less reliably measured with young children when taken in the data collection context described here.

CONCLUSIONS

Reliability estimates >0·95 for weight, length, height, and thigh, waist and head circumference, and >0·90 for triceps and subscapular skinfolds, indicate that these measures can be collected in the field by trained data collectors without compromising data quality. These estimates can be used for interim evaluations of data collector training and measurement protocols.

Intra-rater reliability when using a tympanic thermometer under different self-measurement conditions

[Purpose] This study investigated intra-rater reliability when using a tympanic thermometer under different self-measurement conditions. [Subjects and Methods] Ten males participated. Intra-rater reliability was assessed by comparing the values under three conditions of measurement using a tympanic thermometer. Intraclass correlation coefficients were used to assess intra-rater reliability. [Results] According to the intraclass correlation coefficient analysis, reliability could be ranked according to the conditions of measurement. [Conclusion] The results showed that self-measurement of body temperature is more precise when combined with common sense and basic education about the anatomy of the eardrum.

Measurement of morphological and physiological skin properties in aged care residents: a test-retest reliability pilot study

This test-retest pilot study investigated the intra-rater reliability and reproducibility of non-invasive technologies to objectively quantify morphological (colour, thickness and elasticity) and physiological (transepidermal water loss (TEWL), hydration, sebum and pH) skin properties in an aged care population. Three consecutive measurements were taken from five anatomical skin sites, with the mean of each measurement calculated. The intra-class correlation coefficient (ICC) and the standard error of measurement (SEM) were used to examine the intra-rater reliability and reproducibility of measurements. Non-invasive technologies in this study showed almost perfect reliability for ultrasound measurements of the subepidermal low echogenicity band (SLEB) (ρ = 0·95-0·99) and skin thickness (ρ = 0·95-0·99) across all sites. The ICC was substantial to almost perfect for pH (ρ = 0·76-0·88) and viscoelasticity (ρ = 0·67-0·91) across all sites. Hydration (ρ = 0·53-0·85) and skin retraction (ρ = 0·57-0·99) measurements ranged from moderate to almost perfect across all sites. TEWL and elasticity were substantial to almost perfect across four sites. Casual sebum levels and most colour parameters showed poor ICC. The use of non-invasive technologies in this study provided an objective and reliable means for quantifying ageing skin and may offer future studies a valuable option for assessing skin tear risk.

Intra-rater reliability of hallux flexor strength measures using the Nintendo Wii Balance Board

Background

The purpose of this study was to investigate the intra-rater reliability of a new method in combination with the Nintendo Wii Balance Board (NWBB) to measure the strength of hallux flexor muscle.

Methods

Thirty healthy individuals (age: 34.9 ± 12.9 years, height: 170.4 ± 10.5 cm, weight: 69.3 ± 15.3 kg, female = 15) participated. Repeated testing was completed within 7 days. Participants performed strength testing in sitting using a wooden platform in combination with the NWBB. This new method was set up to selectively recruit an intrinsic muscle of the foot, specifically the flexor hallucis brevis muscle. Statistical analysis was performed using intra-class coefficients and ordinary least product analysis. To estimate measurement error, standard error of measurement (SEM), minimal detectable change (MDC) and percentage error were calculated.

Results

Results indicate excellent intra-rater reliability (ICC = 0.982, CI = 0.96-0.99) with an absence of systematic bias. SEM, MDC and percentage error value were 0.5, 1.4 and 12 % respectively.

Conclusions

This study demonstrates that a new method in combination with the NWBB application is reliable to measure hallux flexor strength and has potential to be used for future research and clinical application.

Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events

Objectives:

The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events.

Methods:

We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later.

Results:

In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89).

Conclusions:

In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.

Intra-rater reliability of an experienced physiotherapist in locating myofascial trigger points in upper trapezius muscle

OBJECTIVES

Myofascial trigger points (MTrPs) are considered the principal clinical feature of myofascial pain syndrome (MPS). An MTrP consists of spot tenderness within a taut band of muscle fibers and its stimulation can produce both local and referred pain. The clinical diagnosis of MPS depends on correct history taking and a physical examination aimed at identifying the presence of MTrP. The purpose of this study was to investigate the intra-rater reliability of a palpation protocol used for locating an MTrP in the upper trapezius muscle.

METHODS

Twenty-four subjects with MTrP in the upper trapezius muscle were examined by an experienced physiotherapist. During each of eight experimental sessions, subjects were examined twice in randomized order using a palpation protocol. An anatomical landmark system was defined and the MTrP location established using X and Y values.

RESULTS

The intraclass correlation coefficient ICC(1,1) values were 0.62 (95% CI: 0.30-0.81) for X and 0.81 (95% CI: 0.61-0.91) for Y. The Bland-Altman plots for X and Y showed a mean of difference of 0.04 and -0.2 mm, respectively. Limits of agreement for X ranged from -26.3 to 26.2 mm and for Y from -27 to 26.4 mm.

DISCUSSION

The ICC(1,1) for the observed values revealed a moderate to high correlation and the Bland-Altman analysis showed means of difference very close to zero with narrow limits of agreement. An experienced physiotherapist can reliably identify MTrP locations in upper trapezius muscle using a palpation protocol.