Statistical methodology: II. Reliability and validity assessment in study design, Part B

Inconclusiveness of psychometric testing of medication adherence questionnaires

PURPOSE

To propose a paradigm change for the validation procedures of medication adherence questionnaires.

METHODS

A total of 121 validation procedures of unique questionnaires for medication adherence were analyzed.

RESULTS

"Construct validity" and "internal consistency" were most often assessed, and test results varied largely. A more in-depth analysis indicated that the assessment of medication non-adherence included distinct but related constructs, such as the extent to which doses are missed, and the attempt to identify different facets of medication-taking behavior. Consequently, each construct requires a different measurement approach with different psychometric tests for establishing its validity and reliability.

CONCLUSION

Results show that assessing the validity and reliability of adherence questionnaires with standard procedures including statistical tests is inconclusive. Refinement of the constructs of non-adherence is needed in pharmacy and medical practice. We suggest a distinction between the (i) extent of missed doses over the past 2 weeks, (ii) modifiable reasons for non-adherence behavior, and (iii) unmodifiable factors of non-adherence. Validation procedures and corresponding statistical methods should be selected according to the specific single constructs.

Emergency Medicine Postgraduate Year, Laryngoscopic View, and Endotracheal Tube Placement Success

STUDY OBJECTIVE

Prior work has found first-attempt success improves with emergency medicine (EM) postgraduate year (PGY). However, the association between PGY and laryngoscopic view - a key step in successful intubation - is unknown. We examined the relationship among PGY, laryngoscopic view (ie, Cormack-Lehane view), and first-attempt success.

METHODS

We performed a retrospective analysis of the National Emergency Airway Registry, including adult intubations by EM PGY 1 to 4 resident physicians. We used inverse probability weighting with propensity scores to balance confounders. We used weighted regression and model comparison to estimate adjusted odds ratios (aOR) with 95% confidence intervals (CIs) between PGY and Cormack-Lehane view, tested the interaction between PGY and Cormack-Lehane view on first-attempt success, and examined the effect modification of Cormack-Lehane view on the association between PGY and first-attempt success.

RESULTS

After exclusions, we included 15,453 first attempts. Compared to PGY 1, the aORs for a higher Cormack-Lehane grade did not differ from PGY 2 (1.01; 95% CI 0.49 to 2.07), PGY 3 (0.92; 0.31 to 2.73), or PGY 4 (0.80; 0.31 to 2.04) groups. The interaction between PGY and Cormack-Lehane view was significant (P-interaction<0.001). In patients with Cormack-Lehane grade 3 or 4, the aORs for first-attempt success were higher for PGY 2 (1.80; 95% CI 1.17 to 2.77), PGY 3 (2.96; 1.66 to 5.27) and PGY 4 (3.10; 1.60 to 6.00) groups relative to PGY 1.

CONCLUSION

Compared with PGY 1, PGY 2, 3, and 4 resident physicians obtained similar Cormack-Lehane views but had higher first-attempt success when obtaining a grade 3 or 4 view.

Risk of Major Adverse Cardiovascular Events After Emergency Department Visits for Hypertensive Urgency

BACKGROUND

Chronic hypertension is an established long-term risk factor for major adverse cardiovascular events (MACEs). However, little is known about short-term MACE risk after hypertensive urgency, defined as an episode of acute severe hypertension without evidence of target-organ damage. We sought to evaluate the short-term risk of MACE after an emergency department (ED) visit for hypertensive urgency resulting in discharge to home.

METHODS

We performed a case-crossover study using deidentified administrative claims data. Our case periods were 1-week intervals from 0 to 12 weeks before hospitalization for MACE. We compared ED visits for hypertensive urgency during these case periods versus equivalent control periods 1 year earlier. Hypertensive urgency and MACE components were all ascertained using previously validated International Classification of Diseases, Tenth Revision Clinical Modification codes. We used McNemar test for matched data to calculate risk ratios.

RESULTS

Among 2 225 722 patients with MACE, 1 893 401 (85.1%) had a prior diagnosis of hypertension. There were 4644 (0.2%) patients who had at least 1 ED visit for hypertensive urgency during the 12 weeks preceding their MACE hospitalization. An ED visit for hypertensive urgency was significantly more common in the first week before MACE compared with the same chronological week 1 year earlier (risk ratio, 3.5 [95% CI, 2.9-4.2]). The association between hypertensive urgency and MACE decreased in magnitude with increasing temporal distance from MACE and was no longer significant by 11 weeks before MACE (risk ratio, 1.2 [95% CI, 0.99-1.6]).

CONCLUSIONS

ED visits for hypertensive urgency were associated with a substantially increased short-term risk of subsequent MACE.

Development of the cervical myelopathy severity index: a new patient reported outcome measure to quantify impairments and functional limitations

BACKGROUND CONTEXT

Existing degenerative cervical myelopathy (DCM) severity scales have significant shortcomings, creating a strong impetus for the development of a practical measurement tool with sound psychometric properties.

PURPOSE

This work reports the item generation and reduction of the Cervical Myelopathy Severity Index (CMSI), a new DCM patient-reported outcome measure of symptoms and functional limitations.

DESIGN

Prospective observational study.

PATIENT SAMPLE

Adult DCM patients belonging to one of three distinct treatment groups: (1) observation cohort, (2) preoperative surgical cohort, (3) 6 to 12 months postoperative cohort.

OUTCOME MEASURES

Patient-reported outcome measure of symptoms and functional limitations.

METHODS

Item generation was performed using semi-structured patient focus groups emphasizing symptoms experienced and functional limitations. Readability was assessed through think-aloud patient interviews. Item reduction involved surveys of DCM patients with a spectrum of disease severity and board-certified spine surgeons experienced in the treatment of DCM. A priori criteria for item removal included: patient median importance/severity <2 (of 4), 30% or more no severity (response of zero), item severity correlations ≤ 0.80 (Spearman), item severity reliability (weighted kappa <0.60) based on a 2-week interval and clinician median importance <2 with retention of items with very high clinical importance.

RESULTS

There were 42 items generated from a combination of specialist input and patient focus groups. Items captured sensorimotor symptoms and limitations related to upper and lower extremities as well as sphincter dysfunction. Ninety-eight patients (43, 30, 25 observation, pre- and postsurgery respectively) and 51 surgeons completed the assessment. Twenty-three items remained after application of median importance and severity thresholds and weighted kappa cutoffs. After elimination of highly correlated (>0.80) items and combining two similar items, the final CMSI questionnaire list included 14 items.

CONCLUSIONS

The CMSI is a new DCM patient-reported clinical measurement tool developed using patient and clinician input to inform item generation and reduction. Future work will evaluate the reliability, validity, and responsiveness of the CMSI in relation to existing myelopathy measurement indices.

Design and Validation of a Soft Robotic Simulator for Transseptal Puncture Training

OBJECTIVE

Transseptal puncture (TP) is the technique used to access the left atrium of the heart from the right atrium during cardiac catheterization procedures. Through repetition, electrophysiologists and interventional cardiologists experienced in TP develop manual skills to navigate the transseptal catheter assembly to their target on the fossa ovalis (FO). Cardiology fellows and cardiologists that are new to TP currently train on patients to develop this skill, resulting in increased risk of complications. The goal of this work was to create low-risk training opportunities for new TP operators.

METHODS

We developed a Soft Active Transseptal Puncture Simulator (SATPS), designed to match the dynamics, static response, and visualization of the heart during TP. The SATPS includes three subsystems: (i) A soft robotic right atrium with pneumatic actuators mimics the dynamics of a beating heart. (ii) A fossa ovalis insert simulates cardiac tissue properties. (iii) A simulated intracardiac echocardiography environment provides live visual feedback. Subsystem performance was verified with benchtop tests. Face and content validity were evaluated by experienced clinicians.

RESULTS

Subsystems accurately represented atrial volume displacement, tenting and puncture force, and FO deformation. Passive and active actuation states were deemed suitable for simulating different cardiac conditions. Participants rated the SATPS as realistic and useful for training cardiology fellows in TP.

CONCLUSION

The SATPS can help improve catheterization skills of novice TP operators.

SIGNIFICANCE

The SATPS could provide novice TP operators the opportunity to improve their TP skills before operating on a patient for the first time, reducing the likelihood of complications.

Optimizing measurement of misdiagnosis-related harms using symptom-disease pair analysis of diagnostic error (SPADE): comparison groups to maximize SPADE validity

Diagnostic errors in medicine represent a significant public health problem but continue to be challenging to measure accurately, reliably, and efficiently. The recently developed Symptom-Disease Pair Analysis of Diagnostic Error (SPADE) approach measures misdiagnosis related harms using electronic health records or administrative claims data. The approach is clinically valid, methodologically sound, statistically robust, and operationally viable without the requirement for manual chart review. This paper clarifies aspects of the SPADE analysis to assure that researchers apply this method to yield valid results with a particular emphasis on defining appropriate comparator groups and analytical strategies for balancing differences between these groups. We discuss four distinct types of comparators (intra-group and inter-group for both look-back and look-forward analyses), detailing the rationale for choosing one over the other and inferences that can be drawn from these comparative analyses. Our aim is that these additional analytical practices will improve the validity of SPADE and related approaches to quantify diagnostic error in medicine.

Towards better understanding the stiffness of nanocomposites via parametric study of an analytical model modeling parameters and experiments

The stiffness of polymeric materials can be improved dramatically with the addition of nanoparticles. In theory, as the nanoparticle loading in the polymer increases, the nanocomposite becomes stiffer; however, experiments suggest that little or no stiffness improvement is observed beyond an optimal nanoparticle loading. The mismatch between the theoretical and experimental findings, particularly at high particle loadings, needs to be understood for the effective use of nanoparticles. In this respect, we have recently developed an analytical model to close the gap in the literature and predict elastic modulus of nanocomposites. The model is based on a three-phase Mori-Tanaka model coupled with the Monte-Carlo method, and satisfactorily captures the experimental results, even at high nanoparticle loadings. The developed model can also be used to study the effects of agglomeration in nanocomposites. In this paper, we use this model to study the effects of agglomeration and related model parameters on the stiffness of nanocomposites. In particular, the effects of particle orientation, critical distance, dispersion state and agglomerate property, and particle aspect ratio are investigated to demonstrate capabilities of the model and to observe how optimal particle loading changes with respect these parameters. The study shows that the critical distance defining agglomerates and the properties of agglomerates are the key design parameters at high particle loadings. These two parameters rule the optimal elastic modulus with respect to particle loading. The findings will allow researchers to form design curves and successfully predict the elastic moduli of nanocomposites without the exhaustive experimental undertakings.

Validity of a Wheelchair Rugby Field Test to Simulate Physiological and Thermoregulatory Match Outcomes

The purpose of the study was to verify the criterion-validity (concurrent) of an existing and reliable, submaximal wheelchair Rugby (WCR) field test by examining the correlations of selected measures of physical performance between the field test and real games. Therefore, ten WCR athletes were observed during two WCR real games and during completing the field test two times. Total distance, mean and peak velocity, playing time, number of sprints, sprints per minute, mean and maximal heart rate, body core temperature (Tc), sweat rate, body weight loss, rate of perceived exertion and thermal sensation were measured. Values were correlated with the data observed by completing the field test two times separated by seven days. The results showed significant correlations between games and field tests for sweat rate (r = 0.740, p < 0.001), body weight loss (r = 0.732, p < 0.001) and the increase of Tc (r = 0.611, p = 0.009). All other correlations were not significant. For perceptual responses Bland−Altman analysis showed data within the limits of agreement. Descriptive statistics showed similarity for mean velocity and total distance between tests and games. In conclusion the study provides the first indications that the submaximal field test seems comparable with the game outcomes in terms of increase in Tc, covered distance, mean velocity and perceptual responses. Nevertheless, more research and additional validation are required.

Criterion Validity of the Newly Developed Occlusal Cant Index

OBJECTIVES

To assess the criterion-related (concurrent) validity of the newly developed occlusal cant index (OCI).

MATERIALS AND METHODS

Four standardized posterior-anterior (PA) cephalometric radiographs of four patients were obtained at a 0° occlusal cant (OC) and manipulated to create various degrees of OC from 1° to 4° on the right and left sides, with a total of 36 PA images. The angle between the actual horizontal line and the occlusal plane was manually drawn on each PA radiographic image. The set of radiographic images was displayed to 36 orthodontists, who were asked to measure the drawn angle and apply the OCI to each PA radiographic image.

RESULTS

The overall criterion-related validity of the OCI was statistically significant among all grades.

CONCLUSION

The OCI is highly valid and recommended for clinical consideration.

A predictive model towards understanding the effect of reinforcement agglomeration on the stiffness of nanocomposites

Nanocomposite technologies can be significantly enhanced through a careful exploration of the effects of agglomerates on mechanical properties. Existing models are either overly simplified (e.g., neglect agglomeration effects) or often require a significant amount of computational resources. In this study, a novel continuum-based model with a statistical approach was developed. The model is based on a modified three-phase Mori-Tanaka model, which accounts for the filler, agglomerate, and matrix regions. Fillers are randomly dispersed in a defined space to predict agglomeration tendency. The proposed model demonstrates good agreement with the experimentally measured elastic moduli of spin-coated cellulose nanocrystal reinforced polyamide-6 films. The techniques and methodologies presented in the study are sufficiently general in that they can be extended to the analyses of various types of polymeric nanocomposite systems.

The promise of computer adaptive testing in collection of orthopaedic outcomes: an evaluation of PROMIS utilization

Background

A crucial component to improving patient care is better clinician understanding of patients’ health-related quality of life (HRQoL). In orthopaedic surgery, HRQoL assessment instruments such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), provide surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s HRQoL. PROMIS has been demonstrated as a valuable instrument in many diseases; however, the extent to which orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research is unclear.

Methods

Systematic scoping methodology was used to investigate the characteristics of studies using PROMIS to assess HRQoL measures as orthopaedic surgical outcomes as well as studies validating computerized adaptive test (CAT) PROMIS physical health (PH) domains including: Physical Function (PF), Upper Extremity (UE), Lower Extremity (LE).

Results

A systematic search of PubMed identified 391 publications utilizing PROMIS in orthopaedics; 153 (39%) were PROMIS PH CAT validation publications. One-hundred publications were in Hand and Upper Extremity, 69 in Spine, 44 in Adult Reconstruction, 43 in Foot and Ankle, 43 in Sports, 37 in Trauma, 31 in General orthopaedics, and 24 in Tumor. From 2011 through 2020 there was an upward trend in orthopaedic PROMIS publications each year (range, 1–153) and an increase in studies investigating or utilizing PROMIS PH CAT domains (range, 1–105). Eighty-five percent (n = 130) of orthopaedic surgery PROMIS PH CAT validation publications (n = 153) analyzed PF; 30% (n = 46) analyzed UE; 3% (n = 4) analyzed LE.

Conclusions

PROMIS utilization within orthopaedics as a whole has significantly increased within the past decade, particularly within PROMIS CAT domains. The existing literature reviewed in this scoping study demonstrates that PROMIS PH CAT domains (PF, UE, and LE) are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. The expanded use of PROMIS CATs in orthopaedic surgery highlights the potential for improved quality of patient care. While challenges of integrating PROMIS into electronic medical records exist, expanded use of PROMIS CAT measurement instruments throughout orthopaedic surgery should be performed.

Plain english summary In orthopaedic surgery, health-related quality of life tools such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), offer patients an opportunity to better understand their medical condition and be involved in their own care. Additionally, PROMIS provides surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s functional status and quality of life. The efficacy of PROMIS has been demonstrated in many diseases; however, its application throughout orthopaedic care has yet to be depicted. This study sought to identify the extent to which all orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research in order to identify its potential as an applicable and valuable tool across specialties. We determined that PROMIS utilization has significantly increased within the past decade. The existing literature reviewed in this scoping study demonstrates that the PROMIS computerized adaptive test domains evaluating physical function status are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. Based on these results, this study recommends the expanded and more uniform use of PROMIS computerized adaptive test measurement instruments in the clinical care of orthopaedic patients.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41687-021-00407-w.

Reliability, Validity, and Responsiveness of the Chinese Learning Accomplishment Profile (C-LAP)

Objectives: To evaluate the reliability, validity, and responsiveness of the Chinese Learning Accomplishment Profile in China. Methods: 12,098 participants aged from 0 to 36 months from 30 provinces (mostly from Shanghai) in China were enrolled between 2013 and 2020. The reliability was reflected by Pearson correlation coefficients, Cronbach’s alpha coefficients and standard errors; the validity was shown by the coefficients between the dimensions, and we also evaluated the responsiveness as a supplement to the validity. Results: Reliability: in six domains among each subgroup, Pearson correlation coefficients between developmental age and chronological age ranged from 0.89 to 0.98, Cronbach’s alpha coefficients from 0.71 to 0.99, and standard errors from 0.15 to 2.76. Validity: after controlling for chronological age, the correlation coefficients between the dimensions were between 0.18 and 0.78, and most of them were below 0.70. Responsiveness: developmental age of all domains obtained via the Chinese Learning Accomplishment Profile system changed significantly (p < 0.001) with time (gap of 1–3 months), and the standardized response mean ranged from 0.66 to 2.45. Conclusions: The Chinese Learning Accomplishment Profile is suitable for assessing children’s development in Shanghai, but still needs confirmation when used in other provinces in China due to the great differences between regions in China.

[Validity and reliability of the Nijmegen Cochlear Implant Questionnaire in German]

Hintergrund

Der Nijmegen Cochlear Implant Questionnaire (NCIQ) ist ein krankheitsspezifischer Fragebogen zur Erhebung der gesundheitsbezogenen Lebensqualität von Patienten vor und nach Cochleaimplantation.

Ziel der Arbeit

Validierung und Reliabilitätsprüfung der deutschen Übersetzung des NCIQ.

Material und Methoden

Es wurde eine prospektive Studie an 100 postlingual ertaubten oder hochgradig schwerhörigen Patienten durchgeführt, welche präoperativ sowie 3 und 6 Monate nach einer Cochleaimplantation mittels NCIQ, Abbreviated Profile of Hearing Aid Benefit (APHAB) und Hearing Participation Scale (HPS) untersucht wurden. Als Kontrolle fungierte ein postlingual ertaubtes oder hochgradig schwerhöriges, unbehandeltes Patientenkollektiv (n = 54). Cronbach‑α und Test-Retest-Reliabilität dienten der Reliabilitätsüberprüfung. Es wurde auf Inhalts‑, Übereinstimmungs- und auf diskriminative Validität getestet. Die Konstruktvaliditätsprüfung basiert auf kürzlich veröffentlichen Daten. Als Gütekriterien wurden die Sensitivität und eine ROC(„Receiver Operating Characteristic“)-Analyse, inklusive AUC(„Area Under the ROC Curve“)-Betrachtung, eingesetzt.

Ergebnisse

Das Test-Retesting ergab nach 3 und 6 Monaten postoperativ stabile NCIQ-Werte. Die Cronbach-α-Werte wiesen auf eine gute interne Konsistenz hin. Der NCIQ diskriminierte valide zwischen behandelten und unbehandelten Patientengruppen. Es ergaben sich statistisch signifikante, wenn auch schwache, Korrelationen zwischen dem NCIQ und dem APHAB (r = −0,22; p = 0,04) und dem HPS (r = 0,30; p = 0,01). Sensitivitäts- und ROC-Analysen zeigten eine gute Messqualität des deutschsprachigen NCIQ.

Schlussfolgerung

Die deutsche Übersetzung des NCIQ misst zuverlässig und valide die Lebensqualität vor und nach Cochleaimplantation und kann zur klinischen Erfolgskontrolle nach Cochleaimplantationen verwendet werden.

Is the international physical activity questionnaire (IPAQ) a valid assessment tool for measuring physical activity of patients with axial spondyloartritis?

BACKGROUND

Determining the level of physical activity (PA) is an essential part of patient evaluation in axial spondylarthritis (axSpA). Subjective and objective methods are both frequently used methods for evaluating PA. Although subjective methods are cost-effective and easy to use, their accuracy for measuring PA is still questionable.

OBJECTIVE

To investigate the concurrent criterion validity of a self-reported questionnaire (IPAQ-Short Form) when compared to an accelerometer (Actigraph wGT3X-BT) for measuring PA level in patients with axSpA.

DESIGN

Cross-sectional design.

METHODS

Fifty-eight patients with axSpA with a median age of 39.0 (IQR 25/75: 30.0/46.0) years were included in the study. An accelerometer (Actigraph wGT3X-BT) was attached to the waist of patients at their first visits and was removed at their second visits, seven days later. Patients were asked to complete the International Physical Activity Questionnaire Short Form (IPAQ) at their second visits.

RESULTS

No significant correlations were determined between IPAQ and accelerometer (p > 0.05), except for the moderate PA (rho: 0.367, p < 0.05), and total PA (rho: 0.330, p < 0.05). It was also observed that IPAQ was underestimating energy expenditure for all types of PA.

CONCLUSION

IPAQ might not be a valid tool for measuring PA level in patients with axSpA. Disease-specific subjective methods for determining the PA should be developed and validated for those patients.

Construct validity and reliability of tests for sacroiliac dysfunction: standing flexion test (STFT) and sitting flexion test (SIFT)

CONTEXT

Sacroiliac dysfunction is characterized by a hypomobility of the range of motion of the joint, followed by a positional change regarding the relationship between the sacrum and the iliac. In general, the clinical tests that evaluate the sacroiliac joint (SIJ) and its dysfunctions lack validity and reliability values.

OBJECTIVES

This article aims to evaluate the construct validity and intra- and inter-rater reliability of the standing flexion test (STFT) and sitting flexion test (SIFT).

METHODS

In this prospective study, the sample consisted of 30 individuals of both sexes, and the evaluation team was composed of five researchers. The evaluations took place on two different days: first day, inter-rater reliability and construct validity; and second day, intra-rater reliability. The reference standard for the construct validity was 3-dimensional measurements obtained utilizing the BTS SMART-DX system. For statistical analysis, the percentage (%) agreement and the kappa statistic (K) were utilized.

RESULTS

The construct validity was determined for STFT (70% agreement; K=0.49; p<0.01) and SIFT (56.7% agreement; K=0.29; p<0.05). The intra-rater reliability was determined for STFT (66.3% agreement; K=0.43; p<0.01) and SIFT (56.7% agreement; K=0.38; p<0.01). The inter-rater reliability was determined for STFT (10% agreement; K=-0.02; p=0.825) and SIFT (13.3% agreement; K=0.01; p=0.836).

CONCLUSIONS

The STFT confirmed the construct validity and was reliable when applied by the same rater to healthy people, even if the rater had no experience. It was not possible to achieve minimum scores using the SIFT either for construct validity or reliability. We suggest that further studies be conducted to investigate the measurement properties of palpatory clinical tests for SIJ mobility, especially in symptomatic patients.

Content and Face Validation of a Novel, Interactive Nutrition Specific Physical Exam Competency Tool (INSPECT) to Evaluate Registered Dietitians' Competence: A Delphi Consensus from the United States

The nutrition-focused physical examination (NFPE) is an integral component of nutrition assessment performed by registered dietitian nutritionists (RDNs) to determine signs of malnutrition and other nutrition-related complications. Increased use of this essential skill among RDNs and the transformation of dietetics education to a competency-based model in the near future calls for appropriately validated tools to measure RDNs' NFPE competence. To fill the need for a validated competency tool, this study developed an Interactive Nutrition-Specific Physical Exam Competency Tool (INSPECT) utilizing the initial 70 items identified in the first phase of the study. The second phase of this study aimed to test the preliminary version of the INSPECT for content and face validity. An expert panel of 17 members provided consensus recommendations through the Delphi process. Internal consistency of the consensus was measured with Cronbach's alpha (α) and α of ≥0.70 was defined as acceptable a priori. Inter-rater agreement among the expert panel was determined using the intraclass correlation coefficient (ICC) and an a priori ICC of 0.75 to 0.9 was established as good and >0.9 as excellent agreement. The results showed acceptable face validity (α = 0.71) and excellent content validity for the INSPECT, with an internal consistency of α = 0.97 in the first round and α = 0.96 in the second round. The inter-rater agreement was also excellent with ICC = 0.95 for each of the Delphi rounds. A total of 52 items were retained from the preliminary version of the INSPECT. Open feedback from the experts allowed for the consolidation of 11 similar items for better scoring and evaluation and thus, a total of 41 items were included in the final version of the INSPECT. The final version of the INSPECT is currently being studied in real-life, multi-site clinical settings among practicing RDNs to examine construct validity, reliability, and item-level psychometric properties. Ultimately, the validated INSPECT will be available for the competency evaluation of RDNs practicing in clinical settings.

The empirical evidence base for the use of the HCR-20: A narrative review of study designs and transferability of results to clinical practice

The HCR-20, a widely used method of assessing and managing risk, relies on the structured professional judgement approach. This paper reports a narrative literature review of the HCR-20 studies to explore the applicability of the study results to the use of the HCR-20 in clinical practice. From a literature search using terms "HCR-20" and "HCR 20", 206 papers were included. Of studies using the HCR-20 version 2 (n = 191), 92% (n = 176) relied on variables based on scores derived by adding item scores, and 50% (n = 95) tested the HCR-20 using predictive validity methodology. Of the HCR-20 version 3 studies (n = 21), the "presence of risk factors" step was the most commonly examined (n = 18, 86%), but 2 of the 7 steps ("scenario planning" and "management") were not examined at all. Amongst those studies whose primary focus was on the HCR-20, 67% (n = 64/95) did so by assessing the predictive validity of the tool. Only one employed a design to test whether the use of the HCR-20 affected violence rates. The predominant study design provides support for the use of the HCR-20 as an actuarial tool, and there is limited empirical evidence in support of its effectiveness as a structured professional judgement approach to the assessment and management of the risk of violence.

Validity of the National Health Security Preparedness Index as a Predictor of Excess COVID-19 Mortality

OBJECTIVE

This study compared 2019 values for the National Health Security Preparedness Index (NHSPI) with 2020 rates of coronavirus disease 2019 (COVID-19)-related mortality as reported by the 50 US states and Puerto Rico during the first six months of the US pandemic (March 1 - August 31, 2020).

METHODS

Data regarding provisional death counts and estimates of excess deaths for COVID-19 according to state and territory were downloaded from the Centers for Disease Control and Prevention (CDC) National Center for Health Statistics website. Reporting included the six-month-long period of March 1 - August 31, 2020. Excess mortality rates were calculated as the number of excess deaths per 100,000 persons in each state population using 2019 US Census Bureau data. Mean values for state and territorial NHSPI domain indices were compared to state and territorial rates of COVID-19-related excess mortality using multiple linear regression, including analysis of variance. Correlations between the 51 state and territorial NHSPI values and corresponding COVID-19 excess mortality rates were calculated using Pearson's correlation coefficient.

RESULTS

These calculations revealed a high degree of variance (adjusted r square = 0.02 and 0.25) and poor correlation (P = .16 and .08) among values for the overall NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.There was also a high degree of variance (adjusted r square = 0.001 and 0.03) and poor correlation (P values ranging from .09 to .94) for values for the six individual domains of the NHSPI as compared to low and high estimates of excess COVID-19 mortality rates for 50 US states and Puerto Rico.

CONCLUSION

The NHSPI does not appear to be a valid predictor of excess COVID-19 mortality rates for 50 US states and Puerto Rico during the first six months of the pandemic.

Evaluation of Pragmatic Methods to Rapidly Assess Habitual Beverage Intake and Hydration Status in U.S. Collegiate Athletes

Fluid intake recommendations have been established for the athletic population in order to promote adequate hydration. The Beverage Intake Questionnaire (BEVQ-15) is a quick and reliable food frequency questionnaire that quantifies habitual beverage intake, which has been validated in children, adolescents, and adults. However, no validated beverage consumption questionnaire is available for collegiate athletes. Urine color (UC), while feasible for determining hydration status, has not been validated within a variety of collegiate athletes. The purpose of this investigation was to evaluate the comparative validity and reliability of pragmatic methods to rapidly assess BEVQ-15 and UC rating in U.S. Division I collegiate athletes. Student-athletes (n = 120; 54% females; age 19 ± 1 years) from two universities were recruited to complete three study sessions. At the first and third sessions, the participants completed the BEVQ-15 and provided a urine sample to determine UC and urinary specific gravity. All sessions included completion of a 24-hr dietary recall. Total fluid intake (fl oz) was 111 ± 107 and 108 ± 42 using the BEVQ-15 and the mean of three 24-hr dietary recalls, respectively, which was not different between methods (p > .05). There were moderate associations between the BEVQ-15 and dietary recall results for total beverage intake fl oz and kcal(r = .413 and r = 4.65; p ≤ .05, respectively). Strong associations were noted between both researcher-rated and participant-rated UC with urinary specific gravity measures (r = .675 and r = .884; p ≤ .05, respectively). Therefore, these rapid assessment methods demonstrated acceptable validity and may be used as practical methods to determine whether athletes are meeting their hydration recommendations.

Accuracy of delirium assessments in critically ill children: A prospective, observational study during routine care

OBJECTIVES

The objectives of this study was to explore the accuracy of the Cornell Assessment for Pediatric Delirium (CAP-D), Pediatric Confusion Assessment Method for the Intensive Care Unit (pCAM-ICU), and Preschool Confusion Assessment Method for the Intensive Care Unit (psCAM-ICU) when implemented in routine care as delirium screening tools, and to assess patient characteristics and clinical variables that may affect their validity.

DESIGN

This is a prospective observational study.

SETTING

The study was conducted in a 36-bed, mixed paediatric intensive care unit (PICU) at an Australian tertiary hospital.

PATIENTS

The study included critically ill children developmentally aged 6 months to 17 years, with a PICU length of stay >18 h.

INTERVENTIONS

No interventions were provided in the study.

MEASUREMENTS AND MAIN RESULTS

Patients were screened for delirium by their bedside nurse (CAP-D and pCAM-ICU/psCAM-ICU) once daily, for up to 5 d. Delirium status identified using screening instruments was compared with delirium diagnosis using the diagnostic criteria for delirium (Diagnostic and Statistical Manual of Mental Disorders, 5th Edition). In this sample, the CAP-D retained its high sensitivity (91.3%) and good specificity (75.2%), whereas the psCAM-ICU and pCAM-ICU had moderate sensitivity (58.8% and 75.0%, respectively) and excellent specificity (89.8% and 84.9%, respectively). There was moderate agreement between the CAP-D and the psCAM-ICU (κ = 0.52, p < .001) and good agreement between the CAP-D and the pCAM-ICU (κ = 0.80, p < .01).

CONCLUSIONS

Although the CAP-D, psCAM-ICU, and pCAM-ICU all appear promising in their validation studies, when implemented in routine care, their performance can be variable. The CAP-D performed well in routine clinical practice, but follow-up diagnosis is required to confirm delirium. The psCAM-ICU and pCAM-ICU both provide valuable, objective assessments of delirium in critically ill children; however, further evaluation of their implementation in routine clinical practice is needed.

Interrater and Intrarater Reliability of the Colloid Cyst Risk Score

BACKGROUND

The Colloid Cyst Risk Score (CCRS) was developed to identify symptomatic patients and stratify risk of hydrocephalus among patients with colloid cysts. Its components consider patient age, cyst diameter, presence/absence of headache, fluid-attenuated inversion recovery (FLAIR) hyperintensity, and location within the third ventricle.

OBJECTIVE

To independently evaluate the inter- and intrarater reliability of the CCRS.

METHODS

Patients with a colloid cyst were identified from billing records and radiology archives. Three independent raters reviewed electronic medical records to determine age, presence/absence of headache, cyst diameter (mm), FLAIR hyperintensity, and risk zone location. Raters made 53 observations, including 5 repeat observations.Fleiss' generalized kappa (κ) was calculated for all of the nominal criteria, whereas Kendall's coefficient of concordance (W) and the intraclass correlation coefficient (ICC) were calculated for the overall score.

RESULTS

Total CCRS score demonstrated extremely strong agreement (W = 0.83) using Kendall's W coefficient and good agreement (ICC = 0.74) using the ICC (P < .001). For interrater reliability of individual criteria, age (κ = 1.00) and FLAIR hyperintensity (κ = 0.89) demonstrated near perfect agreement. Axial diameter (κ = 0.63) demonstrated substantial agreement, whereas agreement was moderate for risk zone (κ = 0.51) and fair for headache (κ = 0.26). Intrarater reliability for total CCRS score was extremely strong using Kendall's W, good to excellent using ICC, and fair to substantial using weighted kappa.

CONCLUSION

The CCRS has good inter- and intrarater reliability when tested in an independent sample of patients, though strength of agreement varies among individual criteria. The validity of the CCRS requires independent evaluation.

Validity and Reliability of a Photoelectric Cells System for the Evaluation of Change of Direction and Lateral Jumping Abilities in Collegiate Basketball Athletes

The validity and reliability of the Optojump system were investigated for jumping height and flight time in vertical jump tests. Conversely, the purpose of the present study was to investigate the validity and reliability of the Optojump system for measuring contact time and lateral displacement in change of direction and lateral jump tests. Thirty basketball collegiate athletes were tested on two 10 m sprints with a 60° (COD60) or 180° (COD180) change of direction, lateral controlled (CLRJ) and maximal (MLRJ) rebound jump, and lateral countermovement (LCMJ) and squat (LSJ) jump with the concomitant use of two force plates and the Optojump system for the measurement of contact time in COD60, COD180, CLRJ, MLRJ, and lateral jumping distance in all the lateral jump tests. Almost perfect coefficients (r ≥ 0.95) emerged for contact time in COD60, COD180, CLRJ, MLRJ, although a systematic bias was found for COD60 (-0.01 s). Good-to-excellent reliability was found for almost all the measurements of contact time and lateral jumping distance for change of direction and lateral jump tests. Therefore, the use of Optojump system for testing change of direction and lateral jumping abilities should be executed with caution, avoiding misinterpretation of data.

Rate of diagnostic errors and serious misdiagnosis-related harms for major vascular events, infections, and cancers: toward a national incidence estimate using the “Big Three”

Background

Missed vascular events, infections, and cancers account for ~75% of serious harms from diagnostic errors. Just 15 diseases from these “Big Three” categories account for nearly half of all serious misdiagnosis-related harms in malpractice claims. As part of a larger project estimating total US burden of serious misdiagnosis-related harms, we performed a focused literature review to measure diagnostic error and harm rates for these 15 conditions.

Methods

We searched PubMed, Google, and cited references. For errors, we selected high-quality, modern, US-based studies, if available, and best available evidence otherwise. For harms, we used literature-based estimates of the generic (disease-agnostic) rate of serious harms (morbidity/mortality) per diagnostic error and applied claims-based severity weights to construct disease-specific rates. Results were validated via expert review and comparison to prior literature that used different methods. We used Monte Carlo analysis to construct probabilistic plausible ranges (PPRs) around estimates.

Results

Rates for the 15 diseases were drawn from 28 published studies representing 91,755 patients. Diagnostic error (false negative) rates ranged from 2.2% (myocardial infarction) to 62.1% (spinal abscess), with a median of 13.6% [interquartile range (IQR) 9.2–24.7] and an aggregate mean of 9.7% (PPR 8.2–12.3). Serious misdiagnosis-related harm rates per incident disease case ranged from 1.2% (myocardial infarction) to 35.6% (spinal abscess), with a median of 5.5% (IQR 4.6–13.6) and an aggregate mean of 5.2% (PPR 4.5–6.7). Rates were considered face valid by domain experts and consistent with prior literature reports.

Conclusions

Diagnostic improvement initiatives should focus on dangerous conditions with higher diagnostic error and misdiagnosis-related harm rates.

Reliability and Validity of Scales for Assessing Child Dental Fear and Anxiety

Introduction

Children with high dental anxiety display uncooperative behavior during dental procedures. Different self-assessment questionnaires are in use to estimate the prevalence of dental fear and anxiety (DFA) in children but they need to be tested for reliability and validity in different populations.

Objectives

The aim of this study was to test the reliability and validity of two scales for the assessment of child dental fear and anxiety in a sample of Croatian children.

Material and Methods

The sample comprised 202 pairs consisting of children and their mothers (97 boys and 105 girls) aged 5 - 12 years. Two scales were used to assess children's DFA: the Children's Fear Survey Schedule - Dental Subscale (CFSS-DS) and the Modified Child Dental Anxiety Scale - face version (MCDAS-f). The Corah's Dental Anxiety Scale (CDAS) was used in the sample of children's mothers. The average scores of cooperative children and children with behavior management problem (BMP) were compared.

Results

Cooperative children displayed significantly lower CFSS-DS scores than children with BMP (M = 28. 46 vs. M = 39. 36; P < 0.001). The average MCDAS-f score was significantly higher in children with BMP than in cooperative children (M = 28. 07 vs. 20. 01; P < 0.001). The CFSS-DS showed high internal consistency (Cronbach's α = 0. 90), while internal consistency of MCDAS-f was good (α = 0. 73). The maternal CDAS showed good internal consistency (α = 0, 89). The correlations with other self-report measurements of DFS show good validity of all scales. The correlations between CFSS-DS and MCDAS-f were highly significant (r = 0.482; P < 0. 01) as well as the correlations of CFSS-DS and MCDAS-f with CDAS scores (r = 0.223; P < 0. 01 and r = 0.198; P < 0. 01, respectively.

Conclusion

The obtained results suggest that both the CFSS-DS scale and the MCDAS-f scale are reliable and valid scales for measuring dental fear and anxiety in children in a sample of Croatian children. The MCDAS-f scale has shown slightly lower internal consistency but it is suitable for use in young children. In addition, the MCDAS-f scale is simple to administer.

Validation of Modified Dental Anxiety Scale for Dental Extraction Procedure (MDAS-DEP)

Background:

Dental anxiety remains a common problem encountered amongst those who are going to have a dental extraction procedure.

Objective:

The objective of the current study was to test the validity and reliability of the modified dental anxiety scale for dental extraction procedure (MDAS-DEP).

Methods:

The current study was conducted by using a cross-sectional survey method on one hundred and thirty-two (69 female; 63 male) participants who were about to go through the dental extraction procedure. A modified version of the MDAS was used in the current study. The questions of the original MDAS were replaced by specific questions about the dental extraction procedure, accompanied by five Likert-type answers. Reliability was measured by referring to the Cronbach’s alpha value whilst construct validity was measured by using Pearson’s correlation.

Results:

The analysis of the current study showed a Cronbach’s alpha value of 0.822 Whilst the Pearson’s correlation analysis revealed that all questions were proven to be valid, with the highest r value gained by question number 5 (r = 0.817, p < 0.01).

Conclusion:

The current scale was proven to be valid and reliable to be used as a specific scale to measure dental anxiety due to dental extraction.

Can Dynamic Ultrasonography of the Hip Reliably Assess Anterior Femoral Head Translation?

BACKGROUND

Hip microinstability has gained attention recently as a potential cause of hip pain. Currently there is a lack of evidence-based objective diagnostic criteria surrounding this diagnosis. Previous studies have shown translation of the femoral head during extreme hip positions. However, reliable assessment of femoral head translation is lacking.

QUESTIONS/PURPOSES

(1) How precise is musculoskeletal ultrasound for measuring anterior femoral head translation during the hip anterior apprehension test? (2) What is the intra- and interrater reliability of dynamic ultrasonography in assessing anterior femoral head translation?

METHODS

We recruited 10 study participants (20 hips) between the ages of 22 and 50 years with no history of hip pain or functional limitations. Test-retest methodology was used. Seven females and three males were enrolled. The mean age of study participants was 27 years (SD 8.7 years); mean body mass index was 22.6 kg/m (SD 2.2 kg/m). All study participants underwent dynamic hip ultrasonography by three different physicians 1 week apart. Each hip was visualized in two neutral positions (neutral and neutral with the contralateral hip flexed [NF]) and two dynamic positions, which sought to replicate the apprehension test, although notably study participants had no known hip pathology and therefore no apprehension. The first maintained the hip in extension and external rotation off to the side of the examination table (EER1), and the second held the hip off of the bottom of the examination table (EER2). One hundred twenty ultrasound scans (480 images) were performed. Mean and SD were calculated using absolute values of the difference in ultrasound measurements (mm) between positions NF and EER1 and NF and EER2 calculated for each physician as well as an average of all three physicians. Intraclass correlation coefficient (ICC) analysis was used to examine intra- and interrater reliability.

RESULTS

The mean absolute difference for NF and EER1 was 0.84 mm (SD 0.93 mm) and for NF and EER2 0.62 mm (SD 0.40 mm) on Study Day 1. Similarly, on Study Day 2, the mean absolute difference for NF and EER1 position was 0.90 mm (SD 0.74 mm) and for NF and EER2 1.03 mm (SD 1.18 mm). Cumulative values of ICC analysis indicated excellent intrarater reliability in all four positions: neutral 0.794 (95% confidence interval [CI], 0.494-0.918), NF 0.927 (95% CI, 0.814-0.971), EER1 0.929 (95% CI, 0.825-0.972), and EER2 0.945 (95% CI, 0.864-0.978). Similarly, interrater ICC analysis cumulative values were excellent for NF, EER1, and EER2 and fair to good for the neutral position: neutral 0.725 (95% CI, 0.526-0.846), NF 0.846 (95% CI, 0.741-0.913), EER1 0.812 (95% CI, 0.674-0.895), and EER2 0.794 (95% CI, 0.652-0.884).

CONCLUSIONS

This study offers the first ultrasound protocol of which we are aware for measuring anterior femoral head translation. Hip dynamic ultrasound may assist in providing precise objective clinical-based diagnostic evidence when evaluating complex hip pain and suspected microinstability. Musculoskeletal ultrasound is a reliable office-based method of measuring anterior femoral head translation that can be utilized by physicians with varying experience levels. Future studies are needed to investigate ultrasound anterior femoral head translation taking into account sex, prior hip surgery, hip osseous morphology, and ligamentous laxity.

LEVEL OF EVIDENCE

Level III, diagnostic study.

Partner Strategic Capabilities for Capturing Value from Sustainability-Focused Multi-Stakeholder Partnerships

As social and ecological problems escalate, the role of collective capacity and knowledge is becoming more critical in reaching solutions. This capacity and knowledge are dispersed among diverse stakeholder organizations. Thus, organizations in the private, public and civil society sectors are experiencing pressure to address these complex challenges through collaborative action in the form of multi-stakeholder partnerships. One major challenge to securing and maintaining partner engagement in these voluntary collaborative initiatives is defining the value proposition for prospective and existing partner organizations. Understanding the relationship between different forms of partner involvement and the subsequent resources that partners stand to gain is necessary to articulate the value proposition of the partnership to partners. This study conducts a survey of partner organizations from 15 different sustainability-focused multi-stakeholder partnerships in Canada. We compare three partner strategies for implementation and value capture and discover that each strategy is associated with different partner-level resource outcomes. Our findings indicate that product stewardship strategies are associated with financial and organizational capital, marketing and promotion with human capital, and internal implementation structures with shared capital. This study has implications for multi-stakeholder partnership researchers and practitioners because it suggests the possibility that certain partner-level outcomes could rely on the partner, as well as partnership implementation strategies.

Investigating the feasibility and acceptability of real-time visual feedback in reducing compensatory motions during self-administered stroke rehabilitation exercises: A pilot study with chronic stroke survivors

INTRODUCTION

Homework-based rehabilitation programs can help stroke survivors restore upper extremity function. However, compensatory motions can develop without therapist supervision, leading to sub-optimal recovery. We developed a visual feedback system using a live video feed or an avatar reflecting users' movements so users are aware of compensations. This pilot study aimed to evaluate validity (how well the avatar characterizes different types of compensations) and acceptability of the system.

METHODS

Ten participants with chronic stroke performed upper-extremity exercises under three feedback conditions: none, video, and avatar. Validity was evaluated by comparing agreement on compensations annotated using video and avatar images. A usability survey was administered to participants after the experiment to obtain information on acceptability.

RESULTS

There was substantial agreement between video and avatar images for shoulder elevation and hip extension (Cohen's κ: 0.6-0.8) and almost perfect agreement for trunk rotation and flexion (κ: 0.80-1). Acceptability was low due to lack of corrective prompts and occasional noise with the avatar display. Most participants suggested that an automatic compensation detection feature with visual and auditory cuing would improve the system.

CONCLUSION

The avatar characterized four types of compensations well. Future work will involve increasing sensitivity for shoulder elevation and implementing a method to detect compensations.

Bowel preparation quality scales for colonoscopy

Colorectal cancer (CRC) is the third most common cancer and second leading cause of cancer-related death in the United States. Colonoscopy is widely preferred for CRC screening and is the most commonly used method in the United States. Adequate bowel preparation is essential for successful colonoscopy CRC screening. However, up to one-quarter of colonoscopies are associated with inadequate bowel preparation, which may result in reduced polyp and adenoma detection rates, unsuccessful screens, and an increased likelihood of repeat procedure. In addition, standardized criteria and assessment scales for bowel preparation quality are lacking. While several bowel preparation quality scales are referred to in the literature, these differ greatly in grading methodology and categorization criteria. Published reliability and validity data are available for five bowel preparation quality assessment scales, which vary in several key attributes. However, clinicians and researchers continue to use a variety of bowel preparation quality measures, including nonvalidated scales, leading to potential confusion and difficulty when comparing quality results among clinicians and across clinical trials. Optimal clinical criteria for bowel preparation quality remain controversial. The use of validated bowel preparation quality scales with stringent but simple scoring criteria would help clarify clinical trial data as well as the performance of colonoscopy in clinical practice related to quality measurements.

Bowel preparation quality scales for colonoscopy

Colorectal cancer (CRC) is the third most common cancer and second leading cause of cancer-related death in the United States. Colonoscopy is widely preferred for CRC screening and is the most commonly used method in the United States. Adequate bowel preparation is essential for successful colonoscopy CRC screening. However, up to one-quarter of colonoscopies are associated with inadequate bowel preparation, which may result in reduced polyp and adenoma detection rates, unsuccessful screens, and an increased likelihood of repeat procedure. In addition, standardized criteria and assessment scales for bowel preparation quality are lacking. While several bowel preparation quality scales are referred to in the literature, these differ greatly in grading methodology and categorization criteria. Published reliability and validity data are available for five bowel preparation quality assessment scales, which vary in several key attributes. However, clinicians and researchers continue to use a variety of bowel preparation quality measures, including nonvalidated scales, leading to potential confusion and difficulty when comparing quality results among clinicians and across clinical trials. Optimal clinical criteria for bowel preparation quality remain controversial. The use of validated bowel preparation quality scales with stringent but simple scoring criteria would help clarify clinical trial data as well as the performance of colonoscopy in clinical practice related to quality measurements.

The validity and reliability of the modified forgotten joint score

We aim to validate the "Modified Forgotten Joint Score" (MFJS) as a new patient-reported outcome measure (PROM) in hip and knee arthroplasty, against the UK's gold standard Oxford Hip and Knee Scores (OHS/OKS). The original Forgotten Joint Score (FJS) (12 items) was created to assess post-arthroplasty joint awareness. We modified the FJS to 10-items to improve its reliability. Postal questionnaires were sent out to 400 total hip or knee replacement (THR/TKR) patients who were 1-2 years' post-op, along with the OHS/OKS. Data, collected from the 212 returned questionnaires (53% response rate), was analysed in relation to construct and content validity. A sub-cohort of 77 patients took part in a test-retest repeatability study, to assess reliability of the MFJS. The MFJS proved to have an increased discriminatory power in high-performing patients in comparison to the OHS and OKS. 30.8% of TKR patients (n = 131) scored highly (87.5% or more) in the OKS compared to just 7.69% in the MFJS TKR patients. The MFJS proved to have increased test-retest repeatability, based upon its intra-class correlation coefficient of 0.968 compared to the Oxford's 0.845, p < 0.001. The MFJS is a more relevant tool, compared to the FJS, with greater discrimination in the assessment of well performing hip and knee arthroplasties in comparison to the OHS/OKS.

Symptom-Disease Pair Analysis of Diagnostic Error (SPADE): a conceptual framework and methodological approach for unearthing misdiagnosis-related harms using big data

BACKGROUND

The public health burden associated with diagnostic errors is likely enormous, with some estimates suggesting millions of individuals are harmed each year in the USA, and presumably many more worldwide. According to the US National Academy of Medicine, improving diagnosis in healthcare is now considered 'a moral, professional, and public health imperative.' Unfortunately, well-established, valid and readily available operational measures of diagnostic performance and misdiagnosis-related harms are lacking, hampering progress. Existing methods often rely on judging errors through labour-intensive human reviews of medical records that are constrained by poor clinical documentation, low reliability and hindsight bias.

METHODS

Key gaps in operational measurement might be filled via thoughtful statistical analysis of existing large clinical, billing, administrative claims or similar data sets. In this manuscript, we describe a method to quantify and monitor diagnostic errors using an approach we call 'Symptom-Disease Pair Analysis of Diagnostic Error' (SPADE).

RESULTS

We first offer a conceptual framework for establishing valid symptom-disease pairs illustrated using the well-known diagnostic error dyad of dizziness-stroke. We then describe analytical methods for both look-back (case-control) and look-forward (cohort) measures of diagnostic error and misdiagnosis-related harms using 'big data'. After discussing the strengths and limitations of the SPADE approach by comparing it to other strategies for detecting diagnostic errors, we identify the sources of validity and reliability that undergird our approach.

CONCLUSION

SPADE-derived metrics could eventually be used for operational diagnostic performance dashboards and national benchmarking. This approach has the potential to transform diagnostic quality and safety across a broad range of clinical problems and settings.

Daboia (Vipera) palaestinae envenomation in horses: Clinical and hematological signs, risk factors for mortality and construction of a novel severity scoring system

Daboia palaestinae is the most common venomous snake in Israel and an important cause of envenomations in humans and animals. Although specific antivenom is produced from horses, little documentation exists regarding the characteristics of envenomed horses. This survey was constructed to describe the clinical, hematological and biochemical characteristics of D. palaestinae envenomation in horses, to identify risk factors for fatality and to construct a prognostic snakebite severity scale (SSS) to be used by veterinarians in the field. Data regarding 123 equine snakebite cases were collected over four years. Clinical signs most frequently described were local swelling (100%), tachycardia (67%), increased respiratory effort (62%), bite marks (44%) and reduced appetite (50%). Blood samples were collected at initial presentation from 23 of these horses. Hematological and biochemical abnormalities described in the majority of the sampled cases included leukocytosis (52%), neutrophilia (78%), increased creatine phosphokinase (77%), hypoproteinemia (50%), decreased total carbon dioxide (54%), and hyperphosphatemia (54%). The overall case fatality rate was 20.3% (25/123 horses). Risk factors associated with increased mortality (P < 0.05) were body weight below 200 kg, hind limb envenomation, severity of local swelling, mental status, respiratory effort, bleeding disorders, dehydration, basophilia, monocytosis, bilirubinemia, hypocalcemia and hyperphosphatemia. These risk factors were used to construct an SSS adapted for the assessment of envenomed horses in the field, consisting of seven dichotomous parameters. A cutoff value of 1.5 predicted mortality with 95% sensitivity and 68% specificity. In the future, this tool could assist clinicians with case management and communication with owners.

Physical activity and healthy eating environmental audit tools in youth care settings: A systematic review

BACKGROUND

There is a growing interest in evaluating the physical activity (PA) and healthy eating (HE) policy and practice environment characteristics in settings frequented by youth (≤18years).

OBJECTIVE

This review evaluates the measurement properties of audit tools designed to assess PA and HE policy and practice environment characteristics in settings that care for youth (e.g., childcare, school, afterschool, summer camp).

METHOD

Three electronic databases, reference lists, educational department and national health organizations' web pages were searched between January 1980 and February 2014 to identify tools assessing PA and/or HE policy and practice environments in settings that care for youth (≤18years).

RESULTS

Sixty-five audit tools were identified of which 53 individual tools met the inclusion criteria. Thirty-three tools assessed both the PA and HE domains, 6 assessed the PA domain and 14 assessed the HE domain solely. The majority of the tools were self-assessment tools (n=40), and were developed to assess the PA and/or HE environment in school settings (n=33), childcare (n=12), and after school programs (n=4). Four tools assessed the community at-large and had sections for assessing preschool, school and/or afterschool settings within the tool. The majority of audit tools lacked validity and/or reliability data (n=42). Inter-rater reliability and construct validity were the most frequently reported reliability (n=7) and validity types (n=5).

CONCLUSIONS

Limited attention has been given to establishing the reliability and validity of audit tools for settings that care for youth. Future efforts should be directed towards establishing a strong measurement foundation for these important environmental audit tools.

Inter- and intra-observer agreement on Miller's classification of gingival tissue recessions

Miller's is the most commonly used classification of gingival tissue recessions, defined as the displacement of the soft tissue margin apical to the cemento-enamel junction. However, data on the reliability of this classification are missing so far, although reliability, which reflects the consistency of repeated measurements, is regarded as a prerequisite for judging the utility of a classification. The aim of the present study was to evaluate inter- and intra-observer agreement on Miller's classification of gingival tissue recessions. Two hundred photographs (50 of each region: maxillary/mandibular anterior/posterior teeth) of gingival tissue recessions were evaluated twice by four observers with different degrees of experience in Miller's classification, gingival phenotype, tooth shape, and identifiability of the cemento-enamel junction. The following inter- and intra-observer agreements were found: Miller's classification, 0.72 and 0.73-0.95; gingival phenotype, 0.29 and 0.45-0.58; tooth shape, 0.39 and 0.44-0.59; and identifiability of the cemento-enamel junction, 0.21 and 0.30-0.59. A higher agreement was detected for anterior teeth. Further, gingival phenotype (thin-high scalloping) significantly correlated with tooth shape (long-narrow) (ρ = 0.662, p < 0.001). Miller's classification of gingival tissue recessions was evaluated by four examiners using 200 clinical photographs and yielded substantial to almost perfect agreement, with higher agreement for anterior teeth. Although limited to photographic assessment, the present study offers the so far missing proof on the sufficient inter- and intra-observer agreement of this classification.

The interproximal clinical attachment level to classify gingival recessions and predict root coverage outcomes: an explorative and reliability study

BACKGROUND

The aims of this study were (i) to test the reliability of a new classification system of gingival recessions using the level of interproximal clinical attachment as an identification criterion and (ii) to explore the predictive value of the resulting classification system on the final root coverage outcomes.

MATERIAL AND METHODS

Patients showing at least one buccal gingival recession were recruited by one operator. Three recession types (RT) were identified. While class RT1 included gingival recession with no loss of interproximal attachment, class RT2 recession was associated with interproximal attachment loss less than or equal to the buccal site and class RT3 showed higher interproximal attachment loss than the buccal site. The classification was tested by two examiners blinded to the data collected by the other examiner. Intra-rater and inter-rater agreement was assessed. Furthermore, the 6-month root coverage outcomes of consecutively treated gingival recessions were retrospectively evaluated in order to explore the predictive value of the proposed classification on the final recession reduction (Rec Red).

RESULTS

The new classification system of gingival recessions was tested in a total of 116 gingival recessions (mean 3.2±1.2 mm) in 25 patients. The intra-class correlation coefficient (ICC) for inter-rater agreement was 0.86, showing an almost perfect agreement between the examiners. The RT classification was predictive of the final Rec Red (p<0.0001) at the 6-month follow-up in 109 treated gingival recessions.

CONCLUSIONS

The evaluation of interproximal clinical attachment level may be used to classify gingival recession defects and to predict the final root coverage outcomes.

Face, content and construct validity of a new realistic trainer for conventional techniques in digestive surgery

PURPOSE

Surgical simulation modules for "open" surgery are limited in contrast to well-studied and validated laparoscopic trainers. In this study, face, content and construct validity of a devised simulation module (Berlin Operation Trainer, BOPT) for handsewn anastomoses in digestive surgery were analysed.

MATERIALS AND METHODS

Participants of a skills course for digestive surgery (novices: 1-3 years of training; experts: more than 5 years of training) were timed on performing four defined handsewn digestive anastomoses on formalin fixed porcine intestine in the BOPT. Questionnaires were answered regarding impression with the simulation module concerning appearance and realism using a five-point Likert and a three-point forced choice scale. Face and content validities were evaluated based on the responses of participants and construct validity by comparing novices to experts. Data collected were analysed with Fisher's exact test and two-sample t test.

RESULTS

Twenty-two novices (median: second postgraduate year) and 26 experts (median: seventh postgraduate year) were enrolled in the study. The BOPT showed strong face and content validities with average scores for satisfaction parameters above 4.2 +/- 0.41 and 4.1 +/- 0.22, respectively. Construct validity was adequate for anastomosis simulation in the BOPT based on different percentages of anastomosis complete during set time between novices and experts as shown for simple (68.2% vs. 92.3%, p = 0.038) and for difficult anastomosis (18.2% vs. 50.0%, p = 0.021).

CONCLUSIONS

The BOPT is a suitable instrument for advanced surgical training for novices and experienced colleagues creating a realistic and demanding situation. Further studies have to evaluate if a more realistic preoperative training will support an effective transfer of learned techniques to the operating room.

Computerized general practice based networks yield comparable performance with sentinel data in monitoring epidemiological time-course of influenza-like illness and acute respiratory illness

Background

Computerized morbidity registration networks might serve as early warning systems in a time where natural epidemics such as the H₁N₁ flu can easily spread from one region to another.

Methods

In this contribution we examine whether general practice based broad-spectrum computerized morbidity registration networks have the potential to act as a valid surveillance instrument of frequently occurring diseases. We compare general practice based computerized data assessing the frequency of influenza-like illness (ILI) and acute respiratory infections (ARI) with data from a well established case-specific sentinel network, the European Influenza Surveillance Scheme (EISS). The overall frequency and trends of weekly ILI and ARI data are compared using both networks.

Results

Detection of influenza-like illness and acute respiratory illness occurs equally fast in EISS and the computerized network. The overall frequency data for ARI are the same for both networks, the overall trends are similar, but the increases and decreases in frequency do not occur in exactly the same weeks. For ILI, the overall rate was slightly higher for the computerized network population, especially before the increase of ILI, the overall trend was almost identical and the increases and decreases occur in the same weeks for both networks.

Conclusions

Computerized morbidity registration networks are a valid tool for monitoring frequent occurring respiratory diseases and the detection of sudden outbreaks.

Interrater reliability of Glasgow Coma Scale scores in the emergency department

STUDY OBJECTIVE

Emergency physicians often use the Glasgow Coma Scale (GCS) to help guide decisions in patient care, yet the reliability of the GCS has never been tested in a typical broad sample of emergency department (ED) patients. We determined the interrater reliability of the GCS between emergency physicians when adult patients with altered levels of consciousness are assessed.

METHODS

In this prospective observational study at a university Level I trauma center, we enrolled a convenience sample of ED patients older than 17 years who presented with an altered level of consciousness. Two residency-trained attending emergency physicians independently assessed and recorded the GCS score and its components (eye, verbal, and motor) in blinded fashion within a 5-minute period. Data were analyzed for interrater reliability by using standard ordinal calculations. We also created scatter plots and Bland-Altman plots for each GCS component and for the GCS score.

RESULTS

One hundred thirty-one patients were screened and enrolled in the study, with 15 excluded because of protocol violations. Of the 116 remaining patients, the agreement percentage for exact total GCS was 32% (tau-b=0.739; Spearman rho=0.864; Spearman rho2=75%). Agreement percentage for GCS components were eye 74% (tau-b=0.715; Spearman rho=0.757; Spearman rho2=57%), verbal 55% (tau-b=0.587; Spearman rho=0.665; Spearman rho2=44%), and motor 72% (tau-b=0.742; Spearman rho=0.808; Spearman rho2=65%). Our Spearman's analyses found that only approximately half (44% to 65%) of the observed variance could be explained by the relationship between the paired component measures. For GCS components, only 55% to 74% of paired measures were identical, and 6% to 17% of them were 2 or more points apart.

CONCLUSION

We found only moderate degrees of interrater agreement for the GCS and its components.

Completeness of state administrative databases for surveillance of congenital heart disease

BACKGROUND

Tracking birth prevalence of cardiac defects is essential to determining time and space clusters, and identifying potential associated factors. Resource limitations on state birth defects surveillance programs sometimes require that databases already available be used for ascertaining such defects. This study evaluated the data quality of state administrative databases for ascertaining congenital heart defects (CHD) and specific diagnoses of CHD.

METHODS

Children's Hospital of Wisconsin (CHW) medical records for infants born 1997-1999 and treated for CHD (n = 373) were abstracted and each case assigned CHD diagnoses based on definitive diagnostic reports (echocardiograms, catheterizations, surgical or autopsy reports). These data were linked to state birth and death records, and birth and postnatal (< 1 year of age) hospital discharge summaries at the Wisconsin Bureau of Health Information (WBHI). Presence of any code/checkbox indicating CHD (generic CHD) and exact matches to abstracted diagnoses were evaluated.

RESULTS

Fifty-eight percent of cases with generic CHD were identified by state databases. Postnatal hospital discharge summaries identified 48%, birth hospital discharge summaries 27%, birth certificates 9% and death records 4% of these cases. Exact matches were found for 52% of 633 specific diagnoses. Postnatal hospital discharge summaries provided most matches.

CONCLUSION

State databases identified 60% of generic CHD and exactly matched about half of specific CHD diagnoses. The postnatal hospital discharge summaries performed best in both in identifying generic CHD and matching specific CHD diagnoses. Vital records had limited value in ascertaining CHD.

Measures of low back function: a review of reproducibility studies

OBJECTIVE

The objective of the present study was to make a systematic literature review with preset quality criteria concerning reproducibility of the tests of the low back regarding strength, endurance and range of motion.

DESIGN

Literature in Medline and local databases was reviewed for articles concerning the reproducibility of strength, endurance, and range of motion measurements.

BACKGROUND

Measures of low back function are widely used, and are important for both clinical and research purposes in relation to low back problems. A review of the reproducibility of these tests has not previously been made.

METHODS

After extensive discussion among all the authors, general evaluation parameters were defined for the quality assessment. Every study was graded from 0 to 2 for each parameter. Parameters evaluated were: number of subjects, subject description, method description, test/retest interval, description of results, and statistics.

RESULTS

The literature search revealed a total of 79 studies. Most studies suffered from methodological weaknesses and only eleven studies received ten or more quality points (maximum 14). The results from the highest graded studies are highlighted.

CONCLUSIONS

It may be concluded that there is a considerable lack of information about the reproducibility of functional measures for the low back, and therefore a recommendation for consensus is difficult. However, most tests performed in the sagittal plane are reliable for use on groups.

RELEVANCE

Measures of low back function are thought to be of great importance for clinicians, and low back researchers in general. A review of reproducibility will be helpful both as a survey of tests, and to provide information on the usefulness in relation to the level of reproducibility.