COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures

PURPOSE

The original COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist was developed to assess the methodological quality of single studies on measurement properties of Patient-Reported Outcome Measures (PROMs). Now it is our aim to adapt the COSMIN checklist and its four-point rating system into a version exclusively for use in systematic reviews of PROMs, aiming to assess risk of bias of studies on measurement properties.

METHODS

For each standard (i.e., a design requirement or preferred statistical method), it was discussed within the COSMIN steering committee if and how it should be adapted. The adapted checklist was pilot-tested to strengthen content validity in a systematic review on the quality of PROMs for patients with hand osteoarthritis.

RESULTS

Most important changes were the reordering of the measurement properties to be assessed in a systematic review of PROMs; the deletion of standards that concerned reporting issues and standards that not necessarily lead to biased results; the integration of standards on general requirements for studies on item response theory with standards for specific measurement properties; the recommendation to the review team to specify hypotheses for construct validity and responsiveness in advance, and subsequently the removal of the standards about formulating hypotheses; and the change in the labels of the four-point rating system.

CONCLUSIONS

The COSMIN Risk of Bias checklist was developed exclusively for use in systematic reviews of PROMs to distinguish this application from other purposes of assessing the methodological quality of studies on measurement properties, such as guidance for designing or reporting a study on the measurement properties.

Conjoint analysis applications in health--a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force

BACKGROUND

The application of conjoint analysis (including discrete-choice experiments and other multiattribute stated-preference methods) in health has increased rapidly over the past decade. A wider acceptance of these methods is limited by an absence of consensus-based methodological standards.

OBJECTIVE

The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Good Research Practices for Conjoint Analysis Task Force was established to identify good research practices for conjoint-analysis applications in health.

METHODS

The task force met regularly to identify the important steps in a conjoint analysis, to discuss good research practices for conjoint analysis, and to develop and refine the key criteria for identifying good research practices. ISPOR members contributed to this process through an extensive consultation process. A final consensus meeting was held to revise the article using these comments, and those of a number of international reviewers.

RESULTS

Task force findings are presented as a 10-item checklist covering: 1) research question; 2) attributes and levels; 3) construction of tasks; 4) experimental design; 5) preference elicitation; 6) instrument design; 7) data-collection plan; 8) statistical analyses; 9) results and conclusions; and 10) study presentation. A primary question relating to each of the 10 items is posed, and three sub-questions examine finer issues within items.

CONCLUSIONS

Although the checklist should not be interpreted as endorsing any specific methodological approach to conjoint analysis, it can facilitate future training activities and discussions of good research practices for the application of conjoint-analysis methods in health care studies.

GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research

Background While the patient and public involvement (PPI) evidence base has expanded over the past decade, the quality of reporting within papers is often inconsistent, limiting our understanding of how it works, in what context, for whom, and why.Objective To develop international consensus on the key items to report to enhance the quality, transparency, and consistency of the PPI evidence base. To collaboratively involve patients as research partners at all stages in the development of GRIPP2.Methods The EQUATOR method for developing reporting guidelines was used. The original GRIPP (Guidance for Reporting Involvement of Patients and the Public) checklist was revised, based on updated systematic review evidence. A three round Delphi survey was used to develop consensus on items to be included in the guideline. A subsequent face-to-face meeting produced agreement on items not reaching consensus during the Delphi process.Results 143 participants agreed to participate in round one, with an 86% (123/143) response for round two and a 78% (112/143) response for round three. The Delphi survey identified the need for long form (LF) and short form (SF) versions. GRIPP2-LF includes 34 items on aims, definitions, concepts and theory, methods, stages and nature of involvement, context, capture or measurement of impact, outcomes, economic assessment, and reflections and is suitable for studies where the main focus is PPI. GRIPP2-SF includes five items on aims, methods, results, outcomes, and critical perspective and is suitable for studies where PPI is a secondary focus.Conclusions GRIPP2-LF and GRIPP2-SF represent the first international evidence based, consensus informed guidance for reporting patient and public involvement in research. Both versions of GRIPP2 aim to improve the quality, transparency, and consistency of the international PPI evidence base, to ensure PPI practice is based on the best evidence. In order to encourage its wide dissemination this article is freely accessible on The BMJ and Research Involvement and Engagement journal websites.

Surgical checklists: a systematic review of impacts and implementation

BACKGROUND

Surgical complications represent a significant cause of morbidity and mortality with the rate of major complications after inpatient surgery estimated at 3-17% in industrialised countries. The purpose of this review was to summarise experience with surgical checklist use and efficacy for improving patient safety.

METHODS

A search of four databases (MEDLINE, CINAHL, EMBASE and the Cochrane Database of Controlled Trials) was conducted from 1 January 2000 to 26 October 2012. Articles describing actual use of the WHO checklist, the Surgical Patient Safety System (SURPASS) checklist, a wrong-site surgery checklist or an anaesthesia equipment checklist were eligible for inclusion (this manuscript summarises all but the anaesthesia equipment checklists, which are described in the Agency for Healthcare Research and Quality publication).

RESULTS

We included a total of 33 studies. We report a variety of outcomes including avoidance of adverse events, facilitators and barriers to implementation. Checklists have been adopted in a wide variety of settings and represent a promising strategy for improving the culture of patient safety and perioperative care in a wide variety of settings. Surgical checklists were associated with increased detection of potential safety hazards, decreased surgical complications and improved communication among operating staff. Strategies for successful checklist implementation included enlisting institutional leaders as local champions, incorporating staff feedback for checklist adaptation and avoiding redundancies with existing systems for collecting information.

CONCLUSIONS

Surgical checklists represent a relatively simple and promising strategy for addressing surgical patient safety worldwide. Further studies are needed to evaluate to what degree checklists improve clinical outcomes and whether improvements may be more pronounced in particular settings.

Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist

BACKGROUND

The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114).

METHODS

75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item.

RESULTS

88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.

CONCLUSIONS

Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

Consensus on the reporting and experimental design of clinical and cognitive-behavioural neurofeedback studies (CRED-nf checklist)

Neurofeedback has begun to attract the attention and scrutiny of the scientific and medical mainstream. Here, neurofeedback researchers present a consensus-derived checklist that aims to improve the reporting and experimental design standards in the field.

Delirium detection in older acute medical inpatients: a multicentre prospective comparative diagnostic test accuracy study of the 4AT and the confusion assessment method

BACKGROUND

Delirium affects > 15% of hospitalised patients but is grossly underdetected, contributing to poor care. The 4 'A's Test (4AT, www.the4AT.com ) is a short delirium assessment tool designed for routine use without special training. The primary objective was to assess the accuracy of the 4AT for delirium detection. The secondary objective was to compare the 4AT with another commonly used delirium assessment tool, the Confusion Assessment Method (CAM).

METHODS

This was a prospective diagnostic test accuracy study set in emergency departments or acute medical wards involving acute medical patients aged ≥ 70. All those without acutely life-threatening illness or coma were eligible. Patients underwent (1) reference standard delirium assessment based on DSM-IV criteria and (2) were randomised to either the index test (4AT, scores 0-12; prespecified score of > 3 considered positive) or the comparator (CAM; scored positive or negative), in a random order, using computer-generated pseudo-random numbers, stratified by study site, with block allocation. Reference standard and 4AT or CAM assessments were performed by pairs of independent raters blinded to the results of the other assessment.

RESULTS

Eight hundred forty-three individuals were randomised: 21 withdrew, 3 lost contact, 32 indeterminate diagnosis, 2 missing outcome, and 785 were included in the analysis. Mean age was 81.4 (SD 6.4) years. 12.1% (95/785) had delirium by reference standard assessment, 14.3% (56/392) by 4AT, and 4.7% (18/384) by CAM. The 4AT had an area under the receiver operating characteristic curve of 0.90 (95% CI 0.84-0.96). The 4AT had a sensitivity of 76% (95% CI 61-87%) and a specificity of 94% (95% CI 92-97%). The CAM had a sensitivity of 40% (95% CI 26-57%) and a specificity of 100% (95% CI 98-100%).

CONCLUSIONS

The 4AT is a short, pragmatic tool which can help improving detection rates of delirium in routine clinical care.

TRIAL REGISTRATION

International standard randomised controlled trial number (ISRCTN) 53388093 . Date applied 30/05/2014; date assigned 02/06/2014.

Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation

The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of CVC insertion by 34 first-year medical residents were reviewed by two independent, trained evaluators. Each evaluator used three assessment tools: a ten-item checklist, a 21-item checklist, and a nine-item global rating scale. Exploratory principal component analysis of the global rating scale revealed two factors, accounting for 84.1% of the variance: technical ability and safety. The two checklist scores correlated positively with the weighted factor score on technical ability (0.49 [95% CI 0.17-0.71] for the 10-item checklist; 0.43 [95% CI 0.10-0.67] for the 21-item checklist) and negatively with the weighted factor score on safety (-0.17 [95% CI -0.48-0.18] for the 10-item checklist; -0.13 [95% CI -0.45-0.22] for the 21-item checklist). A checklist score of <80% was strong indication of incompetence. However, a high checklist score did not preclude incompetence. Ratings using the global rating scale identified an additional 11 candidates (32%) who were deemed incompetent despite scoring >80% on both checklists. All these candidates committed serious errors. In conclusion, the practice of universal adoption of checklists as the preferred method of assessment of procedural skills should be questioned. The inclusion of global rating scales should be considered.

Cross-informant agreement of the Dysregulation Profile of the Child Behavior Checklist

The Dysregulation Profile (DP) of the Child Behavior Checklist (CBCL) (previously called the CBCL-Juvenile Bipolar Disorder or CBCL-JBD profile) characterized by elevated scores on CBCL attention problems, aggressive behavior, and anxious/depressed scales is associated with severe psychopathology and suicidal behavior. The stability of this profile across informants has not been established. In this study, agreement across parent, teacher, and self-reports was examined for the Dysregulation Profile phenotype derived using latent class analysis of a national probability sample of 2031 children aged 6-18. The Dysregulation Profile latent class was found for each informant and accounted for 6-7% of the sample. There was mild to fair agreement on the Dysregulation Profile latent class between parents and youth (Kappa=0.22-0.25), parents and teachers (Kappa=0.14-0.24) and youth and teachers (Kappa=0.19-0.28). When parents and youth reports both placed children into the Dysregulation Profile latent class, 42% of boys and 67% of girls reported suicidal thoughts or behavior. We conclude that the Dysregulation Profile latent class is identified across informants although agreement of specific individuals is mild. Children in this class as identified by parental and youth reports have a very high risk for suicidal thoughts and behaviors.

TIDieR-Placebo: A guide and checklist for reporting placebo and sham controls

BACKGROUND

Placebo or sham controls are the standard against which the benefits and harms of many active interventions are measured. Whilst the components and the method of their delivery have been shown to affect study outcomes, placebo and sham controls are rarely reported and often not matched to those of the active comparator. This can influence how beneficial or harmful the active intervention appears to be. Without adequate descriptions of placebo or sham controls, it is difficult to interpret results about the benefits and harms of active interventions within placebo-controlled trials. To overcome this problem, we developed a checklist and guide for reporting placebo or sham interventions.

METHODS AND FINDINGS

We developed an initial list of items for the checklist by surveying experts in placebo research (n = 14). Because of the diverse contexts in which placebo or sham treatments are used in clinical research, we consulted experts in trials of drugs, surgery, physiotherapy, acupuncture, and psychological interventions. We then used a multistage online Delphi process with 53 participants to determine which items were deemed to be essential. We next convened a group of experts and stakeholders (n = 16). Our main output was a modification of the existing Template for Intervention Description and Replication (TIDieR) checklist; this allows the key features of both active interventions and placebo or sham controls to be concisely summarised by researchers. The main differences between TIDieR-Placebo and the original TIDieR are the explicit requirement to describe the setting (i.e., features of the physical environment that go beyond geographic location), the need to report whether blinding was successful (when this was measured), and the need to present the description of placebo components alongside those of the active comparator.

CONCLUSIONS

We encourage TIDieR-Placebo to be used alongside TIDieR to assist the reporting of placebo or sham components and the trials in which they are used.

Development of a checklist of safe discharge practices for hospital patients

BACKGROUND

Discharge from hospital can be a vulnerable period for patients. Multifaceted "discharge bundles" facilitate care transitions and possibly decrease adverse outcomes. We describe a structured approach to discharge planning, starting from admission and proceeding through discharge, using a standardized checklist of tasks to be performed for each hospitalization day.

OBJECTIVE

To create an evidence-based checklist of safe discharge practices for hospital patients.

METHODS

In the province of Ontario, the Ministry of Health and Long-Term Care convened a panel of expert members from multiple disciplines and across several healthcare sectors. The panel conducted a systematic search of the literature and used a structured approach to review evidence-based practices that ensure efficient, effective, safe, and patient-centered care transitions. A discharge-checklist tool was created to facilitate safe discharge from hospital.

RESULTS

The final checklist describes the processes necessary for a safe and optimal discharge and recommended timeline of when to complete each step, starting from the first day of admission. The checklist domains include (1) indication for hospitalization, (2) primary care, (3) medication safety, (4) follow-up plans, (5) home-care referral, (6) communication with outpatient providers, and (7) patient education.

CONCLUSIONS

The Checklist of Safe Discharge Practices for Hospital Patients summarizes the sequence of events that need to be completed throughout a typical hospitalization. Standardizing discharge planning and initiating processes early on in a patient's hospital stay may ensure a safe transition home.

The discovery of the Amazonian tree flora with an updated checklist of all known tree taxa

Amazonia is the most biodiverse rainforest on Earth, and the debate over how many tree species grow there remains contentious. Here we provide a checklist of all tree species collected to date, and describe spatial and temporal trends in data accumulation. We report 530,025 unique collections of trees in Amazonia, dating between 1707 and 2015, for a total of 11,676 species in 1225 genera and 140 families. These figures support recent estimates of 16,000 total Amazonian tree species based on ecological plot data from the Amazonian Tree Diversity Network. Botanical collection in Amazonia is characterized by three major peaks, centred around 1840, 1920, and 1980, which are associated with flora projects and the establishment of inventory plots. Most collections were made in the 20th century. The number of collections has increased exponentially, but shows a slowdown in the last two decades. We find that a species' range size is a better predictor of the number of times it has been collected than the species' estimated basin-wide population size. Finding, describing, and documenting the distribution of the remaining species will require coordinated efforts at under-collected sites.

STROGAR - STrengthening the Reporting Of Genetic Association studies in Radiogenomics

Despite publication of numerous radiogenomics studies to date, positive single nucleotide polymorphism (SNP) associations have rarely been reproduced in independent validation studies. A major reason for these inconsistencies is a high number of false positive findings because no adjustments were made for multiple comparisons. It is also possible that some validation studies were false negatives due to methodological shortcomings or a failure to reproduce relevant details of the original study. Transparent reporting is needed to ensure these flaws do not hamper progress in radiogenomics. In response to the need for improving the quality of research in the area, the Radiogenomics Consortium produced an 18-item checklist for reporting radiogenomics studies. It is recognised that not all studies will have recorded all of the information included in the checklist. However, authors should report on all checklist items and acknowledge any missing information. Use of STROGAR guidelines will advance the field of radiogenomics by increasing the transparency and completeness of reporting.

Patient-Reported Outcome Measures Related to Laryngopharyngeal Reflux: A Systematic Review of Instrument Development and Validation

OBJECTIVES

Patient-reported outcome (PRO) measures are often used to diagnose laryngopharyngeal reflux (LPR) and monitor treatment outcomes in clinical and research settings. The present systematic review was designed to identify currently available LPR-related PRO measures and to evaluate each measure's instrument development, validation, and applicability.

DATA SOURCES

MEDLINE via PubMed interface, CINAHL, and Health and Psychosocial Instrument databases were searched with relevant vocabulary and key terms related to PRO measures and LPR.

REVIEW METHODS

Three investigators independently performed abstract review and full text review, applying a previously developed checklist to critically assess measurement properties of each study meeting inclusion criteria.

RESULTS

Of 4947 studies reviewed, 7 LPR-related PRO measures (publication years, 1991-2010) met criteria for extraction and analysis. Two focused on globus and throat symptoms. Remaining measures were designed to assess LPR symptoms and monitor treatment outcomes in patients. None met all checklist criteria. Only 2 of 7 used patient input to devise item content, and 2 of 7 assessed responsiveness to change. Thematic deficiencies in current LPR-related measures are inadequately demonstrated: content validity, construct validity, plan for interpretation, and literacy level assessment.

CONCLUSION

Laryngopharyngeal reflux is often diagnosed according to symptoms. Currently available LPR-related PRO measures used to symptomatically identify suspected LPR patients have disparate developmental rigor and important methodological deficiencies. Care should be exercised to understand the measurement characteristics and contextual relevance before applying these PRO measures for clinical, research, or quality initiatives.

Five steps to develop checklists for evaluating clinical performance: an integrative approach

PURPOSE

The process of developing checklists to rate clinical performance is essential for ensuring their quality; thus, the authors applied an integrative approach for designing checklists that evaluate clinical performance.

METHOD

The approach consisted of five predefined steps (taken 2012-2013). Step 1: On the basis of the relevant literature and their clinical experience, the authors drafted a preliminary checklist. Step 2: The authors sent the draft checklist to five experts who reviewed it using an adapted Delphi technique. Step 3: The authors devised three scoring categories for items after pilot testing. Step 4: To ensure the changes made after pilot testing were valid, the checklist was submitted to an additional Delphi review round. Step 5: To weight items needed for accurate performance assessment, 10 pediatricians rated all checklist items in terms of their importance on a scale from 1 (not important) to 5 (essential).

RESULTS

The authors have illustrated their approach using the example of a checklist for a simulation scenario of infant septic shock. The five-step approach resulted in a valid, reliable tool and proved to be an effective method to design evaluation checklists. It resulted in 33 items, most consisting of three scoring categories.

CONCLUSIONS

This approach integrates published evidence and the knowledge of domain experts. A robust development process is a necessary prerequisite of valid performance checklists. Establishing a widely recognized standard for developing evaluation checklists will likely support the design of appropriate measurement tools and move the field of performance assessment in health care forward.

A new screening programme for autism in a general population of Swedish toddlers

The evidence from early intervention studies of autism has emphasised the need for early diagnosis. Insight into the early presentation of autism is crucial for early recognition, and routine screening can optimise the possibility for early diagnosis. General population screening was conducted for 2.5-year-old children at child health centres in Gothenburg, Sweden, and the efficacy of the screening instruments in predicting a clinical diagnosis of autism was studied. The tools used for autism screening comprised the Modified Checklist for Autism in Children (M-CHAT) and an observation made by trained nurses of the child's joint attention abilities (JA-OBS). From the new screening procedure a "definitive" suspicion of autism spectrum disorder (ASD) was raised in 64 individuals in the study population of 3999 young children. Fifty-four of these were clinically assessed in detail. Forty-eight children had a confirmed diagnosis of ASD, three had severe language disorder, and three (6%) were classified as having typical development. The Positive predictive Value (PPV) for the combination of M-CHAT and the JA-OBS was 90%. The combination of instruments used showed promise for early detection of autism as a routine in the developmental programme at child health centres. Trained medical staff is a basic requirement and enables earlier detection and the use of screening tools beyond routine population screening regardless of the age at which a suspicion of autism is raised.

Adapting the Consolidated Framework for Implementation Research to Create Organizational Readiness and Implementation Tools for Project ECHO

The Project Extension for Community Healthcare Outcomes (ECHO) model expands primary care provider (PCP) capacity to manage complex diseases by sharing knowledge, disseminating best practices, and building a community of practice. The model has expanded rapidly, with over 140 ECHO projects currently established globally. We have used validated implementation frameworks, such as Damschroder's (2009) Consolidated Framework for Implementation Research (CFIR) and Proctor's (2011) taxonomy of implementation outcomes, combined with implementation experience to (1) create a set of questions to assess organizational readiness and suitability of the ECHO model and (2) provide those who have determined ECHO is the correct model with a checklist to support successful implementation. A set of considerations was created, which adapted and consolidated CFIR constructs to create ECHO-specific organizational readiness questions, as well as a process guide for implementation. Each consideration was mapped onto Proctor's (2011) implementation outcomes, and questions relating to the constructs were developed and reviewed for clarity. The Preimplementation list included 20 questions; most questions fall within Proctor's (2001) implementation outcome domains of "Appropriateness" and "Acceptability." The Process Checklist is a 26-item checklist to help launch an ECHO project; items map onto the constructs of Planning, Engaging, Executing, Reflecting, and Evaluating. Given that fidelity to the ECHO model is associated with robust outcomes, effective implementation is critical. These tools will enable programs to work through key considerations to implement a successful Project ECHO. Next steps will include validation with a diverse sample of ECHO projects.

Assessment of dysregulated children using the Child Behavior Checklist: a receiver operating characteristic curve analysis

Disorders of self-regulatory behavior are common reasons for referral to child and adolescent clinicians. Here, the authors sought to compare 2 methods of empirically based assessment of children with problems in self-regulatory behavior. Using parental reports on 2,028 children (53% boys) from a U.S. national probability sample of the Child Behavior Checklist (CBCL; T. M. Achenbach & L. A. Rescorla, 2001), the receiver operating characteristic curve analysis was applied to compare scores on the Posttraumatic Stress Problems Scale (PTSP) of the CBCL with the CBCL Dysregulation Profile (DP), identified using latent class analysis of the Attention Problems, Aggressive Behavior, and Anxious/Depressed scales of the CBCL. The CBCL-PTSP score demonstrated an area under the curve of between .88 and .91 for predicting membership in the CBCL-DP profile for boys and for girls. These findings suggest that the CBCL-PTSP, which others have shown does not uniquely identify children who have been traumatized, does identify the same profile of behavior as the CBCL-DP. Therefore, the authors recommend renaming the CBCL-PTSP the Dysregulation Short Scale and provide some guidelines for the use of the CBCL-DP scale and the CBCL-PTSP in clinical practice.

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

BACKGROUND

The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research.

METHODS AND FINDINGS

We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies.

CONCLUSIONS

These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

Gastrointestinal Endoscopy Competency Assessment Tool: reliability and validity evidence

BACKGROUND

Rigorously developed and validated direct observational assessment tools are required to support competency-based colonoscopy training to facilitate skill acquisition, optimize learning, and ensure readiness for unsupervised practice.

OBJECTIVE

To examine reliability and validity evidence of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) for colonoscopy for use within the clinical setting.

DESIGN

Prospective, observational, multicenter validation study. Sixty-one endoscopists performing 116 colonoscopies were assessed using the GiECAT, which consists of a 7-item global rating scale (GRS) and 19-item checklist (CL). A second rater assessed procedures to determine interrater reliability by using intraclass correlation coefficients (ICCs). Endoscopists' first and second procedure scores were compared to determine test-retest reliability by using ICCs. Discriminative validity was examined by comparing novice, intermediate, and experienced endoscopists' scores. Concurrent validity was measured by correlating scores with colonoscopy experience, cecal and terminal ileal intubation rates, and physician global assessment.

SETTING

A total of 116 colonoscopies performed by 33 novice (<50 previous procedures), 18 intermediate (50-500 previous procedures), and 10 experienced (>1000 previous procedures) endoscopists from 6 Canadian hospitals.

MAIN OUTCOME MEASUREMENTS

Interrater and test-retest reliability, discriminative, and concurrent validity.

RESULTS

Interrater reliability was high (total: ICC=0.85; GRS: ICC=0.85; CL: ICC=0.81). Test-retest reliability was excellent (total: ICC=0.91; GRS: ICC=0.93; CL: ICC=0.80). Significant differences in GiECAT scores among novice, intermediate, and experienced endoscopists were noted (P<.001). There was a significant positive correlation (P<.001) between scores and number of previous colonoscopies (total: ρ=0.78, GRS: ρ=0.80, CL: Spearman's ρ=0.71); cecal intubation rate (total: ρ=0.81, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.75); ileal intubation rate (total: Spearman's ρ=0.82, GRS: Spearman's ρ=0.82, CL: Spearman's ρ=0.77); and physician global assessment (total: Spearman's ρ=0.90, GRS: Spearman's ρ=0.94, CL: Spearman's ρ=0.77).

LIMITATIONS

Nonblinded assessments.

CONCLUSION

This study provides evidence supporting the reliability and validity of the GiECAT for use in assessing the performance of live colonoscopies in the clinical setting.

A pilot study of seeking safety therapy with OEF/OIF veterans

PTSD and substance use disorder (SUD) are highly prevalent among veterans returning from Iraq and Afghanistan (Operation Enduring Freedom/Operation Iraqi Freedom; OEF/OIF). Seeking Safety (SS) is a cognitive-behavioral psychotherapy for co-occurring PTSD/SUD. This pilot study with fourteen male OEF/OIF veterans suggests that SS may help to reduce alcohol use, PTSD, and depression in some participants at clinically significant levels, even when providing less than half of the full model. We emphasize several SS features as especially helpful: the case management component to help engage clients in further mental health and SUD care, offering PTSD as an entry point, and emphasis on community resources. Issues particular to veterans include reintegration to civilian life and supporting their connection with other veterans.

COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures

PURPOSE

The original COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist was developed to assess the methodological quality of single studies on measurement properties of Patient-Reported Outcome Measures (PROMs). Now it is our aim to adapt the COSMIN checklist and its four-point rating system into a version exclusively for use in systematic reviews of PROMs, aiming to assess risk of bias of studies on measurement properties.

METHODS

For each standard (i.e., a design requirement or preferred statistical method), it was discussed within the COSMIN steering committee if and how it should be adapted. The adapted checklist was pilot-tested to strengthen content validity in a systematic review on the quality of PROMs for patients with hand osteoarthritis.

RESULTS

Most important changes were the reordering of the measurement properties to be assessed in a systematic review of PROMs; the deletion of standards that concerned reporting issues and standards that not necessarily lead to biased results; the integration of standards on general requirements for studies on item response theory with standards for specific measurement properties; the recommendation to the review team to specify hypotheses for construct validity and responsiveness in advance, and subsequently the removal of the standards about formulating hypotheses; and the change in the labels of the four-point rating system.

CONCLUSIONS

The COSMIN Risk of Bias checklist was developed exclusively for use in systematic reviews of PROMs to distinguish this application from other purposes of assessing the methodological quality of studies on measurement properties, such as guidance for designing or reporting a study on the measurement properties.

Validation of the Kihon Checklist and the frailty screening index for frailty defined by the phenotype model in older Japanese adults

BACKGROUND

The term "frailty" might appear simple, but the methods used to assess it differ among studies. Consequently, there is inconsistency in the classification of frailty and predictive capacity depending on the frailty assessment method utilised. We aimed to examine the diagnostic accuracy of several screening tools for frailty defined by the phenotype model in older Japanese adults.

METHODS

This cross-sectional study included 1,306 older Japanese adults aged ≥ 65 years who underwent physical check-up by cluster random sampling as part of the Kyoto-Kameoka Study in Japan. We evaluated the diagnostic accuracy of several screening instruments for frailty using the revised Japanese version of the Cardiovascular Health Study criteria as the reference standard. These criteria are based on the Fried phenotype model and include five elements: unintentional weight loss, weakness (grip strength), exhaustion, slowness (normal gait speed), and low physical activity. The Kihon Checklist (KCL), frailty screening index (FSI), and self-reported health were evaluated using mailed surveys. We calculated the non-parametric area under the receiver operating characteristic curve (AUC ROC) for several screening tools against the reference standard.

RESULTS

The participants' mean (standard deviation) age was 72.8 (5.5) years. The prevalence of frailty based on the Fried phenotype model was 12.2% in women and 10.3% in men. The AUC ROC was 0.861 (95% confidence interval: 0.832-0.889) for KCL, 0.860 (0.831-0.889) for FSI, and 0.668 (0.629-0.707) for self-reported health. The cut-off for identifying frail individuals was ≥ 7 points in the KCL and ≥ 2 points in the FSI.

CONCLUSIONS

Our results indicated that the two instruments (KCL and FSI) had sufficient diagnostic accuracy for frailty based on the phenotype model for older Japanese adults. This may be useful for the early detection of frailty in high-risk older adults.

Association of Checklist Use in Endotracheal Intubation With Clinically Important Outcomes: A Systematic Review and Meta-analysis

IMPORTANCE

Endotracheal intubation of critically ill patients is a high-risk procedure. Checklists have been advocated to improve outcomes.

OBJECTIVE

To assess whether the available evidence supports an association of use of airway checklists with improved clinical outcomes in patients undergoing endotracheal intubation.

DATA SOURCES

For this systematic review and meta-analysis, PubMed (OVID), Embase, Cochrane, CINAHL, and SCOPUS were searched without limitations using the Medical Subject Heading terms and keywords airway; management; airway management; intubation, intratracheal; checklist; and quality improvement to identify studies published between January 1, 1960, and June 1, 2019. A supplementary search of the gray literature was performed, including conference abstracts and clinical trial registries.

STUDY SELECTION

Full-text reviews were performed to determine final eligibility for inclusion. Included studies were randomized clinical trials or observational human studies that compared checklist use with any comparator for endotracheal intubation and assessed 1 of the predefined outcomes.

DATA EXTRACTION AND SYNTHESIS

Data extraction and quality assessment were performed using the Newcastle-Ottawa Scale for observational studies and Cochrane risk of bias tool for randomized clinical trials. Study results were meta-analyzed using a random-effects model. Reporting of this study follows the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline.

MAIN OUTCOMES AND MEASURES

The primary outcome was mortality. Secondary outcomes included first-pass success and known complications of endotracheal intubation, including esophageal intubation, hypoxia, hypotension, and cardiac arrest.

RESULTS

The search identified 1649 unique citations of which 11 (3261 patients) met the inclusion criteria. One randomized clinical trial and 3 observational studies had a low risk of bias. Checklist use was not associated with decreased mortality (5 studies [2095 patients]; relative risk, 0.97; 95% CI, 0.80-1.18; I2 = 0%). Checklist use was associated with a decrease in hypoxic events (8 studies [3010 patients]; relative risk, 0.75; 95% CI, 0.59-0.95; I2 = 33%) but no other secondary outcomes. Studies with a low risk of bias did not demonstrate decreased hypoxia associated with checklist use.

CONCLUSIONS AND RELEVANCE

The findings suggest that use of airway checklists is not associated with improved clinical outcomes during and after endotracheal intubation, which may affect practitioners' decision to use checklists in this setting.