Validity and reliability assessment of detailed scoring checklists for use during perioperative emergency simulation training. Simul Healthc. 2014;9:295-303. [PMID: 25188486 DOI: 10.1097/SIH.0000000000000048]

Malignant Hyperthermia Preparedness Training: Using Cognitive Aids and Emergency Checklists in the Perioperative Setting

PURPOSE

The purpose of this project was to prepare perioperative staff at a military ambulatory surgical center to respond to a malignant hyperthermia crisis effectively with the assistance the Stanford emergency manual cognitive aid.

DESIGN

This is an evidence-based practice improvement project.

METHODS

Multidisciplinary perioperative staff at a military ambulatory surgical center participated in an in-service about the Stanford emergency manual cognitive aid tool, and a simulation exercise was conducted to practice using the tool during a malignant hyperthermia crisis. Preproject and postproject implementation surveys were to used to measure staff perceptions of and willingness to use cognitive aids/emergency checklists during perioperative emergencies.

FINDINGS

Staff positive perception of and willingness to use the Stanford emergency manual cognitive aid improved after implementation of an education session and simulation exercise with the checklist tool.

CONCLUSIONS

Introduction of the Stanford emergency manual cognitive aid to multidisciplinary perioperative staff with an in-service and simulated malignant hyperthermia scenario improved staff perception of cognitive aid use during emergencies. Use of cognitive aid checklists during simulated perioperative emergencies was shown to reduce missed critical treatment steps.

Competence in Decision Making: Setting Performance Standards for Critical Care

BACKGROUND

Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees.

METHODS

Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards.

RESULTS

For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario.

CONCLUSIONS

Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains.

Medical Emergency Management in the Dental Office (MEMDO): A Pilot Study Assessing a Simulation-Based Training Curriculum for Dentists

In the event of a medical emergency in the dental office, the dentist must be able to identify a patient in distress, assess the situation, and institute proper management. This study assessed the impact of a simulation-based medical emergency preparedness curriculum on a resident's ability to manage medical emergencies. This interventional and pre-post educational pilot study included 8 participants who completed a standard curriculum and 8 who completed a modified curriculum (N = 16). The intervention consisted of a comprehensive medical emergency preparedness curriculum that replaced lecture sessions in a standard curriculum. Participants completed performance assessments using scenario-based objective structured clinical examinations (OSCEs) that were recorded and evaluated by calibrated faculty reviewers using a customized scoring grid. The intervention group performed significantly better than the control group on their summative OSCEs, averaging 90.9 versus 61.2 points out of 128 (p = .0009). All participants from the intervention group passed their summative OSCE with scores >60%, while none from the control group received passing scores. Completion of a simulation-based medical emergency preparedness curriculum significantly improved resident performance during simulated medical emergencies.

Validation of the Cutaneous Lupus Erythematosus Disease Area and Severity Index and pSkindex27 for use in childhood-onset systemic lupus erythematosus

Objective

To determine the measurement properties of the Cutaneous Lupus Erythematosus Disease Area and Severity Index (CLASI) and the paediatric adaptation of the Skindex29 (pSkindex27) when used in childhood-onset SLE (cSLE).

Methods

Patients with mucocutaneous involvement of cSLE were evaluated at the study entry and 6 months later. Besides the CLASI and pSkindex27, the Pediatric Quality of Life Inventory Generic Core scale (PedsQL-GC), its Rheumatology Module (PedsQL-RM), the SLE Disease Activity Index (SLEDAI) and the SLE Damage Index (SDI) were completed.

Results

The CLASI and pSkindex27 had high internal consistency (both Cronbach α >0.82). Children were able to complete the pSkindex27, with self-report and caregiver proxy-reports showing excellent agreement (intraclass correlation coefficient=0.97). The CLASI Activity Score (CLASI-A) was strongly correlated with the mucocutaneous domain score of the SLEDAI as was the CLASI Damage Score (CLASI-D) with that of the SDI (both: Spearman correlation coefficients (r_s) >0.68). pSkindex27 summary scores were moderately correlated with those of the PedsQL-GC and PedsQL-RM (all: r_s>|0.51|), the CLASI-A and CLASI-D (both: r_s> 0.64), respectively. Patients who experienced a >50% improvement of the CLASI-A between study visits had significantly higher PedsQL-GC and pSkindex27 scores than those without improvement of mucocutaneous features.

Conclusion

Both CLASI and pSkindex27 are useful assessment tools in cSLE, active and chronic mucocutaneous lesions and their changes over time can be measured using the CLASI and the pSkindex27 can capture the impact of mucocutaneous involvement on patient health-related quality of life.

Dental students' ability to locate emergency equipment-lessons learned from aviation

PURPOSE

The purpose of this study was to evaluate the dental student's ability to locate medical emergency equipment/items at the University of Michigan School of Dentistry clinic.

METHODS

A total of 138 second-year dental students (traditional group) participated in this study as part of a simulation-based medical emergencies rotation course held during the winter term of 2014 and 2015. Without prior training, students were tested on their ability to locate nine predetermined items on the clinic floor using a self-reported checklist. Six months later, a convenience sample of 18 students (novel group) from the same cohort were later trained on their location and retested individually.

RESULTS

Of the 138 students tested, only 10.14% students could locate seven of the nine items when compared to 100% in the novel group. Only 5.07% of students in the traditional group could locate all items initially, compared with 72.22% students in the novel group.

CONCLUSION

Whilst our students have lecture-based knowledge about medical emergencies, the results of our study identified a gap of knowledge of emergency equipment/item location amongst students. Therefore, an intervention performed with a similar group of second-year dental students supported that proper training may be used to achieve retention of knowledge. Based on our "novel group" results, we have incorporated targeted training in the dental curriculum that leads to students being better prepared in locating emergency equipment/items. This study suggests that other populations, such as faculty or staff, may also benefit from hands-on training.

Development and Testing of a Short Peripheral Intravenous Catheter Insertion Skills Checklist

To date, there is no published, psychometrically validated, short peripheral intravenous catheter (PIVC) insertion skills checklist. Creating a valid, reliable, and generalizable checklist to measure PIVC skill is a key step in assessing baseline competence and skill mastery. Based on recognized standards and best practices, the PIVC Insertion Skills Checklist was developed to measure all the steps necessary for a best practice PIVC insertion. This includes the entire process from reading the prescriber's orders to documentation and, if the first attempt is unsuccessful, a second attempt option. Content validity was established using 3 infusion therapy experts. Evidence in support of response process validity is described. The PIVC Insertion Skills Checklist was used by 8 trained raters to assess the PIVC insertion skills, in a simulated environment, of 63 practicing clinicians working on medical and surgical units in a US teaching hospital. Internal consistency of the PIVC Insertion Skills Checklist was α = 0.84. Individual item intraclass correlation coefficients (ICCs) between rater and gold standard observations ranged from − 0.01 to 1.00 and total score ICC was 0.99 (95% confidence interval, 0.99–0.99). The current study offers validity and reliability evidence to support the use of the PIVC Insertion Skills Checklist to measure PIVC insertion skill of clinicians in a simulated environment.

Interprofessional simulation training for perioperative management team development and patient safety

Establishment of a perioperative management team construct including anaesthesiologists, surgeons, nurses, and other medical staff is essential to optimize safe surgical care. Simulation based education and training provides a unique and effective approach to development of competency and application of relevant technical and non-technical perioperative professional skills such as meta-cognitive ability, caution, shared decision-making, leadership and communication. Development of high functioning perioperative teams can be accomplished through simulation based training.

Contextualization and validation of the interprofessional collaborator assessment rubric (ICAR) through simulation: Pilot investigation

BACKGROUND

Simulation can be used for educating, evaluating and assessing psychometric properties of an instrument. The aim of this study was to contextualize and assess the validity and reliability of the Interprofessional Collaborative Assessment tool (ICAR) in an Iranian context using simulation.

METHODS

In this descriptive study, contextualization of the ICAR was assessed through several steps. Firstly, validity assessment was approved through expert panels and Delphi rounds. Secondly, reliability assessment was done by arranging a simulation video and assessing reproducibility, test-retest (ICC), internal consistency (Cronbach's Alpha) and inter-rater reliability (Kappa).The participants included 26 experts, 27 students and 6 staff of the Standardized Simulation Office of Teheran University of Medical Sciences.

RESULTS

Contextualization and validity of the ICAR were approved in an Iranian context. The reliability of the tool was computed to be 0.71 according to Cronbach´s Alpha. The test-retest was calculated to be 0.76.

CONCLUSION

The Iranian ICAR can be a useful tool for evaluating interprofessional collaborative competencies. The development of the instrument through a simulation scenario has been a positive prospect for researchers.

Internet Training Resulted in Improved Trainee Performance in a Simulated Opioid-Poisoned Patient as Measured by Checklist

INTRODUCTION

Opioid overdose is a leading cause of death in the USA. Internet-based teaching can improve medical knowledge among trainees, but there are limited data to show the effect of Internet-based teaching on clinical competence in medical training, including management of opioid poisoning.

METHODS

We used an ecological design to assess the effect of an Internet-based teaching module on the management of a simulated opioid-poisoned patient. We enrolled two consecutive classes of post-graduate year-1 residents from a single emergency medicine program. The first group (RA) was instructed to read a toxicology textbook chapter and the second group (IT) took a brief Internet training module. All participants subsequently managed a simulated opioid-poisoned patient. The participants' performance was evaluated with two types of checklist (simple and time-weighted), along with global assessment scores.

RESULTS

Internet-trained participants performed better on both checklist scales. The difference between mean simple checklist scores by the IT and RA groups was 0.23 (95 % CI, 0.016-0.44). The difference between mean time-weighted checklist scores was 0.27 (95 % CI, 0.048-0.49). When measured by global assessment, there was no statistically significant difference between RA and IT participants.

CONCLUSION

These data suggest that the Internet module taught basic principles of management of the opioid-poisoned patient. In this scenario, global assessment and checklist assessment may not measure the same proficiencies. These encouraging results are not sufficient to show that this Internet tool improves clinical performance. We should assess the impact of the Internet module on performance in a true clinical environment.

Operator competence in fetoscopic laser surgery for twin-twin transfusion syndrome: validation of a procedure-specific evaluation tool

OBJECTIVES

Fetoscopic laser surgery for twin-twin transfusion syndrome is a procedure for which no objective tools exist to assess technical skills. To ensure that future fetal surgeons reach competence prior to performing the procedure unsupervised, we developed a performance assessment tool. The aim of this study was to validate this assessment tool for reliability and construct validity.

METHODS

We made use of a procedure-specific evaluation instrument containing all essential steps of the fetoscopic laser procedure, which was previously created using Delphi methodology. Eleven experts and 13 novices from three fetal medicine centers performed the procedure on the same simulator. Two independent observers assessed each surgery using the instrument (maximum score: 52). Interobserver reliability was assessed using Spearman correlation. We compared the performance of novices and experts to assess construct validity.

RESULTS

The interobserver reliability was high (Rs  = 0.974, P < 0.001). Checklist scores for experts and novices were significantly different; the median score for novices was 28/52 (54%), whereas that for experts was 47.5/52 (91%) (P < 0.001). The procedure time and fetoscopy time were significantly shorter (P < 0.001) for experts. Residual anastomoses were found in 1/11 (9%) procedures performed by experts and in 9/13 (69%) procedures performed by novices (P = 0.005). Multivariable analysis showed that the checklist score, independent of age and gender, predicted competence.

CONCLUSIONS

The procedure-specific assessment tool for fetoscopic laser surgery shows good interobserver reliability and discriminates experts from novices. This instrument may therefore be a useful tool in the training curriculum for fetal surgeons. Further intervention studies with reassessment before and after training may increase the construct validity of the tool. Copyright © 2015 ISUOG. Published by John Wiley & Sons Ltd.

Reliability and validity of the Chinese version of the Liver Disease Quality of Life Questionnaire (LDQOL 1.0)

AIM: To evaluate the reliability and validity of Chinese version of Liver Disease Quality of Life Questionnaire (LDQOL 1.0)

METHODS: This was an observational, cross-sectional study in Chinese patients from Beijing University Shenzhen Hospital. Feasibility was assessed by analyzing administration times and missing responses. Ceiling and flooring effects were calculated and reliability was tested by examining internal consistency (Cronbach's alpha). Convergent validity was tested by examining correlations between LDQOL disease specific and Short Form health survey with 36 questions (SF-36) dimensions.

RESULTS: A total of 48 patients were included for analysis. Mean age was 39.3 years and 73% of the sample were male. Mean time to complete questionnaire was 35.8 min ±12.2 min. Missing responses were highest on the dimensions of sexual functioning. Ceiling effects were over 20% on 1 of the LDQOL's 12 disease-specific scales. Cronbach's alpha coefficients were over 0.70 on all but 6 dimensions. Correlations between SF-36 and LDQOL disease-specific dimensions generally fulfilled the hypotheses.

CONCLUSION: The Chinese version of the LDQOL 1.0 has shown satisfactory reliability and validity.