Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review

Criteria to evaluate graduate nurse proficiencies in obtaining a health history and perform physical assessment in simulation-based education: A narrative review

BACKGROUND

Simulation is a technique being used increasingly in healthcare education which offers opportunities to evaluate nursing proficiencies. The use of valid and reliable instruments is recognised as the foundation for a robust assessment, however competency-based health assessment courses for graduate nurses can consequently become reductionist in measuring proficiencies.

OBJECTIVE

The specific review question was: In simulation-based education, what are the criteria that evaluate graduate nursing student's competence in obtaining a health history and performance of patient assessment?

METHODS

Eleven studies were included in the review. Papers were critically appraised with The Joanna Briggs Institute quasi-experimental studies checklist. Bloom's taxonomy was used to structure this narrative review.

RESULTS

Seven papers evaluated cognition through questionnaires and two papers used a Likert-scale to determine self-perceived knowledge. Six papers evaluated psychomotor skills with a behavioural checklist. Diversity of application was factored into the studies when testing affective skills. Three papers used Likert-scales to evaluate preparedness, six papers used Likert-scales to evaluate self-confidence and one used a Likert-scale to evaluate autonomy. Three papers used a checklist to evaluate professionalism. Four papers used faculty member/ standardised patient feedback.

CONCLUSION

Reductionist evaluation instruments create a barrier when evaluating competency. The limited validity and reliability of assessment instruments in simulation, as well as the lack of standardisation of affective skills assessment, presents a challenge in simulation research. Affective skills encompass attitudes, behaviours and communication abilities, which pose a significant challenge for standardised assessments due to their subjective nature. This review of the simulation literature highlights a lack of robustness in the evaluation of the affective domain. This paper proposes that simulation assessment instruments should include the standardisation of affective domain proficiencies such as: adaptation to patients' cognitive function, ability to interpret and synthesise relevant information, ability to demonstrate clinical judgement, readiness to act, recognition of professional limitations and faculty/standardised-simulated patient feedback. The incorporation of the affective domain in standardised assessment instruments is important to ensure comprehensive assessment of simulation particularly in the development of health history and physical assessment proficiencies. Attention to all of the domains in Blooms taxonomy during simulation assessment has the potential to better prepare professionals for the patient care setting.

The Sequence of Steps: A Key Concept Missing in Surgical Training-A Systematic Review and Recommendations to Include It

BACKGROUND

Surgical procedures have an inherent feature, which is the sequence of steps. Moreover, studies have shown variability in surgeons' performances, which is valuable to expose residents to different ways to perform a procedure. However, it is unclear how to include the sequence of steps in training programs.

METHODS

We conducted a systematic review, including studies reporting explicit teaching of a standard sequence of steps, where assessment considered adherence to a standard sequence, and where faculty or students at any level participated. We searched for articles on PubMed, EMBASE, CINAHL, Web of Science, and Google Scholar databases.

RESULTS

We selected nine articles that met the inclusion criteria. The main strategy to teach the sequence was to use videos to demonstrate the procedure. The simulation was the main strategy to assess the learning of the sequence of steps. Non-standardized scoring protocols and written tests with variable validity evidence were the instruments used to assess the learning, and were focused on adherence to a standard sequence and the omission of steps.

CONCLUSIONS

Teaching and learning assessment of a standard sequence of steps is scarcely reported in procedural skills training literature. More research is needed to evaluate whether the new strategies to teach and assess the order of steps work. We recommend the use of Surgical Process Models and Surgical Data Science to incorporate the sequence of steps when teaching and assessing procedural skills.

ProDeM: A Process-Oriented Delphi Method for systematic asynchronous and consensual surgical process modelling

Surgical process models support improving healthcare provision by facilitating communication and reasoning about processes in the medical domain. Modelling surgical processes is challenging as it requires integrating information that might be fragmented, scattered, and not process-oriented. These challenges can be faced by involving healthcare domain experts during process modelling. This paper presents ProDeM: a novel Process-Oriented Delphi Method for the systematic, asynchronous, and consensual modelling of surgical processes. ProDeM is an adaptable and flexible method that acknowledges that: (i) domain experts have busy calendars and might be geographically dispersed, and (ii) various elements of the process model need to be assessed to ensure model quality. The contribution of the paper is twofold as it outlines ProDeM, but also demonstrates its operationalisation in the context of a well-known surgical process. Besides showing the method's feasibility in practice, we also present an evaluation of the method by the experts involved in the demonstration.

Achieving Lumbar Epidural Block Competency in Inexperienced Trainees after a Structured Epidural Teaching Model: A Randomized, Single Blind, Prospective Comparison of CUSUM Learning Curves

Aim

The aim of this randomized, prospective study was to investigate whether the use of the structured epidural teaching model (SETM) may affect the learning curve for lumbar epidural block in novice trainees when compared with a standard teaching module.

Introduction

There is a paucity of literature regarding the efficacy of teaching epidural blocks and comparisons between the different educational approaches.

Method

Forty-four PGY3 anesthesia trainees were randomized to receive (study group) or to not receive (control group) the SDM (structured didactic model) before the beginning of their 6 months clinical practice rotation in labor and delivery suites. A CUSUM learning curve was built for every trainee. The scores were assigned by the staff instructor, who was unaware of the group to which the trainee belonged.

Results

The number of subjects who achieved an improvement in performance was 8 trainees from the control group and 14 from the study group. The probability of achieving an improvement was higher (p < 05) in the study group than in the control group, with an aOR of 3.25 (CI: 1.01; 12.1). The proportion of subjects in the study group who completed the epidural without help was 1.21 (1.05–1.41) times the proportion of subjects who completed the epidural without help in the control group. The probability of completing the epidural block without any assistance was 21% higher in the study group than in the control group (p < 05).

Conclusion

We have demonstrated that the use of the structured epidural teaching model (SETM) may improve the learning curve (CUSUM) for lumbar epidural block in novice, entirely inexperienced, anesthesia trainees.

Development and Validation of a Performance Assessment Checklist Scale for Vaccine Administration

Objective Vaccination technique is a crucial skill for medical trainees to learn, especially in the current coronavirus disease 2019 pandemic. To this end, validated assessment tools are essential in teaching appropriate techniques. However, valid instruments for assessing vaccine administration skills have not yet been developed. We therefore explored the development and validation of an assessment tool for vaccination techniques based on expert consensus. Methods We implemented a modified Delphi process to develop a vaccination technique assessment tool. We then conducted a validation study to establish the reliability and validity of the tool. Results Two rounds of the modified Delphi process were performed to generate a 19-item, vaccination performance assessment checklist. In the validation study, the linear weighted kappa value for inter-rater reliability of the overall checklist score was 0.725. Spearman's correlation coefficient between the mean checklist score and the global rating was 0.98 (p<0.01). Conclusions This is a pioneering study examining the development and validation of an assessment tool for vaccine administration techniques. The tool will be widely used in vaccination-related education.

Becoming virtually real: using the Virtual Environment for Radiotherapy Training (VERT™) platform for the summative assessment of performance in a palliative radiotherapy treatment technique

Introduction:

Direct observation in the radiotherapy clinic remains the primary method for the assessment of procedural skills. But with increasing automation and reductions in clinical placement time during the recent pandemic, the role of summative, simulation-based assessment is being revisited. The aim of this article is to share early experiences of using Virtual Environment for Radiotherapy Training (VERT) in the assessment of a palliative, parallel pair, external beam treatment delivery technique in a new pre-registration undergraduate programme.

Methods:

Eight first-year students completed a campus-based individual virtual assessment using virtual patient plans representing metastatic disease in the brain and pelvis and late-stage primary lung tumours. Performance was logged on a 25-item checklist for pre and post-procedure checks and treatment delivery tasks.

Results:

All eight students participated in the assessment with seven students (87·5%) achieving a pass grade at first attempt. The course team observed that participants demonstrated a range of skills and abilities and were able to compare and contrast individual approaches to patient positioning. Specific feed-forward action points were also highlighted as areas for students’ to focus on during their second placement. The project also identified logistical benefits for assessment teams.

Conclusions:

While these experiences are a single snapshot, a VERT simulation-based summative assessment is feasible and identified benefits included controlled observation and structured feedback on individual performance and scheduling pressure reductions for clinical teams. However, more work is needed to determine the psychometric qualities and predictability of performance in more complex techniques.

Sustainable Approach to Certification of Persons: Ensuring Reliability and Quality

Nowadays, sustainability issues are gaining more and more topicality in the context of improving organizational processes, including in the field of conformity assessment. In the field of certification of persons, competence assessment institutions have also become interested in using new approaches as the quality of assessment execution does not fully meet the requirements of stakeholders regarding the ability of a person to apply the acquired knowledge and skills in situations related to the professional activity. The aim of the research is to analyze the aspects of professional competence assessment related to the certification of persons and its performance according to the stakeholder requirements in order to develop a new sustainable approach to the process of certification of persons that would ensure the quality and reliability of its execution. Qualitative and quantitative research methods have been used to analyze the elements of the concept of certification of persons and the main elements of the concept of professional competence. As a result of the research, a new sustainable approach to certification of persons is offered, where certification of persons transforms from a conformity assessment procedure into a professional competence assessment process and ensures that a person’s ability is assessed not only in accordance with the industry requirements, but also with the stakeholder needs.

Developing a tool to assess competence in resuscitative endovascular balloon occlusion of the aorta: An international Delphi consensus study

BACKGROUND

Resuscitative endovascular balloon occlusion of the aorta (REBOA) is an emergency procedure that is potentially lifesaving in major noncompressible torso hemorrhage. It may also improve outcome in nontraumatic cardiac arrest. However, the procedure can be technically challenging and requires the immediate presence of a qualified operator. Thus, evidence-based training and assessment of operator skills are essential for successful implementation and patient safety. A prerequisite for this is a valid and reliable assessment tool specific for the procedure. The aim of this study was to develop a tool for assessing procedural competence in REBOA based on best-available knowledge from international experts in the field.

METHODS

We invited international REBOA experts from multiple specialties to participate in an anonymous three-round iterative Delphi study to reach consensus on the design and content of an assessment tool. In round 1, participants suggested items to be included. In rounds 2 and 3, the relevance of each suggested item was evaluated by all participants to reach consensus. Interround data processing was done systematically by a steering group.

RESULTS

Forty panelists representing both clinical and educational expertise in REBOA from 16 countries (in Europe, Asia, and North and South America) and seven different specialties participated in the study. After 3 Delphi rounds and 532 initial item suggestions, the panelists reached consensus on a 10-item assessment tool with behaviorally anchored rating scales. It includes assessment of teamwork, procedure time, selection and preparation of equipment, puncture technique, guidewire handling, sheath handling, placement of REBOA catheter, occlusion, and evaluation.

CONCLUSION

We present the REBOA-RATE assessment tool developed systematically by international experts in the field to optimize content validity. Following further studies of its validity and reliability, this tool represents an important next step in evidence-based training programs in REBOA, for example, using mastery learning.

LEVEL OF EVIDENCE

Therapeutic, level V.

Development of a comprehensive Percutaneous Dilatational Tracheostomy process model for procedural training: A Delphi-based experts consensus

BACKGROUND

Deconstructing a complex procedure improves skills learning, but no model has covered all relevant Percutaneous Dilatational Tracheostomy (PDT) procedural aspects. Moreover, the heterogeneity of techniques described may hinder trainees' competency acquisition. Our objective was to develop a PDT model for procedural training that includes a comprehensive step-by-step design.

METHODS

Procedural descriptions were retrieved after a structured search in medical databases. Activities were extracted and the adherence to McKinley's dimensions of procedural competence was analyzed. We developed a comprehensive PDT model, which was further validated through a Delphi-based consensus of Spanish-speaking international experts.

RESULTS

The 14 descriptions retrieved for analysis presented a median [interquartile range] of 18 [11-22] steps, covering 3 [2-4] of McKinley's dimensions. The Delphi panel's first model included all McKinley's dimensions, and was answered by 25 experts from nine countries, ending in the second round. The final model included 59 activities divided into six stages (51 from the initial model and eight proposed by experts) and performed by two operators (bronchoscopy and tracheostomy).

CONCLUSIONS

We have presented a PDT model that includes necessary competence dimensions to be considered complete. The model was validated by an experts' consensus, allowing to improve procedural training to promote safer patient care.

Simulation training for emergency skills: effects on ICU fellows' performance and supervision levels

BACKGROUND

The application of manual emergency skills is essential in intensive care medicine. Simulation training on cadavers may be beneficial. The aim of this study was to analyze a skill-training aiming to enhance ICU-fellows´ performance.

METHODS

A skill-training was prepared for chest tube insertion, pericardiocentesis, and cricothyroidotomy. Supervision levels (SL) for entrustable professional activities (EPA) were applied to evaluate skill performance. Pre- and post-training, SL and fellows´ self- versus consultants´ external assessment was compared. Time on skill training was compared to conventional training in the ICU-setting.

RESULTS

Comparison of pre/post external assessment showed reduced required SL for chest tube insertion, pericardiocentesis, and cricothyroidotomy. Self- and external assessed SL did not significantly correlate for pre-training/post-training pericardiocentesis and post-training cricothyroidotomy. Correlations were observed for self- and external assessment SL for chest tube insertion and pre-assessment for cricothyroidotomy. Compared to conventional training in the ICU-setting, chest tube insertion training may further be time-saving.

CONCLUSIONS

Emergency skill training separated from a daily clinical ICU-setting appeared feasible and useful to enhance skill performance in ICU fellows and may reduce respective SL. We observed that in dedicated skill-training sessions, required time resources would be somewhat reduced compared to conventional training methods.

How to rapidly design and operationalise PPE donning and doffing areas for a COVID-19 care facility: quality improvement initiative

Introduction

Effective implementation of standard precautions specific to COVID-19 is a challenge for hospitals within the existing constraints of time and resources.

Aim

To rapidly design and operationalise personal protective equipment (PPE) donning and doffing areas required for a COVID-19 care facility.

Methods

Literature review was done to identify all issues pertaining to donning and doffing in terms of Donabedian’s structure, process and outcome. Training on donning and doffing was given to hospital staff. Donning and doffing mock drills were held. 5S was used as a tool to set up donning and doffing areas. Instances of donning and doffing were observed for protocol deviations and errors. Plan–do–study–act cycles were conducted every alternate day for 4 weeks. The initiative was reported using Standards for QUality Improvement Reporting Excellence (SQUIRE) guidelines.

Results

Best practices in donning and doffing were described. Our study recommends a minimum area of 16 m² each for donning and doffing rooms. Verbally assisted doffing was found most useful than visual prompts.

Discussion

Challenges included sustaining the structure and process of donning and doffing, varied supplies of PPE which altered sequencing of donning and/or doffing, and training non-healthcare workers such as plumbers, electricians and drivers who were required during emergencies in the facility.

Conclusion

Our study used evidence-based literature and quality improvement (QI) tools to design and operationalise donning and doffing areas with focus on people, task and environment. Our QI will enable healthcare facilities to rapidly prototype donning and doffing areas in a systematic way.

The assessment of procedural skills in physiotherapy education: a measurement study using the Rasch model

Background

Procedural skills are a key element in the training of future physiotherapists. Procedural skills relate to the acquisition of appropriate motor skills, which allow the safe application of clinical procedures to patients. In order to evaluate procedural skills in physiotherapy education validated assessment instruments are required. Recently the assessment of procedural skills in physiotherapy education (APSPT) tool was developed. The overall aim of this study was to establish the structural validity of the APSPT. In order to do this the following objectives were examined: i) the fit of the items of APSPT to the Rasch-model, ii) the fit of the overall score to the Rasch model, iii) the difficulty of each test item and iv) whether the difficulty levels of the individual test items cover the whole capacity spectrum of students in pre-registration physiotherapy education.

Methods

For this observational cross-sectional measurement properties study a convenience sample of 69 undergraduate pre-registration physiotherapy students of the HES-SO Valais-Wallis was recruited. Participants were instructed to perform a task procedure on a simulated patient. The performance was evaluated with the APSPT. A conditional maximum likelihood approach was used to estimate the parameters of a partial credit model for polytomous item responses. Item fit, ordering of thresholds, targeting and goodness of fit to the Rasch model was assessed.

Results

Item fit statistics showed that 25 items of the APSPT showed adequate fit to the Rasch model. Disordering of item thresholds did not occur and the targeting of the APSPT was adequate to measure the abilities of the included participants. Undimensionality and subgroup homogeneity were confirmed.

Conclusion

This study presented evidence for the structural validity of the APSPT. Undimensionality of the APSPT was confirmed and therefore presents evidence that the latent dimension of procedural skills in physiotherapy education consists of several subcategories. However, the results should be interpreted with caution given the small sample size.

Delphi Method to Achieve Clinical Consensus for a BPMN Representation of the Central Venous Access Placement for Training Purposes

Proper teaching of the technical skills necessary to perform a medical procedure begins with its breakdown into its constituent steps. Currently available methodologies require substantial resources and their results may be biased. Therefore, it is difficult to generate the necessary breakdown capable of supporting a procedural curriculum. The aim of our work was to breakdown the steps required for ultrasound guided Central Venous Catheter (CVC) placement and represent this procedure graphically using the standard BPMN notation. Methods: We performed the first breakdown based on the activities defined in validated evaluation checklists, which were then graphically represented in BPMN. In order to establish clinical consensus, we used the Delphi method by conducting an online survey in which experts were asked to score the suitability of the proposed activities and eventually propose new activities. Results: Surveys were answered by 13 experts from three medical specialties and eight different institutions in two rounds. The final model included 28 activities proposed in the initial model and four new activities proposed by the experts; seven activities from the initial model were excluded. Conclusions: The proposed methodology proved to be simple and effective, generating a graphic representation to represent activities, decision points, and alternative paths. This approach is complementary to more classical representations for the development of a solid knowledge base that allows the standardization of medical procedures for teaching purposes.

Assessing Competence in Central Venous Catheter Placement by Pediatric Critical Care Fellows: A National Survey Study

OBJECTIVES

To describe the current approach to initial training, ongoing skill maintenance, and assessment of competence in central venous catheter placement by pediatric critical care medicine fellows, a subset of trainees in whom this skill is required.

DESIGN

Cross-sectional internet-based survey with deliberate sampling.

SETTING

United States pediatric critical care medicine fellowship programs.

SUBJECTS

Pediatric critical care medicine program directors of Accreditation Council for Graduate Medical Education-accredited fellowship programs.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

A working group of the Education in Pediatric Intensive Care Investigators research collaborative conducted a national study to assess the degree of standardization of training and competence assessment of central venous catheter placement across pediatric critical care medicine fellowship programs. After piloting, the survey was sent to all program directors (n = 67) of Accreditation Council for Graduate Medical Education-accredited pediatric critical care medicine programs between July 2017 and September 2017. The response rate was 85% (57/67). Although 98% of programs provide formalized central venous catheter placement training for first-year fellows, only 42% of programs provide ongoing maintenance training as part of fellowship. Over half (55%) of programs use a global assessment tool and 33% use a checklist-based tool when evaluating fellow central venous catheter placement competence under direct supervision. Only two programs (4%) currently use an assessment tool previously published and validated by the Education in Pediatric Intensive Care group. A majority (82%) of responding program directors believe that a standardized approach to assessment of central venous catheter competency across programs is important.

CONCLUSIONS

Despite national mandates for skill competence by many accrediting bodies, no standardized system currently exists across programs for assessing central venous catheter placement. Most pediatric critical care medicine programs use a global assessment and decisions around the ability of a fellow to place a central venous catheter under indirect supervision are largely based upon subjective assessment of performance. Further investigation is needed to determine if this finding is consistent in other specialties/subspecialties, if utilization of standardized assessment methods can improve program directors' abilities to ensure trainee competence in central venous catheter insertion in the setting of variable training approaches, and if these findings are consistent with other procedures across critical care medicine training programs, adult and pediatric.

Graduating Fellows' Procedural Comfort Level With Pulmonary Critical Care Procedures

BACKGROUND

Graduating fellows from pulmonary and critical care programs are expected to independently perform bronchoscopy and common medical procedures in the intensive care unit. Given variable exposure and learning opportunities at different training programs, little is known about how comfortable graduating fellows are with these procedures.

METHODS

A survey concerning the ACGME required procedures for pulmonary and critical care fellowship was sent to graduating fellows and program directors at all ACGME accredited subspecialty fellowship programs in pulmonary, critical care, and combined pulmonary critical care medicine.

RESULTS

Critical care fellows performed the most intubations, paracenteses, arterial lines, central venous catheter insertions, and cardioversions, but did not perform as many bronchoscopy related procedures as the other subgroups. Pulmonary and combined pulmonary/critical care fellows performed and felt confident in most procedures. Program directors from all specialties agreed with the fellows overall about the procedural confidence gained during training. There also appeared to be a correlation between number of procedures performed and the confidence of the proceduralist. There appears to be a certain threshold number of successful completions to achieve confidence for independent practice for each surveyed procedure.

CONCLUSION

There is a correlation between a fellow's procedural comfort with independent practice and the number of successful attempts by a graduating fellow. There are no universal guidelines that outline the appropriate number of successful attempts needed for independent practice, and our data strongly suggests that there is a threshold for each procedure across these three subspecialties.

Twelve tips for efficient procedural simulation

Procedural simulation (PS) is increasingly being used worldwide in healthcare for training caregivers in psychomotor competencies. It has been demonstrated to improve learners' confidence and competence in technical procedures, with consequent positive impacts on patient outcomes and safety. Several frameworks can guide healthcare educators in using PS as an educational tool. However, no theory-informed practical framework exists to guide them in including PS in their training programs. We present 12 practical tips for efficient PS training that translates educational concepts from theory to practice, based on the existing literature. In doing this, we aim to help healthcare educators to adequately incorporate and use PS both for optimal learning and for transfer into professional practice.

Nursing students' evaluation of a new feedback and reflection tool for use in high-fidelity simulation - Formative assessment of clinical skills. A descriptive quantitative research design

Clinical skills training is an important part of nurses' education programmes. Clinical skills are complex. A common understanding of what characterizes clinical skills and learning outcomes needs to be established. The aim of the study was to develop and evaluate a new reflection and feedback tool for formative assessment. The study has a descriptive quantitative design. 129 students participated who were at the end of the first year of a Bachelor degree in nursing. After highfidelity simulation, data were collected using a questionnaire with 19 closed-ended and 2 open-ended questions. The tool stimulated peer assessment, and enabled students to be more thorough in what to assess as an observer in clinical skills. The tool provided a structure for selfassessment and made visible items that are important to be aware of in clinical skills.

CONCLUSIONS

This article adds to simulation literature and provides a tool that is useful in enhancing peer learning, which is essential for nurses in practice. The tool has potential for enabling students to learn about reflection and developing skills for guiding others in practice after they have graduated.

Quality of Widely Available Video Instructional Materials for Point-of-Care Ultrasound-Guided Procedure Training in Internal Medicine

OBJECTIVES

Many instructional materials for point-of-care ultrasound (US)-guided procedures exist; however, their quality is unknown. This study assessed widely available educational videos for point-of-care US-guided procedures relevant to internal medicine: central venous catheterization, thoracentesis, and paracentesis.

METHODS

We searched Ovid MEDLINE, YouTube, and Google to identify videos for point-of-care US-guided paracentesis, thoracentesis, and central venous catheterization. Videos were evaluated with a 5-point scale assessing the global educational value and a checklist based on consensus guidelines for competencies in point-of-care US-guided procedures.

RESULTS

For point-of-care US-guided central venous catheterization, 12 videos were found, with an average global educational value score ± SD of 4.5 ± 0.7. Indications to abort the procedure were discussed in only 3 videos. Five videos described the indications and contraindications for performing central venous catheterization. For point-of-care US-guided thoracentesis, 8 videos were identified, with an average global educational value score of 4.0 ± 0.9. Only one video discussed indications to abort the procedure, and 3 videos discussed sterile technique. For point-of-care US-guided paracentesis, 7 videos were included, with an average global educational value score of 4.1 ± 0.9. Only 1 video discussed indications to abort the procedure, and 2 described the location of the inferior epigastric artery.

CONCLUSIONS

The 27 videos reviewed contained good-quality general instruction. However, we noted a lack of safety-related information in most of the available videos. Further development of resources is required to teach internal medicine trainees skills that focus on the safety of point-of-care US guidance.

A systematic review of assessments for procedural skills in physiotherapy education / Assessment von prozeduralen Fähigkeiten in der physiotherapeutischen Ausbildung: Ein systematischer Review

Introduction

Learning of procedural skills is important in the education of physiotherapists. It is the aim of physiotherapy degree programmes that graduates are able to practice selected procedures safely and efficiently. Procedural competency is threatened by an increasing and diverse amount of procedures that are incorporated in university curricula. As a consequence, less time is available for the learning of each specific procedure. Incorrectly performed procedures in physiotherapy might be ineffective and may result in injuries to patients and physiotherapists. The aim of this review was to synthesise relevant literature systematically to appraise current knowledge relating to assessments for procedural skills in physiotherapy education.

Method

A systematic search strategy was developed to screen five relevant databases (CINAHL, Cochrane Central, SportDISCUS, ERIC and MEDLINE) for eligible studies. The included assessments were evaluated for evidence of their reliability and validity.

Results

The search of electronic databases identified 560 potential records. Seven studies were included into this systematic review. The studies reported eight assessments of procedural skills. Six of the assessments were designed for a specific procedure and two assessments were considered for the evaluation of more than one procedure. Evidence to support the measurement properties of the assessment was not available for all categories.

Discussion

It was not possible to recommend a single assessment of procedural skills in physiotherapy education following this systematic review. There is a need for further development of new assessments to allow valid and reliable assessments of the broad spectrum of physiotherapeutic practice

Simulation-Based Training for Residents in the Management of Acute Agitation: A Cluster Randomized Controlled Trial

OBJECTIVE

Simulations are used extensively in medicine to train clinicians to manage high-risk situations. However, to our knowledge, no studies have determined whether this is an effective means of teaching residents to manage acutely agitated patients. This study aimed to determine whether simulation-based training in the management of acute agitation improves resident knowledge and performance, as compared to didactic-based instruction.

METHODS

Following a standard lecture on the management of agitated patients, first-year psychiatry residents were randomized (in clusters of three to four residents) to either the intervention (n = 15) or control arm (n = 11). Residents in the intervention arm then received simulation-based training on the management of acute agitation using a scenario with an agitated standardized patient. Those in the control arm received simulation-based training on a clinical topic unrelated to the management of agitation using a scenario with a non-agitated standardized patient who had suffered a fall. Baseline confidence and knowledge were assessed using pre-intervention self-assessment questionnaires and open-ended clinical case vignettes. Efficacy of the intervention as a teaching tool was assessed with post-intervention open-ended clinical case vignettes and videotaped simulation-based assessment, using a different scenario of an agitated standardized patient.

RESULTS

Residents who received the agitation simulation-based training showed significantly greater improvement in knowledge (intervention = 3.0 vs. control = 0.3, p = 0.007, Cohen's d = 1.2) and performance (intervention = 39.6 vs control = 32.5, p = 0.001, Cohen's d = 1.6). Change in self-perceived confidence did not differ significantly between groups.

CONCLUSIONS

In this study, simulation-based training appeared to be more effective at teaching knowledge and skills necessary for the management of acutely agitated patients, as compared to didactic-based instruction alone. Subjective evaluations of confidence in these skills did not improve significantly compared to controls, corroborating the need for using objective outcome measures when assessing simulation-based training.

Flipped clinical training: a structured training method for undergraduates in complete denture prosthesis

PURPOSE

To design and implement flipped clinical training for undergraduate dental students in removable complete denture treatment and predict its effectiveness by comparing the assessment results of students trained by flipped and traditional methods.

METHODS

Flipped training was designed by shifting the learning from clinics to learning center (phase I) and by preserving the practice in clinics (phase II). In phase I, student-faculty interactive session was arranged to recap prior knowledge. This is followed by a display of audio synchronized video demonstration of the procedure in a repeatable way and subsequent display of possible errors that may occur in treatment with guidelines to overcome such errors. In phase II, live demonstration of the procedure was given. Students were asked to treat three patients under instructor's supervision. The summative assessment was conducted by applying the same checklist criterion and rubric scoring used for the traditional method. Assessment results of three batches of students trained by flipped method (study group) and three traditionally trained previous batches (control group) were taken for comparison by chi-square test.

RESULTS

The sum of traditionally trained three batch students who prepared acceptable dentures (score: 2 and 3) and unacceptable dentures (score: 1) was compared with the same of flipped trained three batch students revealed that the number of students who demonstrated competency by preparing acceptable dentures was higher for flipped training (χ²=30.996 with p<0.001).

CONCLUSION

The results reveal the supremacy of flipped training in enhancing students competency and hence recommended for training various clinical procedures.

Developing a Tool to Assess Placement of Central Venous Catheters in Pediatrics Patients

BACKGROUND

Pediatric critical care medicine requires the acquisition of procedural skills, but to date no criteria exist for assessing trainee competence in central venous catheter (CVC) insertion.

OBJECTIVE

The goal of this study was to create and demonstrate validity evidence for a direct observation tool for assessing CVC insertion.

METHODS

Ten experts used the modified Delphi technique to create a 15-item direct observation tool to assess 5 scripted and filmed simulated scenarios of CVC placement. The scenarios were hosted on a dedicated website from March to May 2013, and respondents recruited by e-mail completed the observation tool in real time while watching the scenarios. The goal was to obtain 50 respondents and a total of 250 scenario ratings.

RESULTS

A total of 49 pediatrics intensive care faculty physicians (6.3% of 780 potential subjects) responded and generated 188 scenario observations. Of these, 150 (79.8%) were recorded from participants who scored 4 or more on the 5 scenarios. The tool correctly identified the expected reference standard in 96.8% of assessments with an interrater agreement kappa (standard error) = 0.94 (0.07) and receiver operating characteristic = 0.97 (95% CI 0.94-0.99).

CONCLUSIONS

This direct observation assessment tool for central venous catheterization demonstrates excellent performance in identifying the reference standard with a high degree of interrater reliability. These assessments support a validity construct for a pediatric critical care medicine faculty member to assess a provider placing a CVC in a pediatrics patient.

Constructing and evaluating a validity argument for the final-year ward simulation exercise

The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student's level of clinical performance. The results are included in each student's portfolio as one source of evidence of the student's capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010-2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students' clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students' clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible.

Diagnosing technical competence in six bedside procedures: comparing checklists and a global rating scale in the assessment of resident performance

PURPOSE

To compare procedure-specific checklists and a global rating scale in assessing technical competence.

METHOD

Two trained raters used procedure-specific checklists and a global rating scale to independently evaluate 218 video-recorded performances of six bedside procedures of varying complexity for technical competence. The procedures were completed by 47 residents participating in a formative simulation-based objective structured clinical examination at the University of Calgary in 2011. Pass/fail (competent/not competent) decisions were based on an overall global assessment item on the global rating scale. Raters provided written comments on performances they deemed not competent. Checklist minimum passing levels were set using traditional standard-setting methods.

RESULTS

For each procedure, the global rating scale demonstrated higher internal reliability and lower interrater reliability than the checklist. However, interrater reliability was almost perfect for decisions on competence using the overall global assessment (Kappa range: 0.84-1.00). Clinically significant procedural errors were most often cited as reasons for ratings of not competent. Using checklist scores to diagnose competence demonstrated acceptable discrimination: The area under the curve ranged from 0.84 (95% CI 0.72-0.97) to 0.93 (95% CI 0.82-1.00). Checklist minimum passing levels demonstrated high sensitivity but low specificity for diagnosing competence.

CONCLUSIONS

Assessment using a global rating scale may be superior to assessment using a checklist for evaluation of technical competence. Traditional standard-setting methods may establish checklist cut scores with too-low specificity: High checklist scores did not rule out incompetence. The role of clinically significant errors in determining procedural competence should be further evaluated.

Measuring competence in central venous catheterization: a systematic-review

Objectives

Central venous catheterization is a complex procedural skill. This study evaluates existing published tools on this procedure and systematically summarizes key competencies for the assessment of this technical skill.

Methods

Using a previously published meta-analysis search strategy, we conducted a systematic review of published assessment tools using the electronic databases PubMed, MEDLINE, Education Resource Information Center (ERIC), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Excerpta Medica, and Cochrane Central Register of Controlled Trials. Two independent investigators abstracted information on tool content and characteristics.

Results

Twenty-five studies were identified assessing a total of 147 items. Tools used for assessment at the bedside (clinical tools) had a higher % of items representing “preparation” and “infection control” than tools used for assessment using simulation (67 ± 26% vs. 32 ± 26%; p = 0.003 for “preparation” and 60 ± 41% vs. 11 ± 17%; p = 0.002 for “infection control”, respectively). Simulation tools had a higher % of items on “procedural competence” than clinical tools (60 ± 36% vs. 17 ± 15%; p = 0.002). Items in the domains of “Team working” and “Communication and working with the patient” were frequently under-represented.

Conclusion

This study presents a comprehensive review of existing checklist items for the assessment of central venous catheterization. Although many key competencies are currently assessed by existing published tools, some domains may be under-represented by select tools.

Electronic supplementary material

The online version of this article (doi:10.1186/2193-1801-3-33) contains supplementary material, which is available to authorized users.

Evidence-based consensus on the insertion of central venous access devices: definition of minimal requirements for training

There is a lack of standard minimal requirements for the training of insertion techniques and maintenance of central venous access devices (CVADs). An international evidence-based consensus task force was established through the World Congress of Vascular Access (WoCoVA) to provide definitions and recommendations for training and insertion of CVADs. Medical literature published from February 1971 to April 2012 regarding 'central vascular access', 'training', 'competency', 'simulation', and 'ultrasound' was reviewed on Pubmed, BioMed Central, ScienceDirect, and Scopus databases. The GRADE and the GRADE-RAND methods were utilized to develop recommendations. Out of 156 papers initially identified, 83 papers described training for central vascular access placement. Sixteen recommendations are proposed by this task force, each with an evidence level, degree of consensus, and recommendation grade. These recommendations suggest central venous access education include didactic or web-based teaching with insertion procedure, infection prevention, complications, care, and maintenance of devices, along with laboratory models and tools for simulation practice incorporating ultrasound. Clinical competence should be determined by observation during clinical practice using a global rating scale rather than by the number of procedures performed. Ensuring safe insertion and management of central venous devices requires standardized education, simulation practice, and supervised insertions.

Development and validation of tools for assessing use of personal protective equipment in health care

BACKGROUND

Incorrect use of personal protective equipment (PPE) may lead to the spread of infectious agents among health care workers and patients. Although novel education programs show promise, there is no standard evaluation for the competencies developed during training.

METHODS

A Delphi methodology was used in which checklist and global rating items for evaluating the performance of PPE skills involving gloves, gowns, masks, eye protection, and hand hygiene were generated and iteratively distributed to a panel of experts. The panel rated the importance of each item until agreement was reached, and the relevant items were used to form the Tools for Assessment of PPE Skills (TAPS), comprising 3 checklist sections (hand hygiene, donning, and doffing) and a global rating scale. Newly trained and experienced PPE users participated in experiments to evaluate the reliability, construct validity, and responsiveness of TAPS.

RESULTS

TAPS demonstrated interobserver reliability, and its global rating scale differentiated the performance of newly trained users and experienced users and was sensitive to changes in performance over time.

CONCLUSIONS

Pending further validation studies, the TAPS may facilitate the development and evaluation of educational programs to support learning and retention of PPE skills, leading to enhanced patient and health care worker safety.

Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation

The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of CVC insertion by 34 first-year medical residents were reviewed by two independent, trained evaluators. Each evaluator used three assessment tools: a ten-item checklist, a 21-item checklist, and a nine-item global rating scale. Exploratory principal component analysis of the global rating scale revealed two factors, accounting for 84.1% of the variance: technical ability and safety. The two checklist scores correlated positively with the weighted factor score on technical ability (0.49 [95% CI 0.17-0.71] for the 10-item checklist; 0.43 [95% CI 0.10-0.67] for the 21-item checklist) and negatively with the weighted factor score on safety (-0.17 [95% CI -0.48-0.18] for the 10-item checklist; -0.13 [95% CI -0.45-0.22] for the 21-item checklist). A checklist score of <80% was strong indication of incompetence. However, a high checklist score did not preclude incompetence. Ratings using the global rating scale identified an additional 11 candidates (32%) who were deemed incompetent despite scoring >80% on both checklists. All these candidates committed serious errors. In conclusion, the practice of universal adoption of checklists as the preferred method of assessment of procedural skills should be questioned. The inclusion of global rating scales should be considered.

Assessing fitness-to-practice of overseas-trained health practitioners by Australian registration & accreditation bodies

BACKGROUND

Assessment of fitness-to-practice of health professionals trained overseas and who wish to practice in Australia is undertaken by a range of organisations. These organisations conduct assessments using a range of methods. However there is very little published about how these organisations conduct their assessments. The purpose of the current paper is to investigate the methods of assessment used by these organisations and the issues associated with conducting these assessments.

METHODS

A series of semi-structured interviews was undertaken with a variety of organisations who undertake assessments of overseas-trained health professionals who wish to practice in Australia. Content analysis of the interviews was used to identify themes and patterns.

RESULTS

Four themes were generated from the content analysis of the interviews: (1) assessing; (2) process; (3) examiners; and (4) cost-efficiency. The themes were interconnected and each theme also had a number of sub-themes.

CONCLUSIONS

The organisations who participated in the present study used a range of assessment methods to assess overseas trained health professionals. These organisations also highlighted a number of issues, particularly related to examiners and process issues, pre- and post-assessment. Organisations demonstrated an appreciation for ongoing review of their assessment processes and incorporating evidence from the literature to inform their processes and assessment development.

Evidence based post graduate training. A systematic review of reviews based on the WFME quality framework

BACKGROUND

A framework for high quality in post graduate training has been defined by the World Federation of Medical Education (WFME). The objective of this paper is to perform a systematic review of reviews to find current evidence regarding aspects of quality of post graduate training and to organise the results following the 9 areas of the WFME framework.

METHODS

The systematic literature review was conducted in 2009 in Medline Ovid, EMBASE, ERIC and RDRB databases from 1995 onward. The reviews were selected by two independent researchers and a quality appraisal was based on the SIGN tool.

RESULTS

31 reviews met inclusion criteria. The majority of the reviews provided information about the training process (WFME area 2), the assessment of trainees (WFME area 3) and the trainees (WFME area 4). One review covered the area 8 'governance and administration'. No review was found in relation to the mission and outcomes, the evaluation of the training process and the continuous renewal (respectively areas 1, 7 and 9 of the WFME framework).

CONCLUSIONS

The majority of the reviews provided information about the training process, the assessment of trainees and the trainees. Indicators used for quality assessment purposes of post graduate training should be based on this evidence but further research is needed for some areas in particular to assess the quality of the training process.

Using checklists in a gross anatomy laboratory improves learning outcomes and dissection quality

Checklists have been widely used in the aviation industry ever since aircraft operations became more complex than any single pilot could reasonably remember. More recently, checklists have found their way into medicine, where cognitive function can be compromised by stress and fatigue. The use of checklists in medical education has rarely been reported, especially in the basic sciences. We explored whether the use of a checklist in the gross anatomy laboratory would improve learning outcomes, dissection quality, and students' satisfaction in the first-year Human Structure didactic block at Mayo Medical School. During the second half of a seven-week anatomy course, dissection teams were each day given a hardcopy checklist of the structures to be identified during that day's dissection. The first half of the course was considered the control, as students did not receive any checklists to utilize during dissection. The measured outcomes were scored on four practice practical examinations and four dissection quality assessments, two each from the first half (control) and second half of the course. A student satisfaction survey was distributed at the end of the course. Examination and dissection scores were analyzed for correlations between practice practical examination score and checklist use. Our data suggest that a daily hardcopy list of anatomical structures for active use in the gross anatomy laboratory increases practice practical examination scores and dissection quality. Students recommend the use of these checklists in future anatomy courses.

HIV prevention among African American youth: how well have evidence-based interventions addressed key theoretical constructs?

Certain constructs are demonstrated in the research literature to be related to HIV risk behaviors among African American adolescents. This study examines how well these constructs are addressed in evidence-based interventions (EBIs) developed for this population. A literature review on variables for sexual risk behaviors among African American adolescents was undertaken. Simultaneously, a review was conducted of the contents of HIV-prevention EBIs. To facilitate comparison, findings from both were organized into constructs from prominent behavior change theories. Analysis showed that environmental conditions and perceived norms were frequently associated with sexual risk behaviors in the literature, while EBIs devoted considerable time to knowledge, skills, and self-efficacy. Findings imply that (a) EBIs might be complemented with activities that focus on important constructs identified in the literature and (b) researchers should better assess the relationship between skill development and HIV risk behaviors. Implications for practice and research are discussed.

Checking the checklist: a content analysis of expert- and evidence-based case-specific checklist items

OBJECTIVES

Research on objective structured clinical examinations (OSCEs) is extensive. However, relatively little has been written on the development of case-specific checklists on history taking and physical examination. Background information on the development of these checklists is a key element of the assessment of their content validity. Usually, expert panels are involved in the development of checklists. The objective of this study is to compare expert-based items on OSCE checklists with evidence-based items identified in the literature.

METHODS

Evidence-based items covering both history taking and physical examination for specific clinical problems and diseases were identified in the literature. Items on nine expert-based checklists for OSCE examination stations were evaluated by comparing them with items identified in the literature. The data were grouped into three categories: (i) expert-based items; (ii) evidence-based items, and (iii) evidence-based items with a specific measure of their relevance.

RESULTS

Out of 227 expert-based items, 58 (26%) were not found in the literature. Of 388 evidence-based items found in the literature, 219 (56%) were not included in the expert-based checklists. Of these 219 items, 82 (37%) had a specific measure of importance, such as an odds ratio for a diagnosis, making that diagnosis more or less probable.

CONCLUSIONS

Expert-based, case-specific checklist items developed for OSCE stations do not coincide with evidence-based items identified in the literature. Further research is needed to ascertain what this inconsistency means for test validity.

How to objectively classify residents based on their psychomotor laparoscopic skills?

In minimally invasive surgery (MIS), a surgeon needs to acquire a certain level of basic psychomotor MIS skills to perform surgery safely. Evaluation of those skills is a major impediment. Although various assessment methods have been introduced, none of them came as a superior. Three aspects of assessing psychomotor MIS skills are discussed here: (i) advantages and disadvantages of currently available assessment methods, (ii) methods to objectively classify residents according to their level of psychomotor skills, and (iii) factors that influence psychomotor MIS skills. Motion analysis has a potential to be the means to deal with assessment of psychomotor skills. Together with classification methods (e.g. linear discriminant analysis), motion analysis provides an aid in deciding whether a resident is ready to move to the next level of training. Presence of factors that influence psychomotor MIS skills results in a high need for standardisation of valid tasks and setups used for the assessment of MIS skills.

Patient safety: Part II. Opportunities for improvement in patient safety

The quality movement in medicine has prompted a shift from a "name, shame, blame" approach to medical errors to one in which each error is regarded as an opportunity to prevent future patient harm. This new culture of patient safety requires the involvement of all members of the health care team and learned skill sets related to quality improvement. A root cause analysis identifies the sources of medical errors, allowing system changes that reduce the risk. In large organizations, sentinel events and signals prompt chart reviews and reduce the reliance on voluntary reporting. Failure mode analysis prompts the development of safety nets in the case of a system failure. The second part of this two-part series on patient safety examines how the culture of patient safety is taught, how medical errors and threats to patient safety can be identified, and how engineering tools can be used to improve patient care. It also examines efforts to measure clinical effectiveness and outcomes in the practice of medicine.

LEARNING OBJECTIVES

After completing this learning activity, participants should be able to improve patient safety through an understanding of both the beneficial and adverse consequences of quality reporting, apply safety engineering tools to the practice of dermatology, and be able to establish a quality improvement plan for a dermatologic practice.

Development of a tool to support holistic generic assessment of clinical procedure skills

CONTEXT

The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic.

OBJECTIVES

This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills.

METHODS

We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff. Results In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was > 70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station.

CONCLUSIONS

This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).