Global rating scale for the assessment of paramedic clinical competence

Video support for prehospital stroke consultation: implications for system design and clinical implementation from prehospital simulations

BACKGROUND

Video consultations between hospital-based neurologists and Emergency Medical Services (EMS) have potential to increase precision of decisions regarding stroke patient assessment, management and transport. In this study we explored the use of real-time video streaming for neurologist-EMS consultation from the ambulance, using highly realistic full-scale prehospital simulations including role-play between on-scene EMS teams, simulated patients (actors), and neurologists specialized in stroke and reperfusion located at the remote regional stroke center.

METHODS

Video streams from three angles were used for collaborative assessment of stroke using the National Institutes of Health Stroke Scale (NIHSS) to assess symptoms affecting patient's legs, arms, language, and facial expressions. The aim of the assessment was to determine appropriate management and transport destination based on the combination of geographical location and severity of stroke symptoms. Two realistic patient scenarios were created, with severe and moderate stroke symptoms, respectively. Each scenario was simulated using a neurologist acting as stroke patient and an ambulance team performing patient assessment. Four ambulance teams with two nurses each all performed both scenarios, for a total of eight cases. All scenarios were video recorded using handheld and fixed cameras. The audio from the video consultations was transcribed. Each team participated in a semi-structured interview, and neurologists and actors were also interviewed. Interviews were audio recorded and transcribed.

RESULTS

Analysis of video-recordings and post-interviews (n = 7) show a more thorough prehospital patient assessment, but longer total on-scene time, compared to a baseline scenario not using video consultation. Both ambulance nurses and neurologists deem that video consultation has potential to provide improved precision of assessment of stroke patients. Interviews verify the system design effectiveness and suggest minor modifications.

CONCLUSIONS

The results indicate potential patient benefit based on a more effective assessment of the patient's condition, which could lead to increased precision in decisions and more patients receiving optimal care. The findings outline requirements for pilot implementation and future clinical tests.

Improving Pediatric Procedural Skills for EMS Clinicians: A Longitudinal Simulation-Based Curriculum with Novel, Remote, First-Person-View Video-Based Outcome Measurement

OBJECTIVE

Emergency medical services (EMS) clinicians are expected to provide expert care to all patients, but face obstacles in maintaining skillsets required in the care of critically ill or injured children. The objectives of this study were to describe and assess the effectiveness of a pediatric-focused, simulation-based, procedural training program for EMS clinicians, delivered on-site by a pediatric simulation education team. We also describe a novel, remote, asynchronous performance outcome measurement system using first-person-view video review.

METHODS

This was a prospective study of simulation-based training and procedural outcomes. The study population involved EMS clinicians at three fire-based EMS agencies stratified as urban, suburban, and rural sites. The primary outcome was performance of intraosseous catheterization (IO), bag-valve-mask ventilation (BVM), and supraglottic device placement (SGD), measured across three time points. Secondary outcomes were identification of differences across EMS agencies and participant survey responses.

RESULTS

We obtained video data from 122 clinicians, totaling 561 videos, with survey response rates of 89.0-91.3%. Pre-intervention scores were high: least-square means (95% confident-intervals) 9.5 (8.9, 10.2) for IO; 9.6 (9.3, 9.9) for BVM; and 11.6 (10.9, 12.2) for SGD. There was significant improvement post-intervention: 11.5 (10.7, 12.3) for IO; 11.0 (10.7, 11.4) for BVM; and 13.6 (12.8, 14.4) for SGD. Improvement was maintained at follow-up after a median of 9.5 months: 10.5 (9.8, 11.2) for IO; 10.2 (9.9, 10.6) for BVM; and 12.4 (11.7, 13.1) for SGD. There were no statistical differences between sites. Of survey respondents, half had not cared for a critically ill or injured child in at least a year, the vast majority had not had hands-on pediatric training in over 6 months, and the majority felt that training should occur at least every 6 months.

CONCLUSIONS

Our pediatric-focused, simulation-based procedural training program was associated with improvement and maintenance of high-baseline procedural performance for EMS clinicians over the study period. Findings were consistent across sites. Remote assessment was feasible. Participant surveys emphasized a desire for more pediatric-focused training and highlighted the low frequency of clinical exposure to procedures potentially needed in the care of critically ill or injured pediatric patients.

Perceptions of paramedic educators on assessments used in the first year of a paramedic programme: a qualitative exploration

BACKGROUND

In Ireland, there are currently three educational institutions (recognised institutions- RIs) providing paramedic programmes, accredited by the regulator, the Pre-Hospital Emergency Care Council (PHECC). Each RI assesses their students in-house, and in order to acquire a licence to practice, students must also pass summative assessments provided by PHECC. These assessments comprise multiple choice questions, short answer questions and skills assessments. The objective of this study was to explore the perceptions and experiences of paramedic educators of assessments used within their institution and by the regulator to provide insights that could inform the future design of paramedic assessments.

METHODS

A qualitative study with an interpretivist approach and purposive sampling strategy was performed. Semi-structured interviews were conducted with educators from one RI, across their three sites. Data were analysed using an inductive approach to thematic analysis.

RESULTS

Four major themes were identified in the data: improving assessments by enhancing authenticity, modifying the current process of assessment, aligning the PHECC and RI examinations and opportunities to use assessment as learning.

CONCLUSIONS

This study identifies perceived deficits and opportunities in the assessments currently used for paramedic students and ways in which these assessments could be improved. While participants were relatively content with their own RI assessments, they identified ways to improve both the RI and PHECC assessments. Modifying some of the current methods could be a useful first step. In particular, assessments used by PHECC could be improved by reflecting 'real-world' practice. The inclusion of additional assessment methods by PHECC, a continuous assessment process or devolvement of the entire assessment suite, to the RI/University has the potential to enhance assessments, particularly summative assessments, for paramedic students.

Assessing the Utility of a Quality-of-Care Assessment Tool Used in Assessing Comprehensive Care Services Provided by Community Health Workers in South Africa

Background

Few studies exist on the tools for assessing quality-of-care of community health worker (CHW) who provide comprehensive care, and for available tools, evidence on the utility is scanty. We aimed to assess the utility components of a previously-reported quality-of-care assessment tool developed for summative assessment in South Africa.

Methods

In two provinces, we used ratings by 21 CHWs and three team leaders in two primary health care facilities per province regarding whether the tool covered everything that happens during their household visits and whether they were happy to be assessed using the tool (acceptability and face validity), to derive agreement index (≥85%, otherwise the tool had to be revised). A panel of six experts quantitatively validated 11 items of the tool (content validity). Content validity index (CVI), of individual items (I-CVI) or entire scale (S-CVI), should be >80% (excellent). For the inter-rater reliability (IRR), we determined agreement between paired observers' assigned quality-of-care messages and communication scores during 18 CHW household visits (nine households per site). Bland and Altman plots and multilevel model analysis, for clustered data, were used to assess IRR.

Results

In all four CHW and team leader sites, agreement index was ≥85%, except for whether they were happy to be assessed using the tool, where it was <85% in one facility. The I-CVI of the 11 items in the tool ranged between 0.83 and 1.00. For the S-CVI, all six experts agreed on relevancy (universal agreement) in eight of 11 items (0.72) whereas the average of I-CVIs, was 0.95. The Bland-Altman plot limit of agreements between paired observes were −0.18 to 0.44 and −0.30 to 0.44 (messages score); and −0.22 to 0.45 and −0.28 to 0.40 (communication score). Multilevel modeling revealed an estimated reliability of 0.77 (messages score) and 0.14 (communication score).

Conclusion

The quality-of-care assessment tool has a high face and content validity. IRR was substantial for quality-of-care messages but not for communication score. This suggests that the tool may only be useful in the formative assessment of CHWs. Such assessment can provide the basis for reflection and discussion on CHW performance and lead to change.

Markers of cognitive skills important for team leaders in emergency medical services: a qualitative interview study

Background

Cognitive skills and other non-technical skills are key for emergency medical services (EMS); however, there have been a limited number of their markers identified. This study aims to identify markers of cognitive skills—situation awareness and decision making—important for team leaders in EMS. The focus is on any markers of cognitive skills that are associated with quality and safety at EMS work.

Method

In-depth semi-structured interviews were conducted with 20 EMS team leaders (10 EMS physician team leaders and 10 paramedic team leaders) and analysed by the structured approach utilising the known framework of cognitive skill elements.

Results

The data analysis revealed a set of 50 markers falling into elements of situation awareness (gathering information, interpreting information, anticipating states), elements of decision making (identifying options, implementing decisions, re-evaluating decisions), and an additional cognitive element (maintaining standards). These markers represented cognitive processes, acts, and communications, therefore, some of them can be observable and others rather unobservable. The identified markers were not too specific, applicable mostly in various challenging situations with patients’ medical problems and in EMS team leaders working in ground ambulances in urban and rural/remote areas.

Conclusion

The findings provide a better understanding of EMS team leaders’ cognitive skills, and can aid in the development of assessment and training tools suited particularly to them.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12873-022-00629-1.

Core Components of a Pediatric Critical Care Transport Communication Curriculum: A Modified Delphi Approach

OBJECTIVE

Interfacility transport of critically ill infants and children is an essential part of the care of children in the United States. However, there is tremendous variation in how transports are coordinated and performed. Pediatric critical care medicine (PCCM) fellows have differing experiences in their fellowships, and there is no standardized way of training medical command for the transport process. The aim of this study was to use a consensus-building process to establish core components of a PCCM transport curriculum focused on communication.

METHODS

A national group of experts in transport medicine rated 51 total possible topics for their importance to include in a fellowship curriculum. Three rounds of surveys were completed.

RESULTS

Fifty-two of 372 invitees (14%) participated in round 1. Consensus was reached to include 15 items in a PCCM curriculum. Twenty of 52 (38%) experts completed round 2, reaching consensus on 2 additional items. Seventeen of 20 (85%) experts completed round 3. No additional items reached consensus.

CONCLUSION

Experts reached consensus on 17 core components to include in a PCCM fellowship transport communication curriculum. This curriculum could likely be adapted to train providers from different disciplines in the transport process.

A Prehospital Emergency Psychiatric Unit in an Ambulance Care Service from the Perspective of Prehospital Emergency Nurses: A Qualitative Study

The prevalence of mental illness is steadily increasing, and ambulance teams frequently attend cases with suspected mental illness. A pilot project, Psychiatric Emergency Response Team (PAP), was carried out in which a prehospital emergency nurse (PEN) was accompanied by a psychiatric specialist nurse in the assessment of individuals with mental illness. The aim of the present study was to evaluate a prehospital emergency psychiatric unit from the perspective of PENs. A qualitative method using content analysis was applied. Seven senior PENs who had worked for 1 year in a prehospital psychiatric ambulance unit were interviewed individually. The analysis resulted in one main theme, "Transition from limited care and insufficient competence to improved and adequate care for psychiatric patients in ambulance care". This emerged from six subcategories: inter-professional development, access to patient records, theambulance vehicle,non-conveyed patients, cooperation with the police and meetings with patients and next of kin. In conclusion, these results suggest that in ambulance care in general, there is a lack of knowledge and skills about mental illnesses and initial care options. The PAP concept opened new avenues for the care of patients with mental illness, which the PENs described very positively as being helpful and valuable.

Nurse preceptors' experience-based strategies for supporting learning in the ambulance service-A combined focus group and dyadic interview study

Ambulance service organizations worldwide report about an expanding professional role, responsibilities and scope of practice for ambulance clinicians, resulting in discussions concerning educational design and desired professional competencies. To face the contemporary demands in ambulance care, non-technical skills are advocated and clinical practice considered fundamental for the development of these abilities. However, there is very little research concerning educational strategies for supporting the desired competencies for novice registered nurses in the ambulance service.

AIM

To describe and explore nurse preceptors' experience-based strategies for supporting registered nurses learning in the ambulance service.

DESIGN

The study had an inductive and data-driven approach, guided by phenomenological epistemology.

METHODS

Twenty-seven Swedish nurse preceptors were interviewed in three focus groups and four dyadic interviews between October 2019 and April 2020. The data were analysed with reflexive thematic analysis.

FINDINGS

The nurse preceptors use several learning strategies, focusing on a socialization process and a clinical competence process, intertwined during clinical practice to support the development of a situated professional identity and a clinical decision-making competence. Supportive structures facilitate a progressive learning strategy when addressing desired skills and cognitive abilities in teamwork processes and clinical judgement.

CONCLUSION

Supporting novice clinicians, prior to and during clinical practice in the ambulance service, should include medical assessment skills, situation awareness and processes for effective teamwork. Further, novice clinicians need to develop complex cognitive abilities to deal with the dynamic nature of decision-making in ambulance care.

IMPACT

The study findings show contextual strategies, previously not described and desired competencies when supporting learning for registered nurses in the ambulance service. A theoretical grounding in episteme, techne, phronesis and situation awareness may guide educators at universities, managers in the ambulance service, preceptors and novice clinicians worldwide in the planning and performance of teaching and learning in the ambulance service.

Principles to Guide the Future of Paramedicine in Canada

Objective: Paramedicine in Canada has experienced significant growth in recent years, which has resulted in a misalignment between existing guiding conceptualizations and how the profession is structured and enacted in practice. As a result, well-established boundaries, directions, and priorities may be poorly aligned with existing frameworks. The objective of this study was to explore emerging and future states of paramedicine in Canada such that guiding principles could be derived. We asked: How should paramedicine be conceptualized and enacted in Canada going forward, and, what might be the necessary enablers? Methods: This study involved in-depth one-on-one semi-structured interviews with Canadian paramedicine thought leaders. We used purposive and snowball sampling strategies to identify potential participants. Interview guide questions were used to stimulate discussion about the future of paramedicine in Canada and suggestions for implementation. We used inductive qualitative content analysis as our analytical approach, informed by a constructivist and interpretivist orientation. Results: Thirty-five key informants from across Canada participated in interviews. Ten themes were identified: (1) prioritizing patients and their communities; (2) providing health care along a health and social continuum; (3) practicing within an integrated health care framework, and partnering across sectors; (4) being socially responsive; (5) enacting professional autonomy; (6) integrating the health of professionals; (7) using quality-based frameworks; (8) enacting intelligent access to and distribution of services; (9) enacting a continuous learning environment; and, (10) being evidence-informed in practice and systems. Six enablers were also identified: shift professional culture and identity, enhance knowledge, promote shared understanding of paramedicine, integrate data environments, leverage advancing technology, advance policy, regulation and legislation. Conclusions: Our results provide a conceptual framework made up of guiding principles and enablers that provide a consolidated lens to advance the paramedicine profession in Canada (and elsewhere as appropriate) while ensuring contextual and regional needs and differences can be accounted for.

Paramedic student encounters with patients during clinical placements: A multi-institutional cohort study

OBJECTIVE

Paramedic students in the US are required to complete clinical placements to gain supervised experience with real patient encounters. Given wide variation in clinical placement practices, an evidence-based approach is needed to guide programs in setting realistic and attainable goals for students. This study's goal was to describe patient encounters and hours logged by paramedic students during clinical placements.

METHODS

A retrospective review of prospectively collected quality assurance data entered by US paramedic students between 2010 and 2014 was conducted. De-identified electronic records entered in the Field Internship Student Data Acquisition Project (FISDAP) Skill Tracker database were included from consenting paramedic students whose records were audited and approved by instructors. Descriptive statistics were calculated.

RESULTS

A total of 10,645 students encountered 2,239,027 patients; most encounters occurred in hospital settings (n = 1,311,967, 59%). The median total number of patient encounters per paramedic student was 206 (142-269) and the median total clinical placement hours per student was 626 (504-752). The median number of team leads per student was 56 (30-84). Students encountered a median of 22 (12-31) pediatric patients, ages 0-12 years, and 181 (126-238) adolescent or adult patients. For pediatric patient encounters, the most common clinical impressions were respiratory distress, other medical complaints, and extremity trauma. Among adult patient encounters, the most common clinical impressions included other medical, trauma, and cardiac conditions.

CONCLUSIONS

US paramedic students experienced a variable range of patient encounter types and volumes. The findings of this study offer an evidence base from which programs can set realistic and attainable clinical placement requirements.

VAD 911: Process Improvement for First Responders Treating Ventricular Assist Device Patients

Ventricular assist devices (VAD) complications including stroke, device failure, gastrointestinal bleeding, and infection all can manifest in the outpatient environment. Often, first responders do not have the knowledge base to respond appropriately. This study conducted an assessment of first responders generalized knowledge and self-reported competency regarding VAD use, provided an education course, and tested for improvement immediately after and 1 month postintervention. Two hundred thirty-six first responders participated from communities with known VAD patients. Responses indicated poor knowledge, experience, and competency. Twenty-one percent had seen a manufacturer training video, 26% had attended prior training, and 38% knew who to contact with a VAD emergency. Generalized knowledge of VAD therapy was poor with majority not understanding use of anticoagulation, when to initiate chest compressions and assessment variations. Self-reported competency was low with 80% of participants reporting unsafe to marginal about VAD awareness. Comparison of 1 month postintervention to preintervention, generalized knowledge improved by a minimum of 38% for each question and competency increased by over 50%. From this study comes the recommendation that VAD centers must include the community first responders as part of their discharge planning for the VAD patient and improve community outreach to ensure appropriate patient outcomes.

Improving the quality of training paramedics by means of cadavers - a pilot study

BACKGROUND

Paramedics are authorised to perform emergency procedures, including trauma assessment according to global standards. The aim of the study was to answer the question whether the use of cadavers in teaching practical competencies to medical rescue students, in the field of trauma assessment, is necessary as a supplement to learning in simulated conditions with the use of mannequins.

METHODS

Research included several stages. The first stage was conduction of classes for 27 students in the field of rapid trauma assessment, in accordance with the guidelines of the International Trauma Life Support. In the second stage, a plan of a test in which students had to perform an analogous procedure of rapid trauma assessment, but with the use of cadavers, human unfixed specimens, was prepared. The Delphi method was used to develop and approve checklists, as well as a scale to assess the global correctness of identification of head, torso and limb injuries by medical rescue students.

RESULTS

The identification rate was 76.54% in the head area, 67.90% in the torso area, while in the limb area it equalled 44.45%. A significant difference in scores, compared to the examination performed on a mannequin, was observed (Wilcoxon = 4.541; p = 0.000). The most difficult to make a correct diagnosis were injuries related to a fracture of the proximal end of the femur and a dislocated wrist (only 18.52% of correct answers). The students highly rated the usefulness of the examination, by awarding it an average of 4.76 points (SD ± 0.56) on the Likert scale (0-5).

CONCLUSIONS

The study shows that the use of cadavers to teach practical competencies in the field of trauma assessment to medical rescue students can be an effective supplement to simulated learning. Students could feel the difference between the human body and the mannequin. More research is needed to assess whether realistic simulation translates into objective endpoints, such as the effectiveness of diagnosis in the examination of trauma patients. However, it should be remembered that the introduction of this teaching method is expensive and requires adequate base, as well as the compliance with a number of formal requirements.

The compatibility principle: on philosophies in the assessment of clinical competence

The array of different philosophical positions underlying contemporary views on competence, assessment strategies and justification have led to advances in assessment science. Challenges may arise when these philosophical positions are not considered in assessment design. These can include (a) a logical incompatibility leading to varied or difficult interpretations of assessment results, (b) an "anything goes" approach, and (c) uncertainty regarding when and in what context various philosophical positions are appropriate. We propose a compatibility principle that recognizes that different philosophical positions commit assessors/assessment researchers to particular ideas, assumptions and commitments, and applies ta logic of philosophically-informed, assessment-based inquiry. Assessment is optimized when its underlying philosophical position produces congruent, aligned and coherent views on constructs, assessment strategies, justification and their interpretations. As a way forward we argue that (a) there can and should be variability in the philosophical positions used in assessment, and these should be clearly articulated to promote understanding of assumptions and make sense of justifications; (b) we focus on developing the merits, boundaries and relationships within and/or between philosophical positions in assessment; (c) we examine a core set of principles related to the role and relevance of philosophical positions; (d) we elaborate strategies and criteria to delineate compatible from incompatible; and (f) we articulate a need to broaden knowledge/competencies related to these issues. The broadened use of philosophical positions in assessment in the health professions affect the "state of play" and can undermine assessment programs. This may be overcome with attention to the alignment between underlying assumptions/commitments.

Translation and further validation of a global rating scale for the assessment of clinical competence in prehospital emergency care

Global rating scales are useful to assess clinical competence at a general level based on specific word dimensions. The aim of this study was to translate and culturally adapt the Paramedic Global Rating Scale, and to contribute validity evidence and instrument usefulness in training results and clinical competence assessments of students undergoing training to become ambulance nurses and paramedics at Swedish and Finnish universities. The study included translation, expert review and inter-rater reliability (IRR) tests. The scale was translated and culturally adapted to clinical and educational settings in both countries. A content validity index (CVI) was calculated using eight experts. IRR tests were performed with five registered nurses working as university lecturers, and with six clinicians working as ambulance nurses. They individually rated the same simulated ambulance assignment. Based on the ratings IRR was calculated with intra-class correlation (ICC). The scale showed excellent CVI for items and scale. The ICC indicated substantial agreement in the group of lecturers and a high degree of agreement in the group of clinicians. This study provides validity evidence for a Swedish version of the scale, supporting its use in measuring clinical competence among students undergoing training to become ambulance nurses and paramedics.

Cognitive skills of emergency medical services crew members: a literature review

Background

Situation awareness and decision making, listed in non-technical skills taxonomies, are critical for effective and safe performance in high-risk professions. These cognitive skills and their behavioral markers have been studied less in emergency medical services (EMS) crew members. This paper aims to review the existing literature and identify important aspects and behavioral markers of situation awareness and decision making in EMS crew members – those who work in the role of prehospital emergency care providers – and to synthesize findings as a basis for developing a rating and training tool.

Method

The search for relevant articles was conducted using electronic databases, reference lists of relevant reviews and included articles and personal collection of articles. The selection process based on the PRISMA statement yielded a total of 30 articles that met the eligibility criteria. Their findings were qualitatively synthesized using the structured approach, informed by the already known structure: situation awareness and its elements (gathering information, interpreting information, anticipating future states), decision making and its elements (generating and considering options, selecting and implementing an option, reviewing outcome/decision). Moreover, the element of maintaining standards also emerged as highly relevant for cognitive skills.

Results

This review found an increased research interest in the non-technical cognitive skills of EMS crew members. The majority of included articles’ research designs were qualitative, then mixed, Delphi, and quantitative. It revealed several specifics of cognitive skills, such as EMS crew members need to holistically assess a wide range of cues and information, to make various health- and safety-related decisions and take EMS standards into account. However, there was only a limited number of observable markers of cognitive skills, such as acts and verbalizations, that could be considered as examples of good behavior. In addition, findings indicate a lack of articles focused on mass-casualty incidents and the interconnection of cognitive skills with other non-technical and medical skills.

Conclusion

Further research is needed to get a more comprehensive view of behavioral markers of cognitive skills and to develop a rating and training tool to improve EMS crew members’ cognitive performance.

The Effect of Foregrounding Intended Use on Observers' Ratings and Comments in the Assessment of Clinical Competence

PURPOSE

Some educational programs have adopted the premise that the same assessment can serve both formative and summative goals; however, how observers understand and integrate the intended uses of assessment may affect the way they execute the assessment task. The objective of this study was to explore the effect of foregrounding a different intended use (formative vs summative learner assessment) on observer contributions (ratings and comments).

METHOD

In this randomized, experimental, between-groups, mixed-methods study (May-September 2017), participants observed 3 prerecorded clinical performances under formative or summative assessment conditions. Participants rated performances using a global rating tool and provided comments. Participants were then asked to reconsider their ratings from the alternative perspective (from which they were originally blinded). They received the opportunity to alter their ratings and comments and to provide rationales for their decision to change or preserve their original ratings and comments. Outcomes included participant-observers' comments, ratings, changes to each, and stated rationales for changing or preserving their contributions.

RESULTS

Foregrounding different intended uses of assessment data for participant-observers did not result in differences in ratings, number or type of comments (both emphasized evaluative over constructive statements), or the ability to differentiate among performances. After adopting the alternative perspective, participant-observers made only small changes in ratings or comments. Participant-observers reported that they engage in the process in an evaluative manner despite different intended uses.

CONCLUSIONS

Foregrounding different intended uses for assessments did not result in significant systematic differences in the assessment data generated. Observers provided more evaluative than constructive statements overall, regardless of the intended use of the assessment. Future research is needed to explore whether these results hold in social/workplace-based contexts and how they might affect learners.

The Effects of Integrated IT Support on the Prehospital Stroke Process: Results from a Realistic Experiment

Stroke is a serious condition and the stroke chain of care is a complex. The present study aims to explore the impact of a computerised decision support system (CDSS) for the prehospital stroke process, with focus on work processes and performance. The study used an exploratory approach with a randomised controlled crossover design in a realistic contextualised simulation experiment. The study compared clinical performance among 11 emergency medical services (EMS) teams of 22 EMS clinicians using (1) a computerised decision support system (CDSS) and (2) their usual paper-based process support. Data collection consisted of video recordings, postquestionnaires and post-interviews, and data were analysed using a combination of qualitative and quantitative approaches. In this experiment, using a CDSS improved patient assessment, decision making and compliance to process recommendations. Minimal impact of the CDSS was found on EMS clinicians' self-efficacy, suggesting that even though the system was found to be cumbersome to use it did not have any negative effects on self-efficacy. Negative effects of the CDSS include increased on-scene time and a cognitive burden of using the system, affecting patient interaction and collaboration with team members. The CDSS's overall process advantage to the prehospital stroke process is assumed to lead to a prehospital care that is both safer and of higher quality. The key to user acceptance of a system such as this CDSS is the relative advantages of improved documentation process and the resulting patient journal. This could improve the overall prehospital stroke process.

Consensus Core Point-of-care Ultrasound Applications for Pediatric Emergency Medicine Training

BACKGROUND

Pediatric emergency medicine (PEM) physicians have variably incorporated point-of-care ultrasound (POCUS) into their practice. Prior guidelines describe the scope of POCUS practice for PEM physicians; however, consensus does not yet exist about which applications should be prioritized and taught as fundamental skills for PEM trainees. Members of the PEM POCUS Network (P2Network) conducted a consensus-building process to determine which applications to incorporate into PEM fellowship training.

METHODS

A multinational group of experts in PEM POCUS was recruited from the P2Network and greater PEM POCUS community if they met the following criteria: performed over 1,000 POCUS scans and had at least 3 years of experience teaching POCUS to PEM fellows, were a local academic POCUS leader, or completed a formal PEM POCUS fellowship. Experts rated 60 possible PEM POCUS applications for their importance to include as part of a PEM fellowship curriculum using a modified Delphi consensus-building technique.

RESULTS

In round 1, 66 of 92 (72%) participants responded to an e-mail survey of which 48 met expert criteria and completed the survey. Consensus was reached to include 18 items in a PEM fellowship curriculum and to exclude two items. The 40 remaining items and seven additional items were considered in round 2. Thirty-seven of 48 (77%) experts completed round 2 reaching consensus to include three more items and exclude five. The remaining 39 items did not reach consensus for inclusion or exclusion.

CONCLUSION

Experts reached consensus on 21 core POCUS applications to include in PEM fellowship curricula.

Asking for Less and Getting More: The Impact of Broadening a Rater's Focus in Formative Assessment

PURPOSE

There may be unintended consequences of broadening the competencies across which health professions trainees are assessed. This study was conducted to determine whether such broadening influences the formative guidance assessors provide to trainees and to test whether sequential collection of competency-specific assessment can overcome setbacks of simultaneous collection.

METHOD

A randomized between-subjects experimental design, conducted in Toronto and Halifax, Canada, in 2016-2017 with paramedic educators experienced in observing/rating, in which observers' focus was manipulated. In the simultaneous condition, participants rated four unscripted (i.e., spontaneously generated) clinical performances using a six-dimension global rating scale and provided feedback. In three sequential conditions, participants were asked to rate the same performances and provide feedback but for only two of the six dimensions. Participants from these conditions were randomly merged to create a "full score" and set of feedback statements for each candidate.

RESULTS

Eighty-seven raters completed the study; 23 in the simultaneous condition and 21 or 22 for each pair of dimensions in the sequential conditions. After randomly merging participants, there were 21 "full scores" in the sequential condition. Compared with the sequential condition, participants in the simultaneous condition demonstrated reductions in the amount of unique feedback provided, increased likelihood of ignoring some dimensions of performance, lessened variety of feedback, and reduced reliability.

CONCLUSIONS

Sequential or distributed assessment strategies in which raters are asked to focus on less may provide more effective assessment by overcoming the unintended consequences of asking raters to spread their attention thinly over many dimensions of competence.

Applying Kane's validity framework to a simulation based assessment of clinical competence

Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies.

Ambulance Nurses' Competence and Perception of Competence in Prehospital Trauma Care

Introduction

We focus on trauma care conducted in the context of a simulated traumatic event. This is in this study defined as a four-meter fall onto a hard surface, resulting in severe injuries to extremities in the form of bilateral open femur fractures, an open tibia fracture, and a closed pelvic fracture, all fractures bleeding extensively.

Methods

The simulated trauma care competence of 63 ambulance nurses in prehospital emergency care was quantitatively evaluated along with their perception of their sufficiency. Data was collected by means of simulated trauma care and a questionnaire.

Results

Life-saving interventions were not consistently performed. Time to perform interventions could be considered long due to the life-threatening situation. In comparison, the ambulance nurses' perception of the sufficiency of their theoretical and practical knowledge and skills for trauma care scored high. In contrast, the perception of having sufficient ethical training for trauma care scored low.

Discussion

This study suggests there is no guarantee that the ambulance nurses' perception of theoretical and practical knowledge and skill level corresponds with their performed knowledge and skill. The ambulance nurses rated themselves having sufficient theoretical and practical knowledge and skills while the score of trauma care can be considered quite low.

i-Assess: Evaluating the impact of electronic data capture for OSCE

INTRODUCTION

Tablet-based assessments offer benefits over scannable-paper assessments; however, there is little known about the impact to the variability of assessment scores.

METHODS

Two studies were conducted to evaluate changes in rating technology. Rating modality (paper vs tablets) was manipulated between candidates (Study 1) and within candidates (Study 2). Average scores were analyzed using repeated measures ANOVA, Cronbach's alpha and generalizability theory. Post-hoc analyses included a Rasch analysis and McDonald's omega.

RESULTS

Study 1 revealed a main effect of modality (F (1,152) = 25.06, p < 0.01). Average tablet-based scores were higher, (3.39/5, 95% CI = 3.28 to 3.51), compared with average scan-sheet scores (3.00/5, 95% CI = 2.90 to 3.11). Study 2 also revealed a main effect of modality (F (1, 88) = 15.64, p < 0.01), however, the difference was reduced to 2% with higher scan-sheet scores (3.36, 95% CI = 3.30 to 3.42) compared with tablet scores (3.27, 95% CI = 3.21 to 3.33). Internal consistency (alpha and omega) remained high (>0.8) and inter-station reliability remained constant (0.3). Rasch analyses showed no relationship between station difficulty and rating modality.

DISCUSSION

Analyses of average scores may be misleading without an understanding of internal consistency and overall reliability of scores. Although updating to tablet-based forms did not result in systematic variations in scores, routine analyses ensured accurate interpretation of the variability of assessment scores.

CONCLUSION

This study demonstrates the importance of ongoing program evaluation and data analysis.

We need support! A Delphi study about desirable support during the first year in the emergency medical service

Background

New and inexperienced emergency medical service (EMS) professionals lack important experience. To prevent medical errors and improve retention there is an urgent need to identify ways to support new professionals during their first year in the EMS.

Methods

A purposeful sample and snowball technique was used and generated a panel of 32 registered nurses with 12–48 months of EMS experience. A Delphi technique in four rounds was used. Telephone interviews were undertaken in round one to identify what desirable support professionals new to the EMS desire during their first year. Content analysis of the transcribed interviews yielded items which were developed into a questionnaire. The experts graded each item in terms of perceived importance on a 5-graded likert scale. Consensus level was set at 75%. Items which reached consensus were removed from questionnaires used in subsequent rounds.

Results

Desirable support was categorized into eight areas: Support from practical skills exercises, support from theoretical knowledge, support from experiences based knowledge, theoretical support, support from an introduction period, support from colleagues and work environment, support from management and organization and other support. The experts agree on the level of importance on 64 of a total of 70 items regarding desirable support. One item was considered not important, graded 1 or 2, 63 items were considered important, graded 4 or 5.

Conclusion

Even with extensive formal competence the EMS context poses challenges where a wide variety of desirable forms of support is needed. Support structures should address both personal and professional levels and be EMS context oriented.

Informing a Canadian paramedic profile: framing concepts, roles and crosscutting themes

BACKGROUND

Paramedicine is a rapidly evolving health profession with increasing responsibilities and contributions to healthcare. This rapid growth has left the profession with unclear professional and clinical boundaries. Existing defining frameworks may no longer align with the practice of paramedicine or expectations of the public. The purpose of this study was to explore the roles paramedics in Canada are to embody and that align with or support the rapid and ongoing evolution of the profession.

METHODS

We used a concurrent mixed methods study design involving a focused discourse analysis (i.e., analysis of language used to describe paramedics and paramedicine) of peer reviewed and grey literature (Phase 1) and in-depth one-on-one semi-structured interviews with key informants in Canadian paramedicine (Phase 2). Data from both methods were analyzed simultaneously throughout and after being merged using inductive thematic analysis.

RESULTS

Saturation was reached after 99 national and international grey and peer reviewed publications and 20 in depth interviews with stakeholders representing six provinces, seven different service/agency types, 11 operational roles and seven provider roles. After merging both data sets three framing concepts, six roles and four crosscutting themes emerged that may be significant to both present-day practice and aspirational. Framing concepts, which provide context, include variable contexts or practice, embedded relationships and a health and social continuum. Roles include clinician, health and social advocate, team member, educator, professional and reflective practitioner. Crosscutting themes including patient safety, adaptability, compassion and communication appear to exist in all roles.

CONCLUSIONS

The paramedic profession is experiencing a shift that appears to deviate or at least place a tension on traditional views or models of practice. Underlying and evolving notions of practice are resulting in intended or actual clinical and professional boundaries that may require the profession to re-think how it is defined and/or shaped. Until these framing concepts, roles and crosscutting themes are fully understood, tested and operationalized, tensions between guiding frameworks and actual or intended practice may persist.

Delivering a sustainable trauma management training programme tailored for low-resource settings in East, Central and Southern African countries using a cascading course model

BACKGROUND

Injuries cause five million deaths and 279 Disability Adjusted Life Years (DALYS) each year worldwide. The COSECSA Oxford Orthopaedic Link (COOL) is a multi-country partnership programme that has delivered training in trauma management to nine sub-Saharan countries across a wide-cadre of health-workers using a model of "primary" courses delivered by UK instructors, followed by "cascading" courses led by local faculty. This study examines the impact on knowledge and clinical confidence among health-workers, and compares the performance of "cascading" and "primary" courses delivered in low-resource settings.

METHODS

Data was collated from 1030 candidates (119 Clinical Officers, 540 Doctors, 260 Nurses and 111 Medical Students) trained over 28 courses (9 "primary" and 19 "cascading" courses) in nine sub-Saharan countries between 2012 and 2013. Knowledge and clinical confidence of candidates were assessed using pre- and post-course MCQs and confidence matrix rating of clinical scenarios. Changes were measured in relation to co-variants of gender, job roles and primary versus cascading courses. Multivariate regression modelling and cost analysis was performed to examine the impact of primary versus cascading courses on candidates' performance.

FINDINGS

There was a significant improvement in knowledge (58% to 77%, p<0.05) and clinical confidence (68% to 90%, p<0.05) post-course. "Non-doctors" demonstrated a greater improvement in knowledge (22%) and confidence (24%) following the course (p<0.05). The degree of improvement of MCQ scores differed significantly, with the cascading courses (21%) outperforming primary courses (15%) (p<0.002). This is further supported by multivariate regression modelling where cascading courses are a strong predictor for improvement in MCQ scores (Coef=4.83, p<0.05).

INTERPRETATION

Trauma management training of health-workers plays a pivotal role in tackling the ever-growing trauma burden in Africa. Our study suggests cascading PTC courses may be an effective model in delivering trauma training in low-resource settings, however further studies are required to determine its efficacy in improving clinical competence and retention of knowledge and skills in the long term.

Applying hospital evidence to paramedicine: issues of indirectness, validity and knowledge translation

The practice of emergency medicine (EM) has been intertwined with emergency medical services (EMS) for more than 40 years. In this commentary, we explore the practice of translating hospital based evidence into the prehospital setting. We will challenge both EMS and EM dogma-bringing hospital care to patients in the field is not always better. In providing examples of therapies championed in hospitals that have failed to translate into the field, we will discuss the unique prehospital environment, and why evidence from the hospital setting cannot necessarily be translated to the prehospital field. Paramedicine is maturing so that the capability now exists to conduct practice-specific research that can inform best practices. Before translation from the hospital environment is implemented, evidence must be evaluated by people with expertise in three domains: critical appraisal, EM, and EMS. Scientific evidence should be assessed for: quality and bias; directness, generalizability, and validity to the EMS population; effect size and anticipated benefit from prehospital application; feasibility (including economic evaluation, human resource availability in the mobile environment); and patient and provider safety.

Selecting and Simplifying: Rater Performance and Behavior When Considering Multiple Competencies

THEORY

Assessment of clinical competence is a complex cognitive task with many mental demands often imposed on raters unintentionally. We were interested in whether this burden might contribute to well-described limitations in assessment judgments. In this study we examine the effect on indicators of rating quality of asking raters to (a) consider multiple competencies and (b) attend to multiple issues. In addition, we explored the cognitive strategies raters engage when asked to consider multiple competencies simultaneously.

HYPOTHESES

We hypothesized that indications of rating quality (e.g., interrater reliability) would decline as the number of dimensions raters are expected to consider increases.

METHOD

Experienced faculty examiners rated prerecorded clinical performances within a 2 (number of dimensions) × 2 (presence of distracting task) × 3 (number of videos) factorial design. Half of the participants were asked to rate 7 dimensions of performance (7D), and half were asked to rate only 2 (2D). The second factor involved the requirement (or lack thereof) to rate the performance of actors participating in the simulation. We calculated the interrater reliability of the scores assigned and counted the number of relevant behaviors participants identified as informing their ratings. Second, we analyzed data from semistructured posttask interviews to explore the rater strategies associated with rating under conditions designed to broaden raters' focus.

RESULTS

Generalizability analyses revealed that the 2D group achieved higher interrater reliability relative to the 7D group (G = .56 and .42, respectively, when the average of 10 raters is calculated). The requirement to complete an additional rating task did not have an effect. Using the 2 dimensions common to both groups, an analysis of variance revealed that participants who were asked to rate only 2 dimensions identified more behaviors of relevance to the focal dimensions than those asked to rate 7 dimensions: procedural skill = 36.2%, 95% confidence interval (CI) [32.5, 40.0] versus 23.5%, 95% CI [20.8, 26.3], respectively; history gathering = 38.6%, 95% CI [33.5, 42.9] versus 24.0%, 95% CI [21.1, 26.9], respectively; ps < .05. During posttask interviews, raters identified many sources of cognitive load and idiosyncratic cognitive strategies used to reduce cognitive load during the rating task.

CONCLUSIONS

As intrinsic rating demands increase, indicators of rating quality decline. The strategies that raters engage when asked to rate many dimensions simultaneously are varied and appear to yield idiosyncratic efforts to reduce cognitive effort, which may affect the degree to which raters make judgments based on comparable information.

On the Assessment of Paramedic Competence: A Narrative Review with Practice Implications

Introduction

Paramedicine is experiencing significant growth in scope of practice, autonomy, and role in the health care system. Despite clinical governance models, the degree to which paramedicine ultimately can be safe and effective will be dependent on the individuals the profession deems suited to practice. This creates an imperative for those responsible for these decisions to ensure that assessments of paramedic competence are indeed accurate, trustworthy, and defensible.

Purpose

The purpose of this study was to explore and synthesize relevant theoretical foundations and literature informing best practices in performance-based assessment (PBA) of competence, as it might be applied to paramedicine, for design or evaluation of assessment programs.

Methods

A narrative review methodology was applied to focus intentionally, but broadly, on purpose relevant, theoretically derived research that could inform assessment protocols in paramedicine. Primary and secondary studies from a number of health professions that contributed to and informed best practices related to the assessment of paramedic clinical competence were included and synthesized.

Results

Multiple conceptual frameworks, psychometric requirements, and emerging lines of research are forwarded. Seventeen practice implications are derived to promote understanding as well as best practices and evaluation criteria for educators, employers, and/or licensing/certifying bodies when considering the assessment of paramedic competence.

Conclusions

The assessment of paramedic competence is a complex process requiring an understanding, appreciation for, and integration of conceptual and psychometric principles. The field of PBA is advancing rapidly with numerous opportunities for research.

TavaresW,BoetS.On the assessment of paramedic competence: a narrative review with practice implications.Prehosp Disaster Med.2016;31(1):64–73.

The use of global rating scales for OSCEs in veterinary medicine

OSCEs (Objective Structured Clinical Examinations) are widely used in health professions to assess clinical skills competence. Raters use standardized binary checklists (CL) or multi-dimensional global rating scales (GRS) to score candidates performing specific tasks. This study assessed the reliability of CL and GRS scores in the assessment of veterinary students, and is the first study to demonstrate the reliability of GRS within veterinary medical education. Twelve raters from two different schools (6 from University of Calgary [UCVM] and 6 from Royal (Dick) School of Veterinary Studies [R(D)SVS] were asked to score 12 students (6 from each school). All raters assessed all students (video recordings) during 4 OSCE stations (bovine haltering, gowning and gloving, equine bandaging and skin suturing). Raters scored students using a CL, followed by the GRS. Novice raters (6 R(D)SVS) were assessed independently of expert raters (6 UCVM). Generalizability theory (G theory), analysis of variance (ANOVA) and t-tests were used to determine the reliability of rater scores, assess any between school differences (by student, by rater), and determine if there were differences between CL and GRS scores. There was no significant difference in rater performance with use of the CL or the GRS. Scores from the CL were significantly higher than scores from the GRS. The reliability of checklist scores were .42 and .76 for novice and expert raters respectively. The reliability of the global rating scale scores were .7 and .86 for novice and expert raters respectively. A decision study (D-study) showed that once trained using CL, GRS could be utilized to reliably score clinical skills in veterinary medicine with both novice and experienced raters.

Enhancing paramedics procedural skills using a cadaveric model

BACKGROUND

Paramedic education has evolved in recent times from vocational post-employment to tertiary pre-employment supplemented by clinical placement. Simulation is advocated as a means of transferring learned skills to clinical practice. Sole reliance of simulation learning using mannequin-based models may not be sufficient to prepare students for variance in human anatomy. In 2012, we trialled the use of fresh frozen human cadavers to supplement undergraduate paramedic procedural skill training. The purpose of this study is to evaluate whether cadaveric training is an effective adjunct to mannequin simulation and clinical placement.

METHODS

A multi-method approach was adopted. The first step involved a Delphi methodology to formulate and validate the evaluation instrument. The instrument comprised of knowledge-based MCQs, Likert for self-evaluation of procedural skills and behaviours, and open answer. The second step involved a pre-post evaluation of the 2013 cadaveric training.

RESULTS

One hundred and fourteen students attended the workshop and 96 evaluations were included in the analysis, representing a return rate of 84%. There was statistically significant improved anatomical knowledge after the workshop. Students' self-rated confidence in performing procedural skills on real patients improved significantly after the workshop: inserting laryngeal mask (MD 0.667), oropharyngeal (MD 0.198) and nasopharyngeal (MD 0.600) airways, performing Bag-Valve-Mask (MD 0.379), double (MD 0.344) and triple (MD 0.326,) airway manoeuvre, doing 12-lead electrocardiography (MD 0.729), using laryngoscope (MD 0.726), using Magill® forceps to remove foreign body (MD 0.632), attempting thoracocentesis (MD 1.240), and putting on a traction splint (MD 0.865). The students commented that the workshop provided context to their theoretical knowledge and that they gained an appreciation of the differences in normal tissue variation. Following engagement in/ completion of the workshop, students were more aware of their own clinical and non-clinical competencies.

CONCLUSIONS

The paramedic profession has evolved beyond patient transport with minimal intervention to providing comprehensive both emergency and non-emergency medical care. With limited availability of clinical placements for undergraduate paramedic training, there is an increasing demand on universities to provide suitable alternatives. Our findings suggested that cadaveric training using fresh frozen cadavers provides an effective adjunct to simulated learning and clinical placements.