Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

INTRODUCTION

Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance.

MATERIALS AND METHODS

A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments.

RESULTS

The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority.

CONCLUSIONS

This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

Exploring the impact of professional soccer substitute players on physical and technical performance

BACKGROUND

Substitutions are generally used to promote the match performance of the whole team. This study aimed to analyze the performance of substitute players and explore the performance difference among substitute players, completed players, and replaced players across each position.

METHODS

Chinese Super Soccer League (CSL) matches in the season 2018 including 5871 individual observation from 395 professional soccer players were analyzed by establishing linear mixed models to quantify the performance difference among substitute players (SP) (n = 1,071), entire match players (EMP) (n = 3,454), and replaced players (RP) (n = 1,346), and then separately for each position (central defenders, fullbacks, central midfielders, wide midfielders, and attackers).

RESULTS

The results show SP display higher high intensity distance and sprint distance significantly (p < 0.05) relative to playing time than RP and EMP. SP in offensive positions (attackers, wide midfielders) showed significantly higher (p < 0.05) passing and organizing performance such as passes, ball control, short passes, and long passes than RP or EMP. The scoring performances of central midfielders of SP including goals, shots, and shots on target are significantly higher (p < 0.05) than RP or EMP. Central defenders of SP showed higher shot blocks and pass blocks (p < 0.05) while lower passing and organizing performance (p < 0.05).

CONCLUSION

Depending on different playing positions, substitute players could indeed improve physical and technical performance related to scoring, passing, and defending as offensive substitute players can boost organizing performance and substitute defenders enhance defending performance. These could help coaches better understand substitute players' influence on match performance and optimize the substitution tactic.

Influence of the embedded participant on learners' performance during high-fidelity simulation sessions in healthcare

BACKGROUND

The embedded participant (EP) plays a key role during a full scale/high-fidelity simulation (HFS) session. He/she is expected to guide the learner towards the achievement of the educational objectives of the scenario. However, his/her influence on learners' performance stands undetermined and this effect remains briefly addressed in the literature. This study primarily aims to assess whether the EP could have an influence on the performance of the learner during a HFS scenario. The secondary aim was to establish an inventory of the EP practices in France.

METHODS

This retrospective study was conducted in Lyon Claude Bernard University Health Simulation Centre (France). Anaesthesia and critical care residents in postgraduate years 1 to 5 who were scheduled for their HFS sessions during the 2016 to 2021 academic years were included. Two investigators independently evaluated the resident performance regarding both technical and non-technical skills from video recordings. In addition, a nationwide survey was sent out by email through the networks of the Francophone Healthcare Simulation Society (SoFraSimS, Société Francophone de Simulation en Santé) to collect information on EP practices in French-speaking Simulation centres.

RESULTS

From a total of 344 HFS videos analysed, a cohort of 19 experienced EPs was identified. The EPs had an influence on the technical and non-technical performances of the learners. The 147 responses to the survey showed that predefined rules of EP good practice existed in only 36% of the simulation centres and 65% of respondents believed that specific EP training would be justified.

CONCLUSION

The EP can exert an influence on the performance of the learners during HFS. For acting as an EP, a great variability of practices and a lack of training are reported by professionals working in simulation centres. These results suggest that more attention must be paid to EP training and attitudes during simulation, especially if summative simulations are extensively developed.

Multiparametric Quantitative Imaging Biomarker as a Multivariate Descriptor of Health: A Roadmap

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

Intraoperative dynamics of workflow disruptions and surgeons' technical performance failures: insights from a simulated operating room

INTRODUCTION

Flow disruptions (FD) in the operating room (OR) have been found to adversely affect the levels of stress and cognitive workload of the surgical team. It has been concluded that frequent disruptions also lead to impaired technical performance and subsequently pose a risk to patient safety. However, respective studies are scarce. We therefore aimed to determine if surgical performance failures increase after disruptive events during a complete surgical intervention.

METHODS

We set up a mixed-reality-based OR simulation study within a full-team scenario. Eleven orthopaedic surgeons performed a vertebroplasty procedure from incision to closure. Simulations were audio- and videotaped and key surgical instrument movements were automatically tracked to determine performance failures, i.e. injury of critical tissue. Flow disruptions were identified through retrospective video observation and evaluated according to duration, severity, source, and initiation. We applied a multilevel binary logistic regression model to determine the relationship between FDs and technical performance failures. For this purpose, we compared FDs in one-minute intervals before performance failures with intervals without subsequent performance failures.

RESULTS

Average simulation duration was 30:02 min (SD = 10:48 min). In 11 simulated cases, 114 flow disruption events were observed with a mean hourly rate of 20.4 (SD = 5.6) and substantial variation across FD sources. Overall, 53 performance failures were recorded. We observed no relationship between FDs and likelihood of immediate performance failures: Adjusted odds ratio = 1.03 (95% CI 0.46-2.30). Likewise, no evidence could be found for different source types of FDs.

CONCLUSION

Our study advances previous methodological approaches through the utilisation of a mixed-reality simulation environment, automated surgical performance assessments, and expert-rated observations of FD events. Our data do not support the common assumption that FDs adversely affect technical performance. Yet, future studies should focus on the determining factors, mechanisms, and dynamics underlying our findings.

Harvesting system sustainability in Mediterranean olive cultivation: Other principal cultivar

In the olive production sector, which is increasingly expanding beyond the borders of the Mediterranean basin, harvesting is the most demanding phase, from both an economic and organisational point of view. Traditional olive orchards are still predominant, with centuries-old and large plants, and are characterised by the gradual ripening of drupes and irregular planting patterns. Even though the structural conversion of these olive orchards into more modern cultivations may be difficult owing to their historical, monumental, and landscaping importance, as well as the existing legal restrictions, supporting a "modernisation" process aimed at mechanising the main farming operations remains a priority. Technological innovation is, therefore, a primary objective for Mediterranean olive growing, as well as for the enhancement of its strengths. The present study aimed at assessing different olive harvesting sites, considering the technical, economic, and environmental aspects, to develop a better version of the "olive harvesting database". The applied methodology, also called the "modular approach", represents a useful tool to apply in unitary process assessment to obtain a comprehensive database of diverse agricultural operations. Eight olive harvesting systems were compared: six highly mechanised scenarios, one based on mechanical-aided harvesting, and the final one based on fully manual harvesting. The mechanised systems obtained a better performance in terms of working capacity, as only 3.5 h ha^-1 were needed to harvest 12 t using a self-propelled trunk shaker. In addition, the economic results revealed that mechanical harvesting, diversely from manual or aided harvesting, is the only way to decrease production costs. From an environmental point of view, manual and mechanical-aided harvesting showed the best performance in terms of impact per hour. However, using the mass-based unit (1 kg of harvested olives), the results were the opposite and this could be very relevant for the ecoprofile of olive oil.

Immersive simulation training at 6-week intervals for 1 year and multidisciplinary team performance scores: a randomized controlled trial of simulation training for life-threatening pediatric emergencies

OBJECTIVES

To demonstrate an effect of 1 year of training using immersive simulations repeated every 6 weeks versus every 6 months to improve the performance of multidisciplinary teams (MDTs) working with children in lifethreatening situations.

MATERIAL AND METHODS

Randomized controlled trial in 12 MDTs of emergency responders in France. Each MDT consisted of 4 persons: a physician, a resident, a nurse, and the ambulance driver. Six MDTs participated in 9 different high-fidelity simulations of pediatric shock over the course of a year. Six control MDTs were presented with 3 of the experimental group's simulations at 3 time points (starting point, 6 months, and 1 year). Technical performance was assessed with the Team Average Performance Assessment Scale (TAPAS) and an intraosseous (IO) access performance scale. Nontechnical performance assessment instruments were the Clinical Teamwork Scale (CTS) and, for leadership, the Behavioral Assessment Tool (BAT). Progress over time was analyzed by comparing the 2 groups during the 3 simulations they experienced in common.

RESULTS

Performance scores rose significantly over the study period in the experimental group (P=.01 for the TAPAS score, P=.008 for IO access, P=.03 for the CTS score, and P=.02 for the BAT score) but did not change in the control group (P=.46 for TAPAS, P=.55 for IO access, P=.62 for CTS, and P=.58 for BAT). All mean (SD) scores were higher in the experimental group than in the control group in the last session: TAPAS, 55.8 ± 6.3 vs 31.2 ± 10.3, P=.01; IO access, 91.7 ± 8.0 vs 62.9 ± 16.2, P=.01; CTS, 63.2 ± 9.3 vs 47.2 ± 13.1, P=.03; and BAT, 72.8 ± 5.1 vs 51.2 ± 14.3, P=.01). The 6-month assessment showed significant between-group differences on 2 technical performance measures (P=.02 for TAPAS and P=.03 for IO access); the experimental group's scores were higher. We also observed close correlations between the performance of the leader and the group on both nontechnical (rho > 0.9) and technical (rho > 0.7) assessments.

CONCLUSION

Simulation-based training should be repeated more than 3 times per year. Our findings suggest the advisability of repeating simulations of infrequent, high-risk scenarios every 6 weeks to improve all performance scores and guarantee acceptable technical and nontechnical performance throughout the year.

Brain activation during laparoscopic tasks in high- and low-performing medical students: a pilot fMRI study

BACKGROUND

Up to 20% of medical students are unable to reach competency in laparoscopic surgery. It is unknown whether these difficulties arise from heterogeneity in neurological functioning across individuals. We sought to examine the differences in neurological functioning during laparoscopic tasks between high- and low-performing medical students using functional magnetic resonance imaging (fMRI).

METHODS

This prospective cohort study enrolled North American medical students who were within the top 20% and bottom 20% of laparoscopic performers from a previous study. Brain activation was recorded using fMRI while participants performed peg-pointing, intracorporeal knot tying (IKT), and the Pictorial Surface Orientation (PicSOr) test. Brain activation maps were created and areas of activation were compared between groups.

RESULTS

In total, 9/12 high and 9/13 low performers completed the study. High performers completed IKT faster and made more successful knot ties than low performers [standing: 23.5 (5.0) sec vs. 37.6 (18.4) sec, p = 0.03; supine: 23.2 (2.5) sec vs. 72.7 (62.8) sec, p = 0.02; number of successful ties supine, 3 ties vs. 1 tie, p = 0.01]. Low performers showed more brain activation than high performers in the peg-pointing task (q < 0.01), with no activation differences in the IKT task. There were no behavioral differences in the PiCSOr task.

CONCLUSIONS

This study is the first to show differences between low and high performers of laparoscopic tasks at the brain level. This pilot study has shown the feasibility of using fMRI to examine laparoscopic surgical skills. Future studies are needed for further exploration of our initial findings.

Surgical timing after chemoradiotherapy for rectal cancer, analysis of technique (STARRCAT): results of a feasibility multi-centre randomized controlled trial

BACKGROUND

The optimal time of rectal resection after long-course chemoradiotherapy (CRT) remains unclear. A feasibility study was undertaken for a multi-centre randomized controlled trial evaluating the impact of the interval after chemoradiotherapy on the technical complexity of surgery.

METHODS

Patients with rectal cancer were randomized to either a 6- or 12-week interval between CRT and surgery between June 2012 and May 2014 (ISRCTN registration number: 88843062). For blinded technical complexity assessment, the Observational Clinical Human Reliability Analysis technique was used to quantify technical errors enacted within video recordings of operations. Other measured outcomes included resection completeness, specimen quality, radiological down-staging, tumour cell density down-staging and surgeon-reported technical complexity.

RESULTS

Thirty-one patients were enrolled: 15 were randomized to 6 and 16-12 weeks across 7 centres. Fewer eligible patients were identified than had been predicted. Of 23 patients who underwent resection, mean 12.3 errors were observed per case at 6 weeks vs. 10.7 at 12 weeks (p = 0.401). Other measured outcomes were similar between groups.

CONCLUSIONS

The feasibility of measurement of operative performance of rectal cancer surgery as an endpoint was confirmed in this exploratory study. Recruitment of sufficient numbers of patients represented a challenge, and a proportion of patients did not proceed to resection surgery. These results suggest that interval after CRT may not substantially impact upon surgical technical performance.

International assessment practices along the continuum of surgical training

BACKGROUND

The objectives of this study were to assemble an international perspective on (1) current, and (2) ideal technical performance assessment methods, and (3) barriers to their adoption during: selection, in-training, and certification.

METHODS

A questionnaire was distributed to international educational directorates.

RESULTS

Eight of 10 jurisdictions responded. Currently, aptitude tests or simulated tasks are used during selection, observational rating scales during training and nothing is used at certification. Ideally, innate ability should be determined during selection, in-training evaluation reports, and global rating scales used during training, whereas global and procedure-specific rating scales used at the time of certification. Barriers include lack of predictive evidence for use in selection, financial limitations during training, and a combination with respect to certification.

CONCLUSIONS

Identifying current and ideal evaluation methods will prove beneficial to ensure the best assessments of technical performance are chosen for each training time point.

Performance evaluation of a new high throughput Mindray BS-2000M1 clinical chemistry system

OBJECTIVE

To evaluate the photometry technical and analytical performance of a newly launched Mindray BS-2000M1 clinical chemistry system (BS-2000M1).

DESIGN AND METHODS

The photometric parameters were evaluated according to the China Food and Drug Administration (CFDA) Automatic Chemistry Analyzer Guideline. The precision, accuracy, linearity and interference were evaluated according to CLSI protocols EP5-A2, EP9-A2, EP6-A and EP7-A2 respectively. The trueness verification on Ca(2+), Mg(2+), P(-) and Cl(-) was conducted by comparing with the reference methods using fresh samples.

RESULTS

The photometer accuracy, precision, linearity, stability and stray light at 340 nm were acceptable. The within-run coefficients of variation (CVs) ranged from 0.16% to 2.13% and the total CV ranged from 0.64% to 4.12%. A good correlation (R>0.95) of method comparison between BS-2000M1 and our reference system was observed for most of the parameters tested with exception of Ca(2+) (R=0.85), Mg(2+) (R=0.71), P(-) (R=0.96, Slope=0.88), Cl(-) (R=0.93), and ASO (R=0.94, Slope=0.93, intercept=-8.81). The trueness verification on Ca(2+), Mg(2+), P(-) and Cl(-) showed acceptable results on both BS-2000M1 and our reference systems. Linearity study showed acceptable linearity range for all parameters. Significant interferences occurred for some evaluated parameters, but were identical to the manufacturer statement.

CONCLUSIONS

Mindray BS-2000M1 achieved the desirable photometry technical and analytical performance, and is therefore suitable for its utilization in modern clinical laboratories.

Match performance and physical capacity of players in the top three competitive standards of English professional soccer

The aim of this study was to compare the match performance and physical capacity of players in the top three competitive standards of English soccer. Match performance data were collected from players in the FA Premier League (n=190), Championship (n=155) and League 1 (n=366) using a multiple-camera system. In addition, a selection of players from the Premier League (n=56), Championship (n=61) and League 1 (n=32) performed the Yo-Yo intermittent endurance test level 2 (Yo-Yo IE2) to determine physical capacity. Players in League 1 and the Championship performed more (p<.01) high-intensity running than those in the Premier League (Effect Size [ES]: 0.4-1.0). Technical indicators such as pass completion, frequency of forward and total passes, balls received and average touches per possession were 4-39% higher (p<.01) in the Premier League compared to lower standards (ES: 0.3-0.6). Players also covered more (p<.05) high-intensity running when moving down (n=20) from the Premier League to the Championship (ES: 0.4) but not when players moved up (n=18) standards (ES: 0.2). Similar Yo-Yo IE2 test performances were observed in Premier League, Championship and League 1 players (ES: 0.2-0.3). Large magnitude relationships (p<.05) were observed between Yo-Yo IE2 test performances and the total and high-intensity running distance covered in both Championship (r=.56 and .64) and Premier League matches (r=.61 and .54). The data demonstrate that high-intensity running distance was greater in players at lower compared to higher competitive standards despite a similar physical capacity in a subsample of players in each standard. These findings could be associated with technical characteristics inherent to lower standards that require players to tax their physical capacity to a greater extent but additional research is still required to confirm these findings.